QueryMatch: A Query-based Contrastive Learning Framework for Weakly Supervised Visual Grounding

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 14:44:25

Abstract: Visual grounding is a task of locating  the object referred by a natural language description.   To reduce annotation costs, recent researchers are devoted into one-stage weakly supervised methods for visual grounding, which typically adopt the anchor-text matching paradigm. Despite the efficiency, we identify that anchor representations are often noisy and insufficient to describe object information, which inevitably hinders the vision-language alignments.  In this paper, we propose a novel query-based one-stage framework for weakly supervised visual grounding, namely QueryMatch. Different from previous work, QueryMatch represents candidate objects with a set of query features, which  inherently establish accurate one-to-one associations with visual objects.  In this case,  QueryMatch  re-formulates weakly supervised visual grounding as a query-text matching problem, which can be optimized via the query-based contrastive learning.  Based on QueryMatch, we further propose an innovative strategy for effective weakly supervised learning, namely Negative Sample Quality Estimation (NSQE). In particular, NSQE aims to augment negative training samples by actively selecting high-quality  query features. Though this strategy, NSQE can greatly  benefit the weakly supervised learning of QueryMatch. To validate our approach,  we conduct extensive experiments on three benchmark datasets of two grounding tasks, i.e., referring expression comprehension (REC) and segmentation (RES). Experimental results not only show the state-of-art performance of QueryMatch in two tasks, e.g., over +5\% IoU@0.5 on RefCOCO in REC and over +20\% mIOU on RefCOCO in RES, but also confirm the effectiveness of NSQE in weakly supervised learning. 
Source codes are available at~\url{https://anonymous.4open.science/r/QueryMatch-A82C}.

Main paper: 
 related work 2.1 weakly supervised visual grounding: 
 Weakly Supervised Visual Grounding includes two tasks, namely weakly supervised REC and weakly supervised RES. These two tasks have attracted considerable attention from numerous researchers recently.Recent developments in weakly supervised learning for RES can be categorized into two types, depending on whether they utilize text-image pairs as weak supervision. For instance, CTS [7] utilized bounding box annotations for weak supervision, involving a two-stage model training process. Despite achieving promising performance in weakly supervised RES, these methods [7, 17] still incur relatively high annotation costs, particularly on large-scale datasets. Thus, an increasing number of researchers have turned their focus to leverage text-image pairs as a weak supervision signal [15, 22, 38] . For example, Fang Liu et al. employed the CLIP encoder to encode both image and text features. The visual information encoded at the grid level by the image encoder is then matched with textual information, and pseudo labels are generated through a two-stage training process based on these matching results [22] . However, current approaches that depend on weak supervision from text-image pairs demonstrate noticeably inferior performance when compared to other kinds of weakly supervised RES methods.Meanwhile, Many existing methodologies in weakly supervised REC [8, 24-26, 39, 41, 46] draw inspiration from two-stage supervised REC frameworks, formulating the weakly supervised REC task as a region-to-text ranking problem. The primary challenge within such approaches lies in effectively utilizing supervisory information from image-text pairings. The primary techniques utilized are semantic reconstruction [24, 25, 39] and contrastive learning [8, 14, 46] . Recent efforts in the field of weakly supervised REC have focused on enhancing inference speed by transitioning from resource-intensive two-stage detectors to more efficient one-stage models. These models, exemplified by [14] , extract anchor features from pretrained detectors like YOLOv3 [35] and employ anchor-text matching to select target anchors for bounding box decoding. Despite their efficiency, these anchor-based frameworks are hindered by fragmented anchor points' inability to accurately represent object information, leading to challenges in precise vision-language alignments.Based on these, we propose a novel query-text matching framework for one-stage weakly supervised visual grounding. After that, QueryMatch filters out queries with low confidence scores and selects the best-matching one for visual grounding prediction. QueryMatch is weakly supervised with only image-text pairs via a novel query-based contrastive learning paradigm.Contrastive learning focuses on minimizing the distances between positive samples while maximizing those between negative samples in the feature space, thereby facilitating the development of effective feature representations. Recently, numerous studies in the multimodal field have embraced contrastive learning [13, 29, 33] . The negative samples in these studies typically comprise real-world instance features, such as images, and it has been observed that the efficacy of contrastive learning tends to improve with an increase in the number of negative samples.In contrast, the cross-modal contrastive learning framework we propose, based on Query-Text Matching, differentiates itself by employing more granular negative samples compared to those representing real-world entities. For instance, a single image can generate as many as 100 query features. While this approach indeed expands the negative sample pool, it simultaneously introduces a higher degree of informational redundancy within these samples. Notably, we have observed that an excessive increase in the number of negative samples can sometimes lead to a marked decline in performance. One of the key challenges lies in effectively harnessing the rich semantic content these samples offer. To tackle this issue, our research introduces a novel method for estimating the quality of negative samples, aiming to maximize the semantic potential inherent in the negative sample set. As such, it presents innovative contributions to the realm of contrastive learning. 
preliminary: 
 We first revisit the construction of query features in computer vision. In particular, query-based object detection is a popular paradigm in computer vision. Starting from DETR [3] , researchers use a set of learnable vectors to represent candidate objects, also known as queries. These queries are used to interact with image features in different Transformer layers and learn the accurate oneto-one association with objects in images. After that, a decoder layer is adopted to predict masks or bounding boxes based on these queries. During the training phase, the objective of set prediction serves as the loss function. To elaborate, we adopt the architecture of the Mask2former [5] . The features of queries, along with their corresponding position encoding, are treated as learnable vectors, initialized randomly through a standard normal distribution. Subsequently, these vectors undergo sequential processing through the cross-attention layer, self-attention layer, and feed-forward layer within each layer of the Transformer decoder, the process can be formatted as follows:EQUATIONEQUATIONEQUATIONhere, 𝑙 denotes layer index, The set 𝑄 𝑙 ∈ R 𝑁 ×𝐶 represents the queries at the 𝑙-th layer, defined as 𝑄 𝑙 = {𝑞 𝑙 𝑖 } 𝑁 𝑖=1 , where 𝑁 signifies the quantity of queries present in each layer. The terms 𝑋 ′ 𝑙 , X𝑙 , and 𝑋 𝑙 ∈ R 𝑁 ×𝐶 correspond to the outputs of the cross-attention, self-attention, and the feed-forward layers in the 𝑙-th layer of the Transformer decoder, respectively. These outputs are characterized by 𝑁 vectors, each of 𝐶 dimensions. 𝑀 𝑙 -1 ∈ {0, -∞} 𝑁 ×𝐻 𝑙 𝑊 𝑙 denotes the attention mask [5] , which is equal to 0 when the mask prediction of the previous 𝑙-th transformer decoder layer is 1, otherwise it is equal to negative infinity. The spatial resolution of image features at the 𝑙-th layer is represented by 𝐻 𝑙 and 𝑊 𝑙 . The transformed image features at the 𝑙-th layer, 𝐾 𝐼 𝑙 and 𝑉 𝐼 𝑙 ∈ R 𝐻 𝑙 𝑊 𝑙 ×𝐶 are under transformation 𝑓 𝐾 and 𝑓 𝑉 respectively. Furthermore, the position encoding is denoted as 𝑃, leading to the equations 𝑄 𝑙 = 𝑓 𝑄 (𝑋 𝑙 -1 + 𝑃), Q𝑙 = 𝑓 Q (𝑋 ′ 𝑙 + 𝑃), and K𝑙 = 𝑓 K (𝑋 ′ 𝑙 + 𝑃), with the exception of V𝑙 = 𝑓 V (𝑋 ′ 𝑙 ). Here, all 𝑓 . notations stand for linear transformations. Additionally, LN signifies LayerNorm. Considering a Transformer decoder comprising 𝐿 layers, the final processed set of queries is denoted as 𝑄 = LN(𝑋 𝐿 ).After the Transformer decoder processes the queries, the network can decode the grounding information embedded in the corresponding query to obtain the result. This process can be defined as:EQUATIONwhere 𝑟 𝑖 denotes the grounding result corresponding to index 𝑖, and 𝑞 𝑖 represents the associated query feature. The term Head is used to refer to the layers within the network that are responsible for decoding grounding information from the queries. 
querymatch 4.1 problem definition: 
 Given an image 𝐼 and a textual expression 𝑇 , Visual Grounding aims to locate target objects through bounding boxes or masks. In the context of weakly supervised visual grounding, the text and the corresponding target grounding are unknown during the training stage, posing significant challenges.Drawing upon the one-to-one correspondence observed between visual grounding results and query features in numerous models utilizing query features for visual grounding [5, 6, 9, 12] , and considering the effectiveness of the query feature derived through the Transformer's cross-attention and self-attention mechanisms in capturing intricate relationships between objects in the image, we propose to reformulate the weakly supervised visual grounding problem as a query-text matching problem, which is defined byEQUATIONHere, 𝑞 * symbolizes the optimal query, and 𝑄 represents the set of queries that have been processed by the transformer decoder within the query-based segmentation model. The notation 𝜙 (•) denotes the mapping module. As delineated in the Section 3, the identification of the optimal query 𝑞 * is pivotal. Once 𝑞 * is determined, it is possible to decode the query features using subsequent visual grounding layers. This process leads to the derivation of the optimal visual grounding results, expressed as 𝑟 * = Head(𝑞 * ).The architecture of the QueryMatch framework is depicted in Fig. 2 . Prior to the alignment of text with query features, QueryMatch initially selects a subset from the pool of queries to be matched.Query-based visual grounding model such as DETR [3] and Mask2former [5] usually have many queries do not contribute to predicting the target. To reduce the complexity of cross-modal matching and to economize on matching efforts, the selection is based on the confidence score of each query feature. More precisely, a selected threshold 𝑂 is established, and only the top 𝑂 queries with the highest confidence scores are chosen, which is defined byEQUATIONwhere 𝑄 𝑠 denotes the queries selected based on confidence, and sort conf refers to the process of descending sorting according to the confidence scores of the query features.Similar to the image-contrastive learning framework, e.g.,, CLIP [33] , QueryMatch likewise projects both visual and textual features into a shared semantic space, facilitating the learning of vision-language alignment.To enable the model to learn how to identify the optimal query that aligns with the text through weak supervision information, thereby achieving visual grounding, we introduce a query-based contrastive learning framework. Specifically, we begin by encoding features of the input images and text to obtain visual features and text features, denoted as 𝑣 ∈ R 𝐻 ×𝑊 ×𝐶 and 𝑡 ∈ R 𝐶 , respectively. The query interacts with the visual features via the transformer decoder, resulting in query features containing the necessary semantic information for visual grounding, denoted as 𝑄 ∈ R 𝑁 ×𝐶 . We then perform confidence-based selection on 𝑄 to obtain 𝑄 𝑠 ∈ R 𝐿×𝐶 , where 𝑄 𝑠 = {𝑞 𝑖 } 𝑂 𝑖=1 . Finally, we map the selected query 𝑞 𝑖 and the text feature 𝑡 linearly onto the same semantic space, written by:EQUATIONEQUATIONwhere 𝑊 𝑞 and 𝑊 𝑡 are the projection matrices, and 𝑏 𝑞 and 𝑏 𝑡 are biases.In visual grounding, a natural language description typically corresponds to a target in the image. Theoretically, one query acts as the positive example, while the rest serve as negative ones. We select the query with the highest matching score to the current text in the image as the positive sample. Negative queries are chosen from queries in images not matching the current text. Our objective is to maximize the similarity between the text and the matching query and minimize the similarity with unmatched queries. Therefore, we define the contrastive loss as follows:EQUATIONhere, 𝑓 𝑗 𝑞 𝑛 represents the positive and negative queries sampled from a batch, with 𝑓 𝑖 𝑞 1 being the positive one. The function sim(•) is utilized for calculating similarity, which is defined by the inner product. The term I ¬(𝑖=𝑗 𝑛≠1) is the indicator function, equal to 0 when 𝑖 = 𝑗 and 𝑛 ≠ 1. 𝐾 is the number of negative queries selected from each unmatched image-text pair, 𝐵 is the batch size, and 𝜏 is the temperature parameter in Hinton et al. [11] . 
negative sample quality estimation: 
 In this section, we detail a negative sample selection strategy aimed at leveraging the semantic information embedded in image-text pairs more effectively to further enhance the performance. Driven by the observation that the mere quantity of negative samples does not guarantee superiority, we propose to estimate their quality. The quality is determined by two factors: distinguishability and uniqueness in the semantic space. In particular, a sample that significantly challenges distinction and exhibits a high degree of uniqueness in the semantic space is considered to be of superior quality. We formulate the quantitative evaluation by:EQUATIONEQUATIONwhere 𝑆 𝑢 𝑖 denotes the uniqueness of the negative samples, and 𝑆 𝑑 𝑖 denotes the difficulty in discrimination. The term 𝑓 𝑞 𝑖 represents the negative query under evaluation for quality, and 𝑓 𝑞 𝑛 𝑗 refers to a previously selected negative query. The variable 𝑚 indicates the number of negative queries selected in the current iteration. The symbol cos denotes the cosine similarity calculation, and Norm signifies the min-max normalization.To elucidate, our fundamental principle asserts that cosine similarity effectively captures the resemblance among negative samples in the feature space. When a negative sample has low similarity with its most similar negative sample in this space, it indicates greater uniqueness. Additionally, higher similarity between negative samples and the current input text makes them more challenging to distinguish.To introduce the final strategy for quality estimation, it is crucial to highlight the significance of uniqueness in negative samples within the feature space, as depicted in Fig. 3 . Excessive clustering of a specific category of negative samples within this space can unintentionally displace positive samples from their ideal locations. This phenomenon can be understood intuitively: an overrepresentation of a single type of negative sample can reduce the relative significance of other negative samples, leading to skewed results. In contrast, a more balanced representation of negative samples contributes to enhanced outcomes.In conclusion, to guarantee that negative samples maintain adequate levels of uniqueness and challenge in discrimination, we propose a straightforward and effective method for estimating their quality, defined by: EQUATION)where 𝑆 𝑞 𝑖 denotes the quality score of the negative sample 𝑓 𝑞 𝑖 . By estimating the quality scores for each negative sample, we can rank them in descending order, enabling the selection of a appropriate number of negative samples for contrastive learning.As illustrated in Fig. 4 , to fully utilize the information present in image-text pairs and to ensure equitable sampling of negative samples across each image-text pair, we conduct our sampling based on the number 𝐾 of negative sample selections for each unmatched image-text pair. It is important to note that the uniqueness of negative samples is dynamic and evolves with the selection of different samples. Initially, when no negative sample is chosen, the uniqueness value 𝑆 𝑢 is set to 1. Consequently, the quality scores of negative samples are dynamically updated throughout the selection process. This process involves 𝐾 iterative steps. In each step, after selecting a negative sample from every mismatched image-text pair, we reassess and update the quality scores of the remaining negative samples based on their revised uniqueness.EQUATIONwhere 𝑖 indicates the 𝑖-th iteration step. The term 𝑓 𝑞 𝑛 denotes the set of negative samples selected up to the current iteration, and 𝑆 𝑖 𝑞 represents the quality score associated with the current negative samples.In addition, the variable 𝑛 𝑔𝑝𝑢 denotes the number of GPUs utilized in distributed training, and 𝐵 represents the batch size. Then we can mathematically articulate the relationship between the total number 𝑀 of negative samples chosen and the sampling unit 𝐾 by:EQUATION)Table 1 : Comparisons with state-of-the-art methods on three RES benchmark datasets. Metric. Consistent with [14, 22] , we utilize IoU@0.5 as the metric for REC, indicating a correct prediction when the IoU between the predicted and ground-truth boxes exceeds 0.5. Additionally, for evaluating RES accuracy, we employ mask Intersection-over-Union (IoU) and Prec@0.5 (P@0.5). 
implementation details: 
 In the QueryMatch framework, we employ a pretrained Mask2former, featuring a Swin-B backbone [5, 27] , to extract Query features. This Mask2former is pretrained on the MS-COCO dataset [20] , with images from the val and test sets excluded for all three datasets under consideration. The language encoding is accomplished using a bidirectional GRU [2] , which is further augmented by a self-attention layer [40] . Consistent with the approach in [14] , Input images are resized to dimensions of 416 × 416. The training regimen is executed on two 24G Nvidia RTX 4090 GPUs, with an allocated batch size of 16 for each GPU. During this phase, the parameters of the Mask2former are kept frozen. For the purpose of Query-based contrastive learning, the linear projection is also set to a dimension of 512 for most datasets, and by default, 20 queries with the highest confidence scores per image are selected. Model training is conducted using the Adam optimizer [16] with a constant learning rate of 1e-4. The number of training epochs is set at 25. RES and REC tasks. This fully demonstrates the cross-modal alignment capabilities of QueryMatch. As shown in Tab.1, the comparison models were trained using weakly supervised information such as bounding box annotations, click annotations, and text. Among the existing weakly supervised RES models trained on image-text pairs, TRIS, which employs a pre-trained CLIP encoder and a twostage pseudo-label generation process for training, previously held the highest performance. However, our QueryMatch significantly outperforms TRIS across all datasets through its novel Query-Text matching framework. For instance, on the RefCOCO dataset, accuracy improved by more than 20%. As can also be seen from Tab.2, QueryMatch surpasses the previous state-of-the-art weakly supervised REC methods by more than 5% on the RefCOCO dataset.Moreover, QueryMatch substantially narrows the performance gap between image-text pair-based RES models and CTS as well as PKS models that utilize more granular weak supervision information like bounding box and click annotations. Compared to the CTS model on the RefCOCO datasets, it improves accuracy by an average of 1%, and notably surpasses the overall performance of the PKS model.We began by evaluating the query confidence selection strategy. In this scenario, the negative sample selection strategy selected the most difficult-to-distinguish query for each unmatched image-text pair. As indicated in Tab.3, implementing the query confidence selection resulted in enhanced outcomes. This approach not only improved computational efficiency but also achieved better accuracy, specifically an average increase of 0.79% on the RefCOCO Concerning the negative sample selection strategy, our initial approach involved random selection among the queries that did not match the image-text pairs. As Tab.4 illustrates, this method sustained low accuracy and exhibited a significant decline when the selection number exceeded 150. Another intuitive strategy was to select a specific number of negative samples from mismatched image-text pairs based on the difficulty of distinguishing them. However, as Tab.4 demonstrates, although this approach generally more effective than random selection, it started to diminish significantly beyond 90 negative sample selections. At 180 selections, there was a 7.22% drop in accuracy compared to the initial 30 selections. This trend suggests that the strategy focusing only on the difficulty of distinguishing has a limited capacity to utilize image-text information effectively.To optimize the utilization of image-text pairs, we proposed a selection strategy that integrates the uniqueness of negative samples with their discrimination difficulty. The results reveal that our method significantly improved accuracy compared to the previous strategies, demonstrating stronger robustness and scalability. Across a range of 30 to 180 negative samples, accuracy remained consistently high. The peak average accuracy, 59% on RefCOCO, was achieved with 120 negative samples. Even with 180 selections, the accuracy remained stable at 58.21%, which is 7.83% higher than the strategy focusing only on discrimination difficulty.Finally, we compared our approach with methods based only on the uniqueness of negative samples and using the sum of uniqueness and discrimination difficulty as a quality measure. As shown in Tab.5, our method exhibited the highest accuracy, substantially outperforming strategies that considered only the difficulty of discrimination. This underscores the importance of negative sample feature distribution in contrastive learning. It's noteworthy that the strategy 𝑆 † 𝑢 𝑖 only considers the uniqueness of negative samples, presupposes selecting the most challenging negative sample in the first iteration step. Opting for random selection in the first iteration step 𝑆 𝑢 𝑖 significantly reduces accuracy, highlighting the critical importance of both metrics in evaluating the quality of negative samples. Additionally, in other datasets, our strategy consistently outperforms, exhibiting a more significant improvement over the 𝑆 † 𝑢 𝑖 strategy than Tab.5 indicates.To provide an intuitive understanding of the efficacy of our proposed method, we conducted visualizations, as illustrated in Fig. and Fig. 6 . Compared to the previous state-of-the-art models, the weakly supervised RES model TRIS and the weakly supervised REC model RefCLIP, our approach demonstrates substantial improvements. Evidently, TRIS often struggles with accurately locating targets, leading to significant prediction errors in many cases. Meanwhile, RefCLIP tends to make localization errors in scenes where the relationships between objects are complex. In contrast, Query-Match exhibits a high degree of precision in target localization in most scenarios. This accuracy persists even in complex situations, such as when objects are intricately interrelated or densely packed. Our method's ability to consistently deliver accurate predictions under such challenging conditions highlights its robustness and effectiveness. 
conclusions: 
 In this paper, we present QueryMatch, a novel weakly supervised visual grounding framework that only relies on image-text pairs. The framework reformulates visual grounding as a Query-Text matching problem, notably simplifying the training complexity by employing an end-to-end paradigm. Further, inspired by experimental observations that highlight the crucial role of negative sample quality in cross-modal contrastive learning, we propose a strategy, namely NSQE, to estimate the quality of negative samples. This strategy significantly boosts performance by selecting high-quality negative samples, emphasizing their uniqueness and difficulty in discrimination. Experimental results validate the effectiveness and robustness of both the Query-Text matching framework and the negative sample quality estimation strategy. 
: 
 tasks [19, 23, 34, 36] , visual grounding not only requires to capture fine-grained object information, but also needs to achieve accurate vision-language alignments. To this end, existing methods often require expensive annotations to learn accurate alignments between natural language descriptions and objects, which is laborious and expensive for practical deployment.To overcome this limitation, researchers have put their efforts into the weakly supervised learning for visual grounding [7, 8, 14, 15, 22, 24, 26, 38, 39, 41, 46] . Among them, early methods [8, 24-26, 39, 41, 46] often resorts to two-stage object detectors [36] to formulate weakly supervised visual grounding as region-text matching problem. However, these methods are often limited in their inference speed due to the expensive processing of two-stage detectors. Therefore, recent endeavors [14] have been devoted to efficient one-stage modeling for weakly supervised visual grounding. As shown in Fig. 1 , they first extract anchor features from a pre-trained one-stage detector, and then conduct anchor-text matching to select the target anchor. Finally, a detection head is used to decode the anchor to the bounding box. This one-stage paradigm does not rely on complex processing like RoI pooling [36] , and shows much better efficiency than the two-stage one.Despite the efficiency, we identify that the anchor-based weakly supervised framework still limits by its object representations, i.e., anchor points. As shown in Fig. 1 , anchor points are grid features extracted by the pre-trained detector, e.g., YOLOv3 [34] . Ideally, these anchor points should represent corresponding object informations so that the anchor-text matching can select the target anchor for grounding. However, compared to region features, anchor points are fragmented and struggle to accurately describe object information. Besides, the large receptive field of an anchor often allows it to receive more noisy object informations, which inevitably hinders the accurate vision-language (VL) alignments.To address these issues, we propose a novel query-text matching framework for one-stage weakly supervised visual grounding, namely QueryMatch. Different from existing methods, QueryMatch resorts to query features extracted from Transformer-based detectors, e.g., DETR [3] and Mask2Former [5] , to represent objects. Benefiting from the bipartite matching based pre-training, query features can achieve one-to-one association with visual objects. Therefore, compared to anchor points, query features contain sufficient and accurate object information. By selecting the target query based on the given expression, we can decode the mask or bounding box of the referent by the pre-trained decoder. To enable weakly supervised learning, QueryMatch performs querytext contrastive learning to achieve vision-language alignments via numerous query-text pairs.Based on QueryMatch, we further propose an innovative strategy for effective weakly supervised learning, namely Negative Sample Quality Estimation (NSQE). As indicated in existing literature [10, 48] , the number and quality of negative samples significantly impact the effectiveness of contrastive learning. Nevertheless, randomly selected negative samples are often noisy and useless, which will practically hurt the weakly supervised performance. In contrast, NSQE aims to actively select informative samples from a batch for negative query augmentations. Specifically, we design two novel metrics in NSQE to evaluate the quality of negative queries from the uniqueness and the difficulty. By dynamically combining these metrics, we can effectively identify and select high-quality negative queries. With NSQE, we can augment the negative samples by 3 times and achieve significant gains in QueryMatch.To validate QueryMatch, we conduct extensive experiments in two grounding tasks, namely referring expression comprehension (REC) and segmentation (RES). Experiments on RefCOCO [30] , RefCOCO+ [30] and RefCOCOg [28, 30] show that QueryMatch outperforms existing weakly supervised methods by large margins e.g., over +20% for RES on RefCOCO. More importantly, qualitative and quantitative results also confirm the effectiveness of NSQE in query-based contrastive learning.In summary, this paper makes three main contributions:• We identify the shortcoming of object representations in existing one-stage weakly supervised visual grounding framework. To overcome this limitation, we propose a novel querybased one-stage framework for weakly supervised visual grounding, namely QueryMatch, which leverages query features to accurately describe object information. • We propose an innovative strategy for effective weakly supervised learning, namely Negative Sample Quality Estimation (NSQE), which can greatly augment negative samples by selecting high-quality query features. • The proposed QueryMatch achieves the state-of-the-art results against existing weakly supervised methods on three benchmark datasets of REC and RES. 
 
 Appendix: Type indicates what kinds of visual features the approach is based on to match text. Sup. denotes the supervision type (B: box-level labels, C: click-level annotations, T: text description labels). † indicates the methods adapted from other related tasks by the related paper[22]. "-" denotes unavailable values. 


Decision: This paper introduces QueryMatch, a novel query-based framework for weakly supervised visual grounding that addresses the limitations of noisy anchor representations. QueryMatch uses query features for accurate one-to-one object associations and employs query-based contrastive learning. It also introduces Negative Sample Quality Estimation (NSQE) to enhance training with high-quality negative samples. Extensive experiments on benchmark datasets demonstrate QueryMatch's state-of-the-art performance and the effectiveness of NSQE in weakly supervised learning.

Pros
- This paper introduces a novel query-based framework for weakly supervised visual grounding, addressing limitations of existing anchor-based frameworks and improving vision-language alignments. The proposed method effectively selects high-quality negative samples, leading to improved performance in weakly supervised learning.
- Comprehensive experiments on benchmark datasets, achieving state-of-the-art results in both referring expression comprehension (REC) and segmentation (RES) tasks. In particular, the results show a substantial improvement in the RES problem.
- The paper is well-organized and easy to follow.

Cons
- There are flaws in the experiments, such as missing results for many methods in Table 2 and no comparison with RefCLIP. Table 1 results exceed expected upper bounds (PKS or CTS) without adequate explanation. There is also an insufficient comparison with well-known baselines and a lack of detailed architecture and hyperparameter settings.
- There are some typos too as revealed by the reviewers.

Overall, this is a decent paper with a novel idea and solid experiments. All reviewers lean towards accepting this paper. The authors are recommended to address all reviewer comments in the camera-ready version.

Reviews: review: paper_summary: paper_summary: The author proposed QueryMatch, a framework target for weakly supervised visual grounding(WSVG). The author reformulates WSVG as a query-text matching problem and optimizes the problem using query-based contrastive learning. To solve the problem, the author proposed NSQE, which can select high-quality negative samples. Experiment results show the effectiveness of the proposed framework.
strength_weakness: strength_weakness: 1. The experiment shows a huge improvement in the RES problem. 
2. The paper is well-organized and easy to follow.
limitations: limitations: 1. In Table 2, the author missed lots of results for other methods. Also, the RefCLIP has better results in their paper, so why not compare with that result?
2. In Table 1, why your result is better than PKS or CTS which provides more information, it should be the upper bound for RES. The author should provide more analysis of why their method can have such a high performance or provide an experiment that also provides C+T or B+T to provide more comprehensive results. 
3. I feel confused about the evaluation metrics that the author used. The author claims that for RES, they use two different evaluation metrics(IoU, P@0.5), but there is only one result.
paper_summary: paper_summary: The paper titled "QueryMatch: A Query-based Contrastive Learning Framework for Weakly Supervised Visual Grounding" proposes a novel query-based framework for weakly supervised visual grounding. The authors identify the limitations of existing one-stage anchor-based frameworks and introduce query features to accurately describe object information. They also propose a negative sample quality estimation strategy to select high-quality negative samples for contrastive learning. Extensive experiments on benchmark datasets demonstrate the effectiveness of the proposed framework and strategy.
strength_weakness: strength_weakness: (1) The paper introduces a novel query-based framework for weakly supervised visual grounding, which addresses the limitations of existing anchor-based frameworks and improves the accuracy of vision-language alignments.
(2) The proposed negative sample quality estimation strategy effectively selects high-quality negative samples, leading to improved performance in weakly supervised learning.
(3) The paper provides extensive experiments on benchmark datasets and achieves state-of-the-art results in both referring expression comprehension (REC) and segmentation (RES) tasks.
limitations: limitations: (1) The paper lacks a detailed description of the implementation details, such as the specific architecture of the proposed framework and the hyperparameter settings.
(2) The paper does not provide a thorough comparison with widely-known baselines in the field, which makes it difficult to assess the novelty and significance of the proposed framework.
(3) The paper lacks a comprehensive analysis of the limitations and potential future directions of the proposed framework.

Overall, the paper presents a novel query-based framework for weakly supervised visual grounding and introduces an effective strategy for selecting high-quality negative samples. The experimental results demonstrate the superiority of the proposed framework over existing methods. However, more details and comparisons with baselines would further strengthen the paper.
paper_summary: paper_summary: The paper introduces QueryMatch, a query-based one-stage framework for weakly supervised visual grounding that improves vision-language alignment by using query features to represent candidate objects. It redefines the task as a query-text matching problem optimized through query-based contrastive learning. The authors also propose Negative Sample Quality Estimation (NSQE) to enhance learning by selecting high-quality negative samples. Experiments on benchmark datasets show QueryMatch's state-of-the-art performance and confirm NSQE's effectiveness.
strength_weakness: strength_weakness: 1. The motivation is clear, and the paper is well-written and easy to follow.
2. The novelty of this paper is significant, which is clearly shown in Figure 1. Authors take a deep look into existing works and re-formulates weakly supervised visual grounding tasks as a query-text matching problem, which seems interesting and inspiring for me.
3. The proposed method achieves the state-of-the-art performance on benchmark datasets with relatively smaller inference costs. Both quantitative and qualitative analyses are very detailed and persuasive.
limitations: limitations: None in particular
paper_summary: paper_summary: The paper introduces a weakly supervised visual grounding method, which is based on the transformer backbone and conducts the contrastive learning by the learnable query. Additionally, the authors propose a negative sample selection strategy that enhances the quality of negative samples. The approach achieves good performance on both the RES and REC tasks.
strength_weakness: strength_weakness: 1) The method improves the strong transformer-based method from full supervised into the weakly supervised manner, which help for the future work.
2) The quality estimation guarantees that the negative sample is orthogonal, and the ablation experiments results are encouraging.
limitations: limitations: 1) As I know for the weakly supervised visual grounding task, the input are the text and the image without grounding box, so the text is known towards models. However, in the line 372, you claim the text is unknown during training but you provide the text input. Can you explain it?
2) I can understand the running process, but the Figure 2 is confusion for me, especially the relation between the green box and blue box. For example, which module's parameters have been updated by contrastive learning? 
3) The experiments only conducted on one backbone cannot demonstrate method generalization.
4) Is the method perform well in the full supervised manner?
Typo Errors
Two comma in Line 412.