Neighbor Does Matter: Global Positive-Negative Sampling for Vision-Language Pre-training

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 11:27:47

Abstract: Sampling strategies have been widely adopted in Vision-Language Pre-training (VLP) and have achieved great success recently. However, the sampling strategies adopted by current VLP works are limited in two ways: i) they only focus on negative sampling, ignoring the importance of more informative positive samples; ii) their sampling strategies are conducted in the local in-batch level, which may lead to sub-optimal results. To tackle these problems, in this paper, we propose a Global Positive-Negative Sampling (GPN-S) framework for vision-language pre-training, which conducts both positive and negative sampling in the global level, grounded on the notion of neighborhood relationships. Specifically, our proposed GPN-S framework is capable of utilizing positive sampling to bring semantically equivalent samples closer, as well as employing negative sampling to push challenging negative samples farther away. We jointly consider them for vision-language pre-training on the global-level perspective rather than a local-level mini-batch, which provides more informative and diverse samples. We evaluate the effectiveness of the proposed GPN-S framework by conducting experiments on several common downstream tasks, and the results demonstrate significant performance improvement over the existing models.

Main paper: 
 related works 2.1 vision and language pre-training: 
 Vision-Language Pre-training (VLP) models are primarily categorized into two types: dual encoder models and fusion encoder models. Dual encoder models, such as CLIP [25] and ALIGN [13] , independently encode images and texts, employing contrastive learning to align image-text pair embeddings. While effective in retrieval tasks, their limited interaction between modalities restricts performance in complex tasks like Visual Question Answering (VQA). Fusion encoder models overcome this by integrating a fusion encoder to meld features from uni-modal encoders. Early iterations [8, 20, 22, 29] relied on pre-trained object detectors for visual feature extraction, facing significant time overhead and capacity constraints. ViLT [16] addressed this by directly inputting image patch features and text token embeddings into a ViT [9] model, but this approach lagged behind object-detector based models due to inadequate unimodal modeling. Subsequent developments, such as ALBEF [19] , combined the benefits of dual and fusion encoders, inspiring further enhancements in models like TCL [32] . TCL introduced intra-modal and global-local contrastive losses to achieve better alignment capabilities.Recent developments in multi-modal large language models have also been based on vision and language pre-training. For instance, BLIP-2 [18] introduced a lightweight Q-Former to bridge the modality gap between vision and text. It employs a two-stage training process, requiring an initial stage of vision and language pre-training to learn how to align with text before connecting to the LLM in the second stage.Our framework distinguishes itself by being model-agnostic, compatible with a broad range of VLP models. It uniquely enhances performance by refining the sampling strategies of these models, a critical factor not explicitly addressed in previous approaches.The primary focus of representation learning lies in extracting semantic information from data, where the representations of similar (positive) pairs are clustered together and those of dissimilar (negative) pairs are spread apart [3] . The sampling of positive and negative pairs is critical in the success of VLP alignment.Positive sampling traditionally focuses on generating semantically similar pairs, particularly through random data augmentation in uni-modal contexts [6, 7, 30] . Approaches like NNCLR [11] have leveraged the nearest-neighbor sample as the positive. It leverages a queue for nearest-neighbor identification in uni-modal data, which is notably trained on the 1000-class ImageNet dataset. In such a clean, categorized environment, a queue suffices for identifying similar neighbors. However, challenges emerge in web-crawled, noisier datasets where identifying semantically similar positive samples requires more nuanced strategies. This underscores the need for advanced positive sampling approaches in cross-modal scenarios, a domain less explored in previous research.Negative sampling is aimed at identifying pairs with dissimilar semantics yet closely embedded representations. Techniques that incorporate hard negative sampling have shown to be beneficial. For instance, ALBEF [19] selects the nearest sample in a mini-batch as the hard negative. VLMo [2] extends this approach by mining hard negatives from a broader pool, gathering training examples across all GPUs, In contrast, GRIT-VLP [5] maintains a queue to identify hard negatives, resulting in more significant improvements.Our work introduces a novel unified positive-negative sampling framework, distinct for its global-level approach. Unlike existing methods constrained by the limited scope of mini-batches or queues, our framework performs sampling across the entire dataset. This comprehensive strategy enables more effective differentiation and identification of positive and negative samples, especially in noisy, diverse datasets, addressing gaps left by previous VLP research and offering a significant advancement in the field. 
method: 
 In this section, we present the Global Positive-Negative Sampling (GPN-S) framework, which is designed to improve the performance of existing VLP models by refining their sampling strategies. We will first introduce the preliminaries about how current VLP models align image and text.For a VLP model composed of a visual encoder 𝑓 𝑉 (•), a text encoder 𝑓 𝑇 (•), and a fusion encoder 𝑓 𝐹 (•, •), the unimodal representations for an image-text pair (𝑣, 𝑡) are obtained as follows:EQUATIONEQUATIONwhere v cls and t cls represent the embeddings of the corresponding [CLS] token. The fusion representation is obtained through𝑓 𝐹 (𝑓 𝑉 (𝑣), 𝑓 𝑇 (𝑡)).Training the VLP model typically involves adopting the imagetext matching objective, which introduces a predict head 𝑓 𝐻 (•) to predict the matching likelihood between the image and text. The probability of a match between 𝑣 and 𝑡 is calculated as:EQUATIONGiven a batch of 𝑁 image-text pairs {(𝑣 𝑖 , 𝑡 𝑖 )} 𝑁 𝑖=1 , where (𝑣 𝑖 , 𝑡 𝑖 ) is referred to as a positive pair and (𝑣 𝑖 , 𝑡 𝑗 ), ∀𝑗 ≠ 𝑖 as a negative pair. The image-text matching loss on the positive pair (𝑣 𝑖 , 𝑡 𝑖 ) is calculated using binary cross-entropy loss:EQUATIONwhere 𝑡 𝑗 ( 𝑗 ≠ 𝑖) is a randomly selected text from the batch. Recent works [2, 19] introduce hard negative sampling by selecting a 𝑡 𝑗 to maximize the dot product (v cls 𝑖 ) 𝑇 t cls 𝑗 within the batch. The hard The proposed method contains two primary strategies: 1) Global Positive Sampling(GP-S), which aims to sample both cross-modal and uni-modal data pairs that share same semantic through the neighboring information. 2) Global Negative Sampling(GN-S), which aims to sample more challenging negative data through the clustering information.negative pairs share similar semantics but differ in fine-grained details, thereby enhancing the model's ability for fine-grained understanding. However, the quality of hard negatives of existing works is constrained by the batch size 𝑁 . Different VLP models may introduce different loss functions, all of which are designed to align positive image-text pair (𝑣 𝑖 , 𝑡 𝑖 ). Without loss of generality, we denote all other losses as L other , and the final training loss is expressed as:EQUATIONWhile current VLP models exhibit outstanding performance across diverse tasks, there are still some limitations in their sampling strategies for positive and negative pairs. Firstly, due to the prohibitive cost of manual labeling, researchers have to train VLP models with web-collected image-text pairs [23, 26] . However, these texts may contain noise and fail to precisely describe the content of corresponding images (e.g., an image of a dog paired with the text "The photo is taken on Sunday"), and such positive pairs hinder the alignment process. Additionally, the current sampling is confined to local-level batches with a limited number of samples, thereby restricting the ability to sample optimal positives or negatives.To address these problems, we propose the Global Positive-Negative Sampling (GPN-S) framework. As shown in Figure 2 , GPN-S utilizes a pre-trained encoder, such as CLIP [25] , to map image and text data from the entire dataset into a unified global embedding space. Within this space, GPN-S samples optimal positives and negatives by leveraging the neighboring relationships among samples. We will introduce how to calculate the neighboring relationships in section 3.2. Then, in section 3.3 and section 3.4, we respectively give the detailed description for how GPN-S samples positives and negatives through neighboring relationships. The pseudocode is provided in the appendix.GPN-S utilizes an off-the-shelf pre-trained model that consists of a visual encoder 𝑔 𝑉 (•) and a text encoder 𝑔 𝑇 (•). These encoders are employed to map image and text from the entire dataset into a unified global embedding space. Given a dataset with 𝐷 image-text pairs {(𝑣 𝑖 , 𝑡 𝑖 )} 𝐷 𝑖=1 , it computes the image embedding v 𝑔 𝑖 = 𝑔 𝑉 (𝑣 𝑖 ) and text embedding t 𝑔 𝑖 = 𝑔 𝑇 (𝑡 𝑖 ) for each pair. Then, we employ Faiss [14] to identify the 𝑘 nearest-neighbor texts for each image 𝑣 𝑖 , determined by the closest cosine distance to 𝑣 𝑖 's embedding, denoted as 𝑉 2𝑇 𝑘 (𝑣 𝑖 ):EQUATIONwhere top 𝑘 identifies the indices of the top 𝑘 largest values. Similarly, we compute 𝑘 nearest-neighbor images 𝑉 2𝑉 𝑘 (𝑣 𝑖 ) for each image 𝑣 𝑖 , and 𝑘 nearest-neighbor images 𝑇 2𝑉 𝑘 (𝑡 𝑖 ) for each text 𝑡 𝑖 . These cross-modal and uni-modal neighbors are crucial for guiding our sampling strategy.Due to textual noise, the pair (𝑣 𝑖 , 𝑡 𝑖 ) does not always serve as a genuinely positive pair that conducive to alignment. We address this issue from two perspectives. From cross-modal perspective, we construct a refined text aligned with 𝑣 𝑖 . From uni-modal perspective, we establish uni-modal alignment among sampled positive image pairs, enabling the representation of 𝑣 𝑖 to align with semantically similar text through other images. This section will elaborate on the utilization of global neighboring information for sampling positive cross-modal and uni-modal pairs.3.3.1 GP-S for Cross-Modal. As mentioned earlier, positive sampling for cross-modal refers to constructing a refined text. We need to extract the most crucial visual information from the image, which pertains to the objects present in the image. We can infer the objects through the neighboring text, as shown in Figure 3 . To be specific, we utilize SpaCy's part-of-speech tagging tool to extract all nouns from the dataset's text. Subsequently, we create a set of nouns, denoted as 𝑆 𝑛 , comprising those nouns that occur more than 10 times. For a given image 𝑣 𝑖 , if the frequency of a particular noun in its 𝑘 nearest-neighbor text surpasses a threshold 𝑝, we conclude that the image 𝑣 𝑖 contains the object represented by that noun, denoted as 𝑜𝑏 𝑗 (𝑣 𝑖 ):EQUATIONwhere the binary indicator EQUATION)where 𝐸 is the total number of epochs and 𝑒 is the number of the current epoch. This strategy ensures that the model is not disrupted by noise during the initial training, leading to a well-optimized starting point. Once alignment performance is sufficiently high, the model can gradually comprehend more complex original text.For uni-modal alignment, the focus is to sample image pairs with both structural and semantic similarity, we define semantic-equivalent images 𝑆𝐸 𝑖 for 𝑣 𝑖 as follows:EQUATIONwhere 𝑘 1 and 𝑘 2 are parameters that control the range of neighbors.Here, information from 𝑉 2𝑉 requires similarity in various aspects such as composition of the images, while information from 𝑇 2𝑉 places greater emphasis on the semantic similarity of the images. When there are no positive images for 𝑣 𝑖 in the dataset, the intersection of the two sets would be empty, and we set 𝑆𝐸 𝑖 as a set containing randomly augmented versions of 𝑣 𝑖 .In each training epoch, we randomly select a positive image 𝑣 pos 𝑖 from 𝑆𝐸 𝑖 and compute the uni-modal contrastive loss defined as follows:EQUATION)where 𝜏 is a learnable temperature parameter, 𝑁 is the batch size.For the input (𝑣 𝑖 , 𝑡 𝑖 , 𝑣 𝑝𝑜𝑠 𝑖 ), the final objective L is achieved by adding the uni-modal contrastive loss to the raw training loss of the model L 𝑓 outlined in its original paper:EQUATIONPrevious works [2, 19, 25] have demonstrated the benefits of sampling hard negatives in VLP. However, in-batch mining is constrained by the batch size. In this section, we introduce global negative sampling, enabling us to acquire more challenging negative samples from the entire dataset. An intuitive approach is to utilize 𝑘 nearest neighbors as hard negatives. However, to avoid potential positives in the negative neighbors, the value of 𝑘 needs to be set relatively high, which in turn results in an increase in computational costs. To enhance efficiency, we opt for the 𝐾-means algorithm [12] for clustering, where any pair of samples within the same cluster serve as a hard negative pair. Specifically, we utilize Faiss [14] to partition the dataset into 𝐾 clusters, with each image-text pair (𝑣 𝑖 , 𝑡 𝑖 ) represented by its image feature v 𝑔 𝑖 . As recent VLP models have incorporated in-batch hard negative mining, introducing our GN-S only requires placing global-level hard negatives into the same batch. Specifically, when a batch of size 𝑁 is constructed, we randomly select 𝑐 clusters from all 𝐾 clusters and 𝑠 samples from each cluster. The remaining 𝑁 -𝑐 × 𝑠 samples are randomly chosen from the entire dataset, as shown in Figure 5 . By employing this approach, the 𝑠 samples from the same cluster act as hard negatives for each other, resulting in a notable increased number of hard negatives compared to batches formed randomly. We observe that the model performs best when 𝑐 × 𝑠 ≈ 𝑁 /4. This is done to maintain diversity within the batch, with still having 3𝑁 /4 completely random samples, thus preventing the model from having difficulty distinguishing the 'easy' negatives.Our framework is highly effective and efficient. Thanks to Faiss [14] , computations such as 500-nearest-neighbor search and 1000-means clustering on a dataset containing 5 million image-text pairs can be completed within 2 hours, utilizing only 2 NVIDIA GeForce RTX-3090 GPUs. As a comparison, training the VLP model ALBEF [19] using 8 A100 GPUs takes 60 hours. The additional computations introduced by our framework do not exceed 10% of the training time. Regarding storage overhead, only the identifiers of the neighbors are stored, such as filenames. The storage space required is significantly smaller compared to that of the original image-text pairs.When applying our framework to a large-scale dataset, we advise dividing the dataset randomly into groups of 5 million data pairs each and sampling neighbors only within each group. This approach has two advantages. First, our experiments have already demonstrated that the neighbor information computed on the 5 million dataset is already sufficient for our global positive and negative sampling, thereby obviating the need for any alteration in hyperparameters. Second, this processing method can be extended to datasets of any size, and the additional costs would also not exceed 10% of the training time. 
experiment 4.1 implementation details: 
 Our GPN-S modifies the existing VLP model's sampling strategy to enhance their performance. We implement GPN-S on ALBEF [19] and TCL [32] model, referring to them as the backbone models. They share similar model architectures and pre-train datasets. Specifically, we utilized their base-size versions. The vision encoder in both models is a ViT-B/16 with 12 layers, and both the text encoder and the fusion encoder are implemented using a 6-layer transformer. The pre-training dataset includes COCO [21] , Visual Genome [17] , Conceptual Captions [27] and SBU Captions [24] , comprising approximately 4 million images and 5.1 million image-text pairs.By default, we use the backbone model as the off-the-shelf pretrained encoder. For instance, when applying our methods to AL-BEF, we obtain the pre-trained checkpoint from their official GitHub repository for neighbor calculation. We empirically set the parameters as follows: 𝑘 = 10, 𝑝 = 0.3 for calculating 𝑜𝑏 𝑗 (𝑣 𝑖 ), 𝑘 1 = 5, 𝑘 2 = 500 for selecting positive image pairs, 𝑐 = 40, 𝑠 = 3 for constructing batches of size 𝑁 = 512 in GN-S, with the number of clusters 𝐾 = 1000. We maintain all settings and training details from the original backbone models, with one modification: we train TCL for 50 epochs, rather than 30, to achieve better convergence. All experiments are conducted on 8 NVIDIA A100 GPUs.To evaluate the effectiveness of our proposed method, we adapt the pre-trained model to various downstream vision-and-language (V+L) tasks, ensuring consistency in all settings with those of the backbone models [19, 32] , as below:Image-Text Retrieval. We evaluate both fine-tune and zero-shot retrieval performance on the MSCOCO [21] dataset and employ the widely used Karpathy split [15] , which comprises 5000 images along with their corresponding 25010 texts. We consider two tasks: imageto-text retrieval (TR), which involves finding the corresponding text for a given image, and text-to-image retrieval (IR), which requires finding the corresponding image with a given text. The R@𝑛 (Recall at 𝑛) measures the proportion of correct answers included within the top 𝑛 retrieved results. The RSUM metric is the sum of TR@1, TR@5, TR@10, IR@1, IR@5, and IR@10.VQA aims to answer natural language questions about images. We fine-tune our model on training and development set of VQAv2 [1] and Visual Genome [17] VQA data, and evaluate on the test-dev and test-std set of VQAv2. We follow ALBEF and TCL, employing a decoder to generate answers from a pool of 3192 candidates.Natural Language for Visual Reasoning for Real (NLVR 2 ) [28] . NLVR 2 presents the task of determining whether a natural language Table 1 : Results on fine-tune(FT) and zero-shot(ZS) retrieval tasks.MSCOCO-FT MSCOCO-ZS TR@1 TR@5 TR@10 IR@1 IR@5 IR@10 RSUM TR@1 TR@5 TR@10 IR@1 IR@5 IR@10 RSUM CLIP [25] - (+0.9) (+0.1) (+0.9) (+0.4) (+0.3) (+4.3) (+1.1) (+1.0) (+0.3) (+1.8) (+1.5) (+0.9) (+6. 6) sentence is true about a pair of images. We follow ALBEF and TCL, extending the fusion encoder to enable reasoning over two images and adding a fully-connected layer to predict if the sentence is true or false.Visual Entailment (SNLI-VE) [31] . SNLI-VE is a novel inference task based on the Stanford Natural Language Inference corpus and Flickr30k dataset. Given a real world image premise and a natural language hypothesis, the goal is to determine if the hypothesis can be concluded given the information provided by image. We follow ALBEF and TCL, considering it as a three-classes classification problem because the relationship could be entailment, neutral or contradiction.For VQA, NLVR 2 and SNLI-VE, we report the average accuracy on the test dataset.This subsection compares the performance of models with and without the GPN-S framework to demonstrate its effectiveness.Retrieval. We evaluate the performance of the model in both fine-tune and zero-shot settings, with the results shown in Table 1 .In both settings, the model incorporating GPN-S significantly outperforms the original backbone model across all metrics. Notably, the improvement is more pronounced in the zero-shot setting, with increases of +14.4(3.1%) for ALBEF and +6.6(1.4%) for TCL in terms of RSUM metrics. This enhancement indicates that GPN-S more effectively aligns images and text, and the more substantial increase in R@1 compared to R@5 and R@10 suggests that GPN-S enhances the model's ability to discern fine-grained differences. 4.3.2 VQA, NLVR 2 and SNLI-VE. We conduct fine-tuning on other downstream tasks requiring deep image-text interaction understanding. Table 2 shows the experimental results. Our method consistently outperforms the corresponding backbone models.Specifically, on the VQA task, we achieve an average improvement of +0.52(+0.7%) and +0.61(+0.8%) for the ALBEF and TCL backbones, respectively. For the NLVR 2 task, our method outperforms the backbone by an average of 0.41(+0.5%). For the SNLI-VE task, our method yields an average improvement of 0.21(+0.3%) over the backbone. These gains are attributed solely to the improved pre-training alignment, as the fine-tuning process remains unchanged. 3 . Each strategy demonstrates improvement across all tasks, except SNLI-VE, compared to the backbone model. Combining two global positive sampling strategies yields better results than employing a single strategy, indicating that uni-modal and cross-modal positive samplings are complementary. Implementing all three strategies leads to the highest performance in most metrics, confirming their collective benefit.The-Shelf Encoder. We investigate the effectiveness of various off-the-shelf models. The results on the ALBEF backbone are detailed in Table 4 . By default, we use the backbone model that is not trained by our framework as the off-the-shelf model (so 'ALBEF' in the table denotes the default method). For comparison, we also utilize CLIP ViT-B/32 and TCL models as alternative off-theshelf encoders, sourced from their official repositories. Our findings indicate that higher-performing off-the-shelf models yield better results in zero-shot retrieval tasks, while maintaining similar levels of performance in other downstream tasks. Given the significant improvements observed across all models, it is advisable to choose the best available off-the-shelf encoder for applying the GPN-S framework. 5 , further support our findings. We achieve comparable model performance through clustering with different modalities.In GN-S, we cluster all samples into 𝐾 clusters. As 𝐾 increases, the number of samples in each cluster decreases, resulting in higher similarity between samples and making it harder to distinguish them.We select different values of 𝐾 and conduct an ablation study on ALBEF backbone, as presented in Figure 6 . Optimal performance is achieved at 𝐾 = 1000 on current dataset. Increasing 𝐾 further Person takes control of the ball from person saturday afternoon as the teams met.A man jumping over an edge of a boat skating.A group of soccer player fighting for the ball.Person --works to control the ball against defenders during game .A boy kicking a soccer ball near a walking girl while others watch.There is a skate boarder that is riding a red pole.A man is about to snowboard across a platform.A snowboarder in the air with a treeline in the background.Negative SamplesRevised: boy soccer fieldRevised: man skating snow results in performance drops, likely due to the issue of false negatives, where semantically similar samples are incorrectly treated as negatives.The GP-S method was proposed to address the impact of textual noise. However, in previous sections, we used relatively clean academic datasets for a fair comparison. In this section, we replace the pretraining dataset with a noisier random 3M subset of CC12M and use GP-S for cross-modal techniques without modifying the parameters 𝑘 and 𝑝. We report the RSUM metric for zero-shot retrieval on the MSCOCO dataset, as shown in Figure 7 . We observe that on this dataset, we achieve a better improvement (RSUM improvement from 354.6 to 382.6, a 7.9% increase). In contrast to a fixed 𝛼, curriculum learning for scheduling 𝛼 proves to be the most effective approach. We believe that curriculum training, progressing from easy to hard, prevents the model from getting stuck in local optima. As pretraining datasets in practical applications continue to grow, they will inevitably contain more noise. As the first work to introduce positive sampling in VLP, we hope this experiment demonstrates the importance of positive sampling in VLP.For a clearer understanding of our sampling strategy, we provide visualizations of the positive and negative samples identified by GPN-S, as illustrated in Figure 8 .In the first example (first row), the text mentions "saturday afternoon" which is not visually represented in the image. We obtain information through neighbors and revise the text accordingly. Moreover, We found neighbor images with similar semantic information and encouraged them to be close to each other in the embedding space. To distinguish hard negative samples within the same cluster, the model is required to fully comprehend the sentence "takes control of the ball from people, " rather than solely recognizing "people" and "ball". In the second example, cross-modal positive sampling successfully captures the key information "snow" which is not explicitly mentioned in the original text. All positive images contain the infomation "skating over a red platform" while negative samples lack certain shared features with the anchor, yet are sufficiently challenging to enable the model to achieve fine-grained discrimination. 
conclusion: 
 This paper introduces the Global Positive-Negative Sampling (GPN-S) framework, a novel framework that expands the sampling strategy of Vision-Language Pre-training (VLP) models to include both positive and negative pairs at a global level. By leveraging neighboring relations, GPN-S effectively identifies globally semantically similar neighbors as positive samples, enhancing the quality of representations in both cross-modal and uni-modal scenarios. Simultaneously, it encourages the model to differentiate between more challenging negative samples within the same cluster. Experimental results validate the effectiveness of this framework, demonstrating significant improvements for various VLP models on widely used benchmarks. 
: 
 model performance. For instance, the CLIP [25] model has shown the benefits of using large mini-batch sizes to obtain hard negatives, thereby increasing model effectiveness. Similarly, ALBEF [19] uses a large negative queue to extract challenging negative examples, leading to notable performance improvements. These developments highlight the potential of leveraging informative samples for further advancements. However, two primary limitations persist:• Current works predominantly focus on sampling hard negatives within mini-batches [2, 19, 32] , a local-level approach that yields less optimal negative samples. • There is limited research on positive sampling strategies, with many studies overlooking the potential of positive sampling in enriching multi-modal alignment information.To overcome these limitations, we propose the Global Positive-Negative Sampling (GPN-S) framework for VLP. This innovative framework conducts both positive and negative sampling on a global level, moving beyond the constraints of local-level minibatches. Specifically, GPN-S maps the entire dataset's samples into a unified embedding space and utilizes their neighborhood relationships to determine which samples should be closer or farther apart, as illustrated in Figure 1 . This approach provides more informative and diverse samples. For global positive sampling (GP-S), we aggregate co-occurring nouns from neighboring texts, replace noisy web-crawled text to enhance the model's cross-modal alignment ability. We also mine semantically equivalent images to improve the model's uni-modal alignment ability. For global negative sampling (GN-S), we employ a global cluster technique to obtain more challenging negative samples. Notably, our method is model-agnostic and can be applied to enhance most existing VLP models.In brief, our contributions can be summarized as follows:• We propose the GPN-S framework, a novel approach that broadens the sampling strategy to encompass both positive and negative pairs at a global level, thus significantly enhancing VLP model performance. • We utilize the relationships between neighboring samples to effectively integrate information from the embedding space, improving the quality of representations in both cross-modal and uni-modal scenarios. • We conduct extensive experiments on several downstream tasks to demonstrate that our GPN-S can significantly yield better performance than the models trained without GPN-S. 
 
 Appendix:  


Decision: The submission proposed to leverage the more informative, positive samples that go beyond the local patch level to help along the VLP models. The submission has been been to consolidate with compelling evidences/justification that can favourably support the claims/contributions. Author responses clinched reviewers concerns to most. Some areas should be considered to improve the quality before publishing.

Critical experiments concerning broader experiments such as image captioning, classification and composition are missing. Those areas are test bed to evaluate any new VLP models. Failing to do so is less convincing to talk the audience around the idea.

The proposed method relies on global sampling spanning the entire dataset, which induces significant computational overheads. However, the authors seem to hedge this issue without providing substantiated proofs to buy this out, albeit the reviewers flagged this for elaborations.

Reviews: review: paper_summary: paper_summary: This paper introduces the Global Positive-Negative Sampling (GPN-S) framework for vision-language pre-training, addressing limitations in current sampling strategies. GPN-S performs both positive and negative sampling globally, based on neighborhood relationships, to bring semantically similar samples closer and push challenging negatives farther apart. Experiments on downstream tasks show significant performance improvements over existing models.
strength_weakness: strength_weakness: - Clear paper writing.

- Decent performance has been achieved.

- Reasonable motivation.
limitations: limitations: - The authors could supplement the paper with experiments related to classification, image captioning, and compositionality.

- It misses the comparison with other positive/negative sampling methods.

This work claims its biggest difference from previous related works is that it selects positive and negative samples from the entire dataset, rather than a mini-batch or mini-batch+memory-bank approach. However, I haven't seen any comparison provided.
paper_summary: paper_summary: This paper proposes a sampling strategy for VLP models that involves global positive and negative samples, differing from previous sampling methods. The authors validate their method on several vision-language downstream tasks, demonstrating its effectiveness.
strength_weakness: strength_weakness: 1. The proposed sampling strategy is a novel strategy that operates on a global level, considering both positive and negative samples beyond the confines of mini-batches.
2. Combine with the proposed strategy, the results shows significant improvements over existing models on various downstream tasks.
3. The GPN-S framework is model-agnostic and can be effectively applied to large-scale datasets with a limited increase in computational costs.
limitations: limitations: 1. How does the GPN-S handle the scalability issue when applied to extremely large datasets that may not fit into memory?
2. The authors used k-means to construct hard-negatives. However, as we know, k-means often encounters the curse of dimensionality when dealing with high-dimensional data, where some clusters may become very large while others are very small or even have zero samples, which clearly does not match the actual situation. How did the authors consider this issue?
3. Figure 6 is a bit blurry.
4. The authors should carefully check the format of the references.
paper_summary: paper_summary: In this paper, a Global Positive-Negative Sampling (GPN-S) is proposed to encompass both positive and negative pairs at a global level, thus significantly enhancing VLP model performance. In addition, this paper jointly considers both positive and negative samples for vision-language pre-training on the global-level perspective rather than a local-level mini-batch, which provides more informative and diverse samples.
strength_weakness: strength_weakness: This paper proposes a Global Positive-Negative Sampling (GPN-S) to encompass both positive and negative pairs at a global level, and conducts experiments to demonstrate the effectiveness of the GPN-S.
1. The paper is easy to understand and flows smoothly.
2. The relevant methods are explained clearly, and they clearly indicate the motivation and novelty.
limitations: limitations: 1. As far as is known, the metrics for each task in Table 2 are greatly influenced by hyperparameters. The approximately 0.5 improvement mentioned in the text might be due to fluctuations during model training and testing, which is concerning.
2. Lines 512: "The additional computations introduced by our framework do not exceed 10% of the training time.", it is recommended to present additional comparative experimental results to demonstrate effectiveness.
3. What impact do N, c, and s in GN-S have on the experimental results? Are there any related experiments or discussions?
4. Please adjust the resolution of Figure 2 and Figure 6 for clearer readability by the readers.
paper_summary: paper_summary: This paper introduces a Global Positive-Negative Sampling (GPN-S) framework for vision-language pre-training. Unlike current methods that focus mainly on negative sampling at the local mini-batch level, GPN-S performs both positive and negative sampling on a global level based on neighborhood relationships. Experiments on several downstream tasks demonstrate that GPN-S significantly improves performance compared to existing models.
strength_weakness: strength_weakness: This paper introduces a Global Positive-Negative Sampling (GPN-S) framework for vision-language pre-training and evaluates it across several visual benchmarks to demonstrate performance. Overall, the paper is well-structured and easy to follow.
limitations: limitations: This paper addresses the limited scope of mini-batches or queues faced by previous methods and proposes performing sampling across the entire dataset. However, I think this approach could result in a significant computational burden. Therefore, it is necessary to compare the computational cost, as well as the training and inference time, between the current method and previous methods.

The authors compared various vision-language models, but the representative work BLIP[1,2] should also be included in the comparison.

Additionally, all the figures are not very clear and are difficult to understand, especially Figure 1.

[1] Li, J., Li, D., Xiong, C., & Hoi, S. (2022, June). Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning (pp. 12888-12900).
[2] Li, J., Li, D., Savarese, S., & Hoi, S. (2023, July). Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning (pp. 19730-19742).