AlignCLIP: Align Multi Domains of Texts Input for CLIP models with Object-IoU Loss

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-09 04:35:02

Abstract: Since the release of the CLIP model by OpenAI, it has received widespread attention. However, categories in the real world often exhibit a long-tail distribution, and existing CLIP models struggle to effectively recognize rare, tail-end classes, such as an endangered African bird. 
An intuitive idea is to generate visual descriptions for these tail-end classes and use descriptions to create category prototypes for classification. 
However, experiments reveal that visual descriptions, image captions, and test prompt templates belong to three distinct domains, leading to distribution shifts. 
In this paper, we propose the use of caption object parsing to identify the objects set contained within captions. 
During training, the object sets is used to generate visual descriptions and test prompts, aligning these three domains and enabling the text encoder to generate category prototypes based on visual descriptions. 
Thanks to the acquired object sets, our approach can construct many-to-many relationships at a lower cost and derive soft labels, addressing the noise issues associated with traditional one-to-one matching. Extensive experimental results demonstrate that our method significantly surpasses the CLIP baseline and exceeds existing methods, achieving a new state-of-the-art (SOTA).

Main paper: 
 introduction: 
 Since the inception of Contrastive Language-Image Pre-training [23] (CLIP) by OpenAI, the field of large-scale vision-language pretraining (VLP) has seen rapid advancements. A multitude of approaches [7, 9, 10, 17, 32] have been proposed, achieving remarkable success across a variety of downstream tasks, thereby revolutionizing our understanding and capabilities in bridging visual and linguistic modalities.Generally, CLIP models are trained using a contrastive learning approach, where they learn to match images with their corresponding text descriptions across a large dataset. This is achieved by embedding both images and texts into a common space and optimizing the model to bring the embeddings of matching pairs closer, while pushing non-matching pairs apart. During the testing phase, models are usually tested on a variety of tasks such as zero-shot classification, retrieval, linear probe, etc. For the zero-shot classification task, test prompt templates such as "an image of class name" are used to generate prototypes for all categories, based on which images are classified without direct training on those specific categories.Although the CLIP model has made great progress in recent times, we observe that they are still sub-optimal:First of all, the distribution of concepts worldwide follows a long-tail distribution. The CLIP models have better zero-shot performance for common concepts and categories. However, their recognition capabilities are significantly weaker for categories that are rare or absent in the training set as shown in Fig. 1(a) . Using CC3M as pre-training, its recognition performance for many tail categories on ImageNet 1K is very low, and even 295 categories have an accuracy of 0. These categories with an accuracy of 0 are because these categories have never appeared in the training set. For example, the class "Airedale Terrier" in ImageNet 1K have a accuracy of 0, which is the largest of the terrier breeds, i.e. a kind of dog. However, this fine-grained dog does not appear in the training set CC3M, resulting in its unrecognizability.A simple and intuitive idea is to use LLM to generate visual descriptions for categories that do not appear in the training set, and use these visual descriptions to generate category prototypes [19, 29] , i.e., classifier weights. However, we found that the visual description and the caption of the training set are in two different domains, that is, they have a distribution shift, as shown in Fig. 1(b) .Furthermore, from Fig. 1 (b), we can see the test prompt templates "an image of class name" also distributed differently from training captions. A classic solution is prompt learning [32, 33] , that is, learning a prompt template to reduce this domain gap. This can be seen as making the prompts in the test phase align as much as possible with the captions in the training phase. This will cause additional testing overhead and is different from the traditional alignment of the training phase to the testing phase.In this paper, we solve the above problem in a unified and novel framework, named AlignCLIP, that is, we align the image captions, test prompt templates and visual description in the training stage, which enables higher performance during testing with one of them as text input with zero additional overhead. To be specific, during the training phase, we first mining the object items contained in image caption with caption object parsing. The caption object parsing operation takes a caption as input and output the object items set in it. For example, the caption 𝑇 𝑖 = "The brown fox is quick and he is jumping over the lazy dog", which contains two objects, namely fox and dog, so caption object parsing outputs the set 𝜋 𝑖 = {𝑓 𝑜𝑥, 𝑑𝑜𝑔}. Then, we input the object set 𝜋 𝑖 into two generation modules, namely the visual description generation and the test prompts generation. Specifically, on one hand, we generate visual descriptions for the tail concepts in the training set, so that the text encoder can align the visual descriptions with image features and have the ability to generate category prototypes based on the visual descriptions. On the other hand, we randomly sample templates from the test prompt templates pool to generate test prompts for the set 𝜋 𝑖 . Since the set 𝜋 𝑖 may contain more than one object, the features of multiple objects will be averaged and normalized, and then aligned with the image features.Currently, a popular improvement direction of CLIP is to change the original one-to-one hard label to soft label [9, 10] , where different negative samples will be assigned different soft labels according to their similarity, instead of 0 for all. However, existing works all share a common shortcoming, i.e, they use an object detector to detect objects in the image to calculate the similarity, which significantly slows down training, such as SoftCLIP [9] . Thanks to the parsed set 𝜋, we can easily measure the similarity between different negative samples and construct the many-to-many relationship with lower computational cost. According to the object sets in a batch, the intersection over union (IoU) between different sets can be used to measure the similarity between images, and further normalized as soft labels. The loss based on the IoU of parsed object sets is denoted as object-IoU loss.In summary, our major contributions are as follows:• We propose to utilize caption object parsing to mine the objects in an image, based on which the tail-classes is enhanced  
related work: 
 Here, we show part of the related work. Please refer to the appendix for the complete related work.In the realm of Vision-Language Pre-training (VLP), the endeavor to synergize visual and textual modalities has been operationalized through extensive training on image-text pairs. Architecturally, VLP models bifurcate into two predominant streams: single-stream and dual-stream frameworks. Single-stream architectures integrate image and text inputs early in the process, utilizing a unified transformer to process the amalgamated embeddings, typified by models such as VisualBERT [14] , OSCAR [16] , UNITER [5] , UNICODER [12] , UNIMO [15] and HAMMER [24] . This architecture facilitates direct interaction between modalities within a shared semantic space. Conversely, dual-stream architectures advocate for a modular approach, encoding images and texts through distinct pathways before convergence. Models like CLIP [23] , ALIGN [13] , DeCLIP [17] , SoftCLIP [9] , PyramidCLIP [10] and LaCLIP [7] exemplify this approach, underscoring the advantage of discrete yet complementary processing of modal information. Most of this work is to improve certain shortcomings of CLIP. For example, DeCLIP [17] speeds up training through self-supervision. PyramidCLIP [10] uses object detectors for more fine-grained alignment. SoftCLIP [9] uses object detectors to construct many-to-many relationships.The proposed AlignCLIP belongs to the dual-stream architecture. Differently, AlignCLIP sets out to solve the long-tail distribution in CLIP and misalignment in multiple text domains. Furthermore, we achieve soft label training at low cost based on caption objectf parsing. Compared with previous methods, AlignCLIP training cost is lower and its performance is better.The long-tail distribution [18] , where few categories are common and many are rare, presents a significant challenge in data mining and machine learning. Addressing this, researchers have developed three main strategies: re-sampling, re-weighting, and transfer learning. Re-sampling [2, 26, 28] methods adjust the dataset to balance the distribution between common and rare classes, either by increasing the presence of rare classes or reducing that of common ones. Re-weighting [4, 22, 30] approaches alter the loss function to prioritize rare classes during training, giving them more importance. Transfer learning [20, 21, 31] techniques use the knowledge gained from common classes to improve the learning of rare classes, enriching their feature representation. These strategies, from adjusting data distribution to modifying training emphasis, offer pathways to mitigate the long-tail problem, aiming for a more balanced learning across classes.However, in multi-modal pre-training, there are relatively few solutions to the long-tail problem, which has been grossly ignored. Although there are some works that use visual descriptions to improve the performance [19, 29] , however, they only generate category attributes at the test stage, which leads to the multi-domain misalignment, limiting model performance. We propose to use visual descriptions while solving the distribution shift of multiple domains during the training stage, achieving better results. 
method: 
 In this section, We first introduce the framework of AlignCLIP. Then we introduce the first step of AlignCLIP, which is caption object parsing. After that, we introduce how to perform multi-domain alignment on the basis of parsed object sets. Finally, we introduce how to perform soft label training based on object-IoU loss.The training of AlignCLIP is shown in Fig. 2 , which is similar to CLIP [23] , that is, matching between image and text pairs is trained through contrastive learning. For the convenience of presentation, only a single image and text are shown in Fig. 2 as an example.To begin with, an image caption 𝑇 𝑖 is input into the caption object parsing module to obtain the object items in the caption, and the parsed objects set is denoted as 𝜋 𝑖 . With set 𝜋 𝑖 , on the one hand, we generate the visual descriptions of objects in 𝜋 𝑖 , if any of objects in 𝜋 𝑖 belongs to the statistical tail concepts. On the other hand, for each object in the collection 𝜋 𝑖 " we randomly sample from the prompt templates pool during testing to generate a simulated test prompt, which is used to solve the distribution shift of training and testing. Since a caption may contain multiple objects, we average obtained multiple visual descriptions and then perform the normalization operation, and the same was done for generated test prompts. Meanwhile, the image 𝐼 𝑖 is extracted feature with image encoder, which will be aligned with the feature of caption 𝑇 𝑖 , averaged visual description and test prompts at the same time. Finally, we propose object-IoU loss to generate soft labels, which utilize the similarity between object sets in a batch to construct many-to-many relationships.For model inference, its cost is similar to ordinary CLIP and does not require additional calculations. During inference, the model can support both visual description input and ordinary test prompts. Users can choose which input form to use based on whether the category is common and accuracy metrics.In order to perform subsequent multi-domain alignment and object-IoU loss, we first need to perform caption object parsing. Compared with previous methods that perform object detection on input image, obtaining object information from the caption is cheaper and faster, and does not rely on a trained object detector.Formally, for an image caption 𝑇 𝑖 , the goal of caption object parsing is to obtain all objects𝜋 𝑖 = {𝑜 𝑖 1 , 𝑜 𝑖 2 , ..𝑜 𝑖 𝑘 𝑖} contained in the 𝑇 𝑖 , where 𝜋 𝑖 is the objects set, 𝑜 𝑖 𝑗 is an object in 𝜋 𝑖 and 𝑘 𝑖 is the number of items in 𝜋 𝑖 .There are many ways to obtain objects in a sentence. In this paper, we include two methods, namely part-of-speech tagging (POS) and large language model (LLM).The task of part-of-speech tagging is to mark the part-of-speech of each word in a sentence, such as nouns, verbs, adjectives, adverbs, etc. Usually, the objects in a caption are nouns. Therefore, we use the part-of-speech tagging algorithm in NLTK [1] to get the nouns in the caption and output it as the result 𝜋 𝑖 .For LLM, we design prompts to enable open source LLM to output the objects in the caption. Due to the simplicity of the task, even an extremely small-sized open source LLM can output very high quality results. In the experiments, for the LLM scheme, we use the Gemma-2B-Chat [27] model, and the specific prompts can be found in the appendix.As mentioned above, the original image caption, the test prompts such as "an image of class name", and the visual description belong to three different domains, and there is a distribution shift between them, which leads to a performance drop. Here, we align these three with image features at the same time during training, so that the model has the ability to use visual descriptions, test prompts, and original captions to generate category prototypes without any distribution shift.To align visual descriptions and visual features, a simple approach is to generate visual descriptions for all objects in the training set, and perform the alignment training. However, this consumes a lot of time, since the training set is large. Here, we adopt a more lightweight approach, which is to generate visual descriptions for only the rarest 𝛼 percent of the tail, where 𝛼 is a hyperparameter set to 30% in our experiments.Object frequency statistics: In order to distinguish which objects in the training set are rare objects in the tail, our first step is to count the frequency of occurrence of objects in the training set and regard objects with less frequency as rare objects.To be specific, we first generate a frequency table Θ that records each object and the number of occurrence the object appears in the training set. Specifically, Θ is formulated as follows:EQUATIONwhere 𝑥 𝑗 is the frequency of occurrence of object 𝑜 𝑗 , 𝑁 1 is the number of objects in total and 𝑁 2 is the number of captions in training set. Then, we sort the objects in Θ according to the number of occurrences 𝑥 𝑗 , and retain the 𝛼 percentile objects with the least occurrences, which is denoted as Θ 𝛼 .Description generation: Since the object set 𝜋 𝑖 of a caption may contain multiple objects, therefore, only a part of the objects in the set 𝜋 𝑖 may belong to the tail objects Θ 𝛼 . In this case, our approach is that as long as 𝜋 𝑖 contains any tail object, other objects also generate visual descriptions together. Specifically, for the set 𝑇 ! 𝐼 ! # 𝐼 ! # 𝑇 ! # 𝒱 ! % 𝒬 ! Figure 2:The training of AlignCLIP. Firstly, for an image caption 𝑇 𝑖 , we perform caption object parsing to obtain the object set 𝜋 𝑖 , as 𝜋 𝑖 = {𝑑𝑜𝑔, 𝑓 𝑜𝑥 } above. Based on the set 𝜋 𝑖 , we perform multi-domain alignment, that is, align the distribution of original captions, visual descriptions, and test prompts to image features. To achieve this, we need to generate visual descriptions of 𝜋 𝑖 if any of objects in 𝜋 𝑖 belongs to tail-classes , as well as test prompts. Since set 𝜋 𝑖 may contain more than one objects, the mean and normalization operation will be performed to obtain the global feature. Finally, the image feature is trained to align with text features from origin caption, visual descriptions and test prompts under the supervision of object-IoU loss.𝜋 𝑖 of caption 𝑇 𝑖 , its visual description set V 𝑖 is formalized as:EQUATIONwhere J 𝑣𝑖𝑠 is a function that inputs an object 𝑜 𝑖 𝑗 and outputs an appearance description of the object. We implement J 𝑣𝑖𝑠 with LLM, which is Gemma-2B-Chat [27] and the prompt instruction is in the appendix.Then, the description features in V 𝑖 will be averaged and normalized to get the global feature that can be aligned with the whole image feature:EQUATIONwhere 𝜙 (•) is the normalization operation. 𝑓 𝑡 is the text encoder. 𝑘 𝑖 is the number of objects in set 𝜋 𝑖 . Here, we first input the features of multiple descriptions into the text encoder separately, and then average and normalize them to obtain a global descriptions representation of multiple objects. Another simple method is to concatenate multiple visual descriptions and then input to text encoder. However, this method can easily causes text to exceed window length because the visual descriptions are usually very long.As mentioned above, we found that the distribution of test prompts and training captions are in two different domains, and there is a distribution shift between them.Existing work attempts to utilize the prompt learning to learn a better prompt template to splice in front of the category name, such as CoOp [33] and CoCoOp [32] . This learned prompt can better conform to the distribution of captions in the training set, thereby improving performance. However, each test set must be learned separately, which increases the cost of inference.Here, we follow the idea of data augmentation and generate simulated test prompts based on the parsed object set 𝜋 𝑖 , and perform alignment during the training phase. Thus, the test prompts will be aligned with image features, and the domain gap will be narrowed.Since the zero-shot classification averages the features of multiple prompt templates to obtain the final category prototype in CLIP, we also need to align multiple prompt templates here. We denote the prompt template collection used in model inference as P = {𝜌 1 , 𝜌 2 , ..., 𝜌 𝑚 }, where 𝑚 is the number of templates. For each object 𝑜 𝑖 𝑗 in the set 𝜋 𝑖 , we randomly sample a template 𝜌 ℎ from the collection P to generate a simulated test prompt as follows: where 𝑔(•, •) is a function that combines templates and objects to format a simulated test prompt 𝑞 𝑖 𝑗 . Then, the features of the test prompt Q 𝑖 will be extracted by encoder 𝑓 𝑡 , and then averaged and normalized to obtain global features Q 𝑖 as follows:EQUATIONEQUATIONIn experiments, we take OpenAI 80 templates for ImageNet as P.In this section, we first present the CLIP preliminaries, then, we introduce how to build a many-to-many relationship based on the IoU of object sets and perform soft alignment. EQUATIONEQUATIONBased on the obtained image embedding and text embedding pairs {( 𝐼 𝑖 , 𝑇 𝑖 )} 𝑁 𝐵 𝑖=1 , CLIP uses contrasting learning with InfoNCE for cross-modal alignment, which bring corresponding images and text embedding closer, and pull those that don't correspond farther away.With embedding pairs {( 𝐼 𝑖 , 𝑇 𝑖 )} 𝑁 𝐵 𝑖=1 , the image-to-text and textto-image similarity matrix can be calculated with:𝑝 𝑖 𝑗 ( 𝐼 𝑖 , 𝑇 𝑖 ) = exp(𝑠𝑖𝑚( 𝐼 𝑖 , 𝑇 𝑖 )/𝜏) 𝑁 𝐵 𝑗=1 exp(𝑠𝑖𝑚( 𝐼 𝑖 , 𝑇 𝑗 )/𝜏) (8) 𝑝 𝑖 𝑗 ( 𝑇 𝑖 , 𝐼 𝑖 ) = exp(𝑠𝑖𝑚( 𝑇 𝑖 , 𝐼 𝑖 )/𝜏) 𝑁 𝐵 𝑗=1 exp(𝑠𝑖𝑚( 𝑇 𝑖 , 𝐼 𝑖 )/𝜏) (9)where 𝜏 is a learnable temperature parameter that controls the concentration level of the distribution. the function 𝑠𝑖𝑚(•) measure the similarity of embedding with dot product.Then, the 𝑖 𝑡ℎ image-to-text similarity vector is denoted as p 𝑖 (𝐼,𝑇 ) = {𝑝 𝑖 𝑗 (𝐼,𝑇 )} 𝑁 𝐵 𝑗=1 and the text-to-image similarity vector is denoted as p 𝑖 (𝑇 , 𝐼 ) = {𝑝 𝑖 𝑗 (𝑇 , 𝐼 )} 𝑁 𝐵 𝑗=1 . Then, the hard label are used to calculate InfoNCE loss as follows:EQUATIONEQUATIONwhere y 𝑖 = {𝑦 𝑖 𝑗 } 𝑁 𝐵 𝑗=1 is one-hot label for 𝑖 𝑡ℎ pair with 𝑦 𝑖𝑖 = 1 and 𝑦 𝑖 𝑗 ( 𝑗≠𝑖 ) = 0. 𝐻 (•, •) is the cross entropy loss function. The final training loss of the original CLIP is calculated by averaging the Eq.( 10) and Eq.( 11) as follows:EQUATION3.4.2 Object-IoU. As mentioned above, the simple one-to-one assumption does not hold in practice. Different negative samples also have different similarities, and should not simply be assigned a label of 0. PyramidCLIP [10] utilize label smoothing to soften the hard label, which is naive and the improvement is limited. SoftCLIP [9] model the many-to-many relationship with the assistance of object detector, which is onerous and inefficient.Here, we exploit the parsed object sets 𝜋 𝑖 to cost-effectively construct many-to-many relations. Specifically, for the object set {𝜋 𝑖 } 𝑁 𝐵 𝑖=1 parsed from multiple captions in a batch, the similarity between 𝜋 𝑖 and 𝜋 𝑗 can be measured as:EQUATION)where | • | represents the size of the collection. Eq.( 13) calculates the similarity of two sets through the intersection of union (IoU) of 𝜋 𝑖 and 𝜋 𝑗 . Generally speaking, the greater the IoU of two sets, the greater the similarity.Based on the target similarity matrix 𝐼𝑜𝑈 𝑖 𝑗 , we align the image features with the original caption, visual description and test prompt respectively. For 𝑖 𝑡ℎ object set 𝜋 𝑖 , the target similarity vector is denoted as𝐼𝑜𝑈 𝑖 = {𝐼𝑜𝑈 𝑖 𝑗 } 𝑁 𝐵 𝑗=1. Following previous work [9] , we utilize KL-Divergence as loss function. The alignment of image caption 𝑇 𝑖 , visual description V 𝑖 and test prompt Q 𝑖 with images feature 𝐼 𝑖 can be expressed as follows:EQUATIONL 𝐼𝑜𝑈 -T2𝐼 = 1EQUATION)where T is variable and T ∈ { 𝑇 𝑖 , V 𝑖 , Q 𝑖 }. I(•) is indicator function, which is used to skip the situation where the visual description is ∅.Then, the object-IoU loss of the image and origin caption is the average of the above two formulas:EQUATIONFinally, the model is trained by a mixture of original CLIP loss L 𝐶𝐿𝐼 𝑃 in Eq.( 12) and object-IoU loss in Eq.( 16) as follows:EQUATIONDuring model inference, the text encoder can accept test prompts, visual descriptions, and captions at the same time, without any distribution shift.In the zero-shot classification experiments in this paper, we first perform inference on all categories using the test prompt to generate category prototypes. Then, for classes with lowest 𝛽 percentile, we generate visual descriptions for them and utilize visual descriptions to generate category prototypes, where 𝛽 is a hyperparameter set to 50% in our experiments. 
implementation details: 
 All models are trained in parallel with 8 V100 GPUs. The proportion of tail classes in the training set and test set, i.e. 𝛼 and 𝛽 is set to 30% and 50%, respectively. We use the AdamW optimizer with the learning rate set to 5e-4. The weight decay rate of AdamW is set to 0.2. Following previous works [9] , cosine learning rate scheduler with a linear warm-up is used to adjust learning rate, where the warm-up takes up about 10% of the total steps to increase the learning rate from 0 to the peak value, and then decreases with a cosine anneal strategy. Meanwhile, we use automatic mixedprecision [8] to train the model to save GPU memory. For CC3M and CC12M, we set the batch size to 1024. For YFCC15M-V2, we set the batch size to 2048. For caption object parsing, we implement it with LLM for comparison and detailed ablation studies can be found in experiments. For more details, please refer to our code.We verified the effectiveness of AlignCLIP on 3 different downstream tasks, namely zero-shot image classification, linear probing and zero-shot image-text retrieval. For zero-shot classification, we utilize the identical prompt templates outlined in the CLIP [23] paper, After the category name is combined with the prompt template, the features will be extracted with the text encoder and used as the category prototype. Images will be classified by calculating similarity with these category prototypes. For linear probing, the image encoder is freeze and a linear layer is added at the end of the image encoder for fine-tuning. For zero-shot image-text retrieval, caption and image features are extracted separately, and the search result with the highest similarity is output.In order to verify the performance of the proposed AlignCLIP, we first compare the performance of zero-shot classification. Here, the caption object parsing is implemented with Gemma.We first present the zero-shot classification results on ImageNet 1K in Tab.1. As we can see, the performance of our AlignCLIP has been greatly improved based on the original CLIP [23] . Our AlignCLIP achieves the highest performance across all pre-training datasets and different image encoder settings. Specifically, using ResNet50 as image encoder, AlignCLIP exceeded the original CLIP by 9.2, 11.4, and 7.1 points respectively when pre-trained with CC3M, CC12M, and YFCC15M. Meanwhile, our method also achieves the highest performance using other image encoders. Not only that, AlignCLIP also has advantages compared with other pre-training methods. Compared with SoftCLIP [9] and LaCLIP [7] , AlignCLIP has improved to varying degrees under different settings with less computing resources. In particular, LaCLIP needs to generate multiple augmentation for each caption, while ours only needs to generate visual descriptions parts of them.In addition, We show the zero-shot classification results of other 6 data sets, where PETS / DTD / F101 / FLOW / SUN / CAL are abbreviations for Pets / Describable Tex-tures / Food-101 / Flowers-102 / SUN397 / Caltech-101 datasets. AVG represents average accuracy. Following previous work [9] , we report the results of model pretrained with YFCC15M-V2. The results are shown in Tab.2. From the table, we can see that AlignCLIP achieves the highest performance in most cases. With Resnet50 as the image encoder, the average accuracy increased from 44.3 in the original CLIP to 50.0, and exceeded SoftCLIP by 0.8 points. Meanwhile, SOTA is also achieved with ViT-B-16 as image encoder.We present the results of linear probing in Tab. 3. Experiments were conducted on 7 datasets, namely Pets, Describable Tex-tures, Image-to-Text Text-to-Image Image-to-Text Text-to-Image R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 R@1 R@5 R@10 with AlignCLIP, the distributions of the three have been better aligned, and the distribution shift has been greatly alleviated. As can be seen from the table: 1) If the model is not aligned using visual descriptions during training, i.e. 𝛼 = 0, the improvement is smaller. 2) Even if 10% of the visual description is added to the training stage, the effect is greatly improved, exceeding the situation without adding visual description.3) However, it is not necessarily the case that a higher proportion of visual descriptions leads to better performance. We observe a sharp decline in performance when the visual descriptions in the test set reach 100%. 
conclusion: 
 In this paper, we introduce a novel image-text alignment training approach named AlignCLIP. To address the issue of CLIP's lower recognition performance for rare, tail-end classes, we propose utilizing visual descriptions to enhance these tail-end classes. Additionally, we introduce multi-domain alignment to synchronize the distributions of image captions, visual descriptions, and test prompts. To achieve this, we first perform caption object parsing on image captions to identify the object sets they contain, thereby generating samples for alignment. Thanks to the parsed object sets, our approach also implements object-IoU loss at a reduced computational cost, facilitating the computation of soft labels. Our method was pre-trained on multiple image-text datasets and evaluated across various tasks. The results demonstrate that our approach significantly outperforms the CLIP baseline and exceeds existing methods, establishing a new SOTA.  
 
 Appendix:  


Decision: The paper initially received three positive indications (2WA, BA) and one negative (R) indication. After the rebuttal, the ratings have remained almost unchanged, with positive reviewers keeping a positive indication (2WA, BA), and the negative reviewer raising his/her score while still maintaining a negative indication (WR). Most reviewers have appreciated the novelty of the approach and the soundness of the experimental results. Concerns raised by the negative reviewer were related to the absence of comparisons with the original OpenAI CLIP model. The AC, however, agrees with the authors and the other three reviewers that it is standard practice to re-train CLIP on publicly available datasets. The paper can be accepted as poster.

Reviews: review: paper_summary: paper_summary: The paper introduces AlignCLIP, a novel approach for enhancing CLIP models through multi-domain alignment and Object-IoU Loss. By leveraging caption object parsing, AlignCLIP generates visual descriptions and test prompts to improve recognition of rare classes. The method aligns visual descriptions, image captions, and test prompts to create category prototypes, leading to enhanced classification accuracy. AlignCLIP surpasses existing methods with lower computational costs, making it a promising advancement in vision-language pre-training.
strength_weakness: strength_weakness: 1, AlignCLIP introduces a unique method for addressing the long-tail distribution issue in CLIP models by aligning visual descriptions, image captions, and test prompts. This innovative multi-domain alignment approach sets it apart from traditional methods . The incorporation of object-IoU loss for generating soft labels based on the similarity between object sets in a batch is a novel contribution to the field.

2, The paper is well-written, presenting the methodology and results in a clear and concise manner, making it accessible to a wide audience . AlignCLIP's application in vision-language pre-training opens up possibilities for improved performance in downstream tasks by addressing the long-tail problem and enhancing alignment between visual and textual modalities.

3, The use of object parsing from image captions for multi-domain alignment is technically robust and offers a practical solution to the distribution shift problem during training . The inclusion of sufficient experimental evaluations across various tasks showcases the effectiveness and robustness of the AlignCLIP model.
limitations: limitations: 1,  The paper lacks a comprehensive and detailed explanation of the pre-training process in AlignCLIP. Insufficient elaboration on the specific pre-training methodology, model architecture, hyperparameter selection, and training procedures may hinder reproducibility and understanding of the pre-training phase. This lack of clarity makes it challenging to assess the benefits brought about by the pre-training process in AlignCLIP accurately.

2, Future experiments could be expanded to include a broader range of datasets beyond COCO and Flickr30K for more comprehensive evaluations.

3, While AlignCLIP builds upon the success of CLIP in vision-language alignment, it is known that CLIP may lack certain reasoning capabilities. Have experiments been conducted using other large language models (LLMs) in Align Multi Domains of Texts Input? Since LLMs are known for their enhanced reasoning abilities, could leveraging these models lead to improved alignment performance?
paper_summary: paper_summary: The paper introduces AlignCLIP, a novel approach to improve the performance of CLIP  models in recognizing rare and tail-end classes, which are often challenging due to their infrequent occurrence in training datasets. The authors propose a method that involves aligning multiple domains of text input, namely image captions, visual descriptions, and test prompt templates, to address the distribution shifts that occur between these domains.
strength_weakness: strength_weakness: 1. The paper introduces a novel approach to address the long-tail recognition problem in vision-language models, which is a significant challenge in the field. The concept of aligning multiple text domains (image captions, visual descriptions, and test prompts) through caption object parsing and the introduction of Object-IoU Loss is innovative.
2. The theoretical foundation of AlignCLIP is sound, as it builds upon the established framework of CLIP models while innovating to address specific limitations. The use of soft labels and many-to-many relationships between images and texts is a theoretically justified advancement over traditional one-to-one matching approaches.
3. The paper is well-structured, with clear explanations of the methodology, experiments, and results. Figures and tables are used effectively to visualize the performance improvements and distribution alignments, which aids in understanding the impact of the proposed approach.
limitations: limitations: 1. The evaluation is conducted primarily on existing benchmark datasets. There might be a need for further evaluation on more diverse or newer datasets to ensure the robustness of the approach. I'd like to take a look at the comparison on a dataset that contains a large number of rare classes, such as Things(https://osf.io/jum2f/).
2. The caption object parsing relies on part-of-speech tagging and large language models. The performance of AlignCLIP could be influenced by the quality and accuracy of these external tools, which may not always be consistent.
3. While the paper claims reduced computational cost due to the Object-IoU Loss, the overall training and inference process might still be resource-intensive, especially when using large language models for parsing and generating descriptions.
4. While the paper compares AlignCLIP with the original CLIP and some variants, it would be beneficial to see comparisons (such as FLIP) with a broader range of contemporary methods to better understand its relative performance.
paper_summary: paper_summary: In this paper, the authors introduce a novel image-text alignment training approach named AlignCLIP. To address the issue of CLIP’s
lower recognition performance for rare, tail-end classes, they propose utilizing visual descriptions to enhance these tail-end classes.
Additionally, they introduce multi-domain alignment to synchronize the distributions of image captions, visual descriptions, and test
prompts. To achieve this, they first perform caption object parsing on image captions to identify the object sets they contain, thereby
generating samples for alignment. Thanks to the parsed object sets, proposed approach also implements object-IoU loss at a reduced
computational cost, facilitating the computation of soft labels. The proposed  method was pre-trained on multiple image-text datasets and evaluated across various tasks. The results demonstrate that proposed approach significantly outperforms the CLIP baseline and exceeds existing methods, establishing a new SOTA.
strength_weakness: strength_weakness: a) The experiment in this paper is thorough, and the presentation is relatively clear.

b) The proposed  method significantly surpasses the CLIP baseline and exceeds existing methods, achieving a new SOTA.

c) The motivation of the paper is reasonable, concise, and effective.
limitations: limitations: a)  The rightmost part of Figure 2 is unclear and not intuitive, it needs to be revised.  

b) The comparative methods in the table need to be accompanied by reference numbers for easy reading.
paper_summary: paper_summary: The paper propose a method to overcome the low performance of CLIP on longtailed concepts. They use existing techniques of generating visual descriptions for rare classes using LLMs. Then they hypothesise that the train, test, and visual description captions have a distribution shift which does not allow to effectively learn the new descriptions. The authors propose to align the three caption types to improve the performance on longtailed classes.
strength_weakness: strength_weakness: Performance improvement in the setting of small scale pretraining.
limitations: limitations: 1) The class wise performance analysis is done on CLIP trained from scratch by the authors on CC3M dataset and not on the original CLIP, which does not represent the statistics of the original model. How does the statistics hold true for the original CLIP?

2) The entire paper is built upon the hypothesis of distribution shift, performed on small scale CLIP trained from scratch on the CC3M. While, CLIP is trained on 400M internet captions, how does it translate on large scale CLIIP.

3) The evaluation and comparison is also done on baseline CLIP trained on CC3M which again might not translate on large scale.

4) How do you claim SOTA when you don't compare at large sale CLIP?