Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work introduces to construct an input pyramid with different semantic levels for each modality. It then aligns visual elements and linguistic elements in a hierarchical way. The proposed PyramidCLIP outperforms CLIP with a large margin. 


strengths_and_weaknesses: Strength:
- The results are very promising. The model outperforms the sota methods across many datasets.
- The overall framework is easy to follow.

Weakness:
- The idea of introducing ROI features may increase the computational cost significantly. 
- In peer-level semantics alignment, the authors introduced coarse-grained global contrast learning and fine-grained local contrast learning. However, I didn’t find the more studies to validate that combining two contrast alignments helps model training.
- The authors described the cross-level relation alignment in Sec 3.3. Three structures are proposed but I suggest the authors providing more analysis to prove their effectiveness.
questions: Please see my comments in the weaknesses part.
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Under the contrastive CLIP learning framework, the work proposes to utilize more fine-grained information to produce multiple views of both the image and text during training, and hence constructs more contrastive loss terms across different views. During inference/evaluation, only the standard view is used.

Empirically, with 3 different architectures (ResNet50/ViT-B32/ViT-B16), different pretraining data scales and several down-stream datasets, authors show that the proposed approach achieves clear gain over the baseline systems. 
strengths_and_weaknesses: The work follows a natural motivation and achieves very good empirical gain over some strong baseline systems. Overall, the paper is well written and the empirical study is solid.

A key concern I have is whether the comparison is fair enough. If I understand correctly, for each view of either the image or the text, we need to feed the view into the model once, which leads to roughly 2x - 3x additional computation cost. Again, if my understanding is correct, despite using the same batch size, the actual pretraining cost might be much higher for the proposed method than baselines, making the comparison less information. A better comparison seems to be make the batch size of baselines larger until the training cost is comparable.
questions: If my understanding above is correct, how would the comparison look like if similar training cost is invested in training the baselines, particularly with soften and LeFF disabled? (NOTE: In my opinion, this is very critical for judging whether the proposed method is really useful. Hence, my final review will largely depend on this.)
limitations: I don't see any particular problem here.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes hierarchical feature alignment for vision language pre-training, called PyramidCLIP, which alleviates semantic mismatch as well as mutual compatibility problems, i.e. false positives and false negatives. PyramidCLIP constructs inputs with three levels of semantics in visual and language modalities respectively and then resolves semantic mismatch through peer-level semantic alignment and cross-level relation alignment. In addition, PyramidCLIP adopts a soft form of InfoNCE to deal with mutual compatibility.
strengths_and_weaknesses: Strengths:

1. This paper is well written and easy to follow.

2. The motivation and solution of the article are clear. More precise hierarchical feature alignment for tackling semantic mismatch, and softening InfoNCE for mutual compatibility.

3. The experiments are well designed, and the results are excellent.


Limitations:

Have the authors adequately addressed the limitations and potential negative social impact of their work? If not, please include constructive suggestions for improvement. Authors should be rewarded rather than punished for being upfront about the limitations of their work and any potential negative societal impact.

1. Needs more explanation about how the training set is constructed.

2. In order to compare with some recently published works(e.g. [1]), it is recommended that the author can supplement the results on smaller scale datasets, such as cc3m.

3. The categories obtained by object detection are simply joined by commas. Whether different joint forms have an impact on the results (e.g. splicing with spaces)? 

[1] Robust Cross-Modal Representation Learning with Progressive Self-Distillation. CVPR 2022.
questions: See the weaknesses.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes to improve CLIP by adding contrastive losses at different levels. Specifically, for each image in the original data, they construct image views at global, local, and region levels, and also create their corresponding text captions. Therefore, for each image-caption pair in the original dataset, they can create two additional image-caption pairs and the model is trained with additional contrastive losses given the newly constructed data. They also soften the original contrastive loss with a label-smoothing-like technique. Experiments demonstrate improvements over several baselines on zero-shot retrieval, linear probing on image classification,  object detection, and semantic segmentation tasks.
strengths_and_weaknesses: Strengths:
1. The empirical results are strong as they can outperform several popular baselines.
2. The paper is well-structured.
3. I like the idea of softening the contrastive loss objective, which makes sense and the paper shows it works well.

Weaknesses:
1. The comparisons between their model and baselines may not be fair. The paper uses multiple pre-trained models for their model (e.g. text summarization and object detection models) which can introduce more supervision. Also, because their method will create more image-caption pairs for their model, they would have more pretraining data than the baselines.
2. Some designs are questionable. For example, the text captions in image-caption datasets are typically short, whereas they use a pre-trained text summarization model, which is designed for summarizing long documents, to further shorten the captions, which does not make sense to me.
3. While it is true that the pre-training datasets can be noisy, the problem may be alleviated if sufficiently large data are used. It would be good to see if they can keep the performance gain as more data are included. For example, they can plot performance-data size curves for both their model and baselines and see the performance gap when the data size varies.
questions: Would it make more sense to use a keyword extraction model than a summarization model?
limitations: I am not sure if their method can be scalable as when more data is included, many of the problems mentioned in the paper may be alleviated.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new framework called PyramidCLIP for vision-language pre-training (VLP). PyramidCLIP first extracts multi-level features from both the visual and linguistic domains and then conducts contrastive learning by aligning visual and linguistic features in both peer-level and cross-level ways. In this way, the vision and language representations learned by PyramidCLIP encode better image-text alignment, which alleviates the semantic mismatch problem that exists in the image-text pairs for pre-training. Moreover, the authors also replace the loss term of the negative samples in contrastive learning with a softened version, to tackle the problem that different image-text pairs may have potential correlation. The empirical results show that PyramidCLIP clearly outperforms the CLIP baseline in a variety of downstream tasks and also achieves SoTA performance on several downstream tasks, as compared with other VLP models.
strengths_and_weaknesses: ### Strengths
* This paper addresses the problem of the quality of image-text pairs (which is an important problem in VLP) using hierarchical feature alignment. The core idea and specific designs of the proposed PyramidCLIP are generally reasonable.
* The authors conduct extensive experiments to demonstrate the effectiveness of PyramidCLIP, covering different backbone architectures (ResNet50 and ViT-B), pre-training datasets of varying sizes and a variety of downstream tasks (zero-shot image classification, zero-shot image-text retrieval, linear probe, object detection and instance segmentation).
* The ablation study is comprehensive, showing that all the components of the proposed method are conducive.
* Qualitative analyses are conducted to intuitively show that PyramidCLIP learns better vision and linguistic representation than CLIP.
* The paper is generally well-written and easy to follow.


### Weaknesses

1. Some specific designs of PyramidCLIP seem counter-intuitive.
* Random crop cannot guarantee the quality, especially for the local view. For example, if the local view is irrelevant to the textual description, minimizing the distance of the corresponding features may confuse the model.
* It is unclear how the cross-level alignment helps the modelling of relations between salient objects. For example, how does the model know that "a table is next to a chair", by contrasting the ROI features to the textual summary or the original text? It is better to provide an intuitive explanation.

2. The difference between PyramidCLIP and existing VLP methods that also introduce multi-level semantics is not clearly discussed. In the related work, the authors state that "Different from methods mentioned above, each level is input to the corresponding encoder individually, without concatenating." Such a difference seems trivial and incremental.

3. Many methods described in the related work are not compared in the experiments, especially the methods that also introduce multi-level semantics (e.g., MVPTR and X-VLM).

questions: * Could you provide an explanation about how the cross-level alignment helps the modelling of relations between salient objects?
* Could provide a more detailed discussion on the difference between PyramidCLIP and the VLP methods that also introduce multi-level semantics?
* Why not compare PyramidCLIP with MVPTR and X-VLM?
limitations: * The soften objective function treats all the negative samples equally using label smoothing, which is suboptimal considering different image-text pairs should have different degrees of correlation.
review_title: null

Meta-review: metareview: This paper proposes PyramidCLIP.  It improve contrastive learning method CLIP with more fine-grained information to produce multiple views of both the image and text during training. During inference/evaluation, only the standard view is used. The empirical results with different network architectures at different pretraining data scales show that the proposed Pyramid achieves clear gain over the baseline methods.

The paper is comprehensively discussed, and receives unanimous accept from all reviewers, leading to an ``Accept'' decision. The authors are highly encouraged to revise the paper accordingly. The authors reported results on a customized benchmark, and showed improvement based on its own baseline. In the future, the authors are highly encouraged to report results based on a common benchmark below [*], so that readers can clearly see the position of PyramidCLIP in the context of all other similar papers in the literature.

[*] https://computer-vision-in-the-wild.github.io/eccv-2022/
confidence: Certain
Decision: Accept