SubCLIP: Sub-caption Inclusivity Objective for Vision-Language Alignment
Authors: Yassine Ouali, Ricardo Guerrero, Swathikiran Sudhakaran, Adrian Bulat, Brais Martinez, Georgios Tzimiropoulos


Reviewer paper summaries: Brief Summary: The paper suggests that VLM models like CLIP often fail at processing input phrases of varying lengths and granularity. To circumvent this issue, the authors suggest a new training loss to capture the sentence hierarchy and additionally fine-tuning on a synthetically curated dataset (from existing datasets). 

Experiments are conducted on image-retrieval, classification and compositionality understanding tasks with the proposed method SubCLIP outperforming competitive baselines such as OpenCLIP and SigLIP by significant margins.
Summary of strengths: Pros:

1. The paper tackles an important task of alleviating the issue of VLMs to process varied length phrases. Being able to use VLMs like CLIP for image tagging can be helpful. As such, the task is well motivated.

2. The proposed model outperforms competitive models on various tasks such as compositionality understanding on SugarCrepe, image-text retrieval on Flickr and COCO, as well image tagging on OpenImages and HICO.

3. A detailed ablation study is provided in Table 1,2 capturing the utility of the sub-caption loss.
Summary of weaknesses: Cons: 

1. The comparison of SubCLIP with models other than SigLIP in Table 4,5,6 is a bit unfair given the base model is trained on more images than its counterparts. Ideally, the authors should show that the formulation can be applied with other initializations from CLIP, EVA-CLIP and OpenCLIP respectively and show corresponding improvements. 

2. One thing I am unclear is whether this is identical in formulation to training original CLIP formulation but with a new dataset of image-text pairs with images repeating for each subcaption, just that the proposed formulation is much more efficient in computation. In that case, the authors should also compare with such a baseline. 

3. How much does the quality of the sentence parser affect the training? Perhaps some ablation with noisy sentence parsing outputs would be helpful.

4. (Minor) It seems like the RAM++ formulation is more suitable for image-tagging which has the most significant distribution shift. And RAM models are able to achieve better performance than the sub-caption method on the Common part but not on the Uncommon part. Why is this the case? The authors report this finding in L551 but don't expand on this finding.
Comments, suggestions and typos: There is some random space after L167.
Overall assessment scores: 3