Image-like retrieval and frequency-based entity filtering for text-only captioning
Authors: Soeun Lee, Si-Woo Kim, Taewhan Kim, Dong-Jin Kim


Reviewer paper summaries: This paper the challenge of generating image captions using text-only training methods and introduces a novel approach called Image-like Retrieval to bridge the modality gap between text data used in training and image data used during inference. Additionally, it proposes a Frequency-based Entity Filtering technique to improve the accuracy of generated captions. These methods are integrated into a unified framework named IFCap, which outperforms state-of-the-art methods in both image captioning and video captioning tasks compared to zero-shot captioning based on text-only training.

The paper introduces IFCap, a framework for zero-shot captioning that leverages text-only training methods to address the modality gap between text data during training and image data during inference. IFCap combines two key components: Image-like Retrieval (ILR) and Frequency-based Entity Filtering (EF). ILR aims to align text features with visually relevant features, mitigating the modality gap by introducing noise into text embeddings to make them more image-like. EF enhances caption quality by filtering entities based on their frequency in retrieved captions. The proposed methods are integrated into a unified framework, IFCap, which has shown some improvements over state-of-the-art methods in both image and video captioning tasks.
Summary of strengths: - **Novel Methodology**: The introduction of Image-like Retrieval and Frequency-based Entity Filtering addresses significant gaps in existing text-only training methods, making the approach innovative and impactful.
- **Superior performance**: IFCap demonstrates superior performance across various benchmarks.
- **Generalization**: The approach's ability to generalize across different domains and tasks, including video captioning, adds to its robustness and utility.

S1: The paper introduces a novel method, Image-like Retrieval(ILR), which effectively addresses the modality gap in zero-shot captioning by aligning text features with image features using noise injection.

S2: The Frequency-based Entity Filtering(EF) technique improves caption accuracy by filtering frequently occurring entities in retrieved captions, providing more relevant information to the captioning model.

S3: The authors conduct extensive experiments on various datasets, including COCO, Flickr30k, and NoCaps, showing the robustness and effectiveness of their approach.
Summary of weaknesses: - Can this be applied to stronger VLMs such as OFA?
- It would be interesting to see how the modality gap is addressed by noise injection.

W1: The optimal value for the noise parameter in ILR is determined heuristically, which may not be practical or efficient. This reliance on heuristic tuning suggests a lack of robustness and scalability in real-world applications. It would be more convincing if the authors had proposed a systematic or automated method for tuning this parameter.

W2: While the paper introduces ILR and EF, the core concepts of retrieval-augmented generation and entity filtering are not entirely new. The novelty lies more in the specific implementation and combination of these methods rather than in fundamentally new techniques. The contribution is a bit incremental.
Comments, suggestions and typos: This paper is well written and has good experiments and ablations.

Please see the weakness.
Overall assessment scores: 3.5

3