VppCap: Text-based Visual Prompts in Probabilistic Embedding Space for Lightweight Image Captioning
Authors: Taewhan Kim, Soeun Lee, Si-Woo Kim, Dong-Jin Kim


Reviewer paper summaries: In this paper, it is hypothesized that refining visual features is crucial for mitigating the modality gap. To address both the modality gap and ambiguity issues, VppCap, a novel approach for generating visual prompts from text, is proposed. VppCap offers several benefits:

i) It mitigates the modality gap within the probabilistic embedding space with minimal parameters, enhancing visual representation through text-generated visual prompts.
ii) It improves performance by integrating with models requiring a visual encoder.
iii) It serves as a prompt-agnostic framework, demonstrating enhanced performance on both simple hard prompts and retrieved text prompts.
Summary of strengths: The VppCap model exceeds the performance of other lightweight models on the COCO dataset and outperforms larger models on the NoCaps dataset.
Summary of weaknesses: The paper is not well-prepared; the motivation for the proposed method is unconvincing in addressing the issues, and there are numerous errors and typos.
Comments, suggestions and typos: The concept of "Visual Prompt" is confusing, as it appears that VppCap employs the CLIP text encoder, which takes text prompts derived from a set of similar image captions.

Additionally, the mapping network is just a standard decoder architecture comprising self-attention, cross-attention, and a feed-forward network. Are there any novel improvements to the mapping network?

Furthermore, in Figure 3, there is a large "Fire" symbol with the text "Cross-attention" underneath it. This is confusing because it is unclear what kind of network with parameters is to be trained, even though it is a crucial trainable module in the entire architecture.

There are numerous errors and typos in the paper. For instance, in Table 4, the delta for the first two numbers (0.4, 0.4) is calculated incorrectly. Additionally, in Table 1, the unit for the numbers in the "Training Param" column, such as “675,” “632,” and “338,” is not specified.
Overall assessment scores: 2.5