Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The visual clues paper tries to leverage the recent advances in both vision-and-language and natural language understand fields to generate comprehensive and rich descriptions of the images. The model works in three steps: (i) generate a set of visual clues (i.e. information) about the image to embed or encode rich visual information (ii) pass these visual clues as a prompt to a large language model to generate candidate paragraphs (iii) select the best paragraph from the candidates. The paper also purposes a new metric to better evaluate long descriptions using scene graph comparisons. The human and automatic evaluations both suggest that the descriptions generated by this method are almost on par with the human annotations.
strengths_and_weaknesses: Legend: S (Strength), W (Weakness), C (General comment)
- C: The paper is very well written and easy to understand. Though, there are some things which are not clear and not well-mentioned probably due to the fact that some of the models used in this paper are proprietary. 
- S: The paper connects the recent advances in the vision-and-language and NLP fields really well by leveraging almost only the pretrained models to generate very coherent and comprehensive descriptions of the images. Since there is almost no training involved, it further suggests how unimodally strong models can be bridged across modalities leveraging abundantly available unimodal data in the world.
- S: The approach to generate visual clues is quite exhaustive and covers various sources of visual information: tags using a contrastively trained VL model, object detection model whose bounding box classes are again calculated by the contrastively-trained VL model, and a full-image caption generator. 
- S: One important contribution of this paper is a new automated metric for evaluating longer paragraph descriptions of the images as existing n-gram based metrics are not suitable for the variety seen in longer descriptions. The metric uses scene graph based similarity and is an extension to SPICE metric.
- S: Human evaluations on the outputs of the model suggest that model's outputs are almost on par in terms of completeness and humanlike but fall behind in accuracy and coherence.
- W: The previous point brings me to one of the weaknesses of this approach is that since we are relying on pretrained models in the complete pipeline, any errors are eventually compounding as we go across the pipeline which means that if any wrong content is passed via visual clues to the text generator, it would add a compound on top which is why maybe the accuracy and coherence are low.
- W: It is hard to understand the significance of SPIPE without seeing results on other metrics for the baselines as well. As of now, only SPIPE results are reported and there is no discussion on why ngram metrics probably don't work on the paragraph comparison.  
- W: The model seems to ignore scene text in the image which might be an issue in real world scenarios as scene text is omnipresent. Even when you are reading this review, you are reading scene text.
- W: There have been works in past such as VisualGPT [1] which try to adapt the language model directly to image captioning. A comparison or discussion with them would be helpful.
- W: The baselines except BLIP are somewhat weak. Even BLIP when finetuned on COCO captions would be more inclined to generate shorter captions. This is also evident from the examples in supplementary. Since COCO captions are small, it would make more sense to use and compare against captioners trained on dataset which contain larger image descriptions such WIT and Localized Narratives which the paper doesn't do as of now. Most of the models used as baselines are tuned for generating single sentence captions.
- W: Most of the models used in this paper are proprietary and not available to the public so it is hard to make any comparison down the line. What would be interesting is to use CLIP instead of Florence along with a publicly available large language model to see how this approach performs.

[1] Chen, Jun, et al. "Visualgpt: Data-efficient adaptation of pretrained language models for image captioning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
questions: - It would be great to see the exact visual clues as well for the images in the supplementary. Without that context, it is hard to understand what was passed to GPT3 to generate the description.
- Will any information about the 400K top bing queries be provided?
- There is no basis behind claiming that n-gram based metrics don't work for larger paragraph generations for images.
- When the tags are matched with the images, are the tags directly matched or is some sort of prompt used?
- What is source/intuition behind the paragraph on line 38
- I disagree with footnote 2 and believe that this overlap is somewhat impactful. The Visual Genome captions are not just semantic propositions but are full captions. Also, in a single image, there are multiple crops for which we have the captions. Now, when these crops along with their captions compound, they do provide a lot of information about the image which is non-trivial leak
- On line 292, how much BEST-generated data was the BLIP-large model trained on?
limitations: Yes, the authors have discussed the limitations. In general, this model because it is developed on proprietary models will not be available so can't have direct impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a general framework (BEST) for generating semantic visual clues by capturing information through tags, captions, object detections by leveraging existing pre-trained visual LMs and textual LMs. The framework enters them in an ascending order of their quality (highest quality predictions at the end): local descriptors, captions, and tags as input to a large-scale LM to generate K candidate paragraphs. The last step leverages the vision language model to select the top K candidates that align best with the image.  The paper presents an extensive set of experiments against state-of-the-art to show the effectiveness of the proposed framework. The paper has both ablation and qualitative results to demonstrate how each component contributes to the final performance and quality of the generated paragraphs. The paper also proposed an evaluation metric: SPIPE that captures Accuracy, Completeness and Coherence.   

strengths_and_weaknesses: Strengths: 
1. Overall, the paper is well written and is easy to read.
2. The paper presents a general framework (BEST) for generating detailed image caption paragraphs. The quantitive/qualitative evaluation results show the effective of BEST against BLIP-large and Socratic model in Table 1. 
3.  Ablation results in Table 2 show the effectiveness of each component of BEST and Table 3 captures Accuracy, Completeness and Coherence components of BEST against Ground-truth/BLIP/Socratic.
4. Fine Tuning BLIP-large also improves results as shown in Table 4.
5. The paper presents many real-world setting where BEST could be useful. May detailed results are presented in Figure 5. 
6. The Section 7 clearly calls out limitations and further improvements to the BEST. 
Weakness: 
1. Given Socrate model is solving a similar problem: it will be good to discuss how and why BEST is so much better than Socrates model in evaluation results. 
2. In related work, more details around how BEST model is different than Socrate model with be different. 
3. Given, there are many pre-trained model involved and also fine tuning is presented. How much it costs in terms of time/resources to train BEST? How much more expensive is training of BEST model when compared to Socrate model?  
 
questions: Detailed questions/comments

In Figure 2, could authors highlight what color is being used to represent the text encoder as f_t(.)? 

Line 35: Florence reference is missing. Please refer at the first introduction. There is a reference later e.g. on Line 106. 

Line 112: How are tags generated {t_i}_{i=1}^{N}? 
Section 3.1 add/create sub-subsections or bold headings to improve organization. 

Line 147/148: Could more intuition be added to what sentences are kept or removed from Candidate selection Step?

Line 204: Some details on how tau temperature needs to be adopted in practice (intuition is needed)? What happens if \tau is 100 vs 0.8 as used in the paper? 

Line 206: Tags are discussed. How you refer this back on Line 113 where it was unclear how these N tags are selected. 

Some more clarification what are the exact inputs to BEST will improve reproduction of the results. What models need to be re-trained/re-run etc? As the paper is leveraging most  of the pre-trained models. It will be good to summarize in a Table/Figure how each component is participating in the final BEST model. 
  
limitations: Already captured in Section 7 of the paper. Authors could  add some "additional societal biases that textual language models suffer from". Add some details on pros/cons of training these large models to environment.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new framework for generating image paragraph captioning in this paper. The framework extracts text descriptions from images using different pretraining modules, subsequently feeds these extracted modules to a language model for paragraph generation, and finally selects the optimal paragraph from several candidate results using a cross-modal selection module as the final result. Also, to better measure this task's results, the authors propose a new evaluation method SPIPE based on the SPICE evaluation method.
strengths_and_weaknesses: Strengths
1. The framework proposed in this paper can generate a paragraph for a given image, which is much longer than the previously widely used image caption.
2. The authors propose a new evaluation metric SPIPE to measure the results of IPC.

Weakness
1. Lack of comparison with other traditional IPC methods. There are many previous IPC methods mentioned in the Related Works section.   but their results are not compared with the proposed methods' results in the experimental section.
2. I think this work seems like a direct extension of PICa to the IPC task. The novelty may be limited.
3. There is no experimental proof in the article that the new evaluation metric SPIPE could evaluate IPC better than the previous metrics, such as SPICE and METEOR. There is also no analysis of whether the SPIPE evaluation metrics can evaluate different aspects. Besides, SPIPE needs human-annotated graphs for each paragraph in the dataset, which limits the generalizability of this metric. As a result, only using the SPIPE metric in all the experiments is not convincing.
4. In Table1, compared with BLIP-large, BEST–general domain improve 1.2 in F-score. Due to more processes (Candidate Synthesis&Selection) and higher demands on computing resources (GPT-3), I don't think the improvement is significant.

questions: See the “weakness”.

Hope the authors improve this paper according to the “weakness”.
limitations: The authors have addressed some of the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper introduces a new method for image paragraph captioning, where the goal is to generate long, coherent and informative textual descriptions of an image. Their method uses several large pretrained models, combined without the need for further training. From the image, open-vocabulary models like CLIP generate tags, object detectors generate a list of objects along with their locations, and image-captioning models generate (shorter) captions of the entire image or image regions. All of these outputs are combined into a natural language prompt (called "visual clues" by the authors), which is then fed to a language model like GPT-3. The language model then generates multiple candidate captions from the visual clues, which are then ranked by a contrastive model like CLIP. The authors also propose SPIPE, a new evaluation metric for measuring performance on image paragraph captioning. Using this metric, the authors show that their method outperform other baselines. The authors also conduct human evaluations, showing that their method typically outperforms previous ones, and can be competitive with human annotations.
strengths_and_weaknesses: **Strengths**

1) The ideas presented in this work are (to the best of my knowledge) novel, and the research direction is timely.

2) The experimental setup and results are solid.

3) The paper is very clear and well written.

**Weaknesses**

1) One concern is that a great portion of the claims that their method is better than previous work rely on a metric proposed by this own work.  The authors explain why other evaluation metrics might not be ideal, but I believe it would still be informative to report them in the paper, especially since there exists image paragraph captioning benchmarks that use other evaluation metrics.
questions: 1) Is SPIPE correlated with human perception?

2) Have authors considered exploring tasks beyond IPC? For instance, VQA.
limitations: To the best of my knowledge, authors adequately discuss the limitations of their work.
review_title: null

Meta-review: metareview: All three reviewers have voted weak accept to this paper; the authors have engaged well with the reviewers and have improved their paper. I also recommend acceptance. 
confidence: Less certain
Decision: Accept