Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper expands prompt engineering techniques to multi-modality scenarios. Specifically, the paper sought to find how best to prompt a set of unimodal and multi-modal models to solve multi-modal tasks that are difficult to solve independently or would require large amount of training data to tackle. The paper also creatively demonstrated the power of in-context substitution where information from non-language domain is substituted into an LLM for contextual reasoning. The paper applied the developed techniques to different multi-modal tasks including image captioning, contextual image description, video-to-text retrieval, and others such as egocentric perception, multi-modal assistive dialog, and robotic perception and planning. The paper also introduce a novel zero-shot evaluation techniques that could be adopted for model selection purposes.
strength_and_weaknesses: Pros:

The paper is generally easy to follow, well motivated, and shows creative tricks to prompt combination of multi-modal models and LLM to solve multi-modal reasoning tasks. The paper attempted to formulate the multi-modal prompt engineering as traversal of a computational graph which is novel and intuitive. The paper also shows good zero-shot performance over strong baseline.

Cons:
While the proposed multi-modal prompting is useful to prompt constrained LLM, VLM, ALM, figuring out the model sequence and the prompt structure to get the best outcome is not trivial. For the cases where the training data is publicly available, could it be possible wouldn't it be best to to just training task specific models. Also, in the case where the multi-modal data is not possible to generate, e.g, egocentric perception, multi-modal assistive dialog, and robotic perception and planning, could it be more useful to use this technique to generate a multi-modal training data that can be used to train task specific model?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and easy to follow. Authors also provided links to Colab where their work could be easily reproduced.
summary_of_the_review: The paper demonstrated that multi-modal prompt engineering is a viable option for utilizing publicly available multi-modal models at zero-shot without retraining a new model for a specific multi-modal reasoning task. The paper is well motivated and experimental results looks convincing.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper explores multimodal prompt engineering for zero-shot multimodal reasoning. Using language as the intermediate representation, the authors propose Socratic Models (SM) where outputs from different modalities are composed into textual prompts to provide zero-shot results on image captioning, description generation and video-to-text retrieval. The paper also discusses potential applications of these methods for dialog, QA and robot perception and planning. 
strength_and_weaknesses: Strength:
- This paper is well-motivated where zero-shot multimodal reasoning is an important research area with vast industrial and research applications.  
- The authors promise the availability of corresponding code and also provide anonymous colab notebooks which is essential for reproducibility. 
- Model-generated examples provide detailed insights into what the prompts and generated outputs were. 


Weakness:

- One might argue that technical novelty is limited in the paper with not much significant research contribution. 
- The vast claims of zero-shot applications go on for 4 pages (Page 6-9) without much experiments and analysis to support them. 
- Continuing on the point above, I believe that the paper needs a major revision where Section 4 could be expanded in more detail with more experiments. For example, Appendix B.2 provides nice examples of ablation where information from VLM is left out and should be a part of the main section compared to unjustified claims in Section 5. Even some applications from Section 5 could be absorbed in Section 4 only when provided with supporting experiments and ablation studies. 
- This paper seems like a mix of position and experimental paper while doing justice to neither. 
- Even though the zero-shot results might be a promising direction, currently the 0-shot results seem far from industry standard applications where there is a huge gap between all the automatic metrics from the baseline method (6.9 vs 40.97 BLEU, 44.5 vs 152 CIDEr)
- It is not clear how prompts were designed for few shot methods like the ones described in Section 4.1 
- Similar experiments with few-shot techniques for Section 4.2 and 4.3 would have made the results more compelling. 
- Even with few-shot learning, have the authors experimented with providing more than 3 examples and comparing the evaluation results as the number of examples is increased. 
- Table 1, Bleu-4 of exact 0 value for ZeroCap model seems like a bug in the experiments. Could the authors please recheck?
- It is not clear how the videos are processed in Section 4.3? Does the model work on a per frame basis? How is CLIP(caption) computed and how does it differ from CLIP(video)? 
- Human evaluation is not provided for the different experiments. 


Questions:
- Could the authors discuss and clarify how they design these multimodal prompts? 
- As the authors also discuss in Section 2 (page 3), simple prompts modifications have a huge impact on performance. Did the authors thus experiment with different wording of the prompts? 
- An ablation on different prompts would be more compelling. How would the results compare when object, place categories or the number of people is omitted from the prompt in Section 4.1? Does the model always generate 3 object and place categories from the VLM? 
- In Section 4.1 (Page 4), have the authors also experimented with nucleus sampling or beam search? How do the results compare against the greedy search? 
- Have the authors also experimented with different prompts for description generation against captioning bot? Example prompting with `I am an intelligent image describing bot` compared to `image captioning bot` to promote longer descriptions and differentiate between the two tasks?
- Could the authors clarify how the long-transcripts subset of the dataset was created and the statistics of the dataset?


Suggestions/Comments:
- Please take care of using citet compared to citep and vice versa (natbib style) appropriately (especially in Section 7 and Appendix). 
- Section 1 can be improved by providing relevant citations on page 2 of the baselines producing results 11.3 and 40.7 R@1 even though they are discussed much later in Section 4.2 on Page 5. 
- Pictorial depiction of the models either in the main section or in the appendices would increase the understanding and interpretability of the proposed approach.

clarity,_quality,_novelty_and_reproducibility: Please see strengths and weaknesses for more details. Availability of corresponding code ideally makes the work reproducible while there is limited technical novelty. The paper is written clearly in most parts. 
summary_of_the_review: The results seem promising; however, the paper could be improved in the presentation and more detailed descriptions. Please see strengths, weaknesses, suggestions and comments. It would be nice to see a revised version of the paper in the future conferences.

--- Rebuttal update ---

I believe the authors have invested fair amount of time to address most of the questions/weaknesses. Even though the 0-shot results are pretty low, they still serve as baseline against other related work. Based on the rebuttal, I am increasing my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies how to incorporate vision language model (VLM), audio language model (ALM) and large language model (LLM) to perform joint predictions for multimodal tasks. The integration of different LMs are based on various language prompts. Results show that the proposed method achieves promising zero-shot and few-shot performance. In addition, the proposed method can be applied to many novel zero-shot multimodal applications.
strength_and_weaknesses: Strengths
1. Interesting methods by effectively incorporating LMs from different modalities
2. Strong results on zero/few-shot learning
3. Demonstrate interesting examples on many zero-shot applications

Weaknesses
1. Combining different LMs would make the inference speed much slower, which could be an obstacle for the application.
2. Section 5 only provides several examples without statistical evaluation results. It it hard to evaluate the model's performance in this application, since the examples could be cherry pick.
3. In the image captioning evaluation (MS COCO) in Table 1, there is still a large gap between SMs and ClipCap. More few-shot baselines are needed for evaluating the image captioning quality of SMs.
clarity,_quality,_novelty_and_reproducibility: This paper is well-written, the proposed idea is original. The proposed method is novel and well-motivated and this paper provides evaluations on multiple tasks, but lack enough statistical evaluations in many zero-shot applications (Section 5).
summary_of_the_review: This paper proposed Socratic Models which aim to incorporate VLM, ALM and LLM for zero-shot multimodal tasks. The proposed idea is well motivated and novel and the experiments show promising and strong zero-shot results. However, this method might slow down the inference speed, and this paper lacks enough statistical evaluations in many zero-shot applications.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose composing multimodal models with language models
in a process they call "socratic models". The key idea is that the
outputs of VLMs/ALMs can be reformulated in text, which can then be
fed to a LLM via a prompt. While each instantiation of a socratic
model is slightly different, this family of approaches can serve as a
surprisingly strong zero-shot baseline for multimodal tasks. The
authors consider several creative compositions and demonstrate that
their general approach is more effective than prior efforts for zero
and few shot multimodal models.
strength_and_weaknesses: My favorite part of this work is the creativity of the compositions.
For captioning, the authors compose several types of object detectors
(scene type, person counter, object detector, etc.), format the
outputs of that in a LLM prompt, which proposes several captions,
which are then re-ranked by CLIP: the resulting model is SOTA for
zero-shot captioning, and even competitive with models that fine-tune
on upaired captions. The rest of the examples are even more creative,
ranging from video retrieval to egocentric video summarization, to
multimodal assistive dialogue, and beyond.

Other positives:

- I think these things make for great baselines, like the authors
  said. I think most papers on multimodal modeling should include one.

- I like that the features passed between models are in text format,
  i.e., they are more interpretable compared to visual features from
  an vision model directly.

- I like that the decoupled nature of these compositions means that
  performance improvements outside of multimodal modeling (e.g.,
  better language-only models or better vision-only models) translate
  directly to improvements in socratic models.

My biggest technical concerns are:

- The granularity of the outputs of the vision/audio models, when put
  into text, may not be sufficient for every downstream task ---
  because information is lost, e.g., when making an object
  classification vs. the whole image, socratic models may hallucinate
  moreso than other approaches. The authors are aware of this, and
  it's interesting to think about how to fix this.

- The design of the prompts, compositions, etc. seem quite ad-hoc,
  i.e., it seems like there's probably a lot of "art" to creating a
  good prompt/set of attributes to textify from a VLM, etc.

My biggest presentation concern is:

- Are we really going to call any composition of vision/audio/language
  model a "socratic" model? Even ones that don't have as much
  "discussion"? It seems like the models are not really engaging with
  a "dialogue" as one might imagine given the name of the method.
clarity,_quality,_novelty_and_reproducibility: see above
summary_of_the_review: Overall, this is a creative paper with some very promising results.  I
am hopeful that future multimodal works will include a socratic model
at least as a baseline. And --- I think the future directions for
these types of models (i.e., models that communicate via textual
representations) are quite promising --- I can't wait to see where it
goes!
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper investigates techniques for prompting pretrained unimodal and multimodal models to perform multimodal tasks in a zero-shot fashion. The paper proposes Socratic Models as a modular architecture that composes the outputs of various pretrained models into textual prompts for large multimodal models. The paper explores the use of Socratic Models in the context of various multimodal tasks, including image captioning, video-to-text retrieval, egocentric (robot) perception, and multimodal dialogue, with zero-shot performance that exceeds that of several baselines and is comparable to some one-shot baselines.

The paper was reviewed by referees who agree that the paper explores an interesting and important problem (i.e., zero-shot mulitimodal reasoning). As they emphasize, the proposed approaches for combining the output of pretrained models as a means of prompting a large mulitimodal model are both creative and effective, as evidenced by detailed insights into the prompts and outputs for the various tasks considered.  One of the core contribution of the paper is the demonstration that multimodal prompt engineering is viable using publicly available multimodel models in a zero-shot manner. Another is the establishment of solid baselines that would be useable by future researchers, which the authors support by providing publicly available implementations. The reviewers identified a few issues with the original submission, notably the concern that identifying the model sequence and prompt structure suitable to a given problem is non-trivial, as well as questions about the zero-shot performance and the need for additional few-shot baselines. The authors made a considerable effort to address these questions/concerns, which the reviewers acknowledged. Overall, this is solid work that provides a valuable contribution to the community.
justification_for_why_not_higher_score: I'm not fully convinced that this merits an oral, with the caveat that it depends on the quality of the other papers.
justification_for_why_not_lower_score: This could be a poster, but I feel it should be accepted.
Decision: Accept: notable-top-25%