Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed to iteratively optimize a system of one generator and multiple classifiers(scorers) for zero-shot generation tasks. It unified four multimodal generation tasks under the same framework. And experiments show that it can have a satisfactory generation ability on those tasks.
strength_and_weaknesses: Strengths:
1. This framework doesn't need further task-specific pre-training and finetuning. It only needs to iteratively update the pre-trained models to refine the generation result. 
2. The idea of using scorers to guide generators to reach a consensus is interesting and seems to be generalizable to four different tasks as in this paper.

Weaknesses:
1. For the methodology, this paper mainly instantiate the proposed method into two procedures: (1) continuous updating on image generation. (2) Discrete optimization over samples.  As for the first one, the whole method is quite similar to VQGAN-CLIP[1] in that they both optimize the image generator's input by a CLIP model to generate a better image. The difference is: at VQGAN-CLIP, the image generator is VQGAN and the optimized input is a latent feature, while this work used a diffusion model and the optimized input is a noisy image. Despiting the minor difference, those two share similar intuition and working mechanisms. For the second procedure, the caption generation optimization is almost the same as ZeroCap[2]. As mentioned in Figure A4's caption, this paper exactly used Equation 5 in ZeroCap to do the optimization. So is the Grade School Math text generation task. Overall, Although I appreciate the author(s)' efforts in unifying the different tasks in the same framework of iterative optimization, I still think the distinction between previous works and methodology contributions to the community is limited.
2. For the experiments, all those four tasks lack enough comparison with previous works. In Zero-shot videoQA, only JustASK is compared. In GSM8K, only GPT is compared. Moreover, the comparison seems not fair to me because the proposed PIC assumes to have a question-solution classifier, while GPT-FT(Pretrain) does not. In image generation and robot manipulation, no other baseline methods are compared.
3. Possibly Wrong/Confusing Illustrations. a). In Figure 2c, how does the input video frame feed into GPT-2? b) The explanation of $L_{CLIP}$ in Video question answering (VQA) of Sec3.1 is so confusing. I cannot really understand until I go back to read ZeroCap.

Ref:
[1] VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance, ECCV2022.
[2] ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic, CVPR2022.
clarity,_quality,_novelty_and_reproducibility: The clarity of some methods part is questionable, it is not friendly to readers. This paper has certain novelty in unification but the core idea is too similar to previous works. The reproducibility and quality seem okay to me.
summary_of_the_review: Overall, I prefer to weak rejection because of limited novelty and lack of comparison.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the problem of leveraging individual expertise of various large pretrained models to compose an ensemble for solving various multi-modal problems in a zero-shot manner. The authors propose a unified framework, called PIC, in which the pretrained models are composed using closed-loop iterative consensus optimization. Specifically, for a given task, the framework employs a single model as a generator and an ensemble of scorers. The generator iteratively generates proposals and each scorer provides a feedback score indicating their agreement, then the generator refines its output until all the scorers achieve a final consensus. Experiments are shown on four zero-shot tasks - image generation, video question answering, grade school math, and robot manipulation. Results show that a composition of the pretrained models with an ensemble of scorers outperform baselines with a single scorer by a considerable margin.
strength_and_weaknesses: **Strengths**:

- The paper writing, presentation, and figures are very good and well-structured.
- The motivation to solve the problem of composing pretrained models is interesting and differences with prior works has been well highlighted.

**Weaknesses**:

-  As the main motivation of the paper is to leverage the individual expertise of the pretrained models, it would be great to have some qualitative analysis on the same. Specifically, as can be seen from Table 1 that the best performance is obtained when all the scorers E1, E2, E3 are used as an ensemble, more insights on identifying individual expertise of the scorers can make the submission stronger. E.g. “which groups of classes does E1 expertise which E2 and E3 fails to?”, “What information does CLS-FREE provide which CLIP and CLS do not?”, “How do the results change with different generators?”. Getting answers to similar questions would be very interesting.

- How about selecting one best scorer for providing feedback to the generative model? I understand that in the context of 3 scores it does not make much sense and the results already show that the combination works the best. However, in many practical scenarios given that HuggingFace models are easily available, there will be access to 100s of diverse models that can be considered as scorers. In those scenarios, is it still advisable to consider the combination for providing feedback to the generator? Authors should perform experiments with a diverse set of models trained using different datasets and of different sizes to verify how the capacity and knowledge of the individual models as scorer effects the final performance. The experiments in the current form are limited to demonstrate the potential of the method.

- What about confidence of predictions for the different scorers? Ideally there should be a learnable weight signifying the importance of a scorer for a particular task and sample. I would encourage the authors to perform experiments for verifying the effect of such weights in the final performance.

- Most of the experiments mainly compare the performance of PIC with different variants of the same method. Given that the proposed framework focuses on composing different models, how is comparable to state-of-the-art methods, e.g, Image generation or video captioning? Due to the lack of comparison with SOTA methods, it is not clear why one should use this framework in their application instead of using any other existing methods.

- What about the memory requirement and FLOPs during inference and training of this method comparable to existing baselines?

- How were the accuracies for the compared baselines JustAsk and JustAsk (FT) computed in Table 2? Was Amazon Mechanical Turk used for them as well? 

- It would be great to have a baseline experiment in which PIC is asked to select an answer from the same pre-defined set used for other baselines instead of using a free-form response approach. How is that baseline compared to state-of-the-art methods on the same dataset? A thorough comparison and analysis should be performed to justify the advantage and utility of the proposed framework.

- As mentioned for Image Generation, analyzing the contribution of each of the scorers would be useful. E.g. “What attributes does CLIP-multilingual specialize in?” Analysis on the usage of GPT-2 is missing. How much does GPT-2 contribute towards the overall performance in video question answering?

- The paper also lacks some implementation details of the optimization procedure. Specifically, details like number of iterative steps used for optimizing for different tasks, learning curves with iterations (can be included in supplementary), optimization time required for each task, details of the context cache - size, initialization, etc.

- Experiments showing applicability of the framework to different permutations of possible generators and scorers are missing. E.g. getting results by using different generators for a given task.

- Experimental comparison with the discussed prior work Socratic Models (Zeng et al., 2022) is also not provided. If it is not possible to compare them in the experimented tasks due to language interface, how about comparing the proposed framework on the tasks used for zero-shot evaluation on Socratic Models. 
clarity,_quality,_novelty_and_reproducibility: The quality and clarity of the work is good with the problem statement properly conveyed to the reader. The introduction is quite comprehensive covering the current state of research in the topic. Related works are well discussed. The method used for solving the four zero-shot tasks have been properly explained and are novel. But as mentioned in the weaknesses section above, extensive experiments and analysis is missing, which is very important for understanding the advantage of the proposed framework.

The use of Amazon Mechanical Turk to get the accuracy for the video question answering task hinders easy reproducibility of the corresponding experiments. An alternate metric not involving the requirement of human feedback would be encouraged to make the experiments more reproducible. Additionally, including details of some hyperparameters (check weaknesses) can increase reproducibility.

summary_of_the_review: I think the identified problem is important but the proposed method is a bit naive and experiments are not convincing. The proposed method should be evaluated and analyzed using extensive experiments resonating with the motivation to solve the problem. In the rebuttal I expect new experiments and analysis listed in the weaknesses to convince me that the proposed method has practical use and outperforms the missing baselines and exiting methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new framework for using pre-trained models for various recognition and reasoning tasks. The idea is to use a "generator" and an ensemble of "scorers" to iteratively improve the output: the generator proposes an output that the scorer scores. This general framework is demonstrated on multiple tasks, including visual question answering, solving math problems, image generation and robot manipulation. 
strength_and_weaknesses: Strengths:
- The framework is very elegant. It uses the complimentary strengths of, for example, language models (as generators) and retrieval models like CLIP (as scorers).
- The notion of iterative feedback to get consistency between the generator's output and the scorer is innovative.
- The results, especially the zero-shot results, are quite impressive.

Weaknesses
- This proposed framework is very similar to the kind of optimization that appears in older literature on MAP estimation in structured prediction. A discussion on this would be useful.
- Also worth discussing would be the effect of the limitations of the pretrained models on the proposed framework.

clarity,_quality,_novelty_and_reproducibility: Approach looks very novel and high quality.
summary_of_the_review: I find the general framework quite interesting and novel. I think the connections to traditional iterative methods for structured prediction problems are very intriguing and should be discussed. That said, I find the approach innovative enough to be published.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an ensembling framework for use the capabilities of multiple foundational models. In cases where there isn't a unified interface to interact with a number of foundational models, it provides a unified framework to allow multiple foundational models to interact with each other. The paper proposes to divide the foundational models as "Generators" and "Evaluators". The "Generator" set of models generates proposals and the "Evaluator" sets of models evaluates the proposals and are used to iteratively refine the proposal using its gradients. 
strength_and_weaknesses: Strengths:

* Elegant technique for combining the strengths of complementary models which are trained on large scale dataset 
* No retraining required which is expensive in most cases with large scale models
* Results show effectiveness of the technique 

Weaknesses:

* The change in the scores with the proposed technique seems minimal. 
* In industry use cases the GPU memory would overflow to fit all the models simultaneously.
* The greatest benefit of this technique seems to be for Robotic Manipulation, reducing the scope of impact. 
clarity,_quality,_novelty_and_reproducibility: The text is quite clear. The experiments have been defined explicitly in the Appendix. The presentation of the ideas in the text can be more coherent but still sets a high bar of excellence. 

The technique is novel, it provides a simple yet effective strategy of combining the strengths of related but relatively orthogonal models. The proposition of using a class of models as diverse as GLIDE, VilD, CLIP in the same framework is novel.

The technique proposed in the paper is quite clearly described and should be straight forward to reproduce it.
summary_of_the_review: This paper proposes a simple technique for combining the strengths of several large scale foundational models. It is especially interesting when large scale data across multiple domains is not readily available such as Robotics and Language Models. It would have been interesting to see some tasks which are only enabled with this system which would not be possible without the presence of this system.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The idea of the paper is to use different pre-trained models to complement the capabilities of each other. For example, GPT-3 is a powerful language model but it does not have the capability of visual understanding. On the other hand, methods such as DALL-E generate realistic images but they fail to capture complex language prompts. The paper proposes to ensemble these different pre-trained models to complement each other. A set of proposals are generated by these models, which are later “scored” by other models. There is a closed-loop optimization strategy that encourages the generators to generate better proposals in the next iterations. 

Strength:
- The idea of iterative feedback to generate better proposals is interesting.
- It is a big advantage that no fine-tuning is required.
- Zero-shot results are quite encouraging.

Weaknesses:
- Lack of automatic evaluation metrics.
- Lack of comparisons with SOTA methods.

justification_for_why_not_higher_score: The paper has some issues regarding metrics and comparisons (explained above). So, it requires some work to become a solid paper. 
justification_for_why_not_lower_score: The community benefits from having this paper in the conference, so I voted for acceptance.
Decision: Accept: poster