Composing Ensembles of Pre-trained Models via Iterative Consensus
Authors: Shuang Li
Yilun Du
Joshua B. Tenenbaum
Antonio Torralba
Igor Mordatch
Venue: ICLR 2023 poster (2023)
Keywords: composing pre-trained models
zero-shot
multimodal
wisdom of the crowds

Abstract: Large pre-trained models exhibit distinct and complementary capabilities dependent on the data they are trained on. Language models such as GPT-3 are capable of textual reasoning but cannot understand visual information, while vision models such as DALL-E can generate photorealistic photos but fail to understand complex language descriptions. In this work, we propose a unified framework for composing ensembles of different pre-trained models -- combining the strengths of each individual model to solve various multimodal problems in a zero-shot manner. We use pre-trained models as "generators" or "scorers" and compose them via closed-loop iterative consensus optimization. The generator constructs proposals and the scorers iteratively provide feedback to refine the generated result. Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e.g. improving accuracy on grade school math problems by 7.5%, without requiring any model finetuning. We demonstrate that consensus achieved by an ensemble of scorers outperforms the feedback of a single scorer, by leveraging the strengths of each expert model. Results show that the proposed method can be used as a general purpose framework for a wide range of zero-shot multimodal tasks, such as image generation, video question answering, mathematical reasoning, and robotic manipulation. 


Published as a conference paper at ICLR 2023
COMPOSING ENSEMBLES OF PRE-TRAINED MODELS VIA ITERATIVE CONSENSUS
Shuang Li +
Yilun Du
Joshua B. Tenenbaum MIT CSAIL,, BCS, CBMM jbt @mi edu
MIT CSAIL Lishuangemit.edi
MIT CSAIL
yi minddd@iit.eem edu
Antonio Torralba MIT CSAIL torralba@mit.edd
Igor Mordatch
Google Brain
imor iimorddttcheeooggee.com
ABSTRACT
Large pre-trained models exhibit distinct and complementary capabilities depen- dent on the data they are trained on. Language models such as GPT-3 are capable of textual reasoning but cannot understand visual information, while vision models such as DALL-E can generate photorcalistic photos but fail to understand com- plex language descriptions. In this work, propose unified framework for composing ensembles of different pre-trained models combining the strengths of each individual model to solve various multimodal problems zero-shot manner. We use pre-trained models as "generators" or "scorers" and compose them via closed-loop iterative consensus optimization. The generator constructs proposals and the scorers iteratively provide feedback to refine the generated result. Such closed-loop communication enables models to correct errors caused by other models, significantly boosting performance on downstream tasks, e.g. improving accuracy on grade school math problems by 7.5%, without requiring any model finetuning. We demonstrate that consensus achieved by an ensemble of scorers outperforms the feedback of a single scorer, by leveraging the strengths each expert model. Results show that the proposed method can be used general purpose framework for a wide range zero-shot multimodal tasks, such as im- age generation, video question answering, mathematical reasoning, and robotic manipulation.
INTRODUCTION
Large pre-trained models have shown remarkable zero-shot generalization abilities, ranging from zero-shot image generation and natural language processing to machine reasoning and action planning. Such models are trained large datasets scoured from the internet, often consisting of billions of datapoints. Individual pre-trained models capture different aspects of knowledge on the internet, with language models (LMs) capturing textual information in news, articles, and Wikipedia pages, and visual- langul-a models (VLMs) modeling the alignments between visual and textual information. While it is desirable have single sizable pre-trained model capturing all possible modalities of data on the internet, such a comprehensive model is challenging to obtain and maintain, requiring intensive memory, an enormous amount energy, months of training time, and millions of dollars. A more scalable alternative approach is compose different pre-trained models together, leveraging the knowledge from different expert models to solve complex multimodal tasks.
Building unified framework k for composing multiple models is challenging. Prior works (Alayrac) ct al., 2022; Zeng et al.) 2022) have explored composing pre-trained models in two main ways: (jointly) finetuning models on large datasets, using common interfaces such as language to combine
*Correspondence to: Shuang Li <lishuang@@n cittaanggmtteiiii indicates equal contribution. Shuang Li did all the experiments image generation, video question answering, and mathematical reasoning. Yilun Du did all the experiments on rohot manipulation.

---

Published as a conference paper at ICLR 2023
Video Question Answering
Grade School Math
Language models Et CLIP models
Language modele
E: QA classifers
Q: A robe takes 2 bolts of blue fiber and white fiber. How many bolts in total does is take! A: Claire makes morning for becalffast. How many doumus will she eat. 1 weeka? A:
holf
omelet every
Q How to makee the food by step?
pot, on saueago, add seusning on top of the pizza
Put
Iterative
What food is being mode? Ar Make pizza
Generator (G)
Scorers
Robot Manipulation
Imaage Generation
Diffusion models
World models
E: CLIP models, Imsge cloestfeers
E: Image
Orange to the right of orange bowl, Orange mug on top ocange bowd
Grasshopper
asguentation modelss
front: of tre
ensemble "scorers" through iterative consensus enables zero-shot generalization across a variety of multimodal tasks.
Figure 1: The proposed framework that composes a "generator"
different models. However, these works have several key limitations: First, simply combining models does not fully utilize each pre-trained model as there is no closed-loop feedback between models. Cascading models, such as Socratic models (Zeng al., 2022), allows one-way communication but prevents information processed by later models from propagating back to earlier models to correct errors. Secondly, common interfaces are limited to particular types of models. Language used as the intermediate connection in Socratic models (Zeng et al. 2022), but language interface insufficient to solve many weal-world tasks, such as continuous robot control, which requires continuous representations. In addition, Socratic models require pre-designed language templates for the communication between models, which limits scalability. Thirdly, jointly finetuning multiple models (Alayrac al., 2022) requires careful optimization ensure that the model behaviors remain stable. Such models also require intensive memory and large datasets and can only be used for solving specific tasks.
To resolve these difficulties, we propose a unified framework to compose models in a zero-shot manner" without any training/finetuiing. Our framework employs single model generator and an ensemble of scorers. The generator iteratively generates proposals, and each scorer provides feedback score indicating their agreement. The generator refines its outputs until all the scorers achieve final consensus. This iterative closed-loop communication between the generator and scorers enables models correct the errors caused by other models, substantially boosting performance.
The ensemble of scorers is inspired by the idea of "wisdom of the crowds". Each scorer provides complementary feedback to the generator, compensating for the potential weaknesses of other scorers. A Vision-Language scorer, for example, may correct the biases of language model. We notice that different pre-trained model instances from the same family have diversity of outputs, which leads to more robust scorers. We demonstrate that guiding the generator with such an ensemble of scorers significantly outperforms generator guided by a single scorer.
To summarize, work has three main contributions.
First, we propose unified framework for composing pre-trained models across variety of tasks, such as image generation, video question answering, mathematical reasoning, and robot manipulation.
Second, we illustrate how the proposed framework can effectively solve zero-shot multimodal tasks without any training/finttninng. The closed-loop communication between the generator and scorers allows the models to interact with each other to improve performance iteratively.
Finally, we illustrate how our framework enables the use of ensembles of different pre-trained models as scorers, significantly improving the zero-shot results by leveraging the strengths of multiple expert models.
These observations point to the effectiveness of the proposed method general purpose framework for composing pre-trained models for solving various zero-shot multimodal tasks.
By zero-shot, we the composed models never trained together the evaluation task.
2

---

Published as a conference paper at ICLR 2023
2 RELATED WORK
Large pre-trained models have shown great success across a variety of domains, such as language generation/trassltion, image generation, and recision-making
Language models. Large language models, such as ELMo (Peters et al. 2018), BERT (Devlin et al., 2018), and GPT-2 (Radford al. 2019), able to achieve -fate-of-the-arr art performance many standard NLP benchmarks. More recent works, such as GPT-3 (Brown et al. 2020), PALM (Chowd hery al. )022), and Chinchilla (Hoffmann al., 2022) further enable few-shot learning from textual prompts. Vision-language models. Large pre-trained vision-language generative models, such DALL-E 2 (Ramesh et al. 2022), Parti (Yu et al., 2022), and Imagen (Saharia al. (022), can generate high-resolution images given natural language descriptions. Large pre-trained vision- language discriminative models, such as CLIP (Radford et al., 202I), convert images and languages into the same feature space, achieving remarkable zero-shot generalization abilityy on downstream tasks. Decision-making models. Large pre-trained models have been widely applied to solve decision-making tasks, such as learning general purpose policies (Reed et al. 2022. [i et al. [222] Shridhar et al. 22022, making planners (Huang et al. [022]] Ahn et al., 2022), and learning world models (Ebert et al. 2018). However, due to the large variability in decision-makinn tasks, no existing pre-trained models can be readily applied across different tasks.
Composing pre-trained models. Composing large pre-trained models has been widely studied recently. The predominant way to compose pre-trained models is to (joint) finetune them on new tasks (Li et al. 2019; |Wang et al. 2021; Alaarac et al.) 2022 Mokady et al. [02II, but such approaches are computationally expensive. Alternttve approaches compose models through a common interface such language(Teve (Tewel et al. 2021 Zeng et al., 2022). Other works compose pre-trained models by composing learned probability distributions of the data, such as energy-based models (Liu et al., 2022 2021 Du al. (2020), which can be applied to generate images or to refine structural prediction (Belanger et al. 2017). In this paper, we propose general framework to compose pre-trate rre-traneed models across variety of domains without any training or finetuning.
METHOD
Given set of large pre-trained models, we aim to utilize the expert knowledge from different models to solve zero-shot multimodal tasks. We separate pre-trained models into two categories generators (G) such as GPT (Brown et al.) 2020; Radford et al. (219)) and Diffusion models (Ho et al. 2020) that can generate candidate solutions, and scorers (E) such as CLIP (Radford et al.) 2021) and classifiers that output scalar score to evaluate each generated solution. We propose PIC (composing ensembles of Pre-trained models via Iterative Consensus), a framework which composes ensembles of pre-trained models for multimodal tasks. The core idea of PIC is generate solutions through iterative optimization, where we leverage the knowledge from different models to jointly construct a consensus solution. In PIC, generator G iteratively and sequentially generate candidate solutions, each of which is refined based on the feedback from a set of scorers. In particular, we seek to obtain a solution x such that
xx" arg min D-G
Σ En(r,,
where {En} is the set of scorers. At each iteration, we refine the solutions have lower score than the previous iterations. This procedure, described in Equation (T). converges to a solution that minimizes the energy across multiple pre-trained models, which maximizes the agreement between the generator and scorers. In contrast to Socratic Models where different pre-trained models are called sequentially, the closed-loop iterative refinement through which we obtain x² enables the generator and scorers to communicate with each other to reach consensus the final solution.
Below, we illustrate how PIC can be broadly applied across tasks in image generation, video question answering, grade school math, and robot manipulation. To optimize Equation (I.. we consider two different optimization procedures either a continuous approach that leverages the gradients each scorer E,, (x)) or discrete approach that directly samples possible solutions.
3.1 APPLICATIONS TO ZERO-SHIOT TASKS
Image generation. We first apply the proposed framework to image generation to generate im- ages conditioned text description or class label. We use the reverse diffusion process of
3

---

Published as a conference paper at ICLR 2023
(a) Unified Framework
(b) Example Image Generation
Used the imput the iteration
Feedback
Update based on gradient of the summed score
Ex (shic)
Generated
from scorers
result
Image y
Result the last iteration
Scorer 1
Generator: Diffusion Model
CLIP score
One moo/more
Generator
proposals
Scorer N
Classifier score
Image 2
Image g
Conditional iuput/
Nome
Input text: Goldfinch
(c) Example Video Question Auswering
(d) Example on Robot Manipulation
Update based on the gradient the summed "ogg""
EarlEE (2
Select the action that has the minimal summed sooze
Word
CLIP 1 score
State Real
bowl with"
Generator: GPT-2
"rice"
Word le41
CLIP score
Generator: MPC-
View 1 score
Caption
World Model
Iaput
State X
Stave he41
score
vidco
frame
Iuput text: red bowl top blue mug
Figure 2: The proposed unified framework and examples on three representative tasks. (a) Overview of the proposed unified framework. Dashed lines omitted for certain tasks. (b) Image generation. A pre-trained diffusion model is used the generator, and multiple scorers, such CLIP and image classifiers, provide feedback to the generator. (c) Video question answering. GPT-2 is as the generator, and set of CLIP models are used as scorers. (d) Robot manipulation. MPC+World model is used as the generator, and a pre-trained image segmentation model is used to compute the scores from multiple camera views to select the best action. Orange lines represent the components used refine the generated result.
used
GLIDE (Nichol et al. 2021). a text-guided diffusion model, as the generator to generate image step of the iterative refinement),
proposals, At each step of the diffusion process (corresponding we use the gradient from an ensemble of scorers, such as CLIP (Radford et al. 2021). to guide and update the generated proposals. We iteratively repeat this procedure until the final step.
As shown in Fig. (b), the image æb generated at iteration k is first sent to the diffusion model to generate an image proposal +1+1 Each scorer outputs score evaluate whether the generated image matches the given text input. For example, CLIP computes the cosine similarity between the image and text features as the score. The scores generated by different scorers are summed, and their gradient with respect to x is used to compute the next reverse prediction x k+I,
N
1+t + + AV
E
x c)
11
where the number of scorers and is the text label. We denote the reverse process prediction as k++1 instead of =-1 (used by most diffusion models) keep the consistent notation across tasks.
Video question answering (VQA). Caption generation for a single video frame is shown in Fig. 2 (c). We use GPT-2 the generator and multiple different CLIP models, trained with different configurations, as the scorers. Given a video frame 1, we generate a sequence of words to describe To integrate feedback from scorers to the generator, similar (Tewel et al. 2021). we define context cache C₁ (a set of embedding functions in GPT-2) that stores the context information generated so far, which is updated iteratively based on the feedback from scorers. The prediction of the word from the generator is given by dc+1 = G(I,, C.). To update C., we first use G to generate a set of candidate words X.+1 = {R++1), }. and then use the feature distance (after softmax) between each sentence (the concatenation of previous words and each new word {x x2, ... ,.+41) where ft-1 E Xx+1) ) and the video frame as the probability of them matching. The CLIP score is the cross-entropy loss LCLP between this probability distribution and the original distribution of the
4

---

Published as a conference paper at ICLR 2023
next word obtained from the generator G. The gradient of the summed score (multiple CLIP models) is then propagated to to update C₁:
N Σ n=1
ct+ t C + AV of
Ceur EE (r1, 02, ,,,++,
(3)
where l is the step of iterative refinement. After several iterations, the updated C₁ is used to generate the next token 1 G( Ce). We repeat this process until we generate the entire caption. We cascade the captions of multiple video frames and questions about this video to prompt GPT-3 for video question answering (See Appendix B.2].
Grade school math. We further apply PIC to solve grade school math problems. We use GPT-2 as the generator and treat the grade school math problem as text generation problem. The scorer, a pre-trained question-solution classifier, provides the generator feedback to guide the next token's s generation Zt+11- We follow the approach used in VQA to iteratively optimize the generations based on the feedback from scorers. Our generator G first generates a set of candidate words X {&(+)}. and then the classifier predicts the probability of each solution (the concatenation of previous words and each new word f #2. .8+1). J. where æ2+1 E Xx+1) matching the given question. The classifier score is the cross-entropy loss between this new probability distribution and the original distribution of the next word obtained from the generator G. The gradient of the classifier score is used to update C, through iterative refinement, same as Eq. (3) The updated C, is used to predict the next word 2 G( Ct). We repeat this process until we generate the complete solution.
Robot manipulation. Finally, we illustrate how PIC can be applied to manipulate objects in the robot environment conform set of object relations such as "red bowl on top of blue mug" shown in Fig. (d). We use the combination of Model Predictive Control (MPC) (Williams et al. 2015) and the World Model as the generator. At each time step, we first use MPC to sample a set possible action) from multiple camera views using
actions and then render the state images (after executing the world model. For each action, the scorer computes a summed score across all camera views as its final score, which is used to select the best action to execute. Thus, in this domain, the ensemble consists of scorers based on different views of the scene.
For the generator, we assume that there is pre-trained model, i.e. world model, that can accurately render and simulate the dynamic changes in the robot world. Since such a large pre-trained model does not directly exist, approximate it using environment simulator combined with MPC the generator. For the scorer, we use the pre-trained ViLD (Gu et al.) (021) to generate segmentation maps for images captured by different camera views n,, and the corresponding text label for each segment, which are used to obtain object relations. We compare the generated object relations and the relations specified by the text description obtain the score, i.e. score equals 0 if they match; otherwise, 1 (here the score means the distance) (see Appendix state 2r that satisfies the specified relations, and the action sequence {a,,... ar) that manipulates the objects into the final state TT, the generator iteratively samples possible actions aلا and gets feedback from scorers. The best action selected as:
details). To obtain final world
a++1 arg min
Σ (
(4)
Each scorer, Eg. outputs score for the resultant state obtained when a candidate action at. is applied to the current world state Iy. We execute 4+41 in the environment and get a new state 1r41- We repeat this process until the task is accomplished or we are at the final step T..
4 EXPERIMENT SETUP
We evaluate the proposed framework for composing pre-trained models four representative tasks, including image generation, vidco question answering, grade school math, and robot manipulation.
Image generation. We first show that composing the pre-trained image generator and scorer models such as CLIP enables effective zero-shot image generation. We evaluate the image generation results on ImageNet (Deng et al.) 2009) with the image resolution of 64 x 64. The class labels used as the text input to guide image generation. Each method generates 50 images for each class. We evaluate the image generation quality using Inception Score (IS) (Salimans et al., 2016). Fréchet
5

---

Published as a conference paper at ICLR 2023
Table 1:
Image generation results on ImageNet. Our PIC can compose the pre-trained generator (G) and
scorers (E) through iterative optimization. Composing multiple scorers further boosts performance.
Method Name
Generator
Scorer
18 1
FID.
KID
PIC (G+EI)
GLIDE
CLIP
25.017
30.462
6.174
PIC (G+E2)
GLIDE
CLS
22.077
30.871
7.952
PIC (G+E3)
GLIDE
CLS FREE
25.926
29.219
5.325
PIC (G+E1+E2+E3)
GLIDE
CLIP CLS + CLS-FREE
34.952
29.184
3.766
Table 2: Video question answering results on ActivityNet-QA JustAsk (FT) is finetuned on ActivityNet- QA, thus achieving the best results. For zero-shot VQA, our method (PIC) significantly outperforms JustAsk (Pretrain), one of the best VQA methods. Using multiple scorers further improves the performance.
Method Name
Zero-Shot
Generator
Scorer
Accuracy f Vocab
JustAsk (FT)
No
64,667
160
JustAsk (Pretrain)
Yes
50.671
210
PIC (G+EI) PIC (G+3I+E2+K3)
Yes
GPT-2
CLIP-32
58.389
267
Yes
GPT-2
CIJP-32+ CL + CIJP-mulilingual
61.168
304
Inception Distance (FID) (Heusel et al., 2017p. and Kernel Inception Distance (KID) Binkkossii et al. 2018). IS measures the distribution of generated images. Higher values mean the models can generate more distinct images. FID considers the distributions of both generated images and real images. Lower scores represent that the generated images are closer to the real images. KID is similar to FID, measuring the similarity between two data distributions, but is in the kernel space.
Video question answering. We evaluate methods for solving VQA tasks on AetivityNet-QA (Yu ie al. 2019). Our method generates free-form language answers instead of selecting an answer from a pre-defined answer set (Yang et al., [021 Lei et al., 2022). To evaluate such free-form VQA, we ask workers from Amazon Mechanical Turk to measure whether the generated answer matches the given question and video (See Appendix
C for IRB approval and experimental details). For fair comparisons, all the approaches answer the same 300 video questions, and each answer is evaluated by three different workers. The accuracy rate and vocabulary size are reported. An answer is correct if at least two workers believe it is correct. The accuracy rate is the percentage of correctly answered questions over all the questions. To evaluate the diversity of generated answers, we also report the vocabulary size (i.e. the number of words) of answers generated by each method.
Grade school math. GSM8K (Cobbe et al. 2021 is a dataset for grade school math problems. Each problem consists of question, intermediate analyses, and a final solution. We evaluate approaches to solving problems on the IK test set. We use beam search to generate candidate solutions. The accuracy of beam size and beam size 5 are reported. For beam size of 1, we mark the result as correct if it matches the final solution. For beam size of 5, we mark the result as correct if any of the five generated results matches the solution.
Robot manipulation. We next evaluate how pre-trained models may be used to manipulate objects in Ravens (Zeng et al. 2020). In Ravens, the action space of robot is to drop an object at a 2D location on the table. The goal is to obtain a scene configuration that satisfies the object relations specified by a textual deseription or a real-world image, such as "blue mug to the left of purple bowl". The task successful if the object relations in the final state satisfy all the relations specified by the input text or image. We report the success rate of tasks with two and three specified object relations.
5 EXPERIMENTS
We compare the proposed method with baselines on the above four zero-shot tasks.
5.1 IMAGE GENERATION
We evaluate the zero-shot conditional image generation on ImageNet in Table [ We first show results of composing single generator (G) and a single scorer (E). We compose GLIDE (Nichol et al. 2021) with three different types of scorers, respectively. E1 is CLIP (Radford et al. 2021 that computes the cosine similarity between the image and text features the score, E2 is the image classifier (CLS) (Dhariwal & Nichol (20TT) that predicts the probability of the image matching the text label the score, and E3 is the classifier-free guidance (CLS-FREE) (Ho & Salimans 2022)

---

Published as a conference paper at ICLR 2023
Oue video
JustAsk (Pretzain)
PIC (G+EI)
PIC (G (G+E1 EE))
JastAsk (FT)
(not scro-shot)
Q: is the person blue man woman? Q: what happened before the pole vault?
A: no
A: woman
A: woman
A: yes
A: the person in the video is stretching
A: audience cheered
A: the athlete rurning
magnesia powder
Q: how many people are there in the viden?
A: no
A: 1
A: 4
A: 2
Q: what kind of trousers does the woman wearing yellow clothes look like?
A: tight trousers
A: short
A: short
A: short
Q: what is the person
A: nursing bicycle
using a pickup
A: cutting grass
A: weed
with hat doing?
Q: what happened to the person in the hat before the engine?
A: someone passed
A: put on the helmet
A: turn on the engine
A: speech
through
Figure 3:
Video question answering example results. Our approach successfully identifies gender and clothing, but its failure count objects reflection of GPT-2 and CLIP's inability to count.
which can be treated as an implicit classifier that directly provides pixel-wise gradient feedback the generated image (Appendix [B.]). We then compose the generator with all scorers, i.e. G+E1+E2+E3 Composing the generator and a single scorer allows zero-shot image generation. Composing multiple scorers significantly outperforms single scorer. We note that the generator is not trained on ImageNet; thus the results in Table cannot be directly compared with methods trained on ImageNet.
5.2 VIDEO QUESTION ANSWERING
Quantitative results. We compare PIC with one of the state-of-the-ard VQA approaches, i.e. JustAsk Yangg et al. (021). on ActivityNet-QA (Yu et al. (019)). In Table 2 JustAsk (FT) is finetuned on ActivityNet-QA thus achieving the best results. We then compare PIC with JustAsk (Pretrain) for zero-shot VQA. The generator of our method, GPT-2 (medium size), is trained on Webtext (Radford et al.] 2019) using the Huggingface library (Wolf al. 2019). Our scorers are CLIP models (Radford et al. 2021: Reimers & Gurevych| 2019) trained different datasets or using different configurations. PIC (G+EI) outperforms JustAsk (Pretrain) by %7.72. Composing more scorers further improves the accuracy by %2.78. In addition, the vocabulary size of answers generated by our method is larger than other approaches, indicating that our method can answer questions using richer language and more diverse phrasing. Note that our method solves "more challenging" problem than JustAsk (Pretrain) and JustAsk (FT). Our method generates open-language answers while JustAsk (Pretrain) and JustAsk (FT) select an answer from pre-defined answer set. Generating free-form responses requires both semantic and grammatical correctness. PIC performs well on both these dimensions while also using a richer vocabulary.
Qualitative results. In Fig. 3 we show answers generated by different approaches given video (only showing single video frame) and questions. Our approach successfully identifies gender and clothing, but none of the approaches know how to count numbers.
5.3 GRADE SCHOOL MATH
Quantitative results. In Table pare PIC with two baselines, i.e. GPT-
3 we com-
Table 3: Grade school math results on GSM8K. Our method (PIC) that composes GPT-2 and a pre-trained
Pretrain and GPT-FT, for solving math problems on GSM8K (Cobbe et al.) 202T).
question-solution classifier significantly outperforms the base- lines, including GPT-FT that is finetuned on GSM8K.
GPT-Pretrain uses the pre-trained GPT-2 (medium size GPT-2 trained on Webtext using Huggingface) to generate numeric strings. GPT-FT is based on GPT-Pretrain and then finetuned on GSMSK. Our method
Method Name
Generator
Scorer
BS=1 t
BS=5
GPT-Pretrain
GPT-2 (Pretrain) GPT-2 (FT)
1.744
12,206
GPT-FT
3.487
18.271
PIC (G+E)
GPT:2 (Pretrain)
16.831
20.773
uses the same GPT-2 (Pretrain) as the generator and a question-solutiin classifier (CLS) as the scorer. The classifier is trained on GSM8K to distinguish whether solution is correct for a given question. We surprisingly find that PIC achieves significantly better performance than GPT-FT (% 13.344 higher on beam size 1), even though the generator has never seen the math problems before. The classifier
7

---

Published as a conference paper at ICLR 2023
Topus object iors
Final scene
Toput object relations
Final sceaee
Topet object rolations
Final scene
blue sug the
bowdl to the right of eyan tang
eyan howd to the right eyan mong gaven bowl on top of
noug
blue bowl on top of eyen mug
orunge to the green
cron bewd to the right blue bowl
beown mig the orange mag
orange man to the righht green bowl
Figure 4:
Robot manipulation example results. The robot manipulates objects to achieve certain object relations that are specified by textual descriptions (first row) or real-world images (second row).
only provides feedback the generator, but through iterative refinement, combining generator and a scorer without joint training is more effective than directly finetuning GPT-2 on GSM8K (we find the overfitting problem when finetuning GPT-2 on GSM8K).
5.4 ROBOT MANIPULATION
Quantitative results. We evaluate the proposed method of manipulating objects to achieve object relations specified by the textual descriptions (Text) or real-world images (Image). In Table we find that using scorers of multiple camera views sub- stantially improves the accuracy both settings.
Table 4: Robot manipulation results on Ravens. PIC can manipulate objects to achieve object relations specified by textual descriptions (Text) real-world images (Image). Using scorers of multiple camera views substantially improves the success rate.
Method Name
2 Relations
3 Relations
Qualitative results. Figure 4 shows the example results of the proposed method manipulating ob- jects to accomplish the given task. Our method enables zero-shot robot manipulation objects with different sizes, colors, and shapes given either the language goal or image goal.
Text 1 Image
Text
Image f
PIC (G+E1) PIC (G+) E.)
35.0
50.0
45.0
67.5
52.6
75.0
65.3
ANALYSIS
PIC exhibits effective zero-shot generalization ability on variety of tasks. To further understand the source of such generalization, we investigate two key components in PIC, i.e. the composition of multiple scorers (consensus optimization) (Section 6.1 and the iterative refinement (Section 6.2
6.1 EFFECT OF CONSENSUS OPTIMIZATION
We have shown that composing multiple scorers contributes zero-shot generalization. We further explore the influence of gradually adding each new scorer on the zeros-shot performance.
Image generation. Table we first show results of composing GLIDE and the CLIP scorer. We then gradually add a new scorer, the image classifier or classifier-fres guidance, each time. Finally, we report the results of composing the generator and all scorers. The performance improves every time we add a new scorer, indicating that composing multiple scorers improves zero-shot performance.
Table 5: Effeect of composing multiple scorers. Image generation results on ImageNet. Gradually adding new scorers keeps improving the performance.
Method Name
Generator
Scorer
IS 1
FID
KID |
PIC (G+E1)
GLIDE
CLIP
25.017
30.462
6.174
PIC (G+E1+E2)
GLIDE
CLS
30.438
29.543
5.435
PIC (G+E1+E3)
GLIDE
CLIP+ FLE-FREE
30.500
29.726
4.304
PIC (G+E1+E2+E3)
GLIDE
CLIP CLS FLE-
34,952
29.184
3,766
8

---

Published as a conference paper at ICLR 2023
Table 6: Effect of iterative refinement. Grade school math results GSM8K. PIC with iterative refinement outperforms baselines where the scorer only provides feedback the generator at end stage. BS the beam search size.
Method Name
Generator
Scorer
Interaction
BS=1
GPT-Pretrain+E
GPT-2 (Medium) (Pretrain)
CLS
t=
9.704
GPT-FT+E
GPT-2 (Medium) (FT)
CLS
[= T
14.481
PIC (G+E)
GPT-2 (Medium) (Pretrain)
CLS
==
{1, .T)
17.210
Robot manipulation. In Table 7 we analyze the effect of composing multiple scores on robot manipulation. The goal is specified by textual descriptions. Composing scores from multiple views, PIC (G+ (G+En Fnn) and PIC (G+ >n=1 En,, leads higher accuracy.
6.2 EFFECT OF ITERATIVE REFINEMENT
Next, we explore the influence of iterative refinement on zero-shot generalization, i.e. the feedback loop between the generator and scorers. We compare PIC with baselines that compose the generator and scorers, but with the scorers only providing feedback to the generator at the end.
school math. In 6 the baselines, GPT-Pretrain+E: and GPT-FT+F, generate five proposal solutions of a given math problem. Then the scorer, Le. the same question-solution classifier used in PIC, selects the best solution based on its score. PIC iteratively refines the generated answer while the baselines refine the entirely generated solutions in the end. PIC and GPT- PupT-Pretainn-E use the same generator and scorer, but PIC outperforms GPT-PretrainEE by %7.507, PIC still achieves better performance than GPT-FT+E, which uses stronger generator (finetuned on the GSMSK dataset).
Robot manipulation. In Table 7 the baseline, No-IR (G+ En), first samples 100 trajectories without using the feedback from scorers. Then the scor- ers select the best trajectories based on the summed score. The generator and scorers of this baseline are the same our method, ie. PIC (G+ > F,,). but our method outperforms the baseline by 737.5 on the "2 Relations" setting, indi- cating the effectiveness iterative refinement in the proposed framework.
Table 7: Effect of composing multiple scorers and itera- tive refinement on robot manipulation. Both components are important for zero-shot generalization.
Method Name
Interaction
Relations
Relations
PIC (G+EI) PIC (G+ (G+E End
{1, .T)
35.0
50.0
{1.... T t [1.
PIC (G+ (G+ En)
T}
67.5
75.0
No-IR (G+) F..)
Together, these results show that the composition of multiple scorers and iterative refinement are both important for zero-shot generalization. These results point to the potential broader applicability of the proposed method general purpose framework for zero-shot multimodal tasks.
CONCLUSION AND FUTURE WORK
In this paper, propose unified framework for composing ensembles of pre-trained models through iterative consensus without any training or finetuning. Our framework consists of generator and an ensemble of scorers. The scorers provide feedback to the generator to iteratively improve its generated results, We show the proposed method allows effective zero-shot generalization on four representative tasks, i.e. image generation, video question answering, grade school math, and robot manipulation, and even outperforms methods that directly finetune models on certain tasks. We further analyze the source of such zero-shot generalization by exploring the effect of the composition of multiple scorers and the iterative refinement, and find that both are important for zero-shot generalization.
As our method does not need any training or finetuning, one drawback is that its performance depends on the pre-trained models. Training large models are complementary to the framework and methods we proposed and may be directly applied. We hope to explore these directions for zero-shot generalization in future work. In addition, our framework enables the composition of separately trained models and boosts performance by leveraging the knowledge from multiple expert models. The scorers can be learned at different times different data in incremental- manner, enabling the combination of incrementally learned knowledge. Our framework thus paves the way for many potential applications in lifelong learning / continual learning settings.

---

Published as a conference paper at ICLR 2023
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not say: Grounding language in robotic affordances. arXiv preprint arXiv:22044 01691, 2022.
Jcan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. /4/98, 2022.
David Belanger. Bishan Yang, and Andrew McCallum. End-to-end learning for structured prediction energy networks. In International Conference on Machine Learning, pp. 429-439, PMLR, 2017.
Mikolaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv: 11100..11450 401, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:18 1877-199,,, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprini arXiv:2204.02311, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Relichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21100 14168, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-877 8794, 2021.
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637-6647, 2020.
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine. Visual foresight: Model-based deep reinforcement learning for vision- robotic control. arXiv 2018
preprint arXiv:1812.000
Xiuye Gu, Tsung-Yr Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint 13X1v 2021.
Louay Hazami, Rayhane Mama, and Ragavan Thurairatnam. Efficient-vdvac: Less is more. arXiv preprint arXiv:2203. 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.1255 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
10

---

Published as a conference paper at ICLR 2023
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint 7rXiv:2207.05688 2022.
Jie Lei, Tamara L Berg, and Mohit Bansal. Revealing single frame bias for iileo-and-languaae learning. arXiv preprint arXiv:2206. 033428 2022.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.035577 2019.
Shuang Li, Xavier Puig, Yilun Du, Clinton Wang, Ekin Akyurek, Antonio Torralba. Jacob Andreas, and Igor Mordatch. Pre-trained language models for interactive decision-making. arXiv preprint arXiv:2202.01771, 2022.
Nan Liu, Shuang Li, Yilun Du, Josh Tenenbaum, and Antonio Torralba. Learning compose visual relations. Advances in Neural Information Processing Systems, 34:23166-231788 2021.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models, arXiv preprint urXiv:2206.017/4 2022.
Ron Mokady, Amir Hertz, and Amit Bermano. Clipcap: Clip prefix for image captioning, arXiv preprint arXiv:2111.091 1..9344, 2021.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 12. 2021.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 2227-2237. Association for Computational Linguisties, June 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-87 8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint (:XXi:22044001255 2022.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.0611 17,, 2022.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL http: /tattp://axxxv..o
araai.iiv........ org/ abs/1908
10084
Robin Rombach, Andreas Blattmann, Domínik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic text-to-image -mage diffusion models with deep language understanding. arXiv preprint 11841 2022.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
11

---

Published as a conference paper at ICLR 2023
Mohit Shridhar, Lucas Manuelli, and Dieter Fox. Cliport: What and where pathways for robotic manipulation. In Conference on Robot Learning, pp. 894-906. PMLR, 2022.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zero-shot image-to-text generation for visual-semantic arithmetic. arXiv preprint arXiv:2111.14447, 2021.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2/08 10904, 2021.
Grady Williams, Andrew Aldrich, and Evangelos Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprint arXiv:1509.01 149, 2015.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowic7, al. Huggingface's transformers: Statc-of-the-an natural language processing. arXiv preprint arXiv:1910.03771,, 2019.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1 1686-1697, 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich rext-to-image generation. arXiv preprint 1rXiv:2206.1078, 2022.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong. Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rearranging the visual world for robotic manipulation. arXiv preprint arXiv:2010. 1440: 2020.
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Puro- hit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXi:2220.000598, 2022.
12

---

Published as a conference paper at ICLR 2023
Appendix
In this appendix, first show additional results in Appendix A We then show experimental details of each task in Appendix B The ethics statement of the Amazon Mechanical Turk experiment for video question answering is in Appendix C
A
ADDITIONAL RESULTS
A.1 IMAGE GENERATION RESULTS
We show more images generation results using different scorer models in Fig. A1 We find that most of the time, the proposed framework, either using single scorer model composing multiple scorer models, work well as shown in the first four examples in the left column of Fig. AA In some hard cases, some scorer models might fail (the rest examples in Fig. Al). However, there no discernible trend on which scorer is better on what tasks. The results of composing multiple scorer models are sigmificantly better than using single one, as different scorer models capture different aspects the information. For example, in the fifth example in the left column of Fig. Al PIC with classifier- free guidance (CLS-FREB and PIC with a pre-trained classifier (CLS) cannot generate an image with "tench", but PIC with the pre-trained CLIP (CLIP) can generate the correct result and the composed model (CLS-FREE CLS CLIP) also works. This is why we consider composing multiple scorers: to leverage the strength of each expert model and improve the worst case.
CLS--RRE
CLS-FREE CLS CLIP
CLS-FREE
CLIP
CLS + CLIP
CLS-FREE
CLS
CLIP
Sulphur-
crested
Hen
cockatoo
Goldfish
Parachute
Carousel
Diamondback
Tiger shark
jacumar
Tench
Beaver
Figure A1: Qualitative results. Image generation results using different models. Composing multiple scorers (CLS-FREE CLS CLIP) achieves best performances.
A.2 IMAGE GENERATION WITH DIFFERENT GENERATOR
Our framework can be applied to other generators as well. For example, we changed the generator, GLIDE, to Stable Diffusion (Rombach et al. 202T). The results of composing Stable Diffusion and classifier-free guidance (CLS-FREE) is shown in Table A1 Using more powerful pre-trained model can further boost the performance.
A.3 GRADE SCHOOL MATH QUALITATIVE RESULTS
Example results of different methods shown in Fig. Our method can solve math problems involving addition, subtraction, multiplication, and division, even for solutions with three-digit numbers. In contrast, GPT-FT often fails to understand math problems.
13

---

Published as a conference paper at ICLR 2023
Table Al: Our framework can be applied to other generators as well. In the image generation task, we use a new generator, boost the performance. Image generation results on ImageNet are reported.
(Rombach et al. 2021). Using a more powerful pre-trained model can further
Method Name
Generator
Scorer
18 1
FID
KID 4
PIC (G+E) PIC (G+E)
GLIDE Dable-Diffusiin
CLS-FREE
25.926
29.219
5.325
CLS-FREE
31.689
29.546
6.562
Grade school
Ground
GPT
GPT FT
PIC (G+E)
math questions
Truth
Pretrain
Q: In a dance class of 20 students, 20% enrolled in contemporary dance, 25% of the remaining eurolled in juzz dance, and the rest enrolled in hip-hop dance. What percentage the entire students enrolled in hip-hop dance?
60
25%
A: 60
Q: Melanie is a door-to-door saleswoman. She sold a third of her vacuum cleaners at the green house, 2 more to the red house, and half what was left at the orange house. Melanie has 5 vacuum cleaners left, how many did she start with?
18
A:5
A: 15
A: 18
Q: A log bank rolls in from the ocean to cover city. It takes 10 minutes to cover every 3 miles of the city. If the city is 42 miles acroes from the occanfront to the opposite
140
A: 140
inland edge, how many minules will it take for the fog bank to cover the whole city?
Figure A2: Grade school math example results. Our method can solve math problems involving addition. subtraction, multiplication, and division.
A.4 COMPOSING SCORER MODELS
The key idea of our method is to compose ensembles of pre-trained models, and the way to combine them can be variant. We add two additional experiments in this appendix: (1) using the best scorer (scorer that provides the highest score) and (2) using the weighted scores and add them together. Our method composes pre-trained models without training or finetuning, thus we did not learn separated weights for different models, But instead we add an experiment that uses the scorers (after softmax) of each scorer model as their weights and then composes the scorers using the weighted summed score. We compare these two baselines with our method that uses the summation of all scores. As shown in Table A2 using a summed score generates the best results.
Table A2: Different ways to compose models. Composing scorers using their summed score generates the best results. Image generation results on ImageNet are reported.
Method Name
Generator
Scorer
18
FID
KID 1
PIC (G+EI)
GLIDE
CLIP
25.017
30.462
6.174
PIC (G+E2)
GLIDE
CLS
22.077
30.871
7.952
PIC (G+E1+E2) Sum
GLIDE
CLIP CLS
30.438
29.543
5.435
PIC (G+E1+E2) Max
GLIDE
CLIP 4 CLS
4LSS
24.782
30.657
6.040
PIC (G+E1+E2) Weighted
GLIDE
CLIP+ CLS
24.728
30.669
A.5 ADDITIONAL BASELINE COMPARISONS
Image Generation. In Table A3 we compare our approach with a generative model specifically trained on ImageNet, Efficient- VDVAE (Hazami et al. 2022). Sitriient-DDAAR is an unconditional hierarchical VAE model. As illustrated in Table A3 despite Efficient-VDVAL is explicitly trained on ImageNet, it is much worse than our approach.
Robot Manipulation. We compare our robot manipulation method with the robot manipulation model, CLIPort (Shridhar et al., (022), used in the Socratic Models paper (Zeng et al. 2022). In Socratic models, the evaluation of robot manipulation consists of two steps: 1) GPT model translates a text goal into set of subgoals and 2) CLIPort model executes each subgoal. In contrast, in our robot manipulation task, 1) a vision-language model is used to translate an image goal into a set of subgoals and 2) an iterative MPC procedure is used to execute each given subgoal. Thus fair comparison is to compare the iterative MPC procedure used in our approach and the CLIPort model used in Socratic models. We thus compare method to the multilingual CLIPort model the performance of executing a single relation subgoal. CLIPort model obtains a success rate of 22.6% in this setting, while approach obtains success rate of 76.3%.
14

---

Published as a conference paper at ICLR 2023
Table A3: Comparison of our method and additional baselines. Image generation results on ImageNet are reported. Our method outperforms the baseline.
Method Name
Generator
Scorer
18
FID
KID
PIC (G+EE)
GLIDE
CLIP
25.017
30.462
6.174
PIC (G+EI+E2)
GLIDE
CLIP CLS
30.438
29.543
5.435
PIC (G+EL+E3)
GLIDE
CLIP + CLS-FREE
30,500
29.726
4,304
PIC (G+E1+E2+E33)
GLIDE
CLIP + CLS CLS-FREE
34,952
29.184
3.766
Efficint-VDVAE Hazami et al. 2022
6.566
143.642
23.007
B EXPERIMENTAL DETAILS
In this section, we provide more experimental details of each task. We use TITAN RTX 24GB GPUs for all the experiments.
B.1 IMAGE GENERATION
We use the reverse diffusion process of GLIDE, text-guided diffusion model, as the generator to generate image proposals. At each step of the diffusion process (corresponding to a step of the iterative refinement), we use the gradient from an ensemble of scorers to guide and update the generated proposals. We iteratively repeat this procedure until the final step.
As shown in Fig. A3 the image x generated at iteration & is first sent to the diffusion model to generate image proposal g4+1 The scorers provide feedback to refine the generated result. The CLIP model computes the cosine similarity between the image and text features as the score (we used the pre-trained CLIP model from (Ho & Salimans 2022).). The image classifier (Dhariwal & Nichol 2021) predicts the probability of the image matching the text label as the score. The scores generated by different scorers are summed, and their gradient with respect to x is used compute the next reverse prediction *+ The classifier- free guidance (Ho & Salimans 2022) can be treated as an implicit classifier that directly provides pixel- gradient feedback the generated image. Our framework enables the use of ensembles of different pre-trained models as scorers, significantly improving the zero-shot results by leveraging the strengths of multiple expert models.
Our implementation for image generation is modified based on the code of GLIDE (Nichol et al.) 202T) and the classifier guidance diffusion (Dhariwal & Nichol, 2021). We use DDIM to sample images from GLIDE in 100 steps. The guidance scale is set to 3.
B.2 VIDEO QUESTION ANSWERING
In video question answering, we use the proposed method to generate captions for the video frames and then use GPT-3 to summarize the captions to answer questions. We use GPT-2 as the generator and a set of CLIP models as scorers to generate captions for each video frame. The CLIP models (Radford
al. 2021
Reimers & Gurevych 2019) from the Huggingface library (Wolf al. 2019):
CLIP-32:
CLIP-14:
Fig. A4 shows the framework for generating frame captions. Given a video frame we generate sequence of words to describe it. To integrate feedback from scorers to the generator, similar to ZeroCap (Tewel et al., 202T), we define a context cache C, set of embedding functions in GPT-2, such as the embedding functions, K, Q. V, in the Transformer blocks.) that stores the context information generated so far, which updated iteratively based on the feedback from scorers. The prediction of the next word from the generator G is given by æ++1 G(z Ci), where G is the pre-trained language model. ZeroCap uses the following loss function to optimize C₁:
arg min argmin (CuP (G (Ix, .) ACEE (It, C,), &i), C,
(AI)
15

---

Published as a conference paper at ICLR 2023
Update based on the gradient of the summed
Image
Generator:
Diffusion
Model
Image,r
Image #+++1
Inpul lext: Goldfinch
CLIP score + Classifier score
Classifier-free guidance
Figure A3: Overview of image generation. We use the reverse diffusion process of GLIDE (Nichol et al. 2021). text-guided diffusion model, the generator generate image proposals. At each step of the diffussio process (corresponding a step of the iterative refinement), we use the gradient from an ensemble of scorers, such as CLIP (Radford et al. 2021), to guide and update the generated proposals. The image = generated at iteration is first sent to the diffusion model to generate an image proposal j**11, The scorers provide feedback refine the generated result. The CLIP model computes the cosine similarity between the image and text features as the score. The image classifier (Dhariwal Nichol (02T) predicts the probability of the image matching the text label as the score. The scores generated by different scorers are summed, and their gradient with respect to x² is used to compute the next reverse prediction *+1 Classifier-free guidance (Ho & Salimans/ 2022) can be treated image. We iteratively repeat this procedure until the final step. Our framework enables the of ensembles of different pre-trained trained models scorers, significantly improving zero-shot results by leveraging strengths of multiple expert models.
implicit classifier that directly provides pixel-wise gradient feedback to the generated
where 7 the feature of a video frame and 8t+1 is the next word predicted by the original language model. The CLIP loss LCLP optimizes C, to make the generated sentence describe the video frame. The second loss Lce ensures the new generated sentence is close to the sentence generated by the original language model.
Our implementation is based on the code of ZeroCap Tewel et al. updated using:
2021 The context cache C₁ is
VCP Ct)
C IVC. (I++1 2
(A2)
where p(Z++1 (C;) is the probability of predicting word Tea- given C.. Optimizing Eq. (Al can be achieved by conducting the gradient descent using Eq. (A2). In our experiments, we use steps of gradient descent. The learning rate is set 0.3.
In the video question answering tasks, we compose multiple CLIP scores and use their composed score optimize Ct:
arg min LaLP (G (rt, Ct), ()) + (G (zz, C)), C. LCI.P.muhiiinnua
(A3)
(G (z,, C,) +XCc (I,, C:), $e..))
(A4)
After several iterations, the updated C, is used to generate the next token F:41 = G( Ct). We repeat this process until generate the entire caption.
To answer the video questions, we cascade the generated captions of the video frames and the questions about this video to prompt GPT-3 to generate answers. For each video, delete the first
16

---

Published as a conference paper at ICLR 2023
CLIP CLIP
word generated after iterative refinement 7++1
candidate word P141
a
pasta
Context information C.
Generator: GPT-2
A
human
is
making
Figure A4: Overview of video frame captioning for video question answering. We use GPT-2 the generator and of CLIP models scorers generate captions for each video frame. To integrate feedback from scorers
define a context cache C2 (a set of embedding functions in GPT-2) that stores the context information generated so far, which is updated iteratively hased on the feedback from scorers. To update Cs, we first use G to generate set of candidate words X,+1 = {f++1), and then use feature distance (after softmax) between each sentence (the concatenation of previous words and each new word 22, , }. where #e+1 E Xx+11) and the video frame the CLIP score is the cross-entropy loss CCUp between this new probability distribution and the original distribution of the next word obtained from the generator G (see Equation in TTwee et al. (021)) The gradient of summed scores (multiple CLIP models) is propagated to G to update Ct (see Equation 5 in (Tewel al. (021)). After several iterations, the updated C: is used to generate the next token 2441 G(x,, Ct). We repeat this process until we generate the entire caption. We cascade the captions of multiple video frames and questions about this video to prompt GPT-3 for video question answering.
generator, similar to ZeroCap (Tewel al. 2021).
probability them matching.
Q: how many people there in the video
A: 2
# Q: what is behind the person white clothes
# A: tree
Q: what is in front of the person with braid
A: chair
# Q: what is the person white doing
# A: tie hair
Q: what happened to the person in gray after he threw goal
A: clap with your teammates
Summarize the following descriptions and answer the question shown above:
a Video showing new Hair tutorial; a video showing young blond hair clip attaching top pony tail of teens hair; ...; a video on the head hair clip website showing blonde long hair twisted in two knots.
Q: the person with a golden hair long hair
Figure A5: Prompt given to GPT-3 for video question answering. Text in black contains the question-answee answer pairs randomly sampled from the ActivityNet-Ou QA training dataset. Text in blue has the video frame captions generated by the proposed method. Text in orange the question about this video that needs answered.
10 frames and the last 10 frames to remove the beginning or ending advertisements. We then take 30 video frames evenly from the rest frames and send them to GPT-3. To guide GPT-3 to generate proper answers, we randomly select 30 question-answe
pairs from the training set of ActivityNer-QA (Yu) et al. 2019) and use them part of the prompt of GPT-3. As shown in Fig. A5 the prompt of GPT3 consists of examples of question-answer pairs, the video frame captions generated by the proposed method, and the question about this video that needs to be answered. The text generated by GPT-3 is used as the answer to the question asked. We also used the profanity check tool remove the improper answers.
thub. http:///gttha.b...ccccoeu222222/
17

---

Published as a conference paper at ICLR 2023
Question-solution classifier
word generated after iterative refinement di-11
candidate word 0241
5
Q: John buys 2 pairs of shoes for each of his 3 children. They cost $60 each. How much did he pay?
Context information C2
Generator: GPT-2
Figure A6; Overview solving grade school math problems. We use GPT-2 as the generator and treat the grade school math problem as a text generation problem. The scorer, a pre-trained question-solution classifier, provides the generator feedback to guide the next token's generation 1++1- We follow the approach used in VQA to iteratively optimize the generations based on the feedback from scorers, Our generator G first generates set of candidate words X2+1 = {अ++1) }. and then the classifier predicts the probability of each solution (the
concatenation of previous words each new word {aı, (2, .2.... .f441) where 2 E (x+1) matching given question. The classifier score is the cross-entropy loss between this new probability distribution and the original distribution of the next word obtained from the generator G. The gradient of the classifier score is used update Ct through iterative refinement (see Equation in (Tewel et al. (202T). The updated C2 is used to predict the next word 2++1 G(z+, Ct). We repeat this process until we generate the complete solution.
B.3 GRADE SCHOOL MATH
We treat the grade school math problem as a text generation problem. As shown in Fig. A6 GPT-2 as the generator and pre-trained question-solution classifier as the scorer. The pre-trained classifier is a binary classifier trained on the training set of GSM8K (Cobbe al.] 202T. Given math problem, such as "Natalia sold clips to 48 of her friends in April, and then she sold half as many clips in May. How many clips did Natalia sell altogether in April and May?", and an answer, such as "72". If the answer is correct for the given problem, then the label is 1; otherwise, the label is O.
After training, the classifier used as the scorer to provide feedback to the generator to guide the next token's 's generation d Similar to VQA, the generator G first generates set of candidate words X1+1 { ), and then the classifier predicts the probability of each solution (the concatenation of previous words and each new word #2, _ where ic+1 E xx+1) matching the given question. The classifier score is the cross-entropy loss between this new probability distribution and the original distribution of the next word obtained from the generator G (the way to compute the classifier score is the same as computing the CLIP score in VQA). We also used the cross-entropy loss fCE in Equation 2 of ZeroCap (Tewel al. 202T) ensure the generated sentence is grammatically sound. The context cache C is updated in the same way as the video question answering task, but we use the classifier score when providing the feedback to C. The updated C, is used to predict the next word #++1 G( C.). We repeat this process until generate the complete solution. Similarly to the video question task, we use 5 steps of gradient descent. The learning rate a is set to 0.3.
B.4 ROBOT MANIPULATION
In robot manipulation, we use the proposed method to manipulate objects in Ravens ZZng et al. 2020) to conform to set of object relations specified by text descriptions or real-world images. We use MPC+World Model as the generator and ViLD (Gu et al. 2021) as the scorer. As shown in Figure A7 given real-world image, our model manipulates objects in the environment to achieve state with objects having the same object relations as the given image. We first use ViLD to generate 2D segmentation of the world image and the corresponding text label, such as "mug". for each segment. We then use the relative pixel-wise offsets of segmentation masks and the text labels to infer a set of object relations (top panel of Figure A7).
18

---

Published as a conference paper at ICLR 2023
Object specifeed Imogee
Scorer
yelloe lowd
pink mag
Exiras object pelations
yslkee
Live mrgg in foor of not bowt
Chmmen View
Gual object relationn
yolkre lowdl pork regg in Boot yelkoe
Match Scoce 1-00
Extrae object
Live uaa Ine of not boot
Generstor: MPC Work
Candidate uzzion Alas
Summee
hood rall of yol lowd
World
VILD
pink
Mutch Scoce
Extrasi
object
love ura Ins..
reiaticus
Assioe sompiad in diffceent teemtoon ...
Artion with the miniarum summeed
View
Goel object rolution
rnd luxl the right ot yellow bowt
VILD
Mutch
Eutireci objeet eelations
pellew blee sang on top
Candidate un fim Medd
Summed ecuen: N
Comera View N
nol levil tha right of yolloe
VILD
park Boos of yeloo boit!
Not Matth Score N--1
Extract
objest
tle
milakion
MPC+World World Model as the generator ViLD as scorer to manipulate objects to conform to a set of object relations specified by text descriptions or wolldworld images. Top: given a real- world image, we first use ViLD to generate 2D segmentation of the real-world image and the corresponding text label, such "mug". for each segment. We then the relative pixel-wise offsets of segmentation masks and the text labels infer state Zt, we aim to generate an action a:-1 so that the new world state after executing a++1 has object relations closer to the object relations in the given image. To do this, we first use the generator (MPC+World World model) to generate a set of candidate actions {ā ) and the corresponding world states [ after executing each candidate action. For each new world state 2 we render N 2D images from N camera views. Each rendered image is sent to VILD to get segmentation map and text labels. We project the objects into 3D space based on the segmentation map and the depth map of the image. We then obtain the object relations hased their 3D positions and predieted text labels. We compare the object relations obtained from each rendered image and the object relations obtained from the real-world image to compute matching; otherwise, 1.
Figure A7: Overview robot manipulation. We use MPC+
of object relations. Bottom: Given the
world
score. The score O the relations are We sum the scores from each rendered image to obtain the final score. We choose the action d that leads to world state with the minimum summed score. We execute 4++1 in the environment and get state 11+1- We repeat this process until the task is accomplished we the final step T.
Given the current world state Jr. we aim to generate action d:+-1 so that the new world state after executing a has object relations closer to the object relations in the given image. To do this, we first use the generator (MPC+World Model) to generate set of candidate actions {a } and the corresponding world states {at after executing each candidate action. For each
world state $ we render N 2D images from N camera views. Each rendered image is sent VILD to get segmentation map and text labels. We project the objects into 3D space based on the segmentation map and the depth map of the image. We then obtain the object relations based on their 3D positions and the predicted text labels. We compare the object relations obtained from each rendered image and the object relations obtained from the real-world image to compute the score. The score is the relations are matching; otherwise, 1. We sum the scores from each rendered image obtain the final score. We choose the action a++1 that leads to a world state with the minimum summed score. We execute O++1 in the environment and get new state T++1- We repeat this process until the task is accomplished or we are at the final step T, where T equals to the number of relations extracted from the real-world image.
19

---

Published as a conference paper at ICLR 2023
Committee On the Use of Humane Experimental Subjecte
IBB Admin:
No greater than minimal risk
Approval Date:
Last Approvel Date:
Expiration Date:
01/29//223
Anticipated Start Date:
Anticipated End Date:
Submission Statue
Figure A8: Screenshot of the approval form from the Committee on the Use of Humans as Experimental Subjects.
B.5 A UNIFIED FRAMEWORK FOR COMPOSING PRE-TRAINED MODELS
Our method shares some similar architecture with existing works, such ZeroCap (Tewel al. 2021) and CLIP- guldee diffusion models (Nichol et al. 2021]. J. However, the focus of our paper is to propose general framework for composing different pre-trained models across variety of tasks, and these particular methods are concrete instantiations of our proposed framework. In addition, in this work, we also illustrate how we may combine ensembles different pre-trained models as scorers to leverage the "wisdom of the crowds" where each scorer provides complementary feedback to the generator, compensating for potential weaknesses of other scorers. Through iterative optimization and the composition of multiple scorers, our method shows effective zero-shot generalization ability on various multimodal tasks.
C ETHICS STATEMENT OF AMAZON MECHANICAL TURK EXPERIMENTS
To evaluate approaches on solving the zero-shot video question answering tasks, we ask workers from Amazon Mechanical Turk to evaluate the generated answer based the video and the asked question. Before showing the questions and answers to the workers, we used the profanity check tool (https://gt) /tttps://github com/ wwhou8422 wwwp:///22//offattt///. http:/oofaattttchccci
remove the improper questions and answers. As shown in Fig. A8, this experiment was approved by the Committee on the Use of Humans Experimental Subjects. screenshot of the task is shown in Fig. A9 The instructions shown to participants are listed as follows:
Instructions: By making judgments about these questions and answers, you are participat- ing in study being performed by [XXX]. Your participation in this research is voluntary. You may decline further participation, at any time, without adverse consequences. Your anonymity is assured; the researchers who have requested your participation will not receive any personal information about you.
Given a video, a question, and a generated answer, the workers from Amazon Mechanical Turk measure whether the answer is correct for the given question and video. Each video shows three question-answer pairs (only one question-answer pair shown in the screenshot). The answers are generated by different methods. The workers not told which method generates each answer. The workers are asked to choose "yes" or "no". If the worker thinks the answer matches the given video and question, they should choose "yes": otherwise, "no".
To control the quality, each task is evaluated by three different workers. The workers are required to have approval rate greater than 98%. Our test shows that each task takes around 10 seconds, but the workers are given up to one hour to complete each task. The workers are paid $0.05 for finishing each task with estimated hourly payment of $18. more than the United States federal minimum wage. There are 33 workers in total who joined our experiment.
20

---

Published as a conference paper at ICLR 2023
you participatin
partcipation nosarch vaturtary
may decine turter partio pation, time, without adverse conoequencces Your ancrymiit issured; the researches who have requesned partcipation will not receive pansonal information about
answer correct comesponding question
video?
Question what color of the clothes does the atfieee wear
green
No. the answer not coneet.
Figure A9: Screenshot Amazon Mechanical Turk we used for the video question answering experiment. Workers are shown video, three questions, and the answer to each question. The answers are generated by different methods. The workers are not told which method generates each answer. The workers are asked to select "yes" "no" based on their measurement of whether the answer is correct for the given video and question.
21


Reviews:
[{"review_id": "vc3JRNL2MJ", "reviewer": "Reviewer_dTDv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem of leveraging individual expertise of various large pretrained models to compose an ensemble for solving various multi-modal problems in a zero-shot manner. The authors propose a unified framework, called PIC, in which the pretrained models are composed using closed-loop iterative consensus optimization. Specifically, for a given task, the framework employs a single model as a generator and an ensemble of scorers. The generator iteratively generates proposals and each scorer provides a feedback score indicating their agreement, then the generator refines its output until all the scorers achieve a final consensus. Experiments are shown on four zero-shot tasks - image generation, video question answering, grade school math, and robot manipulation. Results show that a composition of the pretrained models with an ensemble of scorers outperform baselines with a single scorer by a considerable margin.", "summary_of_the_review": "I think the identified problem is important but the proposed method is a bit naive and experiments are not convincing. The proposed method should be evaluated and analyzed using extensive experiments resonating with the motivation to solve the problem. In the rebuttal I expect new experiments and analysis listed in the weaknesses to convince me that the proposed method has practical use and outperforms the missing baselines and exiting methods.\n", "strength_and_weaknesses": "**Strengths**:\n\n- The paper writing, presentation, and figures are very good and well-structured.\n- The motivation to solve the problem of composing pretrained models is interesting and differences with prior works has been well highlighted.\n\n**Weaknesses**:\n\n-  As the main motivation of the paper is to leverage the individual expertise of the pretrained models, it would be great to have some qualitative analysis on the same. Specifically, as can be seen from Table 1 that the best performance is obtained when all the scorers E1, E2, E3 are used as an ensemble, more insights on identifying individual expertise of the scorers can make the submission stronger. E.g. “which groups of classes does E1 expertise which E2 and E3 fails to?”, “What information does CLS-FREE provide which CLIP and CLS do not?”, “How do the results change with different generators?”. Getting answers to similar questions would be very interesting.\n\n- How about selecting one best scorer for providing feedback to the generative model? I understand that in the context of 3 scores it does not make much sense and the results already show that the combination works the best. However, in many practical scenarios given that HuggingFace models are easily available, there will be access to 100s of diverse models that can be considered as scorers. In those scenarios, is it still advisable to consider the combination for providing feedback to the generator? Authors should perform experiments with a diverse set of models trained using different datasets and of different sizes to verify how the capacity and knowledge of the individual models as scorer effects the final performance. The experiments in the current form are limited to demonstrate the potential of the method.\n\n- What about confidence of predictions for the different scorers? Ideally there should be a learnable weight signifying the importance of a scorer for a particular task and sample. I would encourage the authors to perform experiments for verifying the effect of such weights in the final performance.\n\n- Most of the experiments mainly compare the performance of PIC with different variants of the same method. Given that the proposed framework focuses on composing different models, how is comparable to state-of-the-art methods, e.g, Image generation or video captioning? Due to the lack of comparison with SOTA methods, it is not clear why one should use this framework in their application instead of using any other existing methods.\n\n- What about the memory requirement and FLOPs during inference and training of this method comparable to existing baselines?\n\n- How were the accuracies for the compared baselines JustAsk and JustAsk (FT) computed in Table 2? Was Amazon Mechanical Turk used for them as well? \n\n- It would be great to have a baseline experiment in which PIC is asked to select an answer from the same pre-defined set used for other baselines instead of using a free-form response approach. How is that baseline compared to state-of-the-art methods on the same dataset? A thorough comparison and analysis should be performed to justify the advantage and utility of the proposed framework.\n\n- As mentioned for Image Generation, analyzing the contribution of each of the scorers would be useful. E.g. “What attributes does CLIP-multilingual specialize in?” Analysis on the usage of GPT-2 is missing. How much does GPT-2 contribute towards the overall performance in video question answering?\n\n- The paper also lacks some implementation details of the optimization procedure. Specifically, details like number of iterative steps used for optimizing for different tasks, learning curves with iterations (can be included in supplementary), optimization time required for each task, details of the context cache - size, initialization, etc.\n\n- Experiments showing applicability of the framework to different permutations of possible generators and scorers are missing. E.g. getting results by using different generators for a given task.\n\n- Experimental comparison with the discussed prior work Socratic Models (Zeng et al., 2022) is also not provided. If it is not possible to compare them in the experimented tasks due to language interface, how about comparing the proposed framework on the tasks used for zero-shot evaluation on Socratic Models. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bT5JaSYvb2n", "reviewer": "Reviewer_kbow", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to iteratively optimize a system of one generator and multiple classifiers(scorers) for zero-shot generation tasks. It unified four multimodal generation tasks under the same framework. And experiments show that it can have a satisfactory generation ability on those tasks.", "summary_of_the_review": "Overall, I prefer to weak rejection because of limited novelty and lack of comparison.", "strength_and_weaknesses": "Strengths:\n1. This framework doesn't need further task-specific pre-training and finetuning. It only needs to iteratively update the pre-trained models to refine the generation result. \n2. The idea of using scorers to guide generators to reach a consensus is interesting and seems to be generalizable to four different tasks as in this paper.\n\nWeaknesses:\n1. For the methodology, this paper mainly instantiate the proposed method into two procedures: (1) continuous updating on image generation. (2) Discrete optimization over samples.  As for the first one, the whole method is quite similar to VQGAN-CLIP[1] in that they both optimize the image generator's input by a CLIP model to generate a better image. The difference is: at VQGAN-CLIP, the image generator is VQGAN and the optimized input is a latent feature, while this work used a diffusion model and the optimized input is a noisy image. Despiting the minor difference, those two share similar intuition and working mechanisms. For the second procedure, the caption generation optimization is almost the same as ZeroCap[2]. As mentioned in Figure A4's caption, this paper exactly used Equation 5 in ZeroCap to do the optimization. So is the Grade School Math text generation task. Overall, Although I appreciate the author(s)' efforts in unifying the different tasks in the same framework of iterative optimization, I still think the distinction between previous works and methodology contributions to the community is limited.\n2. For the experiments, all those four tasks lack enough comparison with previous works. In Zero-shot videoQA, only JustASK is compared. In GSM8K, only GPT is compared. Moreover, the comparison seems not fair to me because the proposed PIC assumes to have a question-solution classifier, while GPT-FT(Pretrain) does not. In image generation and robot manipulation, no other baseline methods are compared.\n3. Possibly Wrong/Confusing Illustrations. a). In Figure 2c, how does the input video frame feed into GPT-2? b) The explanation of $L_{CLIP}$ in Video question answering (VQA) of Sec3.1 is so confusing. I cannot really understand until I go back to read ZeroCap.\n\nRef:\n[1] VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance, ECCV2022.\n[2] ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic, CVPR2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DrwZnpAXnK", "reviewer": "Reviewer_BdG8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an ensembling framework for use the capabilities of multiple foundational models. In cases where there isn't a unified interface to interact with a number of foundational models, it provides a unified framework to allow multiple foundational models to interact with each other. The paper proposes to divide the foundational models as \"Generators\" and \"Evaluators\". The \"Generator\" set of models generates proposals and the \"Evaluator\" sets of models evaluates the proposals and are used to iteratively refine the proposal using its gradients. ", "summary_of_the_review": "This paper proposes a simple technique for combining the strengths of several large scale foundational models. It is especially interesting when large scale data across multiple domains is not readily available such as Robotics and Language Models. It would have been interesting to see some tasks which are only enabled with this system which would not be possible without the presence of this system.", "strength_and_weaknesses": "Strengths:\n\n* Elegant technique for combining the strengths of complementary models which are trained on large scale dataset \n* No retraining required which is expensive in most cases with large scale models\n* Results show effectiveness of the technique \n\nWeaknesses:\n\n* The change in the scores with the proposed technique seems minimal. \n* In industry use cases the GPU memory would overflow to fit all the models simultaneously.\n* The greatest benefit of this technique seems to be for Robotic Manipulation, reducing the scope of impact. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7bbBSNGbK7", "reviewer": "Reviewer_To1A", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework for using pre-trained models for various recognition and reasoning tasks. The idea is to use a \"generator\" and an ensemble of \"scorers\" to iteratively improve the output: the generator proposes an output that the scorer scores. This general framework is demonstrated on multiple tasks, including visual question answering, solving math problems, image generation and robot manipulation. ", "summary_of_the_review": "I find the general framework quite interesting and novel. I think the connections to traditional iterative methods for structured prediction problems are very intriguing and should be discussed. That said, I find the approach innovative enough to be published.", "strength_and_weaknesses": "Strengths:\n- The framework is very elegant. It uses the complimentary strengths of, for example, language models (as generators) and retrieval models like CLIP (as scorers).\n- The notion of iterative feedback to get consistency between the generator's output and the scorer is innovative.\n- The results, especially the zero-shot results, are quite impressive.\n\nWeaknesses\n- This proposed framework is very similar to the kind of optimization that appears in older literature on MAP estimation in structured prediction. A discussion on this would be useful.\n- Also worth discussing would be the effect of the limitations of the pretrained models on the proposed framework.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster