Towards Robust Multi-Modal Reasoning via Model Selection
Authors: Xiangyan Liu
Rongxue LI
Wei Ji
Tao Lin
Venue: ICLR 2024 poster (2024)
Keywords: robust model selection
multi-modal learning
multi-step reasoning with LLM
Primary area: general machine learning (i.e., none of the above)
TL;DR: We first formulate model selection in multi-modal reasoning with efficient $\textbf{\textit{M}}^\textbf{\textit{3}}$, capturing sample, model, and subtask dependency, and introduce the MS-GQA benchmark.

Abstract: The reasoning capabilities of LLM (Large Language Model) are widely acknowledged in recent research, inspiring studies on tool learning and autonomous agents. LLM serves as the ``brain'' of the agent, orchestrating multiple tools for collaborative multi-step task solving. Unlike methods invoking tools like calculators or weather APIs for straightforward tasks, multi-modal agents excel by integrating diverse AI models for complex challenges. However, current multi-modal agents neglect the significance of model selection: they primarily focus on the planning and execution phases, and will only invoke predefined task-specific models for each subtask, making the execution fragile. Meanwhile, other traditional model selection methods are either incompatible with or suboptimal for the multi-modal agent scenarios, due to ignorance of dependencies among subtasks arising by multi-step reasoning.

To this end, we identify the key challenges therein and propose the $\textbf{\textit{M}}^\textbf{\textit{3}}$ framework as a plug-in with negligible runtime overhead at test-time. This framework improves model selection and bolsters the robustness of multi-modal agents in multi-step reasoning. In the absence of suitable benchmarks, we create MS-GQA, a new dataset specifically designed to investigate the model selection challenge in multi-modal agents. Our experiments reveal that our framework enables dynamic model selection, considering both user inputs and subtask dependencies, thereby robustifying the overall reasoning process. Our code and benchmark: https://github.com/LINs-lab/M3.

Published as a conference paper at ICLR 2024
TOWARDS ROBUST MULTI-MODAL REASONING VIA MODEL SELECTION
Xiangyan Li Rongxue Li Ji Tao Lin
1iu xi siangyaneu nus nus, Lirongxue@westlake edu. cn; wei m10553523egmal com: lintao@west www.sttlleeeeeeeeee......
Westlake University ZZhejiang University "National University of Singapore
ABSTRACT
The reasoning capabilities of LLM (Large Language Model) are widely acknowl- edged in recent research, inspiring studies on tool learning and autonomous agents. LLM serves as the "brain" of the agent, orchestrating multiple tools for collaborative multi-step task solving. Unlike methods invoking tools like calculators or weather APIs for straightforward tasks, multi-modal agents excel by integrating diverse AI models for complex challenges. However, current multi-modal agents neglect the significance of model selection: they primarily focus on the planning and execution phases, and will mainly invoke predefined task-specific models for each subtask, making the execution fragile. Meanwhile, other traditional model selection methods are either incompatible with or suboptimal for the multi-modal agent scenarios, due to ignorance of dependencies among subtasks arising by multi-step reasoning.
To this end, we identify the key challenges therein and propose the M framework a plug-in with negligible runtime overhead at test-time. This framework improves model selection and bolsters the robustness of multi-modal agents in multi-step reasoning. the absence of suitable benchmarks, we create MS-GQA, a new dataset specifically designed to investigate the model selection challenge in multi-modal agents. Our experiments reveal that our framework enables dynamic model selection, considering both user inputs and subtask dependencies, thereby robustifying the overall reasoning process. Our code and benchmark: https: / / g thub. com LINNs lab/M3.
INTRODUCTION
Large Language Models (LLMs) (Brown al., 2020; Ouyang al., 2022: Chowdhery al., 2022: Zhang et al., 2022b; Touvron et al., 2023) recently emerged to show great potential for achieving human-level intelligence, leveraging the key reasoning ability to tackle complex problems.
As a key step towards artificial general intelligence, the study on multi-modal learning has soon evolved into two paradigms, either training large end-to-end models like PaLMMEE (Driess et al., 2023) and Mini-GPT4 (Zhu et al., 2023) for direct task resolution, or employing LLMs to decompose tasks into subtasks for smaller yet specific models (Gupta & Kembhavi, 2023; Surís et al., 2023; Shen et al., 2023; Gao et al., 2023a). The latter paradigm- as evidenced in the significant attention since tool learning (Schick et al., 2023) and autonomous agents (Reworkd, 2023; Richards et al., 2023) demonstrtts the immense potential in addressing complex real-world problems, where multiple AI models collaborate through malti-modal multi-step reasoning process.
In the realm of multi-modal reasoning scenarios, existing multi-modal agents (Gupta Kembhavi. 2023; Shen et al., 2023) emphasize planning and execution phases, while neglecting the critical model selection phase. As exemplified in Figure/ 1 (a & b), simplistic model selector relies on predefined task-specific models for subtasks, increasing the likelihood of intermediate errors and compromising the overall reasoning process. Moreover, existing traditional model selection methods, though effective in various domains (Zhao et al., 2021; Park et al., 2022; Lee et al., 2022; Zitovsky al., 2023), primarily focus on selecting single model from multiple candidates per sample.
"Equal contribution. Work was done during Xiangyan's visit to Westlake University. "oorrsppnniigg author.

---

Published as a conference paper at ICLR 2024
(a) Muiti-modal Multi-step Reasoning
(b.1) Execution process based simplistic model selection
Question:
8
What color the the object next the cal?
X
"Red"
simplistic model selector
m&
M
M
Answer with reasoning graph:
(b.2) Execution process based robust model selection
Jociaaa the abyje! nast the cat
what coloris is the mose?
localze tits otync!
"Black" T
M
M
Mi
robust model selector
(c) Three model selection paradigms
Localization (Texd-guided
+4.35%
+2.69%
Objjctt Dataction)
Visual Question Anowering
simplistic
traditional e.g. MetaGL
subtask dwnpendency-aware
e.g. HuggingGPT
e.g. MP (ours)
m: My candidate models
Figure 1: I:luutraaiio of the multi-modal multi-step reasoning process and three model selection paradigms
------
within. (a) shows how multi-modal agents utilize LLMs to decompose complex multi- multi-modal tasks, resulting in multi-step reasoning process where each node corresponds simpler yet more specific subtask. (b) highlights that compared robust model selector, simplistic model selection methods are more prone to generating wrong outcomes at intermediate subtask stages, thereby impacting the ultimate reasoning result. Here, ma indicates the i-th model is selected and the color means the corresponding subtask type. (c) numerically illustrates the comparative outcomes of model selection methods from different paradigms in multi-moda reasoning.
Adapting these methods multi-modal reasoning scenarios, which necessitate multiple models for subtasks, is challenging due to the oversight of subtask dependencies.
To this end, we formally define the problem of model selection in multi-modal reasoning scenarios as our first contribution, and then introduce the M framework (Model Selector for Multi-Modal Reasoning) as preliminary remedy for the field. Given the lack of benchmarks, further create a new dataset named MS-GQA (Model Selection in GQA (Hudson & Manning, 2019)) to facilitate research in the field. In detail, M represents the multi-step reasoning process as a computation graph, with nodes corresponding to reasoning subtasks. Multi-modal encoders and models' embedding table transform input and selected models into node features, where a computation graph learner then models the relationships between input, selected models, and subtask dependencies predict the execution status. M showcases superior performance in the model selection process for multi-modal models, with trivial overhead in selection.
Our key contributions are summarized as follows:
We formulate the model selection problem in multi-mode reasoning contexts as an initial endeavor. We introduce M model selection framework for multi-modal models with multi-step reasoning, jointly modeling the relationship between samples, selected models, and subtask dependencies. We create comprehensive dataset called MS- UIS to facilitate the research for the community.
We provide an effective yet efficient model selection solution, with trivial test-time overhead.
2 RELATED WORK
2.1 MULTI-STEP REASONING WITH LLM
A line of work utilizes LLMs to interact with simple APIs, including WebGPT (Nakano et al., 2021). Toolformer (Schick et al., 2023). PAL (Gao et al., 2023b), and ToolkenGPT (Hao et al., 2023). enabling the manipulation of tools like web browser and caleulators. Their capabilities enhanced by behaving as autonomous agents (Richards et al., 2023; Nakajima, 2023; Reworkd, 2023), where they reason, break down complex tasks into subtasks, and iteratively execute those subtasks until the desired goals are achieved. Note that these studies only call simple deterministic APIS for text tasks. To tackle complex multi-modal tasks, researchers are increasingly expanding tool libraries with trained AI models (Wu et al., 2023; Huang al., 2023; Surís et al., 2023). For instance, Vis- Prog (Gupta& & Kembhavi, 2023) and HuggingGPT (Shen et al., 2023) employ LLMs to decompose complex tasks into subtasks and connect various Al models to address them. Chameleon (Lu et al.,
2

---

Published as a conference paper at ICLR 2024
2023) is a plug-and-play compositional reasoning framework using richer set of tools. Unlike the aforementioned approaches that provide a static plan without considering dependent subtasks, AssistGPT (Gao et al., 2023a) and AVIS (Hu et al., 2023) dynamically strategize the utilization of external tools based on the intermediate results of multi-step reasoning. However, all existing methods lack model selection consideration, resulting in reasoning instability.
2.2 MODEL SELECTION
The research on model selection can date back to Forster (2000), and was further extensively investi- gated in various aspects: 1) meta-learning methods (Zhao al., 2021; 2022; Park et al., 2022), 2) non- meta-learning methods (Ying et al., 2020; Zohar et al., 2023), 3) model selection for ensemble (Kotary al., 2023), 4) model selection with language models (Zhao al., 2023; Hari & Thomson, 2023), 5) new metrics for model selection (Zhang et al., 2021; Yang et al., 2023a), and others (Chen et al., 2023; Lee et al., 2022; Zitovsky et al., 2023). We discuss the most relevant two lines of research follows. Meta-learning- based approaches utilize the similarity between new instances and historical instances to predict the performance of candidate models. For example. MetaOD and ELECT (Zhao et al., 2021; 2022) address the unsupervised outlier model selection problem, where they extract meta-features by considering input-specific characteristics. MetaGI. (Park et al., 2022) further extends (Zhao et al., 2021) to select a model for each new graph. However, in our multi-modal multi-step reasoning sce- narios, there is currently a lack of established approaches for extracting instance-wise meta-features. In contrast, non-meta-les meta-learnnngg-- -bsed methods resort to using complex networks to learn the rela- tionship between instances and model choices, without using metau-ecturess Auto-Sclector (Ying et al., 2020) employs a pre-trained model selector and parameter estimator to automatically choose an anomaly detection model for incoming time-series data. LOVM (Zohar al., 2023) employs textual dataset descriptions to train linear model that predicts the performance of vision-language candidate models. EMMS (Meng et al., 2023) employs weighted linear regression to estimate multi-modal model transferabilityy. Note that these methods are restricted to one-step selection and overlook subtask dependencies in multi-step reasoning. and thus infeasible for model selection in multi-step reasoning. In the context of using LLMs for multi-modal models in multi-step reasoning, existing methods either rely on external metrics like download counts for model selection, or explicitly specify the use of a specific version of model for a designated task without any selection process (Surís et al.. 2023). None of the prior work takes the challenges of multi-step reasoning dependency into account.
3 MODEL SELECTION HARNESSES THE MULTI-MODAL REASONING
3.1 ON THE CHALLENGES OF MULTI-MODAL MULTI-STEP REASONING
Given a complicated input query, the contemporary multi-modal multi-step reasoning process nor- mally involves calling LLMs to decompose users' input to subtasks, in which the reasoning logic can be constructed as a task graph by composing dependent intermediate subtasks.
The problem of selecting models for subtasks along the task graph can be defined below:
Definition 3.1 (Model selection for each subtask type on a multi-modal task graph), Let mii. j. t) denote a model choice of subtask type t for the i-th sample x,, where mii, t) indicates the j-th choice from the model zoo for subtask type I on x.. Each subtusk type has n4 choices, forming a set {m((s,,() 1) j E [n=]}- Assume a task graph consists of K subtask types, then all potential model selection choices for the task jj E [nee, E [K]], with size |C] III Rr. The optimal choice of models on the task graph for i-th input can be represented as ct == (m( [K]],
gruph can be defined C C= [mm(- 1)
where we select the optimal model index jī for each subtask type t.
Though Definition 3.1 fully characterizes the procedure of traditional model selection methods as evidenced in Figure 2 and Section 2.2, it leaves the unique challenge emerged in the multi-modal multi-step reasoning untouched, namely the critical subtask dependency defined below.
Definition 3.2 (Subtask dependency on modal task graph). Given the i-th input x. with embeddings from various modalities. Its multi-step reasoning procedure decomposed by an LLM can be described as a directed acyclic computation graph G; = (V,, Tr, Er} with nodes corresponding to multi- modal subtasks, where
V; -(ui,e k E [L] is the subtask nodes in the graph, Ua is the k-th node in Gn and L L:= [Va];
. T [tr. kE [L]} is the set of subtask types in V., t. is the subtask type of k-th node in G::
E, is the set of edges that connect pairs of subtask nodes in the graph G..
3

---

Published as a conference paper at ICLR 2024
Multi-modal Input
Decomposed Computation Graph <ft
Subtask Dependency-Awar
Simplistic
Traditional
Ma
Mpi
Whit the pothern
M
M
of the phirt?
(h)
M
Ma
Mm
Mæ
>
Mie
dag right
mm
Mix
M
How many lage the
uninals both
M
Ma
Mis
Ma
Mis
MM
lege
mai
Inputs differences: (a), (b), (c)
External Metrics
Input Information
Input Information & Subtask Dependency
Similar inputs: (c), (d)
(e.g. Download
a Figure 2: Comparison of three model selection paradigms under various inputs. The model selection pro- cesses the three paradigms, from left (simplistic) to right (subtask dependency-awarry, become progressively more fine-grained. "Siippliticc" is inflexible and can be considered input-agnostic. "Traditional" can solely depend on subtask type and the corresponding original imput information for model selection. When inputs similar, "Traditional" cannot provide diverse model selections "Subtask Dependency-Awares which leverages differences in reasoning logic to offer more varied and suitable model choices. Note that node (green cirelc) in the figure denotes Python module invocation, which does not entail model selection.
Counts)
Remark 3.3 (Generalizing model selection beyond the subtask type). To ease the presentation, Definition 3.1. Definition 3.2, and the main content below only consider selecting the model per subtask type. Though being sufficient for traditional model selection methods, such formulation can be further generalized performing model selection per subtask node (rather than per subtask type) in our case, and we leave it for future research.
Challenges: on the infeasibility of adapting existing model selection methods. Existing methods (see details in our Section 2.2), either the current preliminary strategies for multi-modal agents (Gupta Kembhavi, 2023; Surís et al., 2023; Shen et al., 2023; Gao et al., 2023a), or model selection methods from other domains (Zhao al., 2021; 2022: Park al., 2022: Ying al., 2020; Zohar et al., 2023), do not take the subtask dependency into account-ar evidenced in the following case study-making the model selection for multi-modal model with multi-step reasoning non-trivial. We describe the previous two model selection paradigms (see examples in Figure 2):
1. This paradigm is primarily applied in some recent multi-modal agent frameworks, where each subtask type will be straightforwardly allocated model via some external metrics (e.g..
download counts recent release dates).
2. As has been widely used in other fields like outlier detection and graph learning, this paradigm focuses matching the optimal model through the input information well the subtask type. Given the limitations of previous methods in adapting to the timely multi-modal reasoning scenarios, the following section, we incorporate the graph information G. to capture subtask dependencies.
3.2 M A FRAMEWORK OF MODEL SELECTION FOR MULTI-MODAL REASONING
An ideal model selection solution multi-modal reasoning scenarios discussed above motivates us to jointly model the subtask dependency with input sample features. We depict our unified model selection framework M below towards this goal.
Overview.
The model selection process can viewed as estimating the performance multi- modal input over the corresponding task graph, in which the choice of model selection each subtask
4

---

Published as a conference paper at ICLR 2024
Forward workflow
virtual starting node (mult mouls-modal imp.l)
0000
sublask noces (mult: modal Al lask)
ending (Python module)
Ma Mi ma
canckkow modele
Embedding Table 1/5
Computation Graph Learner %
Ma Mi Meo mo
(scores)
0000
fr
Model Selection Choice
Task Graph
Node Embedding
Joint Modeling
(b) Model selction test-time
multi-modal Information
Learner
sublask dependency
best choice with highest score
What is the patiern the shir??
Figure 3: Illustration of M (a) depicts the forward computation process: 1) Task Graph: An initial virtual mode represents the multi-modal input. Specific models are assigned to each subtask node based on the respective subtask type. 2) Node Embedding: Features are extracted using the multi-modal encoder (-) and embedding table (-) for the initial virtual node and subtask nodes. 3) Computation Graph Learner: The computation
graph, including node features and subtask dependencies (edges &d), serves input learner <(-). contributing to the predicted execution status s!. (b) illustrates the process of ranking and selecting the model selection choice with greater likelihood of success.
node propagates to the next node on the directed task graph. We introduce the notion of meta-training and train a proxy on it to suggest the optimal model choice of task graph on unseen input. In detail,
Training the proxy. Given input sample models the task graph cf E C, aim to model the relationship ø v between (x, G. c,) and the execution status pl {0, 1], where ó and denote the learner and feature extractor, respectively. Here we simplify our setting by only considering the binary execution status, but the principles therein can be generalized
Amain = (xi,... xN for each choice of
continuous case.
Model selection for task graph on the unseen sample. We estimate the status of potential model choices {s} Φ o (x., Gi cf) cf E CJ, and only keep executable choices for further selection.
3.2.1 TRAINING THE PROXY Lookup table and node embedding v.. We adopt similar approach to treatments in multi-modal learning and GNNs (Li et al., 2022; Veličković et al., 2017). Using & := [wh, dèz]-when t/1 denotes the multi-modal encoder and th2 represents the model embedding table-we we map subtask nodes to their corresponding embeddings through H == (xx G. c)) e (L+11)xd
More specifically, we first slightly abuse the notation and form an augmented computation graph G. = (V., T,, E,): we introduce a virtual starting node 0.,0 to incorporate the multi-modal input information x; and use subtask nodes {ui, k [4]] to indicate the execution dependency of subtasks. We then employ two encoders th, the separately to retrieve the node embeddings and use additional W, E dR xd and W2 E Rddaxd to unify them in a shared feature space.
For the virtual starting node Us we utilize the off-the-shelf embedding as b ui (x,) For other nodes, we use lookup table to retrieve node embedding for each subtask type, namely h 2 (m(i, 4.))) TR where 4, E T, and d2 is the model embedding dimension.
multi-modal encoder to extract node Rd,, where dj is the input embedding dimension.
Joint modeling of node embeddings and subtask dependency á. We further leverage computa- tion graph learner to learn the task embedding over 1) sample embeddings. 2) model emheddings, and 3) subtask dependency information. A final linear layer with non-linear activation function is stacked
5

---

Published as a conference paper at ICLR 2024
on top of the learned task embedding to estimate si == Φ (x,, 9 c,), where cf C. In detail, (HH Ei).
Hi (x,, Gi, c
(1)
The output of measures the degree of match between imput sample x, and model selection choice cj, where a higher value signifies a greater likelihood of success in multi-modal reasoning scenario. Note that theoretically, any neural network capable of handling directed acyclic graphs can serve as the backbone for computation graph learner. See Section 4.2 and Appendix D.3 for details.
Optimization over e and ø. For given input sample x,, we aim to learn the model to estimate execution status per the choice cj of models along the task graph, namely correlating 5t E [0. 1] with ground truth pa {0, 1]. Thus, the optimization can be viewed as multi-label classification problem. The conventional choice of using instance- saste Binary Cross Cross-Entropy (BCE) loss in the multi label classification community (Tsochantaridis et al.. 2005; Wehrmann et al., 2018) 8) only aims to build a mapping between (x, (x, c and sy. and thus suffers from the optimization difficulty caused by prediction independency of ef C on the input x,, Therefore, we employ Categorical Croos-EEtroopy (CCE) (Su et al., 2022) as our objective function, using a list-wise approach to model ii and for (x,, Ge C) and {s} c C}. Our goal is to promote higher scores of for executable choices (p: = 1) and lower scores for non-executable choices:
C. = log (1 Eelee ec 1 exp((s)) + log (1+ - exp(s)((
(2)
See Appendix B and D.4 for more details loss design and comparison of different loss functions.
3.2.2 MODEL. SELECTION FOR TASK GRAPH ON THE UNSEEN SAMPL.E
Once we learn the relationship & o tb we can estimate execution status sf for choices of models on the task graph c/ E C, and transfer to the final model selection upon other criteria. For the sake of simplicity, in our evaluation we primarily select the c via the maximal execution probability: c = arg maxc/ec & o (zx,, G. c.). Other metrics, c.g. computation cost, can be further integrated to trade off efficiency and robustness.
4 EXPERIMENTS
4.1 BENCHMARK: MS-GQA
Query
22%
Logical
As our side contribution, introduce the first benchmark, MS- GQA (Model Selection in GQA (Hudson & Manning, 2019)), to explore the model selection methods multi-modal reason- ing scenarios. MS-GQA primarily evaluates model selection choices of scenarios using VisProg (Gupta & Kembhavi, 2023) as the autonomous agent to solve AI tasks on source dataset GQA. The choice of autonomous agents and datasets can be flexibly replaced, such as substituting VisProg with HuggingGPT (Shen et al., 2023) or other related works (Surís al., 2023; Gao et al., 2023a), and replacing GQA with NLVR2 (Suhr et al., 2018) or even other in-the-wild application scenarios (Surís et al., 2023; Yang et al., 2023c).
44%
Choose
Verify
11%
Compare
17%
Figure 4: proportions various structural categories in GQA.
Our benchmark considers the structural variations of tasks in GQA, and thus facilitates comprehensive method evaluation and assesses robustness under diverse test distributions. Currently, we introduce 5 task categories, namely Query, Choose, Compare, Verify, and Logical. The tasks cover 9 functional components (subtask types), of which out components align with specific Python modules, eliminating the need for model selection. The remaining two components involve "LOC" (localization, rext-guided object detection) and "VQA" (visual question answering). offering 10 and 7 candidate models respectively. We have currently collected the binary execution results for 8,426 samples across 70 valid model selection choices. For more details, refer to Appendix A.
4.2 EXPERIMENTAL SETTINGS
Baselines. As discussed in Section 2.2, the multi-modal model currently falls short of model selection methods. To justify the effectiveness of our solution, we extend a range of representative methods to multi-modal reasoning scenarios two groups of baseline references illustrated below. Training-free selects one model for each subtask type based on the prior knowledge or external metrics (e.g., download counts, citations, and publication dates), without considering input information:

---

Published as a conference paper at ICLR 2024
Randomly choose models for each subtask type on the task graph for every sample; VISPROG (Gupta & Kembhavi, 2023): Follow the default choice in the original paper of VisProg and select deterministic candidate model per subtask type:
RANDOM:
EXMETRIC: Incorporate external metrics for model ranking and selection. This baseline can be generalized paradigm that utilizes external metrics for model selection;
GLOBALBEST (Park et al., 2022; Zhao et al., 2021): Select models on the task graph that yield the highest average performance across all training samples, without considering input details. Training-based focuses on leveraging trainable proxy to find the optimal model. We adapt methods from other domains and form our baselines to meet the requirements of the multi-modal reasoning context. Note that tuning and improving these methods are beyond the scope of this paper.
NCF (He et al., 2017): A representative model selection approach using collaborative filtering. It uses a neural network to model the interaction between samples and models, leveraging both features in collaborative filtering manner;
METAGL (Park et al., 2022): A representative model selection approach using meta-learning. It uses input multi-modal features as meta-features, where a multi-relational bipartite graph is utilized to estimate the relationship between the input and the choice of models on the task graph; NCF++ and METAGL++: In comparison to METAGL and NCF, M utilizes additional computa- tion graph descriptions to capture subtask dependencies. To ensure fair comparison, we extend METAGL and NCF to METAGL++ and NCF++, where original multi-modas input features are enhanced by adding extra text features derived from the computation graph descriptions.
Notably, HuggingGPT (Shen et al., 2023) employs the "In-context Task-model Assignment" model selection strategy. However, our Appendix H experiments show its ineffectiveness, so it is not included in the baselines for simplicity.
Evaluation metric. An ideal model selection method for multi-modal reasoning scenarios should allocate the optimal model choice per subtask or subtask type, so as to maximize the execution chance for every input sample. To assess methods on Nest samples from the MS-GQA benchmark, we define the metric Successful Execution Rate (SER) as Nu E/' l..- Here, 1a, indicates the execution result of the i-th sample upon the selected choice of models c;, either (success) or 0 (fail).
SER measures model selector performance: higher SER means better performance and greater overall reliability in multi-modal reasoning. In special cases where all model selection choices either succeed or fail, evaluating the model selector is pointless, so we exclude these cases from our experiments.
Implementation details. To facilitate fair comparison with M we use CCE loss in both NCF and METAGL, including their extensions (NCF++ and METAGL++). The default backbone network of the computation graph learner is GAT (Veličković et al., 2017), which is well-known and capable of handling directed acyclic graphs. When dealing with input features with multi-modal information, we utilize -blip-base-vqa" as the default feature extractor. Additionally, the pure textual encoder baeetbbas-uuncsed" is used for encoding the extra computation graph description information.
The results below are reported over five random seeds. The dataset from MS-GQA is split randomly into training, validation, and test sets, with 6 : 2 ratio. Ablation studies on the choices of feature extractors and backbone in computation graph learner are deferred to Appendix D.I, D.2 and D.3.
4.3 RESULTS: MODEL SELECTION ACROSS DIVERSE TEST DISTRIBUTIONS
In this section, we examine the performance of M and other strong baselines across varied test distributions. The test set is divided into multiple sub-test sets. There are two criteria for division: 1) problem structure; 2) model selection difficulty. noteworthy that the training dataset remains consistent throughout these experiments. The superiority of M² is summarized below.
Training-based methods outperform Training-free methods. In Table 1 and Table 2, the Training-based methods represented by M and METAGL significantly outperform and demonstrate greater robustness than the Training-free methods represented by EXMETRIC and GI.OBAI.BEST. Specifically, on the Full test set in Table 1, sraining-based methods achieve approximately 64%
"ug filters models based on download count on HuggingFace. As our deployed models are not entirely on HuggingFace, we choose the most recently published and targsttparameter model for each subtask. 2The L.M decomposes the original input to generate a textual description of the multi-modal reasoning exccution process, which we refer to as the computation graph description. hhtps:/hugginggfcc.ccm hhans:/hhugigghhccccssssstrrrrrr cosSaeeeoorreeebiinn-----sss http:://ugggnngacc.co//////
7

---

Published as a conference paper at ICLR 2024
Table I: Performance comparison in test scenarios with diverse structural information. The testing set split into five sub-test sets: Query. Choose, Compare, Verify, and Logical. Each subset maintains consistent structural information: for instance, all Compare samples involve tasks related to comparisons. Full refers to the uncategorized test set, which is the complete test datasct. "Improv." indicates the specific numerical improvement of M" compared the corresponding method in this column.
Category
Metries
Traning-free
Training-buee
RANDOM
VISPROG
EXMETRIC
GLORAL BEST
NCF
NCF+
METAGII
METAGL++
MF
Query
Improw. (%)
45.36
+14.17
44.85
+14.68
68.64200
+0.80
51.07.00
53.94ge
45.50
53.63
45.90
57.63_x*
+1.90
65. 08.18
69 6
Sabeet
Choose
SER (%)
Improv, (3)
+8.27
08.26
06.94196
+8.27
08.00.00
+7.93
00.12_00
+7.11
+6.08
70.25g17
73.39
+3.14
+2.81
73.72
+1.82
7477451
76.53
Compure
SER (%)
82, 26
61.29
77.12=00
73.92a59
71.52z21
75.10-44
73.55
76. 45t4
Improx. (%)
+2.53
Verify
SER (%)
Improv. (%8
62.38
+12.71
68
+6.69
65.80200
+9.29
71.76.00
+3.34
75.46
00.37
71.67
+3.42
70.56....
+4.53
73.01
+2.08
g00
Logioal
SER (9)
Improv. (%)
64.72g1.
+12.81
72. 72.4
+5.06
50.55gnn
+17.98
T8.65
+1.03
76.50
77.08
+0.45
74.94
+2.59
75 76.73
+1.80
77.53229
Full
06.61 aus
05.52
64 64.40g17
20.4
65 625
20
Improw. (%)
+12.19
19.66
+2.60
to 69% SER, while Training- free methods only reach 57% to 64%. Furthermore, Table and Table 2 reveal that in sub-test sets, unlike Training-free methods that occasionally excel in one subset while performing poorly in others, Training-base based methods consistently exhibit overall stability.
M demonstrates its robustness in diverse test sets. In direct com- parisons, Table 1 presents that M' stands out as the top performer, showcasing remarkable 2.69% im- provement over the previous state- of-the-art (METAGL) in the com- plete test set (Full). Moreover, M consistently excels in various sub- test sets across both Table 1 and Ta- ble 2, demonstrating its robustness and competitiveness. Even in sub-test sets where does not claim the top spot, M maintains a strong presence, often ranking among the top two or three methods. However, it can be observed that other methods, particularly training-free ones, often perform poorly on specific sub-test sets.
Table 2: Performance comparison with respect the difficulty of model selection. Each increase in the "difficulty level" indicates a one-unit rise in average model selection difficulty for the sub-test set, leading to 20% reduction in the executable ratio (E [, fl/1c.)
Best performances are noted in blue, while the poorest are orange.
Difficulty Level
<d]]]
all
all
EXMETRIC GLOBALBEST
98,35_1.9
86.67=29
59.90
25.53
15.62
92.86
75.46.0.0
65.6220.
54.47
14.06400
NCF
93. 12.19
82.67=29
65 65.94.as
48.08424
15.31+28
METAGL
82.42ga.
77.58g2a
70.10g0.
53,36
20.21,27
M²
93.30
84.97=14
70.50
52.26
20.42+1.4
M' effectively leverages the subtask dependency information. In Table 1, we assess NCF++ and METAGL+++, both of which employ a text encoder to extract textual features describing multi-modal reasoning logic on the task graph. Our experiments reveal that this approach often fails to capture meaningful information and can even harm the performance of the original methods. Conversely, our framework M which integrates subtask dependencies into the modeling process as a whole, is more effective and non-trivial.
4.4 RESULTS: MODEL SELECTION IN DATA MISSING SCENARIOS
Given the challenges of collecting complete execution results for each sample under all choices of models in real-world scenarios, this section will discuss how training- -based methods perform on the fixed complete test set (Full) in two different types of training data missing scenarios:
missing choices of models on the task graph (c), which involves varying levels of incomplete execution results associated with different model selection choices per sample. F.g., with a missing ratio of 0.2, ~ 20% of model selection choices do not have corresponding execution results.
missing samples (x), where all collected samples have execution results for all model choices on the task graph, but some samples are absent compared to the complete training set. In this case, a missing ratio of 0.2 means that 20% of samples are entirely absent.
Data missing results in ad efffects Figure 5(a) and 5(b) reveal that the missing of data gen- crally leads to performance decline across all training-based methods. Specifically, when the missing ratio reaches 0.8, the SER of M' drops to 66.66 66% and 64.67%, respectively, while other baselines exhibit a typical 2% to 3% performance decline. Anomalies in some baselines, where performance improves with increased missing ratios, may be attributed to two factors: 1) certain methods are insen- sitive to training data quantity, and 2) the smaller dataset introduces more experimental randomness.
8

---

Published as a conference paper at ICLR 2024
Sucorsatu Execution Fare (SER)
70%
70%
70%
600%
Successtul Exacution Rana
60%
Execution Rove
68%
NCP
60%
60%
660%
65%
64%
62%
NCF+
6254
80%
60%
0.4 0.6
0.4 0.6
0.9
1.1
(a) Missing ratio of model choices c
(b) Misssing ratio of samples x
(c) Time limit (s)
Figure Performance comparison in scenarios with missing training data and varying time constraints at test-time. (a) and (b) depict two data- missing scenarios with progressively increasing proportions on the x-axis.
(c) illustrates method performance across different time constraints.
Data missing does not impact the superiority of M over baselines. Figure 5(a) and 5(b) demonstrate that M" exhibits superior performance compared to other training-bused baselines in two types of missing scenarios. Specifically, despite the overall decline performance due to missing data in most methods, Md consistently outperforms other baselines, underscoring its robustness. Notably, Figure 5(b), even when up to 80% of samples are missing (64.67%), M performs better than the best training-free method (GL.OBA. BEST, 63.58%).
4.5 RESULTS: TEST-TIME EFFICIENCY
Seconds (s)
Avg. Execution
In practical applications of multi-modal reasoning, beyond enhancing system robustness through model selection, is crucial to consider the associated cost of implementing the selection process. In this section, we discuss the efficiency of M', in terms of 1) its extra model selection time and 2) its comparison with baselines under the same inference time limit budget.
1.6
Avg. Execution Time Model Selection Time
HuggingGPT
1.4
negligible! !
1 0.09
0.8
MetaGL
Best
0.6
0.4
Negligible runtime overhead at test-time. Figure 6 illustrates that Md outperforms all other baselines, particularly training-free methods, despite incurring extra time overhead from model selection. However, this additional time overhead (0.09s) is considerably shorter than the overall task execution time (0.62s) and can be considered negligible.
58
60
Figure 6: Comparing M' and other methods from the perspectives of performance and aver- age time cost. Execution time encompasses the overall task completion time when agents collabo- rate using multiple models, while model selection time is the time spent utilizing a proxy for model selection.
M' continues to perform the best under various time constraints. In practical production settings, due to cost constraints, not all models are available during test-time. To mimic this challenge, gradually decrease the time limit and exclude models from the candidate pool on the task graph if their average execution time exceeds the current limit. Figure 5(c) illustrates the robustness of M across a range of scenarios, from the most stringent time limit scenario (0.3s) to scenarios with no time constraints (xc). Despite decrease in overall performance, M consistently excels other baselines, highlighting its suitability for time-constrained settings at test-time.
5 CONCLUSION
We introduce M a novel framework for assisting autonomous agents in model selection for multi-modal multi-step reasoning scenarios. tackles the issue of subtask dependencies, a new challenge arising from multi-step reasoning, which existing methods fail to address. In MS-GQA experiments, our framework substantially improves performance, enhancing multi-modal reasoning robustness. Despite resource constraints limiting our experiments MS-GQA. MD has broader applications, including subtask node model selection in multi-step reasoning and adaptation to various agents and real-world datasets. For an in-depth discussion our work's 's research significance. the advantages and limitations of our method, and an analysis of our experimental results, please refer to Appendix E, F, and G.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank anonymous reviewers for their constructive and helpful reviews. This work supported in part by the National Science and Technology Major Project (No. 2022ZD0115101) the Research Center for Industries of the Future (RCIF) at Westlake University, and the Westlake Education Foundation.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
Annie S Chen, Yoonho Lee, Amrith Setlur, Sergey Levine, and Chelsea Finn. Contidence-based model selection: When to take shortcuts for subpopulation shifts. arXiv preprint urXiv:2306.11120. 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general purpose vision-language models with instruction tuning, 2023.
ceneral-purpose
Murtaza Dalal, Tarun Chiruvolu, Devendra Singh Chaplot, and Ruslan Salakhutdinov. Plan-seq-leams Language model guided rl for solving long horizon robotics tasks. In CoRL 2023 Workshop on Learning Effective Abstructions for Planning (LEAP), 2023.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 110.80000040 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010.I 1929,, 2020.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lyneh, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
Malcolm R Forster. Key concepts in model selection: Performance and generalizability. Journal of mathematical psychology, 44(1):20-221, 2000.
Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. AssistGPT: general multi-modal assistant that can plan, execute, inspect, and learn. arXiv preprint 2rXiv:2306.088644 2023a.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. International Conference on Machine Learning, pp. 10764-10799, PMLR, 2023b.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14953 14962, 2023.
Shibo Hao, Tianyang Liu, Zhen Wang. and Zhiting Hu. ToolkenGPT Augmenting frozen language models with massive tools via tool embeddings. arXiv preprint arXiv:2305. 1554,, 2023.
Surya Narayanan Hari and Matt Thomson. Tryage: Real-time, intelligent routing of user prompts to large language model. arXiv preprint ar0iv:2308. 11001, 2023.
10

---

Published as a conference paper at ICLR 2024
Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web, pp. 173-182, 2017.
Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-We Chang, Yizhou Sun, David A Ross, Cordelia Schmid, and Alireza Fathi. AVIS: Autonomous visual information seeking with large language models. arXiv preprint 2.0.:22300008/2999 2023.
Rongjie Huang, Mingze Li, Dongchao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang. Jinglin Liu, al. AudioGPT: Understanding and generating speech, music, sound, and talking head. arXiv preprint arXiv:2304. 2023.
Drew A Hudson and Christopher D Manning. GQA: new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3700-6709 2019.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convo- lution or region supervision. In International Conference on Machine Learning, pp. 5583-5594. PMLR, 2021.
James Kotary, Vincenzo Di Vito, and Ferdinando Fioretto. Differentiable model selection for ensemble learning. In Proceedings of the Fifteen International Joint Conference on Artificial Intelligence, IJCAI-23, 2023.
Jonathan N Lee, George Tucker, Ofir Nachum, Bo Dai, and Emma Brunskill. Oracle inequalities for model selection in offline reinforcement learning. Advances in Neural Information Processing Systems, 35:281 194-28207, 2022.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision- vision-language understanding and generation. In International Conference on Machine Learning, pp. 17888-12900 PMLR, 2022.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2 Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning use tools for creating multimodal agents. arXiv preprint arXiv:2311.05433 2023a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-train to-trriinin for open-set object detection. arXiv preprint arXiv:2303.05400 2023b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 16692, 2019.
Pan Lu, Baolin Peng, Hao Cheng. Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. arXiv preprint arXiv:2304.09842, 2023.
Fanqing Meng, Wenqi Shao, Zhanglin Peng, Chonghe Jiang, Kaipeng Zhang, Yu Qiao, and Ping Luo. Foundation model is efficient multimodal multitask model selector. arXiv preprint MrXiv:2308.06262 2023.
M Minderer, Gritsenko, Stone, M Neumann, D Weissenborn, A Dosovitskiy, A Mahendran, A Arnab, M Dehghani, Z. Shen, et al. Simple open-vocabulary object detection with vision transformers. arxiv 2022. arXiv preprint arxiv:2205.062330
Yohei Nakajima. Babyagi. https:: ht ima mmaababyagii... 2023.
11

---

Published as a conference paper at ICLR 2024
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- question-answer with human feedback. arXiv preprint arXtv:2112.09332, 2021.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Namyong Park, Ryan A Rossi, Nesreen Ahmed, and Christos Faloutsos. MetaGL. Evaluation-free selection of graph learning models via meta-leariing In The Eleventh International Conference on Learning Representations, 2022.
Nils Reimers and Iryna Gurevych. Sentence-hert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019, URL https: http:://://aaat /atp:///aavvvvv/.a org/ abs 908.
10084.
Reworkd. Age httpgs///ssttuu..cm////ee//////s 2023.
Toran Bruce Richards et al. Auto-GPT. AutoGPI, 2023.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2330..4443 2023.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- GPT: Solving ai tasks with chatgpt and its friends in huggingface, arXiv preprint arXiv:2303. 2023.
Jianlin Su, Mingren Zhu, Ahmed Murtadha, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Zipr: A novel loss for multi-label classification. arXiv preprint arXiv:2208 022555, 2022.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. arXiv preprint arXiv: 1.00491, 2018.
Dídac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual inference via python execution for reasoning. arXiv preprint arXiv:2303.08722 0888 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann, Yasemin Altun, and Yoram Singer. Large margin methods for structured and interdependent output variables. Journal of machine learning research, 6(9), 2005.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10900 2017.
Jianfeng Wang. Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-ter transformer for vision and language. arXiv preprint a20iv:2205.1 14100, 2022.
Jonatas Wehrmann, Ricardo Cerri, and Rodrigo Barros. Hierarchical multi-labe classification networks. In International conference on machine learning, pp. 5075-5084. PMLR, 2018.
12

---

Published as a conference paper at ICLR 2024
Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, et al. On the road with gpt-4y (ision): Early explorations of visual- language model autonomous driving, arXiv preprint arXi::231.00533, 2023.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Vi- sual chargpt: Talking, drawing and editing with visual foundation models. arXiv preprint 07Xiv:2303.0461,, 2023.
Jianfei Yang. Hanjie Qian, Yuecong Xu, and Lihua Xie. Can we evaluate domain adaptation models without target-domain labels? a metric for unsupervised evaluation of domain adaptation. arXiv preprint arXiv:2305. 18712, 2023a.
Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Chencheng Jiang, Haoran Tan, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, al. Octopus: Embodied vision-language programmer from environmental feedback. arXiv preprint arXiv:2310.08580 2023b.
Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. MM-ReAct: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 1331, 2023c.
Yuanxiang Ying, Juanyong Duan, Chunlei Wang. Yujing Wang, Congrui Huang, and Bixiong Xu. Automated model selection for time-series anomaly detection. arXiv preprint arXiv:2009. 04395, 2020.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision- language understanding. Advances in Neural Information Processing Systems, 35:36067-36080, 2022a.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained transformer language models. arXiv preprint arXiv:2205.010 01068, 2022b.
Yuxiang Zhang, Sachin Mehta, and Anat Caspi. Rethinking semantic segmentation evaluation for explainability and model selection. arXiv preprint arXiv:2101.08411, 2021.
Xu Zhao, Yuxi Xie, Kenji Kawaguchi, Jumxian He, and Qizhe Xie. Automatic model selection with large language models for reasoning. arXiv preprint 30X5::2305.14333 2023.
Yue Zhao, Ryan Rossi, and Leman Akoglu. Automatic unsupervised outlier model selection. Advances in Neural Information Processing Systems, 34:4489-4502, 2021.
Yue Zhao, Sean Zhang, and Leman Akoglu. Toward unsupervised outlier model selection. In 2022 IEEE International Conference on Data Mining (ICDM), pp. 773-782. IEEE, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: En- hancing languaeee understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Joshua P Zitovsky, Daniel De Marchi, Rishabh Agarwal, and Michael Rene Kosorok. Revisiting bellman errors for offline model selection. In International Conference on Machine Learning, pp. 43369 PMLR, 2023.
Orr Zohar, Shih-Cheng Huang, Kuan-Chieh Wang, and Serena Yeung. LOVM: Language-only vision model selection, arXiv preprini arXiv:2306.n8880 08893, 2023.
13

---

Published as a conference paper at ICLR 2024
CONTENTS OF APPENDIX
A Details of MS-GQA
14
A.1
Collection Process and Statistics
14
A.2
Files of MS-GQA
15
A.3
Subtask types of MS-GQA
15
Details of Loss Choice
16
C Training Details
17
Ablation Study
18
D.I
Multi-modal Feature Extractor
18
D.2
Computation Graph Description Feature Extractor
D.3
Backbone of Computation Graph Learner
18
D.4
Objective Funciton
18
Research Significance
19
E.1
Model Selection for Multi-modal Reasoning
19
E.2
Reliable Model Selector: M
19
E,3
Promising Future Work
E,4
Real-World Applications
................................
20
E.5
Techniques
20
F Strengths and Limitations of the M2 Framework
20
F.1
Strengths:
20
F2
Limitations:
21
G Further Analysis
21
Supplementary Experiment on the "In-context Task-model Assignment" of Hugging-
GPT
21
A DETAILS OF MS-GQA
A.1 COLLECTION PROCESS AND STATISTICS
MS-GQA is constructed by deploying the VisProg agent (Gupta & Kembhavi, 2023) on the GQA (Hudson & Manning, 2019) dataset, focusing 9 subtask types, particularly "LoC"(Localizattion Text-guided Object Detection) and "VQA" (Visual Question Answering). The other subtask types ("EVAL". "COUNT". "CROP", "CROPLEFT": "CROPRIGHT". "CROPABOVE", "CROPBELOW" involve Python modules, thus excluding the need for model selection. The construction process involves these key steps:
Model Zoo Expansion: VisProg has integrated recently released, popular, open-source models from Visual Question Answering and Text-guided Object Detection domains into its "VQA" and
"LOC"mooules" modules, expanding the available pool of candidate models.
Execution Results Collection: 10,000 samples are randomly selected from the GQA dataset (test- dev set). Each sample first decomposed using L.M,, resulting in computation graph description representing the reasoning logic (see "program" in Figure 8). Following this description, all models ("VQA": 7, "LOC": 10, total: x 10 70) are sequentially executed for each sample, yielding 700,000 execution records. Each execution record represents the result (0: fail, 1: success) of specific sample's exccution for model selection choice.
Data Cleaning: Samples that experienced reasoning execution failures due incorrect computa- tion graph descriptions generated by LLM are filtered out. These cases are not within the scope of model selection research.
Finally, considering resource constraints, manage to collect total of 8,426 valid samples, which constitute the foundation for constructing the comprehensive MS-GQA benchmark presented in this study. Within the VisProg framework, the LLM employed is the GPT-3.5-ture by OpenAL Regarding the selection of candidate models, we opt for the most recent, widely recognized, high-performing models in each subtask category. The specific candidate models detailed in Table 3.
14

---

Published as a conference paper at ICLR 2024
Table 3: List of models that be selected for each subtask in MS- MS-GQA.
Subtask
Candidate Models
Link
Venue
owlvit-large-putch (Minderer
https//hugginggfce http://inngggccccooooooooo/o/ooo wwggoogllooowllii-looooo -att--arg--ptcch---
ECCV 2022
owlvit-buse-patch (Minderer et al.)
htts://huggingfaee htte://uuuginggeeeeeooooooooooo cofgooglc/owlviiteeeee wwwewwwwt--ib-ttceeeleeee
BCCV 2022
owlvit-base setlit-base-puteleee base-putch32 (Minderer et al.)
http://hugginng face curgoogllcowwlvii base--utcc322
larglarge (Zhang al., 2022a)
hmpp:///ithuuh..
http:///hhhuu...o/mmieenesse com//miccooooo//GG..P
NeurIDS 2022
glip tiny_ (Zhang ct al., 2022a)
http://github
NeurIPS 2022
LOC glip. tiny. (Zhang et 2022a)
NeurIPS 2022
glip_ clp_tiny_c (Zhang al., 2022a)
NeurIPS 2022
glip_tiny tiny_ori ori (Zhang et al., 2022a)
NeurIPS 2022
groundingdino, swinh
al., 2023h)
https://gilaa..com/IDEAAA ResearchKGroumdnnnnDO00
Arxiv
groundingdino_.o swint (Liu al., 2023b)
Anxiv 2023
vilt-h32 1312-/inetuned-uup (Kim al., 2021)
https://uggiinghan highhugggggggcccooo
www.gnnnngg...coocddneeeeiiiaa lnwiaadeeeeeniiiiilll finetunne vqa
ICML 2021
stt-base-textvqa (Wang et 2022)
http:///uggingg http:/gginngaa...ooioooooooooooo oo/microsofigiii-as--etttt
TMLR 2022
blip-vqu-base (Li al., 2022)
https//hugginggfce wtw.saacw...ssssssoorrebei-a-ooe
2022
VQA
hlip2 opt-2-ppt22.bb (1. et al., 2023)
http:://ugginggiice htw.://uuggnggiioo...ooo/eee/ree www..///wwsssssooooooiiiioooooi
ICMI. 2023
blip2-flan-t5-x1
2023)
htnp://huggingficce www.heraa boweee/eeowoceereaa///p//taaaaaa
ICML 2023
instrcthlip--iinn vicuna- 7h (Dai 2023)
http::/huuging http://ugggggggcgeeec hww..n/w......esetette/sessmmana vicuna-- 7h
Arxiv 2023
instructblip-flare flan-t5-xl (Dai et al., 2023)
https//hugging.at htt.au/igw.....oooooooccooccocca
flan--5551 -S
Acxiv 2023
A.2 FILES OF MS-GQA
There are three main files for MS-GQA:
gqa_model selection Enrection_instance_ee results.json provides the results of whether sample can be successfully executed under different model combination choices and the cost time. As shown Figure 7, the sample with index chooses "insrrucbliip-vicuaa-bb "groundingdino swint" as the "LOC" model, then this sample can be executed successfully and the cost time 0.887s.
the "VQA" model and
gqa computation graph deserption.jonn offers the image ID, problem, and programing text of a sample, as shown in Figure
testdev_ balancedeeeeeeeeeeeeeeeee questions.json includes the task type of sample. As shown in Figure 9, we can get a sample's s task type from the attribute [types][structura]]
A.3
SUBTASK TYPES OF MS-GQA
Table 4 shows the examples of five subtasks, including Qurey, Choose, Compare, Verify and Logical.
"1":
{
"vqa": "'nstructbllip-vicaaa77;;
"loc": "groundingdino_swine swintt,sww
"time": 0.886806011199951,
"res":
},
{
"vqa": "blip-vga-base",
"loc": "glip_large",
"time": 0.063403660644499992
"res":
Figure 7: Examples in gqa_maaam aqa__mdeel_ae sew.mmeeeessleecioonnnoooee www.tinnceeesuultssoooooooes
15

---

Published as a conference paper at ICLR 2024
"index": 7967,
"imageId": "n259002",
"question": "What's underneath the ball?",
"program":
haw.aaaee..co.....e.o..iiaaneaao Box
hANNSEEeeesvunniingggellllleeeer WWhtt is underneath the ball?')
Figure 8: Examples in http:/atttooogappppddeppppioioop
"201307251";
"semantic":
"operation": "select",
"dependencies"; [},
"argument": "scene"
{
"operation": "verify weather",
"dependencies": [0],
"argument": "overcast"
"entailed": [],
"equivalent": ["201307251"),
"question": "Is it overcast?",
"imageId": "n161313",
"isBalanced": true,
"groups": ("global": null, "local"; "el-waather_overcast"") "answer": "no",
"semanticStr"; "select: scene-overify weather: overcast [0]",
"aanotations": ("answer": O,, "question": O,, "fullAnswer": (}),
"types": {
"detailed": "weatherverify"",
"semantic": "global", "structural"; "verify"
"fullAnswer": "No, it is clear."
Figure 9: Examples in testdev balanced questions,jsom
B DETAILS OF LOSS CHOICE
In Section 3.2 discussion, observe that in multi-modal reasoning, single sample's execution result (success or failure) is known, creating a binary outcome. Notably, the count of successful executions variable. Consequently, transform the model selection process in multi-modal reasoning into
16

---

Published as a conference paper at ICLR 2024
Table 4: Subtask examples in MS-GQA.
Subtask Type
Question
Image
Qurey
How tall is the chair in the bottom of the photo?
Choose
Is the ground blue or brown?
Compare
Are both the phone and the coffee cup the same color?
Verify
Is the surfer that looks wet wearing a wetsuit?
Logical
Does the utensil on top of the table look clean and black?
multi-label classification issue. Here, the count of positive labels (successful executions) varies, and the total categories are denoted as |C). Previous literature often employs Binary Cross-Entropy (BCE) loss for multi-label classification problems due to its ease of optimization. However, BCE's instance-wist wise nature overlooks the distinct impact different model selection choices on the same sample.
Therefore, an optimal optimization objective for multi-modal reasoning should emphasize differenti- ating between model selection choices corresponding to positive and negative labels for specific sample. Any choice associated with a positive label is deemed optimal. Hence, we choose Categorical Cross- Cross-Entropy (CCE) loss since it ensures that "the score for each target class is not lower than the score for each non-target class".
C TRAINING DETAILS
NCF, NCF++, and 2 all implemented by ourselves within unified pipeline. We employ grid search to filter some crucial hyperparameters. Specifically, we explored hidden sizes [16, 32, 64, 128), learning rates [le-2, 5e-3, le-3, 5e-3, le-4], weight decays [0.01, 0.001, 0.0001], and optimizer
17

---

Published as a conference paper at ICLR 2024
options [AdamW, Adam, SGD]. A batch size of 64 is utilized, along with StepLR Scheduler with parameters step size 100 and gamma 0.7.
What's more, we conduct experiments both METAGL and METAGL++ after adapting the code of MetaGL (Park et al., 2022) to suit our specific scenario. And we train both METAGL and METAGL++ using the optimizer of Adam. The learning rate is adjusted within [le-2, 5e-3, le-3, Se-3, le-4), with a majority of the experiments using 1e-3. The weight decay is set to O, and the batch size is set to 128.
D ABLATION STUDY
D.1 MULTI-MODAL FEATURE EXTRACTOR
To verify the effect of the quality of multi-modal features model selection, we perform ablation experiments on the choices of multi-moda feature extractors. We choose BLiP (Li et al., 2022), BERT (Devlin al., 2018) ViT (Dosovitskiy al., 2020), and ViLT (Kim et al., 2021) as the feature extractors separately. As shown in the Table 5, the quality of the features really affects the performance of model selection methods, but M stays superiority no matter which feature extractor we use.
Table 5: Performance comparison of model selection methods with different multi-modal feature extractors.
Feature extractor
NCF
METAGL
M
BLiP
65.92
66.01
68.704u..
ViLT
64.87
61.90
65.91
BERT+VIT
63.14
62.27+1.6
64.9440.5
D.2 COMPUTATION GRAPH DESCRIPTION FEATURE EXTRACTOR
NCF++ and METAGL++ leverage extra text features derived from the computation graph descriptions (Devlin et al., 2018), "sentence- compared to NCF and METAGL. Here "roberta-base" (Liu et al., 2019) from HuggingFace. bert" (Reimers & Gurevych, 2019), and
respectively, extract the computational graph features given in textual form by LLM. Table show the performance of NCF++ and METAGL++ across different textual encoders.
Table 6: Performance comparison with different computation graph feature extractors.
Extractor
NCF++
METAGL++
bert-base-uncanl uncased
65.62 64.40
sentence-bert
63.95422
64.00+1.8 64.28+1..
roberta-base
65.90+1.1
D.3 BACKBONE OF COMPUTATION GRAPH LEARNER
In Table 7, we report SER 2 when using GAT (Veličković et al., 2017), GRU (Chung et al., 2014) and Transformer (Vaswani et al., 2017) as the backbone, respectively. Since GRU and Transformer cannot be directly applied to model directed acyclic graphs, we made corresponding adjustments; however, their performance still lags behind that of GAT.
D.4 OBJECTIVE FUNCITON
As shown in Table 8, we report the performance comparison of NCF, METAGL and M on Binary Cross-Entropy Loss (BCE) and Categorical Cross- Cross-Entropy loss (CCE), respectively.
"httss//hugginngfccccom www.eeeeeocoeeeeeeeeeeeeeeeeee ww./ert-hase-uneeneeeeee basc-uncased
btttss//huggingfac.cco http://hwwggnnacccccco//ccnnnnna /t/////neetuuoooooo///rrreenenne
"htps://huggnngface
http:://huuggnggfaacccc///bbbbba co/roberta-base
18

---

Published as a conference paper at ICLR 2024
Table 7: Performance comparison with different backbone for M'
Backbone
SER
GAT (GNNs)
68.70+0
GRU (RNNs)
68.02.0.7
Transformer
65.32+0.4
Table &: Performance comparison with different loss function.
NCF
METAGL
M'
BCE
64.49+1.4
66.03+1.4
67.65+1.3 68. 08.70
CCE 65.92=2.8
66.01
E RESEARCH SIGNIFICANCE
E.1 MODEL SELECTION FOR MULTI-MODAL REASONING
Model selection techniques have proven successful various fields. Model selection techniques, recognized in tasks like time series prediction and graph learning, aim match samples with suitable models, enabling task completion without sample labels.
Errors could have a chain reaction effect. Choices in models impact overall robustness, crucial in multi-step reasoning with strong task dependencies, errors lead to chain reactions on subsequent executions.
Underperformance current model selection strategies. Section 4's experimental results reveal poor and lacking robustness in existing multi-modal agent model selectors, with methods from other domains yielding unsatisfactory results due to neglect of subtask dependency.
A significant gap from the oracle model selector. In MS-GQA, the oracle model selector attains 100% success, while a random strategy reaches about 56% SER, and existing multi- modal agents usually achieve around 60% effectiveness. Our M framework, addressing subtask dependency, enhances results to about 69%. Nevertheless, noticeable gap from the oracle's 100% remains, underscoring substantial research potential in this area.
E.2 RELIABLE MODEL SELECTOR: M
Addressed the limited robustness of current model selectors. The industry emphasizes agent robustness, however, current multi-mode agent research mature, and their use simplistic model selection damages robustness. M² addresses these shortcomings as effective and efficient plugin in the model selection stage to enhance overall system robustness.
Reliable performance in various scenarios. M demonstrates reliability in various data missing and restriction scenarios on the MS-GQA dataset. Sections 4.4 and 4.5 highlight its superior performance compared to existing baining-based methods, reflecting the potential applicability of ' in real-worke production environments.
Highly lightweight and efficient. As pioneers in this research, we'v avoided complex network structures and training techniques. Instead, we' 'e tackled a key challenge. subtask dependency, using a straightforward design-a directed acyclic computation graph. This ap- proach models relationships among multi-modal inputs, subtask dependency, and candidate models. The design leads low costs for both training and testing, with overall memory usage around 6GB, as noted in Section 4.5. This emphasizes ' practical potential for real-world production scenarios.
E.3 PROMISING FUTURE WORK
Empower LLM with model selection capabilities. Currently, the MD framework operates orthogonally to LLM in multi-modal agents. The former, following task planning by the
19

---

Published as a conference paper at ICLR 2024
latter, performs model selection for each subtask based its output. Given LLM's robust reasoning abilities, granting it model selection capabilities a promising endeavor. This would allow LLM to simultaneously handle task planning and model selection, eliminating the need for training an additional model selector and reducing deployment costs practical production environments.
Enhance supervisory signals using intermediate results. In the current training of the M' framework, only the final execution results of the multi-modal agent utilized as supervisory signals, indicating the success of execution or the correctness of the provided answer. However, at each step of reasoning, intermediate results generated. If these results are judiciously employed as supplementary supervisory signals, we believe it can further improve the effectiveness of M
More economical unsupervised and semi-supervised training methods. Similar to the second point, if intermediate results are reasonably utilized, even a form of data augmentation, supervisory signals can come not only from the final execution results but also from the intermediate stages. This enables more cost-effective training approach.
E.4 REAL-WORLD APPLICATIONS
Utilizing agents to decompose and gradually solve complex multi-modal reasoning tasks is currently one of the mainstream research paradigms for addressing multi-modal challenges. Relevant work, from pioneers like VisProg (Gupta & Kembhavi, 2023) to the highly regarded HuggingGPT (Shen al., 2023), and more recently LLaVA-Plus (Liu al., 2023a), has been focal point for researchers in this field.
Moreover, AssistGPT (Gao et al., 2023a) and Chameleon (Lu et al., 2023) highlight the potential applications in areas such as video understanding, education, and finance. Mean- while, inspired by these endeavors (Yang et al., 2023b; Dalal et al., 2023; Wen et al., 2023), we reasonably expect that multi-modal agents whose per-step execution relies on other tools will eventually extend their applications to other AI domains, including autonomous driving, robotics, and embodied intelligence.
When agents call upon different multi-modal AI models to tackle various subtasks in reasoning, it gives rise to the need for model selection techniques. Specifically, considering
the richness and abundance of existing multi-modal model types;
the extensive candidate models;
the reliability and feasibility demonstrated by model selection in other domains;
the overly simplistic or less effective model selection strategies employed by current multi-modal agents;
Researching model selection in the context of multi-modal reasoning is highly promising and practically valuable in this new scenario.
E.5 TECHNIQUES
Our technical contributions encompass addressing the challenge of subtask dependency in multi- modal agents by decomposing the original multi-modal task into sub-tasks, as defined in the multi- modal reasoning scenario (see Definition 3.2). Sections 2 and 4 reveal the inadequacy of existing model selection strategies in multi-modal agents, particularly in handling subtask dependencies. Baseline methods, outlined in Appendix E, exhibit notable underperformance compared to the oracle model selector. In response, our proposed model selection framework, M skillfully integrates multi-modal inputs, model embeddings, and subtask dependencies on a directed acyclic graph. This approach, detailed from Section 4.3 to Section 4.5, demonstrates the reliability and robustness of M'. As an initial endeavor, the unified modeling approach of 3 holds promise inspire future researchers.
F STRENGTHS AND LIMITATIONS OF THE Md FRAMEWORK
F.1 STRENGTHS:
Effective performance. The experimental results Tables and ?? demonstrate that approach consistently outperforms baselines across various test distributions the
20

---

Published as a conference paper at ICLR 2024
MS-GQA dataset, particularly when compared simplistic strategies like training- methods.
Efficient design. As pioneers in this domain, we opted for a straightforward yet effective approach. Rather than intricately designing the network structure and optimization strategies for M we focused on exploring the critical aspect of subtask dependeney. This simplicity is reflected in the efficiency of the entire framework, as evidenced by results Section 4.5.
Applicability across diverse multi-modal agents. The M framework comprises three main components: multi-modal imputs, subtask dependency, and candidate models. Although our experiments were conducted with the VisProg agent, these three inputs constitute foundational components of existing multi-modal agents. For example, in Hug if a user inputs an image and a corresponding question, this forms the model's multi-modal inputs. Then, HuggingGPT breaks down the user's original input, identifying various sub-AI tasks and their dependencies, abstracted as subtask dependency. Candidate models in this context refer to all off-the-shelf models or model APIs within HuggingGPT.
F.2 LIMITATIONS:
Data dependency. The experiments in Section 4.4 demonstrate that, despite Md performing better than baselines in scenarios with varying degrees of data missing, there is absolute decline in performance. This underscores the crucial role of data in ' effectiveness.
Supervisory signal underutilization. The source of supervisory signals during training is too singular, failing to fully harness the intermediate results of multi-step reasoning. This limitation may contribute to M higher dependency on data.
G FURTHER ANALYSIS
M2 exhibits the best performance on the complete test set of MS-GQA. Table illustrates that M stands out as the top performer overall compared to other baselines. It achieves significant 2.69% improvement over the previous state-of-the-art (METAGL) in the complete test set (Full).
M² continues to perform exceptionally well on the sub-test sets of MS-GQA. Moreover, in both Table 1 and Table 2, MM consistently excels in several sub-test sets, including the Query and Choose sets in Table 1 and the Group 3, and 5 (difficulty level) sets in Table 2. Even in the remaining sub-test sets where it does not perform best, M² maintains strong performance, usually ranking among the top two or three methods. In contrast, some baselines may excel in one sub-test set but perform poorly in others. For example, VisProg achieves an 82.26% SER on the Compare sub-test set in Table but fares the worst among all methods on the Query (45.36%) and Choose (68.26% sub-test sets. In contrast, the M framework consistently performs well across all sub-test sets, which is why M' significantly outperforms training- ffre methods the complete test (Full). This also demonstrates the robustness of M³.
M depends on the size of dataset. As shown in Figure 5(b), despite Md performing better than baselines in scenarios with varying degrees of data missing, there an absolute decline in performance. Nevertheless, constrained by limited financial resources, the limited dataset, MS-GQA. So according to the above experimental results, we have reason believe that expanding the size of the dataset further will improve the performance of M on both sub-test sets and the the whole test set.
only collected
H SUPPLEMENTARY EXPERIMENT ON THE "IN-CONTEXT TASK- MODEL ASSIGNMENT" OF HUGGINGGPT
In the original HuggingGPT (Shen al., 2023) text, there is mention of utilizing the in-context learning ability of LLM for model selection. Following the configuration in the HuggingGPT source code, we represented each model by using its metadata and relevant descriptions. Based on Table our primary conclusions are as follows:
The selected model consistently remained the same despite changes in question descriptions or structures.
21

---

Published as a conference paper at ICLR 2024
The consistency indicates that leveraging the in-context learning capability of LLM currently falls short of achieving genuine and effective dynamic model selection.
Table 9: Distribution of selected model based on in-context learning
Candidate Models
Query
Choose
Compare
Logical
Verify
vill-b32-finetueed-ee
100%
100%
100%
100%
100%
git-base-textvqa
0%
0%
0%
0%
0%
blip-vqa-base
0%
0%
0%
0%
0%
blip2-opt-2.7b
0%
0%
0%
0%
0%
blip2-flan-t5-xl
0%
0%
0%
0%
0%
instructblip-vicuna-7b
0%
0%
0%
0%
0%
instrucoblip-llan-55-x
0%
0%
0%
0%
0%
The corresponding experimental details and observations are outlined below:
Settings. We randomly selected 100 questions from GQA, and constructed prompts following HuggingGPT'ss description. 100 questions cover 5 different question structures, and each question structure typically has distinct reasoning characteristics. The corresponding experimental code is included in the suppiementary materials. We randomly selected 100 questions from GQA, and constructed prompts following HuggingGPT's description. 100 questions cover different question structures, and each question structure typically has distinct reasoning characteristics. The corresponding experimental code is included in the supplementary materials.
Observations. All questions, irrespective of task type and question itself, are assigned to the first model (ittt-322-inetueed-wee
with five 100% values the first model column. Comment. The experiment was conducted only in a one-step scenario, where one model is selected for a single task. Though simple, we believe this case is sufficient to state the fact that "in-context mode- assignment" may not be particularly effective in multi-step setting.
22


Reviews:
[{"review_id": "fK1XHflWlm", "reviewer": "Reviewer_JZBq", "summary": "This paper concentrates on the issue of model selection for multi-modal reasoning tasks. It introduces the Model Selector for the Multi-Modal Reasoning (M3) framework, designed to model the dependencies among subtasks and enhance model selection in multi-modal reasoning. To explore the model selection challenge in multi-modal tasks, the authors have created the MS-GQA dataset. The experiments demonstrate that the M3 framework improves the robustness of reasoning on the MS-GQA dataset.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper adeptly formulates the model selection problem within multi-modal reasoning contexts and constructs the MS-GQA dataset.\n2. The paper is well-founded in its pursuit to address the overlooked subtask dependencies in previous works. The proposed M^3 framework innovatively and effectively models the relationship between samples, selected models, and subtask dependencies.\n3. The experiments conducted on MS-GQA demonstrate the efficiency and efficacy of the M^3 framework.", "weaknesses": "1. The primary concern is that model selection is a small part of multi-modal reasoning. It remains to be seen whether it is important for the entire task and how it can benefit real-world applications. The selection method proposed in this paper involves complex proxy training and may need to be more universally applicable or scalable for different reasoning tasks.\n\n2. Lack of reproducibility: The paper must include crucial details, such as the LLM used. The constructed MS-GQA dataset is not yet open-sourced, and the paper fails to provide even a single example of the dataset. Furthermore, the paper does not demonstrate how the proposed methods can improve various reasoning tasks and whether they can be applied to open-source models like LLaMA.\n\n3. The implementation of the baselines is weak: The original HuggingGPT paper dynamically selected models for tasks through in-context task-model assignment, yet this paper describes it as only using \"external metrics\" and implements it as \"choosing the most recently published one for each subtask\", which is misleading and causes unfair comparisons.\n\n4. The experiments could be more convincing: This paper only reports results on a newly created MS-GQA dataset. Even though it's compared to simple baselines that do not require training, like directly using the latest or best models (as shown in Table 2), the proposed M^3 method does not show consistent improvements and may even significantly degrade performance. It would be more convincing if experiments were conducted on more reasoning tasks, as done in VisProg and HuggingGPT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AorMzzBhW2", "reviewer": "Reviewer_JtbH", "summary": "This paper addresses the need for improved model selection in multi-modal agents to enhance their robustness in multi-step reasoning tasks. The authors introduce the M3 framework to facilitate dynamic model selection, considering user inputs and subtask dependencies, and they present the MS-GQA dataset as a benchmark for evaluating their framework's performance.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Identification of Critical Challenge: The paper recognizes and addresses a significant challenge in multi-modal agents, which is the selection of appropriate models for subtasks, a crucial aspect often overlooked in prior research.\n\nIntroduction of the M3 Framework: The paper presents the M3 framework, which aims to improve model selection by considering user inputs and subtask dependencies. The framework is designed with negligible runtime overhead at test-time, making it practical for real-world applications.\n\nCreation of the MS-GQA Dataset: The authors introduce the MS-GQA dataset, specifically designed for investigating model selection challenges in multi-modal agents. This dataset is a valuable resource for benchmarking and advancing research in this area.\n\nExperimental Findings: The paper provides experimental evidence that the M3 framework enhances dynamic model selection and, as a result, bolsters the overall robustness of multi-modal agents in multi-step reasoning tasks.", "weaknesses": "Limited Baseline Comparison: The paper could benefit from a more comprehensive comparison of the M3 framework with existing methods. While it claims to outperform traditional model selection methods, a detailed comparison with state-of-the-art techniques would provide a more robust evaluation.\n\nInsufficient Experimental Discussion: The discussion of experimental results could be more in-depth. The paper does not thoroughly analyze the scenarios where the M3 framework performs exceptionally well or falls short. A deeper dive into the results would provide valuable insights into the framework's strengths and limitations.\n\nReal-World Application Discussion: While the paper discusses the practicality of the M3 framework, it could delve further into real-world applications or use cases where this framework could be deployed effectively. This would provide a clearer vision of its potential impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H0nWbr0Avu", "reviewer": "Reviewer_1wtR", "summary": "This paper addresses the importance of model selection in multi-modal agents, where Large Language Models (LLMs) play a central role in orchestrating various tools for collaborative multi-step task solving. Unlike traditional methods that use predefined models for subtasks, these multi-modal agents excel by integrating diverse AI models for complex challenges. However, existing multi-modal agents tend to overlook the significance of dynamic model selection, focusing primarily on planning and execution phases, which can make the execution process fragile.\n\nThe paper introduces the M3 framework as a plug-in with a small runtime overhead at test time. This framework aims to enhance model selection and improve the robustness of multi-modal agents in multi-step reasoning scenarios. In the absence of suitable benchmarks, the authors create a new dataset called MS-GQA, designed to investigate the model selection challenge in multi-modal agents. The experiments demonstrate that the M3 framework enables dynamic model selection by considering both user inputs and subtask dependencies, ultimately enhancing the overall reasoning process. The authors plan to make their code and benchmark publicly available.", "questions": "refer to the above content.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper provides a clear analysis of the challenges.   \nBesides the method, the paper also provides a dataset as one of the contributions.  \nThe experimental results show significant improvements.", "weaknesses": "The method uses a heuristic process to perform selection which the capacity is relying on the pre-trained models themselves.  \nHow about the generalization capacity for the zero-shot tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eiHf8nNVt6", "reviewer": "Reviewer_v23h", "summary": "This paper studied the model selection problem in multi-modal reasoning. It first formulated this problem, and then proposed $M^3$ framework as an initial attempt in this field. This paper also created a new dataset, MS-GQA, as a benchmark to compare different methods.", "questions": "1. Presentation should be improved. I find some of the notations are used without introducing first, which hinders a smooth reading. \nEspecially in Section 3.2:\n* Page 4, bottom line, $\\phi \\circ \\psi$ is used but they have not been introduced.\n* Page 5, Section 3.2.1, Para 1, $\\psi := [\\psi_1, \\psi_2]$ is used without explanation.\n\n2. Why choose SER as the metric rather than accuracy? From Figure 1, I thought a wrong model selection will mainly cause the system to give false answer. But the main metric adopted is to measure the successful execution rate. I think there is a difference between successful execution and final accuracy.\n\n3. What is the difficulty level in tab2? How do the authors define the difficulty level?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper formulated the model selection problem in multi-modal reasoning, which is a new direction worth investigating.\n\n2. This paper made an initial yet comprehensive effort to study this problem, including a model-selection framework, MS-GQA datasets, as well as a comparison between possible baselines.", "weaknesses": "1. The significance of the problem is not well illustrated. While the paper has shown the existence of model selection problem, I am not aware of how important this problem is. There can be lots of problems in multi-modal reasoning, but some may not be of much value. Specifically, is there a huge gap between an oracle model selection and a random selection? Is there a naïve solution that can approach the oracle performance? The authors are suggested to add these preliminary experiments to illustrate the significance of the problem.\n\n2. Lack of ablation study on model inputs. The paper claims that other model selection methods do not take subtask dependency into account. However, the ablation study does not show the effect of using subtask dependency as input. More broadly, because the framework uses various inputs, including multi-modal inputs, node embedding, subtask dependency, a more extensive ablation can be done by removing each component successively. This will show the importance of each component.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)