Mordal: Automated Pretrained Model Selection for Vision Language Models
Authors: Shiqi He
Insu Jang
Mosharaf Chowdhury
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Model
Vision Language Model
Mode Selection
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Incorporating multiple modalities into large language models (LLMs) is a powerful way to enhance their understanding of non-textual data, enabling them to perform multimodal tasks.
Vision language models (VLMs) form the fastest growing category of multimodal models because of their many practical use cases, including in healthcare, robotics, and accessibility.
Unfortunately, even though different VLMs in the literature demonstrate impressive visual capabilities in different benchmarks, they are handcrafted by human experts; there is no automated framework to create task-specific multimodal models.

We introduce Mordal, an automated multimodal model search framework that efficiently finds the best VLM for a user-defined task without manual intervention.
Mordal achieves this both by reducing the number of candidates to consider during the search process and by minimizing the time required to evaluate each remaining candidate.
Our evaluation shows that Mordal can find the best VLM for a given problem using $8.9\times$--$11.6\times$ lower GPU hours than grid search.
We have also discovered that Mordal achieves about 69\% higher weighted Kendall’s $\tau$ on average than the state-of-the-art model selection method across diverse tasks.

Published as a conference paper at ICLR 2026
MORDAL: AUTOMATED PRETRAINED MODEL SELEC- TION FOR VISION LANGUAGE MODELS
Shiqi He, Insu Jang & Mosharaf Chowdhury
Department of Computer Science and Engineering
University of Michigan Ann Arbor, MI 48109, USA
(shiqihe, i iuu mosharaf) @umi ch. edu
ABSTRACT
Incorporating multiple modalities into large language models (LLMs) a powerful way to enhance their understanding of non-textual data, enabling them to perform multimodal tasks. Vision language models (VLMs) form the fastest growing cate- gory of multimodal models because of their many practical use cases, including in healthcare, robotics, and accessibility. Unfortunately, even though different VLMs in the literature demonstrate impressive visual capabilities different benchmarks, they are handcrafted by human experts; there is no automated framework to create task-specific multimodal models.
We introduce Mordal | an automated multimodal model search framework that efficiently finds the best VLM for user-defined task without manual intervention. Mordal achieves this both by reducing the number of candidates to consider during the search process and by minimizing the time required to evaluate each remaining candidate. Our evaluation shows that Mordal can find the best VLM for a given problem using 8.9x-11.6x lower GPU hours than grid search. We have also discovered that Mordal achieves about 69% higher weighted Kendall's on average than the state-of- the-art model selection method across diverse tasks.
Introduction
Vision Language Models (VLMs) bridge the gap between visual and language understand- ing, rising the dominant approach solving visual information-bassed tasks. Notably, GPT- 4o Hurst al. (2024), large-scale multimodal language model, demonstrates impressive vision reasoning capabilities by taking images as in- put and generating detailed natural language de- scriptions. Although the technical details behind GPT-4o remain undisclosed, researchers have proposed a number of publicly available VLMs (e.g., LLaVA Liu al. (2023a), Intern VL Chen et al. (2024b) and Qwen- VL et al. (2023)) that aim to match GPT-4o's capabilities. Many of these open-source VLMs share similar ar- chitecture, which a feature projector converts the image embeddings generated by vision encoder and feeds it to a large language model (LLM) along with the text embeddings.
ChariQA
VizWir.
80
DocVQA
GQA
LLaYA-Neat-
MLaVA-Nerr-Mllrrr
LLaVA-OV-TB
QweaZ-VL-78
InternV1 2-58
SeienceQA
A12D
Figure Benchmark performance of five latest open-source VLMs on six multimodal tasks.
To construct and train VLM, the common approach starts from selecting an appropriate pretrained vision encoder and LLM. Thanks to the ever wer-growwng ecosystem like HuggingFace Face (2025). developers are able to choose from countless pretrained models for their own VLMs. Unfortunately, despite the great number of available models, it is difficult to determine which pretrained models (i.e., vision encoders and I.L.Ms) are the most appropriate ones. Given a user-specific downstream
Mordal is available h htttts:://gtthub..bbbbb
ot cl.ab/Morda]

---

Published as a conference paper at ICLR 2026
task, is unclear which pretrained models can form the VLM that will meet the user's needs most effectively. As shown in Figure no single VLM consistently outperforms the others in accuracy across all dimensions.
VLM capabilities vary significantly depending on their pretrained components Liu et al. 2023c et al. (2023); ; Zhann et al. (2024). It is unreliable and unpredictable rely human "intuitions" to select pretrained models for the given downstream task, such as selecting the latest one or the most well-known one. Existing model selection methods, such EMMS Meng et al. (2023) and LogME You et al. )2 also fall short in the VLM context. These methods are primarily designed for classification, regression, or language-only tasks, where zero-shot performance can serve as an indicator of model quality [rown et al. (2020); [in et al. (2024); | Yi et al. (2024). However, VLMs require vision-text alignment, making zero-shot evaluations unreliable. Without proper alignment, the language model cannot correctly interpret the image embeddings, leading to random and meaningless outputs. With significant constraints on available time and computation cost, it also unrealistic to try every pretrained model combination and train corresponding VLM candidates. Training single VLM with vision-text alignment data could take more than 100 GPU hours Liu et al. (2023a); Karamcheti et al. (2024). This motivates the key question of this work: How effectively find the best pretrained models in a VLM given a downstream task?
To address this question, we formulate the pretrained model selection problem for VLMs and model it as a sesource-constrainne task to predict the alignment performance of a VLM; i.e., the performance on the downstream task after training the feature projector with the vision-text alignment data. We empirically show that existing model selection methods fail to find the best pretrained model for downstream tasks and naive approach like grid search is infeasible in practice.
We present Mordal, a novel pretrained model selection framework, which automatically and effi- ciently explores different pretrained model combinations in VLM. Mordal builds on our observation that efficiently solving this problem needs jointly considering two optimization directions: (1) mini- mizing the number of VLM candidates, where each candidate has different pretrained vision encoders and LLMs; and (2) reducing the evaluation time for each candidate. Overall, we make the following contributions in this work:
We define the pretrained model selection problem in the context of VLMs and demonstrate that off-the-shelf VLMs often do not contain the best pretrained components for a given downstream task.
We propose Mordal, an efficient pretrained model search framework, to find the best VLM for a given downstream task. Mordal clusters VLM candidates by their representation similarities while employing early stopping and scaling prediction mechanisms to reduce evaluation time.
Extensive evaluations show that Mordal efficiently finds the best VLMs with 8.9x-11.6x less computation time than grid search. It also achieves about 69% higher weighted Kendall's T on average than the state-of-the -ortte-rtttttt model selection method across diverse tasks.
2 Background and Motivation
We start by outlining the architecture of typical VLMs. Following this, highlight the challenges in pretrained model selection for VLMS and show that existing VLMs often do not pick ideal pretrained models for downstream tasks. Based these observations, we present the limitations of potential solutions like grid search, which motivate Mordal's 's design.
2.1 Vision Language Model
Common VLM architectures, like LLaVA Liu et al. (203a),, include a pretrained visual encoder to encode visual features, pretrained large language model (LLM) comprehend the user instructions and produce responses, and vision-language cross-modas feature projector to align the vision encoder outputs to the language models:
Vision Encoder (VE). The vision encoder is responsible for processing input images and extracting relevant features. Potential encoder options are CLIP Radford et al. (202I), SigLIP Zhai et al. (2023) and Intern Chen et al. (2024b), etc.
2

---

Published as a conference paper at ICLR 2026
Table I: Evaluation results of capability on tasks of Visual QA, Doc QA, and Knowledge for four VLMs with different pretrained models. CLIP- Vicuna has the same pretrained models and mode structure as LLaVA-1.5-7BB et 2023a The best result for each scenario in bold text. There is no silver bullet.
Visual QA
Doc QA
Knowledge
Langunge Model
GQA
VizWiz
ChariQA
DocVQA
ScienceQA
A12D
CLIP-Vicuna
CLIP-VIT-L/14
Vicuna- L.57B
61.5
41.2
18.2
27.6
70.4
54.8
SigLIP. P.Veuia
Sig S0.PP-s4000-ppai 1atch14
Vicuna- 1.5.7B
66.4
44.8
18.4
24.1
68.5
53.0
CIIP-Llama
CLIP-VIT-/I4
1lmma-3-88
55.8
37.9
13.3
17.3
75.7
58.2
Sgg.IP-s4000mmpam
Lamaa-3-88
Feature Projector (FP). The vision-language cross-modal feature projector aims to align the encoded image features the text token embedding space. The feature projector be achieved directly by a Linear Projector or Multi-Layer Perceptron (MLP) [iu et al. (2023b).
Large Language Model (LLM). The language model processes mixed embeddings generated from both user instructions in text as well as image inputs. The commonly used LMs in VLMs are decoder- only LLMs, which include Llama Touvron et al. (023a)) Qwen Yang et al. 2024) and Mistral Jiang et al. (2023).
LLM
Projection
With this structure, a VLM first processes input image Timg R# with an image processor and passes it a vision encoder V. The vision encoder outputs a sequence of raw vision embeddings (or patches) Pimg Ix where Ping V (Iimg) and hess is the hidden state dimension of the vision encoder outputs. The feature projector P will then map Ping to aligned vision embeddings Cimg E R.xhextt where hezz is the hidden dimension of the corresponding LM token. The aligned embeddings eimg will append to text prompt embedding Étent = embeed(prompll and feed into the LLM to generate output text.
Image Encoder
A
Image
Text
Figure 2: Overview of VLM architecture.
Pretrained Model Selection: No Silver Bullet
VLMS are versatile and powerful because most vision tasks can formulated as next-token prediction. To train VLM for specific downstream task, developers usually cherry-pick pretrained vision encoders and language models for alignment. However, different pretrained models have varying capacities, which affect VLM performance. To investigate the impact of different pretrained models, we conduct grid search on 49 VLM candidates (i.e., seven vision encoders and seven language models) and train each candidate with the same alignment data. While complete evaluation is described in Section we present the performance of four representative VLM candidates on six datasets, across three dimensions: Visual QA (GQA Hudson & Manning (2019) and VizWiz Gurari al.) (2018)) ), Doc QA (ChartQA Masry et al. (2022) and Doc Mathew et al. (202I) ) and Knowledge (ScienceQA Lu et al. (2022) and A12D Kembhavi et al. (2066). As shown in Table both pretrained vision encoders and language models have a significant impact on VLM performance. We further show in Appendix A that fine-tuning on specific datasets does not eliminate these differences, since model performance remains bounded by the quality of its pretrained components. Overall, there is no silver bullet pretrained model or model combination that reigns supreme for all tasks.
Given these observations, one may naturally ask: how models for a specific task? this paper, we address the pretrained model selection problem in the context of VLMs. Given alignment dataset and target task, aim to find the combination of pretrained vision encoder and a language model that achieves the best performance on the target task after alignment.
select the best combination of pretrained
2.3 Limitations of Existing Solutions
Solving the pretrained model selection problem by relying empirical experiences intuitions, such as choosing the newest, largest or most well-known model, is unreliable. For example, in Table 1 VLM with Vicuna-1.5-77 can outperform a VLM with LLaMA-3-8B in certain settings.
3

---

Published as a conference paper at ICLR 2026
Existing model selection methods, such as EMMS Meng et al. (2023) LogME You et al.
(2021)
LEEP Nguyen et al.] (200aa), and NLEEP Li et al. (2021) ). primarily focus on tasks involving
classification regression Brown al. (2020). These methods typically assess models based on transferability metrics, which are inadequate for VLMMs requiring alignment| Vu et al. (2020): Yi
et al. (2024); Lin al. (2024). VLMS like LLaVA Liu et al. (2023a), shown in Figure involve combining vision encoders with large language models (LLMs) through feature projector to achieve multimodal interaction. Without alignment, the LLM cannot interpret image embeddings, rendering transferability metrics less effective.
Performing an exhaustive search for pretrained model selection is also computationally prohibitive. Even within a manageable search space, training a single VLM candidate with the LLaVA-1.5 dataset, using a pretrained 7B L.M backbone, can consume over 100 GPU hours. Considering the vast number of pretrained models available (e.g., over 1 150,000 LLMs on HuggingFace as of January 2025), exhaustively evaluating every candidate becomes impractical. The cost is further exacerbated as new encoders and LLMs continue to be introduced, necessitating constant re-evaluation integrate new components.
To address the inefficiencies of exhaustive search in pretrained model selection, we must reduce the search cost along two key dimensions: (1) reducing the number of candidates and (2) minimizing the time required evaluate each candidate. By optimizing these two dimensions, the exhaustive search process can be replaced with a more efficient pipeline that balances time consumption and selection accuracy.
3 Mordal Design
This section details the core components of Mordal's 's design. The exhaustive search is expensive because (1) needs to evaluate every candidate (large search space), and (2) needs to train each candidate with full dataset see its performance (high evaluation cost). Mordal reduces search space by clustering the candidates based on their similarity and by introducing a two-step inter- and intra-cluster evaluation ($3.)). and reduces evaluation cost of each candidate with early stopping and scaling prediction ($3.2).
3.1 Candidate Clustering
With the rapid increase in the number of pretrained models in popular model zoos (e.g., Hugging- Face |Face| (2025). evaluating every candidate combination is expensive. Based on prior observations that similar models tend to have similar performance Hu et al. (2023); Yu et al. (2024); Lai et al.) (2023), Mordal clusters candidates and evaluates them in two steps: inter-cluster and intra-cluster, to reduce the search space.
Measuring similarity. Measuring the similarity of VLM candidates without training projectors between vision encoders and language models is challenging. Parameter similarity Lai et al. (2023), which has been used to measure similarity between model architectures, does not fully consider the data distribution pattern in the target task. Models with high parameter similarity could still show different performance on different tasks. Therefore. in Mordal, we consider representation similarity between VLM candidates, which depends the target task.
Mordal employs centered kernel alignment (CKA) Kornblith et al. (0099) to evaluate the similarity of representations between two VLM model structures. CKA has been proven to be effective tool for understanding and comparing the information encoded across different layers of neural networks. Formally, CKA operates CKA score is defined as:
datasets by analyzing their corresponding activation matrices. The
HSIC(K, L)
CKA(K,
(1)
/ /HSIC(K, K) HSIC L.
where and I. are the kernel matrices of activations of two models. HSIC is the Silbert-Schhide Independence Criterion (HSIC) defined as:
HSIC(K, L) Tr(KHLH)
(2)
where Tr() is the trace of matrix and H is the centering matrix H = I -11 CKA is particularly useful in this context for two reasons. First, CKA compare representations with differing shapes
4

---

Published as a conference paper at ICLR 2026
Candidate Clustering
Efficient Evaluation
Eadly
Staling
LLM
Stepping
Prufction
Pretrained Models
Clasiers
Inter- Cluster Candidates
Inira Inira-Clusier Cluster Candidates
Best Candicate
Figure 3: An overview figure for Mordal. Gray shapes represent pretrained models and VL. candidates, while white blocks are climinated ones. Mordal clusters similar candidates, evaluates one per cluster, eliminates weak clusters, and uses regression predict the best candidate.
1.0
10
Vision Encoder
ScienceQA
VizWiz
Bapp DPN Squrr cur
0087
0,50
0.57
4.9
0.77
0.52
0.51
09
CLIP-VIT-L/14
67.6
41.2
0.8
Bevs DVN sigh
1aa-P--s400mpppache
67.7
44.8
0.56
0.84
0.48
4.7
0.52
0.54
0.72
0.7
DENSB-CLIP- ViT-H/14
62.3
34.4
066
06
IntemmVTT300M
56.9
30.7
65
0.53
CLP
06% Incmi
CUP Sight
DPN Ince
(a) Evaluation Results
(b) ScienceQA
(c) VizWiz
Figure 4: Evaluation results of VLMs with different vision encoders and the same language model (i.e., Vicuna-1.5-7B)) Similarity scores between four vision encoders on ScienceQA and VizWiz.
generated by different pretrained models, a task where traditional metrics such as cosine similarity fail. Second, as vision representations commonly projected through MLP layers, this transformation does not compromise CKA's properties, making it robust and well-suited for such evaluations.
To validate the effectiveness of CKA, train multiple VLM combinations with four vision encoders (CLIP [43], SigL.P [58], DFN-CLIP [12], Intern ViT [7]) and the same LLM backend Vicuna-1.5-7BB different datasets: ScienceQA and VizWiz. As shown in Figure 4 the image representation (i.c., outputs of projection heads) generated by different vision encoders show different levels of similarity each other. For example, with input images from ScienceQA, CLIP, SigLIP, and DFN-CLIP generate similar representations. Meanwhile, DFN, CLIP and Intern VIT generate similar representations in VizWiz. From the results in Figure 4a| the vision encoders with similar representations will have similar performance with the same language model. Although this observation does not allow us to directly predict the target task's 's performance, it helps reduce the number of VLM candidates by eliminating similar pretrained models.
The trained VLMSs are evaluated
Two-step clustering. Computing the CKA score between each pair of candidates can be expensive, since K and L are activations from batched data input. To reduce the amount of pair-wise CKA computation, Mordal introduces a two-step VLM clustering strategy: (1) clustering vision encoders, (2) clustering language models based on fixed vision representation. We detail the clustering process as follows (see full pseudocode in Appendix B):
Vision encoder clustering. Mordal computes the representation similarity between vision encoders using CKA. A distance matrix Distee is then constructed based on the dissimilarity values. The clustering function will take an input threshold lye and output the vision encoder clusters Cyc-
Language model clustering. Language model clusters are constructed based on vision representations of each vision encoder cluster. Using the medoid vision encoder from each cluster, Mordal generates fixed image embedding for the dataset and a warmed-up feature projector transforms the shape of image embeddings to match the LLM input shape. A distance matrix Distum will record the dissimilarity and language models are then clustered based on an input threshold tum-
For each vision encoder cluster and the corresponding language model clusters, Mordal generates the candidate clusters by conducting the Cartesian product the two clusters. The two-step clustering process reduces the computation costs by avoiding computing the similarity between candidates that have dissimilar vision encoders, which we show in Section 4.3 that usually yield distinct performance.
Inter- and intra-cluster evaluation. After grouping the candidates into clusters, Mordal finds the best candidate with two-step evaluation: inter-cluster evaluation and intra-cluster evaluation.
5

---

Published as a conference paper at ICLR 2026
Samgle
C2
G
G
x
C3
CR
C3
Inter-Cluster
#Samples
Inter-Cluster
#Samples
(a) Inter-cluster evaluation: selects top-K clusters.
(h) Inter-cluster evaluation with carly stopping. Two candidates are early stopped and C3 is selected.
When K = 1, C3 is selected.
Log Emr
Sumple
G
C3
Intra-Cluster
I'Samples
Intra-Cluster
Log(*Samples)
(c) Intra-cluster evaluation: identifies the best candi- date. Among the remaining two candidates, Ca.2 selected.
(d) Intra-cluster evaluation with scaling prediction.
Ca,2 is selected based the predicted performance.
Figure 5: Inter- and intra-cluster evaluation candidate clusters. We accelerate selection by applying early stopping during inter-cluster evaluation and scaling prediction during intra-cluster evaluation. Colors denote different VLM candidates, and the star marks the selected best-performing candidate.
The detailed process is shown in Figures 5a and 5b] Given that candidates in the same cluster have similar performance, inter-cluster evaluation first picks medoid from each cluster as the representative candidate and compares performance of each cluster to eliminate poorly performing clusters. The remaining Top-K clusters will be evaluated in the second step, where K is user-defined parameter.
With the inter-cluster evaluation, many fewer candidates remain in consideration. Intra-cluster evaluation goes back to mondidate-graullart evaluation by aggregating candidates from the remaining Top- K clusters. trains all them on the given alignment dataset, and returns the one with the best performance to the user.
3.2 Efficient Evaluation
Section 3.1 reduces the search space; however, evaluating each candidate in both stages (i.c., inter- and intra-) still requires training each candidate with full dataset, which remains expensive. To address this, Mordal integrates two complementary techniques: early stopping for inter-cluster pruning, and scaling prediction for efficient intra-cluster evaluation. These components are co-designed reuse intermediate checkpoints, further reducing redundant computation.
Early stopping. Mordal adopts the Successive Halving Algorithm (SHA) Jamieson & Talwalkar (2016) to aggressively eliminate low-quality candidates during inter-cluster evaluation. shown in Figure 5c SHA is conducted during inter-cluster evaluation, where all representative candidates are evaluated with the maximum data sample ratio R. It consists of multiple rounds, also known as rung. In each round, Mordal allocates a budget b each candidate, evaluates all of them, and keeps the top 1/n candidates. In the next round, Mordal increases the budget to x 1] per candidate. This repeats until representative candidates number remaining candidates is large, Mordal also applies SHA again to intra-cluster evaluation. With SHA as the early stopping mechanism, Mordal eliminates poor-performing candidates earlier. This mechanism concentrates resources on the most promising clusters and produces intermediate checkpoints that can be reused downstream.
converged or Top- K candidates
determined. In cases where the
Scaling prediction. Scaling laws have traditionally been used characterize the relationship between model size, training data, and performance in language models Kaplan et al. (2020); Hoffmann et al. (2022). These laws are typically formulated as a power-law relationship between LLMs cross-entropy Toss L and their compute scale measures, which take the form:
a D8
L D)
+e
Na
where N the number model parameters, and the number of training samples. The fitted parameters (a, B3 a, e) derived from training models across different scales. These relationships

---

Published as a conference paper at ICLR 2026
typically log-lineat under log-log transformation and have been exploited for model selection in LLMs Lin et al. (2024); Ruan et al. (2024). In contrast to prior work, our focus is on alignment performance in VLMs, where the model size N is fixed, but the alignment data size D is varied. We find that a similar log-linear trend exists between the size of alignment data and task-specific error metrics (details are provided in Section 4.3). This observational scaling law enables us estimate candidate VLM's full-data performance from a small number of low-cost training runs.
With the observational scaling law, Mordal em- ploys a scaling prediction algorithm that auto- matically detects the log-linear scaling with sam- pled alignment dataset to conserve resources. As
Algorithm 1: Scaling Prediction
Input
Maximum data sample ratio R. scaling ratio u, minimum required point p. VLM candidate
shown in Algorithm ! for each candidate e in
Output :Prediction result
remaining candidate list C,, the algorithm starts from maximum data sample ratio (e.g.. }).
Initialize loss-size pair list P
Initialize data sample ratio R
It evaluates checkpoint trained on randomly sampled data with ratio R. The corresponding
while True
/+ Evaluate performances/
performance point (Log(r), Log( Err)) will be
Err = Evaluate(Daaaae Deask, C,T)
(Rog))nd(oog(r))oon) (Err)))
recorded in list P. After that, the algorithm re-
if |
> p then
duces data sample ratio and repeats the above process until enough performance points (i.e.,
Fit a linear regression model fe on Break if fo fitting loss > 6
p) are collected and the log-lineat relationship is observed for given candidate Since data sample ratio is decreasing, we may effectively start the evaluation of r //u from existing inter- mediate checkpoints to save computation costs. 15 Return fa(1)
end
13
r= r=r/u r/u
/* Reduce data samples*/
14 end
Scaling prediction complements Mordal's two-stage evaluation and helps reduce search cost. By fitting log-linear model Ja on few sampled alignment points P, Mordal can estimate the full-data performance fc(1) without fully training the candidate This enables early elimination of weaker candidates while prioritizing promising ones. Since the evaluation proceeds from larger to smaller sample ratios, intermediate checkpoints can be reused to save additional GPU time.
Evaluation
We conducted extensive experiments to thoroughly evaluate Mordal's performance. Our implementa- tion builds on Cornstacchaang al. (2025), multimodal training framework that includes distributed training features. These experiments assessed Mordal's effectiveness in pretrained model selection and included an ablation study. The key findings are summarized as follows:
1. Mordal identifies the best combination of vision encoder and LLM for the target task 8.9x- 1.6x faster than the grid search. It also achieves about 69% higher weighted Kendall's average than the state-of-the-art model selection method across diverse tasks.
2. We further validate the effectiveness of observational scaling law. By conducting the ablation studies, we show that each component in Mordal helps reduce the total search time while ensuring that it can still identify the top- lop-performing candidates.
4.1 Experimental Setup
The experiments are conducted on seven mainstream datasets across three domains with seven vision encoders and seven LLMSs. We deployed Mordal on set of VMs cluster with total of 16 NVIDIA A40 GPUs, Each GPU has 48 GB GDDR6 memory.
Dataset. We use LLaVA-L.5-Instuuttal dataset Liu et al. (2023a) for alignment. In practice, users may use their own alignment datasets. For target evaluation, first select six standard benchmarks across Visual QA, Doc QA, and Knowledge tasks. To test broader generality, we further include MMMU, a multi-disciplinary benchmark covering diverse real-world domains. These datasets commonly used to assess VLM performance Zhang et al. (2024).
Model zoo and training settings. We select seven vision encoders and seven language models based on popularity and performance. The selected vision models include both language-supervised
7

---

Published as a conference paper at ICLR 2026
Table 2: Summary of improvements. Search Time (h) and Top-1 Model Quality (%) results for different datasets. For each task, grid search exhaustively evaluates 49 candidates, requiring 5439 GPU hours. Mordal significantly reduces the amount of training needed (i.e., GPU Saving) while successfully finding the Top-1 VLM candidate.
CI.P--Viuuna
Grid Search
Mordal (ours)
Tisk
Dutaset
(LL.VA-1.5)
Top-1 Score
Model Name
Time
Specdup
Top-1 Score
Visual QA
GQA (Hobonn GQA/Husoon & Manning
2019
61.5
66.4
SigLIP-Vicum
483
11.3x
66.4
VirWa ot
2018
41.2
46.9
SigL.I--Misral
469
11.6%
46.9
ChanQA Maary al.
302
18.2
20.2
CL CLIP-QWen
607
8.9
18.6
Doc
DoeV Mathew ot al.
2023
27.6
28.5
SigLIP Qwen
593
9.2x
28.5
Knowledge
ScienceQA al
1002
70.4
78.5
Sigt Jama
472
11.5%
78.5
A121) Kembhavi et al
2016
54.8
65.2
SigLIP-QWeen
496
10.9
65.2
Miltiedisippinee
MMML Yos et al
2024
35,3
36.6
SigLIP-LLaame
503
10.8
36.6
00000-94 Vises
613
DIND Quoe?
Dra Owee?
Vosu
413
SqUP Owe?
41
TeseVVT Gaoe?
kawnNT Queet
61.5
CUP Qnad
(a) Grid Search (GQA)
(b) Mordal (GQA)
(c) Grid Search (AI2D)
(d) Mordal (A12D)
Figure 6: Top-5 candidates from grid search and Mordal on GQA and A12D.
models (e.g., OpenAl CLIP) and self-supervised models (e.g.. DINOv2). For LLMs, all models follow the decoder-only structure, and we pick the most commonly used 7B LLMs from HuggingFace We follow the same structure of VA.55-..5-7 which has two-layer MLP projector, and finetune pretrained L.M with LoRA. The default training setting uses Adam optimizer with minibatch size 4 and initial learning rate le-4. We use the linear schedule to decrease the learning rate linearly from the initial value. The complete list of models and additional training details are provided in Appendix D
Baseline. We use grid search the primary baseline, where every VLM candidate is fully trained and evaluated to find the top-1 model. We also measure the total training time, in GPU hours, required by both grid search and Mordal. We further compare Mordal with four model selection baselines: EMMS Meng et al. 2023), LogME You et al. (2021). LEEP Nguyen et al. (020a). and NLEEP Li al. (202I). For fair comparison, we use the same alignment data and ensure that the total training time allocated to each candidate matches Mordal's 's overall budget. This allows us to assess both accuracy and efficiency under equal resource constraints. Implementation details can be found in
Appendix D
Metric. To evaluate ranking accuracy, we compare candidates ranked by performance in grid search with those ranked by elimination order in Mordal. Ideally, if a candidate ranks higher in grid search, it should also rank higher in Mordal. This can be captured by Kendall's 7 coefficient defined as:
2
Σ ISKJjSM
sgn - Tgn)sn((S S,) -S)
M(M- 1)
where M is the total number of candidates and sgn() is the sign function. A perfect ranking match results in 7 1. To further focus on top-performing candidates, we adopt the weighted Kendall's coefficient Tw. which is previously used in You et al. (202T) Vigna (201)). The details for implementing Tu can be found in SciPy Virtanen al. (2020).
4.2 Performance Results
This sections evaluates Mordal's effectiveness in identifying top-performing VLM candidates under constrained training budgets. We also report the weighted Kendall's 7 between Mordal's candidate rankings and baselines' candidate rankings to assess ranking quality.
Mordal is significantly faster than grid search. Table shhwws that grid search requires 5439 GPU hours to exhaustively train 49 VLM candidates per task. In contrast, Mordal reduces the total search time by 8.9x to 1 11.6x across tasks, with search time varying by cluster formation and candidate
8

---

Published as a conference paper at ICLR 2026
Table Comparison of different model selection methods with the same time consumption. Kendall represents the differences of hop-performing candidates compared with the groundtruth (i.e., grid search) and larger better.
EMMS
LogMEE
Merdal (Ouns)
GQA Hudson & Manning
2019
0.682
-0.622
0.232
0.435
0.814
QA
VizWiz Gurari
2018
0.657
0.236
0.351
0.502
0.882
ChartQ Masry ct al
2022
0.238
-0.144
-0.071
0.298
0.765
Doc QA
DocVQA Mathew
2021
0.172
0.155
0.111
0.265
0.897
Science QA Lu
2022
0.770
0,269
0.344
0.562
0.960
Knowledge
A12D Kembbaai et al
2016
0.557
0.193
0.316
0.614
0.894
Multi discipline
MMMU
0.526
0.101
0.220
0.50
Erox
0.5
Too Eroo
0.40
0.45
Value
0.9
0.5
GPU hours
2000
1.500
Moria
0.7
1000
0.4
0.35
0.5
500
15
13
Logg("Sumples (D))
Loga(/Sumples (D))
GQA
A12D
GQA
AI2D
(a) GQA
(b) A12D
(c) Value
(d) Time
Figure 7: Observational scaling law validation and ablation study results for efficient evaluation (EE), early stopping (ES) and scaling prediction (SP). The results GQA and AI2D.
convergence rates. Despite this reduction, Mordal successfully identifies the top-1 model six out of seven tasks. For example, on VizWiz, Mordal completes the search in 469 GPU hours and selects SigLIP-Miste -Msttral,, which achieves 46.9% accuracy. Notably, for most tasks, the best VLM candidates have different pretrained model combinations and all of them surpass the performance of the default LLaVA-1.5-7B structure, highlighting the importance pretrained model selection.
Mordal outperforms existing model selecing methods in finding top-performing models. Mordal consistently outperforms existing model selection methods (EMMS, LogME, LEEP, and NLEEP) by achieving higher weighted Kendall's scores across all seven datasets, as shown in Table For example, on the ScienceQA dataset, Mordal achieves a value of 0.96, significantly surpassing the best baseline (EMMS) with a of 0.77. This is because Mordal effectively captures VLM combinations, while baseline methods rely on features from
the alignment performance foundation models that do not fully distinguish between different model pairings. However, Mordal can occasionally miss the best combinations when promising candidates are grouped with weaker prematurely excluded by the early stopping mechanism (See Figure (6). Despite this limitation, Mordal consistently identifies top-performing models more accurately and efficiently than traditional methods.
4.3
Ablation Studies
In this section, we first validate observational scaling law in Mordal. Then we conduct a comprehen- sive ablation study to evaluate the candidate clustering and efficient exploration in Mordal and their contributions to overall performance.
Observation scaling law validation. To validate the existence of scaling laws in VLM alignment, train multiple VLM combinations with different vision encoders and the same language model Qwen2-7B on the sampled LLLVA-1.5 -.A.---.-Instructionn dataset. The trained VLMSs are evaluated on two different datasets: GQA and AI2D. As shown Figures 7 and 7b we observe log-linear scaling across different VLMs, which supports the design of scaling prediction. However, the log-lineat scaling will only appear after a certain number of training samples, which is consistent with the conclusion in previous work Lin et al. (2024: Ruan et al. (002)).
Effect of candidate clustering. Candidate clustering plays vital role in Mordal it enables inter- and intra-cluster evaluation. As illustrated in Figure 77d, inter- and intra-cluster evaluation (i.e., Mordal without efficient evaluation) significantly reduces training time while maintaining

---

Published as a conference paper at ICLR 2026
high 7 value. By grouping candidates with similar characteristics into clusters, Mordal evaluates representative candidates from each cluster first and eliminates candidates in poor-performer clusters.
Effect of efficient exploration. Early stopping mechanism prunes candidates during the early stage of training. While it significantly reduces the search time, applying it during the entire evaluation (i.e., Mordal without scaling prediction) will eliminate some promising candidates (e.g.. SigLIP-Q on A12D shown in Figure 7b). It evaluates candidates based on intermediate performance and leads to low T value. Mordal limits the usage of early stopping and introduces scaling prediction instead to predict the performance of promising candidates. As shown in Figure 7c] this leads to significant improvement the r value while further reducing the total training time.
4.4 Sensitivity Analysis
Sensitivity analysis explores how key hy- perparameters affect Mordal's performance and efficiency, focusing on clustering thresholds tve and fum. exploration param- eters Sinter and Sintras and scaling predic- tion Careful tuning of these parameters ensures efficient operation without com- promising Mordal's 's ability to select top- performing models. Generally, Mordal is robust and consistently identifies the best- performing model across most hyperparam- eter settings.
Category
Config Default
Time
Top. I Score
Mordal
483
66.4
0.81
Candidate
c., = 0.5
446
66.4
0.52
Clustering
t., = 0.9
1041
66.4
0.86
Inter Cluster
toph
417
66.4
0.73
Evaluation
topk....
66.4
Intra-Cluster
topkinn
451
66.4
0.81
Evaluation
lopk
522
66.4
0.81
Prediction
501
66.4
0.81
Table Summary of sensitivity analysis for GQA.
Effect of clustering hyperparameters. The clustering threshold loe and fumm significantly affects Mordal's performance and efficiency. As shown in Table smaller threshold tue = 0.5 creates fewer, larger clusters based on the LLM's general characteristics, which will lower the value by missing finer distinctions and discarding strong candidates with other L.MM backend. On the other hand, a larger threshold tee 0.9 results in more, smaller clusters, capturing subtle differences and improving the 7 value but increasing the number of candidates to evaluate during scaling prediction, leading to longer search time. Balancing the threshold is crucial to ensure diverse clusters while keeping the computational cost reasonable.
Effect of inter- and intra-cluster hyperparameters. Based Table inter-cluster exploration generally has greater impact on Mordal's performance than intra-cluster exploration. A smaller topkinter reduces the number of clusters evaluated, speeding up the search but lowering the value by excluding promising clusters aggressively. A larger topkinter explores more clusters, increasing search time but improving the value by retaining diverse clusters. Intra-cluster early stopping affects candidate selection within clusters, with smaller topkintra focusing on fewer candidates and larger topkintra exploring more candidates for scaling prediction. However, influence smaller, as clusters already limit diversity. Properly balancing these lopkienter and lopkintra ensures efficient exploration and strong performance.
Effect of scaling prediction hyperparameters. The scaling prediction parameter p has minimal impact on Mordal's overall performance but affects search time. Increasing p beyond appropriate values adds the computational cost without improving results. In practice, p 3 is sufficient for constructing the linear regression model used in scaling prediction.
5 Conclusion
We presented Mordal, an automated framework for efficient pretrained model selection vision- language models (VLMs). Mordal reduces search cost by pruning candidate combinations and minimizing evaluation time per model. Experiments show that Mordal achieves up to 11.6x speedup over grid search while maintaining top-1 accuracy, and outperforms existing baselines in ranking quality. This highlights the effectiveness of awsree alignment-required model selection for VLM deployment.
10

---

Published as a conference paper at ICLR 2026
Ethics Statement
This work adheres the ICLR Code of Ethics. Our study does not involve human subjects or sensitive personal data. The datasets we use are publicly available and widely adopted in the research community. We do not anticipate any harmful societal impact from methodology. and we believe that the contributions of this work will positively benefit the community.
Reproducibility Statement
We have taken steps to ensure reproducibility of our results. Detailed descriptions of datasets, experimental settings, and evaluation protocols are included in the main text and appendix. Model configurations, hyperparameters, and training details are provided. In addition, we will release the source code, model checkpoints, and full evaluation results in the future. We believe this will enable other researchers to fully reproduce and extend our work.
Acknowledgements
We thank the ICLR reviewers, as well members of SymbioticLab, for their helpful feedback. This work was supported in part by NSF grants CCF-2450085 and CNS-2106188 and by grants from Ford and Cisco.
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone, arXiv preprint arXiv:2404. /4219, 2024.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433. 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint ar3iv:2308. 2e..230...29660 /2966, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056-27087 2024a.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 24185 24198, 2024b.
Wei-Lin Chiang, Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu. Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90% * chatgpt quality, March 2023. URL https
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung and Steven Hoi. Instructblip: Towards general-purpost vision-language models with instruction tuning, 2023.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 2023.
11

---

Published as a conference paper at ICLR 2026
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In North American chapter of the association for computational linguistics: human language technologies, pp. 4171-4186, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.27783 2024.
Hugging Face. Hugging face models, 2025. URL htt.ttps//huog.nffccccco//oooo sort=download Accessed: 2025-01-30.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T Toshev, and Vaishaal Shankar. Data filtering networks. International Conference on Learning Representations, 2023.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 3608-360 1678-3617, 2018.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09663 2021.
Qinghao Hu, Zhisheng Ye, Meng Zhang. Qiaoling Chen, Peng Sun, Yonggang Wen, and Tianwei Zhang. Hydro: Surrogate-basee hyperparameter tuning service datacenters. USENIX Symposium Operating Systems Design and Implementation, pp. 757-777, 2023.
Runhui Huang, Xinpeng Ding, Chunwei Wang, Jianhua Han, Yulong Liu, Hengshuang Zhao, Hang Xu, Lu Hou, Wei Zhang, and Xiaodan Liang. Hires-llava: Restoring fragmentation input in high-resolution large vision-language models. In Proceedings of the Computer Vision and Pattern 2025.
Recognition Conference, pp.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6700-6709, 2019.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt- system card. arXiv preprint arXiv:2410.21276 2024.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021. URL http:https///e /tteps://do.... or0.0...00000022.8.....
7er:ddo55437733
Kevin Jamieson and Ameet Talwalkar. Non-stochastic best arm identification and hyperparameter optimization. In Artificial intelligence and statistics, pp. 240-248. PMLR, 2016.
Insu Jang, Runyu Lu, Nikhil Bansal, Ang Chen, and Mosharaf Chowdhury. Efficient distributed mllm training with cornstarch. arXiv preprini arXiv:2503. 1367, 2025.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint urXiv:2310.06822 2023.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 2001 00000000001311.000000110101 2020.
12

---

Published as a conference paper at ICLR 2026
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Perey Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditionel language models. arXiv preprint arXfv:2402.07866, 2024.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part IV 14, pp- 235-251, Springer, 2016.
Simon Kornblith, Mohammad Norouzi, Honglak Lec, and Geoffrey Hinton. Similarity of neural network representations revisited. In Interaational Conference on Machine Learning, pp- 3519- 3529. PMLR, 2019.
Fan Lai, Yinwei Dai, Harsha V Madhyastha, and Mosharaf Chowdhury. Modelkeeper: Accelerating dnn training via automated training warmup. In USENIX Symposium on Networked Systems Design and Implementation, pp. 769-785, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-ing image pre- training with frozen image encoders and large language models. arXiv preprint 2.t.::2.011.25997, 2023.
Yandong Li, Xuhui Jia, Ruoxin Sang, Yukun Zhu, Bradley Green. Liqiang Wang, and Boqing Gong. Ranking neural checkpoints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2663-2673, 2021.
Haowei Lin, Baizhou Huang, Haotian Ye, Qinyu Chen, Zihao Wang, Sujian Li, Jianzhu Ma, Xiaojun Wan, James Zou, and Yitao Liang. Selecting large language model to fine-tune via rectified scaling 2024.
law. arXiv preprint arXiv:2402.0233
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023b.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2023c.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-We Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://gg thub com/ httphhuuginnggcce/pettttt 2022.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200-2209, 2021.
Fanqing Meng, Wenqi Shao, Chonghe Jiang, Kaipeng Zhang, Yu Qiao, Ping Luo, et al. Foundation model efficient multimodal multitask model selector. Advances Neural Information Processing Systems, (6:33065-33099 2023.
Ari Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation. Advances in neural information processing systems, 31, 2018.
13

---

Published as a conference paper at ICLR 2026
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pp. 7294-7305. PMLR, 2020a.
Thao Nguyen, Maithra Raghu, and Simon Kornblith. Do wide and deep networks learn the same things? uncovering how neural network representations vary with width and depth. arXiv preprint 17X1::2010.15327, 20200.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Maithra Raghu, Justin Gilmer, Jason Yosinski. and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. Advances in neural information processing systems, 30, 2017.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems, 14-11226-22128, 2021.
Yangjun Ruan, Chris J Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938, 2024.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang. Adithya Iyer. Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal 1lms. arXiv preprint arXiv:2406. 16860, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint vxXiv:2302.13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXivv2307.0920 00988888......... 2023b.
Anh T Tran. Cuong V Nguyen, and Tal Hassner. Transferability and hardness of supervised classifi- cation tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1395-1405, 2019.
Sebastiano Vigna. A weighted correlation index for rankings with ties. In Proceedings of the 24th international conference on World Wide Web, pp. 1166-1176, 2015.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, J Carey, ilhan Polat, Yu Feng. Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientifie Computing in Python. Nature Methods, 17:261-272 2020. doi: 10.1038/s4 292-019-666--2
14

---

Published as a conference paper at ICLR 2026
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. Exploring and predicting transferability across nlp tasks. arXiv preprint XXi::2005.007700 2020.
Jiayu Wang, Yifei Ming, Zhenmei Shi, Vibhav Vineet, Xin Wang, Sharon Li, and Neel Joshi. Is picture worth thousand words? delving into spatial reasoning for vision language models. Advances in Neural Information Processing Systems, 37:75392-75421, 2024.
Ross Wightman. Pytorch image models.
pytorch- image -aaee-iingeeeemeeeeeeeee models 2019.
Shenggiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal Ilm. In International Conference on Machine Learning, 2024.
Peng Xu, Wengi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang. Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306. 09265, 2023.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint (rXiv:2408008877 2024.
An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma. Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang. Peng Wang. Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671 2024.
Linli Yao, Lei Li, Shuhuai Ren, Lean Wang, Yuanxin Liu, Xu Sun, and Lu Hou. Deco: Decoupling token compression from semantic abstraction in multimodal large language models. arXiv preprint arXiv:2405. 20085 2024.
Chao Yi, Yuhang He, De-Chuan Zhan, and Han-Jia Ye. Bridge the modality and capability gaps in vision- languae model selection. Advances in Neural Information Processing Systems, 37: 14429-34452, 2024.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. Logme:
Practical assessment of pre-trained models for transfer learning. In International Conference on Machine Learning, pp. 12133- PMLR, 2021.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first International Conference on Machine Learning. 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Xinyue Zeng, Haohui Wang, Junhong Lin, Jun Wu, Tyler Cody, and Dawei Zhou. Lensllm: Unveiling fine-tuning dynamics for 11m selection. arXiv preprint arXiv:2505.03793, 2025.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 11955-119999 19866 2023.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang. Jingkang Yang. Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407. 12772, 2024.
15

---

Published as a conference paper at ICLR 2026
OIT Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Eattsen- Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18891-189011 2025.
16

---

end Published as a conference paper at ICLR 2026
A
Task-Specific Fine-Tuning Tuning Does Not Eliminate Performance Differences
Table 5: Task-specific fine-tuning results across three datasets.
Model
GQA
ChartQA
ScienceQA
LLIP-Vicuna
62.8
24.4
71.3
SigLIP- Vieeuna
67.4
26.3
70.2
CLIP-LLaMA
56.5
22.9
78.1
SigLIP-LLaMA 58.7
23.2
81.5
An important question whether performance differences across VLMs disappear after task-specific fine-tuning. Prior studies in both moggliiyy and multimodal settings indicate that this is not the case: performance differences often persist even after fine-tuning. For example, Lin et al. (2024) and Zeng ct al. (2025) show that not all pretrained LLMs converge to similar performance after fine-tuning, even with identical data and compute budgets. Our work extends this investigation to the multimodal domain.
We conducted 12 sask-specific fine-tuning runs: the four representative VLM combinations from Table the paper were fine-tuned three target datasets that differ in their overlap with the LLaVA- 1.5 alignment mixture-Got sults shown in Table 5 Even after fine-tuning data from each target distribution, still observe performance differences of up to 10-11 percentage points among models. Notably, the top-1 model remains consistent with the selection reported Table of the paper. These findings indicate that Mordal generalizes well with the alignment data mixture and remains effective under distribution shift. Practically, this result highlights that even if model is fine-tuned, its performance bounded by the quality of its pretrained components. Thus, selecting strong pretrained VLMs remains crucial. When sufficient fine-tuning data is available, Mordal can also applied directly on the target data to prune candidates, further reducing model selection time and compute.
(in-distribution), ChartQA, and ScienceQA -uutof--iittibbution) Re-
B Algorithm
Algorithm 2: Candidate Clustering
Input :Tagget task D, Model Zoo Moe and Müm. clustering threshold for and from Output: Candidate clusters Coim
Def CandidateCCussteringg/ Mue, Mitm]
/+ Visien Encoder Clustering*/ for MA, .M E Mee do
dist = CKA(D, MA, Mp)
Dist M [Ma] = Distov [MR][Ma) distA,B
end Cue Clustering( SonsterinngDisttee. Distve, fee) / Candidate Clusterings/ for Ca E C.. do Mmedaia = PPchMednilMMooe(CC... LLM clustering w/ for MA, MB E Mumm do
10
11
12
distA.B = CK A(Mmedoil (D), MA, .M)) Distam Distum[MA|[[a
Distum [ [M]] dist
15
end Cum Clustering Distums tum) Gatmaappend(Ca, Cum)
18
19
Return Coim
Detailed two-step the MinibatchCKA for computation efficiency, which is introduced in Nguyen et al. (2020b) and later used in Raghu et al. (202T). LLM clustering, we use the last hidden state from LLM as the sentence representation for CKA computation since it produces the best clustering performance. We
clustering algorithm for candidate clustering as described in Section 3.1 We adopt
17

---

Published as a conference paper at ICLR 2026
leverage the hierarchical clustering from scipy. clwster. et al. 2220)). is possible to adopt other clustering methods. Overall, this approach significantly reduces the number of candidates explore.
hierarchy chereer..rrteeeeeeeerrrt library in SciPy Virtanen
C Implementation
import mordol
search biaachh_ith_aordde (mcdel_zoo, alignmeat_date Larget_task_datan
model mordal.query. wttplaalleeere__r__oeeeee for_model
data=aligament
LLaVA 1.5 Xixture
task-targetssask_Gttt
pretrcinad_ve_ ve_ ooo_______ooooooooooo pretrained_ hom_i___oooooeeeeoo Im"],
9 vin_kwargs=;
'projector's 'KEP", "Ereeze_ve'z Srue, "freeze_1im's Palse,
11
1,
mordel_kwarge-
"clustering' ('t_ve': 0.7, 't_1lm/: 0.8),
'exploration' ('top_k_ing k_ntarr' 3, 'top kittrs' 31,
rearly_stopping stoppingg ('R'; 0.125, "bls 0,03, reta': 21 "scaling prwiiciiopeiiiiionss P 0.125, fuf: 2, *delta': 0.01)
16
18
Listing 1: Mordal interface.
This section describes Mordal's implementation details and configurations to ensure efficient and scalable pretrained model selection. We implement Mordal's candidate training pipeline top of Cornstarch Jang et al.] (202)), a multimodal training framework that enables efficient distributed execution. We highlight key design choices that optimize resource utilization without compromising performance. As shown in code snippet Listing to submit a job, users need to provide the alignment data and data for the target task. Users are also allowed submit list of available pretrained models. In 2im_kwargee pretrained components. When unfreezing pretrained components, instead of performing expensive full finetuning, we uses Low-Rank Adaptation (LoRA) Hu et al. (2021) implemented by Parameter- Efficient Fine-Tuning (PEFT) Mangrulkar al. (2022) and manage LoRA configuration for each pretrained model. LoRA injects task-specifi adaptations into the pretrained model by learning low-rank updates for certain layers while keeping the core parameters frozen. This significantly reduces computational and memory overhead, making finetuning feasible under eesuurce-oonttrained settings. We incorporate Flash Attention Dao (2023) for scalable attention computation, which is a memory-efficient implementation of scaled dot-product attention that avoids redundant operations and reduces memory overhead. All models are trained with torch bíloat precision, which balances computational efficiency and numerical stability. Mordal also automatically allocates idle GPU resources to candidates that are not converged to speed the exploration process.
users may specify the projector's architecture and whether to free
D Additional Experiments
Vision Encoders
LLMs
CLIP-ViT-L/14 336 Radford et al. (2021)
Vicuna-1.5-777 Chiang et al
(2023
SggiIP-o4400mppaii @ 384 Zhai al.
(2023
Llama-2-7B Touvron et 20233b
DFN-CLIP- ViT-IPPVVT-HH4-- 378 Fang et al. 2023
Llama-3-8B DubeyyBDDbbbuu et al.
(024)
InternVIT-3000/144 4488 Chen el al. (2024b
Mistral-v0.2. Jiang et al.
(2233)
DINOv2-ViT-L1140511 Oquab et al.
20233
Qwen2-7B Yang et al. (2024)
EVA-CLIP-02-VIT-LIIee 336 Sun et al [2023
Phi-3-Small-7B: Abdin et al.
20024
ConvNeXt-L/140566 Ilharco et al. (2021)
Gemma-1. 1-1BBB
Team et al. 2024
Table 6: List of vision encoders and L.LMs in experiments.
18

---

Published as a conference paper at ICLR 2026
Model zoo. We evaluate seven vision encoders and seven language models shown in Table 6 Most models are available on HuggingFace Face (2025) while EVA-CLIP and ConvNeXt are supported by timm library Wightman (211)). For ConvNeXt, we interpolate the output embeddings to 16x16 patches following Cambrian- Tong et al. 2024).
Hyperparameter settings. For pretrained model clustering, the threshold for vision encoder and LLMs are set to tue 0.7 and fum 0.8, respectively. And the warmup round for the feature projector is 10. When performing an efficient evaluation, we set both topkinter and topkintra to 3, which means that the Top-3 clusters will be selected in inter-cluster evaluation and Top-3 candidates will be selected in intra-cluster evaluation with the early stopping mechanism. We use default reduction factor, which is consistent with typical SHA settings. We further set p = 3 and 5 = 5e 5 by default for scaling prediction. We discuss the effect of hyperparameters in Section 4.4
2 as the
Baseline. We compare Mordal with four model selection baselines: EMMS Meng et al. (2023).
LogME You et al. (202T). LEEP Nguyen et al. (2020a), and NLEEP Li et al. (2021). For a fair comparison, we use the same alignment dataset and ensure that the total training time allocated to all candidates matches Mordal's overall GPU hour budget. Specifically, we divide Mordal's s total training time by the number of candidates, and assign this budget to each candidate. Each VLM candidate is partially trained using this fixed alignment time. We then extract features from the trained model and compute label representations using three foundation models: CLIP Radford et al. (2021). BERTlDevinn et al. (2019). and GPT-2|Radion Radford et al. (2019). These features and label representations used to compute the transferability metries required by EMMS, LogME, LEEP, and NLEEP. This comparison setup follows the standard protocol used in EMMS Meng et al. (2023), ensuring consistency and fairness in evaluating model selection quality under limited compute.
D.1 Evaluation Time Breakdown
Dataset
GPU Hours
Inees-Toooe Exdy Seyping
Early Stpplig
Scalnng
GQA
6.8
VizWiz
7.3
ChartQA
12.3
CIP
VQA
9.5
ScienceQA
5.5
A12D
6.0
6QA
VieWiz
ChenQA
DocVIAA
StienosQA
AlDD
MMMU
MMMU
6.3
(a) Total evaluation time breakdown
(b) Cluster computation time
Figure 8: Evaluation time breakdown and cluster computation time.
When evaluating Mordal different tasks, the total evaluation time required is different. To investigate the differences in time consumption, we analyze the breakdown time for each component of Mordal and present the result in Figure 8 The projector warmup time and CKA computation time is included the inter-cluster stopping time. For vison encoder, we uses minibatches of size 256 for CKA computation and iterate the target dataset for times. For language model, we use the minibatches of size 32 and iterate the target dataset for 32 times. Sampling is conducted without replacement each time Nguyen et al. (202b)). The cluster computation time (Table (8b) varies across tasks-fromm 5.5 to 12.3 GPU hours. Generally, tasks with more vision encoder clusters (e.g.. ChariQA) require generating more candidate activations, leading to longer cluster computation time. However, this cost is negligible relative to the total evaluation time (typically <3%), due to our two-step clustering strategy and cached activations. The projection warm-up stage is similarly lightweight, requiring only 10 iterations in current setup and completing within one GPU hour for any task.
As shown in Figure 8a the early stopping stage consists of most of the evaluation time, and the prediction only takes a small part of time. This is because the scaling prediction is only performed for the candidates left after early stopping. The time varies depending and scaling is observed. The time for inter-cluster and intra-cluster early stopping depends on the number of clusters, controlled by t.e. and hut- The cluster is generally less obvious for difficult tasks
when the model is converged
19

---

Published as a conference paper at ICLR 2026
(e.g.. ChartQA and Doc) leading to many clusters with only one candidate. As fewer candidates are eliminated, the total evaluation time increases.
D.2
Impact Vision Encoders vs. LLMs
Model
CLIP
SigLIP
DFN
InterVVIT
Model
CLIP
SigLIP
DFN
InternVIT
Vicuna
54.8
53.0
52.7
53.4
Vicuna
41.2
44.8
34.4
30.7
Mistral
50.9
50.5
51.8
52.1
Mistral
45.1
46.9
35.6
31.3
Qwen
61.2
65.2
64.1
61.8
Qwen
43.3
44.6
38.9
33.4
LLaMA3
58.2
60.1
58.8
57.0
LLaMA3
37.9
38.1
35.2
32.5
(a) A12D
(b) VizWiz
Figure 9: Performance across combinations of L.MSs and vision encoders on A12D and VizWiz.
To understand the respective contributions of vision encoders and LLMs, we evaluated x combina- tions of LLMS and vision encoders on two tasks: A12D and VizWiz. Results are shown in Tables Da and 9b] On AI2D, the choice of LLM dominates the performance: differences across LLMs (e.g.. Qwen vs. Mistral) are significantly larger than differences across vision encoders within each row. On VizWiz, vision encoders dominate-replacinng Intern with SigLIP improves accuracy by 6-15% across all LLMs. These findings are consistent with prior work (Chen et al.) 2024aa Wang et al. 2024). which shows that some tasks can be solved without strong vision encoder.
Mordal accounts for these differences at a coarse granularity. For vision-centric tasks, the clustering stage effectively eliminates underperforming clusters associated with weak vision encoders. Among the remaining candidates- differiin mainly in their language backbones- applies scaling prediction estimate and rank their performance.
D.3
Small-scale Models
Dataset
Top-1 (Grid)
Top-1 (Mordal)
Time (Mordal)
Kendall's
GQA
62.4 LIgLIP-LLMMA)
62.4
189 (7.7x)
0.707
ChartQA
12.1 SigLIP-Gemma
12.1
217 (6.7x)
0.621
ScienceQA
66.8 (SigLIP-LLMMA)
66.8
166 (8.8x)
0.740
Table 7: Small-scale results. Mordal matches grid search in identifying the top-1 model while reducing training costs.
To evaluate scale robustness, we conducted additional experiments using the same seven vision encoders in the paper and smaller LLMs (i.c., Gemma-2-22. LLaMA-3.2-3B. and Phi-3.5-mini-3B). The results are shown T Mordal consistently selects the same top-1 model as full grid search across all tasks, while reducing total training costs by 6.7x to 8.8x. However, we observe that Kendall's is lower in the small-model setting. This is likely because smaller models exhibit more inconsistent behavior, making scaling prediction less stable (Zohar et al.) 2025). In contrast, larger models tend to demonstrate more predictable scaling behavior, which makes Mordal more effective.
D.4 Training-free Evaluation
To understand the impact vision-text alignment on model selection, we consider training-free variants of all baselines. In this setting, training-free means that no alignment (i.e., no fine-tuning) performed before running the model selection baselines. For Mordal, we set the warmup round in candidate clustering to zero that all methods operate under the same constraint: they only see features from unaligned VLM structures.
Without alignment, the baseline methods (EMMS, LogME, LEEP, and NLEEP) must rely on un- aligned image embeddings. As discussed in Section 2 such embeddings are poorly calibrated and LLM cannot reliably interpret them. Although some tasks (e.g., ScienceQA and AI2D) may still perform well by solely relying on the text, the resulting transferability scores are unstable. This mismatch leads to significantly degraded ranking quality and low Kendall reported in Table g
20

---

Published as a conference paper at ICLR 2026
Table Comparison of different model selection methods. Kendall T represents the differences of top-performing candidates compared with the groundtruth (i.e., grid search) and larger T is better.
Task
Dutaset
EMMS
LogME:
LEEP
NLEE
Moedal wio Warmup
GQA
0.354
-0.088
0.193
0.286
0,776
Visual
VizWiz
0.251
0.184
0.308
0.247
0.802
ChariA
0.201
0.057
0.112
0.143
0.691
Doe QA
DocV
0.139
40.99
0.067
0.212
0.834
SeiencoQA
0.461
0.214
0.253
0.405
0.960
Knowledge
A1213
0.325
0.137
0.169
0.359
0.855
Mulli-disipline
MMMU
0.242
0.148
0.299
0.163
0.757
In contrast, Mordal remains relatively robust in the training-free regime. Even without alignment, Mordal relies only on representation similarity for candidate clustering: we compute CKA-based similarity between candidates and perform inter-/intra-lluster evaluation on this similarity structure rather than zero-shot predictions. As result, the clustering step is much less affected by the lack of alignment, and Mordal still preserves a high weighted Kendall when compared to the fully finetuned grid search ranking. This gap highlights that simply applying training-free transferability metrics to unaligned VLMs is insufficient, whereas Mordal's 's similarity-driven clustering makes more robust to the lack of alignment.
D.5 Q-former Projector
While Mordal's default implementation uses MLP projectors, which are the most prevalent choice in modern VLMs, we investigate the framework's applicability to alternative projector architectures. Specifically, evaluate Mordal with Q-former projectors Li et al. 2023), which use learnable query tokens and cross-attention mechanisms to bridge vision and language representations. Q-former differs from MLP projectors in several key aspects: (1) it uses fixed number of learnable query tokens that interact with vision features through cross-attention, (2) it employs self-attention layers to process query tokens, and (3) it outputs fixed-size representation regardless of input image resolution.
Language Model
EVA-CLIP-G
CLIP-L
0.51
0.9
0.55
Qwen2-7B
54.3
52.7
0.48
0,50
0.44
4.8
0.55
0.42
0.28
08
Llama-3-8B
48.8
48.7
4.7
View Mark
Mistral-v0.2-17
47.8
47.2
Vicuna-1.5-7B
44.9
43.4
0.36
0.44
0.39
0.5
3:41
0.23
0.25
0.5
(a) Evaluation Results
(b) EVA-CLIP-G
(c) CLIP-L
Figure 10: Evaluation results of VLMs with four language models and two vision encoders. Tables show the similarity scores between four language models with EVA- --CIIGI--G and CLIP-L.
We conduct a small-scale experiments on ScienceQA with two vision encoders (EVA-CLIP-G and CLIP-L. and four language models (Qwen2-7B, Llama-3-8B, Mistral-v0.2-7B, and Vicuna-1.5-7B) Note that we use the existing Q-former checkpoints from first stage (i.e., pretraining without involving language model) Dai et al. (2023) and finetune the Q-former projectors with the LLM on a randomly sampled 20% subset of the
dataset. As presented in Figure 10 CKA remains effective with Q-former projectors. Llama-3-8B achieve similar performance Mistral-v0.2-77 and have a high CKA score. However, Mordal primarily focuses on MLP projectors and will discuss the choices of projector architectures Appendix
E Discussion
Limitations. Mordal demonstrates promising results in efficiently selecting pretrained models fo VLM with small vision encoders and 7B LLMs under single-request. However, certain limitations may be addressed to extend its utility and effectiveness. First, while Mordal significantly reduces search time, it does guarantee perfect top-1 prediction. On challenging tasks, poor clustering or
21

---

Published as a conference paper at ICLR 2026
Language Model
Language Model
Feature Projector
Feature Projector
Vision Encoder
Vision Encoder
Image
Text
Image
Text
(a) Frozen pretrained modules
(b) Updatable pretrained modules
Figure 1I: Alternative alignment approaches for VLM instruction tunning.
early stopping may eliminate strong candidates. Second, Mordal's performance can be sensitive to both dataset and model scale; for instance, scaling prediction becomes less accurate with smaller models, which tend to exhibit higher variance and less predictable behavior. Third, Mordal's performance is sensitive to the set of hyperparameters. An automated algorithm that balances the cluster number, exploration time and ranking quality would be an important future extension. Despite these limitations, Mordal achieves balance between efficiency and accuracy. consistently identifies top pop-pffforiing models with speedup of up to 11.6x over grid search, making it a practical solution for real-world VLM selection where exhaustive search is often infeasible.
VLM alignment. One must go through alignment process to ensure that the individual com- ponents of the VLM are well integrated before using it. Developers integrate pretrained LLMs and visual encoders, training the projector from scratch using visual alignment datasets like VQA Antol et al. 2015). During the alignment process, pretrained components may remain frozen or be further finetuned during alignment training Liu al. (2023a). Recently, some proprietary models have employed end-to-end training without using any pretrained models Bai et al. (2023), but it is not common due the excessive training cost. While Mordal addresses pretrained model selection in popular VLM setting, it would be interesting to investigate its effectiveness under other VLM structures with and without pretrained components.
Alternative similarity metrics for candidate clustering. Mordal uses centered kernel alignment (CKA) as the default similarity metric for candidate clustering. While SVCCA Raghu et al. (2017) and PWCCA MMrcoo al. 2018) are popular CCA-based alternatives, replacing CKA with them is practically infeasible for modern VL.M embeddings. Standard CCA algorithms scale at least quadratically in the embedding dimension h (e.g., forming or decomposing h x h covariance matrices), so flattened embeddings with millions of dimensions would require hundreds of gigabytes to terabytes of memory. Moreover, Kornblith et al. (2019) (Theorem 1) show that any similarity index invariant to arbitrary invertible linear transformations (including standard CCA-based indices) becomes degenerate when the representation width exceeds the number of datapoints, making such measures unable to distinguish representations in the preepparamttrrized regime. By contrast, CKA invariant only to orthogonal transformations and isotropic scaling, remains well-defined for large h, and avoids covariance inversion by operating directly on kernel matrices. Therefore, CKA is the most practical and theoretically sound choice for large-scale candidate clustering in Mordal. An interesting direction for future work is to explore layer- weighte variants of CKA that emphasize layers with higher task relevance or variance, potentially refining candidate space without sacrificing scalability.
Non-MLP projectors. We acknowledge the existence of non-MLP projector architectures, such as the Q-former used in 3LIP-2 (Li et al. 2023). However, Q-former is being replaced in recent VLMs like BLIP-3 (Xue al. 2024) due to limitations. First, Q-former's cross-attention mechanism introduces additional computational overhead. Second, the fixed-size output of Q-former may not capture the full richness of vision representations as effectively as MLP projectors Yao et al. (2024); Huang et al. (2025). Currently, MLP-based projectors are the most prevalent choice in the VLM community due to their simplicity, effectiveness. and compatibility with diverse architectures. They have been shown to generalize well beyond dision-language settings- for instance, in multimodal projection for text, audio, and video in NEXT-GPT (Wu et al.) 2024). While our current experiments focus the ML. projector, Mordal is agnostic to the choice of projector architecture, long the
22

---

Published as a conference paper at ICLR 2026
projector outputs a fixed embedding. We will expand our discussion in the future version to clarify this point and outline how Mordal could be extended to handle non-MLP and multimodal projectors.
Extend to smaller and larger models. Mordal's design and evaluation have focused on small size (i.e., 7B) pretrained models, making it efficient and practical for scenarios with limited computational resources. However, extending Mordal to handle smaller (e.g., IB) and larger models (e.g., 70B) introduces new challenges. For small models, as shown in Appendix D.3 the similarity measurements and observational scaling law used to speed up evaluation could become less effective with smaller models due to shifts in their feature spaces, potentially reducing the accuracy candidate selection. For large models, while scaling prediction may becomes more accurate Zohar et al. (2025). the computational overhead associated with larger models increases, requiring more memory and longer processing times for alignment and evaluation. To address these challenges, future iterations of Mordal must incorporate distributed computing frameworks and advanced resource allocation techniques. Adjustments representation similarity metries and scaling law formulations will also be essential to maintain the framework's 's robustness as models vary in size and complexity.
Similar requests among users. Mordal's current implementation is designed to optimize pretrained model selection for a single request at a time, which limits its efficiency in handling multiple similar tasks submitted by users. This approach overlooks opportunities to reduce redundant computations when user requests share overlapping requirements, leading to inefficient use of computational resources. By evaluating shared set of model candidates for grouped tasks, Mordal could eliminate redundant computations and improve throughput. For example, implementing a caching mechanism store and reuse results for previously evaluated models and tasks could further enhance resource efficiency. Addressing these limitations would enable Mordal to support multi-user environments and dynamic workloads more effectively.
F
Related Work
Decoder
Feed Forward
Prediction
LLM
MHA
Projection
Image Encoder
Text Encoder
Image Encoder
Masked MHA
Image Encoder
Image
Text
Image
(shifted right)
A
Image
(a) Clip-based VLM
(b) VisionEnncodereccoder VLM
(c) LLM-based VLM
Figure 12: Vision-language models: (a) CLie-based VLM aligns image and text embeddings via contrastive learning; (b) VisionEnccoeerDecode VLM uses an encoder-decoder structure for classification and OCR tasks; (c) LLM-based VLM combines a vision encoder with a language model for multimodal interactions.
Model selection. Training-free model selection methods such as EMMS Meng et al. (023), LogME You al. (2021). LEEP Nguyen et al. (2020a), and NLEEP Li et (2021) assess the transferability of pretrained features without requiring additional finetuning. However, these methods are primarily designed for architectures like CLIP and VisionEncoderDeccoee models (as illustrated in Figure 12a and Figure [2b]. Other approaches assume that all candidates share the same architecture and differ only in pretraining datasets Tran et al. (2019). LLL--based selection methods [in al.) (2024) are also not directly applicable to VLMs, as vision and language components are pretrained separately and not jointly aligned. Moreover, most existing techniques are tailored for classification or regression tasks, and struggle with the open-ended nature of multimodal generation. Given the growing number of open-source pretrained models and the unique challenges posed by multimodal alignment, there is clear need for pretrained model selection approaches specifically designed for LLM-based VLMs (i.c.. Figure [2c.. Mordal addresses this gap by enabling efficient and alignment-aware selection of pretrained model combinations.
23

---

Published as a conference paper at ICLR 2026
G LLM Usage
We used large language models solely to assist in polishing the writing of this paper. No part of the research ideation, experimental design, or analysis relied on LLMs.
24


Reviews:
[{"review_id": "bt7Xt7kCh2", "reviewer": "Reviewer_Pw84", "summary": "The paper “MORDAL: Automated Pretrained Model Selection for Vision-Language Models” tackles the problem of efficiently identifying the optimal combination of a vision encoder (VE) and a language model (LLM) from an open model library, given a downstream task and a small alignment dataset.\nThe proposed framework, MORDAL, works in three stages. First, it clusters vision encoders using CKA, and within each visual cluster, it clusters language models based on fixed image representations transformed by a lightweight projector. The Cartesian product of these clusters forms candidate groups.\nThen, it performs two-stage evaluation: inter-cluster comparison using cluster medoids to filter top-K candidates, followed by fine-grained intra-cluster ranking.\nDuring evaluation, MORDAL employs Successive Halving for early stopping and fits a log-linear relationship between error and sample ratio using a few “observation points.” This allows it to extrapolate the expected full-data error without full training.\nAcross 49 VE×LLM combinations and 7 datasets, MORDAL achieves 8.9×–11.6× speedup over exhaustive search while matching the top-1 result in 6 out of 7 tasks. Compared to EMMS, LogME, LEEP, and NLEEP, it yields higher ranking consistency in terms of weighted Kendall’s τ.\nOverall, the work frames “pretrained component selection” as an alignment-phase selection problem, combining candidate clustering, staged evaluation, and observation-based extrapolation to significantly reduce cost, demonstrating strong practical value.", "questions": "Please clarify the default setting used throughout the paper (i.e., projector only or projector + LLM-LoRA), and report τ, hit rate, and GPU-hours separately for both modes.\nCould you add results for a strictly training-free baseline, as well as a version under the same extrapolation framework, to isolate the impact of partial training time?\nPlease report mean and variance across multiple random seeds, especially for ChartQA, and quantify the false rejection rate of SHA early stopping.\nRegarding resource accounting, are the CKA computation and warm-up phase included in the total time reported in Table 2? Please supplement GPU-hour absolute values and proportions.\nFinally, could you provide detailed reproduction code and guidelines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "MORDAL enables rapid screening of visual encoder–language model combinations within open model repositories, saving roughly 9–12× the time compared to exhaustive search, while maintaining top hit rates on most tasks. Its two-level clustering greatly reduces the computational cost of CKA, and the SHA and extrapolation strategies allow reusing intermediate checkpoints, achieving overall engineering efficiency.\nExperimental results show that MORDAL’s ranking consistency surpasses that of EMMS, LogME, LEEP, and NLEEP, indicating more stable predictions of downstream adaptability. The paper also provides time decomposition and hyperparameter sensitivity analyses, helping to understand cost distribution. The authors also explicitly acknowledge limitations regarding small models, large models, and non-MLP projectors.", "weaknesses": "The experimental setup and implementation in the paper are not fully aligned; it should be clarified whether LoRA fine-tuning is enabled by default, and performance and resource overhead should be reported separately for both modes.\nThere appear to be typos in the extrapolation and early-stopping logic of Algorithm 1, which may require correction. The terms “maximum sample ratio” and “initial ratio” should also be made consistent.\nThe baseline comparison alters the “training-free” assumption, so a strictly training-free version should be added. The statistical robustness is insufficient — Tables 2 and 3 only report point estimates without confidence intervals, and the false stop rate of early termination is not quantified.\nThe similarity metric is relatively limited, relying solely on CKA without comparing other metrics such as SVCCA, PWCCA, or layer-weighted CKA.\nThe time decomposition does not list CKA computation and the warm-up phase GPU-hours separately, making the resource reporting incomplete. Some wording is potentially misleading: in the abstract, “1.2×–3.3× better performance” should be revised to “higher weighted Kendall’s τ”, and spelling and formula symbols should also be corrected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sudJ4B0l2y", "reviewer": "Reviewer_uzuF", "summary": "This paper introduces Mordal, an automated framework for selecting optimal pretrained vision encoders and language models when constructing Vision Language Models (VLMs). The key insight is that different combinations of pretrained components yield varying performance on different tasks, yet current VLM construction relies on manual selection. Mordal addresses this by: (1) clustering similar model candidates using CKA-based representation similarity, (2) employing a two-stage inter/intra-cluster evaluation process, and (3) utilizing early stopping and scaling prediction to reduce evaluation costs. The authors demonstrate that Mordal can identify top-performing VLM configurations 8.9×-11.6× faster than grid search while achieving 1.2×-3.3× better performance than existing model selection methods.", "questions": "None", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper addresses a real problem faced by practitioners - selecting optimal pretrained components for VLMs is currently ad-hoc and computationally expensive.\n\n- The authors have conducted extensive experiments across 7 datasets, 49 model combinations, with detailed ablation studies demonstrating the contribution of each component.\n\n- The proposed method achieved 8.9×-11.6× speedup over grid search while maintaining great performance.", "weaknesses": "- The evaluation focuses on 7B parameter models with MLP projectors. As acknowledged, extending to smaller (1B) or larger (70B) models presents challenges. The framework's effectiveness on other projector architectures (e.g., Q-former) is unexplored.\n\n- The performance is sensitive to clustering thresholds ($t_{ve}$, $t_{llm}$), which may hurt its generalization to other unexplored LLM and vision encoders.\n\n- The current design optimizes for single tasks independently. It would be better if the authors could address multi-task learning scenarios, which are also common in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G3ifjI3s3u", "reviewer": "Reviewer_pz3Y", "summary": "This paper focus on VLM selection problem and propose Mordal, an automated framework for efficiently selecting optimal pretrained models (vision encoders and LLMs) for VLMs training on downstream tasks. Mordal addresses two optimization dimensions simultaneously:\n1. Reducing Search Space via Candidate Clustering: Mordal use centered kernel alignment (CKA) to clusters vision encoders first and then clusters LLMs based on fixed vision representations. For inter-cluster selection, Mordal evaluates one representative per cluster, eliminates weak clusters and the remaining top K clusters are evaluated in the intra-cluster stage.\n2. Reducing Evaluation Cost per Candidate. Mordal applies: a) Early stopping: Uses Successive Halving Algorithm to eliminate poor performers during inter-cluster evaluation. b) Scaling prediction: Exploits observational scaling laws—trains on small data samples and predicts full-data performance via log-linear regression", "questions": "1. The reviewer is wondering whether it is possible to perform additional experiments on Q-former or other architectures.\n\n2. Could authors provide some analysis on how Mordal would work when the model zoo is scaled to 10 times or more?", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Mordal addresses a practical problem where manual selection for VLM is unreliable and grid search is infeasible.\n\n2. Mordal utilizes the proper metric to cluster VLM combinations.\n\n3. The experiments are extensive and results are strong.\n\n4. Mordal requires no manual intervention", "weaknesses": "1. Limited Scope and Generalizability. Mordal are architecture-specific and only evaluated on MLP projector-based VLMs. It is unclear if it extends to Q-former or other architectures.\n\n2. Only 7 vision encoders and 7 LLMs are tested. It is unclear how it will work when scaling to 10x model zoo.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "10f84vXBll", "reviewer": "Reviewer_J5GP", "summary": "This paper addresses the critical and computationally challenging problem of selecting optimal pretrained vision encoders and large language models (LLMs) for constructing Vision-Language Models (VLMs) for a specific downstream task. The authors identify that existing model selection methods are inadequate for VLMs, as they fail to account for the necessity of vision-language alignment, and that a brute-force grid search is prohibitively expensive. To solve this, they propose Mordal, an automated framework that efficiently searches the model combination space. Experiments across seven datasets (GQA, VizWiz, ChartQA, DocVQA, ScienceQA, AI2D, MMMU) demonstrate that Mordal achieves faster search time and better performance compared to state-of-the-art model selection baselines.", "questions": "1. The experiments are conducted using a simple MLP projector. As noted in Appendix E, other projectors like Q-Former exist. How would the representation similarity (CKA) and the observed scaling laws be affected by more complex, cross-attention based projectors? Have you done any preliminary experiments to validate Mordal's effectiveness in such architectures?\n2. In the case of ChartQA where Mordal did not find the top-1 model, could you provide a more detailed post-mortem analysis? Was the failure primarily due to the clustering step (the best model was in a eliminated cluster) or the evaluation step (it was outranked by others within its cluster during scaling prediction)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper clearly identifies a significant pain point in the VLM development lifecycle: the high cost and unpredictability of selecting pretrained backbones. The problem is timely and relevant given the proliferation of open-source models.\n2.  Mordal is a novel solution that smartly combines several techniques (CKA-based clustering, early stopping, scaling prediction) into a cohesive framework specifically tailored for the VLM selection problem. The two-pronged approach of reducing candidate count and evaluation cost is logical and effective.\n3. The results are impressive. The reported speedups (up to 11.6×) are substantial, and the fact that Mordal finds the top-1 model in 6 out of 7 tasks demonstrates its high effectiveness. The ablation studies and sensitivity analysis provide strong evidence for the contribution of each component.", "weaknesses": "1. The sensitivity analysis (Appendix D.2) shows that Mordal's performance and efficiency are sensitive to clustering thresholds (t_ve, t_lm) and exploration parameters (topk_inter, topk_intra). While the paper provides default values, the need for task-specific tuning could be a practical hurdle for non-expert users. An automated or adaptive method for setting these thresholds would be a valuable future improvement.\n2.  The evaluation is primarily focused on ~7B parameter LLMs. The authors briefly discuss the challenge of scaling to much larger models (e.g., 70B) in Appendix E, noting the increased computational overhead and potential ineffectiveness of similarity metrics. An empirical demonstration of Mordal's performance on a larger scale (even with a smaller subset) would strengthen the claims about its general applicability.\n3. The CKA-based clustering, while reducing overall search time, introduces its own computational cost. Although likely smaller than full training, a quantitative analysis of this overhead (e.g., GPU hours spent on clustering vs. training) would provide a more complete picture of the total cost savings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)