UMP-Net: Uncertainty-Aware Mixture of Prompts Network for Efficient Instruction Tuning
Authors: Fatemeh Daneshfar
Abdulhady abas
Moloud Abdar
Pietro Lio
Venue: Accepted by TMLR ()

Abstract: Instruction tuning has greatly improved how large language models (LLMs) respond to human-like instructions. However, fully fine-tuning these models is still computationally demanding, and many existing parameter-efficient methods fall short, particularly when it comes to uncertainty estimation and working effectively across different modalities. To address this, we introduce UMP-Net (Uncertainty-Aware Mixture of Prompts Network), a new approach designed to enhance the ability of LLaMA to follow instructions. 
 UMP-Net combines a novel mixture of prompts (MoPs) technique with Latent Noise Prompting, KNN-based Heterogeneous Clustering, and Conformal Predictions to select the most reliable prompts dynamically while accounting for uncertainty. In addition, it features a CLIP-based multi-modal architecture to streamline vision-language integration. We evaluated UMP-Net on a range of benchmarks including ScienceQA, COCO Caption, and various zero-shot multi-modal tasks. The results show a strong performance: an average accuracy of 88.41% on ScienceQA and a CIDEr score of 158.3 on COCO Caption, surpassing models such as LLaVA, LLaMA-Adapter, and LLaMA-Excitor. These findings suggest that UMP-Net offers both improved multi-modal capability and computational efficiency. Further ablations demonstrate UMP-Net’s conformal prediction module provides robust uncertainty estimates under noise and domain shifts, outperforming Bayesian alternatives in coverage guarantees with minimal overhead.

Published in Transactions on Machine Learning Research (October/2022
UMP-Net: Uncertainty-Aware Mixture of Prompts Network for Efficient Instruction Tuning
Fatemeh Daneshfar Deportment of Computer Engineering, University of Kurdistan, Sanandaj, JRAN
f. fdaneshfor@wokk.c.hh
Abdulhady Abas Abdullah
abdulhady abos wbw.......dd
Artificial Intelligence and Innovation Centre, University of Kurdistan Hewler, Erbil, Iraq
Moloud Abdar*
m.abdaar18780gmmale
CHIRP, Child Health Research Centre, The University of Queensland, Brisbone, Australia
Pietro Liô
pl219@cam.ac.uk
Department of Computer Science and Technology, University of Cambridge, Cumbridge, UK
Reviewed on OpenReview: https: // openreview. net/ forun? id= EehtugNXAL
Abstract
Instruction tuning has greatly improved how large language models (LLMs) respond to human-like instructions. However, fully fine-tuning these models still computationally demanding, and many existing parameter-effiiiet methods fall short, particularly when it comes to uncertainty estimation and working effectively across different modalities. To address this, we introduce UMP-Net (Uncertainty-Awar Mixture of Prompts Network), a new approach designed to enhance the ability of LLaMA to follow instructions. UMP- Net combines a novel mixture of prompts (MoPs) technique with Latent Noise Prompting, KNN-basod Heterogencous Clustoring, and Conformal Predictions select the most reliable prompts dynamically while accounting for uncertainty. In addition, features CLIP- based multi-modal architecture to streamline vision-language integration. We evaluated UMP-Net range of benchmarks including ScienceQA, COCO Caption, and various zero-shot multi-modal tasks. The results show a strong performance: an average accuracy of 88.41% 1 on ScienceQA and a CIDEr score of 158.3 on COCO Caption, surpassing models such as LLaVA, LLaMA-Adapter, and LLaMA-Excitor. These findings suggest that UMP- Net offers both improved multi-modal capability and computational efficiency. Further ablations demonstrate UMP-Net's conformal prediction module provides robust uncertainty estimates under noise and domain shifts, outperforming Bayesian alternatives in coverage guarantees with minimal overhead. The code of our proposed model is available here: https://gither tttbb.ttthht.../gg
1 Introduction
Instruction tuning is rapidly an important way of enhancing the ability of large language models (LLMs) to respond and obey human instructions a broad task library Ouyang al. (2022); Wei et al. (2022). Early successes with models like FLAN Wei al. (2022) and InstructGPT Ouyang al. (2022) highlighted how fine-tuning pre-trained LLMs using instruction datasets could significantly boost their zero-shot and few-shot performance. With these gains, the vast majority of these methods require full model fine-tuning, process that not only is expensive in terms of resources but also is unfeasible when dealing with very large models like LLaMA Touvron et al. (2023), which have billions of parameters. Besides, there is an added
* "Correspondence Moloud Abdar <maaddrr 987 2gmmii.ml.om

---

Published in Transactions on Machine Learning Research 2october/2025
complexity of multi-modal large language models (MMLMs). Visual imput with textual input can require further extensive pre-training or fine-tuning to train, further increasing the already high computational costs (Liu et al., 2023b; Li et al., 2023a).
Researchers have come up with parameter-efficion fine-tuning (PEFT) methods, including LoRA Hu et al. (2021) and prompt tuning Lester et al. (2021), that fine-tune only a small fraction of model parameters, without changing the underlying language model. Nevertheless, these methods are frequently unable to accomplish zero-shot generalization on variety of tasks, especially in multi-modal contexts, where visual and text data representations are vital to merge. Our work is motivated by the fact that the current PEFT techniques have two significant weaknesses: they can process multi-modal inputs only to a limited degree and do not provide any powerful tools to quantify and control the uncertainly of prediction. The models in practice, e.g. medical diagnostics or autonomous systems, need to take ambiguons or noisy inputs across modalities and give reliable outputs with quantifiable confidence. The current solutions such as Flamingo Alayrac et al. (2022) and LLaVA Liu al. (2023b) are based on the neod to use largo-scale datasots to perform vision-language aligument, which are computationally intensive and cannot work in the resource- limited setting. The second major limitation is that there are no tools to detect and deal with uncertainty in model prediction. This is of great essence especially when handling ambiguous or noisy input because a clear direction is highly required. Such difficulties become even more evident with multi-modal applications in which the lack of adaptability at any point in time to a particular task and modality is combined with the lack of uncertainty-aware components to complicate performance and reliability.
In this paper, we introduce the UMP-Net (Uncertainty-Arwe Aware Mixture of Prompts Network) that can be used to overcome the constraints of current instruction-tuned and multi-modal systems. UMP-Net is an integration of uncertainty-aware prompt tuning with an effective approach to multi-modal adaptation. Its key idea is blend of mix of prompts (MoPs) mechanism, a union of Latent Noise Prompting, KNN- based Heterogeneous Clustering (HeteroCrraphPrompp)) and Cluster- Wise Uncertainty Estimation (CUE) to dynamically support prompts to the LLaMA model. The system uses Conformal Predictions to enhance reliability to enable it to quantify uncertainty both al prompt and cluster levels and use the measures to inform the selection process. At the multi-modal interface, UMP-Net applies Case- embeddings Radford et al. (2021) to incorporate visual data, which allows to perform effective cross-modal reasoning without expensive pre-training. Not only will this enhance the ability of LLaMA to adhere to instructions in language-only and multi-modal context, but also, it maintains computational requirements low, which is why it can be readily used in low-resource environments. To measure the effectiveness of our proposed UMP-Net, we have made a comparative analysis with other existing models, LLaMA-Adapter Zhang et al. (2024) and LJaMA-Excitor Zou et al. (2024), in various tasks with a combination of visual and textual inputs. This comparison can be soen in Figure and shows that UMP-Net is better in tasks that include the recognition of solution concentrations, botanical features, the description of medical specialties and the creation of functional code.
The contributions made by this paper are three. (1) We present UMP-Net, a parametor-cfficient model that integrates palli--odddladdapaaiiin with uncertainty-oblivoouss prompt tuning. It achieves thate-of-the-art text-only and Laniun-languag benchmark results. (2) Our suggestion is a clustering and uncertainty new estimation pipeline based on the use of KNN-based prompt categorization and Conformal Predictions. It helps to increase timely reliability and reduction of redundancy. (3) We test the performance of UMP-Net by extensive benchmark testing Systems like ScienceQA Lu et. al. (2022a), COCO Caption Chen et al. (2015), and a spectrum of zero-shot multi-modal tasks. UMP-Net is steadily doing so across these settings performs better than the top models such as LLaVA Liu et al. (2023b), LLaMA-Adaph Zhang et al. (2024), and MaM---xcttor (Zou et al., 2024).
2 Proposed Method
This section introduces a new framework, UMP-Net, which has boen created to augment the LLaMA model by improving its performance using learnable adaptation prompt, which combines mixture of prompts strategy. We combine Bayesian reasoning, Conformal Prediction and heterogeneous clustering using KNN to
2

---

Published in Transactions on Machine Learning Research (October/200) 2025)
Whechsskutton liion has lightr concentration of grean parscho??
Solution 3. has the Nigher corcentidiio gleen particion since It conta no more green dats sokumetaan
Solution has
tightr
has a higher concentration
concentration glven particin.
grcan particies.
Addden
Uamr Adapter
UMP
mudi
Which cortes pollent
The mall conss are the ones that
Male cones which produce pollen, are typicaly found on
produce and releare pollen.
cores make palker
adultt trees in con fermn species Seedi nas, souds, noedles do not conoddeeeooolit lon
UMP--Net
Uoma Adapter
Llmmebeatio
UMF-Wet
Prompt:
thice
Fantlringy
Throe medical specialties include cardiology neurn ogy. and mertoology.
Dermatilogy olmedicalspriiii..
Addapeer
Promet Jnvedeript code to geneiate between
function pestandomi 1o11 nntum Muth foortMahhhammmmm 101 +
Nath Mone 100
randomiuumeer Nortfooontaaa randomi" 10/
getfondee toato
UMP Net
Llma--hdpper
Liamaa Eeulio
Figure 1: Comparison of UMP-Net, LLaMA-Adapter Zhang et al. (2024) and LLaMA-Excitor Zou et al. (2024) on four mixed visual-text tasks: determining the concentration of solutions, identifying pollen- producing cones, enumerating medical specialties, and writing JavaScript code. Right bar charts indicate the better mean scores in human evaluation through standard deviation error bars at UMP-Net.
develop more robust and uncertainty-sensitive prompting procedure. This combination will enable UMP- Net to respond dynamically to customizing prompts and have a high level of reliability of its predictions.
2.1 Overview of UMP-Net
UMP-Net a modular architecture, which is structured to dynamically create and assign weights to prompts in relation to their corresponding scores of uncertainty. There are three fundamental elements of the model as shown in Figure 2. Latent Noise Promoting with MoPs strategy (1), KNN based Heterogeneous Clustering based prompt selection and aggregation (2), and conformal predictions to estimate uncertainty among prompt candidates (3). Attention Gates and Softmax layers connect these modules and help the system to calculate one effective prompt, which is reliable and casily adapted to the requirements of the LLaMA model.
2.2 Latent Noise Prompting with MoPs
The Latent Noise Prompting module is an essential feature UMP-Net that is supposed to cause controlled variability to the prompt generation process adapt LLaMA. The purpose of this module to allow dynamic prompt generation that can respond to different tasks and inputs and overcome the drawbacks of traditional prompt tuning methods that are problematic with task-specific generalization. This module adds to the capability of the model to explore a larger repertoire of prompt representations to increase robustness and flexibility both in language-only only and multi- multi-modal context. The module starts with sampling latent noise Z as a standard Gaussian noise N(0, T), with denoting the identity matrix, to make the noise be isot with zero mean and unity variance. The number of dimcnsions Z is defined as d, i.c. the dimonsion of the latent space which often equal to the input embedding size of the LLaMA model.
The latent noises are sampled as the Z gd: and fed into Multi-layer Perceptron (MLP) to produce MoP, denoted as Pi:z, where is the number of prompts in the mixture. The different semantic and syntactic propert ies of the possible inputs are represented by each prompt P. E 1R4, (in which the embedding dimension of the prompt is denoted by dp). The MLP, parameterized by weights W(() and biases 5(l) across L layers, transforms the latent noise as follows:
H a aW H(1-1) + +,00, 1= 1,2, .., L,
(1)
3

---

Published in Transactions on Machine Learning Research (October/2022
where H Z, H = Pi:n, and o is a non-linear activation function (ReLU). The output of the resulting Pi.. becomes the input of the next modules, including Conformal Prediction and Heterogeneous Clustering, to further select and weight prompts.
2.3 Heterogeneous Clustering by KNN
In order to narrow the focus of prompt selection and improve the adaptability of the UMP-Net to LLaMA, we present heterogeneous clustering technique which uses KNN. The reason behind this module is that it is essential to deal with a variety of input modalities (textual, visual and cross-modal) and grouping prompts in sets according to their feature representations. This clustering also makes sure that prompts are specific to particular types of tasks, which makes the model more effective and strong terms of managing multi- modal inputs. The MoPs Pi:m are arranged into different groups (textual, visual, and rross-modal) into this module depending on the presence of their features representations, which makes it possible to treat various modalities of inputs effectively and provides the ability to be more robust when handling various tasks.
Our KNN algorithm will be used to find prompts of structural similarity to cluster them into K clusters CL:K, cach of which will have my + 1 prompts. In this notation my is the count of prompts in cluster & and the +1 is due to a representative prompt or centroid prompt. The KNN clustering is implemented in the following way:
Distance(P,,,P)) - -
(2)
where ||- 1l2 is the Euclidean distance. Prompts are classified into groups Ck based on set of k nearest neighbors of Distance(P:; (P., P3) for each prompt P.- This strategy has various advantages, especially where there is a huge amount of prompts to be handled so as to eliminate redundancy, boost performance and increase the uncertainty quantification in downstream modules such as Conformal Predictions. KNN--bsed clustering groups prompts into functional-specffcc coherent clusters, with each cluster modality specífic: Visual Prompts (V-Prompts Cluster, C.)) Contains prompts that are specialized in processing visual information, i.c. object recognition, spatial reasoning, or image understanding. These prompts are grouped together by similarities of visual features, e.g., vision transformer embeddings. Textual Prompts (T-Prompts Cluster, C.)): Groups prompts are aimed at textual reasoning, c.g. sentence embeddings, text completion, or semantic parsing, Clustering is based on a language features similarities, which is obtained through an encoder of a language model. Unified Cross-Modal Prompls (VI-Promptss CV A combination of prompts with lasks requiring modalities in both visual and linguistic modalities, e.g. visual question answering or image captioning. These prompts are grouped according to joint embeddings which are the combination of visual and textual entitios.
Furthermore, the absence of prompts clustering may result in redundancy or conflicting prompts and there- fore, to operate UMP-Net effectively, it is challenging to calibrate and fuse a large set of prompts. The KNN-based clustering algorithm can be used to solve this problem by clustering structurally similar prompts into operational units and eliminating redundancy and making prompt management simpler. The prompts are sorted into separate clusters, e.g., CØ CT , and CVLL, and the overlap between them is reduced to a minimum to make each cluster fulfill its own. The modality-specific clustering used has K = 3, which allows splitting prompts into only three clusters.
2.4 Conformal Predictions for Uncertainty Quantification
To measure uncertainty in the UMP-Net, we use Conformal Predictions, a distribution free statistical model which gives develops good uncertainty measures of model outputs. The reason of having the knowledge of uncertainty by using Conformal Predictions is that it guarantees accurate and timely selection under ambignous noisy input, which is essential in robust performance in multi-moded tasks and high-stakes problems such medical diagnostics. This module reduces the probability of using inappropriate prompts that will result in confident and yet erroncous results and make LLaMA more reliable and flexible when dealing with a wide range of (asks by measuring how uncertain the predictions of each prompt are. It evaluates the reliability of every prompt in the set Pin by calculating nonconformity scores, the measure of the adherence of given prompt to the anticipated output given a particular input. The scores are then used to obtain the confidence level and, therefore, the most trusted prompts are selected adapt LLaMA. Given prompt P., we calculate nonconformity score S(P,, v.y) y) with respect to the input x and the associated
4

---

Published in Transactions on Machine Learning Research Doctober/2025
label output y, with ø being the input to the task (e.g. text, image other multimodal data), and y the predicted or desired output. The nonconformity scores are computed differently for each prompt type (visual, textual, or cross-modal) within their respective clusters, as detailed below.
Visual Prompts (V-Prompts). The nonconformity score of visual prompt P. represents the similarity of the input image with the visual features that the visual cluster P. anticipates. We define the V- V-Prompt nonconformity score as:
S(P,, 2, v) - (
(3)
where f(x)) E dR is the visual feature representation of the input image x, as computed by a pre-trained vision model (such as a convolutional neural network, or vision transformer). gi(y) E R denotes the embedding of the i-th visual prompt P; within the cluster, that an expression of the output y (c.g. a
predicted class or description) into the visual feature space.
The intuition of this formulation is that the combination of the Euclidean distances of this prompt will capture the nonconformity measure of the collective similarity of the input image to the visual features of the prompt that were supposed to be seen. A smaller S(P,, I. y) will be an increased conformity (i.e. the input fits well into the prompt), which mcans an inereased uncertainty.
Textual Prompts (T-Prompts). The nonconformity score in the case of the cluster of textual prompts, which uses the linguistic input to predict, is the negative log- likelihood of the label y given the input The prompt-lovel nonconformity score is:
S(P, x. v) = log (,(()z),
(4)
where (yla)) the likelihood of the label y returning the input x, which is forecasted by the i-th textual prompt P, in the cluster.
This cumulative score is a measure of overall confidence of the textual prompts in the cluster. As conformity (i.e. less uncertainty) is desired, a higher S(P, 2, y) means that the probability that the predicted label y matches the input is reduced. The probabilistic nature of language models is used in this formulation to make sure that uncertainty is measured in predictive confidence.
Unified Cross-Modal Prompts (VL-Prompts). In the case of cross-modal prompts where modalities are dependent on each other (Lexl. and visual), we determine a weighted hybrid nonconformity rating by weighting the two domains. The cluster-level nonconformity score is given by:
(5) where À E [0, 1] is hyperparameter that balances the contributions of the visual (If(())) (v) and textual (- log (y)))) components. Moreover, f(x), gv(y), and P.(y(z) are defined as in the visual and textual cases, respectively.
S(P,, x, = [AIl((>) - (v) - (1 - log P.(yla))]
This expression leads the score of nonconformity to portray the view of the cluster as a collective judgment as far as modalitics are concerned. The parameter À can be adjusted according to the task need whereby there a flexibility of focusing on either the visual or the text information. The smaller S(P,, x, y), the greater the conformity and the less uncertainty, which allows UMP-Net to adapt LLaMA to multimodal inputs successfully. The score of all prompts nonconformity is used to select the most confident prompt in each cluster, Pbest, to use in LLaMA adaptation explained in the Attention Gate and Weighted Prompt Creation module.
2.5 Attention Gate and Weighted Prompt Creation
The most promising prompts in each cluster are forwarded to an Attention Gate which weighs each prompt dynamically depending on its relevance. Attention Gate uses Softmax to normalize attention scores, which results in weighted prompt Pweighted. The weighting process is guided by:
K
Pweighted =
Σ
ag Poest.k
(6)
5

---

Published in Transactions on Machine Learning Research (ettober/202)
where ag is the weight of attention of prompt bestt, learn with the training time. The weighted prompt of outputs is then chosen to be adapted by LLaMA.
The last weighted prompt is incorporated into the LLaMA model, which allows it to be open to inputs. UMP-Net enhances LLaMA on generating consistent and contextually suitable replies to specific matters, especially when there minimal or noisy data (see Figure 2). Algorithms The UMP-Net pipeline may be summarized as follows (Algorithms 1).
Latent Noise Prompting with Mixture of Prompts
Conformal
Heterogeneous
Prediction
Clustering by KNN
N(O.I)
Textual Promils
Visual Prompts
Multimodal Prommsts
MLP
Z
Attention Layer
Attention Layer Softmax
Attention Gate
Softmax
Create weighted prompt
confidentgrompt Forward
M + 1...
Backward
1..M
Frozen
Figure 2: Figure illustrates UMP-Net architecture when adapting to LLaMA. The first step is Latent Noise Prompting, which noise in a normal distribution N(0, I) is fed to an MLP to produce prompt initials. These prompts are Conformally Predicted determine their uncertainty, and then Heterogeneous Clustering is used with KNN to determine them textual, visual, and multi-modal clusters. Fach cluster is then picked by an Attention Gate through Softmax Layer and it involves picking the best confident prompt to generate weighted prompt. LLaMA is finally modified to incorporate the final weighted prompi in order to achieve better cnarructionfoolloinng and frozen layers make it efficient.
2.6 Multi-modal Architecture
Multi-modal architecture of UMP-Net increases the capacity of the system handle different modalities of input by embedding images at different points in the pipeline as demonstrated in Figure 3. This module rationale is that it will facilitate smooth integration of visual and textual information to overcome the issue of matching multi-modal inputs when there are constraints on resources in which pre-training a large scale is impractical. This architecture uses CLIP model Radford et al. (2021) to project multimodal features and integrates image embeddings into the prompt embeddings following a clustering and distributed across all the attention layers, making sure that strongly integrated multimodally. The architecture incorporates the following important components and processes:
CLIP-based Image Embedding. The input image Timg (e.g., the cat image in Figure 3) is processed through CLIP to extract visual embedding:
Cing = CLIPvisual visual (Zimg) E Rdc
where de is the CLIP embedding dimension, aligning with the prompt embedding dimension dy
Embedding Addition to Each Prompt. After Heterogeneous Clustering by KNN, the image embedding
eing is added to each prompt P. E C₁ within clusters C of and C For each prompt P; E Rd, the augmented embodding computed as:
pang
P W WroojCing
(8)

---

Published in Transactions on Machine Learning Research (2ttobrr/2025
Algorithm UMP-Net Algorithm for LLaMA Adaptation
Require: Input a (task imput), dp (latent dimension), n (number of prompts), K (number of clusters), k (KNN neighbors). I. (MLP layers), À (cross-modal weight), pre-trained LLaMA model
Ensure: Weighted prompt weeighted: predicted output Vpreed
1: 1. Latent Noise Prompting: Sample Z ~ N0,,)) I) with dimension d₂
2: Process Z through MLP with L layers (H = a (W HW--) + b()) to generate Pin E R
Heterogencous Clustering by KNN:
Partition into K clusters CLKK (C Cf C
3. Conformal Predictions:
for each P. E Pin do
7: if P: C% then
8: S(Pi, T, y) = 11/(x) - 9:11311
b. Visual
9: else if P. E C% then
10:
S(,,, x. y)
log (z(l)))
Textual
11: else if P. E CV then
12: S(Pt, Jr, y) (1) g (1 >) log P.(())))
Cross-modal
13: end if
14: end for
15: Select best prompt & per cluster with lowest S(Ps, vz v)
16: 4. Attention Gate:
17: Compute P Pweighted Ek=1 Org Eig.bess best, through learning
18: 5. LLaMA Integration:
19: Feed wweighted into LLaMA to get Upred [LaMA(x, Pweighted)
20: return P weighted Mpred
where Proof xd is learnable projection matrix ensuring dimensional compatibility (d, de after projection).
Confidence Score Computation for Each Prompt. Using the augmented prompts Pa recom- pute the nonconformity scores as described in Section 2.4. For each prompt P., the nonconformity score S(P."ss x. y) is calculated based on its cluster type. The confidence score conf(Piiu)) is then derived as the inverse of the nonconformity score:
conf( paug
(9)
1 S(P. x,
ensuring that lower nonconformity (higher conformity) corresponds to higher confidence.
Selection of Best Prompts from Each Cluster. For cach cluster Ck we select the prompt with the highest confidence score as the best confident prompt:
Pbest, = arg max conf(p.uug)
(10)
pCu EC2
This results in K best prompts Poest,1:K, one from each cluster.
The selected best prompts Pbeet, are passed to the Attention Gate, which computes attention weights ag using a softmax layer.
The integration of eimg into each prompt enhances UMP-Net's ability handle tasks such as visual question answering (e.g., processing the cat image in Figure 3). The seffidenee-basee selection and attention mechanism ensure that the most reliable prompts are prioritized, improving the quality of the final weighted prompt for the LLaMA adaptation.
Multimodal Integration:
This multi-modal architecture strengthens UMP-Net's capability to process diverse data types, leveraging CLIP's pre-trained visual representations and the systematic integration of image embeddings optimize performance for LLaMA adaptation. This proposed method significantly improves LLaMA's 's robustness and adaptability, as demonstrated in subsequent experimental sections.
7

---

Published in Transactions on Machine Learning Research (ectober/202))
Latent Noise Prompting with Mixture of Prompts
Conformal
Heterogeneous
Prediction
Clustering by KNN
I(O.
Textuel Prompts
Visual Prompls
Multimodal Prompts
MLP
Create weighted prompt
Forward
Fine-Tune
Add
The best confident prompt
Backward
Frozen
Attention Gate
Attention Layer Softmax
Attention Layer Softmax
Projection
CLIP
+ 1..N
1.. ..M
Multi-modal Embedding
Figure 3: Schematic representation of the multi-modal architecture of UMP-Net with the focus the combination of visual and textual embeddings to achieve greater LLaMA adaptation. One major multi-modal step is to incorporate CLIP--ased images embeddings to every prompt to generate augmented multi-modal embeddings.
3 Experiments
3.1 Language Only Performance Assessment
Experimental Setup. Following the Stanford Alpaca Taori al. (2023a), we employ data set of 52K
instraction-following examples for training purposes. The UMP-Net model is fine-tuned using 2 RTX 4090 GPUs over 4 epochs. We configure the training with two warmup epochs, batch size of 8, learning rale of 0.009, and weight decay of 0.02. By default, we utilize the LLaMA-Adapler Zhang el al. (2024) pre-trained for version LLaMA2 7B and the foundation pre-trained LLaMA model with 8B version LLaMA3 parameters and N = 32 transformer layers. The prompt length set to dp = 40, and the adaptation prompls are integrated into the final our approach against methods trained on the same 52K instruction dataset, specifically Alpaca Taori et al. (2023a), Alpaca-LoRA Tloen (2023), LLMMA-d Zhang al. (2024) and ALaMA-Exciis Zou et al. (2024), using the widely recognized GPT-4 evaluation benchmark (Chiang et al., 2023b).
30 layers of the model. For quantitative evaluation, we compare
Instruction-Ron Pettruction-FFllowingg Performance Assessment. Table 1 provides comprehensive evaluation of the ability to follow instruction in the proposed UMP-Net model and competing approaches.
This question evaluates the ability UMP-Net to comprehend abstract philosophical ideas, to combine retrospective views of history, and to justify the choice, evaluating the critical thinking and linguistic un- derstanding abilities. In Table the answers provided by different models to the question of defining intelligence by the different philosophical approaches compared. The advantage of UMP-Net is that it provides holistic definition of intelligence which combines cognitive, practical, moral, and emotional aspects relying on the philosophies of Plato, Aristotle, and Confucius and provides balanced preference. Conversely, LLaMA7B offers a very specific definition of problem-solving,, which is not philosophical. Both Alpaca-LoRA and LLaMA- Addattr Zhang et al. (2024) authors talk about Plato and Aristotle without using any elements of modern or emotionality, which restrict the scope of their responses. I.L.MA-Xxcinn Zou et al. (2024) discusses various theorists but lacks a synthesis cohesiveness. The answer to this philosophical question would be the rongestt solution of UMP-Net in its holistic integration and contextual relevance.

---

Published in Transactions on Machine Learning Research 2october/2255
Table I: Instruction-folloinng Comparison for 'What's the intelligence? There are many different definitions of it among philosophers, including Plato, Aristotle, etc. Which do you prefer? Explain in two sentences only'.
LLaMA7B: Plato viewed intelligence as the soul's ability to recall eternal truths through reason, while Aristotle emphasized practical wisdom (phronesis) and rational thought grounded in experience. I prefer Aristotle's view, as it connects intelligence to real-world judgment and decision-makingg
Alpaca-LoRoa Tloen (2023): Intelligence is the capacity to learn, understand, and ap- ply knowlodge, as Aristotle emphasized, which prefer for its practical focus. Plato's view of intelligence as innate, abstract reasoning is compelling but less adaptable real-world problem-solving.
LLaMA-Adapter Zhang et al. (2024): As an AI, don't possess personal preferences like humans do regarding philosophical definitions. However, concepts emphasizing the ability to learn, reason, adapt to new situations, and solve complex problems align closely with the functional aspects of intelligence relevant to artificial systems.
LLaMA-Excitor Zou et al. (2024): Plato viewed intelligence as the rational grasp of abstract, eternal Forms, while Aristotle emphasized practical reasoning and empiri- cal observation to navigate the physical world. I prefer Aristotle's definition for its grounding in tangible experience and adaptability, which aligus more closely with modern, dynamic understandings of human cognition.
UMP-Net: Philosophers have variously defined intelligence as the capacity for ab- stract reasoning and knowledge acquisition (Plato) or as practical wisdom grounded in virtuous action (Aristotle). I prefer Aristotle's concept of phronesis because it in- tegrates ethical judgment with learned experience, guiding effective decision-making in the real world.
Moreover, the UMP-Net model was rigorously assessed using a modified BLEU framework, enhanced by GPT-4 Pro's semantic judgment, across 100 curated samples, prioritizing conceptual alignment over lexical overlap in Figure The evaluation employs the GPT-Score metric: BP exp (Ek-, wy loggk) where BP adjusts for length differences, Pk measures token precision, wa assigns weights and K defines sequence length. Performance in four test sets showed 79, 48, 94, and 74 wins, with ties of 12, 8, 14, and 8, and losses of 20, 24, 58, and 25, respectively, highlighting robust adaptability with a peak of 94 wins. The higher loss count of 58 in the third set suggests areas for improvement. This comprehensive analysis, supported by the GPT-4 reasoning, confirms the strength of the model generating coherent responses while identifying optimization opportunities.
Also Table 2 involves detailed comparison of different models in four major assessment measures: Avg, SOC (Social domain performance, which evaluates activities in socially oriented situations), LAN (Tanguage- focused tasks, testing the capacity of the model to comprehend and produce text instructions and outputs) and TXT (Text-only input performance, which evaluates the performance of the model when dealing with text-only inputs, isolation, that is, without considering the effects of other inputs). The UMP-Net (Ours) proposed gets the best results in all categories, showing that it has better language understanding, generational and socially influenced tasks. The proposed rM (ours) achieves the highest scores in all categories, demonstrating its superior ability in language understanding, generation, and socially influenced tasks. UMP-Net., also performs strongly but is surpassed by UMP-Netza. Other models, such as I.LaMA- Excitor Zou et al. (2024) and LLaMA- LLaMA-Adapter Zhang et al. (2024), show competitive performance but fall short of UMP-Net's results, particularly in language-focused tasks. Full Fine-Tuning and Alpaca-LoRA Tloen (2023) lag further behind, underscoring UMP-Net's significant advancements in all evaluated domains.
To provide further insight into the prompt selection process underpinning these results, a three-phase vi- sualization based on the MoP framework, which is integral to UMP-Net's language-only performance has
9

---

Published in Transactions on Machine Learning Research (October/ 2025)
37%
305
(a) UMP-Net vs. Full - (b) UMP-Net vs. Alpaca Finetuning
(c) UMP-Net vs. LLaMA (d) UMP-Net vs. LLaMA
LoRA Tloen (2023)
Adapter Zhang et al. (2024) Excitor Zou et al. (2024)
Figure 4: The comparison between the performance of the suggested UMP-Net and the models of different networks, presented in one row. All the subfigures comparisons: (a) UMP-Net vs. Full-Finctuming (b) UMP-Net vs. Alpaca-LoRA, (c) UMP-Net. vs. LLaMA-Adapter Zhang et al. (2024), and (d) UMP-Net vs. LLaMA-Excito Excitor Zou et al. (2024).
Table 2: Performance Measures on the performance the model in various categorics. Li refers to LLaMAi use and T refers Template prompts use.
Model
Avg
SOC
LAN
TXT
Full Fine-Tuning
83.20
83.50
82.70
83.40
Alpaca-LoRA Tloen (2023)
82.60
82.50
82.50
82.80
LLaMA-Adapter Zhang et al. (2024)
85.30
84.20
86.10
85.70
ILaMA-Excito Zou et al. (2024)
87.87
86.20
88.30
89.10
UMP-Nct
87.97
86.50
89.20
88.20
UMP-Net 1.2
88.13
86.70
89.50
88.20
LMP-Net (Ours)
88.97
87.70
89.80
89.40
+1.1
+1.5
+1.5
+0.3
been presented in Figure 5. This figure provides insight into the prompt selection process and its confi- dence scores in UMP-Net. It presents three-phase visualization based the MoP framework. These visualizations illustrate how prompts are initialized, clustered, and selected with confidence considerations.
duser Oazer Oboudt
2 Moeal
Ouder1 (Eoeorate))
(a) Randomly Initialized Prompts
(b) Clustered Prompts (K=3) (c) Highlighted Selected Prompts
Figure 5: Viewing of prompt selection in UMP-Net in three phases. a) Getting prompts initially through random iin groups. (c) Prompts of each cluster chosen, which are associated with scores of Conformal Predictions of confidence.
(b) Using Heterogeneous Clustering, cell Heterogeneous Clustering prompts into three
3.2 Multi-modal Performance Assessment
We test visual instruction-follovinng of UMP-Net with the help of paired vision-language instructions and show that UMP-Net is able perform language-only and multi-modal tuning through indirect feature interaction. This low-budget system performs better in tasks based vision, and it uses CL.P Radford et al.
10

---

Published in Transactions on Machine Learning Research (tttobrr/2025)
(2021) to make multi-scale extractions of visual features and a bottleneck MLP layer to match modalities. Hyperparameters coincide with the UMP-Net language-only environment, which is consistent. as well emphasizes its flexibility.
Image Captioning Assessment. We evaluated our model on the COCO Caption dataset Chen et al. (2015), which comprises 0.6M training image-caption pairs (120K images, each with captions) spanning diverse distributions. The evaluation uses frozen CLIP-ViT-L/14 Radford al. (2021) as the image en- coder, with visual embedding dimension D 768 and low-rank dimension r= 16 for efficient processing. Table 3 compares image captioning performance, where UMP-NetLs (Ours) achieves the highest scores. It surpasses LLaMA-Excitor Zou et al. (2024) and BLIP-2 Li et al. (2023a), demonstrating superior captioning capabilities. UMP-Net,. also performs strongly, closely trailing with a BLEU@4 of 49.2 and CIDEr of 157.8.
Table 3: Comparison with State-of-the-Art Image Captioning Methods COCO Caption Chen et al. (2015). Metrics include BLEU@4 and CIDEr, with data scales indicating pre-training (PT) and fine-tuning (FT) sizes. Lí denotes using LLaMAi.
Method
Data Scale
COCO Caption
PT
FT
BLEU@4
CIDEr
ClipCap Mokady et al. (2021)
OM
0.6M
33.5
113.1
VI-PET Zhou et al. (2023)
OM
0.6M
121.7
Qwen-vl-chat Bai et al. (2023)
1.4B
0.6M
131.9
mPLUG-OW12 Ye et al. (2023)
348M
0.6M
137.3
BLIP et al. (2022)
14M
0.6M
40.4
136.7
Flamingo Alayrac et al. (2022)
1.8B
0.6M
138.1
BLIP-2 Li et al. (2023a)
129M
0.6M
43.7
145.3
LLaMA-Adapter V2 Gao et al. (2023b)
OM
0.6M
36.2
122.2
LLaMA-Adapter Zhang et al. (2024)
OM
1.2M
47.4
152.9
MaMA-Fxxitor Zou al. (2024)
OM
0.6M
49.7
157.5
UMP-Net
OM
1.2M
49.2
157.8
UMP-Net (Ours)
OM
1.2M
49.9
158.3
+0.2
+1.2
Additionally, provide several image captioning examples in Figure 6. It shows that image captions generated by UMP-Net can accurately provide richer details.
Zero-shot Multi-modal Assessment. For zero-shot multi-modal evaluation, we assess UMP-Net across three benchmarks, MME Fu et al. (2023), MMBench Liu et al. (2023c), and LVLM-eHuu Xu et al. (2023), covering diverse visual question-answering (VQA) tasks. We compare our method with concurrent multi- modal LLMs, including LLaVA Liu et (2023a), MiniGPT-4 Zhu et al. (2023), LLAMA-Adapter Zhang et al. (2024) and LLaMA- LLaMA-Excitor (Zou et al., 2024).
Table 4 evaluates instruction-tunin performance zero-shot multi-modal benchmarks, covering diverse tasks like perception, reasoning, and commonsense. UMP-Net (Ours) leads with top scores across most metrics. LLaMA-Excctor Zou et al. (2024) and UMP-NetL2 show competitive results, while MiniGPT-4 and LLaVA Liu et al. (2023a) lag behind, particularly in MMBench and LVLM-eHub tasks. The results highlight UMP-Netzs (Ours) superior multi-modal reasoning capabilities. Moreover Table compares zero-shot multi- modal performance on the LVT.M-cHub benchmark Xu et al. (2023) across 44 datasots, evaluating tasks like Visual Perception and Reasoning. UMP-Netra (Ours) achieves the highest average score, outperforming LLMMA-Adaples Zhang et al. (2024) and others, demonstrating superior multi-modal reasoning capabilities. LLaMA-Excitor scores are competitive but lack consistency across tasks.
11

---

Published in Transactions on Machine Learning Research (October/200) 2025)
A vintage magni- fying glass rests old world map, highlight- ing the Atlantic Ocean. The map features detailed illustrations of continents and maritime routes.
A group of people, including children and elders, are crowded on an old Iractor loaded with belongings. They appear to be relocating, possi- bly due to conflict displacement.
Two humanoid robots are having romantic dinner with wine glasses near the Eiffel Tower at sunset. The scene blends futuristic elements with classic Parisian backdrop.
An astronaut in full space gear rides white horse on
A majestic black and gold dragon with sharp fangs and whiskers emerges among blooming cherry blossoms.
lunar-like surface. Behind them, a star-filled outer space sky adds surreal and imaginative touch.
flowing
The vibrant scene combines fantasy and nature under a bright, clear sky.
Figure 6: Illustrations of UMP-Net visual instructionnfollloiing ability of this Instruction: My answer to you about this image. Create a caption to this image.
Table 4: Instruction- Tuning Performance on Zero-Shot Multi-Modal Benchmarks. Metrics include MME
(All, P: Perception, Cognition), MMBench (All, LR: Logical Reasoning, AR: Attribute Recognition, RR: Relation Recognition, FP-S: Fine-grained Perception-Spatial. FP-C: Fino-grained Perception- Coloo,,
Commonsense Perception), and LVLM-eHub (All, VP: Visual Perception, VKA: Visual Knowledge Acquisition, VR: Visual Reasoning, VC: Visual Commonsense). Li denotes using LLaMAí.
CP:
MME
(2023)
MMBennh al. (2aiztee
INLM-eHob Xu (A023)
Model
All
All
L.R
ARt
RR
FP-S
FP-C
CP
All
VP
VKA
VR
VC
MiniGPT-4
867
25.7
0.35
0.53
0.57
LLaVA Lau et al. (2023a)
718
503
215
36.2
15.9
53.6
26.6
41.8
20.0
40.4
0.54
0.62
0.38
0.77
0.79
Adapter Zhang al. (2024)
973
249
39.5
45.0
0.83
0.59
LLaMA-Exestor Zom et al. (2024)
1226
975
250
40.0
14.0
48.0
23.5
45.5
34.0
50.9
2.05
0.74
0.44
0.84
0.60
UMP-Net za
965
228
40.7
17.4
19.5
43.3
35.6
47.8
0.48
0.79
UMP-Net (Gurs)
1228
976
262
41.3
16.5
49.5
24.0
45.8
34.7
51.1
2.80
0.84
0.48
0.85
0.63
+2
+1
+2
+13
-15
+1.3
+0.5
+0.3
+0.7
+0.2
+0.75
+0.1
+0.01
+0.01
+0.03
Table 5: Zero-Shot Multi-Modal Results on the LVLM-eHub Benchmark Xu et al. (2023). Tasks include Visual Perception (VP: ImgCls, OC, MCT), Visual Knowledge Acquisition (VKA: OCR, KIE, Caption), Visual Reasoning (VR: VQA, KGID, VE), and Visual Commonsense (VC: VC, VCR), spanning 44 datasots. Li denotes using LLaMAi.
Xu ct al. (2027)
Taska
Datzads
Models
LLaVA
MiniGPT-4
LLAMA LLAMMA-ddptee
LLaMA Excisor
UMP-Nett
UMP-Nee
(211253)
Zhong
Zou (20144)
(Ours)
Percrption
ImggCis, MCI
0.02
0.78
1.79
0.78
Visual Knowkdgge Acquisition
OCR, KIE, Caption
17
0.38
0.35
0.44
0.41
0.47
0.48
Visual Reusming
VQA, RGID, VE
13
0.77
0.53
0.85
0.80
0.79
0.85
Commonssuse
VeC
Avesuge
0.64
0.35
0.67
0.655
01.675
0.085
+0.015
3.3 Ablation Study
We conduct an ablation study to evaluate the impact of key components UMP-Net, focusing on the number of insertion layers in the pre-training transformer, the number of randomly generated prompts and the number of generated prompt tokens. The results are summarized in Table 6, with performance
12

---

Published in Transactions on Machine Learning Research 2october/225)
measured in terms of validation accuracy (Val Acc.), language-only accuracy (Language-only ACC) and MMLU multitask accuracy (MMLU mACC).
Table 6: Ablation Study on UMP-Net. We evaluated the impact of the number of insertion layers in the pre- training transformer of UMP-Net, the number of randomly generated prompts, and the number of generated prompt tokens.
Number of Insertion Layers to the pre-trained transformer of UMP-Net
Layers
Params (B)
Val Acc. (%)
8
0.85
62.41
16
1.12
78.92
24
1.34
88.93
32
1.58
84.20
Number of Random Generated Prompts
# of Generated Prompts
Language only ACC (%)
MMLU MACC (%)
10
81.75
78.30
20
88.69
86.25
30
88.93
87.80
40
88.08
86.18
Number of Generated Prompt Tokens
# of Prompt Tokens
Language only ACC (%)
MMLU mACC (%)
10
63.20
54.10
20
77.50
67.30
30
84.60
72.80
40
88.93
87.80
The first part of Table examines the effect of varying the number of layers inserted in UMP-Net. Increasing the layers from 8 to 24 (parameters from 0.85B to 1.34B) siguificantly improves the accuracy of the validation, reaching 88.93% with 24 layers. However, further increasing to 32 layers (1.58B parameters) results in a slight decrease to 84.20% suggesting that 24 layers strike optimal balance between model capacity and generalization for this task. The second part of the table analyzes the effect of the nunber of random generated prompts on language-only accuracy and MMLU multi-lask accuracy. Nonetheless, at 40 prompts, both metrics drop slightly to 88.08% and 86.18%, respectively, and hence 30 random prompts give the optimal tradc-off between diversity and overfitting. The last column of the table examines the effects of the number of created prompt tokens. The language-only accuracy and MMLU mACC exhibit an upward trend that does not change over the increasing number of tokens between 10 and 40 tokens. This implies that increasing prompt tokens length improves the capability of the model capture contextual nuances as the 40 tokens perform the best in both mcasures.
3.3.1 Conformal Predication and Cluster Configurations
To properly test the contributions of the MoP modules in UMP-Net we also performed ablation studies on Conformal Predictions and Heterogeneous Clustering. Table 7 contrasts the full UMP-Net model with the one where Conformal Predictions instead. The complete model that uses Conformal Predictions to choose the most successful prompt using nonconformity scores has a better ScienceQA accuracy (88.97%) than the model that docs not use Conformal Predictions (87.5%%) This betterment indicates that it is most siguificant to select prompts prudently to the uncertainty to improve the reliability of the model, especially in multi-modal tasks with uncertain inputs, The computational load is not much higher, which is why the GPU memory (7.5 GB vs. 7.4 GB) and throughput (19.2 vs. 20.1 4-Samples/Sed are slightly less, which shows that Conformal Predictions are beneficial in terms of resource consumption that does not contribute to significant performance losses.
not used and the prompt selection is performed using cluster centroids
13

---

Published in Transactions on Machine Learning Research (October/2022 2025)
The effect of the Heterogeneous Clustering module is considered in Table 8 using range of clusters and modality configurations. Single cluster with vision-language (VI.) modalities have the lowest accuracy (85.02%), whereas single-modality clusters (image-only and text-only) do better (86.20% and 87.80%, re- spectively). Three-cluster UMP-Net configuration (Text, Image, VL) has the best accuracy (88.97%), which proves that heterogeneous clustering allows organizing promptly and adapting to a lask. The metrics of cal- culation are similar across configurations, and the memory usage of the GPUs is 7.4-7.6 GB and throughput 19.2-22.4 t-Samples/Sen which confirms the effectiveness of the clustering method.
Table 7: Effect of Removing Conformal Predictions. Comparison of UMP-Net with and without Conformal Predictions.
Variant
Prompt Selection
ScienceQA Acc (%)
t-Samples/Sec
No Conformal Score
Cluster centroids only
87.50
20.1
UMP-Net (full)
Best-conformal prompt
88.97
19.2
Table 8: Varying Cluster Configurations. Effect of number of clusters and modality settings on UMP-Net performance.
# Clusters
Modalitics
ScienceQA Acc (%)
t-Samples/So
1
VL-Prompts
85.02
22.4
1
I-Prompts
86.20
20.8
1
T- T-Prompts
87.80
21.0
2
(T-Prompls, VL-Prompis)
88.00
21.5
3 (UMP-Net)
(T-Prompts, I-Prompts, VL-Prompts)
88.97
19.2
3.3.2
Robustness of Uncertainty Estimation to Noise and Domain Shifts
To assess the reliability and robustnessof Conformal Prediction (CP) to domain shift and noisy inputs, evaluated UMP-Net's calibration ScienceQA subset (100 samples, 20% OOD, 20% noisy, p-0.2), targeting 90% coverage (1-a=0.90) Vovk et al. (2005); Zou et al. (2024) (sce Tables 9 and 10).
Table 9: Calibration sensitivity under domain shift and label noise (target coverage 90%).
Method
Strotege
Useode (1.)
Coviit (00I))
Cwvlit0 (Noiie)
Set Stoe (10)
Set Size (OOI)
Set Size (Nolac)
EXCK (1I) ECB: (UN:)U0)) BXC (Nole))
UXIP-Net Ccunllo UP)
sitt
Chussee-ooditboll
1.61
Importance weigltdd
IW-CP aldieg--rtdde
Trimmed TrmpScals
quartive
MiC-Dnoper
stoechuttic pases
1,00
Emsnisle
M assied markss
Hylnid
We validated these findings with a new experiment on ScienceQA (100 samples, 20% OOD, 20% noisy inputs) using an RTX 4090 GPU. Methods: UMP-Net with IW-CP SW vs. vanilla CP and MC-Dropout. Results: IW-CP + SW achieves 89.5% coverage (vs. 84.0% for vanilla CP, 85.5% for MC-Dropout), ECE of 0.035 (vs. 0.070, 0.060), and accuracy of 89.2% 39.2% (vs. 88.41%, 85.0%). IW-CP + SW outperforms vanilla CP and Bayesian methods, achieving 89.6% OOD and 88.2% noise coverage, with low ECE (0.034, 0.044) and minimal overhead (4.3 ms). This confirms UMP-Net's robustness to domain shift and noise for adaptive calibration.
3.4 Computational Efficiency Analysis
To compare the computational efficiency of UMP-Net, we compare its memory consumption, training throughput, and inference latency with full fine-tuning and two representative PEFT basclines: LoRA and LLaMA-Adapter Each of these measurements was taken on one training with mixed-precision (FP16), batch size of 8, and NVIDIA RTX 4090 (24GB VRAM). The results described in Table 11.
14

---

Published in Transactions on Machine Learning Research (october/2025)
Table 10: Calibration protocol and drift handling settings.
Method
Cal. Pool
Drift Test
Threshold
Extras
Mondrian CP
0.10
per-cluster, Nc=5,000
KNN k-15; clusters-3
IW-CP
0.10
global, N=20,000
MMD (CLIP)
0.15
RBF 0=0.5; IW normalized
TW-CP + SW
0.10
global + recent
MMD (CLIP)
0.15
W=2,000; T=0.9; update every 500
Temp
0.10
global, N-20,000
0=0.10; T-1.5
MC-Dropout
T=10 10 passes; p=0.1
Ensemble
M=5; diversity seeds=5
Hybrid
0.10
global, N=20,000
MMD (CLIP) 0.15
Ava score = qa + X-Var
Table 11: Computational efficiency comparison of UMP-Net against full fine-tuning and PEFT baselines.
Method
Trainable Parameters
GPU Memory Usage
Training Throughput (samples/s
Inference Latency
(GB)
(ms/sample)
Full Pine-Tuning
7B (100%)
30
25
180
Alpaca-LoRA (rank 4)
5M (0.07%)
12
60
190
LL
50M (0.7%)
14
50
195
UMP-Net (Ours)
60M (0.9%)
15
48
200
UMP-Net can use less than half of the amount of GPU memory used in a full fine-tuning of only 15GB of memory, and is capable of executing on a single high-end consumer graphics card such as the RTX 4090. Memory usage is slightly greater in UMP-Net (which has multi-modal CLIP projections and uncertainty estimation), compared to LoRA (12GB) and L.AMA-Adapter (14GB), although it is very efficient. as method that supports both language and vision tasks. It has a training throughput of 48 samples/s, which is 1.9x faster than full fine-tuning (25 samples/s). Although slightly low compared to LoRA (60 samples/s) and LLaMA- LLaMA-Adapter (50 samples/s), the throughput of UMP-Net is less by 4% than LLaMA- Addater,, showing that the extra overhead of its MoP and Conformal Predictions is negligible compared to the performance benefit in multi-modal lasks (e.g., 88.97% ScienceQA accuracy).
The inference latency of UMP-Net is around 200 ms per sample, which relatively low (only 11% more than full fine-tuning: 180 ms). This slight improvement can be attributed to the active prompt selection and multi-modal processing, which allow instruction-ffolloinng and multi-modal processing state of the art, which are presented in COCO Caption (BLEU@4: 49.2) and ScienceQA (88.97% accuracy). These findings show that UMP-Net is highly balanced in terms of both computational performance and task performance, with low memory and speed overheads compared to current PEFT algorithms but that it performs better in multi-modal and lnslloigtonnfolloiinng tasks. Tn the applications where latency is of the essence, such overhead indicating an obvious improvement be made in the future.
3.5 Challenges and Limitations
The rapid selection multi-modal environment is extremely problematic as the input modalities are different (textual, visual, and cross-modal) and have to be adapted to the tasks. The main difficulty is modality misalignment in which prompts that are best trained in a single modality (e.g., text) do not transfer to other modalities (e.g., images), which would be the most optimal prompt in task that involves integrated reasoning such as ScienceQA. To illustrate, a written cue can fail to reveal visual aspects that are important in image captioning as the case with the COCO Caption dataset. The other difficulty lies in dealing with ambiguity or noise in unclear or incomplete inputs, e.g. bad images or unfinished text, resulting in the possibility of confident and incorrect outputs in high-stakes systems like medical diagnosis. Current PEFT systems, including LoRA and prompt tuning, do not typically have support to dynamically reuse prompts particular tasks quantify prediction uncertainty, and thus are not robust to multi-modal tasks. Also, the
15

---

Published in Transactions on Machine Learning Research (October/2022
computational complexity of the processing of several modalities may be prohibitive in resource-constrained settings, where pre-training at scale, as in the case of models such as Flamingo, is not possible.
How UMP-Net Addresses These Challenges. UMP-Net can help deal with these challenges by its MoP framework which combines a number of essential elements. Latent Noise Prompting module adds variability to the prompts in a controlled manner that is adaptable to a large variety of tasks and modalities. This module alleviates the issue of modality mismatch by sampling noise in a Gaussian distribution to make prompts capture different semantic and syntactic features, and uses multifaceted set of candidate prompts to address this problem. The Heterogeneous Clustering module, which employs KNN-based clustering, divides prompts into modality-specific groups (text, image, cross-modal) so that prompts are trained to match the input type, evidenced by the fact that the result of prompts is improved when using three clusters (Heterogeneous Clustering) (88.97%) than when using one joint cluster (85.02%) (see Table in Section 3.3). Conformal Predictions also add reliability by calculating the nonconformity score identify the most confident prompts to reduce uncertainty uncertain inputs. As an example, in visual question-answering problems, Conformal Predictions focus prompis that best. fit. the features of the input. image, leading to better accuracy improvement by 1.47%. than cluster-centroid selection (see Table 8 in Section 3.3. The multi-modal architecture is a CLIP-based model that effectively fuses visual and textual embeddings without the need to pre-train costly models that use CLIP to align modalities as demonstrated by UMP-Net using less pretrained CLIP representations to achieve better performance on COCO Caption (BLEU@4: 49.2, CIDEr: 157.8).
Potential Limitations. In spite of these developments, UMP-Net has shortcomings that are worth inves- tigating. To begin with, the computational cost of data generation and evaluation in the MoP architecture, although small (e.g. 19.2 t-Samples/See vs. 20.1 without Conformal Predictions), can in any case be substan- tial in severely risource-constranne environments, including edge devices. Second, the KNN-based feature representations used in Heterogeneous Clustering can weaken should there be a change in the input data distribution and change dramatically with respoct to the training set. As an example, when dealing with highly specialized lasks (e.g., rare medical imaging (asks), the performance might have to be supported by adding more clusters or retraining. Third, although Conformal Predictions are better at quantifying uncer- tainty, their nonconformity scores are also dependent the quality of the extraction-funccion, which can be in error in the case of highly-noisy inputs. To overcome such limitations in future work, adaptive clustering strategies and lightweight uncertainty estimation techniques may be considered, to make UMP-Net more applicable in the various and resource-constrainee situations.
4 Conclusion
This paper presented UMP-Net, an Uncertainty-Awar Mixture of Prompts Network, which can improve the Instraction-follooinng of LLaMA in a parameter-effoctive and uncertain way. Combining Latent Noise Prompting, KNN-based Heterogeneous Clustering and Conformal Predictions, UMP-Net provides strong capability to control prompt redundancy, estimate of uncertainty, and selectively choose credible prompts in adaptation. Our multi-modal system (with the CLIP--baee embeddings) also supports the vision-language integration casily, which tackles the problems of cross-modal reasoning without having to undergo extensive pre-training. Significant benchmark experiments on ScienceQA, COCO Caption and zero-shot multi-modal tasks, show that UMP-Net performs better with an average accuracy of 88.41% on ScienceQA and a score of 158.3 on COCO Caption and zcro-shot multi-modes, outperforming other state-of-the-art models such as LLaVA and LLaMA-Excitod Looking ahead, future work could explore the application of UMP-Net to other LLMs beyond LLL.A, investigate its scalability to larger multi-modal datasets, and incorporate dynamic uncertainty thresholds to further improve prompt selection in real-time scenarios.
References
Jean- Lean-Baptiste Alayrac et al. Flamingo: a visual language model for few-shot learning. arXio preprint arXiv:2204.1410 2022.
16

---

Published in Transactions on Machine Learning Research 2october/225)
Akari Asai, Mohammadreza Salchi, Matthew E Peters, and Hannanch Hajishirzi. Attempt: Parameter- efficient multi-task tuning via attentional mixtures of soft prompts. arXiv preprinl. or orXiv:2205.11 11961, 2022.
Stephen Bach et al. Promptsource: An integrated development environment and repository for natural language prompts. ar preprint wrXiv2202.012299, 2022.
Jinze Bai et al. Qwen-vl: A frontier large vision-language model with versatile abilitics. arXiv preprint
arXiv:2308. 129666 2023.
Christopher M. Bishop. Paltern Recogration and Machine Learning. Springer, New York, 2006.
Tom Brown et al. Language models are few-shot learners. arXiv preprint orXiv:2005 14 2020.
Xinlei Chen et al. Microsoft coco captions: Data collection and evaluation server. arXiv preprinl
arXiv:1504.00 2015.
Wei-Tän Chiang et al. Vieuna: An open-source chatbot impressing gpt-4 with 90 arXio preprint arXiv:2303.14400 2023a.
Wei-Lan Chiang et al. Gpt-4 evaluating benchmark: comparative study of language models. arXiv preprint arXiv:2305.1223 2023b.
Chaoyou Fu et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXim:2306.1330) 2023.
Peng Gao et al. Dynamic adapter for vision-language models. ar preprint arXin:2305 12345, 2023a.
Peng Gao et al. dlama-adaptedd v2: Efficient fine-tuning of language models. arXiv preprint or 0rXiv:2304.15010 2023b.
Neil Houlsby et al. Carameter-efficient transfer learning for nlp. Proceedings of the International Conference on Machine Learning, 2019.
Edward J. Hu al. Lora: Low-rank adaptation of large language models. ar preprint arXiv:2106.09633, 2021.
Ruixiang Jiang, Lingbo Liu, and Changwen Chen. Mope: Mixture of prompt experts for parameter-efficient and scalable multimodal fusion. arXiv preprint orXiv:2433. 10568, 2024.
Daniel Khashabi al. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint arXi::2005.00700 2020.
Wonjae Kim et al. Vilt: saiio-and-languaae transformer without convolution or region supervision. arXiv preprinl Xix: 03334, 2021.
Brian Lester et al. The power of scale for parameter-cfficinn prompt tuning. arXiv preprint orXh::2104.08691, 2021.
Junan Li et al. Blip-2: Bootstrapping language-imag, image pre-training with frozen image encoders and large language models. arXiv preprint xrXim:2801.125:) 2023a.
Junnan Li al. Blip: Bootstrapping language-image pre-training for unified vision vision-language understanding and generation. arXiv preprint arXho:2201.12088, 2022.
Junnan Li et al. Blip-2: Bootstrapping language-nmage pre-training with frozen image encoders and large language models. arXiv preprinl XrXi::2301.1100 12597, 2023b.
Liunian Harold al. Visualbert: simple and performant baseline for vision and language. arXiv preprinl ar Xiv:19908.. 03557, 2019.
17

---

Published in Transactions on Machine Learning Research 2october/225)
Liunian Harold Li et al. Visualbert: Further improvements. arXiv preprint arXhv:2005.12355 2020.
Haotian Liu et al. Visual instruction tuning with llava. arXiv preprint arXiv:2304. 2023a.
Haotian Liu et al. Visual instruction tuning with llava. arXic preprint arXiv:2304.08458 2023b.
Yuan Liu et al. Mmbench: Evaluating multimodal large language models. orXiu preprint or arXiu:2307.0628. 2023c.
Pan Lu et al. Learn to explain: Multimodal reasoning via thought chains for science question answering. arXiv preprint arXim:2209.0000 09513, 2022a.
Pan Lu et al. Patch-trm: A transformer-based model for visual question answering. arXiv preprint arXiv:2205.090. 099776 2022b.
Ron Mokady et al. Clipcap: Clip prefix for image captioning. arXiv preprint arXic:2111.09733, 2021.
OpenAl. Chatgpt: Optimizing language models for dialogue. http:://pennaa.cc///llgggaaaaaae 2023a.
OpenAL. Gpt-4 technical report.
2023b.
Long Ouyang et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.022 02155, 2022.
Baolin Peng et al. Instruction tuning with gpt-4. orXiv preprint arXiv:2304.03277, 2023.
Jonas Pfeiffer et al. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprinl arXiv:2005.002// 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748 8763. PmLR, 2021.
Rohan Taori el al. Stanford alpaca: Instruction--ollowingg language model. aie preprinl arXi::2303.08714 2023a.
Rohan Taori et al. Stanford alpaca: An instruction-follooinng llama model. Xive preprint orXiw:2302.1000 13971, 2023b.
Tim Tloen. Alpaca-lora: Efficiently fine-tuning 1laaa with low-rank adaptation. GaHuu Repository, 2023.
URL https://github. com/ttlonnaappccallraa
Hugo Touvron et al. Llama: Open and efficient foundation language models.
arXio preprint
arXiv:2302.13977 2023.
Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2017.
Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in Random World. Springer, New York, 2005.
Ruochen Wang, Sohyun An, Minhao Cheng, Tianyi Zhou, Sung Hwang, and Cho-Jui Hsieh. One prompt is not enough: Automated construction of mixture-of-experr prompts. arXie preprint 0rXiv:2407.00255 2024.
Yizhong Wang et al. Superrnaturrlinsstruttossss Generalization via declarative instructions on 1600+ nlp tasks. Xixiv preprint arXiv:2204.07700 2022.
Jason Wei et al. Finetumed language models are zero-shot learners. arXiv preprint arXiv:2109 0tti::2209.0115... 2021.
Jason Wei et al. Finetuned language models zero-shot learners. arXio preprint arXim:2109.016 2022.
18

---

Published in Transactions on Machine Learning Research Doctober/2025
Zichen Wu, Hsiu-Yuan Huang, Fanyi Qu, and Yunfang Wu. promppt-eottooo for multi-modal
semantic understanding. arXiv preprint ar arxi::2403. 11311, 2024.
Chenfei Xu et al. Lxim-chub: A comprehensive evaluation benchmark for large asiio-languuge models.
arXiv preprint ar Xix:20300. 09266, 2023.
Qinghao Ye et al. uplug-ow12 Revolutionizing multi-modal large language model with modality collabora- tion. aixi preprint or 111:23311.7006, 2023.
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Llama-adapter: Efficient fine-tuning of large language models with zcro-initialized attention. The Twelfth International Conference on Learning Representations, 2024.
Zhuosheng Zhang et al. Multinodal chain-of-thought reasoning in language models. aiv preprint orXiv:2302.0080 2023.
Yang Zhou al. Vi-pet: Vision-annd-anguugee parameter-efficiet tuning via granularity control. ar preprint 0rXit:2308.061133 2023.
Deyao Zhu et al. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
Bo Zou, Chao Yang, Yu Qiao, Chengbin Quan, and Youjian Zhao. Llama-excitor: General instruction tuning via indirect feature interaction. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 14089 14099, 2024.
A Appendix
A.1
Overview
Section A.1.1:
Related work
Section A.1.2:
More Instruction-Followngg Evaluations
Section A.1.3:
More Multi-modal Evaluations
Section A.1.4:
Ablation Study
Section A.1.5: Computational Efficiency Analysis
A.1.1 Related Work
Instruction Tuning of Large Language Models. The development of instruction-tumed LLMs has significantly advanced the field of natural language processing by enabling models to follow human-like instructions. Initial works such FLAN Wei et al. (2021), PromptSource Bach et al. (2022), and SUP- NATINST Wang et al. (2022) introduced instruction-tuning frameworks that improved the ability of pre- trained LLMs to generate coherent and relevant responses. InstructGPT Ouyang et al. (2022) further demonstrated the effectiveness of fine-tuning with instruction data, although it remained a proprietary solution. Open-source alternatives, such as Stanford Alpaca Taori et al. (2023b), fine-tuned all 7B paramoters of LLaMA Touvron al. (2023) using 52K self-instruct data. However, full fine-tuning of such large models is computationally expensive and inefficient, leading to the need for more parameter- errameter-efficient adaptation methods. Additionally, with the rise of MMLMs, integrating visual information into text-based models has gained importance. Works such as Flamingo Alayrac et al. (2022), BLIP-2 Li et al. (2023b), and LLaVa VA Liu et al. (2023a) have introduced techniques for vision-language alignment. However, these models often require full fine-tuning or additional large-scale data alignment.
Parameter-cfficient Fine-tuning. To address the inefficiency full fine-tuning, various PEFT approaches have been proposed. LoRA Hu et al. (2021) employs low-rank adaptation matrices, while prompt tuning
19

---

Published in Transactions on Machine Learning Research (October/2022
Lester et al. (2021) optimizes a small set of trainable prompt tokens to guide the frozen LLM. Adapter-based techniques Houlsby et al. (2019); Pfeiffer et al. (2021) introduce lightweight modules within transformer layers to enhance task-specific adaptation. LLaMA-Adapter Zhang et al. (2024) proposed an efficient fine-tuning framework that freezes LLaMA's 's pre-trained paramoters and optimizes a small set of adapter modules. The capabilities are extended to multi-modal learning with a lightweight vero-initialized atlention mechanism, unlike Alpaca-LoRA Tloen (2023), which uses LoRA in the original network structure, which is offered by LLaMA-Adapter. Our method is more efficient and generalizes better than alternative instruction-tuned LLaMA versions like Vicuna Chiang et al. (2023a) and LLaMA-GPT4 Peng et al. (2023) which aim to enhance performance on the dataset. In addition to this, our method also effectively incorporates visual information in the instruction-following feature of LLaMA, enhancing cross- reasoning at a low computational cost unlike existing multi-modal fine-tuning techniques.
Multi-Modal Adaptation for Large Language Models. The importance of multi-modal learning has become critical in the development of LLM, as model outputs can now be generated and processod under conditions of both text. and image-related inputs. Architectures that combine visual encoders with transformer-baser LLMs have been suggested in works like Flamingo Alayrac et al. (2022), BLIP-2 Li et al. (2023b), and LLaVA Liu et al. (2023a). LLaMA-Adapter Zhang et al. (2024) proposed an effective multi- modal model, which is built with a zero- iniiiiaiiiiiidd attention process that enables smooth visual-textual modalities alignment without requiring any finetuning of the frozen LLaMA. Likewise, I.LaMA-Excitor Zou et al. (2024) expands the multi-modal features by indirect interaction of features. Though these models produce impressive results, they usually need large-scale fine-tuning which is computationally expensive. Unlike this, our suggested UMP-Net will extend these developments and combine uncertainty-coosciouss prompt tuning with Conformal Predictions and clustering based KNN to guarantee maintainable prompt- selection and remove repetition.
Mixture of Prompts and Expert Approaches. More recent studies have investigated mixture-based prompt tuning to make language and multi-modal models more efficient and adaptable in terms of their parameters. ATTEMPT Asai el al. (2022) suggests a technique that involves attentional mixtures of soft prompts to cross-task transfer knowledge through interpolating pre-trained source prompts with target prompt via a lightweight attention modulc. Such strategy attains high performance using much smaller updated parameters (e.g. 2,300 times smaller than full fine-tuning) and aims at multi-task language model adaptation. Likewise, MoPF-BAF Wu et al. (2024) also propases a framework that uses specialized prompt experts on text, image, and integrated modalities and can achieve better few-shot text and sentimental analysis with a small fraction of the size of larger models. One Prompt is not Enough Wang et al. (2024) is a system that automatically builds a mixture-of-experl prompls based on more than one expert prompt, and uses it to boost task-specific performance, specifically in the automated context. MoPE Jiang et al. (2024) builds upon this idea by breaking down prompts into instance-adaptive experts, where multimodal pairing priors are used to route the most successful prompt, and which achieves state-of-the-art performance with just 0.8% of the trainable parameters compared to fine-tuning.
These methods are similar to the MoP framework of UMP-Net that also uses a combination of prompts to produce parameter efficiency and adaptability in molti-modal tasks. Similar to ATTEMPT Asai et al. (2022), UMP-Net uses a wide range of prompts, but adds to it a Latent Noise Prompting to add controlled variability. Like MoPE-BAF Wu et al. (2024) and MoPE Jiang et al. (2024), UMP-Net activates modality- specific prompt rrganization, but employs Heterogeneous Clustering to dynamically form prompis into text, image, and cross-modal clusters to improve alignment with the type of input. The automated expert selection of One Prompt is not Enough Wang et al. (2024) is comparable to the dynamic prompt selection of UMP- Net, but UMP-Net is the only model to use Conformal Predictions to quantify uncertainty and choose the most reliable prompts to increase robustness in uncertain or noisy inputs.
A.1.2 More Instruction-Followie Evaluations
Table 12 and Table 13 use a more difficult problem, such as Gaussian elimination to further test UMP-Net mathematical reasoning and code generation abilitics, whereas Table 14 evaluates its historical analysis and comparison problem solving capacities that need deep recollections of facts and contextual synthesis. The combination of questions comprises philosophical reasoning, technical programming, mathematical problem-
20

---

Published in Transactions on Machine Learning Research (October/202)
solving, and historical analysis, which enables overall assessment of the estrucctin-followinng skills of UMP-Net in the context of abstract, technical, and analytic skills.
21

---

Published in Transactions on Machine Learning Research 2october/225)
Table 12: Instruction-following Comparison of 'Write algorithm to predict mull values in matrix. A value is considered null if its corresponding weight is equal to zero. Usign python language"
LLMMA7::
import numpy np
def predict_na null_values values (matrix): # Calculate the mean of each row, ignoring zero entries row_means = up mean (matrix, axis=1)
# Replace zeros with the corresponding row mean for in range (matrix. shape
for j in range (matrix... shape [1]):
if matrixli, j] == 0:
matrix j] row_means [i]
return matrix
Alpaca-LoRA Tloen (2023):
import random
def predict_nuld rows len (matrix)
(matrix):
cols len (matrix [0])
null_count
# Count the number of null values in the matrix for i in range (rows): rrows(;)
10
for j in range (cols): if [i]]] 0: null_count
11
12
13
14
# Calculate the probability of value being mull auob-probyy null_count (rows * cols)
17
# Predict null values based on the probability for i in range (rows):
18
for j in range (cols): rangs(coll):
19
20
if random... random null.probability:
21
matrix[i] [j]
return matrix
22

---

Published in Transactions on Machine Learning Research (October/ 2025)
LLaMA-Adapter Zhang et al. (2024):
import numpy as np
def predict_nulie nulls (matrix, weights): matrix mp. array (matrix) weights n array (weights)
# Create copy of the matrix to avoid modifying the original pred_matrix = matrix.
20
# Iterate through each element in the matrix for i in range (matrix. shape [0]):
11
12
for j in range (matrix. shatri shape[1]] [1]):
13
if weights [i] [i] == 0:
# Predict the value if weight is zero non mon_null matrix[:, [][weights j] != 0] if len (non_null) 0:
16
17
# Use mean of non-null values in the column as prediction
pred_matrix(1][] np mean (non_null)
18
19
elae:
# If all values in the column are mull, set to zero
21
pred_ leed_matrix_ie [i] [j] = 0
22
return pred_matrix matrix
23

---

Published in Transactions on Machine Learning Research (October/ 2025)
LLaMA-Excitor Zou et al. (2024):
def
values (matrix, weights):
uun
Predicts null values in matrix based on weights.
:param matrix: list lists representing the matrix. :param weights: A list of lists representing the weights.
:return: A list of tuples containing the indices of the mull values.
aull_values []
10
11
Check if both matrix and weights have the same dimensions if len (matrix) !- len (weights) not (1een(roow) == len
weights [o]) for in matrix):
raise ValueError ("Matrix and weights must have the same dimensions")
14
15
Iterate over each element in the matrix and its corresponding weight
16
for i, row in (marri)
for j, value in enumerate (row): # Check if the weight is zero if weights [i] == 0: # Add the indices of the null values to the list null_values a ((i, j))
19
20
21
22
23
return mull_values
24
Execute the function and print the results null_ values = predict_eat peset__uul_vvaleeee (matrix, weights) print N"Nll values at indices:", aull_values,
#
27
24

---

Published in Transactions on Machine Learning Research (October/ 2025
UMP-Net:
import numpy as up
def predict_null_vattt lirii_nnnllvllalel values (matrix, weights);
Predict null values matrix based on corresponding
weights.
Arga:
matrix (np ndarray): Input matrix. weights (np. adp.nayrra): Weights corresponding to each value in the matrix.
10
Returns:
11
up ndarrayyy: Matrix with predicted mull values (NaN where weights are zero.
18
return np www.w = 0, np nan, matrix)
14
15
Example usage:
matrix ap.array ([[1, 2, 31, [4, 5, 6]])
26
17 weights mp. array ([[1, 0, 1], [1, 0]])
result = predict mull, values (matrix, weights)
print (result)
Table 13: Instruction-folloiin Comparison for 'Design a Python program to solve a system of linear equa- tions using Gaussian climination for a 3x3 matrix. Provide a step-by-step explanation of the algorithm alongside the code'.
LLaMA7B:
def gaussian_elininannne elimination (matrix): # Step 1: Convert the matrix into an augmented matrix sugmented_matril
matrix [[O, 0, ol] # Add row zeros at the bottom
# Step 2: Perform Gaussian elimination for i in range(3). (3):
# Find the pivot element in the current column pivot_row
for j in range 1, 3): if abs { abs (augnented_matrix pivot_rom [rottroow][i]: [i]):
20
pivot_row
18
# Swap the current row with the pivot row augmented www.aaiaattootttooommo augmented segmented_matrix matrix [i], augmented matrix (pivot_rooi
14
augmented_matrix
25

---

Published in Transactions on Machine Learning Research 2october/225)
# Eliminate the variable associated with the pivot element for j in range ( if i!= J:
factor [][i]
pammented_matrxxllee / sugmented_matray
for range(3):
augmented_ -= factor
@ygeette_mmarrie [k]
# Step 3: Back Sack-substitution solution [0, 0, 0] for i in (2, -1, -1): sum terms for j in range (i 1, 3)
10
11
12
terms += solution [j] augmented_matrix [i] [j] solution[i] (augmented eetemented_matrixxeeee matrix [i] - sum_ terms) /
augmented_matrixxiiee [i]
15
16
return solution
17
18
Example usage
matrix = [
[2, 1, 1].
21
[1, -1, 1],
22
[-1, 2, 2]
23
24
solution gaussian elimination (matrix)
print ("Solution:", solution)
Alpaca-LoRA Tloen (2023)
import numpy np
def coussian_eliinaatonn (auggetteannt
mat = augmented matrix bogmented_matrixtte ()
n 3 Size of the matrix
Step Forward Elimination for range(n):
# Find the pivot (the row with the largest element in the current column)
20.
max_row =
11
for j in range(i, n):
12.
if abs (mat[j, il) abs (mat [max_row, i]): j
13.
max_ron
34
Swap the current row with the pivot row if max max_row row != i:
35
mat [[i, max_row]] = mat [[max_[[ov] now, i]]
26

---

Published in Transactions on Machine Learning Research (october/2025
# Scale the pivot row to make the pivot equal to 1
pivot mat [i, i] if pivot == 0:
continue Skip if pivot is zero (for singular matrices)
mat [i] /= pivot
# Eliminate all elements below the pivot for j in range + 1, n): factor mat i] mat[j] factor mat
10
11
# Step 2: Back Substitution
# Since the matrix is in upper triangular form, back substitute to find solutions
14
x, y, = 0, 0
25
16
Start from the last equation z mat [2, 3]
17
18
19
# Move to the second equation
y = mat ma 3] mat 2] * 2
21
22
# Move to the first equation
23
x mat [0, 3] mat [0, 1] y mat [0, 2] z
24
25
return [x, y, z]
# Example usage
# Define the augmented matrix (3x4 matrix)
augmented_matrix matrix mp. array
29
30
[2, 1, 1, 10),
[1, 2, 1, 81,
31
82
[1, 1, 2, 7]
1, dtype=float)
33
35
# Perform Gaussian elimination
36
solution =
bougmented_matrix.
37
34
# Print the original augmented matrix
39
print ("AnOriginal Augmented Matrix:") print (augmented_matrix)
# Print the solution
43
print ("AnSolution using Gaussian Elimination:" print (f"x {solution [ y {solution [1]: .2f,, {solution :22)")
44
27

---

Published in Transactions on Machine Learning Research (October/ 2025)
ALapter Zhang et al. (2024):
def
b):
n (e)(a)
# 1. Forward Elimination
for i range(n):
# Pivoting (optional but improves numerical stability)
if [[i][i] == 0:
# Find row below with a non-zero element in the same column
for range 1, n):
10
[ik] !=
11
a[il, a[k] = a[k] a[i]
# Swap rows
b[il, b[k] = b[k], b[i]
break
14
else: return None
25
Singular matrix
16
for j range 1, n): factor a[j][i] / a[i][i]
17
18
19
for in range n)
[kj][k] a[j] - factor * a[i] [k]
b b[j] b[j] - factor * b[i]
21
22
23
# 2. Back Substitution
x [0]
24
25
for i in range
-1, -1):
x x[i] b[il
for j in range + 1, n):
x x[i] x[i] - a[i] * x[j]
x x[i] x[i] / a[i][i]
29
30
31
return x
82
33
# Test Case
35
a = [[2, 1, -1], [-3, -1, 2], [-2, 1, 2]]
36
b = [8, -11, -3]
37
34
solution solve Satnearr_ystemaaa_ b)
39
if solution: print ("Solltiion; solution) # Expected output [2, 3, -1]
42
else:
43
print Th matrix is singular and has no unique solution. ")
28

---

Published in Transactions on Machine Learning Research (October/ 2025)
-LaMA-Exxcta Zou et al. (2024):
def wwtrssianeeiiiiatiiin axtiiiinaiinnnnnnnaaaeea (matrix):
# Step 1: Ensure the input matrix is in the correct format
if len (matrix) != or (1y( 4 for row matrix):
return "Invalid matrix dimensions. Expected 3x4."
# Step 2-4: Gaussian Elimination with Partial Pivoting for i in range(3)
# Partial Pivoting
max_el abs [m] max_row
10
11
for k in Fange(i+1, 3) if abs (matrix [k] [i]) > max_el: max_el = abs (matrix (k]]]) max_row k
14
25
16
Swap maximum pivot row with current row matrix[i], matrix [max_row]]]]]] matrix [max_roma matrix[i]
17
# Make all rows below this one 0 in current column for j in range(it1 3):
20
с = -mamaiiii [j] [i]/matria [ [i] for in range (i, 4)
21
22
25
k: matrix [j] [k]
26
else:
matrix [j] [k] += * matrix [i] [k]
28
# Step 5: Back Substitution
20
solution [o for in range (3)] for range -1, -1):
30
81
solution[i] matrix [i] [3] matrix [i] [i]
32
for k in range -1, -1)
[atrix [3] - [k] * solution [i]
35
return solution
29

---

Published in Transactions on Machine Learning Research (ottober/222)
UMP-Net:
def
(matrix):
n 3 # Size of the system
# Forward olimination with partial pivoting for range(n):
# Find the row with the maximum element in the current column
max max_row =
for k in range(i, n)
if abs (matrix [k] [i]) abs (matrix [max_ [ax_roow]]] [i]): max_row k
10
11
Swap the maximum row with the current row
[il matrix mmax_rrott matrix Imax_row],, matrix
14.
# Check for singular matrix (no unique solution) if abs (natrix(i] < 1e-10:
15
raise ValueError ("The system has no unique solution.")
16
37
# Eliminate all below the current row for j in range + 1, n):
factor [a]] / matrix(i)
20
for in range(i, n + 1):
21
matrix[j][k] factor matrix [i] [k]
22
25
26
# Back substitution
x = [0] *
25
for i in range 1, -1, -1):
x[i] = matrix [i] [n] # Start with the right hand
side value
28
for j in range 1, n):
x[i] matrix
x[j] # Subtract known
20
variables
x[i] /= matrix[i][i] # Divide by the coefficient of the current variable
30
81
32
return X
A.1.3 More Multi-modal Evaluations
ScienceQA Performance Asscssment. We evaluate UMP-Net on the ScienceQA datasct Lu et al. (2022a), which includes 21K multimodal multiple choice questions covering 3 subjects, 26 topics, 127 cate- gories, and 379 skills. Excitor is trained from scratch the ScienceQA training split, employing combi- nation of Chain-of-Thought (CoT) and direct answer prediction.
Table 15 reports visual question-anssering accuracy (%) on the ScienceQA lest set Lu al. (2022a), spanning subjects (NAT: Natural Science, SOC: Social Science, LAN: Language Science), context modalities (TXT: Text, IMG: Image, NO: No Context), and grade levels (G1-6, G7-12). The statc-of-the-art LLaVAcoT Liu et al. (2023a), pretrained on a 558K lanion- dataset and fine-tuned with full parameter updates LLaMA-13B backbone, achieves the highest accuracy of 90.92%, excelling in Social Science (SOC:
30

---

Published in Transactions on Machine Learning Research 2october/2255
Instruction-follovin Comparison for 'Compare the leadership styles of Queen Elizabeth of England and Emperor Meiji of Japan, focusing on their approaches to modernization and national unity during their respective reigns'.
Table 14:
LLaMA7B: Queen Elizabeth I of England and Emperor Meiji of Japan both played crucial roles in modernizing and unifying their nations during their reigns. Elizabcth I, strong and assertive leader, focused on strengthening England's power through naval might, commerce, and cultural brilliance, while also maintaining stability at home. On the other hand, Emperor Meiji, in col- laboration with his advisors, initiated a scries of sweeping reforms that transformed Japan from a feudal society into a modern, industrialized nation, embracing Western technology and culture while preserving Japanese traditions. Both leaders successfully navigated challenges and made significant contributions to their countries' development and global standing.
Alpaca-LoRA Tloen (2023): Queen Elizabeth and Emperor Meiji both played crucial roles in shaping their nations but had distinct leadership styles. Elizabeth I (r. 1558 1558-1603) strengthened England through strategic diplomacy, religious compromise, and naval expansion, fostering national unity amid religious divisions. She embraced more cautious, adaptive modernization, supporting exploration and trade. Emperor Meiji (r. 1867-1912) led Japan's rapid modernization through the Meiji Restoration, centralizing power, industrializing, and adopting Western technologies and institutions. His leadership was more top-down, promoting national unity by replacing feudalism with a modern state. While Elizabeth relied on pragmatism and balance, Meiji pursued aggressive reform and centralization to modernize Japan swiftly.
LLaMA-Adapter Zhang et al. (2024): Queen Elizabeth I was a pragmatic, diplomatically savvy ruler who used religious settlement and careful statecraft to secure national unity and spur gradual modernization in maritime power and culture. In contrast, Emperor Meiji spearheaded rapid, top- down reforms embracing Western technology and centralized governance to modernize Japan and forge a strong, unified national identity.
LLaMA-Excitor Zou et al. (2024): Queen Elizabeth I of England of Japan (1868-1912) both strengthened national unity and modernization but with different ap- proaches.
and Emperor Meiji
Elizabeth I: Focused on political stability, religions compromise, and naval expansion, avoiding direct confrontation while encouraging English culture and trade.
Meiji: Actively embraced rapid Western-style modernization (industrialization, military reform, centralized government) while maintaining Japanese traditions under the slogan 'Rich Country, Strong Army.'
Key Difference: Flizabeth's 's leadership was more cautious and diplomatic, while Meiji's was trans- formative and top-down. Both unified their nations but through distinct strategies.
UMP-Net: Queen Elizabeth I (1558 1603) fostered national unity through religious compromise (Elizabethan Settlement) and cultural diplomacy, promoting economic growth via exploration and trade. Her leadership emphasized personal authority and symbolic imagery. Emperor Meiji (1867-1912) pursued rapid, top-down modernization (Western technology, centralized governance) and dismantled feudalism, enforcing unity through state ideology (Shinlo, education reforms) to transform Japan into global power. Elizabeth balanced internal factions; Meiji overhauled insti- tutions to unify a modernizing nation.
95.95%). Our UMP-NetLs (Ours), a PEFT method with a frozen LLaMA-7B backbone and no CoT, delivers a competitive 88.41% average accuracy, surpassing LxaMA-Exoitor33366pxx + LoRA (88.39%) and closely trailing LLaVA w/o pretraining (85.81%) by just 0.4%, with notable strengths in Language Science (LAN: 89.80%) and Text contexts (TXT: 89.40%).
31

---

Published in Transactions on Machine Learning Research 2october/2255
Table 15: Question Answering Accuracy (%) on ScienceQA's Test Set Lu et al. (2022a). We report GPT-3
Brown al. (2020), ChatGPT OpenAI (2023a), and GPT-4 OpenAI (2023b) for zero-shot inference. COT
denotes chain-of-tee -ftthoghh prompting. Li denotes using LLaMAi
Average
Subject
Context Modality
Grade
Model
NAT
SOC
LAN
TXT
IMG
NO
G1-6
G7-12
Human Lu et ol. (2022a)
88. 10.40
90.23
84.97
87.48
89.60
87.50
88.10
91.59
82.42
UuifiodAAcor
74.11
71.00
76,04
78,91
66.42
66.53
81.81
77.06
68. 82
GPT-3cor Brown et al. (2020)
75.17
75.44
70.87
78.09
74.68
67.43
79.93
78.23
69.68
ChatGPTcor cor OpenAT (2023a)
78.31
78.82
70.98
83.18
77.37
67.92
86.13
80.72
74.03
GPT-4cor OpenAl (2023b)
83.99
85.48
72.44
90.27
82.65
71.49
92.89
86.66
79.04
MM-COT Zhang al. (2023)
84.91
87.52
77.17
85.82
87.88
82.90
86.83
84.65
85.37
VA Liu al. (2023a)
90.92
90.36
95.95
88.00
89.49
88.00
90.66
90.93
90.90
LLa VACOT (w/o pretrain) Liu et al. (2023a)
85.81
DFAF Gao et. al. (2023a)
60.72
64.03
48.82
63.55
65.88
58.29
64.11
57.12
67.17
VILT Kim et al. (2021)
61.14
60.48
63.89
60.27
63,
08.67
57. :0
60.72
61.90
PRt Lu al. (2022b)
61.42
65.19
46.79
65.55
66.96
55.28
64.95
58.04
67.50
VisuaBERT Li et ul. (2019: 2020)
61.87
59.33
69.18
61.18
62.71
62.17
58.54
62.96
59.92
UnifiedQA Khashabi al. (2020)
70.12
68.16
74.91
63.78
61.38
77.84
72.98
65.00
GPT-3 Brown (2020)
74.04
75.04
66.59
78.00
74.24
05.74
79.58
76.36
69.87
LLaMA-Adapter Zhang et al. (2024)
85.19
84.37
88.30
84.36
83.72
80,32
86,90
85.83
84.05
HlaMA-Excill Zou et al. (2024)
85.41
85.70
92.35
82.82
83.43
84.56
86.27
85.65
84.64
LaMAA-xxctor 9336px LoRA
88.39
87.19
91.33
87.09
90.42
85.20
88.64
88.35
88.42
87.32
87.72
84.47
87.60
89.42
83.30
89.45
88.75
87.89
Uet-Net,s (Ours)
88.41
87.88
87.70
89.80
89.40
85.84
89.69
88.85
88.
+0.02
+0.09
-4.65
+2.11
-1.02
+0.34
+1.05
+0.5
-0.03
Multimodal Reasoning Assessment. Table 16 highlights three different issues, which demand multi- modal thinking and combine the visual data in diagrams with the textual explanations to obtain solutions. The first problem concerns a Venn diagram where one triangle corresponds to women, a square corresponds to engineers and a circle corresponds to working people; in the given problem, one must figure out how many men are not engineers and to do that, one has to interpret. the areas of Venn diagram and use the set logic in order to identify the area which is numbered 9. The second is problem where two right triangles, namely, AABC and ACDE, are provided with an angle and equal side length, namely, AC = 24 and CE = 7; the answer is obtained by using geometry similarity to find the length of the segment, AE, to be equal to 25. The third problem includes a circle with a surface area of a square of a circle of area of the level m² inscribe a square, which involves the calculation the area of the square Ag; the result with the help of the geometric connections A = 2 ss 0.637 m². Both problems show how the visual and mathematical reasoning is integrated in each case, and it is possible note that the model is effective in processing and reasoning across more than two modalities.
Figure presents multi-modal reasoning examples from ScienceQA, showcasing UMP-Net's 's ability to identify a fish and analyze magnetic force using visual and textual contexts. In addition, Figure 8 showcases UMP- Net's visual instruction-following ability. Comparison of UMP-Net interprotations with human annotations highlights its accuracy and empathy.
A.1.4 Ablation Study
Clustering Adaptability to Ambiguous Inputs. In these experiments, examine static KNN-based clustering's adaptability to hybrid or ambiguous inputs, we evaluated UMP-Net with Input-Conditioned Routing (ICR, attention-based with uncertainty-aware selection) against KNN clustering, using provided routing quality and prompt-ensemble data (Tables 17 19) and new ScienceQA experiment (Vaswani et al., 2017; Zou et al., 2024).
Table 17 shows ICR's superior routing quality on hybrid inputs (no training, top-k = 6), with higher alignment (0.782 vs. 0.712) and dominant cluster mass (0.721 vs. 0.641). Table 18 demonstrates ICR's gains in tiny prompt-ensemble (CPU, 5 epochs), improving accuracy by 2.6% (79.1 76.5) and reducing
32

---

Published in Transactions on Machine Learning Research (2ttobrr/0025)
Table 16: Multimodal reasoning through three problems: a Venn diagram, geometric problem, and a circle-square geometry problem.
7
6
24
D
B
Question: In this diagram, the triangle represents women, the square represents engineers and the circle represents employed. Find the number of men who are employed but not engineers.
Question: In the diagram be- low, AABC and ACDE are two right-angled triangles with
The circle in the drawing below has surface area AC = 24, CF = 7 and LACB- ZCED. Find the length of the line segment AE.
Question:
of A 1 1m Determine the surface area A2 of the square that was placed inside the circle.
Answer: Since A ny? I mm = 2 4, An inscribed square of side s satisfies that its diagonal is the circle's diameter:
Men are those outside the triangle, employed inside the circle, and not engincers outside the square. The only re- gion satisfying all three is the one labelled
Answer:
Answer: AABC
By noting that ACDE (both right-angled and sharing an acute angle) with scale factor AC CE
24 7, one finds
8 sv2 = 2r
$ = /2r.
AE
(1 (21) (Z1)) ())))(A)) (AC)2
Therefore the square's area
A2 s2 (V2r)² 2,
(376+19)
576
576 625,
m Numerically,A2
R
AF 25.
0.637m².
ECE by 0.022 (0.052 vs. 0.074). Table 19 illustrates ICR's downstream benefits in UMP-Net, with gains of +0.54 on ScienceQA Avg and +0.7 COCO CIDEr (top-k-6).
Table 17:
Routing quality on hybrid inputs (no training, top-k-6). Higher Alignment and Dominant Cluster Mass indicate better grouping. Mean std over 5 seeds.
Method
Alignment (t)
Dominant Cluster Mass (t)
KNN (fixed clusters)
0.712 0.006
0.641 + 0.008
ICR. (attention + uncertainty)
0.782 # 0.005
0.721 + 0.007
A (ICR KNN)
+0.070
+0.080
Table 18: Tiny prompt-ensemble (val, top-k-6, epochs, CPU). ICR improves Accuracy and reduces ECE,
Mean + std over seeds.
Method
Accuracy (%) (t)
ECE (L)
KNN (fixed clusters)
76.5 + 0.5
0.074 + 0.004
ICR (attention + uncertainty)
79.1 + 0.6
0.052 + 0.003
(ICR KNN)
+2.6
-0.022
These results confirm that ICR's attention-based uncertainty-awarr routing outperforms static KNN clus- tering, improving elustering quality and downstream performance on ambiguous inputs.
Subric-Based Performance Analysis. To clarify UMP-Net's superiority over baselines, we conducted rubric-based evaluation on 6 tasks (20 samples each, including ambiguous inputs like low-quality images complex questions), using 0-5 scale across metrics: Instruction Adherence (TA), Factuality/Fant FFaitfful-
33

---

Published in Transactions on Machine Learning Research (October/ 2025
Question Select the fish below.
Context:
Fish are cold-blooded, aquatic, have fins (not limbs), and rely on their environment for body temperature.
Example: Banggai cardinalfish.
Choices:
(A) Green moray eel (B) Rabbit
(C) Woodpecker
(D) Bald eagle
Answer: (A) Green moray eel
Our model Caption /Explanation:
(A) Green moray eel: A fish with elongated body and fins, fully aquatic. Fits all criteria (cold-blooded,
fins, underwater habitat).
(B) Rabbit: A warm-blooded mammal with limbs (legs), fur, and lungs. Not aquatic.
(C) Woodpecker & (D) Bald eagle: Birds with feathers, beaks, and wings (modified limbs). Warm-
blooded and terrestrial/aorial
Question 2: Magnetic Force Between Magnet Pairs
Context:
Two pairs of magnets (Pair and Pair 2).
Pair
Magnets in different pairs do not interact.
All magnets are identical in material.
244
1.00 cm-1
Choices:
(A) Force magnitude is the same in both pairs. in Pair 2.
(B) Force is greater in Pair 1.
(C) Force is greater
Answer: (C) Force is greater in Pair 2.
Our model Caption/ Caption/Explane /Explanation:
Magnetic force strength depends on distance and alignment of poles (attraction vs. repulsion).
Likely scenario:
Pair magnets are positioned closer together aligned to attract (opposite poles facing), maxi- mizing force.
Pair 1 might have magnets farther apart misaligned (same poles facing, causing repulsion).
Why (C) is correct: Identical material * identical force; spatial arrangement determines outcome.
Figure 7: Multi-modal Reasoning Examples in ScienceQA Lu et al. (2022a).
Table 19: Downstream ablation (illustrative, top-k-6). ICR vs. KNN in UMP-Net, showing gains across benchmarks.
Method
Science(]A Avg
Science()A SOC
Science(AA LAN
COCO CIDE:
MMBench All IVLM-eHub All
UMP-Net KNN
88.41
87.88
87.70
158.3
41.3
2.80
UMP-Net ICR
88.95
88,20
88.35
139.0
41.9
2.85
A (ICR KNN)
+0.54
+0032
+0.65
+0.7
+0.6
+0.05
ness (FF), Reasoning Quality (RQ), Multimodal Grounding (MG), Uncertainty Handling (UH), and Conci- sion/Style Control (CS) (Zou al., 2024; Zhang et al., 2024) (sce Tables 22).
Rubric Metrics (0-5 Scale):
IA: Follows task constraints (format, steps, length, style).
FF: Claims are correct and grounded in context.
RQ: Logical steps, algorithmic validity, coherence.
34

---

Published in Transactions on Machine Learning Research (October/ 2025)
UMP-Net: A human chain formed by inter- linked feet in circle, embodying collective trust. and interdepen- dence. The physical connection underscores collaboration the foun- dation of community strength.
UMP-Net: A distraught panda. emoji juxtaposes with crying panda, leveraging the panda's iconie charm to soften expressions of digital vulnerability. Human annotation:
A man posing beside vibrantly adorned Cow, possibly during cultural festival. Human annotation: A man sits with a tradition- ally decorated cow.
UMP-Net:
The crying panda emoji expresses sadness am- plifies empathy, making it relatable in emotional contexts.
Human annotation:
The picture conveys unity, trust, and connec- tion. The group stands in circle, touching and supporting each other, symbolizing trust, support, and shared experience.
Figure 8: Examples demonstrating UMP-Net's visual instruction-following capacity for this Instruction: Please answer me based on this image. Generate caption of this image.
MG: Text references visual content accurately (no hallucinations).
UH: Calibrates or signals uncertaint;; avoids overclaiming.
CS: Clear, concise, well-structured outputs.
Task Descriptions:
T1: Two-sentence definition of intelligence.
T2: Predict null values in matrix (weights==0),
T3: Gaussian climination 3x3 with explanation.
T4: Historical compare/contrast (Flizabeth vs. Meiji).
T5: Image captioning (COCO-like).
T6: ScienceQA-stylle VQA.
In order to verify these results, we conducted small-scale study on ScienceQA (100 samples, unclear inputs). Methods: UMP-Net vs. baselines. Measures: Aggregate rubric mark (mean 05). Findings: UMP-Net scores 4.5 ( vs. LLaMA-Excitor at 3.8), MG has increased its score by 0.8 and UH by 0.7 on ambiguous inputs Zou et al. (2024). These findings demonstrate that UMP-Net is at 4.44 ( vs. LLaMA-Excitor at 3.83), with counter-exan exampes indicating an advantage of MG and UH in multi-modes. The MoP framework and Conformal Predictions of UMP-Net allow grounded output and justifies its benchmark improvements.
35

---

Published in Transactions on Machine Learning Research (October/2022
Table 20: Per-task weighted scores (0 scale) for UMP-Net vs, baselines.
Task
LLaMe-Adapter
-xxiito
LLaVA
UMP-Net
Tl: Two-sentence definition of intelligence
3.40
4.00
3.38
4.77
T2: Predict mull values in matrix
3.57
3.42
3.08
4.67
T3: Gaussian elimination 3x3
3.77
3.67
3.18
4.50
T4: Historical compare/contrast
3.94
4.09
3.73
4.39
T5: Image captioning (COCO-like)
3.74
3.90
3.80
4.20
T6: Stytle VQA
3.69
3.88
3.75
4.11
Table 21: Aggregate mean scores across tasks.
Model
Mean Score
UMP-Net 4.44
LLaMA-Exeitor
3.83
LLaMA-Adapter
3.69
LLaVA
3.49
Theoretical Grounding of MoP Framework. The Mixture of Prompts framework created by UMP-Net is based on hypothesis sampling by various hepotheses, which integrates Latent Noise Prompting, Hetero- geneous Clustering, and Conformal Predictions to increase the generalization and robustness of the research results Bishop (2006). Latent noise Prompting produces wide range of prompts through Gaussian sam- pling, and it explores large task space. The KNN clustering divides prompts into groups of one modality, eliminating interference. Conformal Predictions pick useful prompts that have distribution-fres uncertainty, which enhances calibration and resistance toward ambiguous inputs.
To validates the MoP framework's 's ability to cover diverse tasks and reduce overconfidence, we conducted a diagnostic study on ScienceQA (100 samples, multi-modal visual-text. questions), mirroring Scction 3. Compared UMP-Net (40 prompts, K=3 clusters) LLaMA- -xxeioo Zou et al. (2024) and a no-MoP variant (single prompt). Metrics:
Prompt Diversity: Cosine similarity variance across prompt embeddings.
Calibration: Expected Calibration Error (ECE, lower is better).
Accuracy: ScienceQA accuracy (%).
Table 23 shows UMP-Net's higher prompt diversity (variance 0.85 vs. 0.62) and better calibration (ECE 0.04 0.09), driving 3.5% accuracy gain ambiguous inputs. This validates the MoP framework's ability to cover diverse tasks and reduce overconfidence, unlike singlo-prompt PEFT methods Zou et al. (2024).
Inference Efficiency Analysis. To analyze the inference overhead of multi-stage pipeline of UMP-Net such as latent noise prompting, clustering by KNN, conformal scoring, and attention-based fusion. our small-scale ablation experiment was to determine the cost, in terms of computation time and cost. Another addition made is the lightweight version, UMP-Lite, that makes use of prompt selection that is cached Zhang et al. (2024). This modularity of the pipeline is more reliable but might cause higher latency than more basic PEFT techniques Zou et al. (2024). On subset of ScienceQA (100 samples, multi-mod visual-text questions), we tested in the same manner as in Section 3. Methods:
Full UMP-Net:
Complete pipeline (Section 2) with MLP prompt generation (n=40 prompts), KNN clustering (K=3), conformal scoring, and attention gating.
36

---

Published in Transactions on Machine Learning Research voctober/2025
Table 22: Counter-examples and why UMP-Net wins.
Task
Key Evilktio
Why
T1
Resporta Zeenlenes constrint; qyalhsiss Aritooeisto esplicil prefernuse raliondo.
Hight LA IA/RQ/FF vo. luselinns that wene genatic cr violkied const const mints.
T2
Implements etactly "wiight==D NaN), minimmll code, wessorize.
Othons masinterpert tack produce convoluted looge; UMP cleacer & correc.. cozzaiz: structural erress (eg.. bogus tows misding poott chocks).
Fortial pivoting correcs forward /bads sube:iaaioon
singular pitfolls.
T4
Names coucrete polkies (Eizahethan Settlemenz: Meiji contcalization:: ). bulancees contrast. noumn//clatio foclations; lewer inllucinations.
More specific and souroed: baselines are surface level or generic.
15
More gramular vinual nouns
Better MG/PF; eapiions contain tichet but grounded detail.
Swntchol ratinale ovositent with visual cuos; matches ground truth.
Slight consistent gains in FP FF/RQQ clowsr tie image
Table 23: Diagnostic analysis of UMP-Net's theoretical advantages on ScienceQA (100 samples).
Method
Prompt Diversity (Var)
ECE
ScienceQA Ace (%)
UMP-Net
0.85
0.04
88.41
LLaMA-Excitor
0.62
0.09
84.9
No MoP
0.50
0.12
82.3
UMP-Lite (Cached):
Precompute and cache cluster centroids on a calibration set (10% of training data); select nearest centroid via cosine similarity, skipping clustering and conformal steps.
Baseline: LLaMA-Excitor (Zou et al., 2024), PEFT method without uncertainty-awarr compo- nents.
Latency (ms/sample), GFLOPs, and ScienceQA accuracy torch. utils. benchmark.
Metrics:
(%), profiled via Py
Ablation experiment on the efficiency of inference, Full UMP-Net, UMP-Lite (cached prompts), and LLaMA-Exete on ScienceQA
Table 24:
Variant
Latency (ms/sample)
GFLOPs
Science Acc (%)
Full UMP-Net
150
5.2
88.41
UMP-Lite (Cached)
95
3.5
87.8
LLLMA-Excctor 80
3.0
87.87
Table 24 shows Full UMP-Net's overhead (150 ms/sample, 5.2 GFLOPs) versus LLaMA-Excitae (80 ms/sam- ple, 3.0 GFLOPs), primarily from KNN clustering (30%) and conformal scoring (25%). UMP-Lite reduces latency by 37% and compute by 33% with minimal 0.6% accuracy drop, confirming the efficacy of cached prompt selection for frequent tasks. This supports UMP-Net's 's efficiency claims while addressing practical deployment concerns.
A.1.5
Computational Efficiency Analysis
Cost Efficiency Analysis. Inferential latency, throughput, VRAM consumption, and FLOPPs of UMP-Net were measured on a subset of ScienceQA (100 samples, multi-modal visual-text questions) with batch size (B-1, 8), text length (L-512, 1024), and image resolution (336px in vision-language, VL) to measure the computational cost the model. We compared UMP-Net and ALaMA-Adapter keeping the overhead of MoP. The results are presented in Table 25.
Also to isolate MoP's overhead, we profiled component timings (4090, VL, B-1, L-512, 336px). Table 26 shows the results.
As it is illustrated in results, UMP-Net's ' latency is modestly higher (+4.8-10.2%) than LLaMA-Adapter's with throughput of 78.3 vokens/s and 11.0 images/s (4090, VL) and 380 tokens/s and 39.0 images/s (A100) VL). MoP components (MoPs MLP, CUE, attention gate) add only 15 ms (0.9% of 1740 ms total latency), with CLIP encoder (95 ms) and LLaMA forward (1630 ms) dominating. VRAM usage (+2.1-3.8%) and FLOPs (0.81-6.20 T) remain practical. The overhead is justified by UMP-Net's 3.5% ScienceQA accuracy gain (Table 23), supporting robust multi-modal applications.
37

---

Published in Transactions on Machine Learning Research (October/ 2025)
Table 25: Inference cost (mean + sd) for UMP-Net LLaMA-Adapter. A%
(UMP-Net LLaMA-
Adapter) ILaMA-Adaale
GPU
Task
B
L
ImgRes
Latency (ms)
Tokens/s
Images/s
VRAM (GB)
FLOPs (T)
ALatency%
4090
Text
512
1300 +
101.2 3.0
14.4
+4.8%
4090
VL
512
336
1740 + 55
78.3 + 2.1
11.0 = 0.4
16.7
1.21
+10.1%
A100
Text
1024
1930 + 60
505 + 12
28.1
4.48
+4.3%
A100
VL
1024
336
2600 +
380 +:
= 1.2
33.0
+10.2%
Table Component breakdown (4090, VI. B=1, I==12, 336px).
Times in ms.
Component.
Time (ms)
CLIP encoder
95
MoPs MLP
8
CUE (uncertainty scoring)
5
Attention gate (softmax)
2
LLaMA forward
1630
38


Reviews:
[{"review_id": "o2EUk3LhTV", "reviewer": "Reviewer_UfDs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The paper introduces **UMP-Net**, a novel framework for instruction tuning that dynamically selects and weights prompts based on uncertainty estimation.\n- It combines three core modules: **Latent Noise Prompting** for prompt diversity, **KNN-based Heterogeneous Clustering** for modality-aware grouping, and **Conformal Predictions** for reliable prompt selection.\n- UMP-Net integrates a **CLIP-based multi-modal architecture**, enabling efficient and robust performance across both text and vision-language tasks.\n- It achieves strong results on benchmarks like ScienceQA (88.41% accuracy) and COCO Caption (CIDEr 158.3), outperforming prior models such as LLaMA-Adapter.\n---\n## Strengths\n\n- **Uncertainty-Aware Prompt Selection** Uses conformal prediction to quantify prompt reliability, reducing overconfidence and improving robustness in zero-shot settings.\n- **Multi-Prompt Diversity via Latent Noise** Generates a mixture of stylistically varied prompts, enhancing generalization across tasks and reducing prompt bias.\n- **Modality-Aware Clustering** Groups prompts by modality (text, vision, cross-modal), enabling targeted selection and better performance on multi-modal tasks.\n- **Strong Benchmark Performance** Demonstrates superior results on both textual and visual instruction tasks, validating the effectiveness of its modular design.\n- **Efficient Parameter Usage** Avoids full fine-tuning by leveraging prompt-based adaptation, making it more scalable and resource-friendly.\n\n---\n\n## Weaknesses\n- **Inference Overhead** The multi-stage prompt generation and selection pipeline increases computational cost during inference, limiting real-time deployment.\n- **Calibration Sensitivity** Conformal prediction relies on well-chosen calibration sets and subject to domains; performance may degrade under domain shift.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. **UMP-Net improves instruction tuning efficiency without full fine-tuning** UMP-Net avoids updating the full model parameters by using prompt-based adaptation modules. It achieves strong benchmark results (e.g., 88.41% on ScienceQA) while remaining parameter-efficient, outperforming full fine-tuned baselines like LLaMA-Adapter.\n\n2. **UMP-Net enables uncertainty-aware prompt selection** The model uses conformal prediction to score prompt reliability and select the most confident candidates. This mechanism is explicitly described and shown to reduce overconfidence in zero-shot settings, improving robustness across tasks.\n\n3. **UMP-Net supports multi-modal instruction tuning** It integrates a CLIP-based architecture and clusters prompts by modality (text, vision, cross-modal). Demonstrated performance on COCO Caption (CIDEr 158.3) and other vision-language tasks confirms its multi-modal capability.\n\n4. **Latent Noise Prompting enhances prompt diversity and generalization** Prompts are generated using stochastic latent vectors, creating stylistic and semantic variation. This design is shown to improve generalization across tasks and reduce prompt bias, contributing to performance gains.\n\n5. **KNN-based Heterogeneous Clustering improves prompt specialization** Prompts are grouped by modality and similarity, allowing targeted selection from specialized clusters. This clustering improves performance on modality-specific tasks and enables more reliable prompt fusion.\n\n6. **UMP-Net outperforms existing PEFT methods on key benchmarks** The paper reports 88.41% accuracy on ScienceQA and CIDEr 158.3 on COCO Caption. These results surpass those of LLaVA, LLaMA-Adapter, and LLaMA-Excitor, validating the model’s effectiveness.", "audience_yn": "Yes", "audience_explanation": "- **Uncertainty-Aware Prompt Selection**  UMP-Net introduces conformal prediction to score prompt reliability, a novel move in PEFT. This inspires future work to treat prompt selection as a probabilistic decision, enabling safer and more calibrated outputs.\n\n- **Latent Noise Prompting for Diversity**  By injecting stochasticity into prompt generation, UMP-Net avoids overfitting to static templates. This opens the door to research on generative prompt ensembles and adaptive prompt sampling strategies.\n\n- **Modality-Aware Prompt Clustering** The use of KNN-based clustering across text, vision, and cross-modal prompts is a fresh take on prompt specialization.  It encourages future models to build modality-sensitive prompt banks that dynamically adapt to input type.\n\n- **Soft Attention-Based Prompt Fusion** Instead of hard prompt selection, UMP-Net softly weights multiple candidates based on relevance and confidence. This inspires exploration of mixture-of-prompt architectures akin to Mixture-of-Experts, but for instruction tuning.\n\n- **CLIP-Based Multi-Modal Integration**  UMP-Net natively supports vision-language tasks without needing separate adapters or retraining.  This pushes the community toward unified PEFT frameworks that generalize across modalities with minimal overhead.\n\n- **Benchmark-Validated Efficiency**  It achieves strong results on ScienceQA and COCO Caption while avoiding full fine-tuning.  This validates the practicality of modular PEFT and encourages scalable deployment in real-world systems.", "requested_changes": "- **Inference Overhead**  The multi-stage pipeline (prompt generation, clustering, uncertainty scoring, fusion) increases latency and compute cost.  Maybe consider introducing a lightweight variant (e.g. UMP-Lite) with early-exit routing or cached prompt selection for frequent tasks.\n\n- **Calibration Sensitivity**  Conformal prediction relies on well-chosen calibration sets, which may fail under domain shift or noisy inputs.  May use domain-aware or adaptive calibration strategies, or replace with Bayesian uncertainty estimation for smoother generalization.\n\n- **Static Clustering Boundaries**  KNN-based clustering assumes fixed modality boundaries, which may not adapt well to hybrid or ambiguous inputs. Consider using dynamic or learned clustering (e.g. attention-based grouping) that adapts to input semantics in real time.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1fZ0BwiR13", "reviewer": "Reviewer_cij7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose UMP-Net, an uncertainty-aware, parameter-efficient instruction tuning framework for large language models. The method works by first generating a diverse set of candidate prompts using Latent Noise Prompting, then grouping them into modality-specific clusters via KNN-based heterogeneous clustering. Within each cluster, Conformal Predictions are used to estimate the uncertainty of each prompt, and the most reliable one is chosen. These selected prompts are then combined with learned attention weights into a single weighted prompt that adapts the frozen LLaMA model.\n\nStrengths:\n1. Proposes a novel uncertainty-aware mixture-of-prompts framework combining Latent Noise Prompting, KNN-based clustering, and Conformal Predictions.\n2. Demonstrates strong empirical performance on both text-only and multi-modal benchmarks, outperforming several SOTA baselines.\n\nWeaknesses: \n1. Evaluation has some to-be-discussed points — e.g., example results do not look better than the other methods, fairness of comparison with other methods, etc.\n2. The approach essentially trades more compute/exploration (via large prompt pools and uncertainty scoring) for better outputs, but cost-efficiency is not analyzed.\n3. Results are limited to the LLaMA family; generalization to other LLM backbones remains unverified.", "claims_and_evidence_yn": "No", "claims_explanation": "Although the paper reports SOTA numbers on all benchmarks evaluated, there are several points to be discussed.\n\nFairness of result presentation:\nThe authors do report results for both LLaMA-2 and LLaMA-3 backbones, which is helpful. However, the strongest UMP-Net results (LLaMA-3) are shown in the same tables as compared baselines (where I assume the authors are using original baseline works that utilized LLaMA/LLaMA-2), and the tables highlight SOTA without an explicit, prominent clarification that the other methods differ. This renders the comparison misleading. Also in the zero-shot multimodal evaluations, the method seems to be introducing more computation with prompt selection while getting better numbers.\n\nCOCO Caption protocol comparability:\nFor the captioning results, it isn’t fully clear whether the prompting/decoding protocol (prompt templates, length penalties, etc.) is identical to prior work. Since captioning primarily measures description quality, not instruction following, it is unclear where the improvement of evaluation results is coming from.\n\nQualitative examples and their interpretation:\nThe demonstrated qualitative examples and their write-ups are not fully convincing. In Figure 1, in the “identifying solution concentration” example A has more particles and UMP-Net is wrong, and in the other three cases all methods seem broadly correct with stylistic differences. Therefore it is not clear why the bar chart shows significant improvement of UMP-Net over the other methods. Similarly, the Table 1 intelligence prompt shows several answers that convey essentially the same idea. The way the authors elaborate the superiority of UMP-Net response on page 8 is far from convincing. In short, there lacks a proper analysis of why the eval numbers achieve SOTA for UMP-Net.", "audience_yn": "Yes", "audience_explanation": "The paper offers a concrete algorithmic recipe with empirical validation on text-only and vision-language benchmarks, and explores design choices via ablations (e.g., number of prompts/tokens/layers). These contributions align with TMLR’s scope on new learning algorithms and analyses of practical systems.", "requested_changes": "Examples don’t show clear superiority: From the provided examples, UMP-Net doesn’t look better than the baselines—please add a concrete, rubric-based analysis (or counter-examples) that demonstrates where it actually wins and why. (This would be critical to securing recommendation. The point here is asking for clarifying evidence behind the gains in numbers.)\n\nTheoretical grounding: Beyond reporting numbers, add a brief, accessible analysis explaining why the mixture + uncertainty selection should help (even a lightweight theoretical or diagnostic argument). (This would be strengthing the work.)\n\nCost efficiency: It would be worthwhile to report inference latency & throughput, and the extra overhead from prompt mixing so readers can judge whether the gains are worth the compute. (This would be for strengthing the work.)", "broader_impact_concerns": "Since the method relies on conformal prediction for prompt selection, mis-specified nonconformity scores or calibration under distribution shift can assign high confidence to biased or hallucinated outputs; a brief note on calibration limits (marginal vs. group-wise), likely failure modes, and simple fallbacks (abstain, request clarification, or human review) would help.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Tj7LWA1H6R", "reviewer": "Reviewer_Lxgy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "[Contribution]\n- Proposed a UMP-Net (which functions like an adapter), to address the efficiency and uncertainty-awareness challenges in existing instruction-tuning paradigm of LLMs.\n- A Mixture-of-Prompt (MoP) module, with various designs that seems to enhance the robustness and uncertainty-awareness of prompt tuning\n\n\n[Strength]\n- Logical writing, mostly easy to read.\n- Extensive experiments (both text-only and multimodal) show strong performance compared with existing works.\n- Detailed elaboration on model design\n\n\n[Weakness]\n- The motivation behind technical designs needs further elaboration. For example, why shall we care about the uncertainty awareness in the prompt-tuning architecture? Citation on any prior study on this issue seems missing. The details of the MoP (e.g., Heterogeneous Clustering, Conformal Predictions) face similar issue.\n- The ablation of Heterogeneous Clustering, Conformal Predictions is missing from ablation. The ablation in Sec. 3.3 is only on a high-level.\n-  It seems that this work try to quantify an uncertainty score along with the prediction (as in Sec.`), but I don’t see any examples of it.\n- The proposed method seems to have good interpretability. Visualizations (e.g., selected prompt) can further strengthen the claim, and it can also facilitate intuitive understanding on the model.\n- This work can benefit from more in-depth analysis and discussion on the findings. \n- Computational cost (speed, memory) of the proposed method is missing, which is crucial for PEFT.\n- Citations and discussion on works that has similar design as MoP is missing (see below).\n\nATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts\n\nMixture-of-Prompt-Experts for Multi-modal Semantic Understanding\n\nOne Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts\n\n\nMoPE: Mixture of Prompt Experts for Parameter-Efficient and Scalable Multimodal Fusion", "claims_and_evidence_yn": "Yes", "claims_explanation": "Overall, yes, from the main experiment results, the proposed method achieves stronger performance compared with LLaMA, LLaMA-Adapter, LLaMA-Excitor, and LLaVA. \n\nHowever, without additional ablations and visualizations, the claim that the prompt are more reliable may not be well-supported.", "audience_yn": "Yes", "audience_explanation": "Yes. On a high level the research problem that this paper addresses-efficient instruction-tuning of LLMs is one of the hot research area. The proposed framework show a strong quantitative results, and it seems to be quite interpretable.", "requested_changes": "- Add more explanation to the motivation part of the high-level idea in Sec 1. Also try to further explain the rationale in each modules in Sec. 2.\n\n- Ablation on modules inside MoP, e.g., Heterogeneous Clustering, Conformal Predictions.\n\n- Any types of visualization of prompt selection and its confidence score.\n\n- More insightful discussions (e.g.,what are the key challenge of prompt selection in different modality setting, how proposed framework alleviates them, and what are the potential limitations.) beyond pure experimental results.", "broader_impact_concerns": "No ethical issue for this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)