## 6 Greedy Prompt Engineering Strategy  
This section describes the greedy prompt engineering strategy (Greedy PES), which is designed to identify and apply optimal prompt engineering techniques for different MLLM deployment environments, including the various MLLM models and benchmark datasets discussed in the previous sections.  
In addition, the RAG approach was extended by integrating it with CoT, ToT, and SSR, whereby external information is retrieved and reformulated based on each respective reasoning strategy. These variants are denoted as R(C), R(T), and R(S), respectively.  
The greedy prompt engineering strategy (Greedy PES) aims to determine the optimal combination of MLLM models and prompt engineering techniques for each dataset by identifying the highest achievable performance across all possible prompt engineering (PE) combinations. To formalize this, let \(d\) represent a dataset, \(p\) a prompt engineering technique, \(e\) an evaluation metric, and \(m\) an MLLM model. The evaluation score derived from these parameters is denoted as \(S_{d,e}(m,p)\). Furthermore, the weight assigned to each evaluation metric is defined as \(w_{d,e}\), which accounts for the varying dynamic ranges of different evaluation metrics to prevent imbalance when aggregating scores. Additionally, these weights reflect the relative importance of each metric in assessing model performance.  
The applied prompt engineering techniques are represented using the following abbreviations:  
B: base, I: ICL, C: CoT, S: SSR, T: ToT;  
R(B): basic RAG, R(C): CoT-based RAG, R(S): SSR-based RAG, R(T): ToT-based RAG.  
\begin{table}
\begin{tabular}{c c c c} \hline
**Dataset** & **Task** & **Category** & **Description** \\ \hline MSCOCO & Caption & General & \begin{tabular}{c} Image and text matching, \\ caption-based retrieval \\ \end{tabular} \\ \hline Flickr30k & Caption & General & \begin{tabular}{c} Generating natural language \\ descriptions for images \\ \end{tabular} \\ \hline Nocaps & Caption & General & Captioning novel objects and scenes \\ \hline ScienceQA & VQA & Science & \begin{tabular}{c} Evaluating scientific knowledge and \\ reasoning \\ \end{tabular} \\ \hline MathVista & VQA & Math & \
\begin{tabular}{c} Assessing mathematical reasoning \\ with visual context \\ \end{tabular} \\ \hline CVBench & VQA & General &
\begin{tabular}{c} Evaluating visual understanding \\ through QA tasks \\ \end{tabular} \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of benchmark datasets for image–text and visual reasoning tasks.  
Then, the objective is to identify the MLLM model \(m\) and prompt engineering technique \(p\) that maximize the aggregated evaluation score \(S_{e}(m,p)\) across multiple evaluation metrics. This can be formulated as the following optimization equation:  
\[m^{*},p^{*} =\arg\max_{m,p}\{\sum_{e\in E}w_{d,e}\cdot\mathcal{S}_{d,e}(m,p)\ \},\] (11) subject to \[d\in\{\text{MSCOCO},\text{Flickr30k},\text{Nocaps},\text{ ScienceQA},\text{MathVista},\text{CVBench}\},\] \[p\subset\{\text{B},\text{I},\text{C},\text{S},\text{T},\text{R} (\text{B}),\text{R}(\text{C}),\text{R}(\text{S}),\text{R}(\text{T})\},\] \[m\in\{\text{Llama},\text{Pixtral},\text{Phi},\text{Qwen}\},\] \[E=\{\text{BLUE},\text{ROUGE},\text{METEOR},\text{S-BERT},\text{ Mover},\text{CIDEr}\}.\]  
The optimal MLLM model \(m^{*}\) and the optimal prompt engineering technique \(p^{*}\) may vary depending on each dataset \(d\).