# Application of Multimodal AI to Aid Scene Perception for the Visually Impaired
## 3 Results  
The results are presented in Tables 1 and 2, which show the best and worst outcomes for each class, based on five test runs for each model. The full results are provided in Supplementary Material Dataset S1. During the evaluation, predefined keywords from the prompt were searched for within the model's responses. If a given word appeared in the response, it was considered that the model correctly classified the image within the specified scope. Generative models, including commercial ones, frequently deviated from the expected prompt format, sometimes returning alternative data structures (e.g., JSON instead of XML) or paraphrased responses. These inconsistencies reflect a broader limitation of current generative AI: the difficulty in producing reliably structured outputs in response to constrained queries. While this challenge can sometimes be mitigated through prompt refinement, it underscores the need for robust output post-processing or standardized response protocols when deploying such models in assistive applications. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Class** & **LLaVA:13b** & **LLaVA** & **LLaVA-Llama3** & **MiniCPM-V** \\ \hline Indoor/outdoor & 0.990 & 0.996 & 0.765 & 0.864 \\ Stairs & 0.948 & 0.671 & 0.920 & 0.977 \\ Lighting & 0.502 & 0.808 & 0.413 & 0.822 \\ Illumination & 0.136 & 0.474 & 0.737 & 0.756 \\ Doors presence & 0.798 & 0.784 & 0.765 & 0.329 \\ Surface evenness & 0.484 & 0.812 & 0.696 & 0.798 \\ Surface visibility & 0.296 & 0.488 & 0.498 & 0.587 \\ Crosswalk & 0.934 * & 0.877 & 0.540 & 0.371 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Accuracy of scene analysis using multimodal AI models— best results. *—despite five independent runs on the image set, and despite the test set containing 14 crosswalks, the model never detected a crosswalk, not even as a false positive. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Class** & **LLaVA:13b** & **LLaVA** & **LLaVA-Llama3** & **MiniCPM-V** \\ \hline Indoor/outdoor & 0.971 & 0.991 & 0.737 & 0.859 \\ Stairs & 0.934 & 0.671 & 0.901 & 0.972 \\ Lighting & 0.483 & 0.784 & 0.394 & 0.798 \\ Illumination & 0.117 & 0.451 & 0.657 & 0.737 \\ Doors presence & 0.784 & 0.765 & 0.761 & 0.305 \\ Surface evenness & 0.460 & 0.779 & 0.643 & 0.765 \\ Surface visibility & 0.272 & 0.451 & 0.441 & 0.587 \\ Crosswalk & 0.934 * & 0.859 & 0.492 & 0.366 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Accuracy of scene analysis using multimodal AI models—worst results. *—despite five independent runs on the image set, and despite the test set containing 14 crosswalks, the model never detected a crosswalk, not even as a false positive. The visualization of maximum and minimum accuracy values from Tables 1 and 2 is shown in Figure 4.  
It is also worth noting that due to the nondeterministic nature of generative AI, each execution of the same query (even with identical images) can yield different results. Identical input data still resulted in varying outputs across runs, highlighting challenges in ensuring consistency and reproducibility. The evaluation results presented in Tables 3 and 4, and in Figure 5 indicate substantial variability in the classification performance of the analyzed multimodal AI models across different scene analysis categories. While models such as MiniCPM-V demonstrated high effectiveness in structured, visually salient tasks like stair detection and surface assessment, others, such as LLaVA-Llama3, exhibited greater difficulty in handling semantically complex or visually subtle attributes, such as illumination conditions and door presence. Figure 4: The visualization of maximum and minimum accuracy values from Tables 1 and 2. Figure 5: Visualization of precision, recall, F1 score, and accuracy metrics for four multimodal models, based on data from Tables 3 and 4. Overall, none of the evaluated models achieved uniformly high performance across all categories.