# Application of Multimodal AI to Aid Scene Perception for the Visually Impaired
## 3 Results
High accuracy was observed for straightforward binary tasks (e.g., indoor/outdoor classification), whereas tasks requiring the interpretation of nuanced visual cues (e.g., surface visibility, crosswalk detection) exhibited significantly lower precision, recall, and F1 scores. These discrepancies highlight both the potential and the current limitations of multimodal AI models in real-world assistive navigation contexts. While the results are presented in terms of standard classification metrics, it should be noted that this approach primarily captures categorical accuracy and prompt conformity. Broader aspects such as semantic validity, reasoning consistency, or partial correctness in generative outputs remain beyond the scope of this evaluation but are important for real-world assistive use. For metric computation, it was assumed that each classification task had two or three predefined possible responses. Although the models always returned exactly one of these options, the first response listed in the prompt (e.g., Indoor present: Yes) was treated as the positive class (true), while all alternative responses were considered as negative classes (false). This assumption enabled the application of standard binary classification metrics (precision, recall, F1 score, and accuracy) to each category individually, ensuring consistent evaluation across all models. On a system with an Intel Core i9 (Intel, Santa Clara, CA, USA) and NVIDIA RTX 3060 (NVIDIA, Santa Clara, CA, USA), the models processed each image in 2-10 s--a speed comparable to the performance of commercial online models. Additionally, the response speed of the models was evaluated when implemented on a portable device. For this purpose, a Raspberry Pi 5 platform equipped with 8 GB of RAM was used. In this case, after launching the model, the response time was only slightly dependent on the image resolution and ranged from approximately 1 to 5 min, \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Class** & \multicolumn{4}{c}{**LLaVA-Llama3**} & \multicolumn{4}{c}{**MiniCPM-V**} \\ \cline{2-9}  & **Prec.** & **Rec.** & **F1** & **Acc.** & **Prec.** & **Rec.** & **F1** & **Acc.** \\ \hline Indoor scene & 0.52 & 1.00 & 0.69 & 0.76 & 0.66 & 1.00 & 0.79 & 0.86 \\ Stairs visible & 0.98 & 0.71 & 0.83 & 0.90 & 1.00 & 0.93 & 0.96 & 0.98 \\ Lighting: Dark & 0.24 & 1.00 & 0.38 & 0.40 & 0.48 & 0.95 & 0.64 & 0.80 \\ Illumination: Artificial & 0.40 & 0.09 & 0.15 & 0.89 & 1.00 & 0.05 & 0.09 & 0.90 \\ Doors present & 0.00 & 0.00 & 0.00 & 0.77 & 0.25 & 1.00 & 0.40 & 0.31 \\ Surface even & 0.68 & 0.90 & 0.78 & 0.70 & 0.91 & 0.66 & 0.76 & 0.77 \\ Surface visible and empty & 0.59 & 0.78 & 0.68 & 0.56 & 0.59 & 1.00 & 0.74 & 0.59 \\ Crosswalk visible & 0.11 & 1.00 & 0.21 & 0.49 & 0.09 & 1.00 & 0.17 & 0.37 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Summary of precision, recall, F1 score, and accuracy for LLaVA-Llama3 and MiniCPM-V models. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline
**Class** & \multicolumn{4}{c}{**LLaVA-13b**} & \multicolumn{4}{c}{**LLaVA**} \\ \cline{2-9}  & **Prec.** & **Rec.** & **F1** & **Acc.** & **Prec.** & **Rec.** & **F1** & **Acc.** \\ \hline Indoor scene & 0.93 & 1.00 & 0.97 & 0.98 & 1.00 & 0.96 & 0.98 & 0.99 \\ Stairs visible & 0.95 & 0.89 & 0.92 & 0.95 & 0.00 & 0.00 & 0.00 & 0.67 \\ Lighting: Dark & 0.27 & 1.00 & 0.43 & 0.50 & 0.49 & 0.82 & 0.61 & 0.80 \\ Illumination: Artificial & 0.11 & 1.00 & 0.19 & 0.15 & 0.19 & 0.95 & 0.32 & 0.58 \\ Doors present & 1.00 & 0.14 & 0.25 & 0.80 & 0.52 & 0.26 & 0.35 & 0.77 \\ Surface even & 0.79 & 0.09 & 0.16 & 0.46 & 0.79 & 0.90 & 0.84 & 0.80 \\ Surface visible and empty & 0.00 & 0.00 & 0.00 & 0.41 & 0.54 & 0.54 & 0.54 & 0.46 \\ Crosswalk visible & 0.00 & 0.00 & 0.00 & 0.93 & 0.21 & 0.43 & 0.29 & 0.86 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Summary of precision, recall, F1 score, and accuracy for LLaVA-13b and LLaVA models. depending on the model used. This lengthy response time effectively precludes the use of local models for real-time scene classification.