# Application of Multimodal AI to Aid Scene Perception for the Visually Impaired
## 2 Materials and Methods  
The proposed method integrates conventional depth sensing with AI-based scene interpretation to improve the reliability and adaptability of assistive vision systems. In contrast to static parameter selection, this approach leverages multimodal generative models to dynamically analyze the recorded visual data and adjust the processing pipeline according to the current scene characteristics. This adaptive framework is designed to enhance ground plane estimation, improve obstacle detection, and ensure more effective information presentation to the user. The key components of the system include scene type recognition, evaluation of image quality and depth data consistency, and the selection of segmentation and classification parameters tailored to specific environmental conditions. The concept of the AI-assisted algorithm is illustrated in Figure 1. It begins with AI-based scene type and parameter detection, where artificial intelligence analyzes input data to classify the environment and extract relevant parameters. This process may involve image analysis and description to determine the structure and characteristics of the scene. By understanding these factors, the system can optimize further processing steps based on the specific context. Following scene analysis, the extracted parameters inform subsequent processing steps, including segmentation, object detection, and 3D reconstruction. Scene information guides parameter selection for segmentation and object detection algorithms and informs how this information is conveyed to a blind user. For example, this involves selecting thresholds to separate the ground from obstacles, evaluating whether the ground plane can be estimated from the scene and assessing the reliability of the depth map based on lighting conditions. Traditionally, image classification required the preparation of a comprehensive dataset followed by the training of conventional neural networks to assign images to predefined categories. The primary advantage of this approach was the unambiguous classification of an image into a specific class. However, a significant drawback was the necessity of curating extensive training datasets that accounted for all possible variations within a given category. With the advent of generative artificial intelligence (genAI), which enables the creation of novel and unique content, models can now analyze image content without the need for prior task-specific training. Nevertheless, their effectiveness is highly dependent on the  
Figure 1: The idea of using AI-based systems to determine parameters for 3D scene segmentation algorithms. quality of prompt engineering, where carefully crafted prompts are designed to extract relevant information. Despite their advanced capabilities, these models are still constrained by the data they were originally trained on, meaning they may not generalize well to images that differ substantially from their training examples. A key advantage of multimodal AI models is their ability to generate scene descriptions and provide contextual interpretations. They can predict a scene's location, lighting conditions, time of year, and whether it is indoors or outdoors. Additionally, they support multi-class classification in a single prompt, recognizing multiple attributes at once, such as lighting type or the presence of objects like stairs or crossings. This greatly enhances their flexibility and applicability in image analysis. Therefore, it was decided to evaluate the effectiveness of multimodal models in identifying scene parameters that can optimize segmentation for obstacle detection. The following models were tested: LLaVA13b, LLaVA [22; 23; 24], LLaVA-Llama3 [25] and MiniCPM-V [26]. Specifically, the focus was on identifying objects that pose challenges for an assistive navigation system designed for visually impaired individuals in autonomous movement. To achieve this, four multimodal AI models were selected based on their ability to run on a standard PC without relying on commercial cloud-based solutions.