# Application of Multimodal AI to Aid Scene Perception for the Visually Impaired
## 1 Introduction
### Obstacle Detection and the Role of the Ground Plane  
The ground plane serves as a key reference for obstacle detection in assistive navigation. Correct estimation of the ground plane enables the differentiation of walkable and non-walkable areas, reducing false detections. The accuracy of obstacle detection is highly dependent on the quality of ground plane estimation, as errors in its determination can result in misclassification of obstacles or failure to detect potential hazards. Common approaches to ground plane estimation include the Hough transform in disparity space (U-V disparity), which extracts dominant planar structures but is sensitive to roll angle variations [10,11]; normal vector clustering, which groups surface normals for robust plane fitting, as implemented in the Sound of Vision project [12]; RANSAC [13] and least-squares methods, which are used for robust plane estimation from point cloud data; and Singular Value Decomposition (SVD), which provides optimal plane fitting in noisy environments. A common feature of these algorithms is that their use is possible only when it is certain that a ground plane is visible in the scene. Referring to the \"ground plane\" is a simplification mainly relevant indoors with flat floors. Outdoors, uneven surfaces like cracks, curbs, and vegetation can be misidentified as obstacles by recognition algorithms. Lighting conditions impact 3D scene reconstruction and depth estimation accuracy, requiring careful parameter selection for obstacle detection. Moreover, the type of camera sensor used (e.g., stereo, LiDAR, structured light) significantly influences the quality of the 3D reconstruction, which in turn affects the optimal choice of parameters for segmenting obstacles and ground planes [14]. Camera imperfections, such as lens contamination or sensor degradation, can further affect vision-based detection. To effectively use the estimated ground plane for obstacle detection, it is necessary to define a distance threshold that separates points belonging to obstacles from those that lie on the ground. In practice, this threshold is often selected empirically [15; 16]. For instance, in [15], a threshold of 10 cm was used, while in [16], the threshold was set to 15 cm. It is important to note that these values were determined for relatively regular and flat surfaces such as indoor floors or pavements. These studies did not account for more irregular terrains such as grass, gravel, or uneven natural ground. Therefore, the choice of an appropriate threshold should consider the type of depth sensor used, the characteristics of the scene, and environmental conditions. A threshold that is too low may result in false positive obstacle detections, whereas a threshold that is too high can cause real obstacles to be overlooked. Three-dimensional scene analysis faces significant challenges due to the inherent complexity of real-world environments, which feature diverse object shapes, varying appearances influenced by lighting and viewpoint, and frequent occlusions [14; 17]. Moreover, the scarcity of annotated 3D datasets, especially those reflecting diverse indoor and outdoor scenarios, limits the ability to train robust and transferable models [14]. This limitation is particularly critical for traditional machine learning and computer vision models, which require extensive high-quality supervision and do not inherently possess generative or self-adaptive capabilities. Domain-specific differences, such as the variation in architectural styles, materials, and environmental dynamics, further hinder model generalization. Addressing these challenges requires ongoing progress in efficient, reliable methods. Priorities include managing limited annotated data, reducing the computational demands of 3D processing, and improving model generalization across diverse conditions. This creates an opportunity for the use of generative AI and the application of multimodal models for image description and understanding.