# Application of Multimodal AI to Aid Scene Perception for the Visually Impaired
## 2 Materials and Methods
This ensures compatibility with low-power and resource-constrained hardware platforms, such as Raspberry Pi or embedded systems, which are commonly used in mobile assistive devices. Additionally, the selected models are freely available, which facilitates transparency, reproducibility of results, and further customization. Their high performance and flexibility for local deployment, as recognized by the machine learning community, make these models a practical and scalable choice for real-world assistive applications. To evaluate the effectiveness of image classification into predefined categories, a dataset of approximately 200 images was collected using a stereovision system, which serves as an information source in an assistive system for visually impaired individuals. These images were gathered during the testing phase of the system's functionality in real-world conditions. The dataset is diverse and includes both indoor and outdoor scenes, captured during nighttime and sunny daytime as well as across different seasons (Dataset S1). An application was developed (Figure 2) to classify each image into the selected categories. The selected images from this dataset are presented in the Figure 3.  
Figure 2: Application for annotating image datasets. The selected options are saved in JSON format and later compared with the responses generated by AI models. Subsequently, a prompt was prepared, and each image was analyzed using three multimodal models. The prompt used for image analysis is presented below. Analyze the provided image and determine the most appropriate options for each category below based only on what is on the image. Provide your response in exactly 8 images--one line per category--using only the allowed values. Do not include any category names, punctuation, explanations, or extra text. 1. Determine the location type. Choose one from [Indoor, Outdoor]. Indoor means the image shows an enclosed space, such as a room, hallway, or building interior. Outdoor means the image captures an open space, such as a street, park, or other external environments. 2. StairsYes or StairsNo? Determine if stairs are visible. Choose one from [StairsYes, StairsNo]. StairsYes indicates the presence of stairs, defined as a series of steps designed for climbing or descending. Stairs usually have distinct vertical and horizontal surfaces (risers and trends) arranged in a sequence. Horizontal markings, such as those on crosswalks, should not be considered stairs. 3. Lighting conditions: Choose one from [Dark, Bright]. Dark refers to low light conditions, where visibility might be limited. Bright indicates that the scene is well-lit, providing good visibility. 4. Illumination: Choose one from [Artificial, NaturalBandsumy, NaturalBandsumy]. Artificial means the scene is primarily lit by artificial light sources such as lamps, bubles, or other man-made lighting. This is typically observed in indoor environments or outdoor nighttime scenes where natural light is absent. NaturalBandsumy indicates that the scene is illuminated by direct sunlight. This is characterized by well-defined shadows and overexposed areas caused by strong natural light. The weather is clear, and the lighting is consistent with daylight condition. NaturalBandsumy indicates natural daylight is present but without direct sunlight. This is typical in overcast conditions, shaded areas, or when the sum is obscured by clouds. Shadows may be soft or absent, and the lighting appears diffused across the scene. 5. Doors: Choose one from [DoorsPresent, DoorsPresent]. DoorsPresent indicates that the image shows a physical structure designed to open and close, typically mounted on hinges or sliding mechanisms. Doors often separate spaces (e.g., rooms, buildings, or vehicles). Transparent glass panels, open doorways (without an actual door), or windows should not be considered as doors. DoorsNotPresent means there are no visible doors in the image. Note: Openings or frames without a physical door are not considered as doors.