# Human-AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM Era +
## 5 Identified Computer Vision Problems in RSA  
Since the identified challenges are mostly related to visual understanding and spatial perception, we relate one or more CV problems to each challenge in Table 1. The related CV problems include object detection, visual SLAM, camera localization, landmark recognition, visual navigation, scene text recognition, depth estimation, scale estimation, multiple object tracking, human trajectory prediction, field-of-view extrapolation, video stabilization, visual search, video captioning, video super-resolution, and video relighting. These problems are well formulated in the CV literature. Researchers have developed various methods to address these problems and have mostly evaluated them on standard benchmark datasets. In this section, we will analyze the applicability of each of these CV problem settings and solutions when dealing with the identified RSA challenges so as to find out the unique requirements of RSA services for CV research. **Object detection**[PERSON] et al. (2018) aims to detect instances of objects of predefined categories (e.g., humans, cars) in images or videos. Object detection techniques can answer a basic CV question: what objects exist in the image and where are they? It can be used to help address the RSA challenge of difficulty in proving various information, i.e., problem G3. (1) in Table 1. For example, if the user wants to find a trash can, the object detection algorithm can continuously detect objects in the live video feed. When a trash can is detected, it will notify the agent and user. **Visual SLAM**[PERSON] et al. (2018) methods are used for mapping an unknown environment and simultaneously localizing the camera through the video captured in this environment. It can directly address the RSA challenges of scarcity of indoor maps and inability to localize the user, i.e., problem G1. (1)(2)(3) in Table 1. The benefit of visual SLAM for RSA services has been demonstrated in our prior work [112] using Apple's ARKit. **Camera localization**[87] aims to estimate the 6-DOF camera pose (position and orientation) in a known environment. Here, the camera is a general term and could include an RGB camera, RGB-D camera, or 3D LiDAR. Camera localization can address the RSA challenges of inability to localize the user and difficulty in orienting the user, i.e., problem G1. (2)(3) in Table 1. Similar to visual SLAM, our prior work [112] has shown that real-time camera localization can effectively support RSA in indoor navigation. **Landmark recognition**[88] finds the images of a landmark from a database given a query image of the landmark. It is a subtask of content-based image retrieval and has the potential to address the RSA challenges of lack of landmarks on the map, i.e., problem G1. (4)(5) in Table 1. Instead of annotating landmark locations on the map, we can crawl and save the pictures of the landmark in a database. Then, when the user explores the environment, the landmark recognition algorithm can search the database and automatically annotate the landmark on the map if it matches. However, it may be only valid for well-known landmarks. For subtle landmarks, the challenge is open for exploration. **Visual navigation**[89] requires a robot to move, locate itself, and plan paths based on the camera observations until reaching the destination. It may help address the RSA challenges of last-few-meters navigation [26], i.e., problem G1. (7) in Table 1.  
**Scene text recognition**[90] aims to recognize text in wild scenes from a camera and can be regarded as camera-based optical character recognition (OCR). It is still a challenging CV problem due to factors such as poor imaging conditions. The scene text recognition technique could be used to overcome the RSA challenge of difficulty in reading signages and texts from the camera feed, i.e., problem G2. (1) in Table 1.  
**Depth estimation** is a task of estimating the depth of a scene. In particular, monocular depth estimation [91] aims to estimate depth from a single image.