# Human-AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM Era +
###### Abstract  
Remote sighted assistance (RSA) has emerged as a conversational technology aiding people with visual impairments (VI) through real-time video chat communication with sighted agents. We conducted a literature review and interviewed 12 RSA users to understand the technical and navigational challenges faced by both agents and users. The technical challenges were categorized into four groups: agents' difficulties in orienting and localizing users, acquiring and interpreting users' surroundings and obstacles, delivering information specific to user situations, and coping with poor network connections. We also presented 15 real-world navigational challenges, including 8 outdoor and 7 indoor scenarios. Given the spatial and visual nature of these challenges, we identified relevant computer vision problems that could potentially provide solutions. We then formulated 10 emerging problems that neither human agents nor computer vision can fully address alone. For each emerging problem, we discussed solutions grounded in human-AI collaboration. Additionally, with the advent of large language models (LLMs), we outlined how RSA can integrate with LLMs within a human-AI collaborative framework, envisioning the future of visual prosthetics. p 1 2024 16 254 10.3390/616070254 10.  
complex tasks, such as assisting PVI in navigating airports and crossing noisy intersections without veering. With the increased task complexity, researchers [1; 13] have identified that reliance on smartphones' camera feeds can be a limiting factor for agents, affecting their performance and mental workload, which can subsequently degrade the user experience of PVI. To elaborate, [PERSON] et al. [13] and [PERSON] et al. [1], who studied RSA services, reported several challenges for agents, such as lack of confidence due to unfamiliarity with PVI's physical surroundings, lack of detailed indoor maps, inability to continuously track PVI on static maps, difficulty in estimating objects' distances in the camera feed, and describing relevant objects or obstacles in real time. However, these challenges are derived from the agents' perspective, largely overlooking the user experience of PVI, who are the _true_ users of RSA services. As such, the findings in prior works are likely to be incomplete. This paper draws on prior works to holistically understand the technical and navigational challenges in RSA from both the agents' and users' perspectives. Specifically, we aim to address two research questions: _What makes remote sighted assistance challenging?_ and _When does this assistance become challenging to use?_  
To that end, we employed two methodologies. _First_, we conducted a literature review to identify technical challenges in RSA services, mostly derived from agents' point of view. _Second_, we conducted an interview study with 12 visually impaired RSA users to understand navigational challenges from their standpoint. Based on these two studies, we then constructed an exhaustive list of technical and navigational challenges to expand upon prior works and outline how these challenges occur in different real-world navigation scenarios. We organized technical challenges into four broad categories: agents' difficulty in orienting and localizing the users, acquiring the users' surrounding information and detecting obstacles, delivering information and understanding user-specific situations, and coping with poor network connection and external issues. Additionally, we produced a list of 15 real-world scenarios (8 outdoor, 7 indoor) that are challenging for PVI to navigate. This article is an extended version of our work presented at the 2022 International Conference on Intelligent User Interfaces (IUI 2022) [14].