# Human-AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM Era +
## 6 Emerging Human-AI Collaboration Problems in RSA
### Emerging Problem 3: Recognizing Digital Content on Digital Displays  
Digital displays, such as LCD screens and signages, are widely used in everyday life to present important information, e.g., flight information display boards at the airport, digital signage at theaters, and temperature control panels in the hotel. RSA agents reported difficulty in reading texts on these screens when streamed through the users' camera feed, as listed in problem G2. (1) in Table 1. This difficulty can be caused by several technical factors, including the varying brightness of a screen (i.e., the display of a screen is a mixture of several light sources, e.g., LCD backlight, sunlight, and lamplight [130]); a mismatch in the camera's frame rate and the screen's refresh rate; and a mismatch in the dimension of pixel grids of the camera and the screen, resulting in moire patterns, i.e., showing strobe or striping optical effects [130]. Based on the significance and challenges of recognizing content on digital displays through camera feeds, we consider it as an emerging research problem that can be addressed by human-AI collaboration. From the perspective of AI solutions, there exist a few CV systems that assist blind users in reading the LCD panels on appliances [131; 132; 133]. However, these systems are heuristic driven and fairly brittle and only work in limited circumstances. To the best of our knowledge, there is no text recognition method specifically designed to recognize digital texts on LCD screens or signages in the wild. In this regard, we consider scene text detection and recognition [135] as the closest CV method aiming to read texts in the wild. However, these methods are far more difficult than the traditional OCR of texts from documents. For example, the state-of-the-art deep learning methods [136; 137; 138] only achieve \(<85\%\) recognition accuracy on the benchmark dataset ICDAR 2015 (IC15) [139]. Furthermore, existing methods for scene text recognition are likely to suffer from the domain shift problem due to the distinct lighting condition [140], resulting in even worse recognition performance in reading digital content on LCD screens. To formulate human-AI collaboration, we consider scene text recognition methods [135] as the basis for AI models. Next, we consider three aspects of human-AI collaboration. First, CV techniques can be used to enhance the camera feed display [141], while the agents are responsible for the content recognition. In this way, the content in the live camera feed will be transferred to have better lighting and contrast, making it more suitable for the agents to perceive and recognize. Figure 1: Our design prototype for localizing users under poor networks with a split-screen dashboard. The top toolbar shows buttons to toggle a design feature on or off. The left-side screen shows a top–down view of a pre-constructed indoor 3D map, with the pink shape representing the user’s location and orientation. The right-side screen shows a pre-constructed 3D map view, which supplements the live camera feed under poor networks. Second, scene text recognition methods [135] can be used to read digital content for the agents and provide recognition confidence. This approach is particularly useful for recognizing small-scale text that is too small for agents to read on the camera display but contains enough pixels for AI models to process. The agent can ask the user to adjust the camera angle for a better view to achieve more accurate recognition results. Third, the agents are often interested in recognizing specific texts on the screen and can mark the region of interest for the AI to process. This approach helps improve the processing speed of AI models and reduces unwanted, distracting outputs from the models. Note that the above three aspects of human-AI collaboration overlap, e.g., the enhanced camera feed, can be utilized by both humans and AI to improve recognition capabilities. Since it is still an open problem, there may be other aspects of human-AI collaboration to explore in the future.