# Human-AI Collaboration for Remote Sighted Assistance: Perspectives from the LLM Era +
## 6 Emerging Human-AI Collaboration Problems in RSA
### Emerging Problem 5: Predicting the Trajectories of Out-of-Frame Pedestrians or Objects  
In RSA services, the agents need to provide the environmental information in the user's surroundings (e.g., obstacles and pedestrian dynamics) for safety when the user is in a crowded scene. The trajectory prediction of pedestrians or moving objects could assist the agent in providing timely instructions to avoid collision. According to our literature review, it is extremely difficult for RSA agents to track other pedestrians/objects [43; 94] from the users' camera feed and almost impossible to predict the trajectories of out-of-frame pedestrians or objects [8; 9; 10; 11; 13; 42; 94], as listed in problems G2. (3) and G2. (4) in Table 1. The main reasons are the narrow view of the camera and the difficulty in estimating the distance. Based on this observation, we pose an emerging research problem of predicting the trajectories of out-of-frame pedestrians or objects that can be addressed by human-AI collaboration, as illustrated in Figure 2.  
Considering only AI solutions, we can adopt human trajectory prediction technology [96], which has been studied as a CV and robotics problem. Specifically, the motion pattern of pedestrians/objects can be learned by a data-driven behavior model (e.g., deep neural networks). Then, based on the observation from the past trajectories, the behavior model can predict the future trajectories of the observed pedestrians/objects. There are two types of problem settings, i.e., observed from either static surveillance cameras [145; 146] or moving (handheld or vehicle-mounted) cameras [147; 148]. For RSA application, we focus on predicting from handheld cameras. Existing trajectory prediction methods forecast the future pixel-wise locations of the pedestrians on the camera feed without considering the out-of-frame cases. The pixel-level prediction is also not useful for the agents to estimate the distance to avoid collision. Moreover, existing models are learned from the scene without PVI, but the motion patterns of pedestrians around PVI could be rather different. Figure 2: Our design prototype for predicting the trajectories of out-of-frame pedestrians. The top toblar in each figure shows buttons to toggle the design feature on or off. The information on indoor maps and the camera feed is coordinated through colors. Rectangles represent pedestrian detection, lines on the ground are trajectory predictions, intervals between dots symbolize equal distance, arrows represent orientation, and alerts will pop up when collisions may occur. Trajectories of pedestrians turn gray when the pedestrians are out of the camera feed, as shown in (**b**). In terms of human-AI collaboration, to the best of our knowledge, there is no work exploring the problem of pedestrian tracking and trajectory prediction under active camera controls. We consider three aspects of human-AI collaboration in predicting the trajectories of out-of-frame pedestrians. _First_, we need to develop user-centered trajectory prediction technologies. On the one hand, the behavior models need to be trained from a PVI-centered scene. On the other hand, the predicted trajectories should be projected to the real world where even the pedestrians cannot be observed from the camera feed. Based on such trajectory predictions, the agents can quickly plan the path and provide instructions to the user. _Second_, the agents may be only interested in the pedestrian dynamics toward the user's destination. In this case, the agents can mark the region of interest for AI models to conduct prediction. Then, AI models will save some computational resources and also understand the interests of the agents. _Third_, in turn, AI models could suggest moving the camera towards a certain direction (e.g., left) to obtain more observations for better predictions. In this way, AI models can better reconstruct the scene for the agents to make navigational decisions for the user. This problem can be further extended in the human-AI collaboration setting.