Following the Human Thread in Social Navigation

## Abstract

Abstract The success of collaboration between humans and robots in shared environments relies on the robot’s real-time adaptation to human motion. Specifically, in Social Navigation, the agent should be close enough to assist but ready to back up to let the human move freely, avoiding collisions. Human trajectories emerge as crucial cues in Social Navigation, but they are partially observable from the robot’s egocentric view and computationally complex to process. We propose the first Social Dynamics Adaptation model (SDA) based on the robot’s state-action history to infer the social dynamics. We propose a two-stage Reinforcement Learning framework: the first learns to encode the human trajectories into social dynamics and learns a motion policy conditioned on this encoded information, the current status, and the previous action. Here, the trajectories are fully visible, i.e.,  assumed as privileged information. In the second stage, the trained policy operates without direct access to trajectories. Instead, the model infers the social dynamics solely from the history of previous actions and statuses in real-time.
Tested on the novel Habitat 3.0 platform, SDA sets a novel state of the art (SoA) performance in finding and following humans. The code is available at: https://github.com/L-Scofano/SDA .

###### Keywords:

## 1 Introduction

Traditional navigation techniques within Embodied Artificial Intelligence (EAI) have marked a crucial advancement by introducing robots into real-life environments. However, these techniques have primarily focused on agents traversing vacant spaces. Conversely, the significance of social navigation within EAI has steadily increased. Social navigation entails agents’ capacity to navigate human-centric environments while considering human movements and behaviours. These agents need to be able to locate, track, and interact with humans in a safe and socially acceptable manner.
Previous studies predominantly characterized Social Navigation as a variation of PointGoal Navigation, wherein agents strive to reach specified destinations while considering human movements [, , ]. Habitat 3.0 [], a significant breakthrough in EAI, introduces a lifelike environment seamlessly incorporating human avatars. This integration enables the investigation of human-agent interactions within a controlled, risk-free, dynamic environment. What sets Habitat 3.0 [] apart is its ability to replicate complex scenarios where human intentions are constantly changing. Nevertheless, this dynamism also presents particular challenges, such as collision avoidance and achieving success in locating and following humans.

Despite notable efforts in collision avoidance and safety, most existing methods for Social Navigation either rely on privileged information that is not available in real-world scenarios or do not adequately capture the social dynamics and norms of human behaviour. For instance, [] and [] use a GPS and compass sensor to provide the agent with perfect localization, which might be unrealistic even if using SLAM [, ] methods, whenever the human is not in the line of sight. While [] and [] do not account for the social factors that influence human behaviour. Therefore, this limitation hinders their practical applicability and adaptability to dynamic environments. [], instead, models some social factors in the form of Proximity Tasks but fails to account for the cooperative nature that a social agent must possess, restricting itself to merely avoiding collisions with them.
The current SoA model proposed in [] necessitates privileged information, such as humanoid GPS, which offers polar coordinates, detailing the accurate distance and angle of the human from the robot, to attain high-performance outcomes. However, this requirement is highly impractical in real-world environments during inference.

This paper proposes a novel Social Dynamics Adaptation model (SDA), shown in Fig. 2, that effectively solves the robot’s awareness of complex human behaviors, even temporarily losing sight of the person and fast robot motion. Specifically, the first stage trains a base policy considering human trajectories encoded into a latent vector. The latent vector is a low-dimensional, nonlinear projection of the human trajectories, and it is trained end-to-end with the base policy to extract the social factors that led the robots to choose better actions. The subsequent supervised adaptation stage regresses this latent vector using only the robot’s state and action history. Unlike previous methods, such as [] and [], which often rely on simulated privileged information or simplified social behavior models, our approach learns to respond to real-world dynamics in real time. In summary, SDA adapts and accounts for unpredictable human behaviors by exploiting privileged information during training and recovering this fundamental signal during deployment when it is often impractical to compute. Finally, the deployed robot features the motion policy, learned in the first stage, and the social dynamics, inferred from prior statuses and actions.

Out of extensive benchmarking, SDA outperforms the approach proposed in Habitat 3.0 [] and a second adapted best performer method [] from Habitat 2.0 []. We conduct a thorough experimental evaluation on the core contribution of this work, i.e., learning to infer the social dynamics from (privileged) information about the person.
Remarkably, out of ablative studies, we conclude that not only are human trajectories strong input information for the robot control policy, but they also make better supervision for inferring the social dynamics latent for the same policy. Other oracular information such as the humanoid GPS (direction and distance from the person-to-robot) are powerful sensors for the control policy but do not yield adaptable social dynamics to infer the human-robot-scene interaction.

## 2 Related Work

##### Embodied Navigation.

Recently, there has been a surge in exploring indoor navigation within an embodied framework []. This upswing has been facilitated primarily by the availability of large-scale datasets comprising 3D indoor environments [, , ] and simulators designed for simulating navigation within these dynamic 3D spaces [, , ]. Nevertheless, these simulators are not equipped to handle human entities within the environments, restricting the investigation to navigation tasks in scenarios where the agent functions independently or, at most, alongside humans simulated via static meshes that simulate movement [, ]. These simulated humans are treated as dynamic obstacles and lack compliance with any social construct. This constraint has been effectively addressed with Habitat 3.0 [], the simulator used for this research. Habitat 3.0 introduces the capability to simulate the behaviours of humans engaging in tasks within dynamic environments, thus overcoming the limitations mentioned above. Furthermore, there is no longer a need to directly train social models in the real world, where there is a tangible risk of causing harm to actual humans. Thus, Habitat 3.0 establishes itself as a viable proxy for real-world adaptation of social models trained in simulation.

Thanks to these simulators the realm of EAI has witnessed the introduction of numerous tasks [], including PointGoal Navigation [], ObjectGoal Navigation [], Embodied Question Answering [], and Vision and Language Navigation (VLN) [, ]. Various modular approaches [, , , , ] have been proposed to address the challenges of navigating through static, single-agent environments. These approaches utilize maps constructed from depth images and conduct path planning directly on these maps. However, these approaches are unsuitable in social settings where dynamic objects (humans) move within the environment. This is because humans are observable only within the agent’s field of view (FOV). Moreover, the agent must address the additional challenge of tracking and mapping. End-to-end RL-trained policies [, , , , ], should be adapted to learn also social clues similarly to [], where the agent learned proxemics information about the humans moving in the environments thanks to two proximity tasks.
In this paper, instead, we try to learn social behaviours directly by internally modeling the humanoid trajectories in the latent representation of the agent. Furthermore, differently from [], the agent’s aim is no longer just avoiding collisions. Still, it involves locating this dynamically acting human and following them for a specified number of steps while maintaining a safe distance. This evolution of the task demands a heightened level of social comprehension from the agent, requiring the anticipation of the person’s intentions and the ability to trail them closely without compromising safety.

##### Socially-Aware Navigation.

Research in robotics, computer vision, and the analysis of human social behavior explored socially aware representations and models []. Extending from the realm of collision-free multi-agent navigation [, , , ] and navigation in dynamic environments [], researchers have further expanded their investigations to encompass scenarios involving human presence [, , , , ].

The approach presented in [] incorporates collision avoidance algorithms like CADRL [] while introducing common-sense social rules. This integration aims to reduce uncertainty while minimizing the risk of encountering the Freezing Robot Problem []. Other works [, ] seek to model human-agent interaction by employing techniques such as spatiotemporal graphs [].

Typically, these methods undergo testing in a minimal number of environments that offer complete knowledge about the human positions and velocities [, ], featuring simple obstacles and often assuming collaboration between moving agents[, ]. In contrast, our focus revolves around SocialNav within expansive indoor environments, characterized by partial knowledge about them, since the agent perceives the environment only through its sensors from an egocentric perspective, and no information about the velocity or position of the human is given to the agent. Our SDA addresses the missing information from the interacting human position, inferring it from the robot’s history of actions and status.

##### Modelling dynamic environments.

Simulated environments [, , ] offer privileged information about the scene whose exploitation can be computationally intense or unfeasible during deployment. While navigating social environments, it is vital to take into consideration human behaviour[]. Ideally, one would want to forecast people’s position for better path planning [], but forecasting robot-person interactions is significantly slower [] than navigation policies, hence being challenging to be considered for training or deployment.
To overcome this problem, we leverage literature on system identification [, ] to infer the encoded privileged information during robot navigation. Once encoded in a latent space during the first training stage and used to train the primary policy, it is possible to asynchronously derive that same information from the state-action history [, , ], influenced by the signal we want to identify. Unlike previous works, we are the first to identify the social dynamics (under the form of human trajectories), with the intuition that modeling human behavior is fundamental for efficient human-robot collaboration.

Figure: Figure 2: Pipeline of the novel methodology proposed. First, we jointly learn to encode human trajectories and a motion policy. In the next stage, given the previous states and actions, we infer the social dynamics and pass the estimated latent vector to the frozen policy.
Refer to caption: extracted/2404.11327v1/figures/pipeline_new.png

## 3 Methodology

This section introduces a novel framework, SDA , for social navigation that incorporates human trajectories into the sequential decision-making process. The first stage of our approach focuses on encoding human trajectories into a latent social context to represent the social dynamics that are functional to the agent motion policy. In the second stage, we introduce the Adapter module, which enables the estimation of social information from the agent’s past behavior. Recovering this signal allows the robot to operate in the real world without explicitly representing human behavior. We elucidate the training process, trajectory modeling, and optimization techniques utilized in our approach, highlighting its effectiveness in addressing the challenges of social navigation.

##### Problem formalization.

The problem involves the task of locating and following a humanoid in motion within a household environment, maintaining a distance of 1 to 2 meters for at least $k$ steps [].

With the status $x_{t}$, we represent the agent’s “perception” at time $t$. We exploit depth images from different cameras placed on the robot and a preprocessed version containing a humanoid detection bounding box. These perceptions are processed with a ResNet [] before being fed to the recurrent policy network, selecting the best action $a_{t}$ at time $t$. We use Decentralized Distributed Proximal Policy Optimization (DD-PPO) [] to iteratively improve the agent’s policy while maximizing rewards derived from interactions with several environments executed in parallel, ensuring stability through controlled policy updates and lower training times.

The pipeline described above can be considered a baseline implementation that does not contain privileged information but relies only on the robot’s onboard sensors. To address the social aspect required by the task, we consider additional details on the humanoid, e.g., “social dynamics”, defined as $e_{t-N:t-1}$. In the following sections, we describe how exploiting this information at training time can improve performance during deployment when it is absent.

### 3.1 Stage 1: Social Policy

Recurrent policies such as DD-PPO take as input the current status, in our case, a collection of depth images processed via a Resnet $x_{t}$, and the action at the previous time-step $a_{t-1}$. We add another input to this pipeline, namely a latent vector $z_{t}$ built by encoding the humanoid privileged information $e_{t-N:t-1}$:

$$ $\displaystyle z_{t}$ $\displaystyle=\mu(e_{t-N:t-1})$ (1) $\displaystyle a_{t}$ $\displaystyle=\pi(x_{t},a_{t-1},z_{t})$ (2) $$

Intuitively, by training everything with the same objective $z_{t}$ encodes the social dynamics that led the agent to maximize its reward, adapting to human movement patterns. Additionally, including trajectory data allows the agent to learn from past interactions and experiences.
In our approach, the trajectory encoder ($\mu$) is implemented as Multilayer Perceptrons (MLPs). The objective retains its usual formulation without any explicit reference to the human trajectories:

$$ $L^{CLIP}=\mathbb{E}_{t}\left[\min\left(r_{t}(\theta)\hat{A}_{t},\text{clip} \left(r_{t}(\theta),1-\epsilon,1+\epsilon\right)\hat{A}_{t}\right)\right]$ (3) $$

where $r_{t}(\theta)$ is the ratio of the probability of action $a_{t}$ under the current policy and the previous one that is being executed for gathering data.
$\hat{A}_{t}$ represents the advantage function at time $t$, guiding the policy towards actions that yield higher expected rewards.

Defining what information is considered “privileged” and why is essential. In our context, it refers to detailed knowledge about the humanoid in the environment, such as the exact position of humanoids defining a trajectory (traj.), of the relative position with respect to the agent often denoted as humanoid GPS [](hGPS). We can easily gather this information in simulated environments. However, collecting them in real-life scenarios is often impractical. This distinction is crucial as it highlights the challenge of transferring learned policies from simulation to the real world.

### 3.2 Stage 2: Social Dynamics Regression

We aim to extract and exploit social cues directly from the robot’s perception and eliminate the need for auxiliary devices like GPS trackers on humanoids. Inspired by [], we introduce the “social dynamics” module (Adapter), parametrized by an MLP $\psi$ that takes as input the recent history of the robot’s states $x_{t-N:t-1}$ and actions $a_{t-N:t-1}$ to generate a new latent vector $\hat{z}_{t}$:

$$ $\hat{z}_{t}=\psi(x_{t-N:t-1},a_{t-N:t-1})$ (4) $$

We obtain the state-action history by deploying the agent in the environment with optimal policy $\pi^{*}$ obtained after the first stage and the latent vector $\hat{z}_{t}$:

$$ $a_{t}=\pi^{*}(x_{t},a_{t-1},\hat{z}_{t})$ (5) $$

During this process, we optimize the Adapter, MLP, with a supervised regression objective, Mean Squared Error (MSE), to recover the original information contained in $z_{t}$ that we compute relying on the preferential information trajectory:

$$ $\text{MSE}(\hat{z}_{t},z_{t})=\|\hat{z}_{t}-z_{t}\|_{2}^{2}$ (6) $$

Once we finalize the Adapter training, instead of relying on the privileged information, we can depend upon the robot’s states $x_{t-N:t-1}$ and actions $a_{t-N:t-1}$ to generate $\hat{z}_{t}$, serving as an estimate of the actual latent social dynamics vector $z_{t}$. Doing so enables the agent to estimate social dynamics online, improving its performance in dynamic environments and enhancing its social navigation capabilities, freeing it from dependence upon external sensors.

### 3.3 Training details

In our training process, we encode social dynamics using a ResNet, which we train from scratch. The trajectory encoder $\mu$ consists of a 3-layer MLP, and the output $z_{t}$ has a dimensionality of 128. The Adapter $\psi$ comprises alternating spatial and temporal MLP layers, with the output $\hat{z}_{t}$ matching the dimensionality of $z_{t}$.
We jointly train the policy $\pi$, the perceptions encoder (ResNet), and the social dynamics (Trajectory Encoder) encoder $\mu$ during the first stage. In the second stage, everything is frozen except the Adapter.

In Stage 1, we utilize DD-PPO [] for 250 million steps across 24 environments, following the training protocol presented in []. The training process takes approximately four days. In Stage 2, we employ supervised learning for 5 million steps across the same environments. The learning process lasts around 2 hours.
Both stages utilize 4 A100 GPUs for efficient computation.

## 4 Results

Section 4.1 presents our findings regarding Social Navigation, whereas Section 4.2 conducts an ablation analysis concentrating on adaptable information and the role of the trajectory history length. Furthermore, Section 4.3 provides a qualitative exploration of our outcomes.

##### Simulator.

We utilize our SDA on Habitat 3.0[], a simulation platform designed for human-robot interaction within domestic settings. This platform offers precise humanoid simulation capabilities with a strong focus on collaborative tasks such as Social Navigation and Social Rearrangement. It offers a vast library of avatars featuring multiple gender representations, body shapes, and appearances, facilitated by SMPL-X []. Furthermore, it employs an oracle policy to generate movement and behavior, enabling programmable control of avatars for navigation, object interaction, and a range of other movements.

##### Baselines.

The baselines we employ in our study are drawn from Habitat 3.0 [] and consist of:

- •
*Heuristic Expert*: a heuristic baseline equipped with access to the environment map, employing a shortest path planner to devise a route to the current location of the humanoid. The heuristic expert operates on the following principles: When the agent is beyond a distance of 1.5 meters from the humanoid, it employs a “find” behavior, utilizing a path planner to approach the humanoid. Conversely, if the humanoid is within 1.5 meters, the expert executes a backup motion to prevent collision with the humanoid.
- •
*Baseline*: the current SoA method [], a recurrent neural network policy trained with DD-PPO [], operates on a “sensors-to-action” paradigm. Inputs to this policy consist of an egocentric arm and stereo depth sensors, a humanoid detector, and humanoid GPS coordinates, while the outputs are velocity commands in the robot’s local frame. Notably, this policy operates without access to an environment map.
Table 1 compares our model against the most realistic configuration, wherein the humanoid GPS data is unavailable.
- •
*Proximity tasks*: we also adapted the Proximity Tasks defined in [] and applied them to the baseline []. These tasks were proposed for a different setup of SocialNav, where the agent acts in an environment with multiple humanoids and must navigate from point A to point B while avoiding collisions. We adapted the risk and compass proximity tasks to the SocialNav setting addressed in this article. In this context, the risk has a low value (close to 0) when the agent is within 1 to 2 meters from the humanoid and a value close to 1 when the distance is less than 1 meter or greater than 2 meters. Similarly, given the presence of only one humanoid in the environment, the compass was redefined to predict the angle between the humanoid and the agent. This adjustment aims to assist during the following phase, enabling the agent to follow the humanoid while maintaining a safe distance and staying aligned with the human. The proximity tasks are jointly trained with the policy and detached during evaluation.

##### Metrics.

We used the metrics for the SocialNav task as defined in [], briefly summarized in the following.
1. *Finding Success ($S$)*is the ratio of the episodes where the agent successfully located and reached the human.
2. *Finding Success Weighted by Path Steps ($SPS$)*measures the optimality of the path taken by the agent w.r.t. the optimal number of steps needed to reach the human.
3. *Following Rate ($F$)*is the ratio of steps during which the robot maintains a distance of 1-2 meters from the humanoid while facing towards it relatively to the maximum possible following steps.
4. *Collision Rate ($CR$)*is the ratio of the episodes that ended with the robot colliding with the humanoid.
5. *Episode Success ($ES$)*measures the ratio of the episodes where the agent found the human and followed it for the required number of steps, maintaining a safe distance in the 1-2 meters range.

##### Privileged information.

In our research, the privileged information under consideration includes humanoid GPS (hGPS) and human trajectories (traj.). Humanoid GPS is represented in polar coordinates, a method of specifying a point’s position in a plane using two parameters: the distance from the point to the origin (radius) and the angle formed between a reference direction (typically the positive x-axis) and a line connecting the origin to the point (polar angle or azimuth). In our context, the origin is defined as the robot’s position; thus, its position is implicitly known along with that of the human. Conversely, trajectories solely consist of information derived from the human’s position within the environment.

### 4.1 Quantitative Results

In Table 1, we present the results obtained in Habitat 3.0 [] for the Social Navigation task. The table is divided into three sections. The first one displays outcomes achieved by utilizing a heuristic expert endowed with extensive information, including its position, map data, and the humanoid’s position, granting it a competitive advantage over other methods. The subsequent section features models trained and tested using ground truth (GT) data (Baselines and Stage 1), thus establishing an upper limit for techniques utilizing more practical inputs feasible in real-world scenarios. Lastly, the final rows delineate results from methods conducting inference without privileged information. The models that use privileged information (GT) show that S1 has lower performance than the two Baselines, especially in $S$ and $SPS$. This is to be expected, considering that while trajectories solely depict the movement of the human in the environment, hGPS furnishes crucial details on locating the humanoid, offering insights into the distance and angle between them.

In the second stage, despite the absence of human trajectory input, SDA maintains the performance level of Stage 1 by adapting to the social dynamics. Our model outperforms the best techniques in the find task, increasing $S$ and $SPS$ by $6\%$.

Our approach generally improves performance in the episode success ($ES$) metric, which occurs when the agent finds the humanoid and follows it for $400$ steps. However, we emphasize that the test-time episode ends at $1500$ steps or when the agent collides with the humanoid [], not after the $400$ follow steps. In this context, $S$, $SPS$, and $F$ metrics demonstrate how SDA, compared to the Baseline, more frequently locates the humanoid, follows a more optimal path (on average $438$ vs. $540$ steps), and follows it for a longer duration ($390$ vs. $218$ steps). Therefore, as the agent follows the humanoid longer, it has more chances to collide with it, given that the episode does not necessarily end after the required steps. In a scenario where the test-time episode concludes either after $400$ follow steps or immediately upon a collision between the agent and the humanoid, SDA and the Baseline show a comparable collision rate ($CR$), $0.39$, and $0.38$, respectively.

**Table 1: Main results for Social Navigation. Within the table, GT denotes ground truth privileged information and * corresponds to reproduced results.**
| Models | hGPS | traj. | S $\uparrow$ | SPS $\uparrow$ | F $\uparrow$ | CR $\downarrow$ | ES $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Heuristic Expert [] | - | - | 1.00 | 0.97 | 0.51 | 0.52 | - |
| Baseline [] | GT |  | 0.97^±0.00 | 0.65^±0.00 | 0.44^±0.01 | 0.51^±0.03 | 0.55^±0.01* |
| Baseline+Proximity [](^*^**Code refactored and adapted from [] to Habitat 3.0.) | GT |  | 0.97^±0.01 | 0.64^±0.00 | 0.57^±0.01 | 0.58^±0.03 | 0.63^±0.02 |
| SDA - S1 |  | GT | 0.92^±0.00 | 0.46^±0.01 | 0.44^±0.02 | 0.61^±0.02 | 0.50^±0.01 |
| Baseline [] |  |  | 0.76^±0.02 | 0.34^±0.01 | 0.29^±0.01 | 0.48^±0.03 | 0.40^±0.02* |
| Baseline+Proximity [] |  |  | 0.85^±0.02 | 0.41^±0.02 | 0.37^±0.01 | 0.58^±0.02 | 0.41^±0.01 |
| SDA - S2 |  |  | 0.91^±0.01 | 0.45^±0.01 | 0.39^±0.01 | 0.57^±0.02 | 0.43^±0.02 |

**Table 2: Ablation studies for social dynamics estimation. GT denotes ground truth privileged information, and A indicates the adapted information.**
| Models | hGPS | traj. | hGPSA | traj.A | S $\uparrow$ | SPS $\uparrow$ | F $\uparrow$ | CR $\downarrow$ | ES $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| S1 | GT | GT |  |  | 0.94^±0.01 | 0.58^±0.00 | 0.45^±0.02 | 0.64^±0.03 | 0.52^±0.01 |
| S1 | GT |  |  |  | 0.93^±0.00 | 0.62^±0.01 | 0.46^±0.01 | 0.64^±0.02 | 0.48^±0.01 |
| S1 (Proposed) |  | GT |  |  | 0.92^±0.00 | 0.46^±0.01 | 0.44^±0.02 | 0.61^±0.02 | 0.50^±0.01 |
| S2 |  |  | ✓ | ✓ | 0.57^±0.06 | 0.21^±0.04 | 0.05^±0.01 | 0.30^±0.02 | 0.02^±0.00 |
| S2 |  |  | ✓ |  | 0.70^±0.02 | 0.31^±0.02 | 0.05^±0.01 | 0.70^±0.03 | 0.03^±0.01 |
| S2 (Proposed) |  |  |  | ✓ | 0.91^±0.01 | 0.45^±0.01 | 0.39^±0.01 | 0.57^±0.02 | 0.43^±0.02 |

Figure: Figure 3: (left) Training step (x-axis) Vs the number of steps which it takes the robot to find the humanoid (y-axis), in the finding task; (right) Training step (x-axis) Vs the average distance which the robot manages to keep itself at from the humanoid (y-axis), in the task of following.
Refer to caption: extracted/2404.11327v1/figures/stage2_firstenc.png

### 4.2 Ablation studies

In the following section, we investigate the impact of various forms of privileged information and their adaptability. Additionally, we assess how the length of the trajectories fed in Stage 1 affects our model.

#### 4.2.1 Adatptable information

Table 2 presents the results from Stage 1 (S1) and Stage 2 (S2) with the utilization of various privileged information, such as Humanoid GPS (hGPS) and human trajectories (traj.). During S1, particularly in $S$ and notably in $SPS$, incorporating hGPS as an additional input leads to superior results. This advantage is likely attributed to hGPS containing implicit information about human and robot positions, facilitating more efficient path selection, especially in $SPS$ scenarios.

However, this pattern is not evident in S2, where trajectories typically provide more adaptable information. As previously mentioned, hGPS inherently encompasses the robot’s position, posing challenges in regressing this data during the second stage due to the lack of initial context regarding the robot’s location relative to the environment. Furthermore, hGPS may be difficult to adapt when the human is detected and disappears around a wall. Since hGPS comprises polar coordinates, the distance between the robot and the human spans the wall. This issue does not arise when utilizing trajectories, as they do not require any information about the robot and can be adapted solely based on depth images and detection information.

Fig. 3(left) illustrates the trend of the First encounter step over episode during Stage 2. The plot confirms that using trajectories facilitates the robot to find the person, which it achieves after only approx. 450 steps, while the hGPS and hGPS+traj take more than 700 steps. Finding the robot first results in higher $S$ and $SPS$ metrics, but it may incur larger chances of collision ($CR$ metric), due to having to then follow it for longer, cf. Sec. 4.1.
Fig. 3(right) illustrates the average distance between the agent and the humanoid after the first encounter. While hGPS and hGPS+traj stay at approx. 4 meters, the proposed trajectory-based approach ranges between 1.5 and 1.9, thus well within the 1-2 meter range, which yields the success in the task of following (larger $F$ metric, but encompassing a larger risk of collision– $CR$ metric).

**Table 3: Ablation on the length of trajectories to consider during Stage 1**
| Timeframes | S $\uparrow$ | SPS $\uparrow$ | F $\uparrow$ | CR $\downarrow$ | ES $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| 5 | 0.62^±0.01 | 0.25^±0.02 | 0.05^±0.01 | 0.64^±0.01 | 0.06^±0.01 |
| 20 (Proposed) | 0.92^±0.00 | 0.46^±0.01 | 0.44^±0.02 | 0.61^±0.02 | 0.50^±0.01 |
| 50 | 0.70^±0.02 | 0.29^±0.02 | 0.08^±0.02 | 0.78^±0.02 | 0.08^±0.01 |

Figure: Figure 4: The agent and the humanoid start the episode in separate rooms. The agent navigates through the environment in search of the humanoid, and once found, begins to follow it.
Refer to caption: extracted/2404.11327v1/figures/qualitative_strategy3.png

#### 4.2.2 Length of trajectories

Table 3 illustrates the influence of trajectory length on our SDA. Employing 5 past timeframes as input yields generally inferior results, possibly due to the insufficient historical data to capture human dynamics. Similarly, using 50 timeframes leads to comparable outcomes. Although $S$, $SPS$, and $F$ metrics improve relative to the 5-frame scenario, they still lag behind our proposed model. This could be attributed to potential redundancy and counterproductive encoding of information.

Figure: Figure 5: We showcase two different episodes. On the left, the agent successfully follows the human after it swiftly moves in front of the door. On the right, an episode where the human decides to move backward and the agent steps back to make way.

### 4.3 Qualitative Results

We qualitatively demonstrate the results in our proposed SDA. Firstly, we showcase the agent’s ability to locate the humanoid within the environment by moving around, followed by its capability to follow the humanoid within the environment. Subsequently, we present two specific behaviors where the agent briefly spots the humanoid and one where it moves backward to create space for passage.

Fig. 4 shows an episode where the agent and the human are located in different rooms at the start. Then, the agent begins its search for the humanoid by navigating within the environment until the encounter takes place. After the encounter, the agent then transitions into the follow phase.

The episode in Fig. 5 (left) illustrates the agent’s capability to track the human even when it is briefly observed. As the agent searches for the target in the bathroom, the humanoid swiftly passes in front of the door (Step 145) and disappears from the agent’s view again (Step 150). Due to learned social dynamics, the agent exploits the humanoid’s behavior and begins to follow it.

In the right episode in Fig. 5, the agent has already located the human and its objective is to follow it. It is observed that at Step 340, the human decides to move backward, prompting the agent to move backward as well, creating the necessary space for passage by Step 360. Then, it continues its task of following.

##### Limitations.

Table 1 shows a dependency of stage 2 performance on the adaptability of information, highlighting the significance of this factor in determining the model’s effectiveness. Another limitation arises from having benchmarked the proposed model only in simulation and the reliance on information during training that is not easily accessible outside simulation (trajectories). Mitigating these limitations, the advent of Habitat 3.0 [], has greatly minimized the domain gap between simulated and real environments.

## 5 Conclusions

Our study presents the Social Dynamics Adaptation model (SDA) for Social Navigation within interactive settings. Notably, it is the first to integrate privileged human dynamics information during training while adapting it in the following stage, enabling its application in realistic environments without relying on such privileged information. Our findings underscore the non-trivial nature of adapting information, highlighting the necessity for selective processes. In future research, we aim to extend our model by incorporating diverse human dynamics beyond trajectories, enhancing the robot’s comprehension of human movement patterns.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: - Regarding technical novelty, the authors discuss the comparison between the proposed work and the previous work (Cancelli et al., 2023) in Sec.2. It seems that the techniques in both works are similar, but this paper proposes to use a similar structure for the following-human problem, rather than just avoiding collisions with humans as in Cancelli et al., 2023. It would be great if the author could clarify about the difference between these work, or about the extra challenges introduced by following a person rather than just collision avoidance (e.g., perhaps the challenge is more than just changing the reward function of the RL method?).
soundness: 3
strengths: - The problem is well-motivated.
- The writing is clear.
- The method is sound.
summary: This paper proposes a new method enabling a robot to search for a human in a simulated indoor environment and then follow them by maintaining a distance of 1 to 2 meters for at least k steps.

The key challenge is that the robot can only see the human via its first-person egocentric camera, resulting in partial knowledge about the environment, as well as the position and velocity of the human.

To address this problem, the authors propose to rely on privileged information, e.g., human GPS data and human trajectories, that are only available during training, not deployment. Then, the authors propose to train the robot policy in 2 phases. In phase 1, the robot does RL (DD-PPO) to jointly learn an encoder that maps privileged information to the latent state and a policy that depends on the robot camera image and the latent state. In phase 2, the robot will run the learned policy while doing supervised learning to learn a decoder to map the robot's camera image to the latent state (the data for the latent state comes from the ground truth privileged information, and the learned encoder from phase 1). Then, during online deployment, the robot will first convert the camera image to the latent state using the decoder from phase 2, and then run the robot policy learned from phase 1.

The result focuses on the Habitat 3.0 simulator. The proposed method is shown to outperform 3 baseline methods. An ablation study was conducted to evaluate the usefulness of each privileged information. Latent analysis shows that the latent state refers to the robot's subtask in find, seek, lost, and follow. The authors also add more realistic components into the simulator to simulate real-world scenarios, such as making the simulated human aware of the robot's presence with reciprocal collision avoidance (ORCA), lowered planning frequency, and increased noise. In such scenarios, the proposed method still outperforms the baseline methods.
weaknesses: - As far as I can tell, the training and testing assume the same simulated person. In the real world, this means that to follow one person, the robot has to interact with the user to collect offline data for Phase 1 and online data for Phase 2. This can be very cumbersome. It would be great if the authors could discuss some strategies to scale this method up for different people, such as using meta-learning approaches or using more diverse training data.

- The result shown in Tab.1 shows only marginal improvement over the baseline "Baseline+Proximity Cancelli et al. (2023)". Could the authors provide an discussion about the practical implications of these improvements? Another way to demonstrate its significance is to use statistical test.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: Are there any plans to test SDA in real-world environments to evaluate how well it transfers from simulation, particularly in handling variable lighting, unexpected obstacles, and diverse human behaviors?
How does SDA perform when humans change speed or direction quickly? Does the reliance on past state-action history allow the model to adapt in real-time, or are there latency issues?
soundness: 3
strengths: Firstly, the SDA model’s two-stage training framework effectively combines privileged information during training with adaptable inferences during deployment, allowing the model to handle real-world scenarios without needing precise human trajectory data. It outperforms several baselines, showing improved metrics for finding, following, and episode success, which is critical for effective social navigation. Testing on Habitat 3.0, with realistic human motion and environmental noise, strengthens the claim that SDA is robust and adaptable to real-world applications. Comprehensive evaluations, including noise analysis, reduced update frequencies, and alternative metrics, provide a deep understanding of the model’s strengths and weaknesses under various conditions.
summary: This paper introduces the Social Dynamics Adaptation (SDA) model, a two-stage reinforcement learning approach for enhancing robot social navigation. SDA aims to improve a robot’s ability to navigate alongside humans in dynamic environments, where human trajectories serve as social cues that guide robot movement. In the first stage, the model learns a policy using privileged information about human trajectories, encoding social dynamics into a latent representation. In the second stage, this policy operates without access to these trajectories, relying on inferred social dynamics from the robot’s own state and action history.
Experiments conducted on the Habitat 3.0 platform demonstrate that SDA significantly outperforms state-of-the-art baselines in tracking, following, and maintaining appropriate distances from humans. The study includes analyses of SDA’s effectiveness under noisy conditions, different update rates, and with realistic humanoid behaviors, showcasing its potential for adapting to real-world conditions.
weaknesses: Despite testing under realistic noise and movement in simulation, SDA’s real-world performance remains untested, raising questions about its transferability from simulated to physical environments. Besides, the model assumes a certain pattern of human movement and interaction, which may limit its effectiveness in scenarios where human behaviors deviate significantly from training data.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Could you clarify why the problem being solved in this work is hard and important in the introduction? Providing motivating examples early on would help the reader grasp the significance before diving into the methodology, related work or implementation details. Could you revise the introduction to include this context more explicitly?
2. While the paper models sensor noise and signal drops, this may not be enough to claim robust generalization to real-world environments. Could you explain how the model would handle the complexities of real-world social interactions, and consider testing beyond simulated noise scenarios?
3. The paper does not explore how the model adapts to varying human behavior, such as erratic movements. Have you considered testing with more diverse or unpredictable human behaviors? How do you think the model would perform under such circumstances?
4. Given the focus on simulation, could you discuss how the model could be deployed in real-world environments? What are the anticipated challenges of transferring the SDA model to physical robots, and have you considered Sim2Real experiments or alternative testing methods?
5. Could you provide more details on the computational overhead of the two-stage model, particularly during deployment? How do you anticipate the system performing on resource-constrained robots in real-time scenarios, when only stage 2 is run?
6. The current focus is on following a single human in relatively simple environments. Have you considered expanding the model to more complex social settings, such as multi-human interactions?
7. In Figure 4, the text suggests that the behavior clusters are well-separated, but visually, they appear to be overlapping. Could you clarify this discrepancy and perhaps provide a better explanation or adjust the visualization to reflect the claim more accurately?
soundness: 3
strengths: 1. The SDA model cleverly utilizes privileged information (e.g., human trajectories) during training to learn a social navigation policy and then shifts to infer similar dynamics from historical state-action data during deployment. 
2. By modeling human trajectories as latent dynamics, the paper addresses the challenge of partial observability, which is common in robot-centric navigation tasks. The framework effectively models and compensates for missing information, such as losing line of sight of humans, thus proposing to improve the robot's adaptive navigation behavior in dynamic environments.
3. The SDA model is designed to go beyond just avoiding collisions by recognizing and encoding social dynamics, such as keeping an appropriate distance from humans and yielding space when necessary.
summary: The paper introduces the Social Dynamics Adaptation model (SDA) to enhance social navigation, where robots must adapt in real-time to human movements. It proposes a two-stage reinforcement learning framework to infer social dynamics from the robot’s state-action history. In the first stage, the model trains using privileged information, like human trajectories, to learn an optimal motion policy. In the second stage, it operates without direct access to human trajectories, instead inferring social dynamics from past actions and statuses. 
The approach was tested on Habitat 3.0 and achieved state-of-the-art results in finding and following humans. The primary contributions include the SDA framework and extensive testing under realistic scenarios, which bridge the gap between privileged training data and real-world conditions.
weaknesses: 1. The introduction dives too quickly into explaining the simulator, methodology, and related work without clearly stating the problem or why it is hard and important. It lacks motivating examples to contextualize the problem, making it difficult for readers to grasp the significance until section 3. 
2. The paper models sensor noise and signal drops to simulate real-world conditions, but this is not sufficient to claim robust generalization to real-world scenarios. Real-world environments involve complex interactions, unpredictability, and physical constraints that are difficult to replicate in simulation, even with added noise.
3. Although the authors test their model extensively in Habitat 3.0, the paper lacks real-world experiments. Simulation platforms, while powerful, cannot fully capture the complexity and unpredictability of real-world environments. Additionally, real-world computational constraints (e.g., processing speed and memory) were not sufficiently addressed.
4. The paper does not sufficiently explore how variability in human behavior, such as different walking speeds, erratic movements, or group interactions, affects the model’s performance.
5. The SDA model focuses primarily on following a single human in simplified indoor environments, without considering more complex scenarios like navigating among multiple humans or handling other social situations.
6. Figure 4 claims that the behavior clusters are well-separated, but they appear mostly overlapping. 
7. Figure 2 should explicitly define that stage 1 is offline and stage 2 is online. 
8. The paper’s overall presentation could use some work. Many sections of the paper have different writing styles, making the writing really inconsistent, detracting from the message of the paper.


Decision: Accept (Spotlight)