# Pedestrian Simulation with Reinforcement Learning: A Curriculum-Based Approach
## 3 Proposed RL Approach
### Agent Perception Model  
Agents are provided with a limited set of _projectors_ of rays, each extending up to a certain distance (10 m in these experiments) and providing specific information about what is \"hit\" by the ray and the associated distance from the agent. Projectors (and therefore rays) are not uniformly distributed around the agent; they are more densely present in front of the pedestrian to loosely resemble real human visual perception. The angle between the rays and the direction an agent is facing (both positive and negative) follows the rule described in Equation (1):  
\[\alpha_{i}=Min(\alpha_{i-1}+\delta*i,\ max\_vision) \tag{1}\]  
where \(\delta\) was set to 1.5, _max_vision_ to 90 and \(\alpha_{0}\) to 0. As a consequence, projectors emit rays at 0\({}^{\circ}\), \(\pm\)1.5\({}^{\circ}\), \(\pm\)4.5\({}^{\circ}\), \(\pm\)9\({}^{\circ}\), \(\pm\)15\({}^{\circ}\), \(\pm\)22.5\({}^{\circ}\), \(\pm\)31.5\({}^{\circ}\), \(\pm\)42\({}^{\circ}\), \(\pm\)54\({}^{\circ}\), \(\pm\)67.5\({}^{\circ}\), \(\pm\)82.5\({}^{\circ}\), and \(\pm\)90\({}^{\circ}\). Figure 2 graphically depicts this distribution. The overall number of projectors and rays is therefore 23, but the information associated with and conveyed by rays is different for different objects:  
* _Walls and targets_ are associated with four observations that are intended to support basic navigation of the environment (i.e., choice of the direction to be followed): (i) a distance (a numeric value normalized employing a ReLU-inspired function to be between 0 and 1, with distances above 10 m capped at 1), and a one hot encoding indicating if the perceived entity is a wall (ii), a target still not visited (iii), or an already visited one (iv); as shown in Figure 2 this type of ray is not blocked by agents. * _Agents and walls_ are associated with additional information, whose intended meaning is instead to support the regulation of distance between the agent and nearby heterogeneous entities that may generate an impact (which should be avoided): rays bringing this kind of information about agents and walls are therefore associated with a distance (i) (analogous to the one in the previous type of ray), a Boolean type (agent or wall) (ii), and optional information about direction of movement (iii) and walking  
Figure 2: Rays and provided information: yellow = agent, cyan = intermediate target, green = final target, and transparent = wall or none of the others. Pedestrian agents are depicted in red. speed (only relevant for agents) (iv); unlike the previous type of ray, this one is blocked by the first entity that would cause a collision. Some of the acquired information is therefore potentially doubled: in case a wall is hit by the ray, its presence and distance are highlighted by the two different ray types. In the future, we can attempt to remove references to walls in the second type of ray, but the present results show that this redundancy does not seem to cause training convergence issues. Other relevant information for the agent observation is its own current walking speed. To improve the performance of the neural networks typically employed in recent RL algorithms all observations, in addition to those for which normalization has already been introduced, current walking speed was normalized in the interval [0,1]. For the normalization of walking speed, we set the maximum velocity for agents to be 1.7 m/s. The overall agent's observation is summarized in Table 1. Essentially, the overall number of observations for each RL pedestrian agent was 185 (1 due to its own previous walking speed, 23 rays each associated with 4 observations for walls and targets, and 23 rays associated with 4 observations for agents and walls). The overall agent perception model is therefore a simplification of real human perceptive capabilities: the discrete nature of the projected rays makes it possible that, in some situations, objects (obstacles or other agents) might be not immediately perceived, especially when they are not in the center of the field of view.