# Pedestrian Simulation with Reinforcement Learning: A Curriculum-Based Approach
## 3 Proposed RL Approach
### Reward Function  
As briefly discussed in the Introduction, RL employs a feedback signal from the environment to the trained agents to guide their learning process as a form of weaker substitute for labels in supervised learning approaches. This feedback signal is defined as a _reward function_, which represents a central element of an RL approach, because agents are trained to maximize the accumulated instantaneous rewards associated with their actions. Pedestrian decision-making activities are fairly complex: conflicting tendencies are evaluated and sometimes reconciled quickly, almost unconsciously, while we walk. Sometimes, individual and collective actions are reasonable or at least explainable in retrospective, in a combination of individual and collective intelligence, that however leads to suboptimal overall performance (see, for instance, the above-cited studies by [13; 24]). Given the above considerations and exploiting the available knowledge on this topic, we hand-crafted a reward function. Initially, we defined basic components, i.e., factors that are generally agreed upon as elements influencing pedestrian behavior. In a second phase, we performed a sort of initial tuning of the related weights, defining the relative importance of the different factors. A fully fledged sensitivity analysis was not performed, whichwill be object of future studies. The overall reward function is defined in Equation (4)  
\[\textit{Reward}: \begin{cases}+6&\text{Final target reached}\\ +0.5&\text{Intermediate target reached}\\ -1&\text{Reached a previously reached intermediate target}\\ -0.5&\text{No target in sights}\\ -0.5&\text{Agent in very close proximity}<0.6\text{ m}\\ -0.005&\text{Agent in close proximity}<1\text{ m}\\ -0.001&\text{Agent in proximity}<1.4\text{ m}\\ -0.5&\text{Wall in proximity}<0.6\text{ m}\\ -0.0001&\text{Each step complete}\\ -6&\text{Reached the end of steps per episode}\end{cases}  \tag{4}\]  
The only way to increase the cumulative reward is therefore the reaching of intermediate or final targets. It is not uncommon in RL settings to have a _single_ source of a positive reward, i.e., the achievement of the final goal. Let us remind the reader that our goal was to achieve a _generally and directly applicable, yet plausible_, pedestrian behavioral model. In most situations, pedestrians move within an environment structured in several interconnected rooms, whose overall structure is known (e.g., the building hosting their office or workplace, a school or university they attend, or transport stations they use every day). Using just a single positive reward associated with the achievement of the final goal would require the agent to explore the environment through a training process at the end of which the model would have internalized the environment. By instead allowing the annotation of the environment, there are passages and waypoints that are intermediate targets to be pursued, and this allows agents, through training a macro behavioral specification basically guiding them, to reach intermediate targets until the final one is in sight, and then it should be pursued. However, reaching targets that have been previously visited has a negative reward, because it implies moving back away from the final goal, and it makes it much less reasonable to try to \"exploit\" the reward to reach a formally reasonable but totally implausible policy (i.e., reach as many intermediate targets before reaching the final one before the end of the episode). Negative rewards thus are used to suggest that some actions should not be chosen unless they eventually lead to the final goal (and unless better alternatives do the same): a small negative reward granted due to the simple passage of time is usual, which pushes agents to avoid standing still and to actively look for solutions, but we also have negative rewards due to proxemics (as introduced in the foundational work [25]), and to penalize walking too close to walls (again, unless necessary).