# Considerations for Developing Robot-Assisted Crisis De-Escalation Practices
## 5 Training the Robot  
Robots designed to engage in crisis de-escalation cannot be programmed in a fixed, algorithmic, rule-based manner, as for the most part, there are not specific, universal rulesets to govern the interaction (e.g., if client's speech volume increases by 25%, then move 1 m away). Even if there were algorithms governing human-to-human de-escalation, whether those would translate into an effective robot-to-human de-escalation is an empirical question. Whether humans will respond similarly to robots as they do to humans engaged in analogous behaviours is also an open empirical question. It is worthwhile noting that humans undergoing training in de-escalation face the same limitations that robots must overcome, with regard to a lack of specific algorithms to guide their behaviour. Though there are some general principles of de-escalation that beginner crisis responders can draw on to inform their initial approach, the development of expertise in this area requires repeated practice and experience. Through an experience-based or \"data-driven\" learning process, the human brain can detect patterns and develop implicit and explicit \"schemas\" and \"heuristics\", or, in the language of computer programming, \"policies\", to guide behaviour in future de-escalation interactions. At this point in time, there are modern machine-learning techniques that are well suited to data-driven strategies for learning. There exists a range of possible approaches, including, e.g., fuzzy logic [54] and supervised and unsupervised neural networks [55; 56]. However, one approach that might be particularly suitable is reinforcement learning. Reinforcement learning [57] is an adaptive form of artificial intelligence in which the machine learns a new policy to follow through the provision of rewards associated with a target system state. A reward function serves as an incentive mechanism to inform the agent (i.e., robot or machine) what is correct and what is wrong, with the goal of the agent being to maximize the total reward. In standard reinforcement learning, the agent performs an action in the environment and then receives the next state and reward. The agent learns by iteratively interacting with the environment. Reinforcement learning has found a wide range of applications from wireless network optimization [58] to power distribution management [59] and the development of socially aware navigation functions for robots [60]. Developing reward functions for robots to learn crisis de-escalation skills is a challenging task, as de-escalation involves a complex series of behaviours, the sequence of which is neither well understood, nor necessarily consistent within and between individuals and contexts. The potential action space for robots training in de-escalation will also be so large that sparse reward environments (i.e., when a very small number of actions return a reward) will likely pose a challenge because selecting actions at random may not achieve the final reward state (i.e., successful de-escalation, however it may be defined). Reward shaping may offer some promise in overcoming this challenge. In this method, a reward function is designed to provide more frequent feedback on appropriate behaviours by rewarding actions that achieve states that are precursors of or close to the final goal state. Still, setting the reward function for de-escalation is more complicated than most, if not all, of the tasks in which reinforcement learning has been successful. For example, when training an agent to play chess or to stack a block on top of another block, the desired end state is more easily defined (i.e., win vs. lose; block balanced on top of other block vs. not). Operationalizing a successful de-escalation intervention is not as easily done. Broadly, the reward function could reinforce actions that are associated with reductions in a person's agitation, but defining what does and does not represent the end state of reduced agitation is layered, depending on both the context and humans involved.