MODELLING TASK BEHAVIOUR USING HIERARCHICAL REINFORCEMENT LEARNING

This thesis deals with the modelling of an agent that learns adequate task behaviour in a set of complex training scenarios. To ensure adequate training one has to present trainees relevant and representative cases and environments that capture the dynamic and complex character of the real world. The goal of the training is that trainees learn tactical and strategic behaviour. Learning from experience alone is hardly possible: trainees need feedback to determine what behaviours are right and how to adapt their behaviour to improve their performance. Feedback on task results gives them an idea of what is right, feedback on task behaviour provides them with knowledge on how to change their task behaviour successfully. To provide the latter type of feedback, one needs to be able to determine adequate task behaviour at any moment in a scenario. For complex scenarios, this forms a major challenge. Krimp (2004) showed that realistic scenarios are computationally too complex to analyse using backward propagation methods. The focus of the study is formulated in a problem statement that reads: To what extent is it possible to create a model that learns adequate task behaviour in a complex scenario using reinforcement-learning methods? From the problem statement eleven research questions are derived, divided into three groups. They are (1) research questions on the FlatRL model, (2) research questions on the HRL model, and (3) research questions on the differences between the two models. To answer the research questions and problem statement, we investigated two different ReinforcementLearning methods: Flat Reinforcement Learning (FlatRL) (Sutton and Barto, 1998) and Hierarchical Reinforcement Learning (HRL) (Mahadevan and Barto, 2003). A Reinforcement-Learning model consists of five elements, viz. (1) a policy-updating technique, (2) an action-selection technique, (3) a description of all state factors, (4) the available actions, and (5) the reinforcement function. The policy-updating technique used in this research is Monte-Carlo value estimation (Sutton and Barto, 1998). The actionselection technique chosen in this research is ε-greedy action selection (Sutton and Barto, 1998). Elements 3 to 5 of the two models are specific per model. They are described in detail in the thesis. The results of the experiments showed that both models were able to learn adequate task behaviour in a chosen scenario. The performance of the task behaviour is determined by whether the primary objective of the tactical scenario was achieved. The primary objective is elaborated in the thesis. The task behaviour learned using the FlatRL model gained an average performance of 74.7%. The task behaviour learned using the HRL model gained an average performance of 82.6%. Advantages of the HRL model are the ability (1) to apply state abstraction, (2) to use more factors for state determination, and (3) to re-use policies learned for routines. In summary, we may conclude that (1) both Flat Reinforcement Learning and Hierarchical Reinforcement Learning can be used to create an agent that learns adequate task behaviour in tactical scenarios, and (2) Hierarchical Reinforcement Learning is the better one because better results are obtained and more factors for state determination can be processed – enabling a closer resemblance to the real world.