Towards a Biologically Plausible and Efficient Reinforcement Learning Agent

The primary goal of this degree project was to evaluate a selection of biologically plausible reinforcement learning (RL) agents. An agent was assumed to be biologically plausible if it could be constructed using a biologically plausible model of learning. Such a model had to be realizable using an artificial neural network (ANN) with a learning rule that could be implemented within biological constraints. The first step of this project consisted of reviewing the literature on this type of model. Four different models were found. Two of the models were used to construct RL agents: the S-model AR−P weight update rule, and the dual projection BCPNN RL system. The agents were experimentally evaluated by presenting them with a set of challenging tasks. Performance measurements were gathered as they solved the tasks. An implementation of the Sarsa algorithm was used for benchmarking. The evaluations illuminated different strengths and weaknesses of the two agents. Neither agent was found to be very efficient. However, the insights gained could still be used to construct more efficient biologically plausible RL agents in the future. Other insights gained from this project include: (i) a simple and general framework for constructing RL agents from biologically plausible models of learning, and (ii) an evaluation procedure which proved to be very useful in illuminating different features of the agents. Mot en biologiskt plausibel och effektiv agent för förstärkningsinlärning Referat Det primära målet med detta examensarbete var att utvärdera ett urval av biologiskt plausibla agenter för förstärkningsinlärning (RL). En agent antogs vara biologiskt plausibel om den kunde konstrueras utifrån en biologiskt plausibel inlärningsmodell. En sådan modell skulle vara realiserbar med ett artificiellt neuralt nätverk (ANN) vars inlärningsregel kunde implementeras under biologiska inskränkningar. Första steget i exjobbet gick ut på att söka i litteraturen efter den här typen av modell. Fyra olika modeller hittades. Två av modellerna användes för att bygga RL agenter: S-modellens AR−P -viktuppdateringsregel, och det dubbelprojicerande BCPNN RL systemet. Agenterna utvärderades försöksvis genom att låta dem lösa ett antal uppgifter. Data om deras prestation samlades in medan de löste uppgifterna. En implementation av Sarsa-algoritmen användes som måttstock. Experimenten belyste olika styrkor och svagheter hos de två agenterna. Ingen av agenterna ansågs vara särskilt effektiv. Insikterna som uppnåtts kan dock ändå användas för att konstruera effektivare biologiskt plausibla RL agenter i framtiden. Andra insikter som detta projekt har gett upphov till inkluderar: (i) ett enkelt och generellt ramverk för att konstruera RL agenter utifrån biologiskt plausibla inlärningsmodeller, och (ii) en utvärderingsprocedur som visade sig vara mycket användbar för att belysa olika egenskaper hos agenterna. Foreword and Acknowledgments This thesis presents my Master’s project in Computer Science. It was carried out within the research group of Computational Biology & Neurocomputing (CBN) at the Department of Numerical Analysis and Computer Science (Nada), Stockholm University. The project examiner was Professor Anders Lansner, whom I would like to thank for his ideas and advice. Christopher Johansson, my supervisor at Nada, deserves my gratitude for all his support and valuable feedback throughout this project. Instructor Caroline Nordquist at Nada did a great job reviewing and improving the quality of this thesis, thank you. Last, but not least, I want to express my gratitude to my family. Thank you for always being there for me. Thomas Proschinger, Stockholm 2005