Self Reward Design with Fine-grained Interpretability

1 INTRODUCTION . Reinforcement Learning ( RL ) and Deep Neural Network ( DNN ) have recently been integrated to solve problems with remarkable performance . The deep reinforcement learning greatly improves the state-of-the-art of control and , in the words of Sutton & Barto ( 2018 ) , learning from interaction . Among the well-known successes are ( 1 ) the Deep Q-Network ( Mnih et al . ( 2015 ) ) which enabled machine to play Atari Games with incredible performance , and ( 2 ) AlphaGo which is capable of playing notoriously complex game of Go ( Silver et al . ( 2016 ) ) , which has also been further developed and popularized as being capable of defeating human at pro level . Although DNN has proven to possess great potentials , it is a blackbox that is difficult to interpret . To address this difficulty , various works have emerged , thus we have a host of different approaches to eXplainable Artificial Intelligence ( XAI ) ; see surveys Arrieta et al . ( 2020 ) ; Gilpin et al . ( 2018 ) ; Tjoa & Guan ( 2020 ) . They have shed some lights into the inner working of a DNN , but there may still be large gaps to fill . Note that there is no guarantee that interpretability is even attainable , especially when context-dependent interpretability can be subjective . In this paper , we propose the Self Reward Design , a non-traditional RL solution that combines highly interpretable human-centric design and the power of DNN . Our robot ( representing any artificial agent ) rewards itself through purposeful design of DNN architecture , enabling it to solve problem without training . The solution might be sub-optimal , but the use of trainable DNN modules ( we use pytorch , specifically ) addresses the problem . We show that performance is improved while interpretability is retained . This paper is arranged as the following . We start with clarifications . Then we briefly go through related works that inspire this paper . Then our interpretable design and SRD training idea are demonstrated with a 1D toy example , RobotFish . Following that , we introduce SRD on the main robot navigation problem on the 2D lavaland with a large focus on interpretable design , followed by SRD training and unknown avoidance . Its experimental results will be discussed in the section after , and finally , we conclude with limitations and future works . 2 THIS PAPER FOCUSES HEAVILY ON INTERPRETABLE HUMAN DESIGN . What exactly is this paper about ? Demonstrations of how two reinforcement learning problems are solved in an interpretable manner through self-reward mechanism . Readers will find how we design different components taylored to different problem components . See appendix A.1 for more remarks : the paper has been heavily reorganized to direct readers ’ focus on interpretable design . But what is interpretability ? While there may be many ways to talk about interpretability , interpretability in the context of this paper is fine-grained , i.e . we go all the way down to directly manipulating weights and biases of DNN modules . DNN modules are usually optimized using gradient descent from random initialization , thus the resulting weights are hard to interpret . In our SRD model , the meaning and purpose of each neural network component can be explicitly stated with respect to the environmental and model settings . How do we compare our interpretability with existing explainable deep RL methods ? Since we directly manipulate the weights and biases , our interpretability is at a very low level of abstraction , unlike post-hoc analysis e.g . saliency ( Greydanus et al . ( 2018 ) ) or semantically meaningful high level specification such as reward decomposition ( Juozapaitis et al . ( 2019 ) ) . In other words , we aim to be as transparent as possible , allowing users to understand the model from the most basic unit . Baseline . Our main result on 2D robot lavaland example achieves a high performance of approximately 90 % accuracy with 10 % randomness to allow for exploration . Hence , we believe comparing accuracy performance with other RL methods is irrelevant , and , most importantly , distracting from our focus of interpretability . If possible , we want to compare the level of interpretability . Quantitative comparison is tricky , then how do we compare ? What baseline to use ? We have in fact answered this question in the previous paragraph : our interpretability is fine-grained as we directly manipulate weights and biases . Different design for different contexts . There is probably no cookie-cutter framework for SRD . Solution design that is interpretable according to the above definition necessarily and heavily depends on the context , i.e . the shape of the system , thus we might need different formula of losses . This paper demonstrates how purposeful and deliberate designs work on two RL problems . In our robot fish example , the task is simply to survive , hence the model rewards actions that encourages survival e.g . “ eat when hungry ” . In 2D robot lavaland example , each tile is considered as a potential checkpoint to reach before reaching the final target , thus the model uses tile-based module that rewards the tile relative to other tiles : the preferred tile is scored higher . 2.1 RELATED WORKS : FROM RL TO DEEP RL TO SRD . RL imperfect human design . RL system can be set up by human manually specifying the rewards . Unfortunately , human design can easily be imperfect since the designers might not necessarily grasp the full extent of complex problem . For RL agents , designers ’ manual specification of rewards are fallible , subject to errors and problems such as negative side effect of a misspecified reward ( Clark & Amodei ( 2016 ) ) and reward hacking ( Russell & Norvig ( 2010 ) ) . Dylan ’ s inverse reward design ( IRD ) paper ( Hadfield-Menell et al . ( 2017 ) ) addresses this problem directly : it allows a model to learn beyond what imperfect designers specify . An important component of our solution to imperfect designer problem is the implementation of unknown avoidance , particularly wunknown in our 2D robot . Also see appendix A.2 for Reward Design Problem ( RDP ) . From RL to Deep RL . Not only is human design fallible , it might be very difficult , especially for complex problems . In the introduction , we mention that Deep RL ( DRL ) solves this by combining RL and the power of DNN . However , DRL rewards have been produced by the black-box : they become difficult to understand and thus explainable DRL emerges to address that problem . RL papers that address explainability/interptretability problems have been compiled in survey papers ( eg . Heuillet et al . ( 2021 ) ; Puiutta & Veith ( 2020 ) ) . Saliency , a common XAI method , has been applied to visualize deep RL ’ s mechanism ( Greydanus et al . ( 2018 ) ) . Relational deep RL uses a relational module that not only improves the agent ’ s performance on StarCraft II and Box-World , but also provides visualization on the attention heads useful for interpretability ( Zambaldi et al . ( 2019 ) ) . Other methods to improve interpretability include reward decomposition , in which each part of the decomposable reward is semantically meaningful ( Juozapaitis et al . ( 2019 ) ) ; do refer to the survey papers for several other ingenious designs and investigations into the interpretability of deep RL . From DRL to SRD . Our model is called self reward design because our robot computes its own reward , similar to DRL computation of Q-values . However , human design is necessary to put con- straints on how self-rewarding is performed so that interpretability is maintained . Human designer has the responsibility of understanding the problems , dividing the problems into smaller chunks and then finding the relevant modules to plug into the design in a fully interpretable way ( see for example how we use convolution layer to create the food location detector , section 2.1 ) . Computational Efficiency of SRD . The models only use modules from standard DNN modules such as convolution ( conv ) , deconvolution ( deconv ) and fully-connected ( FC ) layers with few trainable parameters ( e.g . only 180 parameters in Robot2NN ) . With proper choice of initial parameters , we can skip the long , arduous training and optimization processes that are usually required for deep learning to learn unseen concepts . We trade off the time spent on training algorithm with the time spent on human design , thus addressing what Kahn et al . ( 2018 ) called sample inefficiency ( the need for large dataset hence long training time ) in a human-centric way . 2.2 OTHER RELEVANT CONCEPTS . Self-supervised Learning . DRL like Value Prediction Network ( VPN , Oh et al . ( 2017 ) ) is selfsupervised . Exploration-based RL algorithm is applied so that data are gathered real-time for training : the model thus optimizes its reward function . Our model is similar in this aspect . Unlike VPN , however , our interpretable design avoids all the abstraction of DNN . Our SRD training is also selfsupervised in the sense that we do not require datasets with ground-truth labels . Instead , we induce semantic bias via interpretable components to achieve the correct solutions . Also see appendix A.2 . Imagination component . Our pipeline includes several rollouts of possible future trajectories similar to existing RL papers that use imagination components , with differences as the following . Compared to Kalweit & Boedecker ( 2017 ) which optimizes towards target value ŷi = ri+γQ′i+1 ( heavily abbreviated ) , SRD ( 1 ) is similar because agent updates on every imaginary sample available , but ( 2 ) has different , context-dependent loss computation . Also see appendix A.2 for more references . 3 ROBOT FISH : 1D TOY EXAMPLE . Problem setting . To broadly illustrate the idea , we use a one-dimensional model Fish1D with Fish Neural Network ( FishNN ) deliberately designed to survive the simple environment . Robot Fish1D has energy which is represented by a neuron labelled F . Energy diminishes over time . If the energy reaches 0 , it dies . The environment is env = [ e1 , e2 , e3 ] where ei = 0.5 indicates there is a food at position i and no food if ei = 0 . The fish is always located in the first block of env , fig . 1 ( A ) . In this problem , ‘ food here ’ scenario is env = [ 0.5 , 0 , 0 ] which means the food is near the fish . Similarly , ‘ food there ’ scenario is env = [ 0 , 0.5 , 0 ] or env = [ 0 , 0 , 0.5 ] , which means the food is somewhere ahead and visible . ‘ No food ’ scenario is env = [ 0 , 0 , 0 ] . Fish1D ’ s Actions . ( 1 ) ‘ eat ’ : recover energy F when there is food in its current position . ( 2 ) ‘ move ’ : movement to the right . In our implementation , ‘ move ’ causes env to be rolled left . If we treat the environment as an infinite roll tape and env as fish vision ’ s on the 3 immediately visible blocks , then the food is available every 5 block . How to design an interpretable component of neural network ? First , we want the fish to be able to distinguish 3 scenarios previously defined : food here , food there and no food . Suppose we want a neuron that strongly activates when there is food nearby ( name it food here neuron , fh ) , another neuron that strongly activates when there is a food nearby ( name it food there neuron , ft ) , and we want to represent the no-food scenario as ‘ neither fh and ft respond ’ . How do we design a layer with two neurons with the above properties ? We use 1D convolution layer and selective activation function as σsa ( x ) = ϵ/ ( ||x||2 + ϵ ) , as the following . The fh and ft neurons . Define the activation of fh neuron as afh = σsa [ convfh ( env ) ] where convfh , a Conv1D with weight array wfh = [ 1 , 0 , 0 ] and bias bfh = 0.5 . When there is food near the fish , we get yfh = convfh ( env ) = [ 1 , 0 , 0 ] ∗ [ 0.5 , 0 , 0 ] − 0.5 = 0 where ∗ denotes the convolution operator , so afh = σsa ( yfh ) = 1 . This is a strong activation of neuron , because , by design , the maximum value of selective activation function is 1 . We are not done yet . Similar to afh above , define aft . The important task is to make sure that when ‘ there is food there but NOT HERE ’ , aft activates strongly but afh does not . They are wft = [ 0 , 1 , 1 ] , bft = −0.5 . Together , they form the first layer called the Food Location Detector ( FLD ) . Generally , we have used aη = σsa [ convη ( env ) ] ( 1 ) Interpretable FishNN . To construct the neural network responsible for the fish ’ s actions ( eat or move ) , we need one last step : connecting the neurons plus fish ’ s internal state ( energy ) ( altogether [ afh , aft , F ] ) to the action output vector [ eat , move ] ≡ [ e , m ] through FC layer , as shown in fig . 2 blue dotted box . The FC weights are chosen meaningfully e.g . ‘ eat when hungry and there is food ’ and to avoid scenarios like ‘ eat when there is no food ’ . This is interpretable through manual weight and bias setting . Is FishNN ’ s decision correct ? The prefrontal cortex ( PFC ) decides whether FishNN ’ s decision is correct or not . PFC is seen in fig . 2 green dotted box . The name ‘ PFC ’ is only borrowed from the neuroscience to reflect our idea that this part of FishNN is associated with internal goals and decisions , similar to real brain ( Miller et al . ( 2002 ) ) . How do we construct PFC ? First , define threshold activation as τ ( x ) = Tanh ( LeakyReLU ( x ) ) . Then PFC is constructed deliberately in the same way FishNN is constructed in an interpretable way as the following . First , aggregate states from FishNN into a vector v0 = [ afh , aft , F , e , m ] . This will be the input to PFC . Then , v0 is processed using convPFC followed by softmax and threshold activation . The choice of weights and biases can be seen in table 2 in appendix B . With this design , we achieve meaningful activations v1 = [ e1 , m1 , ex ] as before . For example , e1 is activated when “ there is food and the fish eats it when it is hungry ” , i.e . fh = 1 , F < 1 and e is activated relative to m1 . The output of PFC is binary vector [ True , False ] = [ T , F ] obtained from passing v1 through a FC layer FCPFC . In this implementation , we have designed the model such that the activation of any v1 neuron is considered a True response ; otherwise it is considered false . This is how PFC judges whether FishNN ’ s action decision is correct . Self reward optimization . As seen above , fish robot has FishNN that decides on an action to take and the PFC that determines whether the action is correct . Is this system already optimal ? Yes , if we are only concerned with the fish ’ s survival , since the fish will not die from hunger . However , it is not optimal with respect to average energy . We optimize the system through standard DNN backpropagation with the following self reward loss loss = CEL ( z , argmax ( z ) ) ( 2 ) where CEL is the Cross Entropy Loss and z = Σmemi=1 [ T , F ] i is the accumulated decision over mem = 8 iterations to consider past actions ( pytorch notation is used ) . The ground-truth used in the loss is argmax ( z ) , computed by the fish itself : hence , self-reward design . Results . Human design ensures survival i.e . problem is solved correctly . Initially , fish robot decides to move rather than eat food when F ≈ 0.5 , but after SRD training , it will prefer to eat whenever food is available , as shown in fig . 3 . New equilibrium is attained : it does not affect the robot ’ s survivability , but fish robot will now survive with higher average energy .

Summary: This paper proposes using individually designed, interpretable neural networks to solve a given task. Each neuron in the network is hand-designed to serve a specific task-dependent purpose. These hand-designed solutions are then optimized using environment interactions. The neuron level design of the network is used to provide human interpretability, as the behavior of the network can be understood based on the activations during environment interactions. The neural network structure of the policy can then be optimized via gradient descent to provide a performance improvement over the hand-designed solution.