MoET: Interpretable and Verifiable Reinforcement Learning via Mixture of Expert Trees

1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) has achieved many recent breakthroughs in challenging domains such as Go ( Silver et al. , 2016 ) . While using neural networks for encoding state representations allow DRL agents to learn policies for tasks with large state spaces , the learned policies are not interpretable , which hinders their use in safety-critical applications . Some recent works leverage programs and decision trees as representations for interpreting the learned agent policies . PIRL ( Verma et al. , 2018 ) uses program synthesis to generate a program in a Domain-Specific Language ( DSL ) that is close to the DRL agent policy . The design of the DSL with desired operators is a tedious manual effort and the enumerative search for synthesis is difficult to scale for larger programs . In contrast , Viper ( Bastani et al. , 2018 ) learns a Decision Tree ( DT ) policy by mimicking the DRL agent , which not only allows for a general representation for different policies , but also allows for verification of these policies using integer linear programming solvers . Viper uses the DAGGER ( Ross et al. , 2011 ) imitation learning approach to collect state action pairs for training the student DT policy given the teacher DRL policy . It modifies the DAGGER algorithm to use the Q-function of teacher policy to prioritize states of critical importance during learning . However , learning a single DT for the complete policy leads to some key shortcomings such as i ) less faithful representation of original agent policy measured by the number of mispredictions , ii ) lower overall performance ( reward ) , and iii ) larger DT sizes that make them harder to interpret . In this paper , we present MOËT ( Mixture of Expert Trees ) , a technique based on Mixture of Experts ( MOE ) ( Jacobs et al. , 1991 ; Jordan and Xu , 1995 ; Yuksel et al. , 2012 ) , and reformulate its learning procedure to support DT experts . MOE models can typically use any expert as long as it is a differentiable function of model parameters , which unfortunately does not hold for DTs . Similar to MOE training with Expectation-Maximization ( EM ) algorithm , we first observe that MOËT can be trained by interchangeably optimizing the weighted log likelihood for experts ( independently from one another ) and optimizing the gating function with respect to the obtained experts . Then , we propose a procedure for DT learning in the specific context of MOE . To the best of our knowledge we are first to combine standard non-differentiable DT experts , which are interpretable , with MOE model . Existing combinations which rely on differentiable tree or treelike models , such as soft decision trees ( Irsoy et al. , 2012 ) and hierarchical mixture of experts ( Zhao et al. , 2019 ) are not interpretable . We adapt the imitation learning technique of Viper to use MOËT policies instead of DTs . MOËT creates multiple local DTs that specialize on different regions of the input space , allowing for simpler ( shallower ) DTs that more accurately mimic the DRL agent policy within their regions , and combines the local trees into a global policy using a gating function . We use a simple and interpretable linear model with softmax function as the gating function , which returns a distribution over DT experts for each point in the input space . While standard MOE uses this distribution to average predictions of DTs , we also consider selecting just one most likely expert tree to improve interpretability . While decision boundaries of Viper DT policies must be axis-perpendicular , the softmax gating function supports boundaries with hyperplanes of arbitrary orientations , allowing MOËT to more faithfully represent the original policy . We evaluate our technique on four different environments : CartPole , Pong , Acrobot , and Mountaincar . We show that MOËT achieves significantly better rewards and lower misprediction rates with shallower trees . We also visualize the Viper and MOËT policies for Mountaincar , demonstrating the differences in their learning capabilities . Finally , we demonstrate how a MOËT policy can be translated into an SMT formula for verifying properties for CartPole game using the Z3 theorem prover ( De Moura and Bjørner , 2008 ) under similar assumptions made in Viper . In summary , this paper makes the following key contributions : 1 ) We propose MOËT , a technique based on MOE to learn mixture of expert decision trees and present a learning algorithm to train MOËT models . 2 ) We use MOËT models with a softmax gating function for interpreting DRL policies and adapt the imitation learning approach used in Viper to learn MOËT models . 3 ) We evaluate MOËT on different environments and show that it leads to smaller , more faithful , and performant representations of DRL agent policies compared to Viper while preserving verifiability . 2 RELATED WORK . Interpretable Machine Learning : In numerous contexts , it is important to understand and interpret the decision making process of a machine learning model . However , interpretability does not have a unique definition that is widely accepted . Accoding to Lipton ( Lipton , 2016 ) , there are several properties which might be meant by this word and we adopt the one which Lipton names transparency which is further decomposed to simulability , decomposability , and algorithmic transparency . A model is simulable if a person can in reasonable time compute the outputs from given inputs and in that way simulate the model ’ s inner workings . That holds for small linear models and small decision trees ( Lipton , 2016 ) . A model is decomposable if each part of a models admits an intuitive explanation , which is again the case for simple linear models and decision trees ( Lipton , 2016 ) . Algorithmic transparency is related to our understanding of the workings of the training algorithm . For instance , in case of linear models the shape of the error surface and properties of its unique minimum towards which the algorithm converges are well understood ( Lipton , 2016 ) . MOËT models focus on transparency ( as we discuss at the end of Section 5 ) . Explainable Machine Learning : There has been a lot of recent interest in explaining decisions of black-box models ( Guidotti et al. , 2018a ; Doshi-Velez and Kim , 2017 ) . For image classification , activation maximization techniques can be used to sample representative input patterns ( Erhan et al. , 2009 ; Olah et al. , 2017 ) . TCAV ( Kim et al. , 2017 ) uses human-friendly high-level concepts to associate their importance to the decision . Some recent works also generate contrastive robust explanations to help users understand a classifier decision based on a family of neighboring inputs ( Zhang et al. , 2018 ; Dhurandhar et al. , 2018 ) . LORE ( Guidotti et al. , 2018b ) explains behavior of a blackbox model around an input of interest by sampling the black-box model around the neighborhood of the input , and training a local DT over the sampled points . Our model presents an approach that combines local trees into a global policy . Tree-Structured Models : Irsoy et al . ( Irsoy et al. , 2012 ) propose a a novel decision tree architecture with soft decisions at the internal nodes where both children are chosen with probabilities given by a sigmoid gating function . Similarly , binary tree-structured hierarchical routing mixture of experts ( HRME ) model , which has classifiers as non-leaf node experts and simple regression models as leaf node experts , were proposed in ( Zhao et al. , 2019 ) . Both models are unfortunately not interpretable . Knowledge Distillation and Model Compression : We rely on ideas already explored in fields of model compression ( Bucilu et al. , 2006 ) and knowledge distillation ( Hinton et al. , 2015 ) . The idea is to use a complex well performing model to facilitate training of a simpler model which might have some other desirable properties ( e.g. , interpretability ) . Such practices have been applied to approximate decision tree ensemble by a single tree ( Breiman and Shang , 1996 ) , but this is different from our case , since we approximate a neural network . In a similar fashion a neural network can be used to train another neural network ( Furlanello et al. , 2018 ) , but neural networks are hard to interpret and even harder to formally verify , so this is also different from our case . Such practices have also been applied in the field of reinforcement learning in knowledge and policy distillation ( Rusu et al. , 2016 ; Koul et al. , 2019 ; Zhang et al. , 2019 ) , which are similar in spirit to our work , and imitation learning ( Bastani et al. , 2018 ; Ross et al. , 2011 ; Abbeel and Ng , 2004 ; Schaal , 1999 ) , which provide a foundation for our work . 3 MOTIVATING EXAMPLE : GRIDWORLD . We now present a simple motivating example to showcase some of the key differences between Viper and MOËT approaches . Consider the N ×N Gridworld problem shown in Figure 1a ( for N = 5 ) . The agent is placed at a random position in a grid ( except the walls denoted by filled rectangles ) and should find its way out . To move through the grid the agent can choose to go up , left , right or down at each time step . If it hits the wall it stays in the same position ( state ) . State is represented using two integer values ( x , y coordinates ) which range from ( 0 , 0 ) —bottom left to ( N − 1 , N − 1 ) —top right . The grid can be escaped through either left doors ( left of the first column ) , or right doors ( right of the last column ) . A negative reward of −0.1 is received for each agent action ( negative reward encourages the agent to find the exit as fast as possible ) . An episode finishes as soon as an exit is reached or if 100 steps are made whichever comes first . The optimal policy ( π∗ ) for this problem consists of taking the left ( right resp . ) action for each state below ( above resp . ) the diagonal . We used π∗ as a teacher and used imitation learning approach of Viper to train an interpretable DT policy that mimics π∗ . The resulting DT policy is shown in Figure 1b . The DT partitions the state space ( grid ) using lines perpendicular to x and y axes , until it separates all states above diagonal from those below . This results in a DT of depth 3 with 9 nodes . On the other hand , the policy learned by MOËT is shown in Figure 1c . The MOËT model with 2 experts learns to partition the space using the line defined by a linear function 1.06x + 1.11y = 4 ( roughly the diagonal of the grid ) . Points on the different sides of the line correspond to two different experts which are themselves DTs of depth 0 always choosing to go left ( below ) or right ( above ) . We notice that DT policy needs much larger depth to represent π∗ while MOËT can represent it as only one decision step . Furthermore , with increasing N ( size of the grid ) , complexity of DT will grow , while MOËT complexity stays the same ; we empirically confirm this for N = [ 5 , 10 ] . For N = 5 , 6 , 7 , 8 , 9 , 10 DT depths are 3 , 4 , 4 , 4 , 4 , 5 and number of nodes are 9 , 11 , 13 , 15 , 17 , 21 respectively . In contrast , MOËT models of same complexity and structure as the one shown in Figure 1c are learned for all values of N ( models differ in the learned partitioning linear function ) .

Summary: The paper proposes an extension to the Viper[1] method for interpreting and verifying deep RL policies by learning a mixture of decision trees to mimic the originally learned policy. The proposed approach can imitate the deep policy better compared with Viper while preserving verifiability. Empirically the proposed method demonstrates improvement in terms of cumulative reward and misprediction rate over Viper in four benchmark tasks.