On Blame Attribution for Accountable Multi-Agent Sequential Decision Making

... a body of people1 , holding themselves accountable to nobody , ought not to be trusted by anybody . —Thomas Paine , A philosopher and a political activist . 1 Introduction . With the widespread usage of artificial intelligence ( AI ) in everyday life [ 1 , 2 , 3 ] , accountability has become one of the central problems in the study of AI . Much recent research studied what constitutes accountability in the context of AI and how to design accountable AI systems [ 4 , 5 , 6 ] , and recent policies and legislations [ 7 ] are increasingly highlighting the importance of accountability , aiming to provide guidelines for developing and deploying accountable AI systems . Accountability is a relatively broad term , and it typically involves an actor ( or multiple actors ) justifying their decisions and facing consequences for actions taken [ 6 , 8 ] . Hence , two critical aspects of accountability are explainability and blame attribution . Recent work proposed various methods for explaining , interpreting , understanding , and certifying algorithmic decision-making and its outcomes [ 9 , 10 , 11 , 12 , 13 , 14 ] . In this paper we study the other critical aspect of accountability – blame attribution . 1Originally , and by modern standards outdated , Thomas Paine used phrasing with the word men . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) In multi-agent decision making , one of the central roles of blame attribution is assigning blame for undesirable outcomes or , broadly speaking , for the system ’ s inefficiency . Prior work on responsibility and blame in AI [ 15 , 16 , 17 , 18 ] has recognized some of the core challenges in attributing blame , including the fact that disentangling agents ’ contributions to the final outcome is not a trivial task . Such challenges are particularly prominent in sequential settings where past decisions influence the future ones [ 16 ] . In this paper , we consider the task of allocating a score to an agent , which represents the degree of its blame , and reflects its contributions to the total inefficiency of the multi-agent system . We focus on cooperative sequential decision making , formalized by multi-agent Markov decision processes ( MMDPs ) [ 19 ] , where the outcome of interest is the expected discounted return of the agents ’ joint policy . Concretely , given an MMDP and the agents ’ joint policy ( true or estimated ) , we ask : How to score each agent so that the agents ’ scores satisfy desirable properties ? To answer this question , we turn to cooperative game theory and consider blame attribution methods that are derived from or inspired by existing concepts in the cost sharing , data valuation , and coalition formation literature [ 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 ] , such as core [ 28 ] , Shapley value [ 29 , 30 ] , or Banzhaf index [ 31 , 32 ] . Taking this perspective on blame attribution , we study blame attribution for accountable multi-agent sequential decision making . More concretely : • We formalize desirable properties that blame attribution methods should satisfy in cooperative multi-agent sequential decision making . We identify properties that are typically not considered in the cost-sharing literature , yet are important for decision making . In particular , we introduce two novel properties : a ) performance monotonicity , which states that , having fixed all the other agents to their policies , the blame assigned to an agent should not increase if the agent adopts a policy that results in a higher expected discounted return ( implying that the method is performanceincentivizing ) ; b ) Blackstone consistency,2 which states that an agent should not receive a higher blame just because the agents ’ policies are not fully known to the blame attribution procedure . • We characterize the properties of the studied blame attribution methods . We show that some blame assignment methods , such as , Shapley value , are not performance-monotonic ( and , hence , performance-incentivizing ) , while others , such as Banzhaf index , may over-blame agents . Motivated by these results , we introduce a novel blame attribution method that trade-offs explanatory power ( by under-blaming agents ) for the aforementioned properties . • We provide algorithms for making the studied blame attribution methods Blackstone-consistent when the agents ’ policies are estimated . We also characterize the effect of uncertainty on blame attribution methods . • Using a simulation-based testbed , we experimentally analyze the studied blame attribution methods , their qualitative properties , as well as their robustness to uncertainty . The experiments showcase the importance of the robustness considerations we study and indicate that typically more efficient blame attribution methods ( i.e. , those that assign more blame in total ) are less robust to uncertainty . 1.1 Other Related Work . Apart from the works mentioned in the previous paragraphs , our work relates to different areas of moral philosophy , law , and AI , and here we highlight some of the most relevant references . Research in moral philosophy and law has extensively studied the problem of blame attribution , both in terms of human actors [ 34 , 35 , 36 ] , as well as AI actors [ 37 , 38 , 39 , 40 ] . We take some of the well known principles in moral philosophy and law in determining properties relevant for blame attribution , e.g. , Blackstone consistency is inspired by Blackstone ’ s ratio [ 33 ] . In AI , blame attribution has been studied through a more formal lens , utilizing causality [ 15 , 16 , 17 ] and/or game theory [ 18 , 41 ] , and primarily focusing on nuances related to defining notions and degrees of responsibility , blame , and blameworthiness . In contrast , we focus on cooperative sequential decision making , and analyze how different blame attribution methods from cooperative game theory fare under different blame attribution properties . Finally , our work is generally related to the credit assignment problem [ 42 , 43 ] , and more specifically to the credit assignment problem in multi-agent reinforcement learning [ 44 , 45 , 46 ] . However , our focus is not on supporting the learning processes of agents by reducing computational and statistical challenges of learning , but on evaluating the agents ’ contributions to the system ’ s inefficiency , ideally in a fair and interpretable manner . 2This property is inspired by Blackstone ’ s ratio : “ It is better that ten guilty persons escape than that one innocent suffer '' [ 33 ] . 2 Formal Setting . In this section , we describe our formal setting , based on multi-agent Markov decision processes ( MMDPs ) , and we formally model the blame attribution problem in sequential decision making . This section also introduces a set of desirable formal properties of blame attribution methods . 2.1 Preliminaries . We consider a cooperative multi-agent setting , formalized as a class of MMDPsM with n agents { 1 , ... , n } . Each MMDP in this class is a tuple M = ( S , { 1 , ... , n } , A , R , P , γ , σ ) [ 19 ] , where : S is the state space ; A = ×ni=1Ai is the action space , with Ai being the action space of agent i ; R is the reward function R : S × A → R specifying the reward obtained when agents { 1 , ... , n } take a joint action ; P specifies transitions with P ( s , a , s′ ) denoting the probability of transitioning to s′ from s when agents { 1 , ... , n } take joint action a = ( a1 , ... , an ) ; γ is the discount factor ; and σ is the initial state distribution . S and A are finite and discrete . A ( stationary ) joint policy π is a mapping π : S → D ( A ) , where D ( A ) is a probability simplex overA , with π ( a|s ) denoting the probability of taking joint action a in s. We assume that a joint policy π is factorizable into agents ’ policies , πi , i.e. , π ( a|s ) = π1 ( a1|s ) · · ·πn ( an|s ) . Therefore , we can define an agent i ’ s policy πi as a mapping from states to a distribution of agent i ’ s actions , i.e. , πi : S → D ( Ai ) . We denote the set of all policies by Π = ×ni=1Πi . We also define a standard performance measure . The expected discounted return of a joint policy π is defined as J ( π ) = E [ ∑∞ t=1 γ t−1R ( st , at ) |s1 ∼ σ , π ] , where the initial state s1 is sampled from σ , and the state-joint action pair of time-step t , ( st , at ) , is obtained by executing joint policy π . We abuse our notation by denoting J ( π′i , π−i ) = J ( π1 , ... , π ′ i , ... , πn ) . Similarly , J ( π′S , π−S ) = J ( π ′′ ) for some S ⊆ { 1 , ... , n } , where π′′i = πi if i /∈ S and π′′i = π′i if i ∈ S . 2.2 Blame Attribution . Our goal is to assign blame to agents for failing to jointly achieve optimal performance . Given the agents ’ behavior policy , denoted by πb , the inefficiency of the considered multi-agent system can be defined as ∆ = J ( π∗ ) − J ( πb ) , where π∗ ∈ arg maxπ J ( π ) is an optimal joint policy . Similarly , we define the marginal inefficiency of a subset of agents S ⊆ { 1 , ... , n } as ∆S = J ( π∗|π b S , π b −S ) −J ( πb ) , as well as the marginal inefficiency of an agent i as ∆i = J ( π ∗|πb i , π b −i ) − J ( πb ) , where π∗|π b S ( resp . π∗|π b i ) denotes an optimal policy of S ( resp . i ) assuming all other policies are fixed , i.e. , π ∗|πb S ∈ arg maxπS J ( πS , π b −S ) ( resp . π ∗|πb i ∈ arg maxπi J ( πi , π b −i ) ) . A blame attribution method is a mapping Ψ : M×Π→ Rn≥0 , where Ψ ( M , πb ) distributes blame for inefficiency ∆ by assigning score βi to agent i . The output of Ψ , i.e. , the blame assignment , is denoted by β . Uncertainty considerations . Since the agents ’ behavior policy πb might not be known to the blame attribution procedure , we also define blame attribution under uncertainty as a mapping Ψ̂ : M×P ( Π ) → Rn≥0 that outputs a blame assignment estimate β̂ . Here , P ( Π ) represents a set whose elements express the knowledge about πb . Inspired by the literature on robust MDPs [ 47 , 48 , 49 ] , we encode such knowledge with uncertainty sets P ( πb ) , one associated to each state s , P ( πb , s ) , defined by the set of probability measures on A . We assume that P ( πb ) is consistent with πb , i.e. , πb ( ·|s ) is in P ( πb , s ) ,34 and that every π ( ·|s ) in P ( πb , s ) factorizes to π ( a|s ) = π1 ( a1|s ) · · ·πn ( an|s ) . Therefore , P ( πb , s ) identifies the set of plausible stochastic actions that agent i takes in state s .

Summary: In the context of MMDPs, this work presents an overview of blame attribution methods from the game theoretic literature, together with a study of desirable properties. It then contributes a novel blame attribution method to mitigate some of the identified issues. All the methods are compared in an empirical study and analysed in terms of robustness in the face of uncertainty.