On Blame Attribution for Accountable Multi-Agent Sequential Decision Making
Authors: Stelios Triantafyllou
Adish Singla
Goran Radanovic
Venue: NeurIPS 2021 Poster (2021)
Keywords: Accountability
Blame Attribution
Multi-Agent Decision Making
Shapley Value

Abstract: Blame attribution is one of the key aspects of accountable decision making, as it provides means to quantify the responsibility of an agent for a decision making outcome. In this paper, we study blame attribution in the context of cooperative multi-agent sequential decision making. As a particular setting of interest, we focus on cooperative decision making formalized by Multi-Agent Markov Decision Processes (MMDPs), and we analyze different blame attribution methods derived from or inspired by existing concepts in cooperative game theory. We formalize desirable properties of blame attribution in the setting of interest, and we analyze the relationship between these properties and the studied blame attribution methods. Interestingly, we show that some of the well known blame attribution methods, such as Shapley value, are not performance-incentivizing, while others, such as Banzhaf index, may over-blame agents. To mitigate these value misalignment and fairness issues, we introduce a novel blame attribution method, unique in the set of properties it satisfies, which trade-offs explanatory power (by under-blaming agents) for the aforementioned properties.  We further show how to account for uncertainty about agents' decision making policies, and we experimentally: a) validate the qualitative properties of the studied blame attribution methods, and b) analyze their robustness to uncertainty. 

On Blame Attribution for Accountable Multi-Agent Sequential Decision Making
Stelios Triantafyllou MPI-SWS strianta@mpi-sws. org
Adish Singla MPI-SWS adishs@mpi-sws.co org
Goran Radanovic MPI-SWS
gradanoviio0mii-ss55 org
Abstract
Blame attribution is one of the key aspects of accountable decision making, as it provides means to quantify the responsibility of agent for decision making outcome. this paper, we study blame attribution the context cooperative multi-agent sequential decision making. As particular setting of interest, we focus on cooperative decision making formalized by Multi-Agent Markov Decision Processes (MMDPs), and analyze different blame attribution methods derived from or inspired by existing concepts in cooperative game theory. We formalize desirable properties of blame attribution in the setting of interest, and we analyze the relationship between these properties and the studied blame attribution methods. Interestingly, show that some of the well known blame attribution methods, such as Shapley value, are not Banzhaf index, may over-blame agents. To mitigate these value misalignment and fairness issues, we introduce a novel blame attribution method, unique in the set of properties it satisfies, which trade-offs explanatory power (by under-blaming agents) for the aforementioned properties. We further show how to account for uncertainty about agents' decision making policies, and experimentally: validate the qualitative properties of the studied blame attribution methods, and b) analyze their robustness to uncertainty.
while others, such as
body of people holding themselves accountable to nobody, ought not to be trusted by anybody. -Thomas Paine, A philosopher and a political activist.
Introduction
With the widespread usage of artificial intelligence (AI) in everyday life 2 3) accountability has become one of the central problems in the study of AI. Much recent research studied what constitutes accountability in the context of Al and how design accountable AI systems 4 S G. and recent
policies and legislations [7] are increasingly highlighting the importance of accountability, aiming to provide guidelines for developing and deploying accountable AI systems.
Accountability is a relatively broad term, and it typically involves an actor (or multiple actors) justifying their decisions and facing consequences for actions taken 6 g. Hence, two critical aspects of accountability are explainability and blame attribution. Recent work proposed various methods for explaining, interpreting, understanding, and certifying algorithmic decision-man makinn and its outcomes
9 10
12 13
[4]. In this paper we study the other critical aspect of accountability blame
altribution.
Originally, and by modern standards outdated, Thomas Paine used phrasing with word men.
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

In multi- decision making, one of the central roles of blame attribution is assigning blame for undesirable outcomes or, broadly speaking, for the system's 's inefficiency. Prior work on responsibility and blame in AI [5, 16 17 [8] has recognized some of the core challenges in attributing blame, including the fact that disentangling agents' contributions to the final outcome is not a trivial task. Such challenges are particularly prominent in sequential settings where past decisions influence the
future ones [16.
In this paper, we consider the task of allocating score to an agent, which represents the degree of its blame, and reflects its contributions to the total inefficiency of the multi- aaltit system. We focus on cooperative sequential decision making, formalized by multi-agent Markov decision processes (MMDPs) [9]. where the outcome of interest is the expected discounted return of the agents' joint policy. Concretely, given an MMDP and the agents' joint policy (true or estimated), we ask: How to score each agent so that the agents' scores satisfy desirable properties?
To answer this question, we turn to cooperative game theory and consider blame attribution methods that are derived from or inspired by existing concepts in the cost sharing, data valuation, and coalition
formation literature (20 21 22 23 24 25 26 27), such as core (28). Shapley value [29] 30 or
Banzhaf index [31 32). Taking this perspective on blame attribution, we study blame attribution for accontabbl multi-agent sequential decision making. More concretely:
We formalize desirable properties that blame attribution methods should satisfy in cooperative multi-agent sequential decision making. We identify properties that are typically not considered in the cost-sharing literature, yet are important for decision making. In particular, we introduce two novel properties: a) performance monotonicity, which states that, having fixed all the other agents to their policies, the blame assigned to an agent should not increase if the agent adopts a policy that results in a higher expected discounted return (implying that the method is performance- incentivizing); b) Blackstone consistency which states that an agent should not receive a higher blame just because the agents' policies are not fully known to the blame attribution procedure. We characterize the properties of the studied blame attribution methods. We show that some blame assignment methods, such as, Shapley value, are not performance-montoon (and, hence, while others, such as Banzhaf index, may over-blame agents. Moti- vated by these results, we introduce a novel blame attribution method that trade-off's explanatory power (by under-blaming agents) for the aforementioned properties.
iefformannc-incentiiiiii
We provide algorithms for making the studied blame attribution methods socckstone-consistee when the agents' policies are estimated. We also characterize the effect of uncertainty on blame attribution methods.
Using a simulation-basse testbed, we experimentally amalyze the studied blame attribution methods, their qualitative properties, as well as their robustness uncertainty. The experiments showcase the importance of the robustness considerations we study and indicate that typically more efficient blame attribution methods (i.e., those that assign more blame in total) are less robust to uncertainty.
1.1 Other Related Work
Apart from the works mentioned in the previous paragraphs, our work relates to different areas of moral philosophy, law, and AI, and here we highlight some of the most relevant references. Research in moral philosophy and law has extensively studied the problem of blame attribution,
35 36), well as AI actors 37 38 39 40). We take some of the well known principles in moral philosophy and law in determining properties relevant for blame attribution, e.g., Blackstone consistency is inspired by Blackstone's 's ratio (33. In AI, blame
both in terms of human actors 34
attribution has been studied through a more formal lens, utilizing causality 115 16 17 and/or game theory [18 411, and primarily focusing on nuances related to defining notions and degrees of responsibility, blame, and blameworthiness. In contrast, we focus on cooperative sequential decision making, and analyze how different blame attribution methods from cooperative game theory fare under different blame attribution properties. Finally, our work is generally related to the credit assignment problem 42 43 and more specifically to the credit assignment problem in multi-agent reinforcement learning 44 45 46). However, our focus is not on supporting the learning processes of agents by reducing computational and statistical challenges of learning, but on evaluating the agents' contributions to the system's 's inefficiency, ideally in a fair and interpretable manner.
"This property is inspired by Blackstone's 's ratio: "It is better that ten guilty persons escape than that one innocent suffer" [33.
2

---

2
Formal Setting
In this section, we describe our formal setting, based on multi-agent Markov decision processes (MMDPs), and we formally model the blame attribution problem in sequential decision making. This section also introduces set of desirable formal properties of blame attribution methods.
2.1 Preliminaries
We consider a cooperative multi-agent setting, formalized as class of MMDPs M with n agents {1, (.....))) n). Each MMDP in this class is tuple M = (S, [1,....11,,, A, R, P, 7. σ) [99]. where: S is the state space; A = x Ai= is the action space, with A, being the action space of agent i; R is the reward function R: S x A - I specifying the reward obtained when agents {I,..., n take joint action; P specifies transitions with P(s, s') denoting the probability of transitioning to s from when agents (1,...,.1) n take joint action (#1, .... 02)); y the discount factor; and is the initial state distribution. S and A are finite and discrete. A (stationary) joint policy π is mapping S + ((4), where ((A)) a probability simplex over .A, with (a s) denoting the probability of taking joint action a in s. We assume that joint policy π is factorizable into agents' policies, Ty, i... (a|s) = Ti (01 [s) Tr (a, s). Therefore, we can define an agent i's policy n, as mapping from states to distribution of agent i's actions, i.e., H; S - (.A)) We denote the set of all policies by II = x , II, We also define a standard performance measure. The expected discounted return of [Ei= " R(ss, 0,)[s2 ~ o, n.. where the initial state $1 is sampled from a, and the state-joint action pair of time-step I, (st,at), ) obtained by executing joint policy æ. We abuse our notation by denoting J( T- = J [xI,... Th,..., un). Similarly, J [ T-S) = (x²)) for some C [.) where T. ny if & S and T. = ,द S
a joint policy
defined as ((x)
2.2 Blame Attribution
Our goal is to assign blame to agents for failing to jointly achieve optimal performance. Given the agents' behavior policy, denoted by x², the inefficiency of the considered multi- -ggent system can be defined as A = -(=²) - J(T"), where x² E arg max, (TT) is an optimal joint policy. Similarly, we define the marginal inefficiency of subset agents S {l,..., n1 as As = J( x² -s) as well as the marginal inefficiency of an agent i as A; = J(T;"" [ _ - J(x²), where T =|=" =11=" denotes an optimal policy of S (resp. i) assuming all other policies are fixed, i.e., E arg mag,a [TT, ="=s) (resp. TH E arg max,, J[T,, --)))) A blame attribution method is mapping v : .M x 11 + R2 where (M, x²) distributes blame for inefficiency A by assigning score B; to agent i. The output of V, i.e., the blame assignment, is denoted by 3.
(resp. T.
Ts =|/T
Uncertainty considerations. Since the agents' behavior policy x² might not be known to the blame attribution procedure, we also define blame attribution under uncertainty mapping V M P(II= + R%o that outputs blame assignment estimate 3. Here, PIII) represents set whose
elements express the knowledge about n²_ Inspired by the literature on robust MDPs 47
48 49 we
encode such knowledge with uncertainty sets P(=²), one associated to each state P( s), defined
by the set of probability measures on A. We assume that P(m²) ) is consistent with x i.e., T (.1s)
is in P(T", s) and that every (-|s)) in P(rr,,) s) factorizes to (als)) = π [s) -.) T (a s). Therefore, P(T", identifies the set of plausible stochastic actions that agent takes in state s.
2.3 Desirable Properties
Our goal is to specify functions Ti and T such that the blame assignments B and 3 satisfy desirable properties. In the following text denote these properties by R. Below, define properties that are taken from or inspired by the game theory literature 126 50 21), but translated to our setting 5
Validity (Ry): We say that V valid if it never distributes more blame than the observed inefficiency A. More formally, Vi satisfies Ry (resp. e-Ry) if for every M and T², Σ 3, A
(resp. Em 3i < A + c), where 3 = *)).
'Such P(x")) can be derived from data containing agents' trajectories and be based confidence intervals. *=(s)))) could be P s) w.h.p., provided Blackstone consistency in Section 2.3 is similarly adjusted. 'Note that the terminology is slightly different.
3

---

Efficiency (Rg): A more strict condition is that the total distributed blame is equal to A. That is, satisfies R (resp. e-Rx) for every M and x", Σi= B3 A (resp. E 3, where B = (M, x²)).
A S c).
Rationality RR)): Similar validity is rationality, which requires that blame distributed any
subset of agents S is not greater than Ag. That is, V satisfies RR (resp. e-Rg) if for every M,
and S S {I..... n). Eies 3, S As (resp. Eies B, I As + c), where / V (M.+ Symmetry (Rs): We say that VP is symmetric if it treats equal agents equally, i.e., agents that equally contribute to the inefficiency should receive the same blame. More formally, v satisfies Rs (resp. g) if for every M and n², Bg 83 (resp. 13, 3y1 S <) whenever Asu(i) = As(
for all S S {I.... n2) {i.j), where 5 (M, x²).
Invariance (R₂): We say that V is invariant if assigns zero blame to agents who do not marginally contribute to inefficiency. More formally, v satisfies R2 (resp. e-R,) if for every M and T², B4 (resp. & S E) whenever Asu(i) Ag for all S, where B v (M,
Note that > 0 in the definitions of c-R, and that we use these properties in our characterization result for blame attribution under uncertainty. Additionally, we consider two properties that relate the blame attribution output to the MMDP structure and the agents' behavior policies.
We say that V is contribution monooooniioo the blame it assigns to agent depends only on its marginal contributions and monotonically so. More
Contribution monotonicity (Rew)) [5[]:
formally, V satisfies RCM (resp. c-RcM) if for every two (AI', .T*) and (M1 n²² ). 3, 2 8,
(resp. 3, 2 8,2 -)) whenever A Su(s) A Z suto - A for all S, where 3 ((M b and 5² (MM, ,²²).
Performance monotonicity (R We say that Vi is performance-monotone if it does not assign greater blame to agent for adopting a policy that results in an equal or higher performance, assuming the other agents' policies fixed. More formally, consider any MMDP M, and any T"_1. и; and n. such that ( π -i) S J T² _<). We say that V satisfies R Per (resp. Per
3, 2 B (resp. B 2 3 c) where 3 = V (M, (T,, -)) and 3' = (M,, (T/, x²==")))
The above definitions directly extend to V except that we require them to hold for all P(m)). Additionally, we identify the following property for L
Blackstone consistency (Rac): We say that V is Blassstone with v if never attributes more blame to an agent than V. More formally, V satisfies (V)) if for any M, x² and P(a²),
B, 3r where 3 vi (M, T²) and 3 (M,
3 Game-Theoretic Approaches to Blame Attribution
In this section, we study blame attribution methods based on well known game theoretic notions, such as the core [28). Shapley value (29 (30, or Banzhaf index [2]. We also introduce a novel blame attribution method, unique in the set of properties it satisfies. The proofs of our results can be found in the supplementary material.
3.1 Max-Efficient Rationality
We start with a relatively simple blame assignment method that puts rationality as strict condition, and maximizes the efficiency of blame assignment under this constraint. We call this blame assignment method max-efficient rationality. More formally, max-efficient rationality can be defined via the following linear program:
V WMER(M, T²) =
3
S.L. ΣBSAs VS CS (L,...,n), iES
(PI)
where As are precomputed. Max-efficient rationality is inspired by the notion of core, but unlike the core, max- max-efficient rationality does not require RE (efficiency) to hold. It is easy to show that the following properties satisfied by any optimizer of (P V MME
Proposition Every solution to the optimization problem (TT), i.e. V MEE satisfies Ry (validity). RR (rationality) and R₁ (invariance).

---

Since there might exist multiple optimal solutions to (PT), a tie breaking rule might be needed to decide on the method's is output, V MER. We account for this fact in the experiments from Section {) Note that the constraints in (PT) are quite restrictive, leading to blame assignments that typically distribute very little blame in total. The amount of total blame assigned is important for explanatory power. Namely, trivial blame attribution method that assigns the score of 0 to every agent satisfies all of the properties from the previous section except Rg (efficiency), but provides no information regarding the agents' contributions to the outcome.
3.2 Marginal Contribution
Another intuitive blame assignment method is what refer to as marginal contribution. This method simply quantifies an agent's s potential to increase the performance of the system, assuming that the other agents keep their policies fixed. That is, the blame assigned to agent is equal to 3r A. The following properties hold:
(Ai; An) satisfies R.s (symmetry), Rr (invariance), RCM (contribution monotonicity) and Per (performance monotonicity).
Proposition 2. V MMc MMc(M,
Unlike max-efficient rationality, marginal contribution does not satisfy validity, i.e., it can over-blame a group agents by assigning them total score that exceeds the improvement they can achieve, i.e., A. Given that an agent's 's marginal inefficiency is not always a good indicator of the agent's 'sss influence on the system's 's performance, this method can be highly inefficient (distributing very little blame) when coordination among agents is required,
show in Section 5
3.3 Shapley Value and Banzhaf Index
In the context of the sequential decision making setting studied in this paper. Shapley value can be defined as B = sv (M, T²) such that
Ws [ x (s) -J(s)) -|=" , -
B
(1)
SC
where coefficients ws are set to ws = 1811=-1511111 We restate (and in the supplementary material, prove the claim for our setting) a well known uniqueness result for Shapley value:
Theorem 1. A57V V WM,
(nn) where Bi is defined by Eq. ① and
(SIlinnSS) 11) is unique blame attribution method satisfying Re (efficiency), Rg (symmetry) and
n!
RCM (contribution monotonicity). Additionally, sv satisfies Ry (validity) and R2 (invariance).
As we show in Section 5 Shapley value does not satisfy properties Rr (rationality) nor R (performance monotonicity).
Banzhaf index, denoted by yBr, is similar to Shapley value, and in fact, it has the same functional form but different coefficients (ws -²rرر leading to slightly different uniqueness result. The supplementary material discusses Banzhaf index and its properties in greater detail. Here, we note that Banzhaf index equivalent Shapley value for two agents. However, in general, Banzhaf index does not satisfy RE (efficiency). but version of it, called 2-efficiency (52). As it is the case with Shapley value, Banzhaf index does not satisfy Per (performance monotonicity) nor R (rationality). Interestingly, Ry (validity) might also not hold (see Section
3.4 Average Participation
Motivated by the fact that Rer (performance monotonicity) is important for incentivizing good performance and Re (efficiency) is important for explanatory power, we introduce a novel blame assignment method, which be seen as a combination of marginal contribution and Shapley value. We first show the following result, which shows that there is an inherent trade-off between Rer and RE assuming Rs (symmetry) and R, (invariance) hold.
Proposition 3. No blame attribution method Vi satisfies RE (efficiency), R.s (symmetry), Rs (invari- ance) and Per (performance monotonicity).
Given this result, we instead consider two new properties RAF (average efficiency) and RePerM (c- performance monotonicity), which are weaker variants of Rg (efficiency) and Per (performance
5

---

monotonicity) respectively. Importantly, RAE is not satisfied by VP MC and RePer by V sv. In ad- dition, we also consider two variants of ReM (contribution monotonicity): R.ParM (e-participation monotonicity) and M (relative c- c-participation monotonicity). To define the new properties, we introduce a contribution function e M x IT x (1....,)) n. - {0, 13 that indicates whether an agent is pivotal, i.e., marginally contributes to the inefficiency of some subset of n):
0 if (u(()) T -u(()) = J x" n -s) AS E m)
cM, x
otherwise
Alternatively, an agent is pivotal if and only if its Shapley value is strictly greater than 0, i.e., c(M, x², i] 1 [3 > 0j. where 8 = (s,(M, and I (.] is an indicator function. The new properties are then defined as follows:
Average efficiency (RAE): satisfies RAE (resp. cARAB) if for every M and n², Ei= 8, Eeg(t..... 2- As (resp. E 9r Esg 2 As! S c), where 3 = (MM, T²). c-Performance monotonicity (R) Consider any MMDP M, and any xh- T. and n. s.t. J[,,, (1,,th,)) S J( nh -ii) and c(M, -;),, cMM, (T// Th -i). for every j. We say that V satisfies RoPer (resp. --RoPeMM) if Be 2 3r (resp. Be 2 3 t) where B (M, (TT, =")) and 3 (M, (x);
c-Participation monotonicity (ReParM:): Vi satisfies ReParns (resp. (-RePar if for every (M), x²") and (M2, , h2) s.t. c(M', ,,th ,d) c(M², x²2 .i) for every i, 83¹ 2 B,² (resp. "3 B,² c) whenever A SUg) 2 A SSut)) for all S. where B = MMM , and 8² = (M,, bb?) Relative c-participation monotonicity (RneParM): V satisfies RnoParM (resp. c- R if for every (M', x²)) and (M², x²²) s.t. c(M nb² .) = c(M², x²² .i) for every i, 3, 3, 2
BE_ 3x² (resp. 8,² - -,,² 2 BBx - -Be² - e) whenever c(M² Tbb .5) = cMM, b1 kk) and
A (U( A (1) 2 A SU( - (k)) for all S E (1,....n]| (J. (j, k), where 3 = ((M,,x))
and 8² = (MM )²
)²
Before describing the main results of this subsection, we briefly outline the intuition behind the above definitions. RAE (average efficiency) is similar to RE (efficiency), however it requires less total blame to be distributed. Whereas Rg requires that the total blame is equal to the total inefficiency A, RAE requires that the total blame is equal to the average marginal inefficiency of subsets of agents, i.e., the average value of As A to R (performance monotonicity), RePerr (C-performance monotonicity) additionally accounts for the pivotality of agents through contribution function c, treating each set of pivotal agents separate case. RaParM (e-participation monotonicity) accounts for agents' pivotality in a similar manner. Moreover, RePorM resembles contribution monotonicity RCM, but instead of requiring blame monotonicity to hold w.r.t. the agent's 's influence on the marginal inefficiency of subsets S (i.c., Asu()) As), it considers blame monotonicity w.r.t. the marginal inefficiency of subsets that contain the agent (i.e.. A {uu())). Relative c-participation monotonicity R.teParM M is similar to ReParM but its blame monotonicity requirement is based on a pairwise comparison of agents with the same pivotality degree. In particular, ReeParr requires that the blame increase is higher for an agent who is in subsets with a greater marginal inefficiency increase (i.c.. 3, 3, 2 3x - 3x² whenever A Auu 4 sunn 2 A ASu(n) A
Average participation: Now, we describe the new blame assignment method, which we call average participation. This blame assignment method can be defined as B = V APP(M, n²) such that
c(M b .) EjeS c(M, t², +1 Asu())
B =
Σ 1166
w. Σ
(2)
where coefficient is set to - Intuitively, AP equally distributes blame for the marginal inefficiency of a subset of agents among the pivotal agents in that subset. Hence, an agent i that is pivotal receives blame for each subset S U {i} equal to Asuçy divided by the number of pivotal agents in S U {i] and scaled by coefficient w. Agents that are not pivotal, obtain blame. Average participation uniquely satisfies the following properties.
'This average does not include 4g. which is equal to O. Note also that A 2 As for every S S so this average is upper bounded by A.

---

Theorem 2. & AMP(M, = (31,...,a.), where 33 is defined by Eq. and 2n is unique blame attribution method that satisfies RAE (average-eftciencyy, Rs (symmetry), R1 (invari- ance), Po (c-participation monotonicity) and Rer P (relative c-participation monotonicity) Furthermore, L AP satisfies RePerM (c-perforanc monotonicity) and Rv (validity).
Unlike marginal contribution, average participation is valid (never over-blames agents), however satisfies a weaker version of performance monotonicity, Still, this version is not satisfied by Shapley value. On the other hand, Shapley value is efficient, unlike average participation, which satisfies a weaker requirement-avee average efficiency. We also showcase these trade-off in Section S
Blame Attribution under Uncertainty
In this section, study blame attribution methods that do not have direct access to n². As mentioned in Section 2 we focus on the case where the knowledge about x² is defined by the uncertainty set P(rT). ) and defined state-wise so that each state is associated with a of probability measures on A identifying plausible candidates for x² (-1s))
We denote E P(r") if is plausible by P(rr)).
4.1 Shapley Value under Uncertainty
In explaining approaches to handling uncertainty, we focus on Shapley value. Arguably, the simplest way to operate under uncertainty is derive a point estimate of denoted by お S and apply V sy on this estimate to obtain blame assignment = sv (M, #*). Albeit being simple, this approach does not satisfy desirable properties, most notably, Ry (validity) and RBC (Blackstone consistency).
Validity. Now, note that 3 V sv (M, サ*) satisfies Σi=1 B 8= J(T)) - J(7"). Therefore, instead of relying on point estimate nh, we could utilize policy 7² for which -===)=(((*** < A. Namely, in that case 3 = sv (M, 7*) results
blame assignment that satisfies Ry (validity). Since this inequality holds for solution to the optimization problem paX (TT), we obtain:
Proposition 4. Let Tb be a solution to the optimization problem max,e J(T). Then Vsv.v P(=*)) sv (M, 7*) satisfies Rv (validity).
Blackstone consistency. As we show in Section stone consistent w.r.t. syv In particular, although the total blame is never overestimated, an agent might receive higher blame than it would receive under V sv. To ensure Blackstone consistency, we can assign blame to agent equal to min 3," s.t. 8 (M), Together with Eq. ( this implies that agent i's blame is obtained by solving
5 although svvv is valid, it might not be Black-
min xEP(
Σ
Wg
[ +/x (U Jmms x/x , m-))
(P2)
where ws [8]1(n IS - 1)! and Fg" +|TT E arg max J(T's, s). We have the following result:
n2.
Proposition 5. Let 8 be the minimum value of the objective in P2 Then (v, P(r*)) = (8)...,3%) (.) satisfies Rv (validity) and Rac (V sv) (Blackstone consistency w.r.d. V sv (M, T²))-
Note that sV distributes less total blame than &s,. since it takes the worst case perspective for each agent separately, while under v Sv, the blame assigned
agents computed with the same joint behavior policy. Moreover, the objective function in (P2) is more complex than in classical robust MDP settings [47 (8). making classical approaches for robust MDPs hard to apply. In practice, (Bc(Wsv (Wsv), ), but at the expense of distributing less blame to the agents. In our experiments, we solve ei [ +/= Su(i' [T) and maxx (=T)" +/ s- for each subset S and with
can relax (P2) and optimize lower bound of the objective; this preserves
appropriately chosen P'(x") 2 (( (see the supplementary material), and we apply Eq. (I to
7 Such definition implics a rectangularity of the uncertainty set 47 For example, this estimate can be derived from data containing the agents' trajectories.
7

---

obtain the blame assignment. This implies that agent i's blame is obtained by solving
Σ ws
min ((T ग. SU(()) max xEP((*")
(EP)(*))
Other Blame Attribution Methods. Similar approaches also work for other blame assign- ment methods discussed in Section 3 For example, and focusing on Blackstone consistency. T (M,, P(=*)) can be obtained in the same way as Vsv,Bc (M, but with ws T. while L (M, P(TTT))) can be implemented as MC (M,, P(=*)) = (Ā,,..., A.)
where Á; = min [ "T - - maX (T)) Implementing Blackstone consistent
MER BC P(T")) and WAP.BC (PT))) is more nuanced, and we discuss it in the supple- mentary material.
4.2 Characterization Result
Notice that the described Blackstone consistent methods (M, P(x*)) are not guaranteed to satisfy the properties that their counterparts (MM, satisfy. However, as long as ((M,P(*7) and x*) output similar enough blame assignments, properties that hold under (M,, x²) will approximately hold under (M, P(=")). More formally, we have the following results.
Theorem 3. Consider T and V s.f. ((M,)( V(M, .²)||, S for any M, x², and P(r²).
Then if V satisfies property R E (Rv, Rg Ru, Rs. R1, RAE ). @ satisfies c-R. Moreover, if V satisfies property R. E ;RRM, RPerrs, RePer ReParM R RePParas), To satisfies 26-R.
This theorem allows us to quantify the robustness of the blame attribution methods- the closer T is V, the more robust is to uncertainty. Interestingly, a trivial blame attribution method that assigns 0 blame to all the agents is robust in this sense. However, as we already mentioned, this trivial blame assignment not informative as it does not attribute any blame. fact, if agents receive no penalties for bad behavior, such blame attribution method might have adverse effects. We provide a broader discussion on the negative side-effects of ander-blaming the supplementary material. Importantly, this example suggests that efficiency (in broad sense, i.e., how much blame is being distributed) and robustness are sometimes at odds, which we also demonstrate in the experiments.
Experiments
To demonstrate the efficacy of the studied blame attribution methods, we consider two environments, Gridworld and Graph, depicted in Fig. [ and Fig. 2 Both environments adapted from (53 and modified to be multi-agent. The experiments evaluate blame attribution methods along three axis:
Performance monotonicity: First, we test blame attribution methods for the PerM (performance monotonicity) property, which we deem important for accountability. To do that, we consider the gridworld environment: this two-agent environment in which one of the agents, A2. optimizes its policy using model of the other agent, A₁. Importantly, by controlling the correctness of A2's model of A,, we can validate whether a blame attribution method satisfies R Namely, if A2 does not receive the minimum blame when its model of A is the correct model, the corresponding method is not performance incentivizing. i.c., it does not satisfy R
Coordination: Second, we evaluate the efficacy of blame attribution methods when a higher degree of coordination among agents is needed to yield improvements over the baseline behavior. For this, we consider the graph environment, which includes configurations where an agent cannot improve the joint performance by unilaterally changing its policy. Thus, this environment is suitable for evaluating whether blame attribution methods incorporate more nuanced counterfactual reasoning. Robustness: Finally, we evaluate the robustness of blame attribution methods under uncertainty. In this case, both environments (Gridworld and Graph) are used for testing purposes, and we control for the level of uncertainty over the agents' behavior policies.
The supplementary material provides more details on the experimental setup and implementation. Below we provide a more detailed description of the considered environments and discuss our findings.
8

---

Environment 1: This is gridworld environment. in which two agents control the same actor but with different priorities. In the single-agent version of the environment, an agent, agent A1, controls the movement of the actor. In our multi-agent version, there is an additional agent, agent Ag, who can intervene and override A 's actions. The two agents select their actions simultaneously. Cells denoted with S are the initial states, blank cells indicate areas of small negative reward, F cells indicate areas of slightly increased cost and H cells are areas of severe penalty. The cell denoted by G is the terminal state of the environment and has a positive reward. When agent A2 intervenes in some state, the actor takes the action that an optimal policy would select the single-agent mode, but also pays cost of intervention C. The behavior policy n²1 of agent A₁ is parameterized by variable a, which specifies the probability that A takes an action determined by an optimal single-agent policy, instead of its personal policy. The personal policy of A1 is mixture of optimal single-agent policy for correctly specified costs and a single-agent policy that is optimal but for misspecified costs of F and H cells-ir it assumes that they have the same cost the blank cells. A2's behavior policy n²2 optimizes the expected discounted return and is trained with a model of A specified by the true personal policy of Ai and variable a' (not necessarily equal to a).. A2 is meant to rectify potential mistakes of A₁ that could inflict cost greater than C. In R experiments we set a = 0.4. In robustness experiments, we only consider uncertainty over the personal policy of Ai, and set 0.2 and a' 0.5.
H
F
G
Fig. 1. Gridworld
Performance monotonicity: Fig. validates our theoretical results regarding P (performance monotonicity). More specifically. methods AP and V MC assign the minimum blame to A2 when it acts optimally w.r.t. the true policy of A1 i.e.. when a' = a. However, this is not the case for methods V and V BI, which implies that these methods are not incentivizing Ag to act optimally w.r.t. its belief about A1. V EER and V Vi assign the same blame to A2 as V MC and Vv,, respectively.
This is a graph environment in which 4 agents simultaneously select actions. The graph consists of one starting and one terminal node, as well as 8 intermediate nodes that can be grouped according to their index number; nodes with even index number are located on the upper level of the graph and nodes with odd index number on the lower level. At each time-step every agent chooses to take either action 0 and move to the upper level or action and move to the lower level. We test multiple variants of this environment, each of which defines a different reward function. In all variants, the reward at each time-step is +1 if some formation constraint is satisfied and - if not. In the first set of experiments (Coordination), we consider 4 different formation constraints: in formation constraint m (I...., ... 4), least m agents need to select action 1 for the constraint to be satisfied. Each behavior policy オ²; takes action a, = 0 in every node. In the second set of experiments (Robustness), we consider one formation constraint that is satisfied if the agents are arranged equally between the two levels. In states where agents are balanced between the levels, each behavior policy n²; takes the action from the previous time-step with probability pæ in unbalanced states, the action that leads to the level with the least number of agents is taken with probability p:-
Environment 2:
Fig. 2. Graph
Fig, seoss how much blame total the blame attribution methods assign for the four different levels of required coordination (m I,..., 4). Observe, that when the constraint can be satisfied by every agent (m = 1), MC violates Rv (validity). For rn- 2, VASER and arc assign zero blame to all agents, while V BI violates Ry (validity). Although always valid, V AP assigns significantly less blame as m increases. V sv is always efficient, and its total blame does not vary with m. sv, V BI, MC and V AP do not satisfy RR (they assign more total blame than MEER).
Coordination:
Robustness: We test the robustness of the blame attribution methods by controlling the amount of uncertainty in the estimates of the agents' behavior policies. To model uncertainty, we consider maximum estimation error Gmazs and to obtain uncertainty sets P(x*), ), we sample (uniformly at random) (s) such that 1 (s) n (+) || S <maz- Moreover, P(m") ,s) contains all policies n such that [[) - (s) S Gmaz- In our experiments, we take 2h to be the point estimate of nh.
Comparing estimation approaches: Fig. 3b and 3f show how the approaches for estimating V from Section 4 fare under different levels of uncertainty. The point estimate approach typically over-blames an agent and the amount of over- weer-blaming increases with the level of uncertainty. SV.BC never over-blames any agent, but becomes less efficient (in distributing blame) as Emaa increases (Fig. 3f.. sv.v is more efficient than V VVV BC, but violates RBC (Blackstone consistency) (Fig. 3b)

---

MER
Targeted Value
Valid Consistent
SV
MER
Point Esti mate
0.15
Blame A2
L1 Distance
Total Blame
Blame
0.30
0.13
0.05
0.02
0.05
0.00
1.00
0.00
0.02
0.1
Eraux
Emax
Erayy
(a) Gridworld: P.r
(b) Gridworld: SV
(c) Gridworld: L.1 Distance
(d) Gridworld: Blame
Total Blame
15
Blame Agent
Distance
Total Blame
Emar
Emix
Emax
(c) Graph: Coordination
(f) Graph: SV
(g) Graph: I.1 Distance
(h) Graph: Blame
Fig. 3. Experimental results for the Gridworld and Graph environments, Plot (a tests methods for
P.r... Plot (e) shows the effect of varying coordination level. Plots db show the effect of varying Emar in different Shapley value approaches
(b and blame attribution methods
Comparing attribution approaches: Fig. 3c and show for each consistent blame attribution method L from Section 4 the L1 distance between its output and the output of its counterpart V/ ("targeted
assignment"). Fig. add 3h show the total blame assigned by these methods. V APP BC consistently outperforms the other methods in terms the L1 distance from its "targeted assignment". Compared to V AP, V sV.BCC is consistently better in terms of efficiency (in distributing blame). Similar, albeit less prominent effects be seen when comparing V AP.BC and T BI,BC. These results indicate a tendency where efficiency (in distributing blame) and robustness are at odds, as we also discuss in Section 4.2 L ERR BC and V MCBBC assign zero total blame even for smaller Emaz. indicating that they are the least robust to uncertainty.
6 Conclusion
In summary, the focus of our work is to provide an overview of possible computational approaches for attributing blame in multi-agent sequential decision making. We discuss the strengths and weaknesses of different methods in order to guide practitioners and policy makers in designing tools that support accountability. We conclude that there is no single best choice for blame attribution methods, since there are inherent trade-offs among properties that one might consider important. Looking forward, we recognize several research directions that could address the limitations of our results, some of which we highlight here. a) In this work we primarily focused on the agents' joint return as the outcome of interest. However, it is often important to pinpoint actual causes that led to more fine grained outcomes. Utilizing causal perspective would be beneficial in this regard and could link our results to prior work (e.g.. [6). b) We considered model-based approaches to blame assignment. Learning blame attribution directly from data (e.g., with model-free counterfactual RL) might be more practical in settings where an approximate model is hard to obtain. c) More generally, ensuring scalability both in the number of agents and the the richness of environments one of the most important steps for making this work more widely applicable. We deem approaches from multi- agent RI. suitable candidate solutions for resolving this problem. d) We primarily studied blame assignment properties that are taken from or closely relate to those from the game theory literature. This list could be extended and include more principles from moral philosophy and law. For example. in this paper, we adopted a consequentialist approach to blame attribution, focusing on the outcomes of the agents' behavior. Alternatively, one could take a deontological perspective, and focus on the alignment of an agent's 's behavior with a set of rules. We further discuss different perspectives on blame attribution the supplementary material. Finally, we would like to draw particular attention to the fact that there is no universal prioritization of properties that applies to all blame attribution problems and hence treating any generic analysis like ours as panacea without further justification, might have a negative impact to the agents that are being blamed. To that end, we would like to emphasize that we see this work not as a final solution to the blame attribution problem, but as a starting point that shows challenges and trade-offs in distributing blame.
10

---

References
[1] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. bias: There's software used across the country predict future criminals. https://www
Machine
and it's biased against blacks. Sichine-bias-eee Seea--assseeeeetsnneeeeeee www.errrimmmeeteeiaaaanaeaeeeeee
httpss/////ww.o.pbbiccaaa propublica. www.eca.rrrrriillle.
2016.
[2] Amir E. Khandani, Adlar Kim, and Andrew W. Lo. Consumer credit-risk models via machine-learnin algorithms. Journal of Banking & Finance, 34(11):2777-2787 2010.
[3] Andre Esteva, Brett Kuprel, Roberto A. Novoa, Justin Ko, Susan M. Swetter, Helen M. Blau, and Sebastian Thrun. Dermatologist-eeve classification of skin cancer with deep neural networks. Nature, 542(7633): 2017.
[4] Finale Doshi Velez. Mason Kortz, Ryan Budish. Chris Bavitz, Sam Gershman, David O' Brien, Kate Scott, Stuart Schieber, James Waldo, David Weinberger, et al. Accountability of AI under the law: The role of explanation. arXiv preprint 17 2017.
[5] Joshua A. Kroll, Solon Barocas, Edward W. Felten, Joel Reidenberg, David G. Robinson, and Harlan Yu. Accountable algorithms. University of Pennsylvania Law Review, 165:633, 2016.
[6] Maranke Wieringa. What to account for when accounting for algorithms: A systematic literature review on algorithmic accountability. Conference on Fairness, Accountability, and Transparency, pages 1-18, 2020.
[7] European Commission. gence. ethics wttegguuiieninssssssttteeeee
Ethics
Guidelines for Trustworthy Artificial Intelli-
URL: https://ec. europa. www.pa..eediiieeeeeeee eu/digital- sasiaalssingele single-marken mww.et/ee/eeess////////ee
2019. [Online; accessed 15-January-2021]
[8] Mark Bovens. Analysing and assessing accountability: A conceptual framework. European Law Journal, 13(4):447-468, 2007.
[9] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Model-agnostic interpretability of machine learning. arXiv preprint arXiv: 0006.::6660053300 2016.
[10] Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In IEEE Symposium on Security and Privacy, pages 398-617, 2016.
[11] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In International Conference on Neural Information Processing Systems, pages 4768 4777, 2017.
[12] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-211 2019.
[13] Finale Doshi Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprint arXiv: / 700.08868, 2017.
[14] Christoph Dann, Lihong Li, Wei Wei, and Emma Brunskill. Policy certificates: Towards accountable reinforcement learning. In International Conference on Machine Learning, pages 1507-1516, 2019.
[15] Hana Chockler and Joseph Halpern. Responsibility and blame: A structural-model approach. Journal of Artificial Intelligence Research, 22:93-115, 115, 2004.
[16] Joseph Y. Halpem. Actual causality. MiT Press, 2016.
[17] Joseph Y. Halpern and Max Kleiman-Weiner Towards formal definitions of blameworthi- ness, intention, and moral responsibility. In AAAI Conference on Artificial Intelligence, page 1853-1860, 2018.
[18] Meir Friedenberg and Joseph Y. Halpern. Blameworthiness multi-agent settings. In AAAI Conference Artificial Intelligence, pages 525-532, 2019.

---

[19] Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In Conference on Theoretical Aspects of Rationality and Knowledge, pages 195-210, 1996.
[20] John Von Neumann and Oskar Morgenstem. Theory of games and economic behavior (com- memorative edition). Princeton University Press, 2007.
[21] Kamal Jain and Mohammad Mahdian. Cost sharing. Algorithmic Game Theory, 15:385-410, 2007.
[22]
Balcan, Ariel D. Procaccia, and Yair Zick. Learning cooperative games. In International Conference on Artificial Intelligence, pages 475-481, 2015.
[23] Eric Balkanski, Umar Syed, and Sergei Vassilvitskii. Statistical cost sharing. In International Conference on Neural Information Processing Systems, pages 6222 6231, 2017.
[24] Ruoxi Jia, David Dao, Boxin Wang. Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J. Spanos, Towards efficient data valuation based on the Shapley value. In International Conference on Artificial Intelligence and Statistics, pages 1167-1176, 2019.
[25] Anish Agarwal, Munther Dahleh, and Tuhin Sarkar. A marketplace for data: An algorithmic solution. In ACM Conference on Economics and Computation, pages 701-726, 2019.
[26] Yoav Shoham and Kevin Leyton-Brown. Multiagent systems: Algorithmic, gume-theoretic, and logical foundations. Cambridge University Press. 2008
[27] Georgios Chalkiadakis and Craig Boutilier. Bayesian reinforcement learning for coalition formation under uncertainty. In International Joint Conference on Autonomous Agents and Multiagent Systems, pages 1090-1097, 2004.
[28] Donald B. Gillies. Solutions to general non-zero-sum games. Contributions to the Theory of Games, 4:47-85, 1959.
[29] Lloyd S. Shapley. 17. A value for n-person games. Princeton University Press, 2016.
[30] Lloyd S. Shapley and Martin Shubik. A method for evaluating the distribution of power in committee system. The American Political Science Review, 48(3):787-792, 1954.
[31] John F. Banzhaf III. Weighted voting doesn't work: A mathematical analysis. Rutgers Law Review, 19:317, 1964.
[32] John F. Banzhaf III. One man, 3.312 votes: mathematical analysis of the electoral college. Villanova Law Review, 13:304, 1968.
[33] William Blackstone and George Sharswood. Commentaries on the Laws of England. In Four Books. JB Lippincott, 1893.
[34] Thomas M. Scanlon. Moral dimensions. Harvard University Press, 2009.
[35] David Shoemaker. Attributabllity. answerability, and accountability: Toward wider theory of moral responsibility. Ethics, (2(3))602-633) 2011.
[36] Ibo Van de Poel, Lamber Royakkers, and Sjoerd D. Zwart. Moral responsibility and the problem of many hands. Routledge, 2015.
[37] Mark Coeckelbergh. Artificial intelligence, responsibility attribution, and relational justifica- tion of explainability, Science and Engineering Ethics, 26(4):2051-2065 2020.
[38] Steve Torrance. Ethics and consciousness in artificial agents. Al & Society, 22(4):495-521, 2008.
[39] Peter M. Asaro. Robots and responsibility from legal perspective. IEEE, 4(14):20-24 2007.
[40] Gabriel Lima, Nina Grgić-Hlača, and Meeyoung Cha. Human perceptions on moral responsi- bility of AI: A case study in Al-assisted bail decision- makiing. CHI Conference on Human Factors in Computing Systems, pages 1-17, 2021.
12

---

[41] Christel Baier, Florian Funke, and Rupak Majumdar. A thee-theoretic account of responsi- bility allocation. Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, pages 1773-1779, 2021.
[42] Marvin Minsky, Steps toward artificial intelligence. Institute of Radio Engineers, 49(1)88-30,
1961.
[43] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT Press, 2018.
[44] Kagan Tumer and Adrian Agogino. Distributed agent-based air traffic flow management. In International Joint Conference on Autonomous Agents and Multi-Agent Systems, pages 1-8, 2007.
[45] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI Conference on Artificial Intelligence, pages 2974-2982, 2018.
[46] Jianhong Wang. Yuan Zhang, Kae-Kyun Kim, and Yumjie Gu. Shapley Q-value: A local reward approach solve global reward games. In AAAI Conference on Artificial Intelligence, pages 7285-7292, 2020.
[47] Garud N. Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257-280, 2005.
[48] Arnab Nilim and Laurent Ghaoui Robust control of Markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-1) 2005.
[49] Aviv Tamar, Shie Mannor, and Huan Xu. Scaling up robust MDPs using function approximation. In International Conference on Machine Learning, pages 181-189, 2014.
[50] Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D. Procaccia. Handbook of computational social choice. Cambridge University Press, 2016.
[51] Peyton H. Young. Monotonic solutions of cooperative games. International Journal of Game Theory, 14(2):65-72, 1985.
[52] Marcin Malawski. Equal treatment, symmetry and Banzhaf value axiomatizations. International Journal of Game Theory, 31(1):47-61, 2002.
[53] Cameron Voloshin, Hoang M. Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning, arXiv preprint 1911.091 2019.
13


Reviews:
[{"review_id": "fzJLSlXXKPV", "reviewer": "Reviewer_iKJr", "summary": "This paper provides a formal definition of blame for MA MDPs, a set of properties (many from the cooperative game theory literature, but also some for this setting) and an analysis of several methods (both existing methods like Shapley value and a new method that has properties Shapley does not have that are useful in this context. They also discuss uncertainty-oriented extensions to these ideas, and provide an experimental evaluation comparing several of the existing and proposed blame attribution methods.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: Are the tasks or methods new? Is the work a novel combination of well-known techniques? (This can be valuable!) Is it clear how this work differs from previous contributions? Is related work adequately cited?\n\nThe work is of course based in traditional cooperative game theory, but I believe it is new in this setting, with extensions to those concepts fitting for the setting. Previous work is well-cited.\n\nQuality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?\n\nIn general I think things are well supported (see clarity issues below), and just about what one would expect for this kind of paper. I’d like to see a little on the complexity of the new method (since Shapley is unusable in practice for this reason). Examples would be even better.\n\nThe experimental section is dense and a little unmotivated (or unclearly motivated). It is not always clear what the experiments are (the domains are reasonably described but I’m still a bit unclear on the policies of the participants and how they were manipulate, and what is being shown on the graphs. At a high level, you would be better to describe *how* you intend to evaluate, how the reader should interpret the results, and then finally show the evaluations, since what is being evaluated here is (rightly!) quite different from the typical NeurIPS paper showing the performance of learning algorithm X.\n\nClarity: Is the submission clearly written? Is it well organized? (If not, please make constructive suggestions for improving its clarity.) Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)\n\nIn general the paper is quite clear (at least, to someone familiar with the traditional cooperative game theory ideas). However, probably due to space issues, I found the run up to the new method in 3.4 to be extremely dense with very little glossing explanation or examples. The high level idea makes sense but not sure I could re-implement. Same for section 4 (where I am less familiar with current approaches to reasoning over uncertain policies). The notational density also starts to wear the reader down (can’t keep the acronyms all straight…need a little cheat sheet to cut out and place by the paper!). Also, comments on experiments above.\n\nSignificance: Are the results important? Are others (researchers or practitioners) likely to use the ideas or build on them? Does the submission address a difficult task in a better way than previous work? Does it advance the state of the art in a demonstrable way? Does it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\n\nYes, I think this is quite a useful porting of these ideas to this setting--I imagine this could be used as a basis of several different future works.\n\n== Thanks for the clarifications; hopefully those can be added to the final version ==", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes; this is somewhat a core idea in the paper.", "review_text": "", "extra_scores": {"ethical_concerns": "none", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "JijV_74Bt9", "reviewer": "Reviewer_4AjN", "summary": "This paper furthers the study of blame in accountability in cooperative multi-agent systems.  They provide many natural desiderata, some natural attribution techniques, and characterizations of how these relate, as well as how they relate to standard game theory ideas.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed this paper, and thought it was generally well written and complete.  My main substantive concern was that many of the desiderata are directly consequentialist in the sense that you can only be to blame if the event would not have happened if you had changed your action.  It seems like this deviates from intuitions around blame; for instance if a pilot took a nap on the job, but the system malfunctioned in a way that would have been unrecoverable even if they were awake, many would still want to blame the pilot for not satisfying the normative commitments we expect of a pilot.  This notion of blame would be more deontological, and would certainly violate many of these desiderata, but is closer to what people do.  The phrase comes to mind \"no one was fired for buying ibm\"; the claim isn't that ibm gives the highest utility, it's that it is a choice that is hard to blame someone for.\n\nAdditionally, I think there is a more causal objection to the desiderata.  You may want to blame the pilot because it would have been their fault if something else where different.  That is, even though nothing they could do would have changed anything, if the autopilot system had worked, or had failed differently, the pilot would have been able to change the outcome.  In a sense this is blame two counterfactual changes away from reality rather than one.\n\nFlagging these kinds of limitations early could further strengthen this work, as it would clearly set out the domain of inquiry.  \n\nOn an organizational note, I think that the paper would be greatly improved if the experimental environments were introduced early and used as running examples.  By the time we got to the discussion of the behavior fo the rules in the environments I had forgotten most of the rules, but this would not have been the case if the behavior in the environments where discussed more gradually. It would have also made the rules more intuitive.\n\nminor comment: \n* 152 should be \"does not assign less blame for a policy with at least as much performance\"\n\n------\n**Update to Review**\n\nI appreciate the interesting comments and clarifications both in the response to this review and to the other reviews.  I believe incorporating the proposed changes will be valuable to readers and will improve the paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A limitation that you mentioned was the negative externality of unjustly blaming someone, but it is also worth mentioning the negative externality of not blaming someone when you should and thus incentivizing bad behaviors.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Bz3cObDYK1", "reviewer": "Reviewer_vy59", "summary": "In the context of MMDPs, this work presents an overview of blame attribution methods from the game theoretic literature, together with a study of desirable properties. It then contributes a novel blame attribution method to mitigate some of the identified issues. All the methods are compared in an empirical study and analysed in terms of robustness in the face of uncertainty. \n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is nicely written. I could follow along even without being familiar with the literature on blame attribution. \n\nMy first major comment for this work: the paper is written more as a survey, will full overview of approaches and properties, so it is difficult for readers to distinguish what the contribution and the novel concepts are, especially if they are not familiar with all the elements. There is no delimitation between existing literature and newly introduced elements. This chips away from the clarity and quality of the work.\n\nAs far as I can tell, the work is technically sound, with no evident issues.\n\nA second major comment concerns the experimental environments. Can the authors explain the choice of the settings? It was especially difficult to grasp Environment 1, and the reasoning behind the manner with which the setting is extended to multiple agents. Is A1 meant to rectify potential mistakes of A2? It was also not clear how the policies of the agents function. For example 'The personal policy of A2 takes an optimal action with a state dependent probability; otherwise it follows an optimal policy that sets the cost of H cells to the cost of F cells'. Do you mean that the optimal policies assumes that H and F cells have the same cost?\n\nAll in all, it was an interesting study, my only thought is that this work could have a better impact in a longer format, that would also incorporate all the numerous details from the appendix.\n\n\nMinor comments:\n\nLines 115 vs. 119: inconsistent $\\mathbb{R}$\nLine 103: the expected return is what is presented there\n\n---Post-rebuttal---\nThank you for addressing the raised concerns and for all the clarifications. I will raise my score accordingly. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No statement for these aspects is present.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers appreciated the general problem being tackled by this paper, and view it as an interesting one for multiagent systems and accountability/interpretability in AI. The axiomatic approach and the novelty and analysis of the new AP mechanism for blame attribution are both novel and interesting. The reviews express some concerns around presentation and explanation of some of the details (e.g., experiment motivation, the remarks around consequentialist vs. deontic perspective on the problem, etc.). The author response helped clarify a few of these issues, and nudged the consensus to acceptance. It is critical however that the authors make the revisions needed to address/clarify the issues raised. While this type of paper is somewhat a bit outside of the (current) set of mainstream topics for NeurIPS, the topic is of relevance to the community (and connects to interpretability and fairness, etc. that are becoming more mainstream in NeurIPS). 

Decision: Accept (Poster)