Distributional Reward Estimation for Effective Multi-agent Deep Reinforcement Learning
Authors: Jifeng Hu
Yanchao Sun
Hechang Chen
Sili Huang
haiyin piao
Yi Chang
Lichao Sun
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-Agent Reinforcement Learning
Reward Uncertainty

Abstract: Multi-agent reinforcement learning has drawn increasing attention in practice, e.g., robotics and automatic driving, as it can explore optimal policies using samples generated by interacting with the environment. However, high reward uncertainty still remains a problem when we want to train a satisfactory model, because obtaining high-quality reward feedback is usually expensive and even infeasible. To handle this issue, previous methods mainly focus on passive reward correction. At the same time, recent active reward estimation methods have proven to be a recipe for reducing the effect of reward uncertainty. In this paper, we propose a novel Distributional Reward Estimation framework for effective Multi-Agent Reinforcement Learning (DRE-MARL). Our main idea is to design the multi-action-branch reward estimation and policy-weighted reward aggregation for stabilized training. Specifically, we design the multi-action-branch reward estimation to model reward distributions on all action branches. Then we utilize reward aggregation to obtain stable updating signals during training. Our intuition is that consideration of all possible consequences of actions could be useful for learning policies. The superiority of the DRE-MARL is demonstrated using benchmark multi-agent scenarios, compared with the SOTA baselines in terms of both effectiveness and robustness.

Distributional Reward Estimation for Effective Multi-Agent Deep Reinforcement Learning
Jifeng Hu
Yanchao Sun?
Hechang Chen 3+
Sili Huang*
Haiyin Piao Yi Chang" Lichao Sun 7+
School Artificial Intelligence, Jlilin University, Changchun, China
2 Department of Computer Science, University of Maryland, College Park, MD 20742, USA
5 Northwestern Polytechnical University, Xian, China
"Lehigh University, Bethlehem, Pennsylvania, USA
1, 1.2nuff210maill jlu edu. huangs1210mails jlu. edu.
"ycs@umd.edu
3,6 chenhce juu. edu. cn, yichang@j edu. cn saaiyiniioomall 71 22 igghige.duu
edu.
Abstract
Multi-agent reinforcement learning has drawn increasing attention in practice, e.g.. robotics and automatic driving, as can explore optimal policies using samples generated by interacting with the environment. However, high reward uncertainty still remains a problem when we want to train satisfactory model, because ob- taining high-quality reward feedback usually expensive and even infeasible. To handle this issue, previous methods mainly focus on passive reward correction. At the same time, recent active reward estimation methods have proven to recipe for reducing the effect of reward uncertainty. In this paper, we propose a novel Distributional Reward Estimation framework for effective Multi-Agent Reinforce- ment Learning (DRE-MARL). Our main idea is to design the multi-action-brandd reward estimation and policy-weighted reward aggregation for stabilized training. Specifically, design the multi-action-brame distributions on action branches. Then we utilize reward aggregation to obtain stable updating signals during training. Our intuition is that consideration of all possible consequences of actions could be useful for learning policies. The superi- ority of the DRE-MARL is demonstrated using benchmark multi-agent scenarios, compared with the SOTA baselines in terms of both effectiveness and robustness.
reward estimation to model reward
Introduction
Multi-agent reinforcement learning (MARL) has achieved substantial successes solving real-time competitive games [30], robotic manipulation [1] autonomous traffic control [32], and quantitative trading strategies [15]. Most existing works require agents to receive high-quality supervision signals, ie., rewards, which either infeasible expensive to obtain in practice [61]. The rewards provided by the environment are subject to multiple kinds of randomness. For example, the reward collected from sensors on robot will be affected by physical conditions such temperature and lighting, which makes the reward full of bias and intrinsic randomness. The interplay among agents will result in more reward uncertainty. For instance, though agent executes the same action under the same state, the reward can still vary because other agents can execute other actions. Thus, handling non-
CCoreesponiing Author. Email address: chenhe @ jlu.edu.cn (H. Chen) and lis221 @lehigh.edu (L. Sun).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

stationary rewards in the learning process is a necessity for successfully learning complex behaviors in multi-agent environments.
in reinforcement learning (RI.) that consider the reward uncertainty during training. The traditional stream of research non-stationary feedback focuses on passive reward correction. For example, Wang et al. [60] adopt the confusion matrix to model the reward uncertainty and obtain a stable reward for learning. While in [[], the authors recover the true supervision signals with peer loss, which punishes over-agreement for avoiding overfitting. However, the assumption of the reward uncertainty of the above works is restrictive. For instance, the reward flipping mechanism [60] restricts the randomness of the reward to countable value set. Recently, another stream of research focuses on active reward estimation. To alleviate the reward uncertainty issue, some works have treated reward estimation as a point-to-pos to-ponnt regression problem where each state-action pair mapped to a reward [38 64]. However, the reward uncertainty, particularly caused by agents' interplay, can not be fully resolved by the point-to tointtt--oiint reward estimation MARL. This is because the regression is good at the one-to-one mapping between state and rewards. But in MARL, the same state-action pair input of one agent will lead to multiple environmental rewards (i.c., one-to-many mapping), which is intractable for regression and thus hurts the performance. Moreover, these methods do not consider the fact that the reward uncertainty comes not only from inherent environment randomness but also from the interplay among agents. These two factors cause blended influence on the received reward and increase the training difficulty. As experimentally show later (See Figure 3 (left) for details.), such a point-to- siinttt--pointt strategy leads to a worse suboptimal outcome with the increasing agent number and the reward uncertainty degree BT.
In fact, there exist numerous works 23
28
61
In this work, we aim to develop distributional reward estimation followed by aggregation for MARL. Intuitively, our idea is like one human constructs reward blueprint of all ac- tion branches in his brain and thoughtfully makes a decision by considering all possible consequences. Traditional methods 27 41 evaluate the policy only with environment rewards, which will bring about more uncertainty when training the critic because they only take into account the reward ri received after taking specific action, i.e., k-th action dg., from the policy. While in our method, we consider not only the environmental rewards but also the potential rewards on other action branches to perform more stable critic updating and thus achieve better performance. So we propose multi-action-
reward
branch distributional reward estimation to model reward distributions { (o', at j== ) on all action branches, where of is the observation and di is the k-th action. Then we aggregate the environment reward F4K and rewards sampled from different brinnch distributions {R' (o', ai))) I 1,mph by weighting them according to the corresponding action selection probability of the current policy, We will obtain the aggregated mix reward It' and lumped reward 7 for training each agent through the reward aggregation. Reward aggregation enables the agent to evaluate its historical decisions thoughtfully. thus providing sophisticated and effective way to reduce the influence of reward uncertainty. Such a model covers the reward uncertainty in MARL environments and achieves better performance in several MARL benchmarks from small to large agent numbers.
Our contribution is foree-fold. 1) We propose a novel framework, called Multi-agent Distributional Reward Estimation (DRE-MARL), to systematically characterize the reward uncertainty in MARL by modeling reward distributions for all action branches. To the best of our knowledge, this the first effort to solve the reward uncertainty of MARL with multi-action- estimation followed by reward aggregation. 2) Policy-weighted reward aggregation is developed in our framework which enables us to perform stable training of the critic and actors. Besides, DRE-MARL is a universal framework that can be expediently integrated into other MARL methods. 3) We validate the performance of our algorithm with function approximation and mini-batch update via extensive simulations in MARL scenarios with different agent numbers and reward uncertainty.
distributional reward
2 Related Work
Reward Uncertainty.
Dealing appropriately with reward uncertainty has received quite a bit of
attention in recent reinforcement learning studies [6] 60 29 16 47 of this line of works is to assume access to the knowledge of the noise and define unbiased surrogate loss functions to recover the true loss or reward. A typical seminal work dates back to [28) which recovers the true loss from the noisy label distribution with the knowledge of the noise rates of labels.
24 [7 228 42 59]. The core idea
Follow-up works offer solutions to estimate the uncertainty level from model predictions 3 65 67
2

---

or clusterable representations [66]. A couple of recent works 660 67 29 have also looked into this problem sequential settings. For example, Everitt et al. 6 analyze the potential sources of uncertainty and provide the impossibility result for training under facultative reward uncertainty. Wang et al. [60] consider modeling the reward uncertainty being caused by a confusion matrix and design statistics-based estimation method to cover the uncertainty. Wang et al. [6T] recover the true supervision signals with peer loss, which punishes over-agreement to avoid overfitting. But the method is inefficient in MARL, because the uncertainty is bigger than single agent RL.
52 47
Reward Estimation. A number of prior literature in robotic and non- non-robotic domains have adopted
reward estimation [8] 38 7 44 II 55 35 56 9 ST 46 57 50 25). In most of these cases, the predicted reward is used for planning. They take advantage of the estimated reward in imagination augmented rollouts with a dynamic model (i.e., world model) accompanying the reward estimator. However, in our case, we aim at handling reward uncertainty rather than planning with nulti-action- branch reward estimation and aggregation, thus avoiding learning system dynamics and multi-step imaginary rollouts. Reward estimation can also be performed by reward shaping [63 12 (RS) and inverse reinforcement learning [10] 18 (IRL). But IRL doesn't I account for the reward uncertainty, and RS focuses on efficient exploration. Recently, Romoff et al. (38) proposes to train reward estimator with function approximation alongside the value function. However, it adopts point-to-point reward estimation, which struggles when generalizing to multi-agent settings.
Distributional RL. Distributional RL has recently gained increasing attention due to its powerful capacity for treating RL's uncertainty uncertainty can be dated back to Sobel problem than single RI. Fortunately, many studies focus on it 48
48
[]. Early studies about return 45 In MARI., the uncertainty issue is more challenging
49
34
37
39
33
For example, Sun et al. 48 propose the Mean-Shape Decomposition method and quantile mixture in value decomposition, bridging the gap between distributional RL and value function factorization. Although our work also belongs to the distributional field, we focus on the reward uncertainty of the environment rather than Q-value uncertainty and model the distribution of reward rather than Q compared with [48 49 Besides, we propose reward estimation followed by reward aggregation, which is novel and effective.
49
34
3 Preliminaries
a multi-agent extension of the Markov decision process, with partial observability can be described as the tuple M = (N, S, (A') JecN (O') JieN, P, ; ( JiEN, 7), where N = (1,..., N} is the set of N agents, S is the state space of the environment, A'' is the action space of agent i, O' is the observation space of agent i, P S x A' x x... A + A(S) (A(z)
A Markov game 20 53
represents the space of distributions over 2.) denotes the state transition probability that is a mapping from the current state and the joint action the probability distribution over the state space, and
E [0. 1) is the discounting factor. We consider the fact that the interplay between agents results in distributional reward feedback: instead of observing R' for each agent i, agent i can only observe the reward sampled from the reward distribution Rº S x A X ..... x A + A(R).
At each time t, N agents receive different observations (o..... of and output certain joint action a, {a k E {L,..., K}. where a' is executed according agent i's policy X' O' A(A'). K = |A| is the size of action space. The environment then transitions to 81411 and rewards each agent by r = (st,,, at. 81+1), where R2 is the reward function. The goal of agent is to find the best policy x' that maximizes the total reward received from the environment from a start state an end state. The expected cumulative discounted reward is expressed as E 7" rr]. In this paper, we consider the N agents, represented by π { ,..,, n NY with policies (actors) parameterized by 0 = {O,,... J. Given a specific policy π, we adopt state value function the V to approximate expected return, where 1/ is the parameters of V/* 7
Problem Formulation
4.1 Reward Uncertainty in MARL
Reward uncertainty is prevalent in many real-world scenarios, but its influence is not well solved in the literature. Handling non-stationary reward during training necessary for successfully learning complex behaviors in multi-agent environments. In many practical scenarios, the agents usually
3

---

(c) Critic
Hidden Block Loss Block
Hidden Block Data Flow
(
Signal
Lineer
Attention
R F
Sampling
TD error
Head
(0,0,0)
(b) DRE
(a) Actor
greedy
Multi-action-branci Reward Estimation
Reward Aggregation
π
AD
Mixed reward Aggregation
R
J
a
AD
Lumped reward Agpregation
Replay-
(g(
Environment
Buffer
dx
Figure I: The overall architecture of Multi-agent Distributional Reward Estimation, which contains (a) Decentralized Actors (b) Distributional Reward Estimation (c) Centralized Critic. As is shown in (a). we perform decentralized execution according to each agent's s observation and store experience in the replay buffer. Then in (b), we perform multi-action-branch reward estimation followed by policy-weighte reward aggregation. Finally, as shown in (c), the centralized critic executes training with aggregated rewards and provides advantage signals for actors.
observe non-stationary reward feedback due to the interplay and inherent randomness rather than perfectly receiving the precise reward for training. Consider a multi- environment with high complexity and large state-action space such that it is intractable for agents to explore the entire state space. Suppose that we can only have access to the non-stationary reward. How can the agents learn cooperative behaviors under the reward uncertainty? We should figure out the factors that cause the reward uncertainty. Assume we are given a multi-agent scenario consisting of agents. the reward contains uncertainty that is caused by multiple factors that can be classified into two aspects.
Mutual interaction. The first factor that causes reward uncertainty is mutual interaction between agents. As mentioned in Section [ the interplay between agents can lead to the one-to me-to-many mapping between state and reward. For example, suppose that we need agents to complete a task cooperatively. The agent will receive various rewards because its partner executes other actions even though the agent executes the same action under the same observation.
Natural disturbance. Another aspect lies the natural disturbance the environment. During the training process, the environment contains inherent randomness that ubiquitous in the real world. For instance, the sensor's 's feedback fluctuates due to the variation in temperature and lighting, which makes the reward feedback inaccurate.
In the following section, we focus on the two aspects and propose estimation and reward aggregation for providing stable updating signals for agents. Here, the intuition is that the decision should be executed after carefully considering all possible consequences. Finally, in Section 6, we empirically investigate the validity of our method in different scenarios.
reward
4.2 Reward Estimation
Usually, many MARL scenarios provide precise human- human-designed rewards, which deviate from the actual situation in practice. To simulate the reward uncertainty, define the stochastic rewards that are provided by the environment according to some stochastic processes. That is, the reward is generated from some distributions with certain probability densities [60]. To capture the reward uncertainty and facilitate training, reward estimation is proposed to reduce the impact of stochastic
rewards [60 38 The reward estimator (40), which core component in scenario with stochastic
rewards, evaluates 8, (x, a), or (s, a,,5 8') pairs at time step / and estimates the possible rewards that are used for guiding the agent through the user goal. Romoff et [38] model the task of learning the reward estimator as a simple point-to-point regression problem by function approximation: [(p)) = [[(r- Ri(T; {)) where R((T; s)) is the reward estimator with parameter based on different inputs T (s, (s, a), (s)a,s')).

---

5 DRE-MARL
In this section, we introduce a general training framework, Multi-agent Distributional Reward Estimation (DRE-MARL), for the reward uncertainty problem in MARL, as shown in Figure T We adopt the architecture of centralized training and decentralized execution (CTDE) 22.. which consists of N decentralized actors (ToJEE and centralized critic V. parameterized by 0 and v.
respectively. In practice, we utilize {T)) hew shown in Figure a) to interact with the environment and collect experiences. Figure [Tb shows the proposed distributional reward estimation structure, which consists of two stages: we first perform multi-action--ue branch reward distribution estimation from observed experiences and sample rewards from the reward distributions of action branches. Then we aggregate the environment rewards and sampled rewards to guide the training of the critic and actors. Figure [c) indicates that utilize a graph attention network with multi-head
attention 121 4] to capture the global information from observations (o'.... o²). The centralized
critic can simultancously produce advantage signals for all agents through one forward calculation
during training. For stability, we construct target centralized critic V, 127 43 and N target policies
Tg. parameterized by v and 0, respectively. The target network parameters (e', 0) are only updated with current network parameters (v, 0) and are held fixed between individual updates [19 [54]
5.1 Multi-action-branch Reward Estimation
Modeling reward distributions is challenging because the reward uncertainty caused by agents' interplay grows exponentially with an increase of agent number. One direct method is to model reward distributions on joint state-action space. But this method suffers from huge uncertainty caused by mutual interaction as agent number grows. Besides, in the ablation study (See Figure (left) for details.), we also verify that modeling joint reward distributions cannot resolve the issue.
To handle the above challenge, we propose another method to achieve our goal. We simplify the issue by viewing other agents part of the environment such that only need to focus the reward estimation of each agent instead of all agents. Additionally, inspired by the fact that humans will imagine the potential consequences based on historical experience, we equip each agent with reward estimator Nt to better capture the reward uncertainty and stabilize the training process. Specifically, for agent at time step f, we propose the multi-action-brant reward estimator R (o], a E D to model the reward distribution based on agent i's observation of and k-th action di,3> where 'D represents reward distribution space, ?2 represents the reward distribution of agent i in k-th action branch, and yo is the parameters of R We use Ñ² DD to represent the K estimated reward distributions for agent i. We can achieve the above goal by optimizing the overall objective function J, = E; (vp",, and the loss function of every agent is as follows:
J+ = B(o.0e.r)~~D) log P|rr| R(oo as: s0) fR).
where we omit the superscript of (y)),, D is the replay buffer, and -log |P] is negative log likelihood. Cn is regular term of reward distributions. In practice, we adopt Gaussian reward distribution D(H, or), but R can be easily extended to other distributions. Then L3 can be defined by a. ||o + var(u), where var is the variance of μ. a and B are hyperparameters.
iiri-acctioobbaanee reward estimation enables us to forecast possible rewards of all action branches. It is just like we only think about each specific situation separately. but in practice, we usually make decision considering all possible consequences. Therefore, to evaluate the experience thoughtfully and stabilize training, we propose reward aggregation introduced in the following section to reduce the impact of reward uncertainty during the training process.
5.2 Training with Reward Aggregation
For each agent at each time step, the agent can only obtain a single reward Fhk while executing k-th action dg. But we augment 7 as a built-up reward vector, where the k-th of the vector F and the following parts are the estimated rewards e sampled from Rt. Then we aggregate the built-up reward vector with the weiihy-eeighted operation. We mainly obtain two types of rewards after reward
5

---

aggregation: the mixed reward R E R and the lumped reward F E R. where R is used to update centralized critic V7,y and F used update decentralized actors {Tg)eE
Specifically, for each agent we first define the built-up reward vector m h(f, rk) RKK which constructed by replacing the k-th value of vector f with true environment reward r% because the agent only takes Og during interaction with the environment, where h(-) is defined by h(f, rk)
[fi,...,Tk, Pk. fk] Then the mixed reward R = g(m² ...., m ). # ==(-l)) is calculated by policy- weighted aggregation, where g(+) represents two operations below. 1) MeanOperation (gMo) we
utilize the average value of m " :..., m as the output of g(-). 2) SimpleSelection (gss): we directly select m² for agent to calculate the mixed reward R. Mathematically, R is defined by
(mean (imaam(m... :..., m ) (-1o) g - gMO
Rt = g(m ..... m ). #² =110) =
mm #110)
g = 9ss
Sampling from the reward distributions has been proven to be beneficial for exploration in some But at the same time, it will also introduce more uncertainty during training. Here, we mainly focus on reducing the influence reward uncertainty. Therefore we use the mean value of reward distributions to perform reward aggregation, usually achieving more stable training process. These two schemes, the sampling option and mean option, can be chosen flexibly. It is a tradeofff: we choose the former option while the environment is hard to explore. The latter is more suitable for an environment with more uncertainty but is easy to explore.
settings [18 60 23
Next, the centralized critic updates parametric state value function Vg.v by minimizing the Bellman residual
Je( E (orr R PV (00) [o)
where to and are the parameters of current state value network and target state value network, respectively. Finally, for each agent during each training iteration, the decentralized actors follow the same training scheme of policy optimization with advantage function [41]:
Ja (0) = E min (u (0), clip (u (0),
+ +))) Å, + H)
where u(f) xa(alo) =z(a(o) is the importance weight, e is clip hyperparameter and usually we choose
c 0.2. H(m)) is the policy entropy of agent i, and 1) controls the importance of entropy penalty
objective. For agent i, the advantage value is defined as follows: A = F + VV, /V, - (o),
where 7 1(m ..... N.,,.n. .kk) is the lumped reward that is defined by
mean (m ...., m N)
if I MO
F' = [(m m N
r()) =
mean(m²)
1 ISMO
(5)
rk
if 1 19s
where represents three operations as follows. 1) MeanOperation (lao)): we average on all agents' built-up rewards m' ie to obtain single output. 2) SimpleMean (IsMo): we utilize the mean value of agent i's built-up rewards m² as output. 3) SimpleSelection (Iss): we directly select the environment reward rf as the lumped reward.
Following the above procedure, we can update the reward estimators and policies iteratively. Further details and the full algorithm for optimizing DRE-MARL can be found in Appendix A
Experiments
6.1 Environment Settings
To demonstrate the effectiveness of the proposed method, we provide experimental results in several benchmark MARL environments based on the multi-agent particle environments [22] (MPE) and several variants of MPE. Specifically, we consider cooperative navigation (CN), reference (REF), and treasure collection (TREA) environments from small to large agent numbers. In the following sections, we use CN-g, REF-q, and TREA-q to represent different environment variants, where q (2,3, 7, 10} and the entire environment variants are (CN-3, CN-7, CN-10, REF-2, REF-7,

---

Table 1: Performance comparison of DRE-MARL and several SOTA MARL algorithms under the
dette rdist, and dic- settings. The values represent mean episodic rewards.
DRE-MARL
p2p-MARL
MAPPO
MAAC
OMIX
MADDPG
IQL
Scenario
9
(ours)
02
13
58
3
-154=1190
-167
425
-227+4338
-116847773
1169
-738
CN-q
7
.3070
-3101am
-674 s
32433
-93201199
5092
:1188:2886
-8488
11377
-8555
-21858
-8938
12658
Tdete
-54.49.599
-58+8.68
-85#3174
-69+28300
-38+3300
44:
+95+177.00
REF--
7
-732
-808
-1187 -11874.50
-1120123055
-135921905
-11332m09
-2400
21424
154221184
-3317
-3380
-4037
-3102:1881
-4921 2:449
-458
-57+
-47 1-9523
+250+52552
-390+37.6
-428+11.7
-924+86637
TREA-9
7
26441
12638
-36482m005
-2809
-3324-3709
-2678
-4733+2447
-50761194
52600
68644
1
-8715
-569911008
-8465 1,998
3
2167
118621377
-773g1019
-270
-12634669
-21326.00
-552.1%
CN-q
7
29300
33222112
-7481 100
3558
100788
3617:2400
-6141 1:161.
-2605-277
-879243029
19696+2907
-8986+002
+2169
-9365+1085
-12693+20s6
-57.4:.544
-6328995
-6
-7243056
-280
43g223
-226:300
REF-g
-752
-800
1947
-007013877
189441995
-1182
-55 2514
-1776=17m
-1570=1560
-2558+58.7
-3884
-2968+1210
+4090
-484
447433612
-691 2191
-51347672
427
-44928006
48.9
TREA-
-273331111
-274551188
295661112
-35532377
288241781
5092
-5340.2236
-5515
-691223063
-5306...03
-889743853
-58472147
-6727
477.1
271
1562
98
367.1616
476118112
2645
CN-q
888
-480+4340
1416
585
17122+
802+3456
-24492+401.
10
133-:4100
-2900+7M4
17379=1510
1003-+1004
-1697744269
-122+7468
-59231
2831
213
2472
1061
255
287.15577
1599
REF-y
3323
18861:295
2949
215
2824:1142
2824
-2618448
10
6340
:3400=415
2884+:559
5301
-230+667
5229+458.1
4981 +7200
181 181+1331
2567
+1239
1708
2742=11225
1776+1247
2384++1057
2712+1077
+3505+7977
TREA-9
7
1424
1627+1154.
14473
8342+465
1436443363
15436=w1.2
117986=2991
29193
20437.1909
3187819905
1730618977
271001
32421
11422
-321361900
normalized
9,802
9.104
6.486
8.091
4.762
9,080
1,990
mean performance
REF-10, TREA-3, TREA-7, TREA-10). In all these environments, the reward is set to collaborative, which means the reward is the summation of individual rewards. Detailed experimental description and settings can be found in Appendix B.T.
Reward formulation. To simulate the reward uncertainty in these classic environments, we investigate several reward settings with induced uncertainty as follows. 1) Deterministic reward (raese): in classic environments, the reward is calculated with respect to distance [3 22). Suppose that the feedback is exactly precise. The deterministic reward refers to that the same observation- action pair is mapped to an identical reward. We use rdete to represent this reward setting. 2) Natural disturbance reward (Fdist): reward uncertainty caused by natural disturbance, mentioned in Section I. can be regarded as the overall impact on the received reward because the sources of dynamics are coupled with each other. Similar to prior works [38 60), we consider the natural disturbance reward that is designed by r'ist ~ Wrractee 1)+0005++rr dretee We use r'ist to represent this reward setting. 3) Mutual interaction reward dist)) in order to simulate the reward uncertainty caused by mutual interaction mentioned in Section 4.1] we add different reward distributions different action branches such that the same observation-action pair will result in different rewards. index(a.)
Specifically, the action dist-reward is defined by r r'c dist ~ NN,, 8) + r d'eete where
and 8 0.001. We use rá rac-dist represent this reward setting. Tdist and Tac-dist settings particularly challenging because agents need to conquer more dynamics.
Baselines. In order to verify the efficacy of distributional reward estimation in MARL with reward uncertainty, we select the following methods as baselines: MADDPG (22) adopts a value- based centralized Q network to concatenate and process all agents' observations. MAAC [3] uses attention mechanism in CTDE paradigm. MAPPO 62 reaches good results with on-policy method. QMIX (36] adopts monotonicity to decompose team reward. IQL (58) adopts completely independent training on each agent. Apart from the above methods, we also compare with point-to-poin (p2p- MARL) reward estimation
38 and global reward estimation (GRE-MARL).
Metries. For all methods we consider, we report the final performance under different reward settings (T (TdetesTdist dist--ocdddell died), which are introduced above. Following prior studies [3] 22), we set the episode
7

---

Average
Deterministic Fam
GJDRE-MARI
(MP2)-MARI
(<MADDPP
CEC
Natural disturbance
3: CN-10
4: RRF-2
S: REF-7
REF-10
TREA-10
Mutual interaction reward Tar Sr-llr
(OMAAРP
KINQL
Figure 2: The normalized mean performance under different reward settings (Tdetes Tdista T Tac-dist) Each polygon (shown right) is averaged three sub-polygon (shown left), where each sub-polygon represents the corresponding reward setting. Each vertex of the sub-polygon denotes the normalized performance, which matches clockwise from CN-3 TREA-10. The detailed calculation method of normalized performance can be found Section 6.1
length as 25 for all experiments and use the mean episode reward as the evaluation metric. Because the value of performance in different scenarios varies considerably, we also consider performance normalization based on all methods. For example, if we have performance values (M1, M2, M3) of
three models, then the normalized performance defined by M maz MM, Mg, (MMM) M,. MM) where w enables us to distinguish the differences of methods appropriately. We set w in the experiments. More details about training and implementation provided in Appendix
w(M, 1-4 1oMM.-mmiiaa M2.M4)
6.2 Results and Analysis
Table Tableessows shows the performance different methods with the three reward settings. We report the true value of each experiment. The results show that in most of the environments, DRE-MARL reaches the best performance, which reveals the effectiveness of distributional reward estimation. In several scenarios, such as TREA-7 with Tdete and raists DRE-MARL is not the best, but its performance is very close to the best. Under the Fac-dist reward uncertainty, DRE-MARL fails in TREA-3, TREA-7, and TREA-10. That is because Foe-dist brings based methods in TREA: DRE-MARL and MAAC all adopt multi-head attention architecture, and they are subject to the same trend of changing. We find that in Idete and Tdist settings, MAAC achieves the competitive performance, while our method reaches the best performance. But in the Tac-dist setting, the performance of MAAC is pretty bad, and also, DRE-MARL is not the best one. If we consider the utility of DRE based on attention in the setting of TREA with Fac-dist (i.e., Comparation between DRE-MARL and MAAC), DRE brings at least 50% improvement.
difficulties for attention-
To reveal the comprehensive abilities of above methods, we normalize the performances of all methods and plot the overall performances as shown in Figure 2 The vertexes (I~9) of the polygon represent the corresponding scenarios R.w....FFF.RRRA.AA. gon, average the normalized values of performances under all reward settings "'etteTdisst aaaaa-dia diat- Figure 2 suggests that all scenarios. In REF-2 and TREA-3, the performance of DRE-MARL is just slightly lower than MADDPG because the agent number is small, and it is relatively easy to train. Additionally, The results also illustrate that DRE-MARL has ward uncertainties because the DRE can capture the reward uncertainty and stabilize training under different reward settings. Besides, exhaustive experiments shown in Appendix B
To get the values on vertexes of poly-
consistently achieves the best overall performance almost
robust performance with respect various
6.3 Ablation Study
Firstly, we perform the ablation study investigate the impact different reward estimation methods on the proposed model: L)DRE-MARL: distributional reward estimation. 2) GRE-MARL:

---

DRE-MARL
GRE MARL
p2p-MARL
0.5
15
0.5
15
0.5
-15
0.5
455
-35
0.5
0.0
40
60
12.0
0.
400
80
12.6
0.
40
120
12.0
0.0
8.0
12.0
0.0
4.0
36
120
7.0
0.33
3.
5.0
3.0
1.3
6.0
4.0
6.0
12.0
4.0
8.0
12.5
0.0
4.0
50
12.0
4.0
0.0
4.0
8.0
12.0
-0.0
4.0
54
124
16.0
35.0
TRP4
2.0
12.0
15.0
TREA
12.0
25.0
4.0
Eit 1
Sex Hight? 4 2.2 ctsc 4
des 20000
4.2 (EER dic
477
4 10
4.7
4 10
Figure 3: Left: Ablation study under reward setting Tac-dist- We change the reward estimation strategy, such as global reward estimation (GRE-MARL) and point-to-point reward estimation (p2p- MARL), and probe the influence of different reward estimation methods. Right: Reward aggregation analysis on different aggregated schemes. We evaluate our model with different and frequently reward aggregation strategies, which enable us to assess our method comprehensively.
global reward estimation based on joint state-action pairs. 3) p2p-MARL: point-to- toiponnt reward estimation. As shown in Figure 3 (left). we report the learning curves of the above methods under the reward setting of Fac-diet. The results show that DRE-MARL is able to achieve better asymptotic performance under different scenarios. GRE-MARL fails to capture the reward uncertainty because the joint state-action space of MARL is huge, which brings more difficulties for global reward estimation. p2p-MARL does not consider the reward uncertainty caused by mutual interactions among agents as the one-to many mapping problem, it performs badly by adopting the one-to-one reward estimation method in MARL scenarios.
To probe the effect of different reward aggregation schemes on the performance, we report the performance of several aggregation schemes in ( under the Tac-dist setting. Recall the reward aggregation introduced in Section 5.2) select the following aggregation schemes. I) Iss + gss. 2) ISMO + gMO. 3) Io + 9M0. 4) ss + gss. 5) lsMo (no aggregation). In Figure 3 (right), remark that the utility of reward aggregation, which makes the agent make decisions thoughtfully, consistently improves performance in various scenarios compared with GRE-MARL and p2p-MARL. Among all aggregation schemes, Iss + 9ss reaches the best performance, and the reported results are also based on this aggregation scheme. More details are shown in Appendix B.2
7 Conclusion, Limitations, and Broader Impact
In this paper, we present DRE-MARL, a general and effective reward estimation method for reward uncertainty in MARL. To capture the reward uncertainty and stabilize the training process, we investigate the benefits of distributional reward estimation followed by reward aggregation in MARL when reward uncertainty is present. The intuition is that careful consideration of all possible consequences is useful for learning policies. For our proposed DRE-MARL, we propose two stages to perform distributional reward estimation: we first design the multi-action-brance reward estimation to model reward distributions all action branches. Then we propose pplicy-weighhen reward aggregation with environment rewards and sampled rewards. Here we emphasize that reward aggregation augments the reward from action branch to all action branches and thus provides more precise updating signals for the critic and actors. Experimental results verify that agents learned by our framework can achieve the best performance and show strong robustness on different types of reward uncertainty. In conclusion, we hope that our work could demonstrate the potential of distributional reward estimation improve the capacity of MARL when reward uncertainty is present and encourage future works to develop novel reward estimation methods.
In terms of limitations, DRE-MARL requires prior assumption about the form of reward distribution. Just like tanglesome signals can be factorized into the superposition of simple but elementary sinusoidal signals, we may consider using a cluster of basic distributions as reward distribution in

---

future work. Another limitation is that reward aggregation is only used in discrete action space. Two solutions may be investigated in the future solve this limitation: 1) Discretizing the range of the available action value. 2) Learning a network with actions as inputs, just like how we convert discrete-action DQN [26] into oontiuuuss-action Q network in DDPG [19]. We do not see any negative societal impacts of our work while using our method in practice.
Acknowledgments and Disclosure of Funding
We thank all anonymous reviewers for their constructive suggestions. This work is partially supported in part by the National Natural Science Foundation of China under grants Nos.61902145, 61976102, and U19A2065; the International Cooperation Project under grant No. 20220402009GH, and the National Key R&D Program of China under grants Nos. 20212D01 12501 and 2021ZD01 12502.
References
[1] OpenAl: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob Mc- Grew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[2] Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforce- ment learning. In International Conference Machine Learning, pages 458. PMLR, 2017.
[3] Gilles Blanchard, Marek Flaska, Gregory Handy, Sara Pozzi, and Clayton Scott. Classification with asymmetric label noise: Consistency and maximal denoising. Electronic Journal of Statistics, 10(2):2780-2822 2016.
[4] Haoqiang Chen, Yadong Liu, Zongtan Zhou, Dewen Hu, and Ming Zhang. Gama: Graph attention multi-agent reinforcement learning algorithm for cooperation. AI, 50(12))4195-4205. 2020.
[5] Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforce- ment learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[6] Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforce- ment learning with a corrupted reward channel. arXiv preprint arXiv: 17111004511111 2017.
[7] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning, arXiv preprint arXiv: 1803.00100 2018
[8] Vincent François-Lavet, Yoshua Bengio, Doina Precup, and Joelle Pineau. Combined reinforce- ment learning via abstract representations. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pages 3582-3589, 2019.
[9] Marek Grzes and Daniel Kudenko. Learning shaping rewards in model-based reinforcement learning. In Proc. AAMAS 2009 Workshop on Adaptive Learning Agents, volume 115, page 30. Citeseer, 2009.
[10] Dylan Madeield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
[11] Mikael Henaff, William F Whitney, and Yann LeCun. Model-based planning in discrete action spaces. 2017.
[12] Hlynur Davíë Hlynsson and Laurenz Wiskott. Reward prediction for representation learning and reward shaping. arXiv preprint arXiv:2105.00112, 2021.
[13] Shariq lqbal and Fei Sha. Actor-ateention-eriticc for multi-agent reinforcement learning. In International Conference Machine Learning, pages 2961-2970. PMLR, 2019.
10

---

[14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[15] Jinho Lee, Raehyun Kim, Seok Won Yi, and Jaewoo Kang. Maps: multi-agent reinforcement learning-based portfolio management system. arXiv preprint wrXiv:2007.05402 2020.
[16] Lisa Lee, Ben Eysenbach, Russ R Salakhutdinov, Shixiang Shane Gu, and Chelsea Finn. Weakly- supervised reinforcement learning for controllable behavior. Advances in Neural Information Processing Systems, 53:2661-26777 2020.
[17] Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label-noise learning without anchor points. In International Conference on Machine Learning, pages 1403-6413. PMLR, 2021.
[18] Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-base reinforcement learning. In Deep RL Workshop NeurIPS 2021, 2021.
[19] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: Ni0.:1100.00971, 2015.
[20] Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pages 157-163. Elsevier, 1994.
[21] Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In AAAI, volume 34, pages 7211-7218, 2020.
[22] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competiive environments. In NIPS, pages 6379-6390, 2017.
[23] Rubén Majadas, Javier Gareía, and Fernando Fernández. Disturbing reinforcement learning agents with corrupted rewards. arXiv preprint arXiv:2102.06887, 2021.
[24] Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class- plosapproyyyility estimation. In International conference machine learning, pages 125-134. PMLR, 2015.
[25] Alberto Maria Metelli, Matteo Pirotta, and Marcello Restelli. Compatible reward inverse reinforcement learning. Advances in neural information processing systems, 30, 2017.
[26] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 13X1v:1312.5602, 2013.
[27] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland. Georg Ostrovski, al. Human-level control through deep reinforeement learning. nature, 518(7540):529-55 -33333,, 2015.
[28] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances in neural information processing systems, 26, 2013.
[29] Jan Corazzal Ivan Gavran2 Daniel Neider. Reinforcement learning with stochastic reward machines. 2022.
[30] OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak. Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz. Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota with large scale deep reinforcement learning. CoRR, abs/1912.066 066600 2019. URL

---

[31] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan. and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
[32] Praveen Palanisamy. Multi-agent connected autonomous driving using deep reinforcement learning. In 2020 International Joint Conference on Neural Networks (ICNN), pages 1-7. IEEE, 2020.
[33] Jorion Philippe. Value at risk: the new benchmark for managing financial risk, NY: McGraw-Hill Professional, 2001.
[34] Wei Qiu, Xinrun Wang, Runsheng Yu, Rundong Wang. Xu He, Bo An, Svetlana Obraztsova, and Zinovi Rabinovich. Rmix: Learning risk-sensitive policies for cooperative reinforcement learning agents. Advances Neural Information Processing Systems, 34:23049- 2021.
[35] Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomenech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al. augmented agents for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[36] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference Machine Learning, pages 4295-4304. PMLR, 2018
[37] R Tyrrlll Rockafellar, Stanislav Uryasev, et al. Optimization of conditional value-at-risk. Journal of risk, 2:21-42, 2000.
[38] Joshua Romoff, Peter Henderson, Alexandre Piché, Vincent Francois-Lavet, and Joelle Pineau. Reward estimation for variance reduction in deep reinforcement learning. arXiv preprint urXiv:1805.03359 2018.
[39] Mark Rowland, Robert Dadashi, Saurabh Kumar, Rémi Munos, Mare G Bellemare, and Will Dabney. Statistics and samples in distributional reinforcement learning. In International Conference on Muchine Leurning, pages 5528-5536. PMLR, 2019.
[40] Tim GJ Rudner, Vitchyr Pong, Rowan McAllister, Yarin Gal, and Sergey Levine. Outcome- driven reinforcement learning via variational inference. Advances in Neural Information Processing Systems, 34, 2021.
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
[42] Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In Artificial Intelligence and Statistics, pages 838-846. PMLR, 2015.
[43] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. PMLR, 2014.
[44] David Silver, Hado Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac-Arnoid, David Reichert, Neil Rabinowitz, Andre Barreto, et al. The predictron: End-to- end learning and planning. In International conference on machine learning, pages 3191-3199. PMLR, 2017.
[45] Matthew Sobel. The variance of discounted markov decision processes. Journal of Applied Probability, 19(4):79-802, 1982.
[46] Jonathan Sorg, Satinder Singh, and Richard I. Lewis. Variance-hased rewards for approximate bayesian reinforcement learning, arXiv preprint arXiv:1203.3518. 2012.
12

---

[47] Sainbayar Sukhbaatar and Rob Fergus. Learning from noisy labels with deep neural networks arXiv preprint arXiv:1406.200 2(3):4, 2014.
[48] Wei-Fang Sun, Cheng- Cheng-Kuang Lee, and Chun- Yi Lee. Dfac framework: Factorizing the value function via quantile mixture for multi-agent distributional q-learning. In International Confer- ence on Machine Learning. pages 9945-9954, PMLR, 2021.
[49] Wei-Fang Sun, Cheng-Kuang Lee, and Chun-Yi Yi Lee. A distributional perspective on value function factorization methods for multi-agent reinforcement learning. In AAMAS, pages 1671-1673, 2021.
[50] Yanchao Sun and Furong Huang. Can agents learn by analogy? an inferable model for pac reinforcement learning. arXiv preprint arXiv: 10329, 2019.
[51] Yanchao Sun, Cong Qian, Ning Yang, and S Yu Philip. Collaborative inference of coexisting information diffusions. In 2017 IEEE International Conference on Data Mining (ICDM), pages 1093-1098. IEFE, 2017.
[52] Yanchao Sun, Da Huo, and Furong Huang. Vulnerability- aware poisoning mechanism for online with unknown dynamics, arXiv preprint arXiv:2009. 0077,7, 2020.
[53] Yanchao Sun, Xiangyu Yin, and Furong Huang. Temple: Learning template of transitions for sample efficient multi-task rl. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9765- 9773, 2021.
[54] Yanchao Sun, Ruijie Zheng. Yongyuan Liang, and Furong Huang. Who is the strongest enemy? towards optimal and efficient evasion attacks in deep rl. arXiv preprint 1rX.v:2/06.000 05087, 2021.
[55] Yanchao Sun, Ruijie Zheng, Xiyao Wang, Andrew Cohen, and Furong Huang. Transfer rl across observation feature spaces via model-based regularization. arXiv preprint arXiv:2201.0022 2022.
[56] Richard S Sutton. Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. In Machine learning proceedings 1990, pages 216-224. Elsevier, 1990.
[57] Erik Talvitie. Learning the reward function for misspecified model. In International Confer- ence on Machine Learning, pages 4838-4847, PMLR, 2018.
[58] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. Proceed- ings of the tenth international conference on machine learning, pages 330-337, 1993.
[59] Brendan van Rooyen and Robert C Williamson. Learning in the presence of corruption. arXiv preprint arXiv: 1500110000009.00 2015.
[60] Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2020.
[61] Jingkang Wang, Hongyi Guo, Zhaowei Zhu, and Yang Liu. Policy learning using weak supervision. Advances in Neural Information Processing Systems, 34, 2021.
[62] C Yu, A Velu, E Yinitsky, et al. The surprising effectiveness of mappo in cooperative, multi-agent games [j/ol]. 2021.
[63] Mingqi Yuan, Mon-on Pun, Dong Wang, Yi Chen, and Haojun Li. Multimodal reward shaping for efficient exploration in reinforcement learning. arXiv preprint
2021.
[64] Kaiqing Zhang, Tao Sun, Yunzhe Tao, Sahika Gene, Sunil Mallya, and Tamer Basar. Robust multi-agent reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems, 33:10571-10583 2020.
[65] Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 10113-10123, 2021.
13

---

[66] Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. In International Conference on Machine Learning, pages 12912-12923. PMLR, 2021.
[67] Zhaowei Zhu, Jingxuan Zhu, Ji Liu, and Yang Liu. Federated bandit: A gossiping approach. In Abstract Proceedings of the 2021 ACM SICMETRICCS/nmeaaiiod Conference on Measurement and Modeling of Computer Systems, pages 3-4, 2021.
14


Reviews:
[{"review_id": "nGRdOL16rc", "reviewer": "Reviewer_TJjg", "summary": "This paper focus on addressing the problem of high reward uncertainty in multi-agent reinforcement learning (MARL). To address this problem, this paper proposes a new Distributional Reward Estimation (DRE) framework, which is composed of multi-action-branch reward estimation and policy-weighted reward aggregation for stable training. The full method, DRE-MARL is built on top with the architecture of centralized training and decentralized evaluation (CTDE), which consists of N decentralized actors and a centralized critic. Empirically, the proposed method is evaluated in several MARL environments, i.e. cooperative navigation, reference, and treasure collection with different number of agents. Compared with other SOTA MARL methods, DRE-MARL achives the best performance in most of the environments using three types of reward settings. ", "questions": "1. Line 55, \"If we can obtain the potential rewards on other action branches, we will perform more stable critic updating and thus achieve better performance.\" Is there any assumption of requirement on the design of action branches to make this sentence correct? What if the other actions are opposite the the current action of a specifict agent? \n\n2. What is the motivation for the design of different aggregation weights?\n\n3. DRE-MARL requires individual reward estimator for each agent. How muc is the computational cost gain of DRE-MARL compared with other SOTA-MARL methods? ", "limitations": "The authors have discussed the limitations and broader impacts of their work in section 7. \n\n==Limitations==\n1. DRE-MARL requires a prior assumption about the form of reward distribution. Although the distribution usually can be chosen arbitrarily, it may hurt performance when we choose a very complex distribution.\n2. Besides, reward aggregation is only used in discrete action space. If would be better if the method can be extented to more general continuous action space. \n\n==Broader impacts==\nThe authors does not see any negative societal impacts of this work while using the proposed method in practice.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "==Originality==\n\nThe proposed DRE-MARL is novel and interesting. The main idea is to develop distributional reward estimation followed by policy-weighted reward aggregation for MARL. This idea is intuitively similar to human's decision making process that considers all possible consequences of all action branches. The connections and differences of this work and previous work are well discussed and the related work are cited in the paper. \n\n\n==Quality==\n\nThe proposed method is technically sound. The experiments are conducted in diverse settings with ablations, and the results well supports the claims. Although there are some limitations, this work is a complete piece of work in addressing the reward uncertainty problem in MARL.  \n\n==Clarity==\n\nThis paper is clearly written and well organized. The problem is well formulated with a clear intruduction of the cause of reward uncertainty. The method is clearly motivated and introduced. The limitations and potential impacts are also discussed. \n\n==Significance==\n\nThis paper provides a novel distributional reward estimation method to address the high reward uncertainty in multi-agent reinforcement learning. The proposed method is novel and can be applied in other MARL methods. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fnoll44TjEG", "reviewer": "Reviewer_JEsw", "summary": "Reward uncertainty is a longstanding problem in multi-agent reinforcement learning, which stems from two aspects: the natural uncertainty in the MDP environment, and the actions of other agents. To deal with the reward uncertainty problem, this paper proposes a new framework for estimating the reward distribution and aggregating the estimated reward distribution. Then the proposed method uses the aggregated rewards to update the centralized critic network and the decentralized actor networks. Experiments in the particle domains demonstrate the effectiveness of the proposed method.", "questions": "Section 6.3 has conducted ablation studies on different reward estimation methods. I wonder what will happen when no reward estimation is conducted (just using the external rewards to update actor and critic networks). This experiment could better demonstrate the utility of reward estimation in multi-agent reinforcement learning.\n\nThis paper refers to the proposed framework as “two-stage” learning (Line 162). Is the reward estimation conducted before the reward estimation? If so, where does the data for estimating the reward distribution come from? Or those two stages are conducted concurrently? This point should be clarified. \n\nWhy is the regularization term $L_R$ needed in the reward estimation objective？ What if this term is removed? \n", "limitations": "The authors have discussed the possible limitations of their work in the last paragraph of the submission. ", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To my best knowledge, the idea of estimating the reward distribution and using the aggregated rewards to optimize the policy is novel, but I get no sense how significant reward estimation is in the multi-agent reinforcement learning domain. Ablation studies on removing the reward estimation module could answer this doubt.\n\nThe writing of this paper is mostly clear. In addition, this paper has conducted thorough comparative experiments to compare with state-of-the-art multi-agent algorithms in the particle experiment domain. A more challenging multi-agent benchmark is Starcraft 2. The authors are encouraged to do more experiments in this domain.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VsQQMAAwNOm", "reviewer": "Reviewer_ugkM", "summary": "this paper propose distributional reward estimation for multi-agent reinforcement learning (DRE-MARL). this paper focuses on the problem of reward uncertainty in MARL. The main idea of this paper is to design the multi-action-branch  reward estimation and policy-weighted reward aggregation for stabilized training. This former part is simply function approximation with historical data, while the latter part is weighted reward aggregation. Experiments show that DRE-MARL outperforms other SoTA algorithms comprehensively.", "questions": "I suggest the authors put the related work in section 1 introduction to section 2. \n\nI think the data volume for reward estimator should be very large, so I suggest the authors conducting some experiments to test how much data should be used. ", "limitations": "the reward distribution in reward estimation is hard to choose", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nit is novel to solve the reward uncertainty problem in MARL\n\npolicy-weighted reward aggregation enables stable training of the critic and actors and it is quite robust. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VIxb9nPV9Uj", "reviewer": "Reviewer_oHz3", "summary": "The paper outlines a new method for multi-agent reinforcement learning settings (DRE-MARL) which primarily focuses on reward estimation for multi-agent reinforcement learning settings. The authors first describe the problem settings and motivate why reward estimation is important and challenging in multi-agent reinforcement learning and how DRE-MARL differs from prior approaches, including reward uncertainty method and reward estimation methods in single-agent reinforcement learning. According to the authors' summary, many reward estimation methods struggle in multi-agent setting due to the additional complexity and greater sources of uncertainty. The authors identify mutual interaction between agents and natural disturbance of the environment as two sources of uncertainty in MARL settings and propose a method simulate reward uncertainty by applying stochastic processes.\n\nSubsequently the authors outline their method (DRE-MARL) for reward estimation based on a distribution of potential action a single agent can take in a MARL setting and also describe different reward aggregation methods that are tested in the experiments. The authors then compare their method against different MARL baselines across a set of collaborative MARL environments showing mostly outperformance and conduct an ablation study for various reward aggregation schemes.", "questions": "**Nits**\n- A is not capitalized in title \"Multi-agent\"\n- first page says 35th Neurips, which is 2021\n- Line 173-175 (page 5) have some spelling errors (\"interplay grows exponentially with an increase of the agent number\")\n\n**General Questions:**\n- Could you clarify why you choose a centralized critic for your method? Is this related to the environment setting (i.e. the environment only provides a single reward for all agents)? Can you see your method working with a de-centralized critic? what would change if anything?\n- Could you say more about how your method might perform in competitive MARL settings? Right now you have looked at cooperative settings and it would be interesting to contrast that with competitive settings. (No new experiments needed, mainly looking for additional detail)\n- It seems like p2p-MARL is the most competitive method to DRE-MARL and also uses reward estimation. Could you clarify the differences between p2p-MARL and DRE-MARL?\n- How did you choose k for the distributional estimation? Was there a significant difference between different values of k?", "limitations": "I think that the paper could be improved by further discussion of the limitations of DRE-MARL. As far as I can tell only one limitation is briefly mentioned on Page 9, which is that DRE-MARL and reward aggregation are only used in discrete action spaces. It would be good to get further clarity on the following from the authors:\n- How could DRE-MARL be applied to continuous action spaces?\n- What limitations does DRE-MARL have that can inspire future work?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality:**\n- Strengths: The paper proposes a new method for reward estimation in MARL settings that differs from prior approaches and shows good performance compared to baseline methods. Relevant work is cited and compared against in the beginning of the paper.\n- Weaknesses: The authors could have provided more detail on the limitations of their method (discussed below as well)\n\n**Quality:**\n- Strengths: The methods and contributions are generally well supported in the experiments and discussed in detail in the paper.\n\n**Clarity:**\n- Strengths: The paper is generally well written and well organized with relevant equations, diagrams and descriptions.\n- Weaknesses: The authors could have made better use of figure captions (in Figure 2 and Figure 3 specifically) to make it easier for the reader to understand the relevant messages conveyed by the figures.\n\n**Significance:**\n- Strengths: The paper proposes a new method in a relevant subject area of MARL.\n- Weaknesses: The authors could have provided more detail on the limitations of their method and put it into the broader context of MARL methods (discussed below as well).\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9qwY83DsjpH", "reviewer": "Reviewer_c9dW", "summary": "Considering that high reward uncertainty remains a problem, the authors propose a novel distributed reward estimation framework to enhance multi-gent reinforcement learning. The training process is stabilized by designing multi-action-branch reward estimation and policy-weighted reward aggregation. Multi-action branch reward estimation is first employed to model the reward distribution of all action branches, and then reward aggregation is used to obtain a stable update signal during training. And its effectiveness is demonstrated experimentally.", "questions": "* Shouldn't this work be discussed and compared with the related work of Distributional MARL? For example. \n    * DFAC Framework: Factorizing the Value Function via Quantile Mixture for Multi-Agent Distributional Q-Learning\n    * A Distributional Perspective on Value Function Factorization Methods for Multi-Agent Reinforcement Learning\n    * RMIX: Learning Risk-Sensitive Policies for Cooperative Reinforcement Learning Agents\n", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths.\n* paper writing is well understood\n* The pictures drawn can help understanding\n* The motivation is clear and seems to make sense\n* table-1 looks like a lot of experiments were done\n* Ablation experiments also look interesting\n\n\nWeaknesses.\n* Lack of discussion and experimental comparison with work related to Distributional MARL\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept