Online Tuning for Offline Decentralized Multi-Agent Reinforcement Learning
Authors: Jiechuan Jiang
Zongqing Lu
Venue: ICLR 2022 Submitted (2022)
Keywords: multi-agent reinforcement learning

Abstract: Offline reinforcement learning could learn effective policies from a fixed dataset, which is promising in real-world applications. However, in offline decentralized multi-agent reinforcement learning, due to the discrepancy between the behavior policy and learned policy, the transition dynamics in offline experiences do not accord with the transition dynamics in online execution, which creates severe errors in value estimates, leading to uncoordinated and suboptimal policies. One way to overcome the transition bias is to bridge offline training and online tuning. However, considering both deployment efficiency and sample efficiency, we could only collect very limited online experiences, making it insufficient to use merely online data for updating the agent policy. To utilize both offline and online experiences to tune the policies of agents, we introduce online transition correction (OTC) to implicitly correct the biased transition dynamics by modifying sampling probabilities. We design two types of distances, i.e., embedding-based and value-based distance, to measure the similarity between transitions, and further propose an adaptive rank-based prioritization to sample transitions according to the transition similarity. OTC is simple yet effective to increase data efficiency and improve agent policies in online tuning. Empirically, we show that OTC outperforms baselines in a variety of tasks.

Under review as a conference paper at ICLR 2022
ONLINE TUNING FOR OFFLINE DECENTRALIZED MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
Offline reinforcement learning could learn effective policies from a fixed dataset, which is promising in real-world applications. However, in offline decentralized multi-agent reinforcement learning, due to the discrepancy between the behavior policy and learned policy, the transition dynamies in offline experiences do not accord with the transition dynamics online execution, which creates severe er- rors in value estimates, leading to uncoordinated and suboptimal policies. One way overcome the transition bias is bridge offline training and online tuning. However, considering both deployment efficiency and sample efficiency, we could only collect very limited online experiences, making it insufficient use merely online data for updating the agent policy. To utilize both offline and online ex- periences tune the policies of agents, introduce online Iransition correction (OTC) to implicitly correct the biased transition dynamics by modifying sampling probabilities. We design two types of distances, i.e., embedding- based and value- based distance, to measure the similarity between transitions, and further propose an adaptive rank-based prioritization to sample transitions according to the transi- tion similarity. OTC is simple yet effective to increase data efficiency and improve agent policies in online tuning. Empirically, we show that OTC outperforms base- lines in a variety of tasks.
INTRODUCTION
In fully decentralized multi-agent reinforcement learning (MARL) (de Witt et al., 2020a), agents interact with the environment to obtain individual experiences and independently improve the poli- cies to maximize the cumulative shared reward. Due to the scalability, decentralized learning would be promising in real-world cooperative tasks. However, in many industrial applications, continu- ously interacting with the environment to collect the experiences for learning is costly and risky, e.g., autonomous driving. To overcome this challenge, offline decentralized MARL (Jiang & Lu, 2021) lets each agent learn its policy from fixed dataset of experiences without interacting with the environment. The dataset of each agent contains the individual action instead of the joint action of all agents. There is assumption on the data collection policies and the relationship between the datasets of agents.
However, from the perspective of each individual agent, the transition dynamics depend on the policies of other agents and will change as other agents improve the policies (Foerster et al., 2017). Since the behavior policies of other agents during data collection would be inconsistent with their leamed policies, the transition dynamics in the dataset would be different from the real transition dynamics in execution, which will cause extrapolation error, i.e., the error in value estimate incurred by the mismatch between the experience distribution of the learned policy and the dataset (Fujimoto al., 2019). The extrapolation error makes the agent underestimate or overestimate state values, which leads to uncoordinated and suboptimal policies (Jiang & Lu, 2021).
One way to reduce the extrapolation error caused by the mismatch of transition dynamics is to bridge offline training and online tuning. However, since both deploying new policies and interacting with the environment are costly and risky, we should consider the deployment efficiency (the number of deployments) and sample efficiency (the number of interactions) in the collection of online experi- ences (Matsushima et al., 2021). Due to the efficiency requirement, the collected online experiences can be very limited. Thus, it is insufficient tune the policies of agents merely using the online

---

Under review as a conference paper at ICLR 2022
data, and the small online dataset may also cause overfitting. To increase data efficiency, it is better to additionally exploit the offline data for online tuning. However, uniformly sampling from the merged offline and online experiences (Nair et al., 2020) cannot address the transition mismatch problem. Therefore, it is necessary to correct the transition dynamics in the offline data to make it close to the online transition dynamics. Nevertheless, the requirement of efficiency also means it is infeasible to accurately estimate the real transition dynamics from the limited online experiences, thus explicit correction is impractical.
In this paper, we introduce a simple yet effective method to correct the transition dynamics of of- fline data for online tuning, without explicitly modeling the transition dynamics. When sampling a transition from the offline experiences, we first search for the best-matched transition in the online experiences, which has the minimum state-action distance to the sampled transition. Then, we com- pute the next-state distance between the sampled transition and the best-matched online transition to represent the transition similarity. After that, a probability function maps the transition similarity to the probability of accepting the sampled transition for update, which is equívalent to modifying the transition probability. Therefore, the objective is to find the optimal probability function which minimizes the KL-divergence between the online transition distribution and the modified transition distribution, given the distance measure. We design two distance measures based on the embed- ding and Q-value of transitions, respectively. The embedding-based distance captures the similarity in feature space, and the value-based distance measures the isomorphism. Due to the limited on- line experiences, it is hard to find the optimal probability function by gradient-based optimization, but we empirically find that the rank-based prioritization in PER (Schaul et al., 2016) is a proper choice of the probability function. Morcover, we propose an adaptive rank-based prioritization to adjust the degree of the correction according to the difference between offline and online transition distributions.
The proposed method, termed OTC (Online Transition Correction), could be applied to any offline RI. method for decentralized multi-agent learning. We construct the decentralized datasets from a variety of D4RL. tasks (Fu et al., 2020) and evaluate OTC on them. Experimental results show that OTC outperforms baselines, and ablation studies demonstrate the effectiveness of the two distance measures, the practicability of rank-based prioritization, and the improvement of adaptive prioritiza- tion. To the best of our knowledge, OTC is the first method for bridging offline training and online tuning in decentralized MARL.
2 RELATED WORK
2.1 OFFLINE RL
In offline RL, the agent could only access to fixed dateset of single-step transitions collected by behavior policy, and no interactive experience collection is allowed during learning. Offline RL easily suffers from the extrapolation error, which is mainly caused by out-of-distribution actions in single-agent environments. To address this issue, constraint-based methods introduce policy con- straints to enforce the learned policy to be close to the behavior policy, e.g., direct action constraint (Fujimoto al., 2019), kernel MMD (Kumar al., 2019), Wasserstein distance (Wu et al., 2019), and KL--divereence (Peng et al., 2019). Conservative methods (Kumar et al., 2020; He & Hou, 2020; Yu et al., 2021) train a Q-function pessimistic to out-of-distribution actions. Uncertainty- based methods quantify the uncertainty by the learned environment model (Yu et al., 2020) or by Monte Carlo dropout (Wu et al., 2021) of Q-function, and use as a penalty or weight the update of fuuncton,, so as to avoid the overestimation of nut-of-distribution actions.
In offline decentralized MARL, besides diitti--iitiittiionn actions, the extrapolation error is also caused by the bias of transition dynamics. For each individual agent, since the transition dynamics depend on other agents' policies which are also updating, there will be a difference between the transition dynamics in the offline dataset and the real transition dynamics during online deployment (Jiang & Lu, 2021). To overcome this, MABCQ (Jiang & Lu, 2021) uses two importance weights to modify the offline transition dynamics by normalizing the biased transition probabilities and increas- ing the transition probabilities of high-value next states. However, the modified transition dynamics in MABCQ are not theoretically guaranteed to be close the real ones. Unlike MABCQ, OTC focuses on online tuning and exploits online experiences to correct the bias of transition dynamics to quickly adapt to the learned policies of other agents.
2

---

Under review as a conference paper at ICLR 2022
deployment 1
deployment
deployment K
Env
Env
Env
D
D2
Dylk
offline training
online tuning
online tuning
online tuning
B
D
B
D
B
B
UD
Figure 1: Overview of offline training and online tuning. After each agent learning its policy my from offline dataset B.. their policies are deployed in the environment to get the online dataset D, Then, n. is finetuned to obtain of using the merged dataset B,. The online tuning is repeated for K times.
2.2 BRIDGING OFFLINE LEARNING AND ONLINE TUNING
Since the offline dataset is usually insufficient to cover the entire transition space, the extrapolation cannot be eliminated entirely in the fully offline learning. It is crucial improve the policy trained using offline data further with online reinforcement learning. Since the online interaction expensive, must consider both the deployment efficiency (the number of policy deployments) and sample efficiency (the number of interactions) in online tuning. The concept of deployment efficiency is adopted in BREMEN (Matsushima al., 2021) and MUSBO (Su et al., 2021), which, however, do not aim to finetune the pre-trained policy but instead train the policy from scratch with limited deployments. AWAC (Nair et al., 2020) employs an implicit constraint that could miti- gate the extrapolation error while avoiding overly conservative updates in offline learning and thus quickly performs online finetuning. Balanced Replay (Lee et al., 2021) adopts prioritized sampling to encourage the use of near-on-polig, samples from the offline dataset. However, they deploy the policy frequently, ignoring the deployment efficiency. Moreover, these methods above are designed for single-agent environments, where offline and online data follow the same transition dynamics, thus they cannot deal with transition bias. Abiding by both deployment and sample efficiency, OTC uses prioritized sampling to reduce the bias of transition dynamics in decentralized MARL, rather than the state-action distribution shift considered in Balanced Replay.
METHOD
3.1 PRELIMINARIES
In offline and decentralized cooperative settings, each agent could only access to an offline dataset B, which is collected by a behavior policy and contains the tuples (s, a,, r, s'), where is the state, a is the individual action of agent r is the shared reward, and s' is the next state. Note that B; dose not contain the joint actions all agents. Each agent independently learns its policy n, using offline RL algorithm, without information sharing among agents. The goal of all agents is maximize the expected return E 7'r; when deploying their learned policies in the environment, where is the discount factor and 7' is the time horizon of the episode. From the perspective of each agent i, the transition probability in B, denoted by PB, (s'|s, a.), depends on other agents' behavior policies during the collection of B, while the real transition probability in execution, denoted by Pe. (s'|s, o,), depends on other agents' learned policies. The difference between Pg, and Ps. would cause severe extrapolation errors, which eventually lead to uncoordinated and suboptimal policies (Jiang Lu, 2021).
To finetune the policies learned from offline datasets, we allow the agents to interact with the envi- ronment to collect online experiences. As illustrated in Figure 1, after offline learning using initial dataset 83 for each agent their learned policies (TT. =²,), where n2, denotes the policies of all agents except i, are deployed the environment and interact with each other for M timesteps. Each agent i obtains of agent rather than the joint actions. We merge D and BS to get B,. and finetune the policy π! to obtain of using B.. Then, deploy the updated policies (m m1,) in the environment and repeat
online dataset DJ with M transitions. D. still only contains the individual actions
3

---

Under review as a conference paper at ICLR 2022
the procedures until K deployments. K represents the deployment efficiency, and K x M repre- sents the sample efficiency. Note that for presentation simplicity, we denote T-4 as also updating, but other agents may may not be learning online, which however does not affect the following problem formulation and our method. That said OTC does not have any assumptions on other agents.
3.2 PROBLEM FORMULATION
For agent i, given and a, the -i-deiveeeence between the transition distributions of next state s' in the online dataset D and in the merged dataset B3 is
PDy (o')s,)))))) PE: (8))),a;)
KL(PD; [Pp;)) Σ Pay (s'1s, a;) log
However, since KL(PDt (P is generally large, order use the merged dataset to finetune the policy, need modify Psy PBς to minimize the KL-divergence between Poy and Pst.
Since the difference of transition distributions means the difference of next-state distributions given the same state-action pair, we first define two distance functions: d(s1, Os, $2, ai,) that measures the similarity of state-action pairs, and d(s1 s4) that measures the similarity of next states. Once sampling a transition (8, dx, dx, r) from B,, we select the best-matched transition "0,0% 8 r*) from Dj, which has the minimum state-action distance to (s, 0z rr,)),
0g
arg min d(s,, a, s .0)).
(2)
D
For the convenience of theoretical analysis, we assume there is always (s" a s't ,r") in DJ that meets d(s, az, s .aa) = O, i.e., = s* and a; a;- If there is more than one transition, we uniformly select one from them. Then, we adopt a probability function f which maps the dis- tance d(s', to the probability of accepting the sampled transition (s, ag,s', r) for update, i.e.
fd(s', )). Therefore, the transition probability can be modified as
Per (5'1,, (8)8,,))))))) Σx (8'ls, a:)
as)
where (s,, az) is a normalization term to make sure Eg Pay (s'(s, = Thus the KL-divergence
between PDF and Poę is
Er Poy (8'1s, ())((((((()))))))))
KL(Pp)||PE)) KL(Po;1|PE)) Poy a;) log
(4)
((s,a;)
To minimize the KL-divergence, we need to design appropriate d-functions to accurately mea- sure the distances between transitions and find the optimal f-function which properly satisfies max, E. Ppt(('(ss log(E> Poy (is)))
3.3 d-FUNCTIONS
Due to the limitations of computational complexity and representation ability, directly measuring the distances the original space impractical, especially high-dimensional environments. There- fore, we design two types of d-functions. The first type is the distance in the embedding space. We employ VAE (Kingma Welling. 2013) to encode the state-action pair and next state into e(s, a,) and e(s'), ) and define the d-functions /1 distance in the embedding space,
(s, a,, s" a,)) |e(s, a.) e(s", aa,),, de(s'ss"))
I|e(s') - e(s"*)|l
(5)
Due to the requirement of sample efficiency, it impossible that could always find the transition from D, with the same state-action pair as the given transition. Relying on the generalization ability of the encoder, similar inputs will be encoded into similar embeddings. We could search for the best-matched transition with the most similar state- atatenaction pair in terms of (s, a,, evaluate the transition similarity using ds )*)
a,, s" at) and then
4

---

Under review as a conference paper at ICLR 2022
Transition a
Transition
Transition
d,
dg
B
D
Agents
Other agents
Landmark
Figure 2: this navigation task, agents are learning to cover all the landmarks. For the transition a from 28, the best-matched transition b selected from D. using d, is still much different from transition a. However, the valuc-based distance dg will select the nearly isomorphic transition which is more helpful for evaluating the trainsition similarity.
The embedding- based distance measures the similarity in feature space. However, since the limited online experiences cannot cover all state-action pairs offline dataset, there must be some transi- tions in the offline dataset which are still much different from the transition with the most similar state-action feature, e.g., transition a and b in Figure ate the transition similarity using d.. Inspired by state representation learning (Gelada et al., 2019; Zhang et al., 2020a;b), we notice that some state-action pairs may have common latent structure, e.g., transition and e in Figure 2. Although they are different in feature space, the topologies of agents are isomorphic. As pointed by DeepMDP (Gelada et al., 2019), nearly isomorphic state- action pairs would have similar Q-values. Therefore, we use the difference in Q-values to evaluate the isomorphism in state-action pairs and select the best-matched transition from the online dataset,
such cases, we cannot accurately evalu-
(s. ass s" aa)) [1Q(s, a;) Q(s", ,at)il
On the other hand, we use the expected Q-value to measure the distance of next states,
d V(s)) E+V(6' VV
V(s'*)
where V(s') Ea, Q(s az).
V('') is the deviation from the expected value, which could mitigate the influence of absolute value. The value-based distance dg has stronger representation ability than the embedding- bbased distance de. The transitions that are close in embedding space will also have small value difference, and the value-based distance could also represent the isomorphism of transitions.
3.4 f Y-FUNCTION
aog The optimal f-function is to maximize Ey PDx (s' a;) log Ex (dds How- ever, since the online experiences are limited, gradient-based optimization. Nevertheless, there must exist such f-functions which do not increase KL.(Pp l ). and trivial example is the constant function. Therefore, we try to find heuris- tic and practical f-function which is able to reduce the KL-divergence and extrapolation error. An important prior is that f f-function should be monotonic, which will produce a larger acceptance probability when fed with smaller distance of next states. The intuition is that if the next state of the transition from 3* is more similar to the next state of the online experience with the same state-action pair, the transition is more likely to follow the transition dynamics D.. Therefore, we should give it larger acceptance probability, which eventually leads to larger transition probabil- ity. Empirically, we find the rank-based prioritized sampling PER (Schaul et al., 2016) is a good solution. Concretely, the probability of accepting transition j is
hard to solve the optimal f-function by
DS
P(j)
(8)
Em P
where the priority = 1/sank(f), and rank(j) is the rank transition j when the transitions are sorted according d(s', s²). The exponent a determines the degree of modifying the transition
5

---

Under review as a conference paper at ICLR 2022
Algorithm OTC for Agent i
Initialize the RL model and the modification degree og
2: Train the RL model using 8 to obtain the policy x!
3: for k == 1 ... K do
4: Deploy the policy :; - in the environment
5: Collect the online dataset Dj with M transitions
6: Merge the experiences B = B/-11 D
7: if h: > then
8:
Adjust a by (9)
9:
end if
for 1, mear_updatee do
11:
Sample minibatch B from B and minibatch D from D for each transition in B do
12:
13:
find the best-matched transition in (2) and compute the transition similarity end for
14:
15:
Sample transitions from B by rank-based prioritization (8)
16: Update the RL model using the sampled transitions
end for
18:
end for
probability, with a = O meaning the f-function degrades into a constant function. The rank-based prioritization ensures that the probability of being accepted monotonic in terms of transition sim- ilarity, and is robust as it is insensitive to outliers.
Another prior is that the modification degree should depend on the distribution difference between DR and B. We should adopt weaker modification degree when the transition dynamics in B is more similar to that in D., and vice versa. Inspired by that, we propose to adaptively adjust a at each deployment & (k 1)
D&
at =
where T. 2,,,,, d(s', 8'* arg min d(s, 8",
D the expected transition similarity for agent at deployment k. the difference between offline and online transition distributions would change along with the update of agents, fixed is not an optimal solution. On the contrary, for example, if the distribution difference grows, adaptive is likely to take large value, and thus more aggressively modifies the transition dynamics.
3.5 IMPLEMENTATION DETAILS
For the embedding distance de, we do not maintain two embeddings e(s, a;) and e(s'), but train conditional VAE G; {Es (p, of 0z, 8'), D; (a.ls, 8', a (H,o))} which encodes (s, Js,0,, a,,s') into the embedding puss, a,, s'). We take the embedding puss a, s') as substitute for both e(s, a;) and e(s'). which more effective and computationally efficient in practice. Moreover, it is costly to sample transitions from B and then search the best-match transitions from DÍ: for every update. To reduce the complexity. for each update we uniformly sample two minibatches, and D respectively from B and D/. and perform the rank-based prioritized sampling on the two minibatches. The complete training procedure is summarized Algorithm 1.
EXPERIMENTS
4.1 SETTINGS
We evaluate OTC in D4RL (Fu et al., 2020) datasets with three types: random, medium, and repium- Following the settings in multi-agent mujoco (de Witt et al., 2020b; Jiang & Lu, 2021), split the original action space of three mujoco tasks (Todorov et al.. 2012), i.e., HalfChee- tah, Walker2d, and Hopper, into several sub-spaces. As illustrated Figure 3, different colors indicate different agents. Each agent obtains the state and reward of the robot and independently

---

Under review as a conference paper at ICLR 2022
(a) HalfCheetah
(b) Walker2d
(c) Hopper
Figure 3: Illustrations multi-agent mujoco tasks. Different colors mean different agents (Jiang Lu, 2021).
controls one or some joints of the robot. For each agent we delete the actions of other agents from the original dataset and take the modified dataset as By- During online tuning, we perform K = 10 deployments. For each deployment k the agent collects very limited online dataset D. of which the size is only one percent of the initial offline dataset (iD)) = 1 17) After online data collection, we finetune the agents by L updates and deploy the updated policies in the environment for next deployment.
We instantiate OTC respectively on two offline RL algorithms, BCQ (Fujimoto et al., 2019) and AWAC (Nair et al., 2020), and also take them as the baselines. During online tuning, the baselines uniformly and randomly sample transitions from the merged offline and online dataset, and they also have the same neural network architectures and hyperparameters trained for five runs with different random seeds, and results are presented using mean and standard deviation. More details about hyperparameters are available in Appendix A.
OTC. All the models are
4.2 EVALUATING d-FUNCTIONS
We summarize the performance of the last deployment of OTC with de (embedding-banl distance) and dg (value-based distance) on BCQ in Table and on AWAC in Table 2, and plot the learning curves along with the number of deployments for a part of tasks in Figure The empirical results show that OTC with de or dy performs more than one standard deviation better than BCQ and AWAC, which verifies that de and dy are capable of properly measuring the transition similarity. Since the online experiences are limited, uniformly sampling from the merged dataset is not effective
Table I: Performance of OTC on BCQ.
d. + BCQ
d, + BCQ
BCQ
halfchectah-randonn
1242
1170 +
1078 + 23
malker2d-randone
446 24
444
374 32
hopper-random
328 + 4
325 + 8
309 + 39
saffihhetah-mmciuu-rrpllhpo
2828 65
2724 +:35 35
2624 + 55
walker2d--medium-rrpla
634 + 40
+30+72
581 + 128
hopper-mediun-reepe
753 - 63
777 1 271
568 1 22
halfchcctah-medine
3725 = 134
3732
34
3638 + 97
malker2d-medium
1449 + 247
1406 = 89
982 + 49
hopper-medium
1286 + 47
1405 33
1169 + 87
Table 2: Performance of OTC AWAC.
d. + AWAC
d, AWAC
AWAC
halfchectah-randon
296 + 276
301 + 243
59 + 110
walker2d-random
660 + 270
278 +78-10 21
262 062+8
hopper-random
552 + 79
459
261 61
maalihhetah-mediiu-rrp....
2990 216
3263 97
2578 +: 188
walker2d- www--mmdddiudmpeeee replay
409 = 67
459 + 37
368 + 60
2943 130
1877
1741 + 455
halfcheetth-mmddium
4253 56
4176 115
4090 + 76
walker2d-medium
2027 + 310
2099 + 588
1059 + 673
hopper-medium
2561 + 533
2275 785
1403 + 384

---

Under review as a conference paper at ICLR 2022
6-400
6,000
d,400
BBQ
Dealonert
(a) halfchectah-rrndom
(b) walker2d-random
(c) hopper-random
DCB
ECO
(d) pallcheetah-mediu--ellla (e) walker2d-
(1)
hopper-mediumn-rellay
AMMMAC
Deplerer::
(g) halfchectah-random
(h) walker2d-random
(i) hopper-random
AMRC
(i) halfchoctah-meddie heppllau--rppyyy (k) faakkrr2d-mmiiuu-rpppaa ()) hopper-mcdium-replay
Figure 4: Learning curves of OTC on BCQ and AWAC.
correcting the transition bias. The value-based distance dy has stronger representation ability than the embedding- distance de, since it could represent both the similarity in feature and isomorphism. However, d, does not commonly outperform de. The reason might be that d, would mistakenly judge the state-action pairs, which different in feature and isomorphism but have similar values, as best-matched pairs. Moreover, since the Q-values are updating, dy is inconsistent during the tuning process.
4.3 EVALUATING f F-FNCCTION
The hyperparameter controls the strength of modifying the transition dynamics. Figure 5 shows the learning curves of OTC (de) with different a. It is observed that if a is too small, OTC has weak effects on correcting the transition dynamics. However, if a is too large, the overly modified transition dynamics would deviate from the real transition dynamies and degrade the performance. Since the agents are continuously updated, the difference between the transition dynamics in B and DŁ will also change every deployment, so fixed a cannot deal with the nonstationarity. We test the fixed a 1.0 and our method for adaptive a where initial a = 1.0. Figure 6 shows that adaptive a could improve the performance of online tuning. Figure 7 shows the curves of a during the online tuning of OTC (de). In malfcheetalh-mediuee -edium-replllyyyy a of both two agents grows, which means the difference between B and DÍ increases as the agents are further improved during online tuning, and a becomes stable in the latter deployments, which means the convergence of online tuning. In aeiicheeaah-andoooe and walker2d-randon the agents are influenced differently by the update other agent. Our adaptive a method could discriminate the transition biases of agents rather than treat them equally.
two agents have different trends of a, which means that
8

---

Under review as a conference paper at ICLR 2022
(a) halfcheetah mandoom Figure 5: Learning curves of OTC (de) on BCQ with different a.
(b) walker2d-random
(c) halfcheetah sawiaaa--miiimm-eee replay
4/a
Ca-333 adocio
(a) balfcheetah-
(b) walker2d-random
(c) halfcheetah wewiihhettalmmiiim--eeee medium-replayy
Figure 6: Learning curves of OTC (d,) on BCQ with adaptive a and fixed a (1.0).
(a) srrcheetaa-randoom
(b) walker2d-randon
(c)
Figure 7: Curves of adaptive of OTC (d.) on BCQ during online tuning. Dotted lines show values, and
violin plots show distributions over seeds.
DTC
(klkm Oibe-raa
Oole-roh
Dephueeen hopper-medium Figure 8: Learning curves OTC, Balanced Replay, uniformly sampling from merged dataset, and only sampling from the online dataset.
Diphay
halfcheetah-mmddimm
walker2d-medium
We additionally provide ablation studies of different sampling strategies in Figure 8. Since the online dataset very limited, finetuning the agents with online samples only is insufficient, and the small dataset could cause overfitting, e.g., Figure 8(b) and 8(c). Balanced Replay (Lee et al., 2021) I) and uniformly sampling are susceptible to the transition bias. OTC obtains the performance gain over the other three sampling strategies.
CONCLUSION
We have proposed OTC to effectively correct the transition dynamics during online interaction for tuning decentralized multi-agent policies learned from offline datasets, given limited online experi- ences. OTC consists two types of distances to measure the transition similarity and an adaptive rank-based prioritization sample transitions for updating the agent policy according to the tran- sition similarity. Experimental results show that OTC outperform baselines for online tuning variety of tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge? arXiv preprint 1.Xxi:2011100553: 2020a.
Christian Schroeder de Witt, Bei Peng, Pierre-llexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint (XXi:20033306709 2020b.
Jakob Foerster. Nantas Nardelli, Gregory Farquhar, Triantafyllos Afouras, Philip HS Torr, Pushmeet Kohli, and Shimon Whiteson. Stabilising experience replay for deep multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), 2017.
Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning (ICML), 2019.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In International Conference Machine Learning, pp. 2170-2179, PMLR, 2019.
Qiang He and Xinwen Hou. arXiv:2012.13662 2020.
Popo: Pessimistic offline policy optimization. arXiv preprint
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2 108.0 108.0/83.....0 2021.
Diederik Kingma and Max Welling. Auto-encoding variational arXiv:1312.61 14, 2013.
bayes.
arXiv preprint
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine Stabilizing off-policy q- learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint 20fi::000000477999 2020.
Seunghyun Lec, Younggyo Seo, Kimin Lee, Picter Abbeel, and Jinwoo Shin. Offline-to- online reinforcement learning via balanced replay and pessimistic q-ensemble. arXiv preprint arXiv:2107.00591, 2021.
Tatsuya Matsushima. Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In International Confer- ence on Learning Representations (ICLR), 2021.
Ashvin Nair, Murtaza Dalai, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.098: 2020.
Xue Bin Peng. Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression:
Simple and scalable off-policy reinforcement learning. arXiv preprint 1rXiv: 2012
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In International Conference on Learning Representations (ICLR), 2016.
DiJia Su, Jason D Lee, John M Mulvey, and Н Vincent Poor. Musbo: Model-based uncertainty regularized and sample efficient batch optimization for deployment constrained reinforcement learning, arXiv preprint arXiv:2102. 11448 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.
In IEFF/RSJ International Conference Intelligent Robots and Systems (IROS), 2012.
10

---

Under review as a conference paper at ICLR 2022
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 19111:191..1131,,222 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offiine reinforcement learning. arXiv preprint arXiv:2105.08/ 400, 2021.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances Neural Information Processing Systems (NeurIPS), 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. arXiv preprint arXiv22102.08363, 2021.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learn- ing invariant representations for reinforcement learning without reconstruction. In International Conference on Learning Representations (ICLR), 2020a.
Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. Learning robust state abstrac- tions for hidden-parameen block mdps. In International Conference on Learning Representations (ICLR), 2020b.

---

Under review as a conference paper at ICLR 2022
A HYPERPARAMETERS
The hyperparameters are summarized in Table 3. For the results Table and Table 2, we use grid search to find the optimal a from [0.6, 0.8, 1.0, 1.2, 1.6..
Table Experimental settings and hyperparameters
Hyperparameter
BCQ
AWAC
discount (7)
0.99
0.99
B [D]
512
512
2000
2000
batch size
128
hidden sizes
(64, 64)
(256, 256)
activation learning rate
ReLU
ReLU
10-4
10
critic learning rate
10 10-4
5 x 10
embedding dimension
10
10
finetuning updates (L)
4000
2000
B OTC ON MABCQ
As the transition distribution of the learned policy in MABCQ does not follow PB, MABCQ is not a suitable backbone algorithm, though we additionally provides some results of OTC on MABCQ in Figure 9. OTC+MABCQ could outperform MABCQ in online tuning. In halfcheetah-randee MABCQ achieves better performance than BCQ in offline learning. However, since value deviation in MABCQ has made the agent be optimistic toward other agents in offine training, the modified transition dynamics in MABCQ is close to the real transition dynamics during the online interaction with improved other agents, and thus MABCQ does not benefit from online tuning in halfcheetah- random.
MARCO
MAVICQ
MAACC
BOQ
BUQ
Deloywes
Deppymert
(a) halfchectah-randonm
(b) walker2d-random
Figure 9: Learning curves of OTC (d.) on MABCQ.
12


Reviews:
[{"review_id": "c8VJm5vl3I", "reviewer": "Reviewer_T8NR", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Pros:\n\n(1) Authors study an interesting and practical setting, offline training and online tuning for multi-agent tasks. \n\n(2) The paper is well-written, with a clear description of the proposed method.\n\n(3) Authors conduct extensive experiments to evaluate OTC.\n\n- Cons:\n\n(1) Related work: The idea of correcting sampling probabilities is very close to that in (Jiang and Lu, 2021), and authors should discuss more about MABCQ in the paper.\n\n(2) Method:\nThe method involves a pairwise comparison of the sampled transition and all transitions from online execution, which can incur some computation overheads when D_i^k becomes larger. In addition, the paper also assumes there is always <s^*, a_i^*, s^{'*}, r^*> in the dataset, which may not always be the case. In the d-function part (Equation 7), it leverages the difference in value functions for measuring the distance. However, Q(s, a) can be over-optimistic for unseen actions. \n \n(3) Experiments: \n\n- There is a missing strong baseline MA-ICQ (Yang et al., 2021) which also studies the offline MARL setting.\n\n- Given Figure 4, there seems to be very large variance, and I'm not very confident about the conclusion.\n\n - In most experiment tasks, there only involves 2 agent, which make it relatively small scale. Can the method scale with more agents? \n", "summary_of_the_paper": "The paper studies offline training and online tuning for multi-agent systems, which focuses on the decentralized cooperative setting where each could only access to its offline dataset. Authors discover that the transition dynamics in the offline dataset and online execution can differ very much, and propose to bridge this gap by introducing online transition correction by modifying the sampling probabilities. To accomplish this goal, authors propose practical implementations for the distance function measuring the similarities of states/state-action pairs. Authors conduct extensive experiments on multi-agent MuJoCo tasks, and show that OTC outperforms baselines (BCQ and AWAC).", "summary_of_the_review": "The paper studies an interesting setting, but should explain more about the potential problem of the method (please check the main review) and compare it with a more recent MA-ICQ method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z4GzDjgr0y", "reviewer": "Reviewer_ZgL6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I don't have issues with the technical proposal of this paper. \n\n* The author studied an important problem in offline MARL. When the opponent agents' actions are not accurate, it can cause mismatch between the transition dynamics in the offline data and the transition dynamics in real online settings. As a remedy, the authors proposed to solve this issue by bridging offline training with online tuning. \n\n\n* OTC essentially tries to find the optimal probability function that minimizes the KL divergence between online transition distribution and modified transition distribution, i.e., $P_{B_i}$ and $P_{\\epsilon_i}$. \n\n\n* The problem formulation of how to select the best-matched transition online data among the offline data is clear. Considering the distance measurement in terms of both embedding space and latent representation is reasonable. \n\n* Using rank-based prioritization in PER to compute $f$ is reasonable. \n\n\n\n", "summary_of_the_paper": "This paper studies the discrepancy between the online and offline transition data in multi-agent RL training. The author proposed OTC to correct the biased transition dynamics and propose a training framework that can leverage offline data for more efficient online training. Results on Mujoco tasks have been provided.  ", "summary_of_the_review": "Though the proposed work is sound on the technical side, I feel not surprising about the proposed method. The novelty is rather limited in the sense that the proposed method is a way to shift the offline data for updates in the context of BCQ. The experiment also seems rather limited; showing its effectiveness over BCQ is also not surprising either. Given the novel setting of offline training with online fine-tuning, the author maybe want to consider transfer learning cases like few-shot settings, and experiments on other multi-agent environment such as StarCraftII will also be favoured.  \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SuMlThIlTBe", "reviewer": "Reviewer_2xuS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty: The bias of transition dynamics in offline decentralized MARL problem is interesting. \n\nSoundness:\n1.This paper studies the deployment constrained setting. However, BREMEN and MUSBO are never compared in the experiments. It is obvious that modifying the transitions given the new data improves the learning performance. \n2.It seems that the OTC technique can also be applied in the single agent setting. Does OTC performs better in MARL than SARL?\n3.In each iteration, OTC needs to find the most similar samples for each sample in the mini-batch, which may incur large computation cost.\n\nSignificance: \nThe performance improvement of OTC compared with BCQ is convincing. But baselines such as BREMEN are missing. \n\n", "summary_of_the_paper": "This paper studies the problem of online tuning for offline decentralized MARL. The online transition correction technique is proposed to correct the bias of the transition probability function between offline dataset and online dataset. Experimental results show that OTC is able to improve the performance of BCQ and AWAC given the same number of deployments and samples.", "summary_of_the_review": "This paper studies an interesting problem in offline MARL. A transition modification technique, OTC is proposed to correct the transition bias during the learning. However, the experimental evaluation is not convincing as important baselines are missing, and the proposed algorithm seems to be too straightforward due to lack of theoretical analysis. Thus, I recommend the rejection.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LYa2KPTFaM6", "reviewer": "Reviewer_a2PP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea that extends deployment efficiency RL to multi agent setting is interesting. Although there are few papers consider offline MARL, they ignore how to consistently improve policy during online deployment. The paper is well-written in most of parts. However, I still have some concerns.\n\n1.\tFor the high-level idea that only using similar transitions to train the policy, is there any analysis? Intuitively, online exploration aims at finding novel samples that not exist in the offline dataset since these novel samples are helpful to know the environment and build a accurate model. However, the novel samples are not prone to be chosen in the algorithm.\n2.\tSome details of the transition function are not introduced. How is the transition function represented? Why can the transition probability be modified by the Eq.3? Any deduction or proof? Do agents share one transition probability function? \n3.\tFor implementation, how many transitions are sampled in Line 16 of the Alg. 1? Since all the transitions should be computed for similarity in Line 13, what is the consumption of time?\n4.\tFor the multi-agent setting, do all agents observe the state rather than their own observations? It seems that the method can be directly applied to single-agent setting. Is there any specific design for multi-agent settings?\n5.\tFor baselines, some offline RL methods are ignored like [1]. I understand that these methods are relatively new and may not be published when authors submit the paper. So I just want to notice authors that offline MARL can also be directly used in the setting of the paper. I would not decrease my score due to this reason.\n\n[1] Offline Decentralized Multi-Agent Reinforcement Learning. Jiechuan Jiang and Zongqing Lu. Arxiv 2021\n", "summary_of_the_paper": "The paper proposes a novel MARL method to bridge offline training and online fine-turning. More specifically, a transition function is maintained and updated according to the data obtained from online deployment. Similar state-action pairs are used to update the RL method.", "summary_of_the_review": "Although the paper is interesting, I still have some concerns on multiple aspects. Three important issues are: 1. The details of the algorithms. 2. The unique design for multi-agent settings. 3. The high-level idea of using similar samples for update.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: ## A Brief Summary of the Paper

 In offline decentralized MARL, the discrepancy between the offline data and the interactions of agents in the environment can cause discrepancy and as a result, the policies will perform suboptimally. This issue in ORL is known as extrapolation error. This paper is trying to address the issue of extrapolation error with offline decentralized agents. It is possible to alleviate this problem by combining offline RL with online fine-tuning. This paper introduces Online Transition Correction (OTC) approach to address this problem which aims to correct the biased transition dynamics with a form of importance sampling based on embedding and value-based distance metrics.
 
 ## Summary of the Reviews
 
 Below I will outline some important concerns raised by the reviewers.
 
 ### Reviewer T8NR
 **Pros:**
 - Interesting and practical setting.
 - Extensive experiments to evaluate OTC.
 **Cons:**
 - Lack of enough discussions about closely related works, for example, the MABCQ algorithm. 
 - Computational cost of OTC.
 - Experiments: comparisons against baselines (MA-ICQ), Large variance in Figure 4. Small-scale (only two agent) setting, can it scale to more agents? 

### Reviewer 2xuS
 
 **Pros:**
 - The bias of transition dynamics in offline decentralized MARL problem is interesting.
 **Cons**
 - Key baselines such as [[BREMEN]] and [[MUSBO]] that looks into the deployment constrained ORL is not compared against in this paper.
 - The proposed OTC algorithm can easily be applied to Single agent settings. MARL vs SARL comparisons would be interesting.
 - Large computational cost incurred from OTC. Because of the search procedure of finding most similar examples to the examples in the dataset.

### Reviewer a2PP
**Pros:**
- Well-written.

**Cons:**
- Analysis on the behavior of the policy, in particular on novelty seeking during the online finetuning phase.
- The missing details of the transition function.
- Compute constraints and budget.
- Unclear experimental protocol: states vs observations...
- Missing baselines.

### Reviewer ZgL6
**Cons**
- Lack of novelty.
- Limited experimental results: transfer learning scenarios, lack of experiments on multiagent environments such as Starcraft II.

## Key Takeaways and Thoughts

Overall, the authors did a good job addressing the concerns raised by the reviewers. For example, the authors ran additional experiments and compared single-agent BCQ with and without OTC on some D4RL tasks. The authors gave detailed responses to the questions related to the computational cost. However, the initial submission version of this paper feels rushed as it is submitted to the conference. I would recommend the authors, go through the reviews carefully and address the points raised by the reviewers carefully in a future version of this paper. As it stands now, it is difficult to evaluate the results reported by the authors during the rebuttal, due to the lack of clarity about their experimental details.

I think the writing can be improved further. There are several typos in the paper, and most reviewers were confused about the novelty of the paper. I would recommend the authors to provide more detailed discussions about the differences from other similar approaches in the literature. Also, justify the selected experimental protocol better.
Decision: Reject