Multi-Agent Domain Calibration with a Handful of Offline Data
Authors: Tao Jiang
Lei Yuan
Lihe Li
Cong Guan
Zongzhang Zhang
Yang Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-agent reinforcement learning
domain transfer
Primary area: reinforcement_learning
TL;DR: This paper formulates domain calibration as a cooperative MARL problem to improve efficiency and fidelity.

Abstract: The shift in dynamics results in significant performance degradation of policies trained in the source domain when deployed in a different target domain, posing a challenge for the practical application of reinforcement learning (RL) in real-world scenarios. Domain transfer methods aim to bridge this dynamics gap through techniques such as domain adaptation or domain calibration. While domain adaptation involves refining the policy through extensive interactions in the target domain, it may not be feasible for sensitive fields like healthcare and autonomous driving. On the other hand, offline domain calibration utilizes only static data from the target domain to adjust the physics parameters of the source domain (e.g., a simulator) to align with the target dynamics, enabling the direct deployment of the trained policy without sacrificing performance, which emerges as the most promising for policy deployment. However, existing techniques primarily rely on evolution algorithms for calibration, resulting in low sample efficiency.
To tackle this issue, we propose a novel framework Madoc (\textbf{M}ulti-\textbf{a}gent \textbf{do}main \textbf{c}alibration). Firstly, we formulate a bandit RL objective to match the target trajectory distribution by learning a couple of classifiers. We then address the challenge of a large domain parameter space by modeling domain calibration as a cooperative multi-agent reinforcement learning (MARL) problem. Specifically, we utilize a Variational Autoencoder (VAE) to automatically cluster physics parameters with similar effects on the dynamics, grouping them into distinct agents. These grouped agents train calibration policies coordinately to adjust multiple parameters using MARL.
Our empirical evaluation on 21 offline locomotion tasks in D4RL and NeoRL benchmarks showcases the superior performance of our method compared to strong existing offline model-based RL, offline domain calibration, and hybrid offline-and-online RL baselines.

Multi-Agent Domain Calibration with a Handful of Offline Data
Tao Jiang 12,33 Lei Yuan 1:,3, Lihe Li Cong Guan
Zongzhang Zhang Yang Yu 12.3
National Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China Schooo of Artificial Intelligence, Nanjing University, Nanjing, China
Polixir Technologies, Nanjing, China (jiangt, yuanl, 1ilh,, guanc) 0lamda. nju. edu. en, (zzzhang, yuy)enju. edu edu. cn
Abstract
The shift in dynamics results in significant performance degradation of policies trained in the source domain when deployed in different target domain, posing a challenge for the practical application of reinforcement learning (RL) in real-world scenarios. Domain transfer methods aim bridge this dynamics gap through tech- niques such as domain adaptation or domain calibration. While domain adaptation involves refining the policy through extensive interactions in the target domain, it may not be feasible for sensitive fields like healthcare and autonomous driving. On the other hand, offline domain calibration utilizes only static data from the target domain to adjust the physics parameters of the source domain (e.g., a simulator) to align with the target dynamics, enabling the direct deployment of the trained policy without sacrificing performance, which emerges as the most promising for policy deployment. However, existing techniques primarily rely on evolution algorithms for calibration, resulting in low sample efficiency. To tackle this issue, we propose a novel framework Madoc (Multi-agent domain calibration). Firstly, we formulate a bandit RI objective to match the target trajectory distribution by learning couple classifiers. We then address the challenge cooperative multi-agent reinforcement learning
large domain parameter space by
modeling domain calibration (MARL) problem. Specifically, we utilize a Variational Autoencoder (VAE) to automatically cluster physies parameters with similar effects the dynamics, grouping them into distinct agents. These grouped agents train calibration policies coordinately to adjust multiple parameters using MARL. Our empirical evaluation on offline locomotion tasks D4RL and NeoRL benchmarks showcases the su- perior performance of our method compared strong existing offline model-based RL, offline domain calibration, and hybrid offline-and-online RL baselines.
Introduction
Reinforcement learning (RL) has gained significant traction various fields [] such as sequential recommendation systems [2] and robotic control [3], demonstrating tremendous potential in real- world applications. However, the inherent trial-and-errof nature of RL limits its application, especially in safety-critica areas such as healthcare [ and autonomous driving [5]. as extensive interactions with the target environment can entail prohibitive costs and pose substantial safety risks. To address this problem, a range of studies have proposed collecting training samples from surrogate source domain (e.g., simulation environment) learn policies, which are then deployed to the downstream
"Equal Contribution CCorreponning Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

target domain 71. Nonetheless, due to complex system dynamics and the characteristics of open environments, a high-fidelity simulator may not always be available [8) leading to severe dynamics shifts between the source and target domains [ Consequently, policies trained optimally in the source domain may fail catastrophically in the target domain. To bridge the dynamics gap, various kinds of solutions have been developed recently. Domain randomization (10) methods, for instance, randomly sample the physics parameters of the source domain and train policies across multiple simulated environments to approximate the target domain. However, as the target domain is often unknown and set in an open environment [8], these methods can also suffer from unpredictable policy degradation, which hinders further development.
Integrating the source domain with some data from the target domain offers a promising solution to the mentioned problem [T]] A class of methods, known as domain calibration, attempts to use data from the target domain as feedback to calibrate the easily obtained source domain and then transfer the policy directly to the target domain. This approach shows enormous potential when the parameters are adjusted accurately enough. Some typical methods automatically tune the physics parameters by minimizing the transition discrepancy between the source and target domains [22 [3] or by maximizing the expected return in the target domain [14]. While these methods can successfully transfer learned policies in robotics 15] they still require interaction feedback from the target domain during training. Instead, DROID 16 and DROPO (17] introduce an offline setting for domain calibration, where the physics parameters adjusted using offline demonstrations pre-collected in the target domain, showing potential for real-world applications.
Nevertheless, in complex real-world scenarios, numerous physics parameters may require calibration.
The ubove-mentioned methods primarily employ evolutionary algorithms [T6 18] or sampling- based methods [19 20 for black-box box optimization, often results in low sample efficiency [1 22
Recently, some algorithms have attempted to mitigate this issue by learning sampling strategies [22]
or leveraging causal discovery 23 to eliminate parameters that have little impact the environment. Despite the effectiveness of these methods, a significant challenge remains in handling complex scenarios where all physics parameters critically influence the dynamics, and different parameters may have varying, or even opposite. impacts on these dynamics (24). A method for efficiently addressing the interrelations among different parameters is urgently needed.
From the perspective of whole-domain calibration, each physies parameter contributes to different aspects of the calibration process. This can be modeled as a typical multi-agent system (MAS) problem (25,, where each agent adjusts a group of domain parameters, and all agents cooperate to reduce the domain gap. This problem can be addressed using cooperative multi-agent reinforcement learning (MARL) (26), leading to the development of the Madoc (Multi-agent domain calibration) framework. Specifically, we first formulate domain calibration matching problem and derive a bandit RL optimization objective by introducing a couple of classifiers to act as the reward model. We then formulate the problem into the MAS where multiple agents calibrate different parameters to reduce the dynamics gap between the source and target domains. Concretely, propose automatic grouping technique cluster physics parameters based on their impacts on the dynamics. We then employ popular value decomposition methods in MARL to train cooperative calibration policies adjust domain parameters. We conduct experiments on popular locomotion tasks to showcase Madoc's superior performance against baselines and highlight the contributions of its core design components. The source code is available at |https //gi Cattuu..comLLAMM..c.. Lom//LMDA--RL//aaacc
target trajectory distribution
2 Related Work
Domain transfer in RL. Transferring RL policies learned from imperfect souree domains to the target domain is a crucial step in the practical use of RL algorithms [II.. However, the trained policy often suffers from severe performance deterioration when directly deployed into the target domain due to the distribution shift between different domains with varying transition dynamics (27]. Previous works have addressed this problem with three common strategies: domain randomization (DR), domain adaptation (DA), and system identification (SI). DR attempts to train a generalizable policy that works well across a variety of randomized simulated dynamics 28 9. While the motivation is simple and often effective, these methods require manually determining which parameters to randomize and may result in underfitting or failing policies due to hand-tuning parameter ranges. DA involves using huge amount of data from source domains to improve policy performance

---

different target domain [29] 30 311. However, these efforts are constrained by the quality and quantity of target domain data and often still require interaction with the target domain. Another line of work, SI, uses measured data to build mathematical models of dynamical systems [3 These methods rely on numerous interactions with the target domain to study how to learn a model of the system dynamics [33 [
which results in learning biased policy with fewer interactions. Most recent works calibrate the parameters of the biased source domain to bridge the domain gap [9 13 [5].
also known domain calibration, and try to improve efficiency by learning a parameter sampling strategy (22] or leveraging causal discovery [23] However, these methods still require interacting with the target domain, posing potential safety hazards during the training process. To mitigate this problem, DROID [16] and DROPO [7 use offline datasets to adjust the source domain parameters via evolutionary algorithms [8] with different optimization objectives, perform poorly when the number of domain parameters is large [35]. Unlike the above methods, we propose to adjust the source domain with handful of offline data, enabling the domain parameters to match the target trajectory distribution with high sample efficiency.
Cooperative multi-agent RL. Many real-world problems are inherently large-scale and complex, making it inefficient and impractical to model them as single-agent systems. Instead, they are more suitably addressed as multi-agent systems (MASs) 255. Multi-agent reinforcement learning (MARL) provides frameworks for modeling and solving such challenges [26]. In scenarios where agents within MAS share common objectives, these problems
categorized under cooperative MARL, which has demonstrated significant advancements in domains like power management [3], path planning (7], and dynamic algorithm configuration (38]. One of the primary challenges in cooperative MARL is the scalability issue 13 40 41, exacerbated by the exponential growth of the search space with the number of agents, complicating policy exploration and learning. Various approaches have been proposed to enhance agent coordination recently. These include policy- methods such as MADDPG [42] and MAPPO (43), value-based techniques like VDN [44] and QMIX 45 and innovations like the transformer architecture [46). Among these methods, value-based approaches
have demonstrated promising results in diverse and complex settings 47 48
VDN leverages
additivity to factorize global value functions, QMIX further enforces monotonicity in global value functions, and DOP [49) introduces value function decomposition within multi-agent actor-critic frameworks. These methods exhibit remarkable coordination capabilities across various tasks such as SMAC, Hanabi, and GRF (26). In this paper, our method formulates domain calibration as a cooperative MARI problem, improving efficiency and fidelity.
3 Background
Reinforcement Learning can be generally modeled as a Markov decision process (MDP) [50]. formulated as a tuple M := (S, AT, r..7% Pu), where S and A denote the state and action spaces, T(s'ls, a) E [o, 1] and r(s, a) represent the transition and reward functions, ? E [0, 1) implies the dis- count factor, and (s(s) is the initial state distribution. The agent running in the environment perceives the state 8, E S at time step l. performs an action d, E AA based on learnable policy x (als) E [0. 1). then the environment receives the action, transits to new state 8t+1, and rewards the agent according to the transition function T(*++1 ||s, ax) and reward function r(8,, 0z) at next time step. The above process is continuously iterated until termination,
can record the whole trajectory of length // + 1 as r = (50, do; To, $1, d1, rl, ....... ......... , dH, TH) and the trajectory distribution over agent's policy and the environment can be defined as (T) = po((o) II T( (a))((()ss)))) The objective of RL algorithms is learn policy (aas)) which maximizes the expected discounted return across the distribution of trajectories, i.c., J(M, x) = E,-dd, (()))(()))))) with R(T) = E 7'r(s, a.).
Domain Calibration aims to adjust manipulable source domain to close the domain gap between it and the target domain. Both the target and source domains can be modeled as MDPs, and the only difference between them is the transition functions which are determined by the physics dynamics parameter vector& & E C R (e.g., friction, mass, damping). Here, denotes the physics parameter space, and N represents the dimension of the physics parameters. Each parameter Ç is bounded on closed interval, which can only be inferred roughly based on experience and expert knowledge, i.c.. We assume the unknown physics parameters of the target domain e* are included
çi E in the parameter space E if the physical modeling is reasonable, i.e., 5" E. as we can set sufficient wide parameter ranges. We now denote the transition function conditioned on domain parameters as T'g T(s' ((s'|s, a, E) and the corresponding MDP as Mg := AS 7, Po). However, the
3

---

Imperfect souree domain
Calibrated source domain
Multi-agen
Target domain
Online RL algorithm
domain Calibration (Madoc)
Final policy
Simulated data ((s, a, Sono
(is, Sire
Cross entropy loss
Domain parameters
Assign
Agest
Simulated datu
Offline
( ] 1
Calibrate % (
((5, G. (ire
Agent 2
[(5, Sares
Reward model
Encoder
Apent
Encode
Value decomposition
Muximize
Team reward
group
MARL.
log TarlSSuels
32 (
Embedding space Multi-agent system
Tere (ere
Figure The conceptual workflow of the multi-agent domain calibration framework. The or- ange arrow represents the simulated data flow in the source domain, with the transition function T(s'ls, a, E), while the blue represents the offline data in the target domain, with the transition function T(s'|s, o, (*). The subscripts "sre" and "tar" are used to distinguish between the source and target domains, respectively. After learning the grouping scheme, we use the red arrow represent the process of domain calibration by MARL value decomposition methods.
manipulable source domain is typically non-differentiable, we can only calibrate the distribution of the source domain parameters (o(:), and the optimal policy learned under this distribution is marked as T (g+) = arg max, Rg-aa( [MM, The objective of domain calibration to learn the source domain parameters that maximize the expected discounted return under the target domain: max4, J (Me-, n (gx)).
4 Method
In this section, we propose the Madoc (Multi-agent domain calibration) framework for leveraging modest amount of offline data from the target domain to calibrate the biased source domain, thus
facilitating optimal policy transfer. The overall workflow of the Madoc framework is shown in Fig. [
We first deduce a bandit RL objective to adjust the domain parameters in Sec. [4.]] improving the synthetic data sampled from the source domain align with the target trajectory distribution. We further model it as a cooperative multi-agen reinforcement learning problem Sec. 4.2 44.2and and use an automatic grouping technique improve the efficiency and fidelity of domain calibration. Finally, practical algorithm under the Madoc framework is presented Sec. 4.3
4.1 Domain Calibration via Reinforcement Learning
Domain calibration the process tuning the parameter distribution of a mismatched source domain to better align with the target domain, which can be realized by comparing the divergence between target domain interactions and simulated synthetic rollouts based on the same policy [3] [3]. However, since interacting with the target domain may not be feasible for sensitive fields, propose an alternative approach to minimize the trajectory discrepancy between the two domains by employing handful of offline target domain data.
Formally speaking, the static offline dataset D {πi, (T,,T2,". ,2....,777, contains A trajectories where = (s6: agj rý. si at, rf......... & Str,ate a r.y), which are collected previously by an unknown behavior policy μ from the target domain, i.c., Ti ~ dy.,,-- (-). By introducing a prior normal parameter distribution p(E) to foster better generalization [51], we intend learn a sample policy æ and calibrate the domain parameters to match the target trajectory distribution:
min DKL (T (T)). 8.98
(1)

---

where the Kullback-Leible (KL) divergence can be further derived as:
t st 7 (8t+ 1841111,, ar, 5)
log (5(
E (-)
log
(2)
tt= (axl(s))
p(5)
H
π (ar()t)) u(())s1))
+ log T(s ar,E)
log (5(()
=E (mga(t) Med
Σ
log
+ 18 B
(3)
T(s++1 18, a1, at E*)
pl()
log (als)
(f log T( T(s'|s,a.E)
BB (s,0)~B
+ DKI (Ko((a(())ll(E))
(4)
p(aas)
where Eq. is an approximation of Eq. als the parameter distribution and trajectory distribution used to calculate the expectation are difficult to compute. Consequently, we use Monte Carlo sampling on the source domain to approximate the expected results. To enhance sampling efficiency, we sample a domain parameter ~ (=(+), generate the trajectories ~ dr.,ee (+), and store the rollouts (s, s', {) in the replay buffer B. By doing so, we are able to convert the trajectory- objective into a transition- based one, following the classic off off-policy RI. paradigm.
delighted to discover that the objective in Eq. 4 can be clearly divided into three terms: the first term, i.e., min, (e,))== log x(ale)
Rs min, (xL attempts to minimize the [{a[s)
KI. divergence between I (a]s) and p(als), we can consider it as variant of behavior cloning; the second term is formulated as maXqs (ss (B)BB log T(a'|s.a.E")
T(ss s,0,g] which can be seen bandit RL T(d')a,.E"), and the last term is regarded objective for policy (o(E) to maximize reward (5) = log T(s'|s,a.))
as a policy regularizer added on (E) to prevent it from collapsing. is worth noting that the policy (5( here is not the one running (sampling) on the source domain, but the one outputting physics parameter vector & as an action to adjust the source domain. To prevent confusion, in the following paper, the policy running on the source domain, i.e., T T(als), is referred to as the "running policy", while the one adjusting the domain parameters, i.e., (C) is referred to the "calibration policy". Additionally, we similarly define the "calibration critic", which is responsible for evaluating the accuracy of the parameters output by the calibration actor. The calibration critic and the calibration policy (actor) together constitute a calibration agent.
The key challenge lies in how estimate the stochastic reward log T(s")s..g") T(.'s.a.E) given the offline data
and simulated rollouts. According to Bayes' rule, we can transform the transition probability 29 as:
a, P(E|s, a, s's,(', P(E)s, a, 8'),(s,,,8''. P(s))(ssa(E)
(5)
P({ls,a)P(s,4))
and the reward can be derived as:
(5) = log P(5*Is, - log P(C*|s (s, a) - log P(Ols a, s')) + log P(E)s.a),
train couple of binary classifiers De (-|s, a, 8') and Dp... (-|s,a) a) to discriminate whether state-actio sitee-accion-state and state-action pairs come from the offline dataset (referred samples (referred to as the binary variable source). These two discriminators form a reward model with certain generalization ability (52), and the corresponding cross-entropy losses are written as:
where e* and stand for the target and source domains respectively. Hence,
the binary variable target) or synthetic
Lerm (,,, flog p (target|ss, a, s')] E(s,a,,"==_[log D Dr.... (sourco (s a, }
Cym (xxa) (log Den.. (target|s, a)] [ss,)) Dw... (source) s,
(6)
4.2 Multi-Agent Domain Calibration
As the complexity of the source domain grows, characterized by expanding number of physics parameters, the calibration policy learned by single-agent reinforcement learning often struggles to consistently reduce the domain gap. To address this challenge, we employ multi-agent reinforcement learning (MARL) algorithms to effectively reduce the search space for single calibration agent, thereby enhancing the efficiency and fidelity of domain calibration.
5

---

We conduct experiments on the HalfChee- tah [53] environment and train to calibrate the gravity coefficient along with other physics pa- rameters. The preliminary results are shown in Fig. 2 we capture snapshots for both ap- proaches at the same training step to investi-
SA Pomson -0.072 077
Absolute calibvation error
10.0
gate Pearson correlation (54) between the critic value of the gravity coefficient and the absolute calibration error. Here, the absolute calibra- tion error represents the absolute difference be- tween the parameters output by the calibration actor and the target parameters. The single- agent method, represented by the blue dots, utilizes just a shared calibration critic for pa- rameter adjusting, facing challenges in assess- ing a specific action within a huge action space, In contrast, the multi-agent method, depicted by the red dots, employs value decomposition algorithms 4 to narrow the action space of each individual calibration policy. This decom- position leads more accurate evaluations of domain parameters, thereby reducing calibra- tion errors and improving policy transfer.
2.5
0.1
0.0
0.2
0.4
0.5
0.5
1.0
Normalized critic value
Figure 2: The Pearson correlation between the critic value of the gravity coefficient and the absolute cal- ibration error. Each dot represents sampled action, which is then fed into the corresponding critic to compute the critic value. When the parameters out- put by the calibration actor are closer to the target parameters (indicating smaller absolute calibra- tion error), the evaluation value output by a "good" critic should be higher.
To utilize the cooperative MARL for domain calibration, we now formally define N agents (since the source domain has a total of N physics parameters) to perform domain calibration coordinately where each of them (5*) attempts to adjust the single parameter 5¹, therefore the joint calibration
policy can be decomposed as g = I (E)) Utilizing any off-the-shelf value decomposition algorithm VD, we employ the single global rewards (R](E)), R. (g (E), to guide individual calibration policy updates:
max Bei ( RG((") - DEL (c)(E))llz((e)))
%
Nonetheless, when the number of the domain parameters N is large, employing agents for domain calibration leads to low exploration and optimization efficiency (55). To mitigate this problem, clustering physics parameters with similar effects on the transition dynamics, e.g., the mass symmetrical robot's left and right feet, into one calibration agent is an advisable choice. Hence introduce an automatic grouping technique by adopting a Variational Autoencoder (VAE) [56]. By adjusting a specific parameter Ç while keeping all other parameters & fixed [57], we assume the identity i of each parameter (i.e., the one-hot encoding) to be representative of the transition ir (8,a, a, 8', )²), the Evidence Lower BOund (ELBO) of the transition is then derived
log P(tx) 2 Ez-)) [log fa [tr[2))] DKL (fo(=li)ll(=)).
(8)
where fe, fa stand for the encoder and decoder, is the latent variable, and p(2) represents the corresponding prior distribution. We can deduce the reconstruction term as:
log (trlz) = log [fa(s']s. a, ff),)).(ss <'|=)],
log fa a, 2)
(9)
where e is a constant as 8, a, and ç do not depend on the latent variable 2. We parameterize the encoder fe and the decoder fa with V, making them fv. and fwa, and optimize the VAE model with the following loss:
(10) Before domain calibration, first minimize the VAE loss (Eq. [O]. Then apply the k-means
Lv -
[log fw, (s'|s, a 2)] + DKL (fu. (=10)]p((2))
clustering method (58) to the means generated by the encoder fu, (2)1) for all i N, in order to group the domain parameters [39]. The resulting n (1
S N) grouping scheme is recorded as [691, .... _. each group forms one agent equipped with calibration actor o (c")) and
calibration critic (E*0), following the multi-agent actor-critic framework.

---

4.3 Practical Algorithm
Based on the above analysis, we now present a practical algorithm under the Madoc framework, the pseudo-code is shown in App. We apply DOP [49], popular multi-agent policy gradient algorithm as the value decomposition method. Concretely, there are n agents for calibrating the source domain after automatic grouping, we factor the joint critic as a weighted summation of individual crities:
Vitot
S0 (5")) b,
(11)
where h; 2 0 and are denoted as learnable weights and biases. The individual critics are learned by pack-propagating gradients from global Temporal Difference updates:
(Vg R2(())
C+ = (s,a,'''
(12)
2
Given individual critics, we use SAC [59] to update the stochastic actors in an off-policy manner:
Ls = E Egs og (ox log to qh
(50h) vo (E )= + ADKI. (1% (c" [1"))) )]
(13)
where a and Å control the relative importance of the entropy and regularization term respectively. Besides, the prior domain parameter distribution (59") is set to be exponentially moving average of the current calibration policy 9% (E9)), which has been shown stabilize training like target network. Finally, parameterize the policy rumning in the source domain with @ and enable it to clone the behavior policy on offline data during domain calibration. Once domain calibration is complete, train the policy me from scratch on the source domain using SAC, and directly deploy it to the target domain.
5 Experiments
In this section, we present the empirical evaluations of our proposed Madoc framework. We first describe the experiment environments and related baselines in Sec. 5.I and then conduct series of experiments to answer the following questions: (1) How the comprehensive performance of Madoc against multiple baselines (Sec. 5.2? (2) How do core components of Madoc contribute to the overall performance (Sec. (.3) 2 (3) How is the generalization capability of Madoc across datasets of varying sizes and source domains with different initial ranges (Sec. 5.4)2
5.1 Experiment Setup
In our experiments, we evaluate Madoc on classic continuous control tasks from the MuJoCo (53 engine and choose two offline benchmarks to serve as offline datasets collected in the target domain. On the popular D4RL benchmark (60), we choose four locomotion tasks (HalfCheetah, Hopper, Walker2d, Ant), each with three types of datasets (medium, medium-replay, medium-experth to evaluate different algorithms'' performance when faced with datasets of varying quality. Considering more challenging scenarios, three environments (HalfCheetah, Hopper, Walker2d) along with three levels of datasets (low, medium, high) from NeoRL benchmark [6]] are also selected. The main difference between the two benchmarks lies in that the static datasets in the NeoRL benchmark occupy more narrow distributions. During the training process, we are given imperfect source domains, and only aware of the initial range of specific physics parameters (gravity, body_mass, _mass, dof_ dammpnng. Each environment has different parameter dimensions, initial ranges, and ground truth values, refer to App. D for detailed information.
Madoc utilizes static offline datasets calibrate biased source domains, and we choose several baseline algorithms with identical or similar settings for comparison. DROPO [17] and DROID [16] use fixed offline datasets from the target domain to optimize the distribution bounds with different objectives via evolution algorithms like CMA-ES [8]. OTED [51] models the parameter optimization process as bandit RL problem, which is similar to our method, but the objective different and cannot cope with large parameter space. H2O 62 and DR+BC are two hybrid iffiin-andd-olingg algorithms where the former penalizes the Q-function learning on simulated state-action pairs with large dynamics gaps, and the latter directly combines uniform domain randomization with behavior
7

---

Table I: Normalized average returns on D4RL benchmark. The results are evaluated in the target domain and we bold the highest mean.
DROPO
DROID
OTED
H20
DR+BC
CQL
MOREC
Madoc-S
Mudoc
hietab-mme
41.0+ 7.9
29.9+ 7.5
761++1177
57.34 3.7
31.9+14.5
52.0+
3.0
73.9+
3.0
93.3- 9.4
91.9+ 7.7
hfctabb-mod-rop
51,2=16.2
45.3#
67.5+134
50.42
38.62
48.9:
84.7=18.6
95.71
hrctab--mddexx
15,4=10.4
46.9+11.3
78.9+12.9
55.3+ 4.1
40.5+ 4.1
53.0+ 2.9
72.0= 3.1
70.7-23.8
96.9 5.3
hogper-med
59.1+34.6
13.9+10.4
49,8+2144
83.7+19.0
43.24224.7
50.34-1888
105.0= 1.0
57.5-17.0
76.0+-13.9
hopper- med rep
43.0:11..7
47.6
65,44126.1
84.1-11.3
13.5
70.0::13.9
28.74
0.7
79.9-34.
90.2::11.7
sugper-med-exp
80.4421.3
21.54 8.3
41.1120.7
89.0111.3
63.1122.0
68.511221
106.02
0.8
47.7=13.0
81.51.18.6
walkerr-eed
61.5::21.1
63.2::12.1
58.8131.6
75.5.1 8.7
57,2
4,5.1: 3.5
84.12 0.8
69.9
90.5117..5
malket-med-rep
19.8=16.6
16.84 8.7
71.2:422.5
83.44 1.3
43.74 5.5
62.4+13..1
85.4= 0.3
60.6=33.1
85.8::20.88
walker-med-exp
60.0=13.8
73.8+ 9.6
74.8:428.2
91.74
61.1+
12:24
86.2+
60.2-18.3
79.9++12.8
ant-med
16,4+12.2
20.8++1..8
65.3++41.8
60.0++26.6
29.2+:12.6
58.04-20.6
64.9+41.2
76.5-29.6
58,7+-24.8
med-rep
64.19-31.9
64.41135.0
62.4-141.9
98.4
14.8-115.0
43.81:33.7
6.1114.11
42.9
81.24416..5
ant-med-cxp
76.7=34.0
64.2:+41.1
70.0+33.5
66.5++22.88
30.3+ 9.2
19.6
67.8=35.6
65
65.6=24.5
101.0:211.5
Average
52.4
47.2
65.1
74.6
43.1
44.8
71.2
68.9
88.3
Table 2: Normalized average returns on NeoRL benchmark. The results are evaluated in the target domain and we bold the highest mean.
Task
DROPO
DROID
OTED
H20
DR+BC
COL
MOREC
Madoc-S
Madoc
HallChhetah--L
49.5216.8
64,8=21.3
60.92 8.7
33.12 3.0
31.34 3.1
32.02 29
50.61 0.3
63,2+:2.77
71.9=10.8
HollChectah-M
33.6t 9.8
30.5= 8.1
64.6=1599
37.1=13.7
34.8: 4.4
51.94
4,8
1.1 0.6
63.9=18.0
85.6= 7,3
HalfCheetah-H
45.4+16.6
47.1-1755
61.0+154
2.5 3.4
10.5
30.6+25.5
58.642229
80.1+10.5
84.2+
Hopper-L
62.6217.0
37,8214.0
39.8219.8
24.1=
2.0
40.3:11.5
20.7+ 8.1
25.64 0.7
56.9216.5
71.21169
Hopper-M M
67,211199
68.3118.1
10.4116.5
39.2.1
6.0
49.4116.4
35.111004
$4.2110.2
47.21134
61.4123.8
Hopper-H
56.6225.9
27.1=14.9
29.4216.1
55.2=
6.5
42.4:15.7
55,1214.1
40.62 3,0
56,0=2444
62.6218.1
Walker2d.11
25.8+ 8.5
9.9 2,9
25.14-20.9
47.3+
3.3
34.1++1430
14.0
33,84
2.9
41.04-25.1
48.74-24.5
62.6 13.0
Walker2d-M
16.81 6.2
44.81174
53.1121.9
44.3
5,3
37.0.L
4.5
57.41
3.1
78.11 1.1
47,5133.1
69.6120.5
Walker2d-H
20.9110.1
11.9 3.8
41.7=26.7
36.0=
7.9
37.8:
9.7
77.1
5.7
20.3::30.7
50.9=22.0
67.9=18.7
Average
42.0
37.2
48.4
35.4
38.8
43.7
41.1
57.2
70.8
cloning. CQL (63 is the classic pure offline RL algorithm using value regularization, MOREC [52 the state-of-thee offline model-based RL algorithm by learning generalizable dynamics reward function, representing the upper- upper-bound performance by solely utilizing offline data. Madoc-S is an ablation algorithm of our proposed Madoc, which only uses a single calibration agent for parameter tuning, without applying the multi-agent decompasition and automatic grouping technique. Madoc and all the above baselines use SAC [59] as the online algorithm after domain calibration for its practicality and convenience on MuloCo tasks.
All the numerical results in our experiments are reported in terms of mean and 95% confidence interval computed over 6 random seeds, and we give more details about the model implementation of our method in App. C.
5.2 Performance Comparison on the Benchmarks
We first evaluate the overall performance of our method and baselines on the two benchmarks and use the normalized average scores introduced by D4RL [60] for intuitive comparison. In realistic scenarios, both the size of offline datasets and the usage of the source domain can be restricted, thus we constrain each algorithm to access static dataset with up only 1 x 10t simulated samples for domain calibration (when used). Due to these constraints, the performance of the baseline algorithms has experienced varying degrees of degradation. As shown in Tab. and Tab. 2 we observe that our methods outperform prior methods on both benchmarks: two pure offline algorithms. CQL and MOREC, suffer from severe performance degradation in some scenarios, especially when encountering narrow NeoRL datasets; two hybrid offline-and-onie algorithms are limited by the dataset quality and domain fidelity, and cannot achieve the expected performance; DROPO and DROID employ evolutionary algorithms to optimize domain parameters, encountering challenges in transferring learned policies when the parameter space is large; the comparison with OTED demonstrates the rationality of our optimization objective described in Sec. 4.1 while the comparison with Madoc-S further illustrates the effectiveness of modeling domain calibration as a multi-agent system formulated in Sec. 4.2 To verify that Madoc indeed reduces the dynamics gap between the source and target domains, we report the mean absolute calibration error for different algorithms in App. E.I Furthermore, given the significant variance in domain
10 transitions and execute
8

---

110
Madec
Normalizee ined averagee retur
M
100
perfomaanc gop
60
Dimension domaii paramelens
(a) Visualization of the automatic grouping technique
(b) The performance gep between Madoc and Madoc-S
Figure 3: (a) The visualization results of the automatic grouping technique. The left part is schematic of the Hopper robot, each point on representing physies parameter to be calibrated. and different colors indicate the final grouping results. For example, the four parameters encircled by the yellow rectangle are clustered into one group in the embedding space on the right part. (b) The normalized average return of Madoc and Madoc-S on the Ans environment. We can observe that as the parameter dimension of the source domain increases, the performance gap between them (indicated by the black shadow) becomes more pronounced.
calibration algorithms, we provide detailed discussion on the stability of the experimental results in App. E33
5.3 Effectiveness of Different Components
To investigate the impact of the design components of Madoc, we first design experiments on the Hopper environment visualize the automatic grouping technique, shown in Fig. 3(a). The Hopper is a one-legged robot simulation with four distinct body sections: the torso, thigh, leg, and foot. These components connected by three joints, which serve the articulation points between each pair of bodies. We need to calibrate the gravity coefficient, the mass of each body, and the damping coefficients each joint. After projecting the embedding space onto three-dimensional space, we can discover that the robot's physics parameters are divided in an orderly manner from top bottom, while the gravity coefficient forms separate group on its own. The result aligns perfectly with our expectations, as parameters that have similar impacts on dynamics can be adjusted using one calibration policy. Additionally, in Fig. ()), we conduct ablation studies on the Ant environment of the D4RL benchmark, to verify the effectiveness of modeling domain calibration as an MARL problem. When the dimension of the domain parameters gap between Madoc and Madoc-S; however, as the source domain becomes more complex, the performance of both declines, but the drop is faster when only one agent employed for parameter calibration. This experimental result favorably supports that modeling as MARL can enhance the efficiency of domain calibration large parameter spaces.
there no significant performance
5.4
Generalization across Various Conditions
Madoc leverages static offline datasets adjust the domain parameters, driving our curiosity toward its generalization capacity under varying dataset sizes and initial parameter ranges the source domain. Here we choose all the above-mentioned tasks on the NeoRL benchmark and calculate the averaged normalized returns for comparison. As illustrated in Fig. 4(a), the algorithms access datasets of different magnitudes, 5 x 10* (small), x 10a (medium), and x 1 (large), to reflect spectrum of data availability. The results reveal that the effectiveness of both the hybrid dffline-and-olligg H2O algorithm and the purely offline MOREC algorithm rises with the expansion of the dataset size. This discovery suggests that their dependency on the size of offline data for improved performance. On the contrary, our method maintains stable and excellent performance, unfazed by the dataset size. Besides, for source domains with various initial parameter ranges, categorized easy, medium, and hard (see App. [ more details), Madoc exhibits remarkable performances across all levels, particularly excelling in "hard" cases with the largest parameter search space, shown in Fig. 4b).

---

Normalized average return
50-
Normalizeed
48-
30-
50
20-
20
10-
10
Medium
Lange
Hard
Narmral
Faac
(a) Performance with different datasets
Performance under different initial domain parameter ranges
Figure 4: The generalization ability of Madoc compared to the baselines under different conditions.
This underscores our algorithm's ' effectiveness coping with challenging scenarios, affirming its robustness and adaptability in diverse conditions.
6 Conclusion and Discussion
In this paper, we introduce Madoc, framework for closing the dynamics gap by calibrating the source domain with handful of offline data via multi-agent reinforcement learning. Concretely, the target domain data serve as a guide for target transition dynamics, which is leveraged to train classifiers generating rewards and derive bandit RI. objective for domain calibration, To improve calibration efficiency with a large number of parameters, we further model it cooperative MARL problem and propose to group parameters with similar effects on dynamics. Experiments on popular control tasks demonstrate that our method can calibrate the source domain with sufficient accuracy, allowing the optimal trained policy to be transferred to the target domain without severe performance deterioration. One possible constraint of our method is that, when dealing with high -inh-eminniinall vision tasks [64], using handful of offline data may not guarantee the accuracy and generalizability of the reward model. This challenge could be mitigated by deploying more expressively powerful tools like diffusion models [65] which is left for future work.
Acknowledgements This work supported by the National Science Foundation of China (62276126, 62250069), the Natural Science Foundation of Jiangsu (BK20221442, BK20243039, and BK2024119), and the Fundamental Research Funds for the Central Universities (0221/1438002))
References
[1] Xu Wang, Sen Wang, Xingxing Liang. Dawei Zhao, Jincai Huang, Xin Xu. Bin Dai, and Qiguang Miao. Deep reinforcement learning: A survey, IEEE Transactions Neural Networks and Learning Systems, 35(4):5064-500 5078, 2024.
[2] M Mehdi Afsar, Trafford Crump, and Behrouz Far. Reinforcement learning based recommender systems: survey. ACM Computing Surveys, 55(7):1-38, 2022.
[3] Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applications: A comprehensive survey. Artificial Intelligence Review, 55(2):945 990999999 2022.
[4] Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: survey, ACM Computing Surveys, 55(1):1-36, 2021.
[5] B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions Intelligent Transportation Systems, 23(6):4909 2021.
[6] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladien Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robotics, 5(47):5986, 2020.
10

---

[7] Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In Robotics: Science and Systems, 2018.
[8] Zhi-Hua Zhou. Open-environment machine learning. National Science Review, 98)) 123, 2022.
[9] OpenAI, Ilge Akkaya, Marcin Andrychowicz, Macick Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. Solving rubik's 's cube with robot hand. arXiv preprint arXiv:1910.07113 2019.
[10] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In International Conference on Intelligent Robots and Systems, pages 23-30, 2017.
[11] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: A survey. In IEEE Symposium Series on Computational Intelligence, pages 737-744, 2020.
[12] Yuqing Du, Olivia Watkins, Trevor Darrell, Pieter Abbeel, and Deepak Pathak. Auto-tuned sim-to-real transfer. In IEEE International Conference on Robotics and Automation, pages 1290-1296, 2021.
[13] Fabio Ramos, Rafael Possas, and Dieter Fox. Bayessim: Adaptive domain randomization via probabilistic inference for robotics simulators. In Robotics: Science and Systems, 2019.
[14] Fabio Muratore, Christian Eilers, Michael Gienger, and Jan Peters. Data-efficient domain randomization with bayesian optimization. IEEE Robotics and Automation Letters, 6(2):911- 918, 2021.
[15] Marius Memmel, Andrew Wagenmaker, Chuning Zhu, Dieter Fox, and Abhishek Gupta. ASID: Active exploration for system identification and reconstruction in robotic manipulation. In International Conference on Learning Representations, 2023.
[16] Ya-Yen Tsai, Hui Xu, Zihan Ding, Chong Zhang, Edward Johns, and Bidan Huang, DROID: Minimizing the reality gap using single-shot human demonstration. IEEE Robotics and Automa- tion Letters, 6(2):3168-3175, 2021.
[17] Gabriele Tiboni, Karol Arndt, and Ville Kyrki. DROPO: Sim-to-real transfer with offline domain randomization. Robotics and Autonomous Systems, 166:104432. 2023.
[18] Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolu- tion strategies. Evolutionary Computation, 9(2):159-19)) 2001.
[19] Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In International Conference on Robotics and Automation, pages 8973-8979, 2019.
[20] Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In AAAI Conference on Artificial Intelligence, pages 1607-1612, 2010.
[21] Marcin Andrychowicz, Bowen Baker, Macick Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[22] Bhairav Mehta, Manfred Diaz, Florian Golemo, Christopher Pal, and Liam Paull. Active domain randomization. In Conference on Robot Learning, pages 1162-1 176, 2020.
[23] Peide Huang, Xilun Zhang, Ziang Cao, Shiqi Liu, Mengdi Xu, Wenhao Ding, Jonathan Francis, Bingqing Chen, and Ding Zhao. What went wrong? closing the sim-to-real gap via differentiable causal discovery. In Conference on Robot Learning, pages 734-760, 2023.
11

---

[24] Nima Fazeli, Russ Tedrake, and Alberto Rodriguez. Identifiability analysis of planar rigid-body frictional contact. Robotics Research: Volume 2. pages 665-682, 2018.
[25] Ali Dorri, Salil S Kanhere, and Raja Jurdak. Multi-agent systems: A survey. IEEE Access, 6:28573-28599 2018.
[26] Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. A survey of progress on cooperative multi-agent reinforcement learning in open environment. arXiv preprint arX1v:2312.01058. 2023.
[27] Manuel Kaspar, Juan D Muñoz Osorio, and Jürgen Bock. Sim2real transfer for reinforcement learning without dynamics randomization. In International Conference on Intelligent Robots and Systems, pages 4383-4388, 2020.
[28] Fereshteh Sadeghi and Sergey Levine. CAD2RL: real single-image flight without single real image. In Robotics: Science and Systems, 2017.
[29] Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhut- dinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In International Conference on Learning Representations, 2020.
[30] Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross-domain policy adaptation via value-guided data filtering. In Advances in Neural Information Processing Systems, pages 73395-73422 2023.
[31] Xiong-Huu Hui Chen, Shengyi Jiang, Feng Xu, Zongzhang Zhang, and Yang Yu. Cross-modal domain adaptation for cost-efficient visual reinforcement learning. In Advances in Neural Information Processing Systems, pages 12520-12532, 2021.
[32] Lennart Ljung. System identification. In Signal Analysis and Prediction, pages 163-173. Springer, 1998.
[33] Horia Mania, Michael Jordan, and Benjamin Recht. Active learning for nonlinear system identification with guarantees. Journal of Machine Learning Research, 23(3)): 2022.
[34] Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification. In Robotics: Science and Systems, 2017.
[35] Zhi-Hua Zhou, Yang Yu, and Chao Qian. Evolutionary learning: Advances in theories and algorithms. Springer, 2019.
[36] Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim Green. Multi-agent reinforcement learning for active voltage control on power distribution networks. In Advances in Neural Information Processing Systems, pages 3271-3288 2021.
[37] Guillaume Sartoretti, Justin Kerr, Yunfei Shi, Glenn Wagner, TK Satish Kumar, Sven Koenig, and Howie Choset. PRIMAL: Pathfinding via reinforcement and imitation multi-agent learning. IEEE Robotics and Automation Letters, 4(3):2378-233 22878 2019.
[38] Ke Xue, Jiacheng Xu, Lei Yuan, Miging Li, Chao Qian, Zongzhang Zhang, and Yang Yu. Multi-agent dynamic algorithm configuration. In Advances in Neural Information Processing Systems, pages 20147-20161, 2022.
[39] Filippos Christianos, Georgios Papoudakis, Muhammad A Rahman, and Stefano V Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning, pages 1989-1998, 2021.
[40] Lei Yuan, Tao Jiang, Lihe Li, Feng Chen, Zongzhang Zhang, and Yang Yu. Robust coopera- tive multi-agent reinforcement learning via multi-view message certification. Science China Information Sciences, 142102,, 2024.
[41] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Interna- tional Conference on Machine Learning, pages 330-337, 1993.
12

---

[42] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative environments. In Advances in Neural Information Processing Systems, pages 6379-6390, 2017.
[43] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Advances in Neural Information Processing Systems; pages 24611 -4611-24624, 2022.
[44] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. sdlee-dccopoositoon networks for cooperative multi-agent learning based on team reward. In International Conference on Autonomous Agents and MultiAgent Systems, pages 2085-2087, 2018.
[45] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt. Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1-51, 2020.
[46] Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. In Advances in Neural Information Processing Systems, pages 16509-16521, 2022.
[47] Rihab Gorsane, Omayma Mahjoub, Ruan John de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius. Towards standardised performance evaluation protocol for cooperative marl. In Advances in Neural Information Processing Systems, pages 1510-5521, 2022.
[48] Lei Yuan, Lihe Li, Ziqian Zhang, Fuxiang Zhang, Cong Guan, and Yang Yu. Multiagent continual coordination via progressive task contextualization, IEEE Transactions on Neural Networks and Learning Systems, 2024.
[49] Yihan Wang, Beining Han, Tonghan Wang, Heng Dong, and Chongjie Zhang. DOP: Off- policy multi-agent decomposed policy gradients. In International Conference on Learning Representations, 2020.
[50] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. Adaptive computation and machine learning. MIT Press, 1998.
[51] Izzeddin Gur, Ofir Nachum, and Aleksandra Faust. Targeted environment design from offline data. In Deep RL Workshop NeurIPS, 2021.
[52] Fan-Ming Luo, Tian Xu, Xingchen Cao, and Yang Yu. Reward-consisten dynamics models are strongly generalizable for offline reinforcement learning. In International Conference on Learning Representations, 2023.
[53] Emanuel Todorov, Tom Erez, and Yuval Tassa. MuJoCo: A physics engine for mosel-based control. In International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[54] Philip Sedgwick. Pearson's 's correlation coefficient. Bmj, 345, 2012.
[55] Jayesh K Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Best Papers, pages 66-83, 2017.
[56] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[57] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In AAAI conference on Artificial Intelligence, 2018.
[58] Tapas Kanungo, David M Mount, Nathan S Netanyahu, Christine D Piatko, Ruth Silverman, and Angela Wu. An efficient k-means clustering algorithm: Analysis and implementation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(7):881-895 2002.
13

---

[59] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870, 2018.
[60] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
[61] Rong-Jun Qin, Xingyuan Zhang. Songyi Gao, Xiong-Hur Chen, Zewen Li, Weinan Zhang, and Yang Yu. NeoRL: A near real-world benchmark for offline reinforcement learning. In Advances in Neural Information Processing Systems, pages 24753-24765, 2022.
[62] Haoyi Niu, Shubham Sharma, Yiwen Qiu, Ming Li, Guyue Zhou, Jianming Hu, and Xianyuan Zhan. When to trust your simulator: Dynamics-aware hybrid offline-and-onling reinforcement learning. In Advances in Neural Information Processing Systems, pages 36599-36612, 2022.
[63] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Advances in Neural Information Processing Systems, pages 1179-1191, 2020.
[64] Md Tanzil Shahria, Md Samiul Haque Sunny, Md Ishrak Islam Zarif, Jawhar Ghommam, Sheikh Iqbal Ahamed, and Mohammad H Rahman. comprehensive review of vision-based robotic applications: Current state, components, approaches, barriers, and potential solutions. Robotics, 163):139, 2022.
[65] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, pages 6840-6851, 2020.
[66] Yang Yu. Towards sample efficient reinforcement learning. In International Joint Conference on Artificial Intelligence, pages 5739-5743, 2018.
[67] Petar Kormushev, Sylvain Calinon, and Darwin G Caldwell. Reinforcement learning in robotics: Applications and real-world challenges. Robotics, 222):122-148, 2013.
[68] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rêmi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine. Çaglar Gülçehre. Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy P. Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi multi-agent reinforcement learning. Nature, 575(7782)330--335 2019.
[69] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, pages 1889-1897, 2015.
[70] Hans-Georg Beyer and Hans-Paul Schwefel. Evolution strategies A comprehensive introduc- tion. Natural Computing, 1:3-52, 2002.
[71] Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems, pages 2960-2968, 2012.
[72] Shimon Whiteson.
Evolutionary computation for reinforcement learning. Reinforcement Learning: State-of-the-art, pages 325-355, 2012..
[73] Thomas M. Moerland, Joost Broekens, Aske Plaat, and Catholijn M. Jonker. Model-based reinforcement learning: A survey. Foundations and Trendso in Machine Learning, 16(1):1-118, 2023.
[74] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: survey. The International Journal of Robotics Research, (2(11)) 2013,
14

---

[75] Ahing-An Cheng, Andrey Kolobov, and Alekh Agarwal. Policy improvement via imitation of multiple oracles. In Advances in Neural Information Processing Systems, pages 5587-5598, 2020.
[76] Eliseo Ferrante, Alessandro Lazaric, and Marcello Restelli. Transfer of task representation in reinforcement learning using policy-based proto-value functions. International Conference on Autonomous Agents and MultiAgent Systems, pages 1329- 1239, 2008
[77] Richard S Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1- 2):181-211, 1999.
[78] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.0164. 2020.
[79] Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International Conference on Machine Learning, pages 28701-28717, 2023.
[80] Tianhe Yu, Garrett Thomas, Lantao Yu. Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. In Advances in Neural Information Processing Systems, pages 14129-14142, 2020.
[81] Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology, 61(3)611--62,, 1999.
[82] Christian Schroeder De Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the StarCraft multi-agent challenge? arXiv preprint 20Xv:2001.095333 2020.
15

---

Appendix
A Algorithm Description
The pseudo-code of Madoc presented in Alg. utilize SAC [59 and DOP 49 algorithms for domain calibration. From lines I to 9, we group the domain parameters by learning VAE and employing k-means; lines 13 to 16 aim at training discriminators as the reward model and regularizing the running policy; the calibration actors and critics are updated from lines 17 to 19; finally, we train the running policy from scratch and deploy the target domain.
backbone
Algorithm Madoc
Input: A pre-collected dataset D, an imperfect manipulable source domain M
Initialize: Policy иe. joint calibration actor ds and critic Up. couple of binary discriminators Da VAE fe. prior distributions p(E) and p(z), replay buffer B = 0 for simulated rollouts, step for parameter grouping T Igroup. step for domain calibration calibraaioon learning rate 1]
1: for step t = 1, group do
2: Behavior cloning:
3: Ta +- BC(==, D)
4: Collect simulated data:
5: B + ROTLOOUTT M)
6: Update VAF:
7: Jw fv vL according to Eq. 10
8: end for
9: Get groups by k-means, clear replay buffer B 0. and reinitialize policy we
10: for step t = T group + +, 1.. Tgroup + Tealibration do
Collect simulated data:
12: B < B - SUROOLOUUT( (T......M))
Update discriminators:
14: D& + De 1/ VVVLy according to Eq.
Behavior cloning:
16:
Wo +- BC( D)
Update the joint calibration actor and critic:
18: Uq. 14 V Lop according to Eq. T2
19:
Go - Uo - V aLs according to Eq. 13
20:
end for
21: Train policy from scratch:
22: Th < SAC((T. 9a, M))
23: Output: Policy xe
B Extended Related Work
Sample efficient RL. A significant drawback of current reinforcement learning methods is their poor sample efficiency, leading to extensive environmental interactions [66]. This results in prohibitive costs in real-world applications [67] and hinders policy learning even in complex digital environments, like the full StarCraft game [68]. Multiple factors may restrict the sample efficiency, while several aspects can alleviate these limitations. Different optimization methods possess varying capabilities for exploring and exploiting data. Mainstream algorithms in reinforcement learning commonly rely on the gradient of the objective or surrogate objectives 66 Still, there are also sample-based methods, a.k.a. derivative-free optimization, that offer their unique advantages. Typical algorithms like evolutionary algorithms (70) and Bayesian optimization [7T] have been applied for reinforcement learning 722. showcasing better performance on some tasks. Nonetheless, these methods also encounter certain drawbacks, including slow convergence, difficulties scaling, sensitivity to noise, and lack of theoretical guarantee. From another perspective, model-based algorithms can be much more efficient as planning in the model free of real-world samples in ideals 73]. Nevertheless, in high-dimensional environments, learning an accurate transition model using supervised learning is challenging. Employing manually constructed environments, i.e., simulators, is more efficient and practical, especially in the field of robotics [74] However, it also encounters problems due to
16

---

the sim2real dynamics gap, and simulators are generally non non-differentiable, unlike neural models. Besides the aspects discussed above, the ability to transfer knowledge is also crucial for improving the sampling efficiency. Rather than starting from scratch for each task, humans continuously learn and build upon experiences from a variety of tasks. Many methods have proposed various types of knowledge transfer, such as policy transfer [75], representation transfer (76), and skill transfer [77]. However, these methods work effectively only in specific cases. A general approach to transfer reinforcement learning is yet to be developed.
Offline RL. In the offline RL setting. the agent no longer has the ability interact with the environ- ment 778. Instead, the learning algorithm is access to static offline dataset, collected previously by an unknown behavior policy from the target domain. The main obstacle to this data-driven learning paradigm is the distribution shift problem due to the discrepancy between the learned and behavior policies, leading to severe extrapolation error. Previous works tackle this problem by constraining the learned policies against the behavior policy [79] or penalizing the value function on
(OOD) actions [63] both of which require large enough datasets. Besides, offline model-based RL algorithms learn dynamics model from offline data to enhance the efficiency of offline RL [80]. Benefiting from generating the synthetic data by the learned dynamics, model-based algorithms improve the coverage of the dataset. However, this depends on accurate dynamics models and still requires sufficient offline data. It is worth noting that in this work we use offline datasets to calibrate domain parameters, thereby avoiding the selection of OOD actions, and alleviating the requirements for dataset quality and quantity.
C Implementation Details
Discriminators. We train two discriminator networks De (-Is, a') and Des, (+|s, a) to classify whether the rollouts comes from the offline dataset or the source domain in our method. We follow the implementation in DARC (29), propagating gradients back through both discriminators as follows:
Dem (-|s, a) SoftMax((fa(s.o),
Dinm (-18, a, 8') Soft a, 8') (s)(s,
where (s, and [sas(s, (s, a, 8') represent the outputs of the two classifier networks, Moft.aa((::) = exp mi Exyex expay" X = (target, source]. Both networks contain two hidden layers, while dropout layers
and ReLU activations are used between layers. Before the final Maxt layer, employ the Tanh activation function to map the outputs into probabilities.
Grouping VAE. We train both the encoder and decoder with two hidden layers, both dropout layers and ReLU activation functions are used between layers. We employ MSE loss for state reconstruction and eliminate KL divergence, as our goal is to encode the agent identity rather than generating states. For each domain parameter, we encode its identity into a 16-dimensional latent space. Then the latent variable is concatenated with the current observation and action, and the combined data is fed into the decoder to predict the observation for the next time step. The latent variables of all parameters are projected to three dimensions using Principal Component Analysis (PCA) [8I) after which they grouped using the k-means algorithm extract schemes from the last phase of group training and determine the final grouping scheme by choosing the one that occurs most frequently.
[58]. To reduce randomness in the group results, we repeatedly
the multi-agent decomposition method because it is popular off-policy multi-agent policy gradient method. As claimed in the paper, individual critics vg are learned by backpropagating gradients from global TD updates, guiding the updates of individual calibration actors by similar objective in SAC
Calibration policy. We use DOP 49
[59]:
L = E ggga (-)
[x log 0% (g#t) g (E9i) ADRL (1% ((o))
Here, we do not employ the double-Q function and the automatic tuning scheme of a. Instead we update > min (10, exp( and prior pd* (Est) as target network of (E with smoothing coefficient x 10 3 to stabilize training. The prior parameter pd ((*)) ) is distribution initialized as normal distribution A(0, 1) enhance exploration the early stages of training.
We scale the output of the calibration actors by using a Gaussian with Tanh squashing, e.g., we rescale the output of gravity to be within [-30, 0] in our preliminary experiments. To further enhance the
17

---

Table 3: The comparisons of method complexities.
Method
GPU Memory Cost
Contained Modules
Madoc
39SMB
reward models, grouping VAF, calibration agents, and running agents
CQL MOREC
286MB
running agents
1053MB
dynamics reward function, dynamics models, and running agents
Table The comparisons of computational costs.
Method
Total Training Time
Average Training Time per Epoch (over 1000 epochs)
Madoc
5 hours
1.8s for grouping (200 epochs), 14s for calibration, 4s for SAC
CQL. MOREC
2 hours
7s for policy training 5s for training dynamics reward function, 16s for policy training
6 hours
Table 5: The common hyper-parameter Madoc.
Attribute
Value
Calibration actor learning rate
3 x 10
Calibration critic learning rate
3 x 10-4
The temperature coefficient of calibration actor
0.05
Hidden layers of the calibration critic network
[64, 64
Target network smoothing coefficient for calibration actor Target network update interval for calibration actor
x 10-3
10
Discriminator learning rate Hidden layers of the discriminator network The dropout rate of the discriminator network Running actor learning rate Running critic learning rate Hidden layers of the running actor network Hidden layers of the rumning critic network Target network smoothing coefficient for running actor Target network update interval for running actor
1 x 10-3
[256, 256)
0.2
3 x 10
3 x 10
[256, 256
[256, 256
5 x 10-3
VAE learning rate
3 x 10-4
Hidden layers of the VAE
[256, 256)
Latent variable dimension of VAE The dropout rate the VAE Batch size
16
0.5
256
Optimizer
Adam
Discount factor ?
0.99
Buffer size
2 x 10
exploration of domain parameters, we sample series of domain parameters every 100 rollouts the source domain. After calibration, we set the physics parameters with the means outputted by actors, and train SAC from scratch with default hyper-parameteer
Most experiments were conducted on server outfitted with 13th Gen Intel(R) Core(TM) 19-13900K CPU, 2 NVIDIA RTX A5000 GPUs, and 125GB of RAM, running Ubuntu 22.04. We also conduct comparison between Madoc and various offline RL algorithms with regard to method complexities and computational costs on the hfftah- med---eedrppppp- task of the D4RL benchmark, the results are presented in Tab, B3 Tab, 4 Compared to the traditional pure offline RL algorithm CQL, Madoc incorporates additional modules that lead to increased GPU memory cost. Furthermore, it encompasses three training stages, which consequently require more computational cost. Nevertheless, given the significant performance improvement, these extra expenditures are deemed justifiable. We list the default hyper- parameter settings for Madoc in Tab.
18

---

aa) Half' Cheetah
(b) Hopper
(c) Walker2d
(d) Ant
Figure 5: illustration of environments used our experiments.
D Experiment Details
D.1 Extended Environment Descriptions
We adopt the popular Gym-MuJoCo tasks 53 as our benchmarks used in the experiments. To investigate the performance of method and baselines, conduct experiments four locomotion tasks shown in Fig. 5 including HalfCheetah, Hopper, Walker2d, Ant.
HalfCheetah is 2D robotic simulation featuring links and 8 joints, including two paws, designed to move forward rapidly by applying torque on 6 specific joints. The main objective of this robot centers maximizing forward progression to carn positive rewards while minimizing backward movement to avoid penalties.
Hopper is designed to enhance the complexity with more state and control variables than traditional control settings, using a wwoddimensional, one-legged hopper composed of a torso, thigh, leg, and foot. The overall aim is to propel the hopper forward by exerting torque the three joints that link its four body sections.
Walker2d a two-dimensional figure with two legs, including a torso, two thighs, two legs, and two feet, focusing on coordinated movement advance. The objective is to utilize torque on six joints synchronize the movement of all six body parts toward the desired direction.
Ant is 3D model with freely rotating torso and four two-linked legs, designed for coordinated forward movement. The aim to maneuver the ant by applying torques on eight hinges to effectively control the movement of its nine parts toward the desired direction.
The number groups n for each task is 6, 4, 6, and 6, respectively. We verify in extended experiments (shown in Fig. ([)) that the performance results do not differ significantly as long as it is within reasonable range. report the initial ranges and ground truth value of each physics parameter in
Tab. Tab. Tab. and Tab. g
19

---

Table 6: The setting of HalfCheetah physics parameter at easy/normal/hard level.
Physics Parameter
Initial Range (casy/norrmalhhard)
Ground Truth
gravity
[ 15, - 6]/[--3, 0)/[-50, 0]
9.81
66. 7//55 7]
6.36
body body_mass_? mass__as_2
[1,2]/(0. [0.
1.54
mady_mass_3
[1. [1.2]/[0, 21/[0,3]
1.58
body_mass.
[0.5, 1.5]/[000000 [],
1.07
body_mass_5
[1,2]/[0, 2]/[0,3]
1.43
body_man mady_mass_9
[0.5, 1.5]/[0, 2),
1.18
body mass_ 7
[0.5, 1.5] [0
0.85
dof damping 3
[4.5, 7.5) 7.5] /00 7.5]
dof _of_dnmping.... 4
33 (].5, 7.5| / 7.5]
4.5
dof damping 5
[1.5, 4.5]/[1.5, 7 7.5)
/ [0, 7.5]
3
dof dof_ping 6
33 6) (0, 6)/ 6)/ 7.5]
4.5
dof diippnnnnn 7
[1.5, 4.5]/[0 [o. 6)/ 7.5)
dof damping_S
[o. 3 (0 7.5]
1.5
Table The setting of Hopper physics parameter easy/normal/hard level.
Physies Parameter
Initial Range (easy/normal/hard)
Ground Truth
gravity
15, 5| 0|/[-50,0
-9.11
body_ mady_mass__ 1
[3, 53/
3.53
body _aa__mas____
[3.5, (..5]/(,,55/2,,
3.93
/2,,
2.71
body_ fiily_mass_
(.5,5.5)// 6),
5.08
dof damping_3, 33, 3,4,5
(0,2]/10,33/1044]
1
Table 8: The setting of Walker2d physics parameter easy/normal/hard level.
Physics Parameter
Initial Range (eass//orrmm////arrd))
Ground Truth
graviry
| 15, 5//1-30,00//111
-9.81
body mass 1
33, [3.4]/
3.53
body mass 2,
[3.5, 43.]/[3]33... 5]/22,
3.93
Mody_mass__
[2,3]/[2, 4]
2.71
body paal_mass_A4 4,7
[2.5, 4]/[1,5]
2.94
dof ddmddaamingg 3,4,5, 5, 6, 7,8
[0,0.2)/(0, 0.5) 10, [0
0.1
20

---

Table 9; The setting of Ant physics parameter at normal level.
Physies Parameter
Initial Range
Ground Truth
gravity
1-30,00 0]
-9881
body_mass
[0,0.5] 0.5]
0.33
body mass mass__,,3 3, 5,
0.1]
0.036
body body_mass ____mass__44
0.1]
0.065
dof_ damping 7, 27,889
[0.3]
1
D.2 Baselines
Here introduce the baselines used in experiments, including offine domain calibration, hybrid offline-and-onling RL, and pure offline RL algorithms.
DROPO 17 adapts a distribution of dynamics parameters match offline dataset by employing probabilistic distance measure, aimed at directly maximizing the likelihood of replicating real- world data within a simulation. Consequently, the simulator be envisioned model, where the inherent randomness is attributed to variations in the scene's physical parameters.
stochastic forward
DROID [16 harnesses human demonstrations to synchronize the simulator's trajectories with those observed in the real world, rather than relying on guesswork or exhaustive adjustments to establish the domain randomization (DR) range. This process helps in finding the most suitable range of parameters for the simulator, which can be formulated as a statistical model. Subsequently, this model can be sampled to inform the training process of RL agents.
OTED [5]] designed to autonomously learn a of simulator parameters that align with a given offline dataset. Using the calibrated simulator, it proceeds to train Reinforcement Learning (RL) agent using conventional online methods. An objective is formulated for the tuning of simulator parameters, aiming to minimize a divergence metric between the state-action distribution generated by the simulator and the provided target offline dataset.
presents a novel policy evaluation framework that is aware of dynamics, adaptively impos- H20 62
ing penalties on Q-function training for simulated state-action pairs that exhibit significant dynamics discrepancies. At the same time, it permits learning from a predetermined dataset originating from the real world without direct interactions with it.
CQL [63 enhances the conventional Bellman error objective by adding an uncomplicated Q-value regularization term, which is easy to apply most current deep Q-learning and actor-critic models. The objective of CQL is to overcome existing limitations by training a conservative Q-function, ensuring that the policy'ss 's expected value, estimated by this Q-function, is conservative estimate of its actual value, thus avoiding selecting OOD actions.
acquires dynamics reward function that can be generalized from offline data. This reward function is then utilized as a transition filter within any offline Model-Based Reinforcement Learning (MBRL) approach. During the transition generation process, the dynamics model produces a set of possible transitions, from which the one with the highest dynamics reward value chosen for selection and used for policy update.
MOREC 52
E
Additional Experiment Results
E.1
The Absolute Calibration Error
We report the corresponding mean absolute calibration error of experiments in Sec. S.c. As shown in Tab. an Tab. [T] there are five methods performing domain transfer by tuning simulator parameters. It is apparent that the evolutionary algorithm-based methods, DROPO and DROID, result in huge calibration errors, which suggests their limited effectiveness within the realm of high-dimensional parameter spaces. Additionally, Madoc-S outperforms OTED, highlighting the more rational design
21

---

Table 10: The mean absolute calibration enror on D4RL. benchmark, We bold the lowest mean.
Task
DROPO
DROID
OTED
Madoc-S
Madoc
hfetah-med
0.78+1018
1.13:0.34
0.28:0.03
0.13+0.04
0.0610.01
hedtrhh-md-ree
0.64+0.15
1.35==0.40
0.61+0.11
0.23++010
0.12+00.02
heetah-med-exp
0.66++0.14
0.87=0.22
0.33++0.06
0.10++0.03
0.09
hopper-med
1.20=+0.35
0.74+0.29
0.42+00.05
0.34
0.33
hopper-med-rep
1.27=0038
1.21=0.33
0.48+00.7
0.53
0.39
hopper-med-exp
0.73++0223
1.43=0.40
0.65+
0.38++0008
0.34
walker-med
0.55+0.16
0,45=0.09
0.30
0.26++0.05
0.18+0.02
walker-med-rep
1.16#0.20
1.13=0.34
0.38
0.33
0.25
walker-med-exp
0.64
0.48+00.1
0.29+0..4
0.21+00.04
0.19+0.03
ant-med
1.12::027
1.04=0.29
0.41
0.25
0.16:10.02
ant-med-rep
0.74+0.22
0.84+0031
0.38+0.12
0,25++0.06
0.14+0002
ant-med-exp
0.54++0.13
0.60=#.30
0.27+00066
0.21+0007
0.14+
Table 11: The mean absolute calibration enror on NeoRL benchmark. We bold the lowest mean.
Task
DROPO
DROID
OTED
Madoc-S
Madoc
HalfCheetah-L.
0.55+ 14
0.40
0.46+10.07
0.43+00.27
0.13
HalfCheetah-M
0.92+0.37
0.73+
0.34
0.24+0003
0.20=10.05
HalfCheetah-H H
0.50:10.17
0.73+
0.47+0.05
0.32+10.06
0.21=0005
Hopper-L
0.76:10.19
0.59
0.30+0005
0.22+10.66
0.24+0.05
Hopper-M
0.99::021
1.06=0.29
0.42+00.07
0.34
0.26=#0.04
Hopper-H
0.93++0.28
1.90
0.38+00.07
0.34
0.26
Walker2d-L
0.86+0.37
1.22
0.27
0.24
0.22
Walker2d-M
1.16+0.34
0.62+0022
0.26+0006
0.29++0.03
0.17+00.2
Walker2d-H
1.17+00.99
1.09
0.24
0.28
0.18+0003
of our reward model. Madoc minimizes the mean absolute calibration errors in almost all tasks, which also serves as a basis for its efficient domain transfer.
F.2 Ablation Studies
In order to further validate the effectiveness of the automatic grouping technique, in Fig. 6 we implement a variant of Madoc, which is not equipped with the automatic grouping technique, i.c.. there are N agents each responsible for calibrating a single parameter. We the same hyper- parameters, and train both policies on the HalfCheetah tasks of the NeoRL benchmark. We can find the policy performance Madoc significantly decreases without the use of the automatic grouping technique, underscoring the importance of this component.
Besides, we additionally consider independent learning methods and different value decomposition methods. Independent learning [41] treats each agent as an independent individual, optimizing each policy with shared rewards without considering the joint policy, has gained traction again due their surprising performance in some domains 82 Therefore, we have implemented the Madoc-ISA AC algorithm for domain calibration. VDN 44 is classic value decomposition method representing the global value function as a simple sum of individual value functions, and we denote the corresponding version as Madoc-VDN, We train these policies with the same hyper-parameters on the HalfCheetah tasks of the D4RL benchmark. The results, shown in Tab. 12] demonstrate that Madoc-IS. ISC and Madoc- VDN perform worse than Madoc on all three datasets. All physics parameters in the source domain are interrelated and cooperative. Consequently, the independent learning method overlooks policy changes of other calibration agents, which leads to non-stationary problems and decline in performance. The gap between Madoc-VV Madoc-VDN and Madoc is small, we speculate the reason is domain calibration is essentially a bandit RI problem, where there no state space (therefore, we do not compare with more complex value decomposition algorithms [45] either, as their implementation
22

---

HallCheetah-L
HalCCheetah-M
HaltChessah-H
10000
10000
10000
8000
Relumm
6000
6000
6000
2000
2000
2000
0.5M
1.0M
6.6M
0.2M
1.0M
Rollout
Rollout step
Rollout
with automatic grouping
without automatic grouping
Figure 6: The average return of rollout steps in the source domain with without the automatic grouping technique.
Table 12: Normalized average return of Madoc and its variants. The results are evaluated the target domain and we bold the highest mean.
hfctah- med
hfctah-med-rep
hfctaah-ee-eepp
Madoc
91.9 7.7
95.7 9.9
96.9 5.3
Madoc- VDN
89.9 6.9
92.71 7.1
88.2: 5.7
Madoc-ISAC
81.1=11.77
75.0
75.2+1
would be the same without the global state). The main purpose of value decomposition methods is to perform eredit assignment and both methods achieve this. This also reflects that algorithm can be integrated with any existing MARL value decomposition methods.
E.3 Stability of the Experimental Results
The performance results presented in Tab. and Tab. reveal that Madoc exhibits greater variance in certain task scenarios compared to offline RL algorithms. this subsection, we explain the large variance and have designed corresponding adjustments to improve upon this instability.
Madoc has achieved a trade-off between high mean and low variance return performance. On the one hand, Madoc requires online interaction with the source domain to search for the domain parameters that best match the offline dataset. Consequently, the random seed significantly influences exploration and exploitation, leading algorithms like CQL and MOREC learn on a fixed dataset conservative manner and do not need to explore. Therefore, they are less influenced by the random seed and have smaller variance. On the other hand, algorithms with lower variance, namely H2O, DR+BC, CQL, and MOREC, obtain conservative policies by penalizing the value functions policies against the behavior policies. As a result, their mean performances are also limited by the dataset. Our method has achieved trade-off between high mean and low variance, attaining optimal performance compared to baselines in most scenarios.
larger variance for Madoc. In contrast, pure offline
OOD actions or directly constraining the
Regarding the large variance problem of Madoc, we have made some improvements. Once domain calibration completed, we no longer use pure SAC to train the policy on source domain from scratch. Instead, we combine SAC with BC to impose appropriate constraints on the learned policy, referred to as Madoc+BC. The results are shown in the Tab. I3 We can observe that the mean performance of Madoc+BC decreased slightly but became more stable, confirming our approach.
E.4 Sensitivity of Hyper-parameters
By utilizing automatic grouping technique, Madoc clusters physical parameters into several groups; thus, we investigate the impact of the number of groups on algorithm performance in the hedttah-med-rep task. As illustrated Fig. a the best performance achieved when the number of groups is 6; however, the impact is not significant as long the value is within appropriate range.
23

---

Table 13: Normalized average return of Madoc and its variants. The results are evaluated in the target domain and we bold the highest mean.
hfctah-med
hfctahh-ed-reep
hlctahhmed-exp
Madoc
91.9:17.7
95.7:19.9
96,9:15.3
Madoc+BC
88.9:14.6
90.18
97.2:13.33
MOREC
73.9=3.0
74.1
72.0#3.11
Normallzed average return
80
Nornalized avernge
40-
40-
0.001
0.01
0.05
0.5
(a) Sensitivity of group number i
(b) Sensirivity of temperature coefficient at
Figure 7: Sensitivity of
Therefore, default setting 6 the number of groups for the three tasks HalfCheetah, Walker2d, Ant, and for the Hopper task, which has fewer domain parameters, we choose as the number of groups. Additionally, for the hyper-parameter that affects the entropy of the calibration policy during the domain calibration process, we also design experiments on this task to verify parameter sensitivity. Fig. ())) demonstrates that 0.05 the best choice, values that are too large or too small will both lead to reduced search efficiency, affecting the final policies' performance, We also set this
the same value for all experiments, and showcases stable results.
Sloennlizedd ever retur
70
Normalized average retur
TD-
D03ID
-0-
42
40-
30
30.
Lage
Exsy
Performance with different sized datasets
Performance under diffeerent initial domain parameter ranges
Figure 8: More results about the generalization ability of Madoc compared to the basclines under different conditions.
E.5 More Generalization Results
In Sec. 5.4 we have already verified the generalization capability of Madoc across different datasets and under different search spaces. Here, we list the comparative results with more baseline algorithms. As shown in Fig. ( the pure offline RI. algorithm CQL is inferior to Madoc across various datasets, highlighting the limitations of conservative algorithms; DROID employs an evolutionary algorithm for parameter optimization and achieve performance comparable Madoc in simple cases, but collapses in complex ones. Madoc, on the other hand, can stably handle different scenarios and achieve excellent performance, supporting its generalization stability.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state the main claims in the abstract and introduction, which accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the possible limitations in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
25

---

Justification:
All assumptions and proof' are clearly stated in Sec. and Sec.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We present the implementation details in Sec. and App. C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
26

---

Answer: [Yes]
Justification:
We provide the code at hutttpss///tthiii cOw.LuDAA..OAAMAAAAAAAAAAAA RL/Madoc
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli///uuui wwwiii/gguie///eeeeeeeecc CoodeSubeiiieeee ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We present the experimental setting in App. and App.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results and accompanied by error bars and confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
27

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide sufficient information in App- C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that NeurIPS Code of Ethics https: //t::://neuiippeeeeee http.//c/b...ttttccccsseeeeeeeee
didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the
Answer: [Yes]
Justification: I have read and followed the ethics guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
28

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: I cite the origin codebase in the last paragraph in App. C
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee www.rde.wthcee....c... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
29

---

Answer: [Yes]
Justification: We provide the code in the supplementary material, and document our method in a README.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
30


Reviews:
[{"review_id": "VEwDsPAW0v", "reviewer": "Reviewer_Cdsr", "summary": "The authors, introduce Madoc, a novel framework for domain calibration. By leveraging offline data from the target domain, it dynamically adjusts physics parameters, enabling direct policy deployment. To tackle the challenge posed by a large domain parameter space, the authors propose modeling domain calibration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. Experimental results demonstrate that Madoc surpasses existing techniques across most tasks in the D4RL and NeoRL benchmarks.", "questions": "- What is the computation cost of training your method compared to other Offline RL methods? Please add a discussion. \n\n- The reward model seems to be a crucial component of this method. How good is the reward model? Is there a way to quantify its accuracy? \n\n- Recently, in MARL community, independent learning methods have gained traction again due to their suprising performance in some domains [1] due to their robustness to large and complex tasks. How do you think independent SAC agents wll perform in this problem?  \n\n[1] de Witt et al, \"Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?\"", "limitations": "Please refer to the weaknesses section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses the critical research topic of domain calibration, which plays a pivotal role in enabling effective domain transfer of reinforcement learning (RL) methods. The authors present a well-written and organized paper, meticulously explaining their ideas and methodology. The motivation behind their approach is clearly articulated, and the experimental section thoughtfully addresses key questions. The experimental results are particularly compelling, demonstrating significant improvements over baselines on two standard benchmarks (D4RL and NeoRL). One of the paper’s novel contributions lies in its use of Cooperative Multi-Agent Reinforcement Learning (MARL) for adjusting source domain parameters. The authors delve into the specifics of their method, conducting ablation studies to highlight the impact of different components. Overall, I find this paper both interesting and potentially useful for the community.", "weaknesses": "*Major comments:*\n\n- Method Complexity and Comparisons with Offline RL: \n\nI noticed that the method appears more complex and computationally intensive compared to offline reinforcement learning (RL) methods like MOREC and CQL. This complexity might impact the fairness of the comparisons in Tables 1 and 2.\nAdding discussion on the trade-off between complexity and performance in specific scenarios or domains could improve the paper.\n\n- Surprising Results in Table 2: \n\nI found it interesting that MOREC performs significantly poorly in HalfCheetah-M and Walker2d-H, which differs from the original paper results. It’s possible that differences in experimental settings, hyperparameters, or implementation details contribute to this discrepancy. Accurate baseline implementation is crucial for fair comparisons.\n\n- Smallest Target Task Dataset Size: \n\nThe paper’s title suggests tackling domain transfer with handful of handful of offline data. The method seems to clearly perform well in small dataset regims. However, the smallest target task dataset still contains 5e4 samples. It’s reasonable to expect results with much smaller data sizes to better support the claim about domain transfer in sensitive contexts.\n\n*Minor comments:*\n\nIn Section 4.2, the first paragraph refers to multi-agent (MA) methods depicted by red dots. However, the legend in Figure 2 uses blue for MA (presumably short for Multi-agent).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Method Complexity and Comparisons with Offline RL.\n**Madoc adds more modules compared to CQL, but the model complexity and GPU memory cost remain acceptable.** \n\nWe conducted experiments on the hfctah-med-rep task of the D4RL benchmark to evaluate the model complexity of different methods. The results are shown in the table below, with all networks in the form of MLP. Our method achieves a balance between model complexity and performance.\n| Method | GPU Memory Cost and Hidden Layers                                                                                                       |\n| ------ | --------------------------------------------------------------------------------------------------------------------------------------- |\n| Madoc  | 398MB: 2 * [256, 256] for reward models, 2 * [256, 256] for VAE, 6 * [64, 64] for calibration agents, 2 * [256, 256] for running agents |\n| CQL    | 286MB: 2 * [256, 256] for running agents                                                                                                |\n| MOREC  | 1053MB: [128, 256, 128] for dynamics reward function, 40 * [200, 200, 200, 200] for dynamics models, 2 * [256, 256] for running agents  |\n### Q2: Surprising Results in Table 2.\n**The performance of MOREC is not as good as in the original paper because we used a smaller amount of offline data.** \nMOREC is sensitive to the quantity of offline data as it initially needs to train a generalizable dynamics reward function from offline data and accordingly learn dynamics models for generating transitions. When the quantity of offline data is insufficient, both the dynamics reward function and the dynamics models tend to underfit. As depicted in Fig. 4(a), when using the same large dataset as MOREC, the performance gap is small. However, as the amount of offline data decreases, the gap becomes increasingly larger. We will provide an explanation of it in the revised manuscript.\n### Q3: Smallest Target Task Dataset Size.\n**Madoc utilized a sufficiently small amount of offline data.** \n\nWe present the smallest target task dataset size of different methods below:\n| Method     | Size |\n| ---------- | ---- |\n| Madoc      | 5e4  |\n| SSR [1]    | 2e5  |\n| CQL, MOREC | 1e6  |\n\nSSR [1] proposes a data-efficient pipeline for offline RL with limited data and still used 2e5 training samples. This is because the continuous control tasks possess larger state and action spaces compared to discrete RL environments. Thus, the smallest target task dataset of Madoc, which contains only 5e4 samples, can be regarded as using a handful of offline data.\n### Q4: What is the computation cost of training your method compared to other Offline RL methods?\n**Madoc incurs higher computational costs compared to CQL, but this is acceptable given its performance improvement.** \n\nWe conducted experiments on the hfctah-med-rep task to evaluate the computational costs of different methods. The results are shown below:\n| Method | Total Time | Seconds per Epoch (1000 epochs)                                  |\n| ------ | ---------- | ---------------------------------------------------------------- |\n| Madoc  | 5 hours    | 1.8s for grouping (200 epochs), 14s for  calibration, 4s for SAC |\n| CQL    | 2 hours    | 7s for CQL                                                       |\n| MOREC  | 6 hours    | 5s for dynamics reward function, 16s for policy training         |\n\nThe table indicates that Madoc involves three training stages and consumes more computational cost compared to CQL. However, given the significant performance improvement, this additional computational cost is justifiable. We will also include this discussion in the revised appendix.\n### Q5: How good is the reward model?\n**The reward model can accurately reflect the dynamics gap under different parameters.** \n\nTo verify the role of the reward model, we designed an experiment to test whether it can act as a scoring function to evaluate the truthfulness of transitions. As shown **in the right half of the PDF of the \"global\" response**, we stored the checkpoint of the trained reward model and used it to test the reward results (the reward range should be [-4, 4]) it outputs under different dynamics. \nIn this simple test, we only changed two parameters: the gravity coefficient and body_mass_1 (the target values are -9.81 and 6.36, respectively). We sampled 256 transitions with the behavior policy under each parameter condition to calculate the mean reward. The results, shown in the heat map, indicate that when the domain parameters are closer to the target parameters, the output reward is higher. This suggests that it can accurately reflect the dynamics gap under different parameters.\n### Q6: Comparison with independent learning methods.\n**Independent SAC agents perform worse than Madoc agents in domain calibration.** \n\nIndependent SAC (ISAC) treats each agent as an independent individual, optimizing each policy with shared rewards without considering the joint policy. We conducted relevant experiments and the results are shown below:\n| Task           | Madoc    | ISAC      |\n| -------------- | -------- | --------- |\n| hfctah-med     | 91.9±7.7 | 81.1±14.7 |\n| hfctah-med-rep | 95.7±9.9 | 75.0±13.1 |\n| hfctah-med-exp | 96.9±5.3 | 75.2±16.1 |\n\nThe results indicate that ISAC performs worse than Madoc in domain calibration. We believe the main reason is that all physics parameters are interrelated and cooperative. The independent learning method ignores the policy changes of other calibration agents, leading to non-stationary problems [2]. IPPO [3] performs well on some SMAC tasks because these tasks do not have high requirements for cooperativeness.\n\nReference:\n\n[1] Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Data. NeurIPS, 2022.\n\n[2] Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning. arXiv, 2019.\n\n[3] Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? arXiv, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yAE4M9dlHu", "reviewer": "Reviewer_dRio", "summary": "This paper introduces Madoc, a domain transfer method that calibrates a source domain using small amount of offline data from the target domain via multi-agent reinforcement learning (MARL). More concretely, MARL is used to tune physics parameters governing dynamics in the source domain to more closely match those in the target domain. Empirically, Madoc enables transfer in D4RL and NeoRL benchmark tasks.", "questions": "1. I'm a bit confused by Figure 2. Could the authors provide more detail on what is being plotted here and why it is being plotted? In particular, what exactly are at the plotted metrics and what are they intuitively related?", "limitations": "Limitation are noted in the conclusion.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "2. The domain transfer problem is clear and the method is well-motivated.\n1. The empirical analysis consider many relevant baselines and considers appropriate benchmark tasks.", "weaknesses": "I lean to reject primarlily because the paper's empirical analysis does not seem to support the two core claims on the paper -- namely that Madoc reduce the difference in dynamics between source and target domains and outperforms existing methods.\n\n1. **Weak empirical results.** In most if not all tasks, the confidence regions for Madoc overlap with those for other baselines.\n2. **It unclear if Madoc is indeed reducing the dynamics gap between source and target domains, since this gap is not evaluated in experiments.** Additional experiments should be included showing that Madoc indeed reduces the dynamics gap. First instance, one could report the difference between source and target domain physics parameters and/or the KL divergence between source and target domains. As it stands, it looks like Madoc offers little to no improvement over baselines, which leads me to believe that Madoc might not be reducing the gap by much.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Weak empirical results.\n**Our method Madoc has achieved a trade-off between high mean and low variance in return performance.**\n- On the one hand, Madoc requires online interaction with the source domain to search for the domain parameters that best match the offline dataset. Consequently, the random seed significantly influences exploration and exploitation, leading to larger variance for Madoc. In contrast, pure offline algorithms like CQL and MOREC learn on a fixed dataset in a conservative manner and do not need to explore. Therefore, they are less influenced by the random seed and have smaller variance.\n- On the other hand, algorithms with lower variance, namely H2O, DR+BC, CQL, and MOREC, obtain conservative policies by penalizing the value functions on OOD actions or directly constraining the policies against the behavior policies. As a result, their mean performances are also limited by the dataset. Our method has achieved a trade-off between high mean and low variance, attaining optimal performance compared to baselines in most scenarios.\n  \nRegarding the large variance problem of Madoc, we have made some improvements. Once domain calibration is completed, we no longer use pure SAC to train the policy on the source domain from scratch. Instead, we combine SAC with BC to impose appropriate constraints on the learned policy, referred to as Madoc+online_bc. The results are shown in the table below. We can observe that the mean performance of Madoc+online_bc decreased slightly but became more stable, confirming our approach.\n  \n| Task           | MOREC    | Madoc    | Madoc+online_bc |\n| -------------- | -------- | -------- | --------------- |\n| hfctah-med     | 73.9±3.0 | 91.9±7.7 | 88.9±4.6        |\n| hfctah-med-rep | 74.1±2.8 | 95.7±9.9 | 90.1±4.8        |\n| hfctah-med-exp | 72.0±3.1 | 96.9±5.3 | 97.2±3.3        |\n\nThe concern you mentioned is of crucial significance, and we will incorporate its discussion in the experimental section of the revised manuscript.\n\n### Q2: It's unclear if Madoc is indeed reducing the dynamics gap between the source and target domains.\n**Our method Madoc indeed reduces the dynamics gap between the source and target domains.**\nSome experimental results are as follows:\n\n| Task           | DROPO     | OTED      | Madoc     |\n| -------------- | --------- | --------- | --------- |\n| hfctah-med     | 0.78±0.18 | 0.28±0.33 | 0.06±0.01 |\n| hfctah-med-rep | 0.64±0.15 | 0.61±0.11 | 0.12±0.02 |\n| hfctah-med-exp | 0.66±0.14 | 0.33±0.06 | 0.09±0.02 |\n\nAs shown in the table, we used \"mean absolute calibration error\" (i.e., the absolute difference between the physics parameters of the source and target domains) to measure the dynamics gaps of different algorithms. It can be observed that Madoc minimized the mean absolute calibration errors in all three tasks. The complete results are presented in Appendix E.1. We sincerely apologize for the confusion caused by the lack of appendix guidance in the manuscript and will add this guidance in the revised version.\n\n### Q3: Could the authors provide more details about Fig. 2?\n**Fig. 2 demonstrates that, compared to the single-agent method, the multi-agent method can more effectively evaluate the accuracy of each physics parameter output by the calibration policy.**\nThe calibration agent consists of a calibration critic that evaluates the accuracy of the physics parameters and a calibration actor (policy) that adjusts the output physics parameters. The plotted metrics are the values output by the calibration critic for a specific physics parameter (i.e., the gravity coefficient) under different parameter conditions. When the parameters output by the calibration actor are closer to the target parameters (indicating a smaller absolute calibration error), the evaluation value output by a \"good\" critic should be higher. The results in Fig. 2 show that our multi-agent method can train such a good critic, whereas the single-agent method fails to correctly evaluate the accuracy of the physics parameters due to the lack of reasonable reward allocation and the large search space.\nWe have corrected the legend of Fig. 2 **in the left half of the PDF of the \"global\" response** and will provide more detailed explanations about it in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zu5A9718Ta", "reviewer": "Reviewer_EfDo", "summary": "The paper addresses the challenge of performance degradation when RL policies trained in one domain are deployed in another with different dynamics. The proposed solution, Madoc (Multi-agent domain calibration), uses a small amount of offline data from the target domain to adjust the source domain's physics parameters. This approach improves upon existing domain calibration techniques by employing cooperative Multi-Agent Reinforcement Learning (MARL) to handle a large parameter space more efficiently.", "questions": "- Regarding the objective at the start, eq. (1), why do we need to match action distribution as well? Usually if the transition is varying, the objective is to match the state distributions. If the transitions are different, the actions to make the similar state distributions will be different, and it seems natural to not match the action distributions.\n- Why do we need the multi-agent method? in section 4.2. it is motivated in a way that shared critic struggles to optimize policy in a huge action space. Does it mean that it is beneficial to use multi-agent algorithm for any RL environments for better optimization of policy? As far as I know, multi-agent algorithms are harder to optimize to the optimal policy (often converge to sub-optimal), and I am curious why this case is different.\n- Why can't we apply algorithms like \"imitation learning from observation\" for this problem setting after concatenating the domain parameter to the action space? How is this paper different from such algorithms?", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Strong empirical performance", "weaknesses": "- The paper is so hard to follow.\n  - In 4.1, we start with KL minimization objective. It is difficult to understand how it is optimized, as the paper lacks sufficient explanation and some derivations are incorrect. Eq (4) seem to be completely different formula compared to eq (3). regarding q_\\phi, in eq (4), the only relevant term is the third term, which makes no sense.\n  - Section 4.2. refers to \"calibaration critic\" and some terms that are not defined.\n  - especially the latter part of 4.2, explaining about the parameter grouping, is mostly not understandable. what does \"the identity i of each parameter\" mean? How can $i$ itself determine s' given s,a,\\zeta^i without knowing \\zeta^-i? \n\nminor comments:\n- in figure 2 legend, blue dots are said to be MA, whereas the text on left says blue dots are single-agent method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The objective in Sec. 4.1.\n**Eq. 4 is obtained through reasonable approximations.**\n\nThe explanations for each derivation step are as follows:\n- Eq. 2 is derived from Eq. 1 based on the definitions of trajectory distribution and KL divergence, transforming the objective into the expectation under the parameter distribution and trajectory distribution of the source domain. \n- Eq. 3 further decomposes Eq. 2. Since $\\log \\frac{q_{\\phi}(\\xi)}{p(\\xi)}$ is independent of the trajectory distribution $d_{\\pi,\\mathcal{M}_{\\xi}}$, the expectation can be decomposed into two terms. \n- Eq. 4 approximates Eq. 3. As the parameter distribution and trajectory distribution used to calculate the expectation are difficult to compute, we use **Monte Carlo sampling** on the source domain to approximate the expected results. To enhance sampling efficiency, we store the samples in the replay buffer, **converting the trajectory-based objectives into transition-based objectives**, following the off-policy RL paradigm.\n\nWe will add the detailed explanations in the revised manuscript.\n### Q2: Some terms are not defined in Sec. 4.2.\nThe calibration critic and the calibration policy (actor) together constitute a calibration agent. The calibration critic is responsible for evaluating the accuracy of the parameters output by the calibration actor. The absolute calibration error represents the absolute difference between the parameters output by the calibration actor and the target parameters.\n### Q3: The latter part of Sec. 4.2 is not understandable.\n\"The identity $i$ of each parameter\" represents a specific identification symbol for each parameter. Specifically, in the experiments, these identities refer to a set of one-hot encodings used to distinguish different parameters. \nBesides, some key contents were missed: the identity can determine $s'$  given $s, a, \\xi$ by **keeping the other parameters $\\xi^{-i}$ fixed**, similar to the approach in COMA [1]. By only changing $\\xi_i$, the effect of this parameter on the dynamics can be reflected, thereby facilitating parameter grouping . \nWe apologize for the confusion and will add explanations in the revised manuscript for better understanding.\n### Q4: Why do we need to match action distribution?\n**We need to match the action distribution to make the dynamics of the source domain closer to that of the target domain.**\n\nEq. 1 is used to match the trajectory distribution (state distribution and action distribution). The underlying motivation is to minimize the transition dynamics gap. If only the state distribution is matched, multiple combined solutions of the policy and the transition function can exist. Therefore, to obtain the target transition function, we need to match the action distribution as well. To verify our idea, we conducted relevant experiments: for Eq. 1, we no longer matched the action distribution, denoted as Madoc_st. The experimental results, shown below, indicate performance degradation in all three scenarios.\n| Task           | Madoc    | Madoc_st  |\n| -------------- | -------- | --------- |\n| hfctah-med     | 91.9±7.7 | 78.2±8.9  |\n| hfctah-med-rep | 95.7±9.9 | 85.2±12.4 |\n| hfctah-med-exp | 96.9±5.3 | 88.1±10.1 |\n### Q5: Why do we need the multi-agent method?\n**We use the multi-agent method for better optimization efficiency.**\n\nThe search space of domain parameters grows exponentially with the number of parameters, hindering the optimization of both single-agent methods and previous multi-agent methods. Fortunately, with the advent of linear value factorization algorithms like VDN [2], researchers have improved scalability by decomposing the joint action space theoretically and practically [3], resulting in a much lower action space for each agent. Consequently, Madoc has higher optimization efficiency and can calibrate parameters more accurately.\nCertainly, the multi-agent method is not applicable to all RL environments. When there is only one agent in the environment with a low action dimension, a single-agent RL method might be more straightforward, providing satisfactory policy optimization without the added complexity.\n### Q6: How about methods like \"imitation learning from observation\"?\n**Madoc is different from and superior to these methods.**\n\nImitation from observation [4] methods (denoted as IfO) can be formalized as learning a discriminator $D(s, s')$ and a policy $\\pi(a,\\xi|s)$ simultaneously, while Madoc involves training a discriminator $D(s, a, s')$, a calibration policy $q(\\xi)$ and a running policy $\\pi(a|s)$. There are mainly two differences: \n- Madoc decouples tuning the domain parameters and the policy, making it more accurate and efficient in reducing the dynamic gap. \n- IfO requires matching the state distribution of the expert trajectory, which imposes additional quality requirements on the dataset [5]. However, Madoc only requires datasets of arbitrary quality to match the transition dynamics. \n\nWe also tested the performance of IfO. The first table below reports the mean return during the test stage.\n| Task           | Madoc    | IfO       |\n| -------------- | -------- | --------- |\n| hfctah-med     | 91.9±7.7 | 43.2±28.9 |\n| hfctah-med-rep | 95.7±9.9 | 55.2±19.4 |\n| hfctah-med-exp | 96.9±5.3 | 42.1±22.1 |\n\nThe second table below compares the mean absolute calibration error.\n| Task           | Madoc     | IfO       |\n| -------------- | --------- | --------- |\n| hfctah-med     | 0.06±0.01 | 0.98±0.12 |\n| hfctah-med-rep | 0.12±0.02 | 1.02±0.30 |\n| hfctah-med-exp | 0.09±0.02 | 0.79±0.08 |\n\nReference:\n\n[1] Counterfactual Multi-Agent Policy Gradients. AAAI, 2018.\n\n[2] Value-Decomposition Networks for Cooperative Multi-Agent Learning Based on Team Reward. AAMAS, 2018.\n\n[3] Towards Understanding Cooperative Multi-Agent Q-Learning with Value Factorization. NeurIPS, 2021.\n\n[4] Recent Advances in Imitation Learning from Observation. IJCAI, 2019.\n\n[5] Robust Learning from Observation with Model Misspecification. AAMAS, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate valuable comments from all reviewers. We have carefully clarified the ambiguous parts and supplemented our work with additional experiments to address the raised issues. Our revisions can be briefly summarized as follows:

- Method.
    - We have further elaborated on the motivation and derivation process of the optimization objective, supplemented the explanations of relevant definitions, and improved the representation to make it easier to understand.
    - The supplementary PDF presents the revised version of Fig. 2.  
- Experiments.
    - We have provided a more detailed explanation of the empirical results and supplemented them with relevant experiments to demonstrate the superiority of our method.
    - We have added the comparison results with new baseline algorithms in some scenarios, highlighting the differences and demonstrating the superiority of our method.
    - We have presented the comparison of model complexity and computational cost with offline algorithms, reflecting the feasibility of our method.  
  
The supplementary PDF consists of two parts:
- The left half presents the revised version of Fig. 2 from the manuscript, with corrected errors in the legend. Thank you for pointing out this crucial typo, which has helped us improve our work.
- The right half contains the verification of the reward model, demonstrating that it can reflect the dynamics gap under different physics parameters. The complete description can be found in Q5 for Reviewer Cdsr.

We hope that our responses address all the questions and concerns. If we have missed anything, please let us know. We are always willing to resolve any further issues and look forward to the ensuing insightful discussions.

Meta-review: This paper frames domain calibration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem, and finds significant performance gains in smaller data regimes than tackled in previous works.  Reviewers originally raised a number of concerns regarding the cost of the method, clarity, the motivation for such an approach, and the strength compared to existing alternatives in the literature.  Ultimately, the authors were able to address these points satisfactorily in the discussion, and I’m happy to recommend acceptance on the basis of the strong empirical results and novelty of the framework.
Decision: Accept (poster)