Multi-Agent Domain Calibration with a Handful of Offline Data

Authors: tao  jiang
lei  yuan
lihe  li
cong  guan
zongzhang  zhang
yang  yu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 14:25:48

Abstract: The shift in dynamics results in significant performance degradation of policies trained in the source domain when deployed in a different target domain, posing a challenge for the practical application of reinforcement learning (RL) in real-world scenarios. Domain transfer methods aim to bridge this dynamics gap through techniques such as domain adaptation or domain calibration. While domain adaptation involves refining the policy through extensive interactions in the target domain, it may not be feasible for sensitive fields like healthcare and autonomous driving. On the other hand, offline domain calibration utilizes only static data from the target domain to adjust the physics parameters of the source domain (e.g., a simulator) to align with the target dynamics, enabling the direct deployment of the trained policy without sacrificing performance, which emerges as the most promising for policy deployment. However, existing techniques primarily rely on evolution algorithms for calibration, resulting in low sample efficiency.
To tackle this issue, we propose a novel framework Madoc (\textbf{M}ulti-\textbf{a}gent \textbf{do}main \textbf{c}alibration). Firstly, we formulate a bandit RL objective to match the target trajectory distribution by learning a couple of classifiers. We then address the challenge of a large domain parameter space by modeling domain calibration as a cooperative multi-agent reinforcement learning (MARL) problem. Specifically, we utilize a Variational Autoencoder (VAE) to automatically cluster physics parameters with similar effects on the dynamics, grouping them into distinct agents. These grouped agents train calibration policies coordinately to adjust multiple parameters using MARL.
Our empirical evaluation on 21 offline locomotion tasks in D4RL and NeoRL benchmarks showcases the superior performance of our method compared to strong existing offline model-based RL, offline domain calibration, and hybrid offline-and-online RL baselines.

Main paper: 
 introduction: 
 Reinforcement learning (RL) has gained significant traction in various fields [1] , such as sequential recommendation systems [2] and robotic control [3] , demonstrating tremendous potential in realworld applications. However, the inherent trial-and-error nature of RL limits its application, especially in safety-critical areas such as healthcare [4] and autonomous driving [5] , as extensive interactions with the target environment can entail prohibitive costs and pose substantial safety risks. To address this problem, a range of studies have proposed collecting training samples from a surrogate source domain (e.g., simulation environment) to learn policies, which are then deployed to the downstream target domain [6, 7] . Nonetheless, due to complex system dynamics and the characteristics of open environments, a high-fidelity simulator may not always be available [8] , leading to severe dynamics shifts between the source and target domains [9] . Consequently, policies trained optimally in the source domain may fail catastrophically in the target domain. To bridge the dynamics gap, various kinds of solutions have been developed recently. Domain randomization [10] methods, for instance, randomly sample the physics parameters of the source domain and train policies across multiple simulated environments to approximate the target domain. However, as the target domain is often unknown and set in an open environment [8] , these methods can also suffer from unpredictable policy degradation, which hinders further development.Integrating the source domain with some data from the target domain offers a promising solution to the mentioned problem [11] . A class of methods, known as domain calibration, attempts to use data from the target domain as feedback to calibrate the easily obtained source domain and then transfer the policy directly to the target domain. This approach shows enormous potential when the parameters are adjusted accurately enough. Some typical methods automatically tune the physics parameters by minimizing the transition discrepancy between the source and target domains [12, 13] or by maximizing the expected return in the target domain [14] . While these methods can successfully transfer learned policies in robotics [15] , they still require interaction feedback from the target domain during training. Instead, DROID [16] and DROPO [17] introduce an offline setting for domain calibration, where the physics parameters are adjusted using offline demonstrations pre-collected in the target domain, showing potential for real-world applications.Nevertheless, in complex real-world scenarios, numerous physics parameters may require calibration. The above-mentioned methods primarily employ evolutionary algorithms [16, 18] or samplingbased methods [19, 20] for black-box optimization, often results in low sample efficiency [21, 22] . Recently, some algorithms have attempted to mitigate this issue by learning sampling strategies [22] or leveraging causal discovery [23] to eliminate parameters that have little impact on the environment. Despite the effectiveness of these methods, a significant challenge remains in handling complex scenarios where all physics parameters critically influence the dynamics, and different parameters may have varying, or even opposite, impacts on these dynamics [24] . A method for efficiently addressing the interrelations among different parameters is urgently needed.From the perspective of whole-domain calibration, each physics parameter contributes to different aspects of the calibration process. This can be modeled as a typical multi-agent system (MAS) problem [25] , where each agent adjusts a group of domain parameters, and all agents cooperate to reduce the domain gap. This problem can be addressed using cooperative multi-agent reinforcement learning (MARL) [26] , leading to the development of the Madoc (Multi-agent domain calibration) framework. Specifically, we first formulate domain calibration as a target trajectory distribution matching problem and derive a bandit RL optimization objective by introducing a couple of classifiers to act as the reward model. We then formulate the problem into the MAS where multiple agents calibrate different parameters to reduce the dynamics gap between the source and target domains. Concretely, we propose an automatic grouping technique to cluster physics parameters based on their impacts on the dynamics. We then employ popular value decomposition methods in MARL to train cooperative calibration policies to adjust domain parameters. We conduct experiments on popular locomotion tasks to showcase Madoc's superior performance against baselines and highlight the contributions of its core design components. The source code is available at https: //github.com/LAMDA-RL/Madoc. 
related work: 
 Domain transfer in RL. Transferring RL policies learned from imperfect source domains to the target domain is a crucial step in the practical use of RL algorithms [11] . However, the trained policy often suffers from severe performance deterioration when directly deployed into the target domain due to the distribution shift between different domains with varying transition dynamics [27] . Previous works have addressed this problem with three common strategies: domain randomization (DR), domain adaptation (DA), and system identification (SI). DR attempts to train a generalizable policy that works well across a variety of randomized simulated dynamics [28, 9] . While the motivation is simple and often effective, these methods require manually determining which parameters to randomize and may result in underfitting or failing policies due to hand-tuning parameter ranges. DA involves using a huge amount of data from source domains to improve policy performance on a different target domain [29, 30, 31] . However, these efforts are constrained by the quality and quantity of target domain data and often still require interaction with the target domain. Another line of work, SI, uses measured data to build mathematical models of dynamical systems [32] . These methods rely on numerous interactions with the target domain to study how to learn a model of the system dynamics [33, 34] , which results in learning a biased policy with fewer interactions. Most recent works calibrate the parameters of the biased source domain to bridge the domain gap [19, 13, 15] , also known as domain calibration, and try to improve efficiency by learning a parameter sampling strategy [22] or leveraging causal discovery [23] . However, these methods still require interacting with the target domain, posing potential safety hazards during the training process. To mitigate this problem, DROID [16] and DROPO [17] use offline datasets to adjust the source domain parameters via evolutionary algorithms [18] with different optimization objectives, perform poorly when the number of domain parameters is large [35] . Unlike the above methods, we propose to adjust the source domain with a handful of offline data, enabling the domain parameters to match the target trajectory distribution with high sample efficiency.Cooperative multi-agent RL. Many real-world problems are inherently large-scale and complex, making it inefficient and impractical to model them as single-agent systems. Instead, they are more suitably addressed as multi-agent systems (MASs) [25] . Multi-agent reinforcement learning (MARL) provides frameworks for modeling and solving such challenges [26] . In scenarios where agents within MAS share common objectives, these problems are categorized under cooperative MARL, which has demonstrated significant advancements in domains like power management [36] , path planning [37] , and dynamic algorithm configuration [38] . One of the primary challenges in cooperative MARL is the scalability issue [39, 40, 41] , exacerbated by the exponential growth of the search space with the number of agents, complicating policy exploration and learning. Various approaches have been proposed to enhance agent coordination recently. These include policy-based methods such as MADDPG [42] and MAPPO [43] , value-based techniques like VDN [44] and QMIX [45] , and innovations like the transformer architecture [46] . Among these methods, value-based approaches have demonstrated promising results in diverse and complex settings [47, 48] . VDN leverages additivity to factorize global value functions, QMIX further enforces monotonicity in global value functions, and DOP [49] introduces value function decomposition within multi-agent actor-critic frameworks. These methods exhibit remarkable coordination capabilities across various tasks such as SMAC, Hanabi, and GRF [26] . In this paper, our method formulates domain calibration as a cooperative MARL problem, improving efficiency and fidelity. 
background: 
 Reinforcement Learning can be generally modeled as a Markov decision process (MDP) [50] , formulated as a tuple M := (S, A, T, r, γ, ρ 0 ), where S and A denote the state and action spaces, T (s ′ |s, a) ∈ [0, 1] and r(s, a) represent the transition and reward functions, γ ∈ [0, 1) implies the discount factor, and ρ 0 (s) is the initial state distribution. The agent running in the environment perceives the state s t ∈ S at time step t, performs an action a t ∈ A based on a learnable policy π(a|s) ∈ [0, 1], then the environment receives the action, transits to a new state s t+1 , and rewards the agent according to the transition function T (s t+1 |s t , a t ) and reward function r(s t , a t ) at next time step. The above process is continuously iterated until termination, we can record the whole trajectory of length H + 1 as τ = (s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , • • • , s H , a H , r H ) and the trajectory distribution over agent's policy and the environment can be defined as d π,M (τ ) = ρ 0 (s 0 ) H t=0 T (s t+1 |s t , a t )π(a t |s t ). The objective of RL algorithms is to learn a policy π(a|s) which maximizes the expected discounted return across the distribution of trajectories, i.e., J (M, π)= E τ ∼d π,M (•) R(τ ) with R(τ ) = H t=0 γ t r(s t , a t ).Domain Calibration aims to adjust a manipulable source domain to close the domain gap between it and the target domain. Both the target and source domains can be modeled as MDPs, and the only difference between them is the transition functions which are determined by the physics dynamics parameter vector ξ ∈ Ξ ⊂ R N (e.g., friction, mass, damping). Here, Ξ denotes the physics parameter space, and N represents the dimension of the physics parameters. Each parameter ξ i is bounded on a closed interval, which can only be inferred roughly based on experience and expert knowledge, i.e.,ξ i ∈ [ξ i low , ξ i high ].We assume the unknown physics parameters of the target domain ξ * are included in the parameter space Ξ if the physical modeling is reasonable, i.e., ξ * ∈ Ξ, as we can set sufficient wide parameter ranges. We now denote the transition function conditioned on domain parameters as T ξ = T (s ′ |s, a, ξ) and the corresponding MDP as M ξ := (S, A, T ξ , r, γ, ρ 0 ). However, the ). The subscripts "src" and "tar" are used to distinguish between the source and target domains, respectively. After learning the grouping scheme, we use the red arrow to represent the process of domain calibration by MARL value decomposition methods. manipulable source domain is typically non-differentiable, we can only calibrate the distribution of the source domain parameters ξ ∼ q ϕ (•), and the optimal policy learned under this distribution is marked as π * (q ϕ ) = arg max π E ξ∼q ϕ (•) J (M ξ , π). The objective of domain calibration is to learn the source domain parameters that maximize the expected discounted return under the target domain: max q ϕ J (M ξ * , π * (q ϕ )). 
method: 
 In this section, we propose the Madoc (Multi-agent domain calibration) framework for leveraging a modest amount of offline data from the target domain to calibrate the biased source domain, thus facilitating optimal policy transfer. The overall workflow of the Madoc framework is shown in Fig. 1 . We first deduce a bandit RL objective to adjust the domain parameters in Sec. 4.1, improving the synthetic data sampled from the source domain to align with the target trajectory distribution. We further model it as a cooperative multi-agent reinforcement learning problem in Sec. 4.2 and use an automatic grouping technique to improve the efficiency and fidelity of domain calibration. Finally, a practical algorithm under the Madoc framework is presented in Sec. 4.3.Domain calibration is the process of tuning the parameter distribution of a mismatched source domain to better align with the target domain, which can be realized by comparing the divergence between target domain interactions and simulated synthetic rollouts based on the same policy [12, 13] . However, since interacting with the target domain may not be feasible for sensitive fields, we propose an alternative approach to minimize the trajectory discrepancy between the two domains by employing a handful of offline target domain data.Formally speaking, the static offline datasetD = {τ 1 , τ 2 , • • • , τ k } contains k trajectories where τ i = (s i 0 , a i 0 , r i 0 , s i 1 , a i 1 , r i 1 , • • • , s i H , a i H , r i H ), which are collected previously by an unknown behavior policy µ from the target domain, i.e., τ i ∼ d µ,M ξ * (•). By introducing a prior normal parameter distribution p(ξ) to foster better generalization [51] , we intend to learn a sample policy π and calibrate the domain parameters to match the target trajectory distribution:EQUATIONwhere the Kullback-Leibler (KL) divergence can be further derived as: E ξ∼q ϕ (•) τ ∼d π,M ξ (•) log H t=0 π(EQUATIONwhere Eq. 4 is an approximation of Eq. 3 as the parameter distribution and trajectory distribution used to calculate the expectation are difficult to compute. Consequently, we use Monte Carlo sampling on the source domain to approximate the expected results. To enhance sampling efficiency, we sample a domain parameter ξ ∼ q ϕ (•), generate the trajectories τ ∼ d π,M ξ (•), and store the rollouts (s, a, s ′ , ξ) in the replay buffer B. By doing so, we are able to convert the trajectory-based objective into a transition-based one, following the classic off-policy RL paradigm.It is delighted to discover that the objective in Eq. 4 can be clearly divided into three terms: the first term, i.e., min π E (s,a)∼B log π(a|s) µ(a|s) ≈ min π D KL (π(a|s)||µ(a|s)), attempts to minimize the KL divergence between π(a|s) and µ(a|s), we can consider it as a variant of behavior cloning; the second term is formulated as max q ϕ E (s,a,s ′ ,ξ)∼B log T (s ′ |s,a,ξ * )T (s ′ |s,a,ξ) , which can be seen as a bandit RL objective for policy q ϕ (ξ) to maximize reward R q (ξ) = log T (s ′ |s,a,ξ * )T (s ′ |s,a,ξ) ; and the last term is regarded as a policy regularizer added on q ϕ (ξ) to prevent it from collapsing. It is worth noting that the policy q ϕ (ξ) here is not the one running (sampling) on the source domain, but the one outputting physics parameter vector ξ as an action to adjust the source domain. To prevent confusion, in the following paper, the policy running on the source domain, i.e., π(a|s), is referred to as the "running policy", while the one adjusting the domain parameters, i.e., q ϕ (ξ), is referred to as the "calibration policy". Additionally, we similarly define the "calibration critic", which is responsible for evaluating the accuracy of the parameters output by the calibration actor. The calibration critic and the calibration policy (actor) together constitute a calibration agent.The key challenge lies in how to estimate the stochastic reward log T (s ′ |s,a,ξ * )T (s ′ |s,a,ξ) given the offline data and simulated rollouts. According to Bayes' rule, we can transform the transition probability [29] as:T (s ′ |s, a, ξ) = P (ξ|s, a, s ′ )P (s, a, s ′ ) P (ξ)P (s, a|ξ) = P (ξ|s, a, s ′ )P (s, a, s ′ ) P (ξ|s, a)P (s, a) ,and the reward can be derived as: R q (ξ) = log P (ξ * |s, a, s ′ ) -log P (ξ * |s, a) -log P (ξ|s, a, s ′ ) + log P (ξ|s, a), where ξ * and ξ stand for the target and source domains respectively. Hence, we can train a couple of binary classifiers D ψsas (•|s, a, s ′ ) and D ψsa (•|s, a) to discriminate whether state-action-state and state-action pairs come from the offline dataset (referred to as the binary variable target) or synthetic samples (referred to as the binary variable source). These two discriminators form a reward model with certain generalization ability [52] , and the corresponding cross-entropy losses are written as:EQUATIONAs the complexity of the source domain grows, characterized by an expanding number of physics parameters, the calibration policy learned by single-agent reinforcement learning often struggles to consistently reduce the domain gap. To address this challenge, we employ multi-agent reinforcement learning (MARL) algorithms to effectively reduce the search space for a single calibration agent, thereby enhancing the efficiency and fidelity of domain calibration. The Pearson correlation between the critic value of the gravity coefficient and the absolute calibration error. Each dot represents a sampled action, which is then fed into the corresponding critic to compute the critic value. When the parameters output by the calibration actor are closer to the target parameters (indicating a smaller absolute calibration error), the evaluation value output by a "good" critic should be higher.We conduct experiments on the HalfCheetah [53] environment and train to calibrate the gravity coefficient along with other physics parameters. The preliminary results are shown in Fig. 2 , we capture snapshots for both approaches at the same training step to investigate Pearson correlation [54] between the critic value of the gravity coefficient and the absolute calibration error. Here, the absolute calibration error represents the absolute difference between the parameters output by the calibration actor and the target parameters. The singleagent method, represented by the blue dots, utilizes just a shared calibration critic for parameter adjusting, facing challenges in assessing a specific action within a huge action space.In contrast, the multi-agent method, depicted by the red dots, employs value decomposition algorithms [44] to narrow the action space of each individual calibration policy. This decomposition leads to more accurate evaluations of domain parameters, thereby reducing calibration errors and improving policy transfer.To utilize the cooperative MARL for domain calibration, we now formally define N agents (since the source domain has a total of N physics parameters) to perform domain calibration coordinately where each of them q i ϕ (ξ i ) attempts to adjust the single parameter ξ i , therefore the joint calibration policy can be decomposed as q ϕ (ξ) = N i=1 q i ϕ (ξ i ). Utilizing any off-the-shelf value decomposition algorithm VD, we employ the single global rewards[R 1 q (ξ i ), • • • , R N q (ξ N )] = VD(R q (ξ))to guide individual calibration policy updates:EQUATIONNonetheless, when the number of the domain parameters N is large, employing N agents for domain calibration leads to low exploration and optimization efficiency [55] . To mitigate this problem, clustering physics parameters with similar effects on the transition dynamics, e.g., the mass of a symmetrical robot's left and right feet, into one calibration agent is an advisable choice. Hence we introduce an automatic grouping technique by adopting a Variational Autoencoder (VAE) [56] . By adjusting a specific parameter ξ i while keeping all other parameters ξ -i fixed [57] , we can assume the identity i of each parameter (i.e., the one-hot encoding) to be representative of the transition tr = (s, a, s ′ , ξ i ), the Evidence Lower BOund (ELBO) of the transition is then derived as:EQUATIONwhere f e , f d stand for the encoder and decoder, z is the latent variable, and p(z) represents the corresponding prior distribution. We can deduce the reconstruction term as:EQUATIONwhere c is a constant as s, a, and ξ i do not depend on the latent variable z. We parameterize the encoder f e and the decoder f d with Ψ, making them f Ψe and f Ψ d , and optimize the VAE model with the following loss:EQUATIONBefore domain calibration, we first minimize the VAE loss (Eq. 10). Then we apply the k-means clustering method [58] to the means generated by the encoder f Ψe (z|i) for all i ∈ N , in order to group the domain parameters [39] . The resulting n (1 ≤ n ≤ N ) grouping scheme is recorded as [ξ g1 , • • • , ξ gn ], each group forms one agent equipped with a calibration actor q gi ϕ (ξ gi ) and a calibration critic v gi Φ (ξ gi ), following the multi-agent actor-critic framework.Based on the above analysis, we now present a practical algorithm under the Madoc framework, the pseudo-code is shown in App. A. We apply DOP [49] , a popular multi-agent policy gradient algorithm as the value decomposition method. Concretely, there are n agents for calibrating the source domain after automatic grouping, we factor the joint critic as a weighted summation of individual critics:EQUATIONwhere k i ≥ 0 and b are denoted as learnable weights and biases. The individual critics are learned by back-propagating gradients from global Temporal Difference updates:EQUATION)Given individual critics, we use SAC [59] to update the stochastic actors in an off-policy manner:EQUATIONwhere α and λ control the relative importance of the entropy and regularization term respectively. Besides, the prior domain parameter distribution p gi (ξ gi ) is set to be an exponentially moving average of the current calibration policy q gi ϕ (ξ gi ), which has been shown to stabilize training like target network. Finally, we parameterize the policy running in the source domain π with θ and enable it to clone the behavior policy on offline data during domain calibration. Once domain calibration is complete, we train the policy π θ from scratch on the source domain using SAC, and directly deploy it to the target domain. 
experiments: 
 In this section, we present the empirical evaluations of our proposed Madoc framework. We first describe the experiment environments and related baselines in Sec. 5.1, and then conduct a series of experiments to answer the following questions: (1) How is the comprehensive performance of Madoc against multiple baselines (Sec. 5.2)? (2) How do core components of Madoc contribute to the overall performance (Sec. 5.3)? (3) How is the generalization capability of Madoc across datasets of varying sizes and source domains with different initial ranges (Sec. 5.4)?In our experiments, we evaluate Madoc on classic continuous control tasks from the MuJoCo [53] engine and choose two offline benchmarks to serve as offline datasets collected in the target domain. On the popular D4RL benchmark [60] , we choose four locomotion tasks (HalfCheetah, Hopper, Walker2d, Ant), each with three types of datasets (medium, medium-replay, medium-expert), to evaluate different algorithms' performance when faced with datasets of varying quality. Considering more challenging scenarios, three environments (HalfCheetah, Hopper, Walker2d) along with three levels of datasets (low, medium, high) from NeoRL benchmark [61] are also selected. The main difference between the two benchmarks lies in that the static datasets in the NeoRL benchmark occupy more narrow distributions. During the training process, we are given imperfect source domains, and only aware of the initial range of specific physics parameters (gravity, body_mass, dof_damping). Each environment has different parameter dimensions, initial ranges, and ground truth values, refer to App. D for detailed information.Madoc utilizes static offline datasets to calibrate biased source domains, and we choose several baseline algorithms with identical or similar settings for comparison. DROPO [17] and DROID [16] use fixed offline datasets from the target domain to optimize the distribution bounds with different objectives via evolution algorithms like CMA-ES [18] . OTED [51] models the parameter optimization process as a bandit RL problem, which is similar to our method, but the objective is different and cannot cope with large parameter space. H2O [62] and DR+BC are two hybrid offline-and-online algorithms where the former penalizes the Q-function learning on simulated state-action pairs with large dynamics gaps, and the latter directly combines uniform domain randomization with behavior cloning. CQL [63] is the classic pure offline RL algorithm using value regularization, MOREC [52] is the state-of-the-art offline model-based RL algorithm by learning a generalizable dynamics reward function, representing the upper-bound performance by solely utilizing offline data. Madoc-S is an ablation algorithm of our proposed Madoc, which only uses a single calibration agent for parameter tuning, without applying the multi-agent decomposition and automatic grouping technique. Madoc and all the above baselines use SAC [59] as the online algorithm after domain calibration for its practicality and convenience on MuJoCo tasks.All the numerical results in our experiments are reported in terms of mean and 95% confidence interval computed over 6 random seeds, and we give more details about the model implementation of our method in App. C.We first evaluate the overall performance of our method and baselines on the two benchmarks and use the normalized average scores introduced by D4RL [60] for intuitive comparison. In realistic scenarios, both the size of offline datasets and the usage of the source domain can be restricted, thus we constrain each algorithm to access a static dataset with up to 2 × 10 5 transitions and execute only 1 × 10 6 simulated samples for domain calibration (when used). Due to these constraints, the performance of the baseline algorithms has experienced varying degrees of degradation. As shown in Tab. 1 and Tab. 2, we observe that our methods outperform prior methods on both benchmarks: two pure offline algorithms, CQL and MOREC, suffer from severe performance degradation in some scenarios, especially when encountering narrow NeoRL datasets; two hybrid offline-and-online algorithms are limited by the dataset quality and domain fidelity, and cannot achieve the expected performance; DROPO and DROID employ evolutionary algorithms to optimize domain parameters, encountering challenges in transferring learned policies when the parameter space is large; the comparison with OTED demonstrates the rationality of our optimization objective described in Sec. 4.1, while the comparison with Madoc-S further illustrates the effectiveness of modeling domain calibration as a multi-agent system formulated in Sec. 4.2. To verify that Madoc indeed reduces the dynamics gap between the source and target domains, we report the mean absolute calibration error for different algorithms in App. E.1. Furthermore, given the significant variance in domain calibration algorithms, we provide a more detailed discussion on the stability of the experimental results in App. E.3.To investigate the impact of the design components of Madoc, we first design experiments on the Hopper environment to visualize the automatic grouping technique, as shown in Fig. 3 (a). The Hopper is a one-legged robot simulation with four distinct body sections: the torso, thigh, leg, and foot. These components are connected by three joints, which serve as the articulation points between each pair of bodies. We need to calibrate the gravity coefficient, the mass of each body, and the damping coefficients at each joint. After projecting the embedding space onto a three-dimensional space, we can discover that the robot's physics parameters are divided in an orderly manner from top to bottom, while the gravity coefficient forms a separate group on its own. The result aligns perfectly with our expectations, as parameters that have similar impacts on dynamics can be adjusted using one calibration policy. Additionally, in Fig. 3 (b), we conduct ablation studies on the Ant environment of the D4RL benchmark, to verify the effectiveness of modeling domain calibration as an MARL problem. When the dimension of the domain parameters is 1 or 2, there is no significant performance gap between Madoc and Madoc-S; however, as the source domain becomes more complex, the performance of both declines, but the drop is faster when only one agent is employed for parameter calibration. This experimental result favorably supports that modeling as MARL can enhance the efficiency of domain calibration in large parameter spaces.Madoc leverages static offline datasets to adjust the domain parameters, driving our curiosity toward its generalization capacity under varying dataset sizes and initial parameter ranges of the source domain. Here we choose all the above-mentioned tasks on the NeoRL benchmark and calculate the averaged normalized returns for comparison. As illustrated in Fig. 4 (a), the algorithms access datasets of different magnitudes, 5 × 10 4 (small), 2 × 10 5 (medium), and 1 × 10 6 (large), to reflect a spectrum of data availability. The results reveal that the effectiveness of both the hybrid offline-and-online H2O algorithm and the purely offline MOREC algorithm rises with the expansion of the dataset size. This discovery suggests that their dependency on the size of offline data for improved performance. On the contrary, our method maintains stable and excellent performance, unfazed by the dataset size. Besides, for source domains with various initial parameter ranges, categorized as easy, medium, and hard (see App. D for more details), Madoc exhibits remarkable performances across all levels, particularly excelling in "hard" cases with the largest parameter search space, as shown in Fig. 4(b) . This underscores our algorithm's effectiveness in coping with challenging scenarios, affirming its robustness and adaptability in diverse conditions. 
conclusion and discussion: 
 In this paper, we introduce Madoc, a framework for closing the dynamics gap by calibrating the source domain with a handful of offline data via multi-agent reinforcement learning. Concretely, the target domain data serve as a guide for target transition dynamics, which is leveraged to train classifiers generating rewards and derive a bandit RL objective for domain calibration, To improve calibration efficiency with a large number of parameters, we further model it as a cooperative MARL problem and propose to group parameters with similar effects on dynamics. Experiments on popular control tasks demonstrate that our method can calibrate the source domain with sufficient accuracy, allowing the optimal trained policy to be transferred to the target domain without severe performance deterioration. One possible constraint of our method is that, when dealing with high-dimensional vision tasks [64] , using a handful of offline data may not guarantee the accuracy and generalizability of the reward model. This challenge could be mitigated by deploying more expressively powerful tools like diffusion models [65] , which is left for future work. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: I cite the origin codebase in the last paragraph in App. C. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 
 
 Appendix: The pseudo-code of Madoc is presented in Alg. 1, we utilize SAC [59] and DOP [49] as our backbone algorithms for domain calibration. From lines 1 to 9, we group the domain parameters by learning a VAE and employing k-means; lines 13 to 16 aim at training discriminators as the reward model and regularizing the running policy; the calibration actors and critics are updated from lines 17 to 19; finally, we train the running policy from scratch and deploy it to the target domain.Input: A pre-collected dataset D, an imperfect manipulable source domain M Initialize: Policy π θ , joint calibration actor q ϕ and critic v Φ , a couple of binary discriminators D ψ , VAE f Ψ , prior distributions p(ξ) and p(z), replay buffer B = ∅ for simulated rollouts, step for parameter grouping T group , step for domain calibration T calibration , learning rate η 1:for step t = 1, • • • , T group do 2:Behavior cloning: 3:π θ ← BC(π θ , D) 4:Collect simulated data:5: B ← B ROLLOUT(π θ , q ϕ , M) 6:Update VAE: v Φ ← v Φ -η∇ Φ L Φ according to Eq. 12 19: q ϕ ← q ϕ -η∇ ϕ L ϕ according to Eq. 13 20: end for 21: Train policy from scratch: 22: π θ ← SAC(π θ , q ϕ , M) 23: Output: Policy π θ B Extended Related Work Sample efficient RL. A significant drawback of current reinforcement learning methods is their poor sample efficiency, leading to extensive environmental interactions [66] . This results in prohibitive costs in real-world applications [67] and hinders policy learning even in complex digital environments, like the full StarCraft game [68] . Multiple factors may restrict the sample efficiency, while several aspects can alleviate these limitations. Different optimization methods possess varying capabilities for exploring and exploiting data. Mainstream algorithms in reinforcement learning commonly rely on the gradient of the objective or surrogate objectives [69] . Still, there are also sample-based methods, a.k.a. derivative-free optimization, that offer their unique advantages. Typical algorithms like evolutionary algorithms [70] and Bayesian optimization [71] have been applied for reinforcement learning [72] , showcasing better performance on some tasks. Nonetheless, these methods also encounter certain drawbacks, including slow convergence, difficulties in scaling, sensitivity to noise, and a lack of theoretical guarantee. From another perspective, model-based algorithms can be much more efficient as planning in the model is free of real-world samples in ideals [73] . Nevertheless, in high-dimensional environments, learning an accurate transition model using supervised learning is challenging. Employing manually constructed environments, i.e., simulators, is more efficient and practical, especially in the field of robotics [74] . However, it also encounters new problems due to the sim2real dynamics gap, and simulators are generally non-differentiable, unlike neural models. Besides the aspects discussed above, the ability to transfer knowledge is also crucial for improving the sampling efficiency. Rather than starting from scratch for each task, humans continuously learn and build upon experiences from a variety of tasks. Many methods have proposed various types of knowledge transfer, such as policy transfer [75] , representation transfer [76] , and skill transfer [77] . However, these methods work effectively only in specific cases. A general approach to transfer reinforcement learning is yet to be developed.7: f Ψ ← f Ψ -η∇ Ψ L Ψ according to Eq.Offline RL. In the offline RL setting, the agent no longer has the ability to interact with the environment [78] . Instead, the learning algorithm is access to a static offline dataset, collected previously by an unknown behavior policy from the target domain. The main obstacle to this data-driven learning paradigm is the distribution shift problem due to the discrepancy between the learned and behavior policies, leading to severe extrapolation error. Previous works tackle this problem by constraining the learned policies against the behavior policy [79] or penalizing the value function on out-of-distribution (OOD) actions [63] , both of which require large enough datasets. Besides, offline model-based RL algorithms learn a dynamics model from offline data to enhance the efficiency of offline RL [80] . Benefiting from generating the synthetic data by the learned dynamics, model-based algorithms improve the coverage of the dataset. However, this depends on accurate dynamics models and still requires sufficient offline data. It is worth noting that in this work we use offline datasets to calibrate domain parameters, thereby avoiding the selection of OOD actions, and alleviating the requirements for dataset quality and quantity.Discriminators. We train two discriminator networks D ψsas (•|s, a, s ′ ) and D ψsa (•|s, a) to classify whether the rollouts comes from the offline dataset or the source domain in our method. We follow the implementation in DARC [29] , propagating gradients back through both discriminators as follows:D ψsa (•|s, a) = SoftMax(f sa (s, a)), D ψsas (•|s, a, s ′ ) = SoftMax(f sas (s, a, s ′ ) + f sa (s, a)),where f sa (s, a) and f sas (s, a, s ′ ) represent the outputs of the two classifier networks, SoftMax(x i ) = exp xix j ∈X exp xj , X = {target, source}. Both networks contain two hidden layers, while dropout layers and ReLU activations are used between layers. Before the final SoftMax layer, we employ the Tanh activation function to map the outputs into probabilities.Grouping VAE. We train both the encoder and decoder with two hidden layers, both dropout layers and ReLU activation functions are used between layers. We employ MSE loss for state reconstruction and eliminate KL divergence, as our goal is to encode the agent identity rather than generating states. For each domain parameter, we encode its identity into a 16-dimensional latent space. Then the latent variable is concatenated with the current observation and action, and the combined data is fed into the decoder to predict the observation for the next time step. The latent variables of all parameters are projected to three dimensions using Principal Component Analysis (PCA) [81] , after which they are grouped using the k-means algorithm [58] . To reduce randomness in the group results, we repeatedly extract schemes from the last phase of group training and determine the final grouping scheme by choosing the one that occurs most frequently.Calibration policy. We use DOP [49] as the multi-agent decomposition method because it is a popular off-policy multi-agent policy gradient method. As claimed in the paper, individual critics v gi Φ are learned by backpropagating gradients from global TD updates, guiding the updates of individual calibration actors by a similar objective as in SAC [59] :L ϕ = E ξ gi ∼q gi ϕ (•) α log q gi ϕ (ξ gi ) -v gi Φ (ξ gi ) + λD KL q gi ϕ (ξ gi )||p gi (ξ gi) . Here, we do not employ the double-Q function and the automatic tuning scheme of α. Instead we update λ t = min(10, exp( t 2×10 5 )) and prior p gi (ξ gi ) as a target network of q gi (ξ gi ) with smoothing coefficient 1 × 10 -3 to stabilize training. The prior parameter p gi (ξ gi ) is distribution is initialized as a normal distribution N (0, 1) to enhance exploration at the early stages of training.We scale the output of the calibration actors by using a Gaussian with Tanh squashing, e.g., we rescale the output of gravity to be within [-30, 0] in our preliminary experiments. To further enhance the exploration of domain parameters, we sample a series of domain parameters every 100 rollouts in the source domain. After calibration, we set the physics parameters with the means outputted by actors, and train SAC from scratch with default hyper-parameters.Most experiments were conducted on a server outfitted with a 13th Gen Intel(R) Core(TM) i9-13900K CPU, 2 NVIDIA RTX A5000 GPUs, and 125GB of RAM, running Ubuntu 22.04. We also conduct a comparison between Madoc and various offline RL algorithms with regard to method complexities and computational costs on the hfctah-med-rep task of the D4RL benchmark, the results are presented in Tab. 3 and Tab. 4. Compared to the traditional pure offline RL algorithm CQL, Madoc incorporates additional modules that lead to increased GPU memory cost. Furthermore, it encompasses three training stages, which consequently require more computational cost. Nevertheless, given the significant performance improvement, these extra expenditures are deemed justifiable. We list the default hyper-parameter settings for Madoc in Tab. 5. We adopt the popular Gym-MuJoCo tasks [53] as our benchmarks used in the experiments. To investigate the performance of our method and baselines, we conduct experiments on four locomotion tasks shown in Fig. 5 , including HalfCheetah, Hopper, Walker2d, Ant.HalfCheetah is a 2D robotic simulation featuring 9 links and 8 joints, including two paws, designed to move forward rapidly by applying torque on 6 specific joints. The main objective of this robot centers on maximizing forward progression to earn positive rewards while minimizing backward movement to avoid penalties.Hopper is designed to enhance the complexity with more state and control variables than traditional control settings, using a two-dimensional, one-legged hopper composed of a torso, thigh, leg, and foot. The overall aim is to propel the hopper forward by exerting torque on the three joints that link its four body sections.Walker2d is a two-dimensional figure with two legs, including a torso, two thighs, two legs, and two feet, focusing on coordinated movement to advance. The objective is to utilize torque on six joints to synchronize the movement of all six body parts toward the desired direction.Ant is a 3D model with a freely rotating torso and four two-linked legs, designed for coordinated forward movement. The aim is to maneuver the ant by applying torques on eight hinges to effectively control the movement of its nine parts toward the desired direction.The number of groups n for each task is 6, 4, 6, and 6, respectively. We verify in extended experiments (shown in Fig. 7(a) ) that the performance results do not differ significantly as long as it is within a reasonable range. We report the initial ranges and ground truth value of each physics parameter in Tab. 6, Tab. 7, Tab. 8, and Tab. 9. Here we introduce the baselines used in our experiments, including offline domain calibration, hybrid offline-and-online RL, and pure offline RL algorithms.DROPO [17] adapts a distribution of dynamics parameters to match an offline dataset by employing a probabilistic distance measure, aimed at directly maximizing the likelihood of replicating real-world data within a simulation. Consequently, the simulator can be envisioned as a stochastic forward model, where the inherent randomness is attributed to variations in the scene's physical parameters.DROID [16] harnesses human demonstrations to synchronize the simulator's trajectories with those observed in the real world, rather than relying on guesswork or exhaustive adjustments to establish the domain randomization (DR) range. This process helps in finding the most suitable range of parameters for the simulator, which can be formulated as a statistical model. Subsequently, this model can be sampled to inform the training process of RL agents.OTED [51] is designed to autonomously learn a set of simulator parameters that align with a given offline dataset. Using the calibrated simulator, it proceeds to train a Reinforcement Learning (RL) agent using conventional online methods. An objective is formulated for the tuning of simulator parameters, aiming to minimize a divergence metric between the state-action distribution generated by the simulator and the provided target offline dataset.H2O [62] presents a novel policy evaluation framework that is aware of dynamics, adaptively imposing penalties on Q-function training for simulated state-action pairs that exhibit significant dynamics discrepancies. At the same time, it permits learning from a predetermined dataset originating from the real world without direct interactions with it.CQL [63] enhances the conventional Bellman error objective by adding an uncomplicated Q-value regularization term, which is easy to apply to most current deep Q-learning and actor-critic models.The objective of CQL is to overcome existing limitations by training a conservative Q-function, ensuring that the policy's expected value, as estimated by this Q-function, is a conservative estimate of its actual value, thus avoiding selecting OOD actions.MOREC [52] acquires a dynamics reward function that can be generalized from offline data. This reward function is then utilized as a transition filter within any offline Model-Based Reinforcement Learning (MBRL) approach. During the transition generation process, the dynamics model produces a set of possible transitions, from which the one with the highest dynamics reward value is chosen for selection and used for policy update.We report the corresponding mean absolute calibration error of experiments in Sec. 5.2. As shown in Tab. 10 and Tab. 11, there are five methods performing domain transfer by tuning simulator parameters. It is apparent that the evolutionary algorithm-based methods, DROPO and DROID, result in huge calibration errors, which suggests their limited effectiveness within the realm of high-dimensional parameter spaces. Additionally, Madoc-S outperforms OTED, highlighting the more rational design of our reward model. Madoc minimizes the mean absolute calibration errors in almost all tasks, which also serves as a basis for its efficient domain transfer.In order to further validate the effectiveness of the automatic grouping technique, as in Fig. 6 , we implement a variant of Madoc, which is not equipped with the automatic grouping technique, i.e., there are N agents each responsible for calibrating a single parameter. We set the same hyperparameters, and train both policies on the HalfCheetah tasks of the NeoRL benchmark. We can find the policy performance of Madoc significantly decreases without the use of the automatic grouping technique, underscoring the importance of this component.Besides, we additionally consider independent learning methods and different value decomposition methods. Independent learning [41] treats each agent as an independent individual, optimizing each policy with shared rewards without considering the joint policy, has gained traction again due to their surprising performance in some domains [82] . Therefore, we have implemented the Madoc-ISAC algorithm for domain calibration. VDN [44] is a classic value decomposition method representing the global value function as a simple sum of individual value functions, and we denote the corresponding version as Madoc-VDN. We train these policies with the same hyper-parameters on the HalfCheetah tasks of the D4RL benchmark. The results, shown in Tab. 12, demonstrate that Madoc-ISAC and Madoc-VDN perform worse than Madoc on all three datasets. All physics parameters in the source domain are interrelated and cooperative. Consequently, the independent learning method overlooks the policy changes of other calibration agents, which leads to non-stationary problems and a decline in performance. The gap between Madoc-VDN and Madoc is small, we speculate the reason is domain calibration is essentially a bandit RL problem, where there is no state space (therefore, we do not compare with more complex value decomposition algorithms [45] either, as their implementation with automatic grouping without automatic grouping would be the same without the global state). The main purpose of value decomposition methods is to perform credit assignment and both methods can achieve this. This also reflects that our algorithm can be integrated with any existing MARL value decomposition methods.The performance results presented in Tab. 1 and Tab. 2 reveal that Madoc exhibits greater variance in certain task scenarios compared to offline RL algorithms. In this subsection, we explain the large variance and have designed corresponding adjustments to improve upon this instability.Madoc has achieved a trade-off between high mean and low variance in return performance. On the one hand, Madoc requires online interaction with the source domain to search for the domain parameters that best match the offline dataset. Consequently, the random seed significantly influences exploration and exploitation, leading to a larger variance for Madoc. In contrast, pure offline algorithms like CQL and MOREC learn on a fixed dataset in a conservative manner and do not need to explore. Therefore, they are less influenced by the random seed and have smaller variance. On the other hand, algorithms with lower variance, namely H2O, DR+BC, CQL, and MOREC, obtain conservative policies by penalizing the value functions on OOD actions or directly constraining the policies against the behavior policies. As a result, their mean performances are also limited by the dataset. Our method has achieved a trade-off between high mean and low variance, attaining optimal performance compared to baselines in most scenarios.Regarding the large variance problem of Madoc, we have made some improvements. Once domain calibration is completed, we no longer use pure SAC to train the policy on the source domain from scratch. Instead, we combine SAC with BC to impose appropriate constraints on the learned policy, referred to as Madoc+BC. The results are shown in the Tab. 13. We can observe that the mean performance of Madoc+BC decreased slightly but became more stable, confirming our approach.By utilizing the automatic grouping technique, Madoc clusters physical parameters into several groups; thus, we investigate the impact of the number of groups n on algorithm performance in the hfctah-med-rep task. As illustrated in Fig. 7 (a), the best performance is achieved when the number of groups is 6; however, the impact is not significant as long as the value is within an appropriate range. Therefore, we default to setting 6 as the number of groups for the three tasks HalfCheetah, Walker2d, Ant, and for the Hopper task, which has fewer domain parameters, we choose 4 as the number of groups. Additionally, for the α hyper-parameter that affects the entropy of the calibration policy during the domain calibration process, we also design experiments on this task to verify parameter sensitivity. Fig. 7 (b) demonstrates that α = 0.05 is the best choice, values that are too large or too small will both lead to reduced search efficiency, affecting the final policies' performance. We also set this hyper-parameter to the same value for all experiments, and it showcases stable results. As shown in Fig. 8 , the pure offline RL algorithm CQL is inferior to Madoc across various datasets, highlighting the limitations of conservative algorithms; DROID employs an evolutionary algorithm for parameter optimization and can achieve performance comparable to Madoc in simple cases, but collapses in complex ones. Madoc, on the other hand, can stably handle different scenarios and achieve excellent performance, supporting its generalization stability.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification: I have read and followed the ethics guidelines. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper frames domain calibration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem, and finds significant performance gains in smaller data regimes than tackled in previous works.  Reviewers originally raised a number of concerns regarding the cost of the method, clarity, the motivation for such an approach, and the strength compared to existing alternatives in the literature.  Ultimately, the authors were able to address these points satisfactorily in the discussion, and I’m happy to recommend acceptance on the basis of the strong empirical results and novelty of the framework.

Reviews: review: paper_summary: paper_summary: The authors, introduce Madoc, a novel framework for domain calibration. By leveraging offline data from the target domain, it dynamically adjusts physics parameters, enabling direct policy deployment. To tackle the challenge posed by a large domain parameter space, the authors propose modeling domain calibration as a cooperative Multi-Agent Reinforcement Learning (MARL) problem. Experimental results demonstrate that Madoc surpasses existing techniques across most tasks in the D4RL and NeoRL benchmarks.
strength_weakness: strength_weakness: *Major comments:*

- Method Complexity and Comparisons with Offline RL: 

I noticed that the method appears more complex and computationally intensive compared to offline reinforcement learning (RL) methods like MOREC and CQL. This complexity might impact the fairness of the comparisons in Tables 1 and 2.
Adding discussion on the trade-off between complexity and performance in specific scenarios or domains could improve the paper.

- Surprising Results in Table 2: 

I found it interesting that MOREC performs significantly poorly in HalfCheetah-M and Walker2d-H, which differs from the original paper results. It’s possible that differences in experimental settings, hyperparameters, or implementation details contribute to this discrepancy. Accurate baseline implementation is crucial for fair comparisons.

- Smallest Target Task Dataset Size: 

The paper’s title suggests tackling domain transfer with handful of handful of offline data. The method seems to clearly perform well in small dataset regims. However, the smallest target task dataset still contains 5e4 samples. It’s reasonable to expect results with much smaller data sizes to better support the claim about domain transfer in sensitive contexts.

*Minor comments:*

In Section 4.2, the first paragraph refers to multi-agent (MA) methods depicted by red dots. However, the legend in Figure 2 uses blue for MA (presumably short for Multi-agent).
questions: questions: - What is the computation cost of training your method compared to other Offline RL methods? Please add a discussion. 

- The reward model seems to be a crucial component of this method. How good is the reward model? Is there a way to quantify its accuracy? 

- Recently, in MARL community, independent learning methods have gained traction again due to their suprising performance in some domains [1] due to their robustness to large and complex tasks. How do you think independent SAC agents wll perform in this problem?  

[1] de Witt et al, "Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge?"
limitations: limitations: Please refer to the weaknesses section.
paper_summary: paper_summary: This paper introduces Madoc, a domain transfer method that calibrates a source domain using small amount of offline data from the target domain via multi-agent reinforcement learning (MARL). More concretely, MARL is used to tune physics parameters governing dynamics in the source domain to more closely match those in the target domain. Empirically, Madoc enables transfer in D4RL and NeoRL benchmark tasks.
strength_weakness: strength_weakness: I lean to reject primarlily because the paper's empirical analysis does not seem to support the two core claims on the paper -- namely that Madoc reduce the difference in dynamics between source and target domains and outperforms existing methods.

1. **Weak empirical results.** In most if not all tasks, the confidence regions for Madoc overlap with those for other baselines.
2. **It unclear if Madoc is indeed reducing the dynamics gap between source and target domains, since this gap is not evaluated in experiments.** Additional experiments should be included showing that Madoc indeed reduces the dynamics gap. First instance, one could report the difference between source and target domain physics parameters and/or the KL divergence between source and target domains. As it stands, it looks like Madoc offers little to no improvement over baselines, which leads me to believe that Madoc might not be reducing the gap by much.
questions: questions: 1. I'm a bit confused by Figure 2. Could the authors provide more detail on what is being plotted here and why it is being plotted? In particular, what exactly are at the plotted metrics and what are they intuitively related?
limitations: limitations: Limitation are noted in the conclusion.
paper_summary: paper_summary: The paper addresses the challenge of performance degradation when RL policies trained in one domain are deployed in another with different dynamics. The proposed solution, Madoc (Multi-agent domain calibration), uses a small amount of offline data from the target domain to adjust the source domain's physics parameters. This approach improves upon existing domain calibration techniques by employing cooperative Multi-Agent Reinforcement Learning (MARL) to handle a large parameter space more efficiently.
strength_weakness: strength_weakness: - The paper is so hard to follow.
  - In 4.1, we start with KL minimization objective. It is difficult to understand how it is optimized, as the paper lacks sufficient explanation and some derivations are incorrect. Eq (4) seem to be completely different formula compared to eq (3). regarding q_\phi, in eq (4), the only relevant term is the third term, which makes no sense.
  - Section 4.2. refers to "calibaration critic" and some terms that are not defined.
  - especially the latter part of 4.2, explaining about the parameter grouping, is mostly not understandable. what does "the identity i of each parameter" mean? How can $i$ itself determine s' given s,a,\zeta^i without knowing \zeta^-i? 

minor comments:
- in figure 2 legend, blue dots are said to be MA, whereas the text on left says blue dots are single-agent method.
questions: questions: - Regarding the objective at the start, eq. (1), why do we need to match action distribution as well? Usually if the transition is varying, the objective is to match the state distributions. If the transitions are different, the actions to make the similar state distributions will be different, and it seems natural to not match the action distributions.
- Why do we need the multi-agent method? in section 4.2. it is motivated in a way that shared critic struggles to optimize policy in a huge action space. Does it mean that it is beneficial to use multi-agent algorithm for any RL environments for better optimization of policy? As far as I know, multi-agent algorithms are harder to optimize to the optimal policy (often converge to sub-optimal), and I am curious why this case is different.
- Why can't we apply algorithms like "imitation learning from observation" for this problem setting after concatenating the domain parameter to the action space? How is this paper different from such algorithms?
limitations: limitations: The authors adequately addressed the limitations.

Comments: title: RE multi-agent approach
About author responses
Should have another review in by tomorrow
comment: Thank you once again for your thorough review and engaging discussion. We would like to explain why accurate calibration is essential for achieving high performance from the following two perspectives:

- **Performance Improvements with Accurate Calibration**: When the domain parameter space is large, Madoc significantly reduces the mean absolute calibration error to the lowest level on both benchmarks, leading to performance improvements of at least 28% and 23% over other domain calibration methods (DROPO, DROID, OTED, Madoc-S), respectively. This substantial boost in performance is attributed to accurate calibration, as all methods use the SAC algorithm after calibration is completed.

- **Explaining the "Slim Gains"**: We observed that DROPO and DROID, despite their extremely poor parameter calibration, still manage to achieve mediocre performance. This can be explained by a "Floor Effect" [1], where, after reaching a certain threshold of poor calibration, further increases in calibration error do not result in significant performance declines. To validate this, we designed a toy task using the HalfCheetah environment, where only the gravity parameter requires calibration, with -9.81 as the target value. We trained policies on source domains with gravity coefficients of -10, -11, -12, -13, and -14, and then deployed them to the target domain. The performance changes are shown in the table below (tested with 4 seeds due to time constraints). 
  
| Gravity | Return    | $\Delta$ |
| ------- | --------- | -------- |
| -10     | 93.5±8.9  | 6.5      |
| -11     | 68.2±17.6 | 25.3     |
| -12     | 57.4±25.2 | 10.8     |
| -13     | 47.1±17.6 | 10.3     |
| -14     | 44.3±10.5 | 2.8      |

In the table, $\Delta$ represents the change in performance between successive gravity coefficients. Mathematically, for a given gravity coefficient $g_i$ and its corresponding return $R(g_i)$, $\Delta$ is defined as:

$
\Delta = |R(g_{i-1}) - R(g_i)|
$

where $g_{i-1}$ and $g_i$ are consecutive gravity coefficients, with $g_{i-1}$ being closer to the target value of -9.81 than $g_i$. This metric illustrates how sensitive the performance is to changes in the gravity coefficient. We observed that as the gravity coefficient deviates further from the target value—from -10 to -13—the performance change $\Delta$ is substantial. However, as the gravity coefficient decreases further to -14, the performance change becomes less pronounced. This helps explain why "the gains are slim."

**Reference:**

[1] Floor and ceiling effects in the OHS: an analysis of the NHS PROMs data set. BMJ open, 2015.
Regarding returns achieved by Madoc vs baselines: Experiments show that Madoc is reducing the calibration gap, though I maintain that Madoc does not offer a significant performance gain in many tasks, and will keep my (updated) score. Given that Madoc is calibrates well, it would be worth better understanding why the gains are slim in the chosen tasks -- perhaps there are benchmarks that would better highlight the benefits of Madoc. Is there a simple toy task the authors could create where accurate calibration is necessary to achieve high performance? Such an example would be instructive.
Thank you for your thoughtful feedback. 

We appreciate your suggestion about the vertical bar and will certainly consider it for the final revision. If you have any additional comments or concerns regarding the performance (return) of Madoc versus other baselines, we'd be happy to discuss them.
Dear AC,

My concerns are resolved by the authors, especially from the point that the proposed method only requires particularly small amount of data, and change in the equations. - and I raised the score.

Thanks!
Thanks for addressing my concerns. I changed my mind to raise the score.
Thank you once again for your thorough review and enthusiastic discussion.
Regarding your first concern, we have decided to revise Eq. (4) from 
$$\approx-\mathbb{E}\_{(s,a)\sim\mathcal{B}}\left[\log\frac{\mu(a|s)}{\pi(a|s)}\right] - \mathbb{E}\_{(s,a,s',\xi)\sim\mathcal{B}}\log\left[\frac{T(s'|s,a,\xi^*)}{T(s'|s,a,\xi)}\right]+D\_{\mathrm{KL}}(q\_{\phi}(\xi)||p(\xi)),$$
to 
$$\approx-\mathbb{E}_{a\sim\pi(\cdot|s)}\left[\log\frac{\mu(a|s)}{\pi(a|s)}\right]-\mathbb{E}\_{\substack{\xi\sim q\_\phi(\cdot) \\\\ (s,a,s')\sim d\_{\pi,\mathcal{M}\_{\xi}}(\cdot)}}\log\left[\frac{T(s'|s,a,\xi^*)}{T(s'|s,a,\xi)}\right]+D\_{\mathrm{KL}}(q\_{\phi}(\xi)||p(\xi)),$$

where both the second and third terms optimize the parameter distribution $q_\phi$. While the underlying approach remains the same, the revised expression is now clearer. Our initial intention was to emphasize the off-policy training paradigm by introducing the replay buffer $\mathcal{B}$ in the equation, but this led to some confusion. We will update this section in the revised version and add an explanation to address your concern.

Furthermore, addressing the concern about the necessity of a multi-agent approach, it's important to emphasize that the motivation behind using multi-agent methods is to mitigate optimization challenges posed by the large number of parameters. By employing the widely used value decomposition approach, the multi-agent method reduces the search space for each individual agent while preserving cooperation between different parameters. Additionally, we have chosen MARL for the following reasons:

- With the advancement of deep learning and the development of techniques such as value decomposition in MARL [1, 2], research in multi-agent reinforcement learning (MARL) has made significant strides, overcoming numerous challenges and achieving notable progress across various fields. Consequently, the optimization challenges in multi-agent algorithms have been substantially alleviated in recent years.

- Transformers and Diffusion models have recently shown exceptional results across various domains of deep learning. In reinforcement learning, approaches like TT [3], Diffuser [4], and DD [5] have also achieved impressive outcomes using these models. However, their success heavily relies on large datasets—TT, Diffuser, and DD required 1e6, 1e6, and 2.5e6 transitions, respectively, for training on continuous control tasks, while our method uses only 5e4 transitions. Consequently, their performance may not be as strong in our small-sample setting. Nonetheless, in future work, we plan to explore integrating Transformers and Diffusion models into our framework to enhance the expressive power of the reward model, particularly in scenarios involving high-dimensional image and text inputs.

- Multi-agent methods can solve problems that single-agent approaches cannot. For example, MA-DAC [6] models dynamic algorithm configuration problems as multi-agent systems and addresses them using cooperative MARL algorithms, thereby improving optimization efficiency. Similarly, MA2ML [7] effectively tackles optimization learning challenges in the connection of modules in automated machines through MARL. The study in [8] introduces a generic game-theoretic credit assignment framework to enable decentralized execution in continuous control. MARLYC [9] presents a novel method called MARL yaw control, which optimizes the yaw of each turbine, ultimately enhancing the total power generation of the farm. Additionally, the work in [10] models image data augmentation as a multi-agent problem, offering a more fine-grained automatic data augmentation approach by dividing an image into multiple grids and determining a jointly optimal enhancement policy. These successful applications highlight the effectiveness of modeling problems as multi-agent systems.

Thank you for your valuable feedback. We will continue to refine our framework to enhance domain transfer, guided by the reviewers' suggestions, and further advance the practical application of MARL.

Reference:

[1] Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 2022.

[2] A survey of progress on cooperative multi-agent reinforcement learning in open environment. arXiv, 2023.

[3] Offline reinforcement learning as one big sequence modeling problem. NeurIPS, 2021.

[4] Planning with diffusion for flexible behavior synthesis. ICML, 2022.

[5] Is conditional generative modeling all you need for decision making? ICLR, 2023. 

[6] Multi-agent dynamic algorithm configuration. NeurIPS, 2022.

[7] Multi-agent automated machine learning. CVPR, 2023.

[8] Multiagent model-based credit assignment for continuous control. AAMAS, 2022.

[9] Marlyc: Multi-agent reinforcement learning yaw control. Renewable Energy, 2023.

[10] Local patch autoaugment with multi-agent collaboration. IEEE Transactions on Multimedia, 2023.
Thank you for the response!

* Ah, I see, the paper *does* compare the source/target calibration gap in Appendix E. I must've missed this; I don't think there's a direct reference to Appendix E in the main body. I agree, Madoc is indeed achieving a lower calibration gap. I'm raising my score now that this has been clarified. As a side note, I suggest adding a vertical bar separating Madoc-S and Madoc from the other baselines. When I was comparing numbers, I found myself accidentally comparing Madoc with Madoc-S rather than the baseline methods.
* Fig 2 is now clear to me. If the discussion in your rebuttal is incorporated into the main paper, readers like myself should have a much easier time understanding the punchline here. I think my confusion primarily arose because the legend was incorrectly labeled in the original submission. 
* I'll make another comment about the performance (return) of Madoc vs other baselines later today once I have some time to carefully look over the results again. I'll respond as soon as I can to ensure we can discuss if needed!
Thanks for the detailed response. The responses addressed all of my concerns and integrating them to the paper could improve the paper. I still keep my score and support acceptance as the methodology of the paper is both novel and well communicated in my opinion.
Thanks for weighing in!

Can you clarify your comment that you are "not convinced that multi-agent approach is mandatory"?  
* Do you view this as a (e.g. central) claim of the work?
* Is this related to some weakness or limitation of the experimental results?  What experiments would you want to see in order to be convinced?
* Are there any particular works you can mention that use "complex neural architectures as Transformers/Diffusion models" to solve these or similar problems?  Or can you expand on how you envision this working?
Thanks for the detailed response. These responses address some of my concerns. However:

1. My question on (4) was that it no longer includes $q_\phi$ except the last term. optimizing it results in $q_\phi=p$, and I am sure that it is not what the paper is aiming to do. 

2. I am still not convinced that multi-agent approach is mandatory in small-sized experiments of Mujoco locomotion tasks. I believe that the optimization challenge addressed with multi-agent approach can be also handled with complex neural architectures as Transformers/Diffusion models. In this sense, considering the difficulty in optimizing multi-agent algorithms, there won't be much interested researchers willing to explore similar directions.

Due to these reasons, for now, I decide to keep the score.
I've recruited an emergency reviewer who should finish by tomorrow at the latest.
Apologies for the lateness.