Title: A Generalized Bisimulation Metric of State Similarity between Markov Decision Processes: From Theoretical Propositions to Applications
Venue: NeurIPS 2025
Accepted: yes
Keywords: Markov decision process

reinforcement learning

bisimulation metric
Abstract: The bisimulation metric (BSM) is a powerful tool for computing state similarities within a Markov decision process (MDP), revealing that states closer in BSM have more similar optimal value functions. While BSM has been successfully utilized in reinforcement learning (RL) for tasks like state representation learning and policy exploration, its application to multiple-MDP scenarios, such as policy transfer, remains challenging. Prior work has attempted to generalize BSM to pairs of MDPs, but a lack of rigorous analysis of its mathematical properties has limited further theoretical progress. In this work, we formally establish a generalized bisimulation metric (GBSM) between pairs of MDPs, which is rigorously proven with the three fundamental properties: GBSM symmetry, inter-MDP triangle inequality, and the distance bound on identical states. Leveraging these properties, we theoretically analyse policy transfer, state aggregation, and sampling-based estimation in MDPs, obtaining explicit bounds that are strictly tighter than those derived from the standard BSM. Additionally, GBSM provides a closed-form sample complexity for estimation, improving upon existing asymptotic results based on BSM. Numerical results validate our theoretical findings and demonstrate the effectiveness of GBSM in multi-MDP scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper rigorously introduces a definition for Generalized Bisimulation Metric (GBSM). The authors establish the existence and convergence of this metric and prove two fundamental properties analogous to those of standard BSM: symmetry and an inter-MDP triangle inequality. While not proving that the distance between identical states is zero, they provide an upper bound. The utility of GBSM is demonstrated through theoretical analyses of policy transfer, state aggregation, and sampling-based estimation, showing tighter error bounds compared to traditional BSM. Few numerical experiments validate these theoretical improvements and conclude the paper.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS:\n\n- The paper successfully defines GBSM and formally proves its existence and convergence.\n\n- The work provides complete proofs for the symmetry and inter-MDP triangle inequality properties of GBSM, filling a critical gap in multi-MDP analysis.\n\n- The authors derive explicit and tighter error bounds on policy transfer (Theorem 4.1), state aggregation (Theorem 4.4), and sampling-based estimation (Theorem 4.5) compared to BSM.\nThe theoretical claims are  validated through numerical experiments that directly compare the bounds derived from GBSM with BSM and the ground truth.\n\nWEAKNESSES:\n\n- The authors claim to have proven the pseudometric properties of GBSM, but a key property - that the distance of a state to itself is zero - is not established. They only provide an upper bound.\n\n- The paper mentions in the introduction the prior work of Song et al. (2016), which already demonstrated the utility of GBSM experimentally. However, the current paper's numerical results are never contextualized by or compared with these previous findings.\n\n- The MDPs used for the numerical experiments in Section 5 are not defined, making the results difficult to interpret and impossible to reproduce. The authors answered \"NA\" to the checklist question about experimental details, but this information is essential for a complete submission.\n\n- The paper does not clearly articulate the broader impact or potential consequences of the work. The conclusion would benefit from an expanded discussion of the work's limitations and its potential applications.\n\nQUESTIONS:\nGiven my low confidence in assessing this paper, my questions are focused on clarifying points of potential misunderstanding.\n\n- In line 25, the triangle inequality appears to be defined with an equal sign. I assume this is a typo and should be \"less than or equal to\".\n\n- In line 166, the paper states it has \"rigorously proved the metric properties\". Given that the distance between identical states is only shown to be bounded - and not strictly zero - is using the word \"metric\" not an abuse of language?\n\n- In Theorem 3.6, a cost is defined for the shared value between distributions. Could you clarify the justification for this? My understanding is that the 1-Wasserstein distance should primarily concern the cost of moving the probability mass that is not shared.\n\n- The theoretical results are strong, but the conclusion is brief. Could you please expand on the potential impact this work could have on the field? Furthermore, how do your experimental findings relate to or advance the previous experimental work by Song et al. (2016)?\n\nLIMITATIONS:\nLimitations are not clearly stated. As previously mentioned, I encourage the authors to expand their conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a generalized bisimulation metric (GBSM) to measure state similarities across multiple Markov Decision Processes (MDPs) and rigorously establishes its fundamental metric properties, including symmetry, inter-MDP triangle inequality, and the distance bound on identical states. The paper applies GBSM to policy transfer, state aggregation, and sampling-based estimation, deriving explicit theoretical bounds for policy transfer performance, aggregation error, and estimation error. It also demonstrates that the error bounds of GBSM are significantly tighter than those of the standard BSM. Furthermore, the paper introduces a closed-form sample complexity approximation method, improving upon existing asymptotic results based on BSM, and supports its theoretical findings with numerical experiments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Leverages Wasserstein distances and existing MDP structures in a zero-shot manner, requiring no additional data collection or model training while maintaining rigorous theoretical guarantees.\n2.GBSM achieves significantly tighter theoretical bounds in policy transfer (regret bound reduced by 1-γ), state aggregation (error bound compressed to1\\/(1-γ) of BSM baselines), and sampling estimation (closed-form sample complexity K), with experimental validation showing 60-80% improvement over existing methods.\n3.The paper presents a rigorous and comprehensive theoretical framework for the GBSM, defining it clearly, proving key properties (e.g., symmetry, triangle inequality, distance bounds), and establishing convergence using fixed-point theory.\n\nWeakness:\nmajor concern：\n1. GBSM relies on exact transition probabilities P and reward functions R across MDPs (Eq. 2), which are often impractical to obtain in real-world RL settings. Many applications, such as robotics and healthcare, involve stochastic dynamics that are challenging to model accurately. Additionally, the computation of the Wasserstein distance over entire state spaces introduces significant computational overhead, particularly in high-dimensional environments, thereby limiting the scalability of GBSM. Exploring approaches to approximate P and R, or employing scalable approximations of Wasserstein distance, could address these limitations and improve the applicability of GBSM in real-world scenarios.\n2. The numerical validation in Section 5 relies solely on synthetic MDPs with random perturbations. Could the authors clarify whether these experiments sufficiently capture the complexity of real-world RL tasks? Would testing on standard benchmarks help demonstrate the applicability of GBSM in practical scenarios? Additionally, the aggregation and estimation simulations appear to simplify the complexity of real distribution shifts. Could the authors comment on the potential limitations this might impose on the generalizability of the results?\n\nQUESTIONS:\nAlthough the proposed GBSM framework demonstrates strong theoretical guarantees and significant improvements over baseline methods, certain aspects of its practicality and design motivations are not clearly articulated. The reliance on exact transition probabilities and reward functions, which are often impractical in real-world RL settings, raises concerns about its scalability and applicability. Additionally, the numerical experiments, conducted solely on synthetic MDPs, do not sufficiently demonstrate the method's efficacy in complex, real-world tasks. While the paper provides rigorous theoretical foundations, the lack of clarity in these aspects may confuse readers and limit the perceived utility of the method.\nGiven these issues, I am inclined to suggest a \"Borderline\" decision. However, I am open to revising my score if the authors address these concerns by providing clearer justifications, exploring practical approximations, and validating the method on more realistic benchmarks.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper studies the Generalized Bisimulation Metric (GBSM) as a distance measure across multiple MDPs. Specifically, it proves that GBSM has pseudo metric property by showing symmetry, the triangle inequality, and a zero distance for identical states. Then, leveraging these properties, this paper analyzes policy transfer, state aggregation, and sampling-based estimation. Numerical experiments on a toy example further demonstrate GBSM’s advantages over the classical bisimulation metric.

STRENGTHS AND WEAKNESSES:
Strengths:
This paper first establishes that the Generalized Bisimulation Metric (GBSM) satisfies some metric axioms, marking a solid theoretical advance. The presentation is clear and well structured, and the toy experiments illustrate effectiveness of GBSM. Overall, the work offers a meaningful contribution to the reinforcement learning theory literature.

Weaknesses:
I did not find major shortcomings. However, the extent of its theoretical novelty remains unclear. Frankly, although I appreciate the usefulness of GBSM, I’m not convinced the contribution is strong enough for acceptance at this conference.

QUESTIONS:
Could GBSM be extended to continuous-state MDPs or to the average-reward setting?

Could GBSM be incorporated into Value Iteration or policy gradient methods?

Were any critical obstacles or techniques involved in proving the theorems?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the question of formalizing a bisimulation distance between distinct MDPs, with possibly different state spaces. They begin by introducing a desirable fixed-point formulation for such a distance, and proving its existence using the Knaster-Tarski fixed point theorem. They then provide a number of theoretical properties on the distance, such as its relationship to optimal value differences between the MDPs, as well as regret bounds on policy transfer. They next consider its sample complexity, and provide sample complexity results on how many samples are needed in the empirical MDP. They then compare it empirically to standard bisimulation acrossa a task of randomly-generated MDPs.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper has a high level of theoretical rigour, with each result carefully proven. \n- The paper is clearly written.\n\n---\n**Weaknesses**\n- The variant of the bisimulation metric used is the original (which takes $\\max$ over actions), which has been shown to have a number of drawbacks (lack of connection to non-optimal policies, biased estimation, etc). I think that these issues will only be exacerbated in the cross-MDP setting, and that the resulting metric would be vacuous. I think the paper can be strengthened greatly by either (i) generalizing the provided results to one of the relaxations of the original bisimulation metric, or (ii) evaluating the metric across a pair of two small-scale, interpretable MDPs and demonstrate non-trivial bounds. \n- Although it is clearly written and correct, the theoretical results all lack some novelty. In particular, to my knowledge, every proven result has a very similar equivalent in literature. \n- The theory is set up to require that both MDPs have the same action space, even though their state spaces are allowed to differ. I think that arguably different action spaces is a more realistic setting, and should be relatively easy to adapt to (as done in lax bisimulation metrics), if the authors agree.\n- The empirical evaluation is very limited: they are currently only done on randomly-generated MDPs. The results themselves are difficult to interpret currently, as for example it is not stated what the x-axis represents.\n\nQUESTIONS:\n- Can you elaborate more on the construction of the MDPs for the experiments? (Are they Garnet-style, what is the state space, how different are the state spaces of the pairs, etc). \n- What is the x-axis in the experiments? What is the y-axis as well (seems to be incorrectly labelled \"values\")?\n- Why do you allow for different state spaces but require the same action space? Can this be relaxed?\n- Is it possible to perform any experiments on more interesting MDPs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}