Title: Distances for Markov chains from sample streams
Venue: NeurIPS 2025
Accepted: yes
Keywords: Optimal Transport

Markov Chains

bisimulation

primal dual

stochastic methods
Abstract: Bisimulation metrics are powerful tools for measuring similarities between stochastic processes, and specifically 
Markov chains. Recent advances have uncovered that bisimulation metrics are, in fact, optimal-transport distances, 
which has enabled the development of fast algorithms for computing such metrics with provable accuracy and 
runtime guarantees. However, these recent methods, as well as all previously known methods, assume full knowledge of the transition 
dynamics. This is often an impractical assumption in most real-world scenarios, where typically only sample trajectories are available.
In this work, we propose a stochastic optimization method that addresses this limitation
and estimates bisimulation metrics based on sample access, without requiring explicit transition models.
Our approach is derived from a new linear programming (LP) formulation of bisimulation metrics, which we solve using a 
stochastic primal-dual optimization method. We provide theoretical guarantees on the sample complexity of the algorithm 
and validate its effectiveness through a series of empirical evaluations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new stochastic optimization method for computing bisimulation distances between finite Markov chains using only sample trajectories, without requiring knowledge of the transition kernels. They extend results from   Calo et al. [2024], where computation of this metric required explicit transition kernels. The method is grounded in a novel linear programming (LP) formulation that characterizes the so-called bicausal couplings, enabling an estimation of the optimal transport distance between Markov processes. The authors introduce a saddle-point stochastic optimization algorithm (SOMCOT), provide rigorous sample complexity guarantees, and validate the approach with experiments. The paper builds upon recent work connecting bisimulation metrics with optimal transport distances and aims to generalize this theory to settings with limited observability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n* **Theoretical rigor**: The proposed method is based on a solid and novel LP formulation of bisimulation metrics, with full theoretical guarantees and a clean stochastic primal-dual algorithm.\n\n* **Nontrivial contribution**: Reformulating the problem to eliminate explicit dependence on transition kernels and make it sample-based is meaningful and potentially useful.\n\n* **Relation to OT and RL**: The work connects two active lines of research—optimal transport and bisimulation in RL—and contributes a new theoretical perspective.\n\n\n**Weaknesses:**\n\n* **Unclear practical motivation**: Although the paper highlights real-world relevance (e.g., only having access to sample trajectories), it lacks concrete scenarios where this distance is actually needed. No concrete application or decision-making scenario is provided where this distance is clearly useful.\n\n* **Limited experimental insight**: The experimental section includes illustrative examples (e.g., block Markov chains, parameterized random walks), but it is unclear what questions these experiments are intended to answer or what insight is gained. No direct comparison with competing estimators or baselines is provided.\n\n* **No clear real-world application**: It remains open how this framework would be applied in, for instance, reinforcement learning, model comparison, or structure discovery—despite several such references in the introduction.\n\n\n*Comments*\n* While the paper focuses on optimal transport-based distances between Markov processes, it might benefit from briefly situating itself within the broader literature on estimating or testing properties of Markov chains from a single trajectory. You should e.g. consider mentioning:\n    * [a] Hsu, D., Kontorovich, A., and Szepesvári, C. (2019). Mixing Time Estimation in Reversible Markov Chains from a Single Sample Path. The Annals of Applied Probability, 29(4), 2439–2480.\n    * [b] Daskalakis, C., Dikkala, N., Kamath, G., and Tixier, R. (2022). Identity Testing of Reversible Markov Chains. AISTATS 2022,  151, 1516–1540.\n    * [c] Wolfer, G., and Kontorovich, A. (2019). Estimating the Mixing Time of Ergodic Markov Chains. Proceedings of the 32nd Annual Conference on Learning Theory (COLT), PMLR 99, 1–40.  \n    * [d]  Wolfer, G. (2020). Mixing Time Estimation in Ergodic Markov Chains from a Single Trajectory with Contraction Methods. Proceedings of the 31st International Conference on Algorithmic Learning Theory (ALT), PMLR 117, 1–15.  \n\n* **To clarify**: my main concern relates to the lack of a clearly articulated practical use case for the proposed method. However, since optimal transport and RL  are not my  area of expertise, it's possible that I’m overlooking applications that others may appreciate. If this point is addressed or judged less important by other reviewers, I would be glad to see the paper accepted.\n\nQUESTIONS:\n* Q1:  page 3, line 3: Why do the authors refer to the processes as “stationary,” despite having deterministic initial states? Was it meant “ergodic” or “time-homogeneous”?\n\n* Q2: in “Preliminaries” there are notations $\\mathcal{X}^\\infty$ and $\\mathcal{X}^\\mathbb{N}$? Are they the same?\n\n\n* Q3: How does the proposed bicausal coupling compare to classical Markovian couplings? Could this be clarified more explicitly (is one a subset of the other?) (for Markovian vs non-Markovian couplings see e.g. [R1])\n\n\n* Q4: Could the authors provide a concrete example (e.g., from RL or model selection) where computing this distance would lead to a practically useful decision?\n\n\n* Q5: Why are the experiments focused solely on synthetic examples? Have any real-world applications been attempted or considered?\n\n \n* Q6:  The proposed formulation removes the need to sample from transition kernels \\( P_X(\\cdot|x) \\), \\( P_Y(\\cdot|y) \\), and instead relies on samples from the joint distributions over consecutive states. Could the authors clarify what assumptions are made about the availability of such samples? For example, if only a single long trajectory per process is available, under what mixing or subsampling conditions can these empirical joint distributions be used reliably in the stochastic optimization algorithm?\n\n* Q7: Each SOMCOT iteration costs $\\Theta(|X|^2|Y|^2)$ time and $O(|X|^2|Y|^2)$  memory, while the sample-complexity bound scales only with $|X||Y|(|X|+|Y|)$.  For state spaces beyond a few hundred states this becomes prohibitive. Do the authors foresee low-rank, sparse, or neural parameterizations that could make the algorithm practical? A brief discussion would strengthen the experimental narrative.\n\n* Q8: The distance is defined with a discount factor $\\gamma\\in(0,1)$. How sensitive is the estimator to the choice of $\\gamma$in practice? Are there guidelines (or theory) for selecting $\\gamma$ when the horizon of interest is finite or unknown?\n\n[R1] Czumaj, A., Vöcking, B. (2014). Thorp Shuffling, Butterflies, and Non-Markovian Couplings. In: Esparza, J., Fraigniaud, P., Husfeldt, T., Koutsoupias, E. (eds) Automata, Languages, and Programming. ICALP 2014. Lecture Notes in Computer Science, vol 8572. Springer, Berlin, Heidelberg.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal of this work is to approximate the Wasserstein distance between two distributions over an infinite-dimensional data space, from samples. Specifically, these distributions are discrete-time Markov processes. \n\nThe idea is as follows:\n- Classicaly write the Wasserstein divergence as minimizing a cost over couplings between the two distributions (Eq 1)\n- Decompose the distributions autoregressively using the Markov transition probabilities or occupancy measures (Eq 2)\n- Characterize the occupancy measures by three constraints (Eqs 6-7-8)\n- Dualize these constraints (Eq 9)\n- Solve the resulting min-max problem using a stochastic primal-dual method (Algorithm 1). \n\nThe authors provide computational and sample complexity guarantees (Section 4) and experimental validation (Section 5).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**. The writing is exceptionally clear. The main steps are clearly conveyed (see summary). Furthermore, the writing and notations make the parallels clear between this setup and the finite-dimensional case. It is appreciable to have a a theoretical analysis of the computational and sample complexity and Figure 2.b. is a particularly convincing use-case. \n\n**Weaknesses**. None in particular.\n\nQUESTIONS:\nQ1. Is this analysis amenable to continuous-time Markov chains, so as to compute the Wasserstein distance between two path measures?\n\nQ2. This work shows how to *estimate* the Wasserstein divergence between two Markov chains from samples, using a primal-dual optimization algorithm to solve a min-max formulation of the divergence. Suppose we additionally wanted *optimize* through the divergence, for example if one Markov chain is the \"ground truth\" and the other Markov chain is a \"parametric model\". Do the authors think  that the extra minimization would be numerically stable?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of estimating the adapted (bicausal) Wasserstein distance between two finite-state Markov chains from sample trajectories. Building on the observation that the bisimulation metric used in reinforcement learning coincides with the adapted OT distance under discounted cost, the authors propose a stochastic primal-dual algorithm (SOMCOT) that operates using only sample access to the underlying chains.\n\nA key contribution is a new linear programming formulation over occupancy couplings that eliminates dependence on the transition kernels, making the problem amenable to stochastic approximation. The algorithm performs mirror descent on the primal variables and projected gradient ascent on the duals. A non-asymptotic error bound of order O\\left( \\frac{1}{\\sqrt{K}(1 - \\gamma)} \\right) is established, with dimension dependence matching that of standard OT estimators.\n\nEmpirical results illustrate the method’s capacity to recover latent state structure and perform model selection from sample data. The formulation is theoretically sound and avoids the heuristics common in earlier deep RL applications of bisimulation. While the algorithm is framed in the language of bisimulation, the underlying quantity is precisely the discounted adapted Wasserstein distance, and the contribution is best viewed as a sample-based estimation method for this object.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\t1.\tWell-Formulated Estimation Problem:\nThe paper develops a stochastic algorithm for estimating the adapted (bicausal) Wasserstein distance between two Markov chains from sample streams. The problem is well-posed and connected to classical constructions via occupancy measures and causal couplings.\n\t2.\tSample-Based Optimization Framework:\nBy introducing a linear program over normalized occupancy couplings, the authors construct a saddle-point formulation that can be optimized without access to transition kernels. The method—SOMCOT—uses mirror descent and projected gradient ascent, and is supported by finite-time convergence guarantees.\n\t3.\tSample Complexity Analysis:\nThe convergence rate in \\varepsilon is \\widetilde{O}(1/\\varepsilon^2), which is consistent with minimax rates for empirical optimal transport. The dependence on 1/(1 - \\gamma)^2 is correct under the authors’ normalization, and avoids the more pessimistic 1/(1 - \\gamma)^3 rate that arises in standard Monte Carlo estimation of discounted costs.\n\t4.\tEmpirical Utility and Side Products:\nThe dual variables and conditional marginals (e.g., \\lambda_X(y|x), \\lambda_Y(x|y)) admit interpretations as encoder-decoder maps, and are shown to capture latent structure in block Markov chains. The method also shows promise for model selection and abstraction tasks.\n\n⸻\n\nWeaknesses\n\t1.\tFraming Ambiguity:\nAlthough the paper cites the bicausal OT literature and works with bicausal couplings, it frames the contribution primarily in terms of bisimulation metrics. This obscures the broader significance of the estimator as a sample-based method for adapted Wasserstein distances.\n\t2.\tPer-Iteration Computational Burden:\nThe per-iteration cost is O(|X|^2 |Y|^2), due to updates over full occupancy tensors. This limits scalability and raises the question of whether dynamic-programming-based or low-rank approximations could yield more efficient alternatives.\n\t3.\tMissing Normalization Clarification:\nThe paper does not explain that the discounted cost is normalized by (1 - \\gamma) in the definition of occupancy measures, and that this normalization is what permits the 1/(1 - \\gamma)^2 rate. Without this, readers familiar with standard cost accumulation may incorrectly expect a 1/(1 - \\gamma)^3 dependence.\n\t4.\tNo Discussion of Known Low-Rank Cases:\nIn the special case where both chains have rank-1 transition matrices, the problem reduces to static OT between stationary distributions. This is a classical LP (packing type), and the paper does not compare its approach or complexity with recent optimal running times for optimal transport. \n\t5.\tAbsence of Lower Bounds:\nThe paper asserts that its rate is optimal (up to logarithmic factors), but does not prove a matching lower bound or explore dimension-dependent regimes where structural assumptions (e.g., rank, sparsity) could yield improved complexity.\n\nQUESTIONS:\n1.\tComparison with Calo et al. (2024):\nThe proposed LP formulation appears closely related to the one in Calo et al. (2024), replacing conditional transition kernels P(x{\\prime}|x) with the marginal occupancy distribution \\nu_X(x, x{\\prime}). While the authors argue that their formulation is more suitable for the sample-access setting, conditional distributions can also be estimated from trajectories. Could the authors clarify precisely why their formulation offers an advantage over that of Calo et al., especially since this work builds directly upon it? A more careful comparison—both conceptual and empirical—seems warranted.\n\t2.\tSampling Cost under Discounting:\nThe algorithm requires samples from the distribution\n(1 - \\gamma) \\sum_{t=0}^\\infty \\gamma^t \\mathbb{P}(X_t, X_{t+1}),\nwhich may involve simulating long trajectories when \\gamma is close to 1. How is this distribution sampled in practice? Does the sample complexity bound in Theorem 1 account for the cost of generating such samples, particularly in terms of the expected number of steps required per sample?\n\t3.\tNormalization and Adapted Wasserstein:\nIn Equation (2), the objective \\langle \\mu, c \\rangle lacks an explicit (1 - \\gamma) factor. This raises concerns about alignment with the standard definition of the adapted (bicausal) Wasserstein distance, which includes discounting in the cost. Are the authors implicitly including the (1 - \\gamma) normalization in the definition of \\mu? If so, could this be clarified more explicitly to avoid confusion?\n\t4.\tConnection to Optimal Control and Dynamic Programming:\nThe adapted Wasserstein distance between Markov chains can be interpreted as the value of an optimal control problem over causal couplings. The variable \\lambda_X(y|x) in the paper resembles a policy mapping in this context. Is there a direct connection between the proposed algorithm and dynamic programming or optimal control formulations of adapted OT, such as those explored in Moulos (2021) or O’Connor et al. (2022)? Clarifying this connection could help contextualize the algorithm within the broader literature.\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCalo et al. (2024) showed that bisimulation metrics are a kind of optimal transport (OT) distance.\n\nThe present paper extends this by deriving a new stochastic optimization algorithm (SOMCOT) for estimating bisimulation metrics between Markov chains based on sample observations, without knowledge of the transition law. Key to this is proposition 1, which offers a new characterization of occupancy couplings, opening up a novel path to optimization through stochastic primal-dual methods. \n\nTheoretical guarantees are provided and the method is demonstrated on several toy examples\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- This is a sophisticated contribution with extensive theoretical guarantees. It seems to me a rigorous and interesting piece of work. It's also well written.\n- The approach relies only on sampling and not on knowledge of the transition law. This relaxation is a meaningful step forward and non-trivial.\n- A key contribution is the new LP formulation in proposition 1 which, if I’ve understood it correctly, I find to be a clever result allowing us to describe occupancy couplings without using the transition kernel explicitly. It does so by introducing auxiliary variables representing conditional distributions under the optimal coupling.\n- The paper offers an analysis of sample complexity with explicit constants.\n\nWeaknesses\n\n- There are at least two clear limitations to the work: it relies on finite state spaces and is extremely computationally intensive.\n- The finite state space assumption. While the authors acknowledge this limitation, it really does limit the applicability of the methods and there is little in the way of evidence to suggest an extension is possible.\n- The computational complexity. The sample complexity [O(|X||Y|(|X| + |Y|)/eps^2)] is clearly highly undesirable, as is the cost per iteration [Theta(|X|^2 |Y|^2)]. This implies for example that in a fairly modest system of 100 states, you would need ~2M/eps^2 samples, which for eps=0.1 (a low bar in some problems) implies 200M samples. I would assume, though I’m no expert, in realistic reinforcement learning scenarios, this will balloon to trillions of samples.\n- As far as I can see, the paper does not compare the approach empirically to any other existing method. I take the authors’ point that this method is the first of its kind (p.2) in some respects, but without any kind of comparison it’s hard to say whether the algorithm offers any advantage over simpler baselines, or how its complexity compares in practice to alternatives. They could have restricted their attention, at least in one example, to a case where this comparison was possible, e.g. some tractable small example where many methods could be used. The lack of empirical comparison is an issue: (i) exact methods exist in some small state space examples, (ii) I cannot tell from this work when to prefer this method over alternatives that may be available.\n- Nit: is it standard in this field to use x bar to mean an infinite-length vector x? In statistics this commonly refers to a sample mean, so this seems a little confusing.\n\nQUESTIONS:\nI would certainly be willing to increase my evaluation if some or all of the following could be addressed. They are all essentially related to practical utility. My feeling is that this work may be useful to the community, and so worth publishing, but that it doesn't make this case clearly enough at the moment. \n\n1. Better comparison with existing methods\n\nI wouldn't mind if the example is very small, but I do think there should be at least some way of comparing the current approach to previous methods, even just to see what is lost in terms of accuracy moving to a sample-based approach from explicit transition kernel knowledge. Would it be possible to use the approach of Calo et al. (2024) or maybe Chen et al. (2012)? I'm not particularly familiar with those works, but there must be some other method that could be compared to. I'd be very happy to see competitive accuracy from SOMCOT and perhaps demonstrating a regime where SOMCOT is superior. \n\n2. Empirical scalability\n\nHow does performance degrade on practical problems when we are limited in computation?\n\n3. Could the authors elaborate on a plan towards extending the approach to infinite state spaces using function approximation etc.? This isn't vital, but if they can show anything relating to it then that does strengthen the claim that the paper is a foundational step into a promising direction, and not a dead end.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-14"}