Linear Mixture Distributionally Robust Markov Decision Processes
Authors: Zhishuai Liu
Pan Xu
Venue: NeurIPS 2025 poster ()
Keywords: off dynamic reinforcement learning
distributionally robust MDP
linear mixture MDP
Primary area: reinforcement_learning
TL;DR: A new framework for DRMDP with a novel uncertainty set design.

Abstract: Many real-world decision-making problems face the off-dynamics challenge: the agent learns a policy in a source domain and deploys it in a target domain with different state transitions. The distributionally robust Markov decision process (DRMDP) addresses this challenge by finding a robust policy that performs well under the worst-case environment within a pre-specified uncertainty set of transition dynamics. Its effectiveness heavily hinges on the proper design of these uncertainty sets, based on prior knowledge of the dynamics. In this work, we propose a novel linear mixture DRMDP framework, where the nominal dynamics is assumed to be a linear mixture model. In contrast with existing uncertainty sets directly defined as a ball centered around the nominal kernel, linear mixture DRMDPs define the uncertainty sets based on a ball around the mixture weighting parameter. We show that this new framework provides a more refined representation of uncertainties compared to conventional models based on $(s,a)$-rectangularity and $d$-rectangularity, when prior knowledge about the mixture model is present. We propose a meta algorithm for robust policy learning in linear mixture DRMDPs with general $f$-divergence defined uncertainty sets, and analyze its sample complexities under three divergence metrics instantiations: total variation, Kullback-Leibler, and $\chi^2$ divergences. These results establish the statistical learnability of linear mixture DRMDPs, laying the theoretical foundation for future research on this new setting.

Linear Mixture Distributionally Robust Markov Decision Processes
Zhishuai Liu
Pan Xu
Department of Biostatistics & Bioinformatics Duke University
Department of Biostatistics & Bioinformatics Duke University
Durham, NC 27708 zhishuai liu@duke.edi
Durham, NC. 27708 p xu@duke edu
Abstract
Many real-world decision-making problems face the off-dynamics challenge: the agent learns policy source domain and deploys it in a target domain with different state transitions. The distributionally robust Markov decision process (DRMDP) addresses this challenge by finding a robust policy that performs well un- der the worst-case environment within pre-specified uncertainty set of transition dynamics. Its effectiveness heavily hinges on the proper design these uncertainty sets, based prior knowledge of the dynamics. In this work, we propose novel linear mixture DRMDP framework, where the nominal dynamics is assumed to be a linear mixture model. In contrast with existing uncertainty directly defined a ball centered around the nominal kernel, linear mixture DRMDPs define the uncer- tainty based ball around the mixture weighting parameter. We show that this new framework provides more refined representation of uncertainties com- pared to conventional models based on (s, a) rectangullartt and d-rectangularity, when prior knowledge about the mixture model is present. We propose a meta algorithm for robust policy learning in linear mixture DRMDPs with general f- divergence defined uncertainty sets, and analyze its sample complexities under three divergence metries instantiations: total variation, Kullback-Leibe divergences. These results establish the statistical learnability of linear mixture DRMDPs, laying the theoretical foundation for future research on this new setting.
and x²
Introduction
In off-dynamics reinforcement learning (RI.) [21, 48, 10, 45), an agent trains policy in source domain and then deploys the learned policy in an unknown target domain with possible dynamics shift from the source one. The dynamics shift may arise from the sim-to-real gap [34, 57, 7] when the policy is trained on a simulator that mimics the real environment. The dynamics shift may cause catastrophic failure since the policies learned from the source domain would admit distinct behaviors in the target domain which thereby deviate the agent from achieving its goal, e.g., maximizing a long term cumulative reward in the target domain.
line of literature have tried to solve off-dynamics RL problems from various perspectives differentiated by the extend of information [21, 48, 10, 45, 24]. In this work, we are particularly interested the case where we can only collect data from the source domain for policy learning. In other words, the agent has little knowledge about the target domain but only holds the belief that its dynamics should be structurally close to the source domain. Such large uncertainty in the target domain necessitates robust policy learning to train policies that generalize well across a wide range of target domain. To this end, the agent usually has to adopt a pessimism philosophy in the face of large uncertainty prepare for the worst case dynamics.
collect from the source and target domains
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In this work, we tackle off off-dynamics RL through a framework widely studied in recent decades that adopts the pessimism philosophy described above, the distributionally robust Markov decision process (DRMDP) [38, 18, 32, 49, 47]. In DRMDPs, the term dynamics specifically refers to the transition kernel P that takes as input the state-action pair and outputs the next state. The uncertainty of the target domain dynamics is modeled as uncertainty set defined by some probability measure centered around the transition kernel de of the source domain. Given a specific construction of the uncertainty set, an agent aims to learn robust policy that secures a fair amount of cumulative reward uniformly across all target domains represented within the uncertainty set. In essence, the DRMDP formulates max- min optimization problem, under which the goal is to find policy that is guaranteed to perform well even in the worst-case environment within the uncertainty set. Recent studies [18, 47, 12] have shown that both the tractability of the max-min optimization problem and the performance of the learned robust policy depend heavily on the design of the uncertainty set. On the one hand, Wiesemann et al. [47] show that arbitrary uncertainty sets could render the max-min optimization problem NP-hard. On the other hand, an appropriately designed uncertainty set- on suitable prior information can effectively model distributional shifts and prevent the robust policy from becoming overly conservative.
To demonstrate this further, let first consider the case where no prior information on the struc- ture of transitions is available. It is then reasonable to independently construct uncertainty sets centered around transitions at each state-action pair, i.e., ((P)) ( U (-)s, a)). This gives us the commonly studied (s, a) recttnngull uncertainty set [18, 52, 16].
well- that (s, a)-recaangal uncertainty sets may include transitions that would never occur in the target domain Jomain--prticulatte when the state-action space is large, as is often the case in many real-world roplications-reslltii in overly conservative policies [12, 29, 25]. To address this issue, one option is to incorporate transition structure information into the uncertainty set design.
A commonly studied class of transition is the mixture distribution [19, 2, 6, 58, 22], which frequently arises in practice [30, 35, 36). Assuming we are equipped with the prior information that the source domain transition kernels are linear mixture distributions of some basis modes ((-1s, a) that we have access to, i.e., (-|s, a) = (0", Φb-1s, a)). where 0 is some unknown mixture weighting parameter. Then it is reasonable to hold the belief that the target domain dynamics maintains the linear mixture structure, i.e., P(-|s, a) (0, Φ a)), while the parameter is subject to some perturbation from 0¹ This kind of perturbation cannot be precisely characterized by existing uncertainty set designs in literature. In this work, we propose the novel linear mixture uncertainty ser. We formally establish new framework for DRMDPs with linear mixture uncertainty sets, dubbed as the linear mixture DRMDP. This formulation is intrinsically different from existing frameworks due to the introduction of structural information into both the dynamics and the uncertainty set design. Focusing on the offline RL setting where we only have access to an offline dataset pre-collected from the source domain by behavior policy n², we provide answers to the following fundamental questions:
How many samples are required to learn c-optimal robust policy for linear mixture DRMDPs?
In this paper. we provide the first ever study on linear mixture distributionally robust Markov decision processes. We summarize our main contributions as follows:
We show that the novel design of the linear mixture uncertainty can achieve more refined quan- tification of the dynamics shift compared to the standard (s. a) -recttngga and the d-rectangular uncertainty set, hence could potentially be more favorable. This justifies the need of linear mixture DRMDPs. Further, we prove that the dynamic programming principles hold for linear mixture DRMDPs, which motivate the algorithm design and theoretical analysis.
We propose a meta algorithm based on the double pessimism principle [5] and transition targeted ridge regression [22] for linear mixture DRMDPs with general probability divergence metric defined uncertainty sets. From the theoretical side. we prove that when instantiating to the commonly studied TV, KL and x² divergences, our proposed algorithm achieves upper bound on the suboptimality in order of O(dH22=** /VK), O(dH
and Ó(d( H²) /VK)' respectively, showing the statistical learnability of linear mixture DRMDPs.
Here d is the number of basis modes, H is the horizon length, C"" is a coverage parameter of the offline dataset (see Assumption 5.1). A is the lower bound on the dual variable of KL- KL-divergence (see Assumption 5.5), p is the uncertainty level and K is the number of samples in the offline dataset.

---

Notations We provide the notations used in this paper for the reader's ' reference. We denote ( as the set of probability measures over some set S. For any number H 2 we denote [H] as the
set of 2,, .... H). For any function V : S - R. we denote [P, a) P.'mPa(-),,aa [Vs')] as the expectation of V with respect to the transition kernel Pr. and [s)]a = (V) a}, given a scalar a > 0, as the truncated value of V. For a vector æ. we denote 2; as its j-th entry. And we denote [ as a vector with the i-th entry being 24. For a matrix A, denote (A) as the i-th eigenvalue of A. For two matrices A and B, we denote = B as the fact that semidefinite matrix, and A > B as the fact that A - B is a positive semidefinite matrix. For any function f: S:S + IR we denote |l = sE [s)). Denote A as the d dimensional simplex. For any two probability distributions P and Q on S such that P is absolutely continuous with respect to Q, we define the total variation (TV) divergence as DTv (PIQ) = 1/2 Ss P(s) - (ds,s
A is a positive
the Kullback-Leibler divergence as (Pl()) = Ss P(s) log P(s)/ ((sss(s((// ({))) ds and the x² divergence
as x²= (PIIQ) = Is (P(s) - Q(+))
2 Related works
A substantial body of empirical research also explores off-dynamics RL through the lens of domain adaptation and transfer learning [10, 8, 54, 50, 46, 14, 45, 28, 7, 13], among others. In this paper, we focus on the DRMDP formulation of off-dynamics RL. Readers are referred to the aforementioned works for complementary approaches that explore this orthogonal line of research.
Several lines of studies have extensively studied DRMDPs from different perspectives [49, 61, 3, 53, 5, 41, 27, 25, 42, 26], including offline robust policy learning, online data exploration, and function approximation, etc. We particularly focus on works studying uncertainty set design in this part. The seminal works of [18, 33] proposed the DRMDP with (s, reettanguulrr where the uncertainty set is constructed independently at each state-action pair. Wiesemann et al. [47] then studied the s-rectangular uncertainty set, which includes the (s, a) rectangular uncertainty set special case. They also showed that solving DRMDPs with general uncertainty sets can be NP-hard. When the state and action spaces are large, DRMDPs with s-rectangular and (s, a) -rectangular uncertainty sets suffer from issues of conservative policies and intractable computation complexity. Goyal and Grand-Clement [12] proposed to leverage the latent structure of the transition kernel and propose the r-rectangular uncertainty set, which was then shown to be significantly less conservative than prior approaches. Motivated by the design of r-rectangular uncertainty set, Ma al. [29] proposed the setting of d-rectangular linear DRMDPs, where the transition kernel is assumed to be linear combination of a known feature mapping and unknown factor distributions, and the uncertainty is constructed upon perturbations onto the factor distributions.
Beyond the conventional rectangularity framework, Zhou et al. [60] proposed two novel uncertainty set, one is based on the double sampling and the other on an integral probability metric, to achieve more efficient and less conservative robust policy learning. Zouitine et al. [62] proposed the time- constraint DRMDPs, where the uncertainty set is modeled to be dieeddepnnenn,, to accurately reflect real-world dynamics and thus solve the conservativeness issue. Li et al. [23] studied DRMDPs with non-rectangular uncertainty sets, where dynamic programming principles do not hold, and they provided a policy gradient algorithm to learn the optimal robust policy. Our work also aims to address the issues of conservativeness and computational tractability in DRMDPs. However, we tackle this problem from a completely different perspective. Specifically, with appropriate prior information, we assume that the uncertainty originates from perturbations in the underlying parameters that define the model. Given the great interest linear mixture model from both practical [20, 1 11] and theoretical [2, 19, 58] sides, our linear mixture DRMDP framework provides the first result on robust policies learning when the source and target transitions are linear mixture model.
3 Linear mixture distributionally robust Markov decision process
An episodic distributionally robust MDP is denoted as DRMDP(S,, A, H, HP with the H horizon length H, time homogeneous nominal transition kernel pe (PA) 1 deterministic reward
function {rA 1 and uncertainty set set (P*) Un (P)). The robust value function and
3

---

robust Q-function are defined as
H
(o) = inf
F²
(1 = 8,
(3.1)
PE
t=h
(s, =
inf (PU(
EP
Σ re(s at) 8/ = ap a,
We assume transition kernels are mixture distributions in the following assumption. Assumption 3.1 (Linear Mixture Models [2]). For any (s,a, S x A, there exists feature mapping Φ S:S x A + (A(S)), where 6(-1s, a) = [-1 a), oa(-1s, a)) and oi(-|s, a) E A(S), Vi E [d]. Assume there exists d-dimensional vector 0h, where 0g Ad-1, such that
( = (1((1s, BR), ((s.a)
(3.2)
(-(s|s, a) is often referred to as the basis latent mode [19, 2]. The nominal transition (- a) is a probabilistic mixture of the basis latent modes, and 0% are the weighting parameters. We assume the agent knows the basis latent modes, but doesn't know 0% In the literature, there several kinds of assumptions parameter norms. The current formulation leads to 10% ll S L. and [ox(-|s, (ss a) || 1, for any (i, s, a) E ld] x x A. For any function V S - [0, 1] and (s, a) E A, we denote Φ (s, = Is (ss a) (s')d's then we have | V (s, a)
Vd.
Now given the structure of the nominal kernel pa, we define the uncertainty set. In particu- lar, we assume the parameter 0 can be perturbed in the test environment, that is, Uh. UP(()) ((s.))Exx1l((s..0 az 0%), where
UP(s, az 0% = (P(+s, E A(S)) = (0(-18, a): :0: :0 ((0) S p), (3.3)
D(-II.) is probability divergence metric that will be instantiated later. Denote O = { A A (0)) S p} as the parameter uncertainty set. Notably, the uncertainty set of the transition 24 is determined by the uncertainty set of the weighting parameter, θ), We refer to the uncertainty set defined in (3.3) as the linear mixture uncertainty set and the DRMDP equipped with the linear mixture uncertainty set as the linear mixture DRMDP. We highlight due to the fact that the perturbation on parameters 0₂ decoupled among times steps, the linear mixture uncertainty set belongs to the family of rectangular- uncertainty sets, which also includes the (s, a--rectangularity [18], s-rectangularity [47] and r-rectangularity [12], etc. It is well known that the rectangularity property ensures the dynamic programming principles hold for DRMDPs. Following the proof of Propositions 3.2 and 3.3 in [24], establish the (optimal) robust Bellman equation and the existence of deterministic optimal policy.
Proposition 3.2 (Robust Bellman Equation). Under the linear mixture DRMDP setting, for any nominal transition kernel pe and any stationary policy [T the following robust Bellman equation holds: for any (h, 8, a) E [H] x SXS. A.
(s, a) = ())
inf Pr... (8.00)))
(())(s) (a 10 (s,
Proposition 3.3 (Existence of the optimal policy). Assume the nominal transition kernel po satisfies (3.2) and the uncertainty satisfies (3.3). Then there exists a deterministic and stationary pilicy m such that V /0 (s) = V and Q "(s, a) = QT (s, a), for any (h, a,a) E [H] S A.
Then have the robust Bellman optimality equation:
(7 a) = (8, +
inf (a(le.))E(((e.
V = max
a
Then it suffices to estimate the optimal robust Q- fuuctioo " to find n²

---

12
dz (2:002) 02)
p (1102,
po
du (-1111 a1)
$1 (152,, a2)
(2(1111.11)
61 (11811 a1)
(111,a1)
01(- (1)))
33
(a) Comparison with (s, re-rtanggn uncertainty set. (b) Comparison with d-rectangular uncertainty
Figure I: Illustrations of the comparison among (s, a) -eretanngl d-rectangular, and linear mixture uncertainty sets in R3, where simplex and each point in the region is probability distribution. Φ1 and de are two basis modes, which are represented by blue dots. The nominal kernel pe represented by red dots, with 1/2] The linear mixture uncertainty set with radius p 1/8 is the magenta segment. (a) The orange hexagons represents the smallest (s, a) rectanguull uncertainty set centered around pd that cover the linear mixture uncertainty set. (b) The orange hexagons represent the uncertainty sets centered around factor distributions n and pq- The green hexagons represent the ddrretangular uncertainty sets, which are linear combinations of the uncertainty sets of the factor distributions and cover the lincar mixture uncertainty set.
{I1, 22,,,2,50 233.. The light blue region represents the probability
the offline dataset consisting K trajectories collected from the nominal environment by the behavior policy x². The goal of the agent is to learn an optimal robust policy π only using the offline dataset D. Denote the learned policy as f, we define the suboptimality of क SubOpt(ft, p) == VT ( (s1) (81) which is used to evaluate the performance the estimated policy
Offline dataset and learning goal Denote D = {[( 4% ( H.K
3.1 Comparison with (s, reettangul uncertainty sets
Now we compare our linear mixture uncertainty set with the most commonly studied (s, a)- rectangular uncertainty set [18, 32]. For the ease of illustration, focus on the total variation (TV) divergence.
Lemma 3.4. When the probability is specified to the TV divergence for both the linear mixture uncertainty set and the (s, a)-reccang uncertainty set, the regular (s, rectangulll uncertainty set can be strictly more conservative.
As illustrated by the example in Figure la, the TV divergence defined linear mixture uncertainty set strictly smaller than the TV divergence defined (s, a) rectangular uncertainty set. We conclude that (1) the linear mixture uncertainty set can achieve refined representations of the dynamics perturbation, and (2) the size and shape of the linear mixture uncertainty set essentially depends on the pre-specified basis latent modes. Finally, show in the following lemma that the linear mixture uncertainty set can be adapted to recover the standard (s, -rreenggga uncertainty set.
Lemma 3.5. In tabular DRMDPs with finite state and action spaces, the linear mixture uncertainty set can recover the standard (s, -recttanga uncertainty set by design.
3.2 Comparison with d-rectangular uncertainty sets
Another well-studied uncertainty set with linear structure is the d-rectangular uncertainty set [12, 29, 24, 25, 44]. Next we explore the relation between the d-rectangular uncertainty set and the linear mixture uncertainty set. Given the fact that the standard linear MDP and standard linear mixture
5

---

MDP do not cover each other [59]. it is not straightforward to compare the more complex uncertainty sets. However, we find case that the d-rectangular uncertainty set can be more conservative, as illustrated in Figure 1b. Specifically, S = {81. 82, 83), 14" = [4%.1(] [0] is the factor distribution and the orange hexagons are factor uncertainty sets [24] of the d-rectangular linear DRMDP. There exists feature mapping +b(:;-) S x A + I2² such that Po a) (eb(s, a),, (.)) Moreover, let 0 = [1/2, 1/2] and -() be basis modes such that (- a) = (Ø(-1s, a), 0²) in the linear mixture DRMDP. We can observe that the smallest d-rectangular uncertainty sets covering the linear mixture uncertainty sets are the green hexagons. We can conclude that the di d-rectangular uncertainty set is strictly larger than the linear mixture uncertainty set in this instance.
Remark 3.6. We note that the comparison with (s. re-etaggga uncertainty sets and d-rectangular uncertainty sets is only to show that in particular cases, the linear mixture uncertainty set has advantages in modeling the uncertainty while existing uncertainty sets fall short. It does not lead to the conclusion linear mixture is always better or always less conservative. We highlight the linear mixture DRMDP assumes prior information, ie.. the linear mixture dynamics and known basis modes, it's 's not fair to directly compare it with other DRMDP settings because they either require other prior information, e.g,, the linear MDP structure and feature mapping Vb for d-rectangular DRMDP, or do not require prior information at all, e.g., the (s, rectanguull DRMDP.
4 Algorithm design
In this section, we design a meta algorithm to learn the optimal robust policies under various probability divergence metrics. Due to technical reasons (see Remark 4.2), we assume that for each state & E S, there are finitely reachable states.
Assumption 4.1. v(h, s, a) E [H] x A. we denote the feasible set corresponding (s, a) as (s, = {s' E S P (s' s, ss > 0}. Assume there exists positive constant Pmin > 0, such that min Ph [8, a) > Punin v(h, 8,8) a) E [H] x S x A.
A direct implication of Assumption 4.1 is that for any (h, s, a) E [H] x S x A. we can construct a feasible set (s, with cardinality (s, a)] [1/Pminn that contains Sa (s, a), i.e., Sp (s, a) C
Sa (s...a a). Specifically, for s' (f.ssaa/S.. (s, a), we have (s((''s, a) O. Let 5, f0, 1] (] 1//mial be the one-hot vector with 6x(s) = 1, we propose to estimate the mixture weights (0 Th=1 by solving the regularized transition targeted regression:
min BER k=
(Ø(s)s T dek (s)) + tlll
(4.1)
( (
Then we have the close form solution 0% = An brh where
K
A
Σ Σ
sss)sh (ss)s (t) T + ALd, by = Σ Σ
5 (s)0(s(sh.
k=1 (ES
k= 1-L
Remark 4.2. Though the most commonly studied method for parameter estimation in standard linear mixture MDP literature is the value-targe targeted regression [2, 58, 55), we find data coverage issue that is hard to bypass in the suboptimality analysis for algorithms using the value-targeted targeted regression for parameter estimation. Instead, we propose substitute the value- target by the transition information 156, 22j. With Assumption 4.1, we are able to construct feasible sets Sy(8, a) and conduct the transition-targele ridge regression (4.1). Besides the difference in the larget, the transition-turgeted regression estimation also induces a notable problem in the concentration analysis. To see this, we note that typically we resort to the self-normalized concentration lemma for vector-valued martingales regression. However, as discussed
[1, Theorem 1] to bound the estimation error of the in [22], the errors, E [P(s]s a 6.h,, (s))esSs aty) Vk: E [K), in the transition-targert
ridge regression are not independent due to the fact that ExEsatefaat c]((s) = 0. Thus, the self- normalized concentration lemma in [1] does not apply anymore, as the independence between errors is an essential condition. To solve this issue, we resort to the concentration lemma proposed by [22], which is specifically tailored to the dependent error structure in (4.1). With the concentration result,
we can construct confidence sets as follows.

---

Define the confidence set , (0 RS - < BBh), where the radius 3h, is to be determined. By adapting the Lemma [22] (see Lemma D.2 for details), we immediately have the following lemma stating that, with high probability, the true parameter lies in O
Lemma 4.3. Let & (0, 1). For all h E [ if we set 3h
v 7x (2 " ? + dolog(4 +
4 4[1/Pmiu / K Ad)), then with probability at least 1 - Ç, it holds that 0% E O
Given the confidence set of the parameter, next we define the confidence region for transition kernel. Specifically, denote
P= @hene Pr. and Pa (-(-1-))) ) 0, @
(4.2)
the confidence region, where θ O A d-1 Leveraging the double pessimism principle proposed in [5], we define the value estimator as
J (T) =
inf PAEPA.ISASH (gg())
inf V. (T1)(s)),
where (si) represents the robust value function, with the nominal kernel P. The policy that maximizes the doubly pessimistic value estimator as the estimated optimal robust policy,
# == argmax Peas (T).
(4.3)
=ell
We present our meta-algorithm in Algorithm 1.
Algorithm Meta Algorithm of Policy Optimization for Linear Mixture DRMDP
The offline dataset D, the regularizer X, and the robust level p. 2: Construct the confidence region P according to (4.2).
Input:
3: Get the estimated optimal robust policy मे by (4.3).
4: Return: Policy f.
Theoretical guarantees
In this section, we provide corresponding finite sample guarantees for Algorithm 1, In particular, we instantiate the probability divergence metric (-I--) in (3.3) as the TV, KL. and diviivergssces, which are most commonly studied in literature [18, 52, 42].
5.1 TV-divergence
Before present the result the finite sample suboptimality upper bound, introduce the following coverage assumption on the offline dataset.
Assumption 5.1 (Robust Partial Coverage: TV-divergence), For any P E P. denote the worst-case transition at (s, a) as
"t-ss())a))== arg inf Egr an) [V [V
(5.1)
(P)))
the problem dependent robust value covariance matrix as AT (o: dx' h.prrit Vita.r) = (o h pn
( Vitt. an)] (oliter [s))al (5.2)
and the sampling covariance matrix as
Ag
Σ
1st. at ( sh+ 1st, TT
(5.3)
Then assume there exists positive constant C > 0, such that
T A (a; d= pepit Vi
C" H²
sup sup sup aE(o. PEB xeRd
A/
K

---

Remark 5.2. Assumption 5.1 is robust partial rype coverage assumption on the offtine dataset. It only requires the offline dataset have good coverage on the state-action space visited by the optimal robust policy x², and it considers the worst case transition in (5.1). Assumption 5.1 resembles the partial coverage assumption for standard linear mixture MDP in [43] (see Table their paper for more details). Nevertheless, we highlight two major differences arising from the robust setting we considered in this work. First, the value covariance matrix AT is defined under the worst case transition. Second, there is an additional supremum over which arises from the dual formulation of the TV divergence defined robust value function. Lastly, Assumption 5.1 considers all transitions in the confidence region P. which depends on the offline dataset. Thus, it implicitly imposes a constraint on the offline dataset. We note that P can be replaced by the set of all feasible transition kernels a) TT 0 0 E TR Ei= 0; = 1]. which would lead to a stronger assumption though.
M = { MM
Theorem 5.3 (TV-divergence). Assume Assumptions 3.1, 4.1 and 5.1 hold. For Ç E (0, 1), there exists an absolute constant e. >> O, such that if we set X d in Algorithm I, then for TV-divergence uncertainty set, with probubility at least 1 C, we have
SubOpi(#t 81) S edH2C= K- 1og/ log(K///o/oog K/Pmind ()-
This result is the first of its kind since are studying new framework. Nevertheless, we can compare it with results for standard linear mixture MDPs. We note that [2] present a suboptimality bound in the order of (dHH3/2/K) They assume in their work the transition is homogeneous, say, P =....-= P'= = P. which means the weighting parameter 0 is shared across stages. This would reduce their bound by vH, which is indicated by [58]. If their analysis modified inhomogeneous transitions, an additional O ((H) factor would be added up. Thus, the bound in Theorem 5.3 matches that in Theorem 1 of [2] in terms of dimension d and horizon length H. Next, we translate the suboptimality bound in Theorem 5.3 the sample complexity bound.
Corollary 5.4. Under the same assumptions and setting as in Theorem 5.3, to learn an e-optimal policy with probability at least 1 - S, we require the sample size K satisfying
d (H+((C)) (C*( )c
K K=Ō 0
2
5.2 KI.-divergence
For KL divergence, we require different assumptions due to its distinct geometry and dual formulation. Assumption 5.5 (Regularity of the KL-divergence duality variable). We assume that the optimal dual variable X for
[exp___exp { (Ei-o, Texp {
Φ. Vi /xH) - ~xp}
sup XER
is lower bounded by >> for any O₁ P (o(-).) On PM and step h E [H]. Remark 5.6. Assumption 5.5 is a condition on the dual variable of the KL divergence, which is specific to the linear mixture DRMDP with KL divergence defined uncertainty set. We note that a similar assumption also appears in Assumption F.1 of [5], which is proposed to guarantee the provably efficient offline learning of the d-rectangular robust linear MDP.
Assumption 5.7 (Robust Partial Coverage: KL-divergence). Define the worst case transition kernel pr àt and the sampling covariance matrix AQ as (5.1) and (5.3), respectively. For any P E P. denote the problem dependent robust value covariance matrix as
KILP
VKILP 37.. (5.4)
A (X; dee ta VHH1.) = =Eron partt exp
exp {
Then we assume there exists a positive constant C > 0, such that
TT AKL (A; d
h.pp Viti
CTT
sup sup sup XEAA.//p PEP ZeRd
x A
K
8

---

Remark 5.8. Assumption 5.7 shares the same spirit with Assumption 5.1, except that it is designed for the KL divergence based uncertainty set, which leads to the different definition of the robust value covariance matrix AR different supremun operation over the dual variable X, and different order of the partial coverage upper bound C /K.
Theorem 5.9 (KL-divergencc). Assume Assumptions 3.1, 4.1, 5.5 and 5.7 hold. For S E 1). there exists an absolute constant с O, such that if we X = d in Algorithm I, then for KL-divergence uncertainty set, with probability at least - C, we have
SubOpt(ft $1) S edH2C" p ( Pu
The above bound on suboptimality gap matches that of Theorem in [2] in terms of feature dimension and horizon length H. However, it has an additional exponentially large term e//A in the numerator and an additional p in the denominator. Both of them expected they stand for the unique characteristics of DRMDP with KL divergence defined uncertainty set. Similar terms have also appeared in previous literature tabular DRMDPs with KI. divergence defined (s, a) rrectanngll uncertainty set (Proposition 4.8 of (5]) and d-rectangular linear robust regularized MDPs with KI. divergence regularization (Theorem 5.1 of [42]). This reflects the hardness in learning robust policies for DRMDPs with KL divergence defined uncertainty sets. Next, we translate the suboptimality bound in Theorem 5.9 to the sample complexity bound.
Corollary 5.10. Under the same assumptions and setting as in Theorem 5.9, to learn an 6-optimal policy with probability at least G, we require the sample size K satisfying
K x( (HH (CT')R22212 )222122
p²e²
5.3 x2 -dviergee
For linear mixture DRMDPs with --divergeence defined uncertainty sets, we additionally introduce the following coverage assumption on the offline dataset.
Assumption 5.11 (Robust Partial Coverage: x2 di-deeergence). Define the worst case transition kernel
Ph: it and the sampling covariance matrix AQ as (5.1) and (5.3), respectively. For any P E P. denote problem dependent robust value covariance matrix
R (a; dr pert V. (oon h.prtit [[oVitir (sh, a (o an)) 2T
(5.5)
Then we assume there exists a positive constant C > 0, such that
x A (o;; d=¹ h.ptttt V
C²'H* H
sup sup sup a (o.0 H1 PEP zeRd
TTA
K
Theorem 5.12 (x2 diveergge)) Assume Assumptions 3. 5.1 and 5. hold. For S E (0. 1), there exists an absolute constant <> O, such that if we set X d in Algorithm I, then for x² -divergence uncertainty set, with probability at least - G, we have
SubOpt(#t ft $1) S cd(vpH + H2)0='H 112 log K d/
When p O() basically saying that the dynamics perturbation is negligible, the bound in Theorem 5. matches that for non-robust linear mixture MDP (Theorem I of [2]) in terms of d and H. This makes sense as when p + O, the linear mixture DRMDP degrades to standard linear mixture MDP. When p is large, our bound suggests that policy learning would be harder due to the complex geometry of the x2 divergence uncertainty set. This aligns well with findings for (s, a) rectanguul tabular DRMDP with x² divergence defined uncertainty set (see Table 2 in [41] for details).
Next, we translate the suboptimality bound in Theorem 5.12 to the sample complexity bound. Corollary 5.13. Under the same assumptions and setting as in Theorem 5.12, to learn an e-optimal policy with probability at least - Ç, we require the sample size K satisfying
K ō of (Vpll3 + 2 (0
2

---

Remark 5.14. The robust partial coverage assumptions Assumptions 5.1, 5.7 and 5.11 are specifically designed for linear mixture DRMDPs. They posses distinct features compared to robust partial coverage assumptions in literature, such as the robust partial coverage coefficient [40, 5] for (s, d)>rectangular tabular DRMDPs, and the robust partial coverage covariance matrix [5, 42] for d-rectangular linear robust (regularized) MDPs. Specifically, the formulation of the robust partial coverage assumption for linear Mixture DRMDPs varies according to the choice of uncertainty sets. According to the definition in (5.2), (5.4) and (5.5), different uncertainty set leads to different problem dependent robust value covariance matrix. In contrast, the robust partial coverage coefficient for tabular DRMDPs is simply defined in the form of visitation ratio, and the robust partial coverage covariance matrix for d-rectangular linear robust (regularized) MDPs is defined by the known feature mapping Φ Both share the same form across uncertainty sets defined by different divergences.
6 Discussion and conclusion
We proposed novel framework, termed the linear mixture DRMDP, for robust policy learning. Focusing on the offline reinforcement learning setting, we introduced a meta-algorithm and formalized the assumptions necessary to study mixture DRMDPs. Our work lays the theoretical foundation for future research in this direction and highlights several promising avenues for exploration.
guarantees under various instantiations of linear
First, online learning in linear mixture DRMDPs would be an important extension, particularly relevant in applications such robotic training with simulators. In the online setting, the agent must actively and efficiently collect data to balance the exploration-exploiatto trade-off, which necessitates the design of non-trivial strategies tailored to the linear mixture DRMDP framework.
Second, learning linear mixture DRMDPs assumes access prior knowledge, such as the structure of the linear mixture dynamics and the basis modes. Investigating when it is appropriate to model changes in dynamics using linear mixture uncertainty set snt how to construct meaningful basis modes in complex environments like MuloCo- would represent an interesting future step. Several open questions arise this context, including: How does misspecification of the basis modes affect the performance of robust policies? How do linear mixture DRMDPs compare to (s, a) rectanguul DRMDPs in terms of robustness and sample efficiency?
On the practical side, the proposed meta-algorithm in Algorithm 1 depends on the planning oracle defined in (4.3), rendering computationally intractable. To demonstrate the practical utility of the linear mixture DRMDP framework, we introduced two computationally tractable algorithms in Appendix based on an iterative estimation subroutine, and evaluated them on simple simulated environments in Appendix F. Experimental results showed that the learned policies were robust to environment perturbations, validating the effectiveness of the proposed framework.
Nevertheless, the reliance on an iterative estimation subroutine introduces a gap between the theo- retical analysis and the practical algorithms. While the primary focus of this work is to introduce the linear mixture DRMDP framework and establish its finite-sample guarantees in the offline RL setting, an important open question remains: Can we design algorithms that both statistically and computationally efficient? We leave this as an exciting direction for future research.
Acknowledgments
We would like to thank the anonymous reviewers for their helpful comments. ZL and PX was supported in part by the National Science Foundation (DMS-23231 12) and the Whitehead Scholars Program at the Duke University School of Medicine. The views and conclusions in this paper are those of the authors and should not be interpreted as representing any funding agency.
References
[1] Yasin Abbasi- Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in Neural Information Processing Systems, 24, 2011. 6
10

---

[2] Alex Ayoub, Zeyu Jia, Csaba Szepesvari, Mengdi Wang, and Lin Yang. Model-based rein- forcement learning with salue-targeted regression. In International Conference on Machine Learning, pages 464-474. PMLR, 2020. 2, 4, 6, 9,
[3] Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning, pages 51 1-520. PMLR, 2021. 3
[4] Isin M Balci and Efstathios Bakolas. Density steering of gaussian mixture models for discrete- time linear systems. In 2024 American Control Conference (ACC), pages 3935-3940. IEEE, 2024.7 22
[5] Jose Blanchet, Miao Lu, Tong Zhang, and Han Zhong. Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. Advances in Neural Information Processing Systems, 36, 2024. 2, 3, 8, 10
[6] Qi Cai, Zhuoran Yang. Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning, pages 1283-1294, PMLR, 2020.
[7] Longchao Da, Justin Turnau, Thirulogasankar Pranav Kutralingam, Alvaro Velasquez, Paulo Shakarian, and Hua Wei. A survey of sim-to-real methods in rl: Progress, prospects and challenges with foundation models. arXiv preprint arXiv:2502.1118,, 2025. 1,
[8] Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. An imitation from observation approach to transfer learning with dynamics mismatch. Advances in Neural Information Processing Systems, 33:3917-39299 2020. 3
[9] Maico HW Engelaar, Micha PP Swaanen, Mircea Lazar, and Sofie Haesaert. Stochastic mpe for finite gaussian mixture disturbances with guarantees. In 2025 European Control Conference (ECC), pages 1609-1615. IEEE, 2025. 22
[10] Benjamin Eysenbach, Shreyas Chaudhari, Swapnil Asawa, Sergey Levine, and Ruslan Salakhut- dinov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. In International Conference on Learning Representutions, 2021. 1,3
[11] Boris Vladimirovich Gnedenko and Igor Nikolaevich Kovalenko. Introduction to queueing theory. Birkhauser Boston Inc., 1989. 3
[12] Vineet Goyal and Julien Grand-Clement. Robust markov decision processes; Beyond rectangu- larity. Mathematics of Operations Research, (8(1):203-222 2023. 3, 4,5
[13] Jingwen Gu, Yiting He, Zhishuai Liu, and Pan Xu. Policy regularized distributionally robust markov decision processes with linear function approximation. arXiv preprint 250.::::5200.444. 2025.
[14] Yihong Guo, Yixuan Wang, Yuanyuan Shi, Pan Xu, and Anqi Liu. Off-dynamics reinforcement learning via domain adaptation and reward augmented imitation. In Advances in Neural Information Processing Systems, volume 37, pages 136326-136360, 2024. 3
[15] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Ad- vances in neural information processing systems, 31, 2018. 22
[16] Yiting He, Zhishuai Liu, Weixin Wang, and Pan Xu. Sample complexity of distributionally robust off-dynamics reinforcement learning with online interaction. In Forty-second Interna- tional Conference on Machine Learning, 2025. URL https: /tttps://openrevier/ wtw.ppperrrriiieeeettrrrnrn net/forum? id=pJaMDKqdSV. 2
[17] Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust optimization. Available at Optimization Online, 1(2):9, 2013. 30
[18] Garud Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257-280, 2005, 2, 4, 7

---

[19] Zeyu Jia, Lin Yang. Csaba Szepesvari, and Mengdi Wang. Model-based reinforcement learning with value-targeted regression. In Learning for Dynamics and Control, pages 666-686. PMLR, 2020. 2, 3, 4,31
[20] Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey,
The International Journal of Robotics Research, 32(11):1238 1274, 2013. 3
[21] Sylvain Koos, Jean-Baptin Baptiste Mouret, and Stéphane Doncieux. The transferability approach: Crossing the reality gap in evolutionary robotics. IEEE Transactions on Evolutionary Computa- tion, 17(1): 145, 2012. 1
[22] Long-Fei Li, Peng Zhao, and Zhi-Hua Zhou, Improved algorithm for adversarial linear mixture mdps with bandit feedback and unknown transition. In International Conference on Artificial Intelligence and Statistics, pages 3061-3069. PMLR, 2024. 2, 6, 7, 29, 30
[23] Mengmeng Li, Daniel Kuhn, and Tobias Sutter. Policy gradient algorithms for robust mdps with non-rectangular uncertainty sets. arXiv preprint arXiv:2300. 19004, 2023. 3
[24] Zhishuai Liu and Pan Xu. Distributionally robust off-dynamics reinforcement learning: Prov- able efficiency with linear function approximation. International Conference on Artificial Intelligence and Statistics, pages 2719-2727. PMLR, 2024. 1, 1,4,5, 4,5, 6, 32
[25] Zhishuai Liu and Pan Xu. Minimax optimal and computationally efficient algorithms for distributionally robust offline reinforcement learning. In Advances in Neural Information
Processing Systems, volume 37, pages 86602-86654, 2024. 2, 3,5
[26] Zhishuai Liu, Weixin Wang, and Pan Xu. Upper and lower bounds for distributionally robust off-dynamics reinforcement learning. arXiv preprint 20X552240,.2052,, 2024. 3
[27] Miao Lu, Han Zhong, Tong Zhang, and Jose Blanchet. Distributionally robust reinforcement learning with interactive data collection: Fundamental hardness and near-optimal algorithms. In The eighth- Annual Conference on Neural Information Processing Systems, 2024. 3
[28] Jiafei Lyu, Kang Xu, Jiacheng Xu, Jing-Wen Yang, Zongzhang Zhang, Chenjia Bai, Zongqing Lu, Xiu Li, et al. Odrl: A benchmark for off-dynamics reinforcement learning. Advances in Neural Information Prucessing Systems, 37:59859-5991 2024.
[29] Xiaoteng Ma, Zhipeng Liang. Li Xia, Jiheng Zhang, Jose Blanchet, Mingwen Liu, Qianchuan Zhao, and Zhengyuan Zhou. Distributionally robust offline reinforcement learning with linear function approximation. arXiv preprint arXiv:2209.0660 2022. 3,
[30] Geoffrey J McLachlan, Sharon X Lee, and Suren I Rathnayake. Finite mixture models. Annual review of statistics and its application, 6(1):355-378, 2019. 2
[31] John A Nelder and Roger Mead. A simplex method for function minimization. The computer journal, 7(4):308-313 1965.
[32] Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005. 2,5
[33] Arnab Nilim and Laurent Ghaoui. Robustness in markov decision problems with uncertain transition matrices. Advances in Neural Information Processing Systems, 16, 2003.
[34] Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv: 18 10.0.10 12282, 2018.
[35] Carl Rasmussen. The infinite gaussian mixture model. Advances in Neural Information Processing Systems, 12, 1999. 2
[36] Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663);
3. 2009. 2
[37] Steindór Sæmuundsonn, Katja Hofmann, and Mare Peter Deisenroth. Meta reinforcement learning with latent variable gaussian processes. arXiv preprint 81X1v:1803.07511, 2018. 22
12

---

[38] Jay K Satia and Roy E Lave Jr. Markovian decision processes with uncertain transition probabilities. Operations Research, 21(3):728-740, 1973. 2
[39] Andreas Sedlmeier, Michael Kölle, Robert Müller, Leo Baudrexel, and Claudia Manhoff-Poppie Quantifying multimodality in world models. arXiv preprint (rxxv:2112.072000 2021. 22
[40] Laixi Shi and Yuejie Chi. Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity. Journal of Machine Learning Research, 25(200):1-91, 2024. 10
[41] Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, Matthieu Geist, and Yuejie Chi. The curious price of distributional robustness in reinforcement learning with a generative model. Advances in Neural Information Processing Systems, 36, 2024. 3, 9,9,3. 30
[42] Cheng Tang, Zhishuai Liu, and Pan Xu. Robust offline reinforcement learning with linearly structured StS-divergence regularization. In Forty-second International Conference on Machine Learning, 2025. URL tatttps//oppeneeee //pp///vepeeeeeeeeeeeeeeeeeeeeee
3, 7,9, 10
[43] Masatoshi Uchara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference Learning Representations, 2021. 8, 30
[44] He Wang, Laixi Shi, and Yuejie Chi. Sample complexity of offline distributionally robust linear markov decision processes. In Reinforcement Learning Conference, 2024. 5
[45] Ruhan Wang, Yu Yang, Zhishuai Liu, Dongruo Zhou, and Pan Xu. Return augmented decision transformer for off-dynamics reinforcement learning, arXiv preprint arXiv:2410.23450, 2024.
1,3
[46] Xiaoyu Wen, Chenjia Bai, Kang Xu, Xudong Yu, Yang Zhang, Xuelong Li, and Zhen Wang. Contrastive representation for data filtering in cross-domain offline reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, pages 52720-52743,
2024. 3
[47] Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes.
Mathematics of Operations Research, 30(1):1533333 153311531 2013. 3,4,4
[48] Markus Wulfmeier, Ingmar Posner, and Pieter Abbeel. Mutual alignment transfer learning. In Conference on Robot Learning, pages 281 1-90. PMLR, 2017.
[49] Huan Xu and Shie Mannor. The coansttess--erffmmann tradeoff in markov decision processes. Advances in Neural Information Processing Systems, 19, 2006. 2,
[50] Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. Cross-domain policy adaptation via aalue-guided data filtering. Advances in Neural Information Processing Systems, 36:73395-73421, 2023. 3
[51] Mengdi Xu, Wenhao Ding. Jiacheng Zhu, Zuxin Liu, Baiming Chen, and Ding Zhao. Task- agnostic online reinforcement learning with an infinite mixture of gaussian processes. Advances in Neural Information Processing Systems, 33:6429-64400 64400 2020. 22
[52] Zaiyan Xu, Kishan Panaganti, and Dileep Kalathil. Improved sample complexity bounds for distributionally robust reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages
PMLR, 2023. 2,7
[53] Wenhao Yang. Liangyu Zhang, and Zhihua Zhang. Toward theoretical understandings of robust markov decision processes: Sample complexity and asymptotics. The Annals of Statistics, 50 00:3223-32433 2022. 3
[54] Tan Zhang. Kefang Zhang, Jiatao Lin, Wing- learning of obstacle avoidance for robotic manipulators in uncertain environments. IEEE Robotics and Automation Letters, 7(1):65-72, 2021.
Ting-Yue Yue Geoffrey Louie, and Hui Huang. Sim2real
[55] Weitong Zhang, Dongruo Zhou, and Quanquan Gu. Reward-free model-based reinforcement learning with linear function approximation. Advances Neural Information Processing Systems, 34:1582--599 2021. 6
13

---

[56] Canzhe Zhao, Ruofeng Yang. Baoxiang Wang, and Shuai Li. Learning adversarial linear mixture markov decision processes with bandit feedback and unknown transition. In The Eleventh International Conference on Learning Representations, 2023. 6
[57] Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE symposium series on computational intelligence (SSCI), pages 737-744. IEEE, 2020.
[58] Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory, pages 4532-4576. PMLR, 2021. 3, 6, 8,
[59] Dongruo Zhou, Jiafan He, and Quanquan Gu. Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning. pages 12793 12802. PMLR, 2021.
[60] Ruida Zhou, Tao Liu, Min Cheng, Dileep Kalathil, Panganamala Kumar, and Chao Tian. Natural actor-critic for robust reinforcement learning with function approximation. In Thirty-seventh Conference on Neural Information Processing Systems, 2023."
[61] Zhengqing Zhou, Zhengyuan Zhou, Qinxun Bai, Linhai Qiu, Jose Blanchet, and Peter Glynn. Finite-sample regret bound for distributionally robust offline tabular reinforcement learning. In International Conference on Artificial Intelligence and Statistics, pages 3331-3339. PMLR, 2021.
[62] Adil Zouitine, David Bertoin, Pierre Clavier, Matthieu Geist, and Emmanuel Rachelson. Time- constrained robust mdps. Advances in Neural Information Processing Systems, 37:35574-35611 2024.7
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. We propose novel framework for DRMDP in Section called the linear mixture DRMDP. A meta-algorithm is proposed for policy optimization under the linear mixture DRMDP and finite sample complexity results are provided in Section 4 and Section 5.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations are comprehensively discussed in Section 6. We leave them for future study.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide detailed discussion and justification of assumptions used in the main context and rigorous proof of theorems in the supplementary material.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details on experiment setup and implementation are provided in Appendix F, They are enough for reproduction of all experiment results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
All experiment results can be reproduced by the code in this link: htwps///oopppooocooo/eeeeaaeaaaa Guidelines:
Justification:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:/nnip)) cc/
Baalliicguidee//odeSuubbae wwtplbmmssiooooiccyy... for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //iiss... wwwpcccpppc/..ee..c...ccce.cetle ppwbli//guides....//eepp// httpc/gggideeeoeeuuuubiiiiiiiiii SCodeSubmissiond sow.bbiiissiooicoccccoooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Details on experiment setup and implementation are provided in Appendix More details can be found in the released code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[No]
Justification:
Not applicable to our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: All numerical experiments were conducted on a MacBook Pro with a 2.6 GHz 6-Core Intel CPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https //neurrris.... wwweerrps..cppuullppeeeeeeeeeeee htcc/pbblic/thhcsciieeeeee
Answer: [Yes]
Justification: We have checked that the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA
that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed. This work focuses on the theoretical side of robust RI., and methods this paper do not lead direct path to any negative applications.
18

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
The paper does not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //neruurpp.... wct//oooeeeeccc//22///1/111 for what should or should not be described.
21

---

A On practical motivation of the framework
Mixture models, in particular Gaussian Mixture Models (GMMs), are widely studied in fields of optimal control and RL. For example:
Model Predictive Control (MPC) Balci and Bakolas [4] study the density steering for discrete- time linear systems, a variant of optimal mass transport problem widely used in applications such controlling swarm of robots/drones to achieve a desired spatial distribution. They model the stochastic dynamical system using GMMs and derive its optimal control policy. Engelaar et al. [9] study stochastic MPC algorithm for linear systems subject to additive Gaussian mixture disturbances. They consider a vehicle control case study in which the vehicle must maintain its position on an ill-maintained road.
Meta RL Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from set of training tasks unseen, but related, tasks. Recent works on Meta RI model the dynamics of a new system as a mixture of expert systems, thus realizing knowledge transfer. In particular, Sæemundsson et al. [37], Xu et al. [51] model the dynamics of the expert systems using Gaussian process with latent embedding. And the dynamics of the new system constructed by mixture of Gaussian processes with the distribution over the latent embedding being the mixture weights, which should be learned from data. In experiments, their meta-learning models effectively generalizes to novel tasks under various environments such as Cart-pole& & Double-pendund swing-up -uinggup, HalfCheetah and
World Models for model-based RL Model-based Deep Reinforcement Learning (RI.) assumes the availability of model of an environment's underlying transition dynamics, which are stochastic in nature and oftentimes multimodal. Ha and Schmidhuber [15], Sedlmeier et al. [39] use Mixture- density Networks to construct World Models to capture the multimodality in dynamics. Then model-based RL methods are implemented based on the constructed World Models solve various tasks such as car racing and Inverse Sine Wave.
The basis modes in our work be the Gaussian components in GMMs, dynamics of expert systems, or mixture components in the Mixture-density Networks. These applications naturally give rise to distributionally robust MDP problems, for example 1) robust control of drones or vehicles is needed to hedge against environmental uncertainties and distributional perturbations: 2) given established expert systems, usually the mixture weights are estimated from limited data collected from the unseen new task, and thus of high uncertainty. Our framework could potentially enhance robustness of decision making in unseen tasks by modeling the weighting parameter uncertainty: 3) by modeling the uncertainty in the World Model (specifically, in the weights), our framework enables robust decision making in model-based RL.
B Proof in Section 3
In this section, we prove Lemma 3.4 and Lemma 3.5.
B.1 Proof of Lemma 3.4
Proof. To see this, the one hand, for any as S x A, assume (S ((s', s,a) a) 0 is the nominal kernel. For any O E (0 E 1 (0) s p). we have P(s'(s,a) (ss' T.. then
1 Σ
Div (P) (-1s,a)) =
P(s' - P
s'ES
1
Σ
8))) a T(O 8a)
2
e'S
1
Σ
Gls', s,a)
T 0 ---0 8")
2
E'S
(Tv(0 Sp.
22

---

where in the last line we used the fact that Eres fi(s', = 1 for all s, a. Thus, whenever P lies in the linear mixture uncertainty set, must also lie in the (s, a) )-rectangula uncertainty set with the same radius. Consequently, the linear mixture uncertainty set contained the (s, d)-rectangular uncertainty set.
B.2 Proof of Lemma 3.5
Proof. To see this, we consider the tabular setting where |S] and [A|] are finite, and a slight modi- fication on Assumption 3.1. In particular, let d |||||| S) and a(;;--)) S A x S + [d] be the function that maps the state-action-nexxtstaee tuple (s, a, 8 to its index in the space S x A x S. Letting
0i(5'1s, = =(0
if if otherwise.
and 00 P if a, a, = i. Then we have (s'ls, (ss'ss,a) (Ø(5'ss,)) 0%) To define the uncertainty set, fix any (s, E E x we define (s, a) E RISI as the segment of 0 corresponding to (s, a). For any & E Ad-1 we define (s, a;; E) as the vector of replacing the segment (s, a) in 60 by g. Then we define the uncertainty set ((s, a) = (5 E ((E) (s,a)) S p} and us,s a; 8 (s) E O(s, a)). Then by the construction of basis latent modes, the linear xl"
{) is exactly the standard (s, a) -eectttnnn uncertainty set. For simplicity, we focus on the linear mixture DRMDP defined in Assumption 3.1 in the main context.
mixture uncertainty set U (P))
Next, provide some cases where there exist distributions in the (s, .)-rcttang uncertainty set that are not in the linear mixture uncertainty set. Let's consider two simple examples.
Example B.1. Let's assume {or(s', a are identical. Then is trivially to check that the linear mixture uncertainty set contains only the nominal kernel since there is only one basis mode and the perturbation on the weighting parameter 0 does not result different probability distribution. While the (8, -reettnnnal uncertainty set defined based on the nominal distribution certainly contains more distributions other than the nominal one.
Apart from the above degenerated example, show another example as follows.
Example B.2. Let's define S
x2, 12, 2, 0 2" and
(.) = 0.76,, ()
(2(-) 0.16,, (1) + 0.20 ())
pd 0.46.,, ( ()) 0.25 (.).
Note that p0 (Φ, 0). Then for any p s 1 we have (P(P)) (0.70, + +..18)),,,() (0.10) 2. 0.702) 6zz + 0.28 (+), where 0, + 02 = and (109 0, + 09 021)/2 < Then for any
PE E PEUP(P) we have
Drv < 0.8p,
and the equation can be achieved by 0 = [1/2 + 1/2 p] Next we show that in the standard (s, a)- -reeettangulrrityy uncertainty set around po with radius 0.8p, there exist kernels such that they are not in the linear mixture uncertainty set defined above. In particular, for any 0 < S 0.8p, we have
Qs = -044-)oom + +00.4 -0.4--) ))0m + (0.2 - 00)
-0.4-
Since the weight of 6,g is not equal to 0.2, thus we can conclude that Qo is not in the Up (P*).
23

---

C
Suboptimality analysis
In this section, we prove Theorems 5.3, 5.9 and 5.12.
Proof. The following proof assumes that the event in Lemma 4.3 holds. Then by definition, we have
SubOpt(ft a, 81. p) 1,
1, (81)
Vi,roo
inf V (s1) inf
1.P (s1) V,, (s))
PEP
PEP
(8))6 inf V V (s1)
inf PEP
(()) (1,Po(11)
PEP
Vi,po((s) (81)
inf PEP -T. (s1)
1, (s1)
= sup (Vi.p( PEB
For any P E P and any step h [H], we denote that (sh, ( an) = Qhi (sh, an) - Qh, (sh. an)
inf (PP))
Ba ( h41, (5'] po (rEtiP(P)
inf
inf E, Ba V po (5)] (P)))
E By (ntiipp())) (r)))(Pg)
inf
inf E (VT (P.)
inf
Pa (VTti(p'''))))
Pr (rdu(())))
For term I, define
Ph arginf BywArHha (T v(s,a) Sxx.
A (.(())
Then we have
< E, Ey :t ((1sn.0))
ro (s'] - E d' (Hlon ond [V
E Eywp," ((le 4 p aa))))
For the term II, we denote by A (II) (Sh, an) for simplicity. Then we have
(srrss, an) + 11 E
Rewgg" " (lss (1r) (A (() + A (T (C.1)
Recursively applying (C.1) and the plugging in the definition of A (II) (sh, an), obtain that
E Bes (s1,,()1,
H
[ inf (1 (Vitt,(s)))
and h.rgt (PE))(P))
an
inf P.EU)((P))
(C.2)
Next, we study
inf E, E.r.Pal [VT] (.)))
inf P.EUP(PP) BewPh (VTii((s')]
under different kinds of divergences.
24

---

I. TV-divergence Le defined by the TV-divergence, we have
inf (PE))
E BernPaCllenen Vi [ inf PREU((PP)
(errna((leaan) [ (s))
inf (,0)) Bmdn oi
Vatiirr
(sh, an)
inf (O.))
(sh, an)
{ (BLORR os Wiille (# (sh. an)
=(
p a min (
Willle (sp. an) [alb
= sup ae|o,H
{ (Rewon (i Villle (on; aw) p(a- min (o (
(on, an))
sup a0(0.8]
{ (Bmoog Binow) (0) (sh: (sh.aa))) )n)
S sup aE(o,H]
Denote
{ o = sug (Ri-OR - Rewon) (0, (0 Vittiin (Nh, an) [.) oE(O,H]
then we have
inf (P)))(PP))
Pa (s))) inf (.E))(Ps) S (Erwog -Eioon) pf little (shs (on an)
= 00 On, (phitiirsnn an) oo
< 110% 0 0illan || 1 (sn, an) aa AR
(C.3)
Combining (C.2) and (C.3), we have
E Bas (-1o1) (81,
|108- - -Oallna // (sp: an) cjj A7
1=1
BA (Beenien |l| 1 (1ittii (Sn, and an loglaa))))
1/2
(C.4)
h=1
$ 8m/
Tr (Ecenon) sprerst [[ [[ (Sh, an) og
( an)] oj.
A {
H
= Σ
Bn v
Tr (A TV (af;
d. h. 10 (
h=1
11
T (a d
V )
<
Σ
BA
sup
x TTAYe
Tr (A
(C.5)
1=1
SER"
BN V
H Ct Ran
(C.6)
K
h=1
1
Σ
3N V
H2. CF dd
K
h=1
log(A (og(K/a K/dppa plpmmi))
(C.7)
VK
25

---

where (C.4) holds due to Jensen's 's inequality, (C.5) holds by Lemma D.3 and the fact that Ag, = An. (C.6) holds by Assumption 5.1, and (C.7) holds by the fact À d and bounding 3n as follows
2 H Ví
+ dooo (4+ 4 111/mmmn|
K K) )) - cVd
K
Bn
V +
log
Ad
Puin
Thus, we have
cdH2C* log( (/dPpmin()
SubOpt(ft, $1, p)
VK
We complete the proof of Theorem 5.3.
II. KL-divergence
inf (PE)
E B M (Vtiip(ss)) inf Bar BawPael [ (PE)
= inf ))Eu)((R) Er (0 Villi (sh an))
Ri [0, Vail'e (sk; ( an)
inf (o.))
{ -AAlog (RiwoR exp
Vhile
(sh, an)/ an)/ 20}
sup He(X.H/p)
di
sup xE(x.H/ol
Alog
Binon exp
Vitiee
(sh. an) /2}l) - 20}
variur
Einon cxp
Φi
(sh: and/
P (sh, /n)/2/ }
sup H/pt
Alog
E Ei exp
Denote
E, Ei-0, O exp
(sn, an) }
X arg sup AE(A.H/p|
A Alog
Viiife
(sh, an) /n)
En exp
0
Then we have
inf A. (P)
B Parnnaa)
inf [V (rE)
Eivo, expl { -0 Vi
(sh
an)/ {
< A log
Er exp
{ -
0i
Vitiie
(sh, an)/ N
VallLe (sn, (sn,an) an)/ /2
(Eiwon Bewor) exp
= Aí log
1+
Eiooy exp {
0 VitliP (Shs an)/ Xn
(Eion - (lwog) exp {
Vittii (Sh, and xh
0.
< 2/
Ener exp { di
Vhille (sh an)/ Xā
HeH/2
(Bi.o. - (Emog) exp { - dr VhiliP (sn,
( an) (nh) >
<
0.
H/2
<
| - -0%1l /loxp {
$ Vatiii (sh, and/ X1
(C.8)
0
Combining (C.2) and (C.8), we have
H// -1
Egy Eggoet A ((1,
26

---

H
(on an bd A.pn [100% -0%lla ||exp {
x Vu (S an3/ Mi Am
(Brennewwaaii it
exp Φ Viitier (sh, an)/ Ni bn>}1 311 (AA
(A,:1) 1/2
1/:
(C.9)
h=1
exp {
Φ Vinii (shr (sh an)/ / exp o (Kiiir and//// Ni} |
A
3N
Tr (Eisnan
H
Σ
(A de port )A
h=1
H
T A (Ats d
V 1)
B
sup DERd
x AZ
Tr (A%A7')
(C.10)
H
V
O (Ann(AA))
(C.11)
K
4=1
H
Σ
C .d
h=1
eHdC"
log( K ((dPPennn))
(C.12)
VK
where (C.9) holds by the Jensen's 's inequality, (C.10) holds by Lemma D.3 and the fact that A N An (C.11) holds by Assumption 5.7, and (C.12) holds by the fact A d and bounding 3r follows
2 (2 log Vi Ç
H
+ dloog (1+ 4[1/Pmin K )) S cvdlog Puinn
Bn x +
Xd
Thus, have
CdH e//2 log K/ddp/ppuule Priia()
SubOpt(ft, $1. p) S
VK.
We complete the proof of Theorem 5.9.
III. x² --dvvergence
inf (PR)
Ey Bgr [VTii inf P.E)
E (l (
inf (. Ewon 0 WhlL Wiillp(sn (sn, and]
Ene, or ( inf (
(sh, an)
sup HE[O.H] { (Er of
patiil (sh)
-V Varaos (lof (sh, an)) }
{ Enwo (0 yiatte (sh and V - Varion (1o) (sns and }
[Einor[o!
sup aE(o,
{ (BmoR Bmon) (0 Vatiii (sn VV Varnor (oohiiiie (sh, an))
< sup aE|o.
+ | p Varien (lo) (itheiie (sn, anna ((nhi((((),,a)))))
Denote
oj arg sup { (Bmog - Bmee)) [o) Vittii [0
- -V p (loyitiir (sh.
27

---

+ v Varsmen (1of (sh an)))
then we have
inf Ps (PP)) E. B [V inf E Ex (onnan (V (PN)
(0 aattle (sh. (shnaan)ass v Varswog (lo (sh (shr an)) ot)
(Emog - Biwon)
+ p Varewon (1o (ons (onnan)) and) aij)
110 0bllan -I (ons (sn) an) Iap
+ Varswan (1o [oohiiii (sn. and oz ) -p Var (1o, viiii (sh, a
-1 0illan |1(0, (oofitii ( (sh l (Rimog) [o Vhille (on, an) agj
vp Erwoy (6 Vaille (sn.
1/2
[o Vitii (st, an)) at
( + (Binon ( vatiie (sn, an) oa
Enrrs
110n - 0%11 An 11(0% Vitili (sn, and) ag an
(sh,
Villif
(ons an) [3, -Fewon (ot Villle (sn, an)
VPV
Bn 0i
(Eaoay de
Whille (ons an) 1..))² (Eiwon
vo
(on: and) la3
+ VPV
11
0illan || Vittie (sn, (sn.an)
IAy + +vp/
1 -0%llan
||[o
Vatii'
(sh, and
aj An
aj A,
11.
(C.13)
Vp 0H - 0gll,, ||(0 Viiie (sn, an)
(n)
(C.14)
IlIa
Combining (C.2) and (C.14), then we have
H
E Eur e1) (fi, 01)] S
[I. + 1I + +m,.-
an h.prt.
h=1
Note that, by the similar proof as that of the Case TV-divergence, we immediately have
11
E (n per [h + mml] < cf log(K) K dd dlpPm)))
+ cVpH5/2dC log( K K/d (dppann))
K
K
1=1
Next, we study
(en h,p [[n]]
28

---

H
E
as Jove h,per / 110, - -8%lAA,
-1(0! hil. (sh: and) a3 Ag
11
Bn (eenn Mar [ll(0!
- (3hx and
1
11 ci
1|2
1/2
(C.15)
1=1
H
Tr (n horee [1 Vallle (sh: an)
12
Vhille
2, an) (11
PP
Bmy
Pi
A
ooj
h=1
H1
vp
Bny
Tr (A (at;
d por. V (
h=1
x A (att d pusity V k111p)
Vp
BN
sup #ERd
Tr(A)4777
(C.16)
Ag
h=1
P
BI
Ha. C Rank(A))
(C.17)
h=1
H
VP
B
H4. -F
K
h=1
cypll"d log( K k/ddppu p(
(C.18)
VK
where (C.15) holds due to Jensen's inequality, (C.16) holds due to Lemma D.3 and the fact that A% < An. (C.17) holds due to Assumption 5.11, and and (C.18) holds by the fact À = d and bounding 3n, as follows
2
H + dloog (4+ 4[1/Pmin x)) S cvdlog
K
B V
2 log Ç VT
Ad
P
Thus, we have
SubOpt(z2,s,.)
o/pH&d log( K K/dPpmins)
cH2dO=' log(K (s((//) /d Pminn)
c/pH5/2dC log K /KKPPpinn
Pinin K el (VpH'3 H2
Pmin K
Pinin K
+H22)d** dCx log( K Bog(K/d2pun /ddppminn
VK
We complete the proof of Theorem 5.12.
D Auxiliary lemmas
Lemma D.1 (Lemma 1 of [221). Let (Fi) be filtration. Let (8 be an ----ull stochastic process such that 8, is Fe measurable one-hot vector. Furthermore, assume E|6, F(-1] = Pt and define 6 = Pt Sc. Let {x be a sequence of TR. xdd stochastic process such that x, is Fi measurable and || I 1, [N]. Let {2, be sequence of non-negative scalars. Define
N V1
N
+ AeTd,
St
C.jai.j
==
j=
i=1
Then, for any Ç E (0, 1), with probability at least 1 - S, we have for all
Vx
2d (V))
d/2
|| ly, 1 1
+
log
4
S
29

---

Lemma D.2 (Lemma 2 of (22]). Let Ç E (0. 1), then for any & E [K] and simultaneously for all [H] with probability at least 1- 8. it holds that
0 C where CN,A {110- - ORRAAllen S BB3 2)
with 8x = =
(2log( dlog(( (4 Aad )).
Lemma D.3 (Lemma 15 of [43]). Suppose A1, A2, A3 E RRdxd are semipositive definite matrices, then we have
Tr(A) o max (Ag A, A, Tr(A3A2).
where
T A1
mmax (Aa (A7 1/22111115
sup Z Rd
Aza
Lemma D.4. (Strong duality for TV [41, Lemma 1J). Given any probability measure μD over S, fixed uncertainty level the uncertainty set UP (p²)) = μ A(8), (rvllll)))))) S p}, and any function V S - 10, H), we obtain
inf Egn V())= af VV(Vnn Viass] {Esnno(V(s)) (V(s))a
max
weur (1O)
- p(a - (dd-min(V(d))aaa)
(D.1)
where [V (s)) a). Vimin = min, V8)) and Vmma max, V(s). Notably, the range of can be relaxed to [0,, H] without impacting the optimization.
Lemma D.5. (Strong duality for KL [17, Theorem]) Suppose f(x) has finite moment generating
function in some neighborhood around = O, then for any a > O and a nominal distribution po, we have
exp [(())
sup E Ex-plf(X)] inf AlogEx (PO)
X20
Lemma D.6. (Strong duality for x2 [41, Lemma 21) Consider any probability vector P EE A(S) any fixed uncertainty level o, and the uncertainty set U (P) == (P) For any vector V E R
obeying V 2 0, one has
max (xw aE(min, max, v(s) PEu*
inf
(Eanr(V(s)]o - vo Varp (VVo)
where Varp(V) = 1EaPV22( - (E (E (s))2
E Practical algorithms
Despite Algorithm 1 is provably efficient and enables robust policy learning, the construction of the robust policies in (4.3) relies an optimization oracle. Hence, Algorithm 1 is not computational tractable. In this section, we propose practical algorithms for numerical experiments.
We use value itcration to iteratively estimate the optimal robust Q-function in backward fashion and take the corresponding greedy policy the robust policy estimation. Specifically, for any step h [H]. given an estimated robust value function Vi S H). dive into the
one step robust Bellman operation on De Pht First, for any (s,a) E S x A, define & Ph (8, (8, =
Is (ss''s, V (s') )s'' One step robust Bellman operation on h+1 leads to
(s an( +
inf (A(1
BArNPa( Th
o,e a)) + inf / (ols')s, a), 0n) Vip (s))ds'
= =s(ssa) + inf e, e,
((5's, (5))sss
30

---

(8, 0n) V/ = mm(s,a) + inf Ei 0z (s)
(s) + inf
OrEOn
(E.1)
B,EO,
The infimum term in (E.1) can be solved by optimizing their duality. By the duality formulation in Lemma D.4, for TV-divergence defined O)., we have
[E 0: Vit (s. a)) p(a-- min o
Pet (s, [.)]
(s = ra(s,,) + max aE(U,H]
(s.
For KL-divergence defined Or by the duality formulation in Lemma D.5, we have A log Bi exp { - Vit+ (s, -Ap_.
a)(,,a)= = rs. sup He(o.H/p)
{
For x2 diverggee,, by the duality formulation in Lemma D.6, have
(%(,,a) = a)) max
{Er_
V (s, (s. [o.
-ell:-VVVoaaaaa Var
(s))))))
oE(0.H]
Finally, substituting the unknown 0% in above equations with an estimation On. We estimate Popk
in Es,o, 0z at1 (s, as follows.
Ponk En o h+l 0,
Pati (s. - -p(a- min 0 D/- (0/--1(6 (s, a)) {
TV: inf
(s,0) = max aE|O.H]
E (o
(E.2)
{ - Alog AlogEEno Brnd, exp { --0 Dh ( (s, - -22 (E.3)
KL: inf Eive, oi n+] (s,a) H/p)
sup
01 /
2. inf x E
max aE(0.11 E Ei 02 Ail1(10 (s,
Dark
W ( (os. (1 V V Varimon ((o
(8, (.)
(E.4)
where 0 an estimation 0-- In implementation, the optimization in (E.2) (E.4) can be approximately solved by the heuristic Nelder- method [31]. Finally, propose two approaches to estimate the unknown parameter 0% from the offline dataset.
transeeeion-aargeeedd regression Follow the cunsitionntaaretter regression proposed in (4.1), we get the estimation of 0% O,. Note that the parameter estimation procedure of the transition-targetee regression only depends on the transition information in the offline dataset. Thus, it can be conducted simultaneously for all stages h E [H] and obtain [θж]иени] all at once.
Value- targett regression As an alternative choice for parameter estimation, the value targeted regression an approach well studied in theory studies on standard RL [19, 2, 58]. To illustrate its application in robust RL, we also proposed an algorithm to learn robust policies based on the value
targeted regression for parameter [0 h=1 estimation. In particular, at stage with the estimated robust value function Vht we estimate 0% via the following ridge regression:
argmin Σ BERd
[( PRi R1 (sh (a..a)). o)), O) 1)) 2l1l1k = A
Σ
Ph o Phin at
Vi (s
1),
(E.5)
where Ae Ek=1 Pr (
AI. Note that the parameter estimation procedure of the malue-targeted regression is conducted iteratively with the value function estimation. For example, the parameter estimation stage h depends on the estimated value function at the next stage h + 1, which in turns depends on the parameter estimation at stage + 1. This is different from the parameter estimation procedure of the transition-targeted regression.
Algorithms based on the transition targeted regression (termed as the DRTTR) and the value targeted regression (termed as the DRYTR) are presented in Algorithm 2.
31

---

Algorithm 2 Distributionally Robust Transition (Value) Targeted Regression (DRTTR and DRVTR)
Require: Regularization parameter X, offline dataset D, robust level p. initialization VH+1(+) = 0.
I: for h H, ...... do
2: For DRTTR, estimate 0% by (4.1); For DRVTR, estimate 0₂ by (E.5).
3: Estimate (:))) using (E.2), (E.3) and (E.4) for TV-, KL- and x²- divergences, respectively.
4: (-)(.) +- argmax (f(,,a) a), (-) < max.E (3,a)
5:
end for
Simulation study
We now conduct experiments in a simulated environment to show the robustness of policies learned by our algorithms.
F1 Experiment setup
We adapt the simulated linear MDP instance proposed by [24] linear mixture DRMDP. Note that we have access to difference information in those settings. In particular, for linear DRMDPs, an agent has access the feature mapping & (s, a) + Rd. While for linear mixture DRMDPs, an agent has access basis modes Φ (s, a) A(S) Thus, though the DRMDP environments actually remain the same, our implemented algorithms are essentially different from that in [24] due to the different prior information available.
T.4
JA
p(1
(E. a))
(5,a))
P(1 5 ())
1 5 - (R.0)
o5 ())
5 (3.a)
I1
23
T2
T3
p][1
p)(1
(E.
DX1 (E.a)
(E.a) a)
6 (E
(1 9)(5 (E.
5 (E.0)
8 ()
+ (E.0)
35
JTSS
(a) The source environment.
(b) The target environment.
Figure 2: The source and the target linear MDP environments. The value on each arrow represents the transition probability. For the source MDP, there five states and three steps, with the initial state being F1, the fail state being I4, and ø; being an absorbing state with reward 1. The target MDP on the right obtained by perturbing the transition probability the first step of the source MDP, with others remaining the same.
We present the source domain linear mixture MDP Figure 2a. Specifically, there are five states S = {x1, T2, T3, 34, ). 16 actions A {-1, and H 3 stages. The basis modes ((-1..)) are
32

---

probability measures defined on S/ry as follows:
T (-11, a
8 (E.0)
8+ (E.a)
a() -
22, Øs 11,
=
1 5 - (5
0
8+ (
0
(g.0)
8 (.
T (-12,,22 a)
T
1 8 (g.a)
8 (5.
66-122,, a)
(22-1122,4
-
0
- 6
(5,a)
6 (E,
dds (22,
0
1 8 (5,a)
6 (E,0)
Thi (-124,
0
0
1
07
((1121,a) =
Φ2 11,
=
0
Lds( [1.
0
0
1
0
Φ (s)
0
0
0
17
(0 a)
2 ( [s.
=
(F.I)
ses [r5,a)
0
where and are hyperparameters. Apparently, the dimension d equals three in this instance. is
trivial to verify that the source transition kernels can be represented as Ph (-l,-) = <(-1.-) 0% The reward functions Tn (7,8,0) a) are designed as
(x (b(5,0 a) TOVTTA Vh, v(h, 8, [H] x x
where
(x1, a) = (1 - 8 (5, (x22 a) (0, -
(233 a) 10,,118-(E5.0) (E.0), (c,
(144 a) = (0,0,1,0)*,
(5 a) = (0,0,0,1)*,
and
1 = (0.0,0.0) 122 = (0, 0, 1) and V3 = (0.0,0,1)
We construct target domains by perturbing the weighting parameter 0 in the first stage of the source MDP. Specifically, we 01 (q. q. 0) where q is factor that controls the perturbation level. The perturbed target environment is shown in Figure 2b.
Implementation For the offline dataset collection, simply use the random policy that chooses ac- tions uniformly at random at any (s, a, h) E S Ax [H] as the behavior policy T² to collect the offline dataset D. The offline dataset contains 500 trajectories collected by the behavior policy n from the source environment po_ For the setting details of the source environment po, we set hyperparameters the defining the source and target environments as { = (1/I|El) /ll 1 E/| ( 0.1, 8 0.4, and q E [0, 1]. We implement Algorithm 2 with TV. KL and X divergences on the collected offline dataset D, and denote them as DRTTR-TV, DRITR-KL DRTTR-chi2, DRVTR-TV, DRVTR-KI and DRVTR-chi2, respectively. Denoting the robust lev- els the TV, KL and x2 uncertainty set as PIV. Px2. we consider two sets of robust levels: PKL, Px²) {00.35, 5, 10), (0.7, 10, 20)). We compare DRTTR and DRVTR with the non- robust algorithms, dubbed as the TTR and VTR respectively, which basically set the robust level p 0 in DRTTR and DRVTR. To show the robustness of the learned policies by DRTTR and DRVTR, we test the learned policies on various target environments with different levels of perturbation. We the 'average reward', which is defined the averaged cumulative reward among 100 episodes, as a criterion to evaluate performances robust policies under different target environments. All experiment results are based on 10 replications, and were conducted on a MacBook Pro with 2.6 GHz 6-Core Intel CPU.
|| 0.4.
(PIv,PKL:
Experiment results
Simulation results Algorithm 2 are shown in Figure 3. As the perturbation level increases, the performances of policies learned by the non-robust algorithms TTR and VTR drop drastically when the magnitude of perturbation increases. When the perturbation is large, all robust policies outperform the non-robust policy. This illustrates the robustness of our proposed algorithms. We also conduct additional ablation studies to test the performances of Algorithm 2 in various environments.
33

---

TIR DRTTE 1N,0N2
TIL BRTTR,
VTR BRVTR
VTR
BRVTR.
DRVTR
DRVTR
DRV
Paras
(a) (0.4,0.4, 0.4,0.55 5, 10) (b) (0.4, 0.4,0.44,0011 0.7, 10, 20) (c) (0.4, 5, 10) (d) (0.4, 0.4. 10, 20)
Figure 3: Simulation results of Algorithm 2. Policies are learned from the nominal environment featuring 0 = (0.1, 0.8, 0.1). Numbers in parenthesis represent (ò, || PTV, PKL, Px*) respectively, The a-axis represents the perturbation level corresponding different target environments. PTV, PKL and 2x² are the input uncertainty levels for our DRTTR algorithm.
Ablation study
We set the weighting parameters in the source environment as
0. = 0 (0, - p.p)"
(F2)
where p= 0.1 is the hyperparameter that controls the mixture weights. We set the hyperparameter (1/I|E|Iı, 1/I|Ella, 1/1 I and consider different choices of [lll E {0.3, 0.4). We consider difference levels of hyperparameter and x² divergence, well as DRVTR with TV, KL and x² divergence on the collected offline dataset D, and denote them as DRTTR-TV, DRTTR-KL DRTTR-chi2, DRVTR-TV, DRVTR-KL and DRV2R-chi2, respectively. Denoting the robust levels of the TV, KL and x2 uncertainty set as Prv. PKL Py2, we consider two sets of robust levels: (pIv. PKL: Pv²) E {(0.35, 5, 10), (0.7, 10, 20)}- We test the learned polices in testing environments with q [0, 1].
{0.3, 0.4]. We implement DRTTR with TV, KL
Experiment results are shown Figure 4 and Figure 5. Moreover, we also consider the source domain with the weighting parameter at the first stage 0 (p, 1 2p, p)) while all other parameters remain the same. Experiment results are shown in Figure and Figure We can conclude that in most cases Algorithm 2 can learn robust policies compared to their non-robust counterparts. In particular, when the perturbation becomes larger, all robust policies outperform non-robust policies terms of the cumulative reward.
As for the choice of uncertainty set, it requires certain prior knowledge of the nominal dynamics problem in this field that limited theoretical
and distribution shift. This actually is a guarantees are available to guide the choice of uncertainty sets. practice, one can learn policies corresponding to different divergence defined uncertainty sets and then compare their performances testing environments.
TTA DRTTRTTV DATTR, KL
DRTTR, TV DETTR Ki
DR DRTTR, K:
DATV DATTR KL
CETTR, ch12
DRTTA, ch2
DATTR, ch2
CATTR chi
Pattraaoon
Penrtalion
Peturtation
(a) (0.4, 0.4, 0.35, 5, 10)
(b) (0.4, 0.35, 5, 10)
(c) (0.4, 0.7, 10, 20)
(d) (0.4, 0.3, 0.7, 10, 20)
TTR
mwand
Aestapoo cowrdd
DXTILIT TV
roard
Avsrage Rwardd
ONTIUIV
DATIU KL DATTR dN2
DMTILS DATTR
DV DR 1R,
DRTTR.TV BRTTR DRTTR
returaaion
CA
88
(e) (0.3, 0.3, 0.35, 5, 10)
( 0.2 0.35, 5, 10) (g) (0.3, 0.3, 0.7. 20) (h) 0.2, 10, 20)
Figure 4: Simulation results of DRTTR under different source domains. Policies are learned from the nominal environment featuring 0 (0, 0.9. 0.1). Numbers in parenthesis represent (6, [IEl PKl. Px=), respectively. The r-axis represents the perturbation level corresponding to different target environments. Prv: PKt and p,= are the input uncertainty levels for our DRTTR algorithm.
34

---

DRVTR
DRVTR DRVTR chi
DRVTR
DRVT
DRVTR
DRVTA
Feturration
Pernrtation
Permrtation
Pertrtadion
(0.4, 0.4, 0.35, 5, 10)
(b) (0.4, 0.35, 5, 10)
(0.4.0.4.0. 0.7, 10, 20)
(d) (0.4, 0.3, 0.7 10, 20)
DAVIR DRVTA 2h22
DAVTR DAVTR,02
VTB
(e) 0.3, 5, 10)
(1) (0.3, 0.2, 0.35, 5, 10) (g) 0.3,0.3,0013 10, (h) 0.2.0.2,...7. 10, 20)
Figure 5: Simulation results of DRVTR under different source domains. Policies learned from the nominal environment featuring 0 = 0.9, 0.1). The a-axis represents the perturbation level corresponding to different target environments. PTV, PKL and px2 the input uncertainty levels for our DRVTR algorithm.
(0.4, 0.4, 0.35, 5, 10)
(0.4, 0.3, 0.35, 5, 10)
(0.4.0.4, 0.7, 10, 20)
(d) (0.4, 0.7 10, 20)
DXTILIV
TIR 1h,
ANOUE
TTA BRTTR, FUKL
TEA
Natur sulton
(e) (0.3, 0.3, 0.35, 5, 10) (1 (0.3, 0.2, 0.35, 5, 10)
(g) (0.3,0.3,0.77 10, 20) (h) (0.3, 10, 20)
Figure Simulation results of DRTTR under different source domains. Policies are learned from the nominal environment featuring 0 = (0.1.0.8, 0.1). Numbers in parenthesis represent
(8, || PTV, PKL. Px²), respectively. The r-axis represents the perturbation level corresponding to different target environments. Prv. PKL and 2x² are the input uncertainty levels for our DRTTR algorithm.
35

---

12
(0.4, 0.4, 0.35, 10)
(b) (0.4, 0.3, 0.35, 5, 10) (c) (0.4, 0.4, 0.7, 10, 20)
(0.4, 0.3, 0.7, 0.3, 10, 20)
VTR
VTR
DRVVIL
(c) (0.3, 0.3,0.5,, 5, 10) (f) (0.3, 0.2, 0.35, 5, 10) (g) (0.3,0.3,0.3 10, (h) (0.3, 0.2.0.2 10, 20)
Figure 7: Simulation results of DRVTR under different source domains. Policies learned from the nominal environment featuring 0, = (0.1, 0.1). Numbers in parenthesis represent (6, |I PTV, PKL. Px²), respectively. The æ-axis represents the perturbation level corresponding different target environments. PTV, PKL and px* are the input uncertainty levels for our DRVTR algorithm.
36


Reviews:
[{"review_id": "A4fs5BxBNz", "reviewer": "Reviewer_8wNc", "summary": "This paper introduces a novel framework called linear mixture DRMDPs to tackle the off-dynamics reinforcement learning problem, where an agent must learn a policy in a source environment but deploy it in a target environment with different dynamics. This work assumes that both the source and target transition kernels share a linear mixture structure and proposes a meta-algorithm for offline robust policy learning in this setting using transition-targeted ridge regression and a double pessimism principle. The authors derive finite-sample guarantees for learning an approximately optimal robust policy under three different f-divergence-based uncertainty sets.", "questions": "I have a few follow-up questions that I understand may be beyond the intended scope of this work. However, I would greatly appreciate it if the authors could provide any insights or clarifications in their response:\n1. How sensitive is the framework to mis-specified basis modes in practice?\n2. Would it be possible to extend the framework to linear MDPs and low-rank MDPs?\n3. Would it be possible to extend the framework to online settings with exploration guarantees?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper introduces a new variant of DRMDPs that leverages linear mixture structures in the transition dynamics. This is a compelling and practically motivated idea that brings additional structure to the uncertainty set, going beyond conventional (s,a)- or d-rectangular formulations.\n2. The proposed framework is rigorously analyzed under three widely used f-divergences. The authors derive finite-sample suboptimality bounds, showing the statistical learnability of robust policies under the linear mixture setting.\n3. The linear mixture uncertainty sets are shown, both theoretically and illustratively, to be strictly smaller than standard rectangular sets in several scenarios. This leads to less conservative policies and tighter robustness guarantees.\n4. The proposed algorithm is modular and adaptable, capable of working with different divergence metrics and scalable with known basis modes. The adoption of transition-targeted regression is also well-motivated given the limitations of value-targeted methods in this context.\n\n**Weaknesses:**\n\n1. The meta-algorithm depends on a planning oracle that optimizes over a doubly pessimistic objective involving both parameter and transition uncertainty. As acknowledged by the authors, this oracle is computationally intractable, and no approximation strategy is integrated into the theoretical analysis. This creates a gap between theory and practice.\n2. While the specific combination of linear mixture dynamics and distributional robustness is new, many of the underlying components—such as f-divergence uncertainty sets, ridge regression for parameter estimation, and pessimistic value estimation—have been studied extensively. The novelty lies in integrating these ideas, but the conceptual leap is incremental compared to prior work on d-rectangular or linear DRMDPs.\n3. Some theoretical sections (e.g., the coverage assumptions and concentration analysis in Section 4) are dense and could benefit from clearer intuition or visual aids. This may hinder accessibility to readers unfamiliar with the robust RL literature.", "was_revised": true, "final_justification": "I appreciate the authors' detailed responses to my questions. I will maintain my rating.", "rebuttal": "We thank the reviewer for their valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all of the reviewer's points.\n\n---\n> **Q1:**  The theoretical section is dense and could benefit from clearer intuition or visual aids.\n\n\n**A1:** Thank you for the suggestion. The current layout of section 4 is due to the space limit. Given additional space in the camera ready, we will separate section 4 into two sections, Algorithm Design and Theoretical Guarantees, and expand our current discussion and interpretation on our analysis and assumptions.\n\n\n---\n> **Q2:** How sensitive is the framework to mis-specified basis modes in practice?\n\n\n**A2:** From a theoretical point of view, mis-specification in basis modes can cause error and result in suboptimal performance, then there would be no guarantee for the algorithm to learn the optimal robust policy. In practice, the basis modes are typically constructed from data, such as the Gaussian components in Gaussian Mixture Model [1,2], or dynamics of expert systems in Meta-RL [3,4], thus approximation error is inevitable. Nevertheless, these prior works have shown certain empirical success in using mixture models to approximate the complex true dynamics and performances actually vary from tasks to tasks. Another empirical evidence is from the recent work [5] on d-rectangular linear DRMDPs. They use linear MDP to model nonlinear dynamics. Even with large mis-specification errors in the feature mapping, their algorithm still achieves certain robustness performance. Based on these evidences, we conjecture that sensitivity of our framework with respect to the misspecification in basis modes should depend on specific tasks. But even with certain approximation errors, we can still expect certain robustness performances in practice. It serves an interesting future research direction to include the uncertainty in basis modes into consideration.\n\n\n---\n> **Q3:** Extend the framework to linear MDPs and low-rank MDPs.\n\n**A3:** We would like to note that distributionally robust MDPs have been separately studied in prior works under the framework of linear MDPs [5] and low rank MDPs [6]. They study different types of uncertainty sets constructed based on the posed structures of transition models, propose corresponding algorithms and provide theoretical guarantees. In our work, we particularly focus on a mixture structure of the dynamics and modeling uncertainty in the mixture weights. Thus, our work is orthogonal to previous ones on linear MDPs and low-rank MDPs.\n\n\n---\n> **Q4:** Would it be possible to extend the framework to online settings with exploration guarantees?\n\n**A4:** Thanks for the great question. Yes, our framework can potentially be extended to online settings where the agent has to actively interact with the nominal environment to collect data. Challenges lie in how to design algorithms capable of reasoning about the exploration and address the exploration-exploitation trade off in a data efficient manner. A potential solution would be leveraging the principle of optimism in the face of uncertainty [7]. For example, at each episode $k$, we can maintain a confidence set $\\hat{\\mathcal{P}}^k$ based on the weighting parameter $\\theta$ estimation and update it when new trajectories are collected. Then we define the value estimator $J^k(\\pi)$ as the largest robust value function with nominal kernel included in the confidence set: $J^k(\\pi) = \\sup_{P\\in \\hat{\\mathcal{P}}^k}\\inf_{\\tilde{P}\\in \\mathcal{U}_h^{\\rho}(P)}V_{\\tilde{P}}^{\\pi}(s_1)$, instead of the double pessimism estimator in our work. Then we use the optimal policy wrt $J^k(\\pi)$ as the execution policy at the current episode to collect data. Another potential challenge would be quantifying the difficulty of exploration in online DRMDPs. There is a line of work studying this problem and several interesting phenomenons have been identified that are unique to the online setting. We leave a rigorous theoretical development for future research.\n\n---\n**References:**\n\n[1] Balci et. al., 2024, Density steering of gaussian mixture models for discrete-time linear systems\n\n[2] Engelaar et. al., 2024, Stochastic MPC for Finite Gaussian Mixture Disturbances with Guarantees\n\n[3] Sæmundsson et. al., 2018, Meta reinforcement learning with latent variable gaussian processes\n\n[4] Xu et. al., 2020, Task-agnostic online reinforcement learning with an infinite mixture of gaussian processes\n\n[5] Liu et. al., 2024, Distributionally robust off-dynamics reinforcement learning: Provable efficiency with linear function approximation\n\n[6] Hu et. al., 2024, Efficient Duple Perturbation Robustness in Low-rank MDPs\n\n[7] Azar et. al., 2017, Minimax Regret Bounds for Reinforcement Learning\n\n[8] He et. al., 2025, Sample Complexity of Distributionally Robust Off-Dynamics Reinforcement Learning with Online Interaction\n\n---\nWe hope we have addressed all of your questions/concerns. If you have any further questions, we would be more than happy to answer them and if you don’t, would you kindly consider increasing your score?"}, {"review_id": "2ouyP6I2gO", "reviewer": "Reviewer_i9w6", "summary": "This paper introduces a framework for distributionally robust reinforcement learning, namely the linear mixture DRMDP, where the uncertainty in dynamics is modeled as perturbations to the mixture weights $\\theta$ in a known set of basis transition kernels $\\phi$. The authors propose a meta-algorithm that first estimates the nominal model through regression, then applies a doubly pessimistic policy optimization procedure. They analyze the sample complexity of this approach under total variation, KL, and $\\chi^2$ divergence-based uncertainty sets, providing suboptimality bounds in the offline RL setting.", "questions": "Are there any practical examples that naturally fit the linear mixture structure assumed in this paper? In many real-world applications where model dynamics are uncertain and robustness is desired, it often seems more appropriate to model uncertainty in the features \n$\\phi$, rather than in the mixture weights $\\theta$.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well-organized and clearly written, which makes it easy to follow the technical developments. Robustness in reinforcement learning—especially in the offline and off-dynamics setting—is a meaningful and timely topic, and the proposed linear mixture DRMDP framework offers an interesting perspective that incorporates structural assumptions into the uncertainty set design. Although I have not verified all the proofs in detail, the analysis appears technically sound.\n\nA key concern I have is regarding the practical motivation for the linear mixture distributionally robust setting. While the theoretical framework is cleanly laid out, the paper does not provide any concrete or compelling real-world example where such a structure naturally arises. The simulated environments used in Appendix F appear toy-like, which makes it difficult to judge whether the linear mixture structure meaningfully captures uncertainty in practical applications, or if it was primarily chosen for analytical convenience.\n\nAdditionally, the algorithm itself appears relatively straightforward: first estimate a nominal model via regression, then apply a \"double pessimism\" policy optimization using that estimate. The learning and planning phases are essentially decoupled, and the optimization uses known techniques adapted to this specific uncertainty set. While the theoretical guarantees are solid, the technical depth of the algorithmic development seems limited, and the analysis benefits from the structure already assumed.", "was_revised": true, "final_justification": "The rebuttal clarifies my major doubts and concerns. However, I still think that the practical contribution seems limited and the authors makes no efforts in terms of validating there approach even on some simple practical examples.", "rebuttal": "We thank the reviewer for their valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all of the reviewer's points.\n\n---\n> **Q1:** Practical motivation for the linear mixture distributionally robust setting.\n\n**A1:** Mixture models, in particular Gaussian Mixture Models (GMMs), are widely studied in fields of optimal control and RL. For example:\n\n**Model Predictive Control (MPC):** [1] studies the density steering for discrete-time linear systems, a variant of optimal mass transport problem widely used in applications such as controlling a swarm of robots/drones to achieve a desired spatial distribution. They model the stochastic dynamical system using GMMs and derive its optimal control policy. [2] studies a stochastic MPC algorithm for linear systems subject to additive Gaussian mixture disturbances. They consider a vehicle control case study in which the vehicle must maintain its position on an ill-maintained road. \n\n**Meta RL:** Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from a set of training tasks to unseen, but related, tasks. Recent works on Meta RL model the dynamics of a new system as a mixture of expert systems, thus realizing knowledge transfer. In particular, [3,4] model the dynamics of the expert systems using Gaussian process with latent embedding. And the dynamics of the new system is constructed by a mixture of Gaussian processes with the distribution over the latent embedding being the mixture weights, which should be learned from data. In experiments, their meta-learning models effectively generalizes to novel tasks under various environments such as Cart-pole & Double-pendulum swing-up, HalfCheetah and Highway-Intersection. \n\n**World Models for model-based RL:** Model-based Deep Reinforcement Learning (RL) assumes the availability of a model of an environment’s underlying transition dynamics, which are stochastic in nature and oftentimes multimodal. [5,6] use Mixture-density Networks to construct World Models to capture the multimodality in dynamics. Then model-based RL methods are implemented based on the constructed World Models to solve various tasks such as car racing and Inverse Sine Wave. \n\n\nThe basis modes in our work can be the Gaussian components in GMMs, dynamics of expert systems, or mixture components in the Mixture-density Networks. These applications naturally give rise to distributionally robust MDP problems, for example 1) robust control of drones or vehicles is needed to hedge against environmental uncertainties and distributional perturbations; 2) given established expert systems, usually the mixture weights are estimated from limited data collected from the unseen new task, and thus of high uncertainty. Our framework could potentially enhance robustness of decision making in unseen tasks by modeling the weighting parameter uncertainty; 3) by modeling the uncertainty in the World Model (specifically, in the weights), our framework enables robust decision making in model-based RL.\n\n\n---\n> **Q2:** In many real-world applications where model dynamics are uncertain and robustness is desired, it often seems more appropriate to model uncertainty in the features $\\phi$, rather than in the mixture weights $\\theta$.\n\n**A2:** Thank you for raising this insightful point. Take the Meta-RL application in our response to Q1 for an example, the expert systems are constructed from huge amounts of historical data and thus of low uncertainty. Once constructed, they can be deemed as fixed. To estimate the dynamics of the new tasks, we have to estimate the mixture weights from data collected from the new tasks, which is usually of limit size and cause large uncertainty in the mixture weights estimation. This motivates us to model the uncertainty in the mixture weights $\\theta$ to enable robust decision making in the new task, which turns out to be natural and of great importance.\n\nMoreover, we agree that accounting for the uncertainty in the feature $\\phi$ is also important for achieving robust decision-making. However, addressing it requires careful and non-trivial treatment in both algorithm design and theoretical analysis—particularly due to its interaction with uncertainty in the weighting parameters. A thorough investigation of this issue is beyond the scope of our current work, and we leave it as an important direction for future research.\n\n---\n> **Q3:** Algorithm design: Decoupling of the learning and planning phase\n\n**A3:** Thank you for this interesting question. We would like to provide more insights on our choice of algorithm design. We highlight that the decoupling of the learning and planning phase is the key to make theoretical guarantees possible. In particular, the analysis of the meta-algorithm only needs to control one-source of estimation error, which arises from the transition-targeted estimation of $\\theta_h$. Moreover, the transition-targeted estimation simply uses $\\phi$ as the feature vector. These two attributes are essential to allow Assumptions 3.4, 3.10 and 3.14 to neatly control the coverage of the offline dataset. And our analysis also leverages recently established concentration lemma for dependent error structure as we discussed in Remark 4.2.\n\nFor algorithms that embed learning with learning step-wisely, like the value-targeted regression algorithm widely studied in standard linear mixture MDP literature [7], they involve two layers of estimation error, i.e., the estimation error in $\\theta$ and the estimation error in $\\hat{V}$. During the preparation of this work, we found that the consideration of dynamics shift in DRMDP induces essential hardness to control these two sources of estimation error, unlike in standard MDPs. It is still unclear how to quantify/control the data coverage of the offline dataset with this type of algorithm. We would like to leave this problem for future study.\n\n\n---\n**References:**\n\n[1] Balci et. al., 2024, Density steering of gaussian mixture models for discrete-time linear systems\n\n[2] Engelaar et. al., 2024, Stochastic MPC for Finite Gaussian Mixture Disturbances with Guarantees\n\n[3] Sæmundsson et. al., 2018, Meta reinforcement learning with latent variable gaussian processes\n\n[4] Xu et. al., 2020, Task-agnostic online reinforcement learning with an infinite mixture of gaussian processes\n\n[5] Sedlmeier et. al., 2021, Quantifying multimodality in world models\n\n[6] Ha et. al., 2018, Recurrent world models facilitate policy evolution\n\n[7] Jia et. al., 2020, Model-based reinforcement learning with value-targeted regression.\n\n\n---\nWe hope we have addressed all of your questions/concerns. If you have any further questions, we would be more than happy to answer them and if you don’t, would you kindly consider increasing your score?"}, {"review_id": "84qFoXdJ12", "reviewer": "Reviewer_sAJt", "summary": "The authors tackle the problem of solving Markov decision process problems with uncertainty in the transition matrices. \nThey introduce the notion of distributionally robust linear mixture MDPs, where the transition matrices are assumed to be expressed as linear combinations of known \"basis latent modes\". The weights used for this combination are assumed to lie in an uncertainty set.\nThey consider various f-divergence-based ambiguity sets and develop a meta-algorithm to solve the problem. \nThey provide theoretical guarantees on the suboptimality of the policy obtained using this algorithm. \nFinally, in the appendix, they provide some numerical results demonstrating their approach and the benefit of using a distributionally robust policy under increasing noise.", "questions": "Questions\n1. Can you discuss the dependence of suboptimality on the size of the uncertainty set?\n3. Can you discuss the dependence of your models on the number of data trajectories available for training? DRO approaches are expected to be particularly useful if limited data is available. Does that appear true in your setting? \n4. Could you discuss the impact of the robustness approach on scalability, especially compared to the standard (s, a) and d-uncertainty sets? \n5. Can the authors also discuss a bit about how difficult or easy it is to obtain good estimates of the latent modes, which are combined to obtain the transition matrices?", "limitations": "Limitations\nThe work is primarily theoretical, so there is not much to address in terms of negative societal impact. \nI would like it if the authors discuss the potential limitations of their choice to use mixture models. What kinds of worst cases can they address or not, and so on.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The idea behind the paper makes sense and is somewhat original. \n2. The authors provide performance guarantees for their estimated solution\n3. They also theoretically compare their uncertainty models to other uncertainty sets.\n\nWeaknesses\n1. The notation presented in the paper is quite challenging to read. \n2. The example provided in the appendix involves very few states. \n3. Lack of numerical comparison to standard (s, a) or d-uncertainty sets, which makes it difficult to evaluate the presented uncertainty models.", "was_revised": true, "final_justification": "The authors have responded to my questions in detail. As I had already recommended the acceptance of the paper, I will maintain my score.", "rebuttal": "We thank the reviewer for their valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all of the reviewer's points.\n\n---\n> **Q1:** Dependence of suboptimality on the size of the uncertainty set.\n\n**A1:** Thank you for your question. For the case of TV divergence, the suboptimality bound does not depend on the radius $\\rho$, as shown in Theorem 4.6 of our submission. For the case of KL divergence, the suboptimality bound is inversely proportional to the radius $\\rho$ as shown in Theorem 4.11 of our submission. This dependence aligns with prior works on DRMDP with KL divergence defined uncertainty set (see Theorem 3 in [1] and Corollary 4.1 in [2]), arising from the geometry of KL divergence in analysis. For $\\chi^2$ divergence, the suboptimality bound is proportional to $(\\sqrt{\\rho}H^3+H^2)$ as shown in Theorem 4.13 of our submission. For $\\rho=O(1/H^2)$, this term is in order of $O(H^2)$ and our bound matches that for non-robust linear mixture MDP; for moderate $\\rho$, our bound suggests that policy learning would be harder due to the complex geometry of the $\\chi^2$ divergence uncertainty set. A similar pattern has also been identified by a prior work [3] on tabular DRMDPs with $\\chi^2$ divergence defined uncertainty sets.\n\n---\n> **Q2:** Dependence of your models on the number of data trajectories available for training.\n\n**A2:** Our suboptimality bound can be translated into sample complexity bounds. To learn an $\\epsilon$-optimal policy, we need sample sizes in order of $\\tilde{O}(\\frac{d^2H^4(C^{\\pi^{\\star}})^2}{\\epsilon^2})$ for TV divergence, $\\tilde{O}(\\frac{d^2H^4(C^{\\pi^{\\star}})^2e^{2H/{\\underline{\\lambda}}}}{\\rho^2\\epsilon^2})$ for KL divergence, and $\\tilde{O}(\\frac{d^2(\\sqrt{\\rho}H^3+H^2)^2(C^{\\pi^{\\star}})^2}{\\epsilon^2})$ for $\\chi^2$ divergence. \n\nClassic DRO approaches are expected to be effective in scenarios with limited data because they are explicitly designed to guard against **uncertainty from insufficient data samples**. In contrast, our current framework mostly focuses on uncertainty from dynamics shift, which is also common in real world applications.\nMoreover, our DRMDP framework hedging against dynamics shifts can also be adapted to achieve robustness w.r.t. parameter estimation uncertainty due to limited data. Actually this idea has been explored in the prior work [4], where they show DRMDP can be used to solve offline RL. Experiments therein demonstrate DRO approaches achieve significantly better performance than non-robust approaches with limited samples. Thus, we can expect the same is true for our framework. We leave a comprehensive examination for future study. \n\n---\n> **Q3:** The impact of the robustness approach on scalability, especially compared to the standard (s, a) and d-uncertainty sets?\n\n**A3:** From a theoretical perspective, our suboptimality bound does not depend on the size of state-action space. Thus even when the state-action space is of high dimensional and infinitely large, our approach scales well thanks to the linear mixture structure. d-rectangular DRMDPs also can scale, leveraging a different modeling on latent structure of dynamics and uncertainty set. The linear mixture MDP or linear MDP assumption on the nominal transition kernel naturally admits linear representation in terms of the dynamics or Q-function, thus enabling linear function approximation for scalability. While for $(s,a)$-rectangular DRMDPs, most existing model-based and model-free approaches [1,3,6] cannot scale, they either rely on frequency-based transition estimation or rely on a Q-table for robust Bellman update. Existing literature has also studied general function approximation for scalability in the context of $(s,a)$-rectangular DRMDPs. For example, [5] study $(s,a)$-rectangular DRMDPs with an infinitely large state-action space, they leverage a double layer function approximation routine for dual optimization approximation and robust Q-function estimation. Their proposed algorithm, Robust Fitted Q-Iteration, can scale in applications with continuous states and actions, like CartPole and Hopper. [2] study function approximation in $(s,a)$-rectangular DRMDPs from a theoretical perspective. They present suboptimality bounds depending on the complexity measure of the function class and independent on the size of the state-action spaces.\n\n\n---\n> **Q4:** how difficult to obtain good estimates of the latent modes.\n\n**A4:** Thanks for the interesting question. To answer this question, we would like to lay out several practical motivations for our framework. Mixture models, in particular Gaussian Mixture Models (GMMs), are widely studied in fields of optimal control and RL. For example:\n\n**Model Predictive Control (MPC):** [1] studies the density steering for discrete-time linear systems, a variant of optimal mass transport problem widely used in applications such as controlling a swarm of robots/drones to achieve a desired spatial distribution. They model the stochastic dynamical system using GMMs and derive its optimal control policy. [2] studies a stochastic MPC algorithm for linear systems subject to additive Gaussian mixture disturbances. They consider a vehicle control case study in which the vehicle must maintain its position on an ill-maintained road. \n\n**Meta RL:** Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from a set of training tasks to unseen, but related, tasks. Recent works on Meta RL model the dynamics of a new system as a mixture of expert systems, thus realizing knowledge transfer. In particular, [3,4] model the dynamics of the expert systems using Gaussian process with latent embedding. And the dynamics of the new system is constructed by a mixture of Gaussian processes with the distribution over the latent embedding being the mixture weights, which should be learned from data. In experiments, their meta-learning models effectively generalizes to novel tasks under various environments such as Cart-pole & Double-pendulum swing-up, HalfCheetah and Highway-Intersection. \n\n**World Models for model-based RL:** Model-based Deep Reinforcement Learning (RL) assumes the availability of a model of an environment’s underlying transition dynamics, which are stochastic in nature and oftentimes multimodal. [5,6] use Mixture-density Networks to construct World Models to capture the multimodality in dynamics. Then model-based RL methods are implemented based on the constructed World Models to solve various tasks such as car racing and Inverse Sine Wave. \n\n\nThese applications naturally give rise to distributionally robust MDP problems, for example 1) robust control of drones or vehicles is needed to hedge against environmental uncertainties and distributional perturbations; 2) given established expert systems, usually the mixture weights are estimated from limited data collected from the unseen new task, and thus of high uncertainty. Our framework could potentially enhance robustness of decision making in unseen tasks by modeling the weighting parameter uncertainty; 3) by modeling the uncertainty in the World Model (specifically, in the weights), our framework enables robust decision making in model-based RL. \nThe basis modes in our work can be the Gaussian components in GMMs, dynamics of expert systems, or mixture components in the Mixture-density Networks. As for the basis modes construction, we can adopt the estimation procedure in prior works. The difficulty should depend on specific tasks. It serves an interesting future research direction to also incorporate the uncertainty in basis modes into consideration for robust decision making. \n\n\n---\n> **Q5:** Potential limitations of the choice to use mixture models.\n\n**A5:** \nOur model assumes the dynamics is a mixture distribution with finite modes/component distributions. This restricts its application in scenarios where mixture distributions cannot approximate the dynamics well. Furthermore, the current manuscript assumes perfect knowledge of the basis modes and focuses solely on uncertainty in the weighting parameters. However, in practice, these basis modes often need to be estimated from data, introducing an additional layer of uncertainty that can significantly affect decision-making. Addressing this source of uncertainty is nontrivial and falls outside the scope of the current work; we leave it for future investigation.\n\n---\n**References:**\n\n[1] Shi et. al., 2024, Distributionally robust model-based offline reinforcement learning with near-optimal sample complexity\n\n[2] Blanchet et. al., 2023, Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage\n\n[3] Shi et. al., 2023, The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Model\n\n[4] Wang et. al., 2024, A Unified Principle of Pessimism for Offline Reinforcement Learning under Model Mismatch\n\n[5] Panaganti et. al., 2022, Robust reinforcement learning using offline data\n\n[6] Liu et. al., 2022, Distributionally Robust Q-Learning\n\n[7] Balci et. al., 2024, Density steering of gaussian mixture models for discrete-time linear systems\n\n[8] Engelaar et. al., 2024, Stochastic MPC for Finite Gaussian Mixture Disturbances with Guarantees\n\n[9] Sæmundsson et. al., 2018, Meta reinforcement learning with latent variable gaussian processes\n\n[10] Xu et. al., 2020, Task-agnostic online reinforcement learning with an infinite mixture of gaussian processes\n\n[11] Sedlmeier et. al., 2021, Quantifying multimodality in world models\n\n[12] Ha et. al., 2018, Recurrent world models facilitate policy evolution"}, {"review_id": "VsWce3Bxi1", "reviewer": "Reviewer_okkx", "summary": "In this paper, the author proposes a linear mixture distributionally robust Markov decision process (DRMDP) framework, which differs from the more common setting where the ambiguity set is defined as a ball centered around a nominal transition kernel. The author also provides proofs that the dynamic programming principles hold for linear mixture DRMDPs. This result serves as a solid theoretical foundation for future research. In datails,  the author analyzes the case of $\\phi$-divergence-based ambiguity sets and presents an error bound analysis.", "questions": "1. As mentioned in the weaknesses, the motivation for using the linear mixture ambiguity set is not entirely clear. It would be helpful if the author could explain under what conditions or in which scenarios this type of ambiguity set is preferable to more common settings,. Are there specific applications where the linear mixture ambiguity set offers clear advantages? A more detailed discussion or empirical comparison would strengthen the motivation for this approach.\n\n2. Since the linear mixture ambiguity set is formed by combining several latent base models in a linear way, it is unclear how well the method performs when the true distribution is non-convex or multimodal. For example, if the underlying distribution contains multiple distinct modes, the linear mixture structure may not be flexible enough to capture such complexity. Can the author extend the method to better handle multimodal distributions?\n\n3. In the experiments, the author compares the non-robust method VTR with the DRTTR-based method. Can we have a comparison between the proposed distributionally robust method using the linear mixture ambiguity set and a method based on an $(s,a)$-rectangular ambiguity set?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths: The structure of the paper is clear, and the author uses several figures to illustrate the differences between the linear mixture ambiguity set and the more common settings. These visualizations help the reader better understand the key ideas and contributions of the work.\n\nWeaknesses: Although the author provides strong theoretical results, the analysis relies on a number of assumptions. This raises the question of whether real-world applications can satisfy all these assumptions, especially since each theorem introduces new conditions. While the theoretical contributions are well-supported, the performance evaluation is based on training with an offline dataset. A demonstration using a real-world application would be more convincing and could better highlight the practical effectiveness of the proposed approach. Moreover, common ambiguity sets, such as those based on Wasserstein distance or $\\phi$-divergence, have been shown to perform well in many practical settings. Therefore, it would be helpful if the author could provide more detailed guidance on when to choose the linear mixture ambiguity set over these existing alternatives.", "was_revised": true, "final_justification": "Keep score", "rebuttal": "We thank the reviewer for their valuable time and effort in providing detailed feedback on our work. We hope our response will fully address all of the reviewer's points.\n\n---\n> **Q1:** Practical motivation for linear mixture DRMDPs. When should we choose linear mixture DRMDPs over other frameworks?\n\n**A1:** Thanks for the interesting question. To answer this question, we would like to lay out several practical motivations for our framework. Mixture models, in particular Gaussian Mixture Models (GMMs), are widely studied in fields of optimal control and RL. For example:\n\n**Model Predictive Control (MPC)**: [1] studies the density steering for discrete-time linear systems, a variant of optimal mass transport problem widely used in applications such as controlling a swarm of robots/drones to achieve a desired spatial distribution. They model the stochastic dynamical system using GMMs and derive its optimal control policy. [2] studies a stochastic MPC algorithm for linear systems subject to additive Gaussian mixture disturbances. They consider a vehicle control case study in which the vehicle must maintain its position on an ill-maintained road.\n\n**Meta RL**: Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from a set of training tasks to unseen, but related, tasks. Recent works on Meta RL model the dynamics of a new system as a mixture of expert systems, thus realizing knowledge transfer. In particular, [3,4] model the dynamics of the expert systems using Gaussian process with latent embedding. And the dynamics of the new system is constructed by a mixture of Gaussian processes with the distribution over the latent embedding being the mixture weights, which should be learned from data. In experiments, their meta-learning models effectively generalizes to novel tasks under various environments such as Cart-pole & Double-pendulum swing-up, HalfCheetah and Highway-Intersection. Our framework could potentially enhance robustness in unseen tasks.\n\n**World Models for model-based RL**: Model-based Deep Reinforcement Learning (RL) assumes the availability of a model of an environment’s underlying transition dynamics, which are stochastic in nature and oftentimes multimodal. [5,6] use Mixture-density Networks to construct World Models to capture the multimodality in dynamics. Then model-based RL methods are implemented based on the constructed World Models to solve various tasks such as car racing and Inverse Sine Wave.\n\nThe basis modes in our work can be the Gaussian components in GMMs, dynamics of expert systems, or mixture components in the Mixture-density Networks. One potential application of our framework would be the meta-RL. For example, assume we have constructed expert system dynamics (basis modes) from huge amounts of historical data, and would like to solve a new task whose dynamics is a mixture distribution of expert system dynamics. With limited data collected from the new task, there is high uncertainty in weighting parameter estimation. In such cases, the linear mixture DRMDP framework leverages the latent structure in dynamics of the new task, and thus is conceptually more favorable than other DRMDP framework. Other applications involve robust control of drones or vehicles to hedge against environmental uncertainties and distributional perturbations, and robust decision making in model-based RL by modeling the uncertainty in the World Model (specifically, in the weights).\nMoreover, it serves as an interesting future research direction to also incorporate the uncertainty in basis modes into consideration for robust decision making. \n\n\n---\n> **Q2:** Linear mixture structure for multimodal distributions.\n\n\n\n**A2:** Thank you for the interesting question. We highlight that the linear mixture structure can naturally model multimodal transitions. In particular, each basis mode can represent a component distribution with corresponding local maximum in the probability density function or probability mass function, such as gaussian distributions with different means and variances. And the mixture of basis modes capture the multimodality in the dynamics, say, for a given state-action pair, the next state follows multiple distinct patterns. For example, [5] discuss the multimodality in dynamics of many real-world RL applications. They introduce world models using mixture distribution networks to approximate the multimodal dynamics, which could potentially be adapted to our framework to construct the basis modes. \n\n---\n> **Q3:** Comparison between the proposed distributionally robust method using the linear mixture ambiguity set and a method based on an (s,a)-rectangular ambiguity set.\n\n**A3:** We can justify this point. In this work we mainly focus on establishing the framework, which is shown in Section 2 to have advantages in modelling the uncertainty in cases where existing frameworks fall short, from scratch and designing algorithms that achieve provable efficiency. The simple simulation environment is just to validate our theoretical results, i.e., to showcase the robustness of our proposed algorithms. We also would like to note that the linear mixture DRMDP assumes prior information, i.e. the access to basic modes, for algorithm design. It’s not fair to directly compare algorithms designed for different DRMDP settings because they either require other prior information or do not require prior information at all. Thus, we instead compare those two algorithms designed for the linear mixture DRMDP. \n\nAs an interesting future work, we would like to explore when it is appropriate to model the change in the dynamics as a linear mixture uncertainty set and how to construct the latent modes in complex environments like MoJuCo. In those real-world applications, we can have a fair comparison between different DRMDP frameworks. However, this requires considerable effort, which is beyond the scope of this work. \n\n---\n**References:**\n\n[1] Balci et. al., 2024, Density steering of gaussian mixture models for discrete-time linear systems\n\n[2] Engelaar et. al., 2024, Stochastic MPC for Finite Gaussian Mixture Disturbances with Guarantees\n\n[3] Sæmundsson et. al., 2018, Meta reinforcement learning with latent variable gaussian processes\n\n[4] Xu et. al., 2020, Task-agnostic online reinforcement learning with an infinite mixture of gaussian processes\n\n[5] Sedlmeier et. al., 2021, Quantifying multimodality in world models\n\n[6] Ha et. al., 2018, Recurrent world models facilitate policy evolution\n\n---\nWe hope we have addressed all of your questions/concerns. If you have any further questions, we would be more than happy to answer them and if you don’t, would you kindly consider increasing your score?"}]

Author rebuttal:

Meta-review: Summary: This paper introduces a new class of distributionally robust Markov decision processes (DRMDPs) that leverage linear mixture structures in the transition dynamics. Unlike conventional rectangular uncertainty sets, the proposed linear mixture ambiguity sets exploit structural assumptions to yield less conservative policies and tighter robustness guarantees. The authors develop a modular algorithm, provide rigorous theoretical analysis under several f-divergences, and establish finite-sample suboptimality bounds that demonstrate the statistical learnability of robust policies. Illustrative examples highlight how the linear mixture sets can be strictly smaller than standard ones.

Comments: We received 4 expert reviews, with scores 4, 4, 4, 5, and the average score is 4.25. 

The reviewers acknowledge that the paper is technically solid, with an interesting theoretical contribution in robust RL. In particular, the linear mixture ambiguity sets are shown to provide advantages over standard (s,a)- and d-rectangular sets, both theoretically and illustratively. The work establishes meaningful finite-sample guarantees and variance bounds. The paper is well-written. 

The reviewers have also pointed out many weaknesses. Reviewer 8wNc has pointed out that the planning oracle assumed in the theoretical framework is computationally intractable, and no approximation method is analyzed, creating a gap between theory and practical deployment. More than one reviewer has pointed out that the experimental validation is minimal. 

Based on my reading of the paper, reviews, and rebuttal discussion, I believe that, overall, this paper provides strong theoretical contributions and advances the literature on robust RL by proposing and analyzing a new ambiguity set structure. While the lack of empirical evaluation is a limitation, the primary strength lies in the theory, and the technical results are substantial enough to merit acceptance. The authors should, however, improve the presentation of the intuition behind their assumptions, clarify the conditions under which linear mixture ambiguity sets offer clear advantages over existing alternatives, and provide a more explicit discussion of potential applications. Including even modest additional experiments or comparisons to standard rectangular sets would significantly strengthen the case for practical impact.
Decision: Accept (poster)