Offline Reinforcement Learning via Inverse Optimization
Authors: Ioannis Dimanidis
Tolga Ok
Peyman Mohajerin Esfahani
Venue: Accepted by TMLR ()

Abstract: Inspired by the recent successes of Inverse Optimization (IO) across various application domains, we propose a novel offline Reinforcement Learning (ORL) algorithm for continuous state and action spaces, leveraging the convex loss function called  "sub-optimality loss" from the IO literature. To mitigate the distribution shift commonly observed in ORL problems, we further employ a robust and non-causal Model Predictive Control (MPC) expert steering a nominal model of the dynamics using in-hindsight information stemming from the model mismatch. Unlike the existing literature, our robust MPC expert enjoys an exact and tractable convex reformulation. In the second part of this study, we show that the IO hypothesis class, trained by the proposed convex loss function, enjoys ample expressiveness and reliably recovers teacher behavior in MuJoCo benchmarks. The method achieves competitive results compared to widely-used baselines in sample-constrained settings, despite using orders of magnitude fewer parameters. To facilitate the reproducibility of our results, we provide an open-source package implementing the proposed algorithms and the experiments. The code is available at https://github.com/TolgaOk/offlineRLviaIO.

Published in Transactions on Machine Learning Research (03/2026)
Offline Reinforcement Learning via Inverse Optimization
Toannis Dimanidis
Gounndina@grmaal.ccmmm
Delft Center for Systems and Control Delft University of Technology Netherlands
Tolga Ok
tok
tok tokktudeff..dl
Delft Center for Systems and Control Delft University of Technology Netherlonds
Peyman Mohajerin Esfahani
Industrial & Mechanical Engincering
University of Toronto Conada
Reviewed on OpenReview: https: // openreview. net/ forum? id= 18j2gXoeCy
net/forum?
Abstract
Inspired by the recent successes of Inverse Optimization (1O) across various application domains, we propose a novel offline Reinforcement Learning (ORL) algorithm for continuous state and action spaces, leveraging the convex loss function called "sub-optimality loss" from the 10 literature. To mitigate the distribution shift commonly observed in ORL problems, further employ robust and non-causal Model Predictive Control (MPC) expert steering nominal model of the dynamics using in ha-diidsight information stemming from the model mismatch. Unlike the existing literature, our robust MPC expert enjoys an exact and tractable convex reformulation. In the second part of this study, we show that the 10 hypothesis class, trained by the proposed convex loss function, enjoys ample expressiveness and reliably recovers teacher behavior in MuJoCo benchmarks. The method achieves competitive results compared to widely-usod basclines in constriiod settings, despite using orders of magnitude fewer parameters. To facilitate the reproducibility of our results, provide open-source package implementing the proposed algorithms and the experiments. The code is available at https:: cotToo/aa// ine.
1 Introduction
In dynamic environments where real-world interactions are impractical, there is often the need to work with datasets of previously collected interactions. Decision-making in these contexts typically follows one of two paradigms. (i) Imitation learning (IL), subclass of the Supervised Learning (SL) paradigm, in which the aim is imitate given expert's decisions (i.e,, labels in SL terms) and (ii) offline Reinforcement Learning (RL), where the aim is to learn policy that improves upon the performance observed within the dataset. SL in general, and IL in particular, has proven to be successful in a wide range of applications (Husscin et al. 2017). while oflline RL is known to be a notoriously hard task (both computationally and statistically) (Bertsekas) 2021). One of the primary challenges in offline RL the mismatch between the dataset and the policy distributions. Hence, naively applying existing online RI. algorithms combined with high-capacity Q function approximation leads optimistic and potentially biased value functions, which, turn, leads poorly performing and unstable policies that do not generalize in the online evaluation.

---

Published in Transactions on Machine Learning Research (03/2026)
To combat these issues, in this work, we approach the offinne RL problem in two steps: (1) by utilizing non-causal expert, we perform an "action improvement" step over the dataset; and (īi) using the improved actions, we fit a Q-function using a novel "sub-optimality loss" to obtain an efficient and causal policy that generalizes over online evaluations. Specifically, in the first step, by leveraging a nominal model and in-hindsight model mismatch information, unknown at runtime, we rntuee an expert in the form of a non-causal Model Predictive Control (MPC). While we assume access to a nominal model, the true dynamics are influenced by unknown disturbances and model mismatches. A traditional MPC approach in this setting would either ignore these disturbances- -eadiigg to significantly degraded performance or require significant human effort to manually model the residuals and incorporate them into the control loop. By contrast, our non-causal teacher/canl caaual student setup streamlines this process: the non-causal expert utilizes in-hindsight data to determine the optimal response lo disturbances, which the student then distills into a causal policy. This enables the agent to learn directly from the data distribution without the need for explicit disturbance modeling or heavy human supervision. To realize the non-causal expert, we propose replace the Bellman residual loss with the "sub-optimality loss" drawn from the Inverse Optimization (IO) literature that fits the optimal Q function given the improved dataset. The proposed optimization problem enjoys the convexity of the loss function, yiekding efficient and causal policy that can generalize over unseen states. Before proceeding with further details regarding our proposed approach and the related literature, we introduce some notations.
The dimension of variable x is denoted by Rg. We denote with N the MPC horizon and with T the size of dataset. With bold, we denote the stacking of variables, i.e., X1:N (41, (2........... IN), IN), unless noted otherwise. When no exact range is given in the subscript, the default length of a bold variable is N (i.e., =..NN. We denote by (+-) an inner product with the respective norm ||^||| = x). For any A>00 0, we define ||0]/== - (2r, Ax). With 8, we denote the Kronecker product. As the letter "Q" will be used to indicate both matrices and Q-functions, we denote with Q the former and with Q(s, u) the latter, although it should usually be clear from the context. The operators diag(-) and blkdiag(-) construct square or block matrix, respectively. Finally, with MPC-N, we refer to policies stemming from the minimization of an N-stage cost that predicts the future behavior of the system using some model.
Notation:
1.1 Problem statement and Contributions
We consider the constrained control of discreto-time dynamical systems with unknown dynamics f, where we have access lo a deterministic nominal model fo: and an oflline trajectory of stale-action pairs Dr = & âg 2 collected under some behavior policy applied to f- The control u is constrained to belong to some (xx), which is also assumed to be known, and there is an N-stage control objective defined through the known stage- and terminal-cost functions c(æ: u) and e (z). We aim to learn causa| stationary parameterized policy vy that distills a non-causal MPC expert on So with access to the full future model-mismatch sequence inferred from Dr and fo. Concretely, our proposed RL scheme comprises two key steps:
Non-causal action improvement: From Df and fo infer the mismatch trajectory (e.g., ui+1 = 4++1 fu( d.)). Feed the full future mismatch sequence into an N-stage recoding-horte horizon non-causal MPC defined on fo to produce an expert control sequence [T A robust variant replaces the known mismatch with a worst-case element from specified uncertainty set.
Imitation learning/polin pplicy distillation: Fit a cansal policy Fe to the new improved state-action dataset {āe, a;x) 17 via a tractable convex TO objective (sub-optimality loss) yielding a computationally cheap policy for deployment.
The above procedure is documented in detail in Algorithm I Building on this setting, we summarize our contributions as follows:
(i) Two-step offline RL via IO-based policy distillation: We use the Inverse Optimization framework to bridge oflline RL with Imitation Learning, distilling the non-causal MPC expert from the above
AA policy deemed be causal iff depends only on past and present data, i.c,, ie = a(+/17 $ f)-
2

---

Published in Transactions on Machine Learning Research (03/2026)
two-step process into causal policy. This approach circumvents the need for explicit disturbance models by learning the environment's nuances directly from trajectory data. The resulting formulation is computationally attractive due to the convexity of the training landscape and results in policies that are efficient to evaluate, while also opening the door for tools from online convex optimization to be readily used for the control lasks considered here.
(ii) Tractable robustification of the MPC expert: For the case of linear dynamics, quadratic stage/ stage/terminal /terminal costs, and polytopic constraints, we derive an exact convex reformulation of a robust non-causal MPC expert that optimizes against worst-case model-mismatch trajectories within a prescribed uncertainty set. This allows us to incorporate adversarial robustness without introducing conservatism or sacrificing tractability. From the empirical analysis of Appendix [ and Section [.1]] we show that the robustification helps combat the distribution shift from the training to the test phase and the mismatch between the nominal model and the true dynamics.
(ii) Empirical validation of IO expressiveness: Through experiments on nonlinear control problems and MuJoCo benchmarks, we provide evidence that the 10 hypothesis class is expressive enough for high-quality policy distillation imitation learning. In particular, our method achieves statc-of-the-art performance in low-data regimes while using ofders-of-magniuude fewer parameters than neural-network--bssed baselines.
1.2 Related works
Offline Reinforcement Learning: To prevent the value function from exploiting any dataset bias, offline RL approsches typically attempt to enforce pessimistic policy learning (Rashidinejad et al. 2021;; this be achieved by constraining the policy learning within the region supported by the dataset FFjiimooo el al. 2018) or by penalizing the value function for the state-action pairs outside the dataset (Kumar et al. 2019 Wu et al.) 2019: Kostrikov et al.) 2021 Kumar et al. 2020). Model- approaches employ similar ideas but instead try to exploit the model information to learn a less conservative value function. For instance, COMBO (Yu et al., (2021) approximates the true model dynamics and utilizes both simulated and dataset. samples to learn conservative value estimation by penalizing out-of-support state-action pairs obtained by running the simulated model. On the other hand, our proposal uses nominal model to improve the actions of the state-action pairs present in the dataset; finally, in contrast to the aforementioned works, our work is more computationally attractive, as the resulting program for learning the policy is convex.
Imitation Learning: The second step of our algorithm, where we employ Inverse Optimization to fit a policy on the improved state-action pairs, is analogous to IL. Similar to our dataset improvement scheme, several other II. algorithms employ augmentation strategies to further improve policy learning. For example, BAIL (Chen et al. 2020) first estimates the Monte Carlo returns of each state-action pair in the dataset, an infinite horizon and discounted extension of our objective function, and employs a neural network-based estimate to the returns. Based on this estimate, BAIL selects only the highest-valued state-action pairs and learns a policy via IL. On the other hand, our approach makes use of the entire dataset, improving actions through our robust MPC formulation, and utilizes a convex "sub-optimality loss" to perform the IL step. The decision to employ relatively small model together with a convex loss function (in the parameter space) is justified by empirical studies (see Section and aligns with similar findings reported by Emmons et al. (2021). particularly in limited data regimes.
Terminal value function approximation: Since MPC projects its internal model into the future, it also act as approximation to the Bellman equation. This observation is exploited by (Zhong al. 2013) to effectively increase the planning horizon by constructing approximate terminal Value Functions (VF) from MPC simulation data. Using the same principle, (Lowrey et al.) 2018) showcased an algorithm that promoles exploration and, therefore, accelerates VF learning. Finally, (Bhardwaj et al. 2020) propose a blended approach that combines elements from model-free and model-based methods to reduce model bias. Similarly, our work can be viewed as a specific instance of VF approximation, where learning the Q-function reduces the horizon to a single step. Additionally, in contrast to the papers mentioned above, our approach is computationally tractable.
3

---

Published in Transactions on Machine Learning Research (03/2026)
Trajectory Augmentation: A large body of work has studied the augmentation of offline datasets to mitigate distributional shift and synthetically generate high-return trajectories. Among recent approaches, Generative Trajectory Augmentation (GTA) (Lee et al. 2024) employs a trajectory-level conditional diffusion model to enrich the offiin RL dataset toward high-return regions, while remaining consistent with observed data. Similarly, Diflusion-based Trajectory Stitching (DiffStitch) (Li et al.] 2024) synthesizes bridging sub-trajectories that connect low- and high-return trajectories, effectively stitching them together to form an expanded dataset. More closely related to our setting are model-based augmentation methods (Wang et al. 2021 Lyu et al. 2022. Zhang al., 2023), which learn the dynamics and the rollout policy to generate synthetic trajectories, while enforcing conservatism via model agreement uncertainty-baser truncation. By contrast, we do not synthesize new states or transitions; instead, given a nominal model, we improve the actions along observed trajectories by solving the robust non-causal MPC problem introduced in Section 3
2 Inverse Optimization for RL
In what follows, we briefly review the existing literature on IO and its potential learn a control law, We then introduce the first contribution of this study: how in-hindsight information be exploited to devise offline RL algorithm.
2.1 Inverse Optimization as Supervised Learning
The goal Inverse Optimization is to learn the behavior an expert whose actions depend on an external signal. Specifically, for a given E S C RR",, the expert's decisions u E U(s)) RR" stem from deterministic policy: uxx = (s). We wish to approximate (s) with a policy in similar spirit in Q-Learning that is defined as:
Te(s) arg min Qa(s, u) aEl(()
where Qa is parameterized function belonging to the hypothesis class Q. Throughout this work, we consider the strongly convex quadratic hypothesis class
Q = =QQoss u) = (u, Ouuu) + 2(s,, Ouuu) Oua b Inn.
(1)
To learn the optimal 0", we use the "sub-optimality loss", which was first introduced in Mohajerin Esfahani et al. 2018):
(a ) Qu(s, u w²*) min Qa(s,u).
wEM(()
Notice that the mapping + Qo u) is linear, and thus, the "sub- isub-optimality loss" ( is convex in 0 for convex U(s). Given dataset {Se, 01*² ) of states 3, and expert actions 0. x (8),, and polytopic constraint set U(s) {u: G(s)u < h(s)}, we have that (Akhtar et al.
2021
T Σ Qa 0(x) + 7 (hesde)
min Bub (St, 0,*)=
min 0.yu: Auer
(3)
t=1
1=1
s.t.
Onm
> Inui
A, 2.20 0,
t S T,
Oum
2 GT 2AT
> 0,
t t
7
where we use the shorthand G, G(8.) and hy = h(&)). The convex optimization oflers an efficient way to learn the policy (.). It is important to highlight that a key part upon which this program is built the sequence of the "ground-truth" expert actions n While the actions contained within an offline RL datasct. can be regarded expert actions, we propose to improve them by leveraging the hindsight information of controlled dynamical system.
4

---

Published in Transactions on Machine Learning Research (03/2026)
2.2 Imitating an MPC expert with Inverse Optimization
Given a delerministic nominal model fa, and denoting state and inpul constraints for each step X and U respectively, we formulate the deterministic MPC-N problem as follows:
N
Vmmee N
(x) = min
dx., uk) + (g((N)
(4)
s.t. E (N
where (N""(() (x) = (u ( Nnu ur U, Tk+1 fo(#R; ux) X, N, 2o = = Thanks to the principle of
optimality, express the Q-function of (4) as Qupe(z: (z, u) clz, u)+ VN u)), which is defined over
the 1-step constraint set (x) = {u E I u E u, fo(z, u) E X}. To approximate Q with Inverse
Optimization, we solve 33 with & = Lt, and tht = (+t), where
π ( (x) = arg min QuP u). wEL("i"(a) (x)
(5)
Remark 2.1 (Approximating MPC with IO). For the MPC problem 4 be tractable, common assumption is that fo is linear in z and and the sets X and U are polytopic. In such a setting, the Q-function Qmue is piecewise quadratic where the number of pieces may be exponential in the horizon length N. Therefore, approximating Quipe using the quadratic hypothesis class (I] may likely not be exact. Nonetheless, as reported in (Akhtar et al. 2021), such an approximation can work quite well. If there are no constraints, then (4) becomes a horizon LQR problem, whose Q-function is known to be quadratic and positive definite, and as such, we can have an exact approximation within the hypothesis class [ In this case, the approximate policy becomes TT(s)) (s) = -0 s,, which implies that we essentially learn an optimal linear control policy.
2.3 Exploiting in-hindsight information
This section contains the first contribution of this study, aiming to bridge the gap between IO and offline RL settings. To this end, we consider an exlended nominal model with additive disturbances w E R"", ie., fo(2, u, w) = fo(2, u) + Far where Et F = Denoting the N-length disturbance trajectory by w, we define the non-causal MPC-N problem via
V-o-mpe (I, (x,w)= w) = min Σ
clEK, uk) + cg(xx)
(6)
0=0
s.t.
u E UN
une-mp (z, w).
with Un nc-mpe (I, w) = (u Nn. E U, 0k+1 = Jo uk, Wx+1) E X, k N, xn = =x). Then, akin to Sec- tion 2.2 we can define Que-mporr, (I, u. w) and 2 u (I, w) accordingly, and therefore we obtain the non-causal
MPC expert policy
no-mpe (x, w) π
arg min
Q n (I, u, w)
uEt me-mpi (2.w)
We construct expert actions by leveraging in-hindsight disturbance trajectories extracted from data. Given the extended nominal model fo and a measured transition (ś, â, f4), we define the residual Eu = 4 - fo( 0). By concatenating such residuals across the datasct Dr, we obtain disturbance sequences that can be injectod into the non-causal MPC problem (6) to compute expert actions. Because this policy requires future disturbances WI+1++N, not available online at time t, it is inherently non-causal and can only be used offline.
To make this expert usable in practico, we approximate it causally. We introduce feature map 6 that summarizes past information and define the augmented state 8, = ((xia, unz). In general, the design of o is a feature-engineering problem and lies outside the scope of this paper; we assume that, given the application at hand, one has access to features that can capture predictive structure the disturbances; for example, if disturbances evolve lincarly, a natural feature choice is the most recent H residuals, ie., ((x1a; Uaa) = Wi- H- la, with H chosen sufficiently large.
5

---

Published in Transactions on Machine Learning Research (03/2026)
Algorithm Using in-hindsight information for 10
Input:
Offline trajectory Dr {(Br, (e)
Extended nominal model fo(2,, u) + Ew
4: Non-causal expert policy TN (z. w) with horizon N
5: Feature map ((--)
6: Initialize dataset of training pairs Dex 5
for 1 1
do
Compute residual mismatch uh+1 t- (de+1 folSe: a.)) Compute augmented state Φ(811, un:e)
10:
Let T -
11:
if T 21 then
12:
Query expert action 0F* + N (fr, W Wr+1)++)) Append (8,, (.*) to D.x
13:
14: end if
15: end for
16: Solve the 1O training problem
33 with dataset Dox to obtain 0*
17: Return: policy parameters l
We then use Inverse Optimization to train a causal policy T(s;) that imitates the non-causal MPC expert policy 7 implicitly learning both the predictive relationship between past and future disturbances and the corresponding optimal response. The procedure used to approximate the non-causal MPC expert with IO is outlined in Algorithm 11.
Remark 2.2 (Validity of in-hindsight trajectories). The validity of this construction depends on the source of the mismatch. If disturbances are exogenous, i.e, generated by an external process independent of the state-action action trajectory, then the non-causal MPC problem with in-hindsight disturbances (6) is equivalent to optimizing directly on the true system f, and the expert corresponds to the true optimizer. If disturbances depend on the state-action path, the disturbance sequence is path-dependent and cannot be reused counterfactually; in this case the in-hindsight expert remains useful surrogate teacher, but not the true optimizer.
Remark 2.3 (Literature on disturbance feedback and non-causal control). The idea of "disturbance feodback control" has also been explored in recent works related to online control for advensarial disturbances (Hazan et al. 2020 Agarwal et al.) 2019 Foster & Simchowitz 2020). Additionally, similar problem is also considered (Goel & Hassibi 2021) where a non-causal controller is approximated by causal one in an offline setting. Contrary to these works, which consider a lincar policy class with no constraints on state or input, our proposed policy is nonlinear nature and can handle constraints.
Remark 2.4 (Feature engineering). While the proposed quadratic hypothesis class 11 is affine the feature space, it does not limit the policy to linear functions of the raw state. By selecting appropriate feature maps 6 (e.g., polynomial expansions, sines/cosines of the state), one can model highly nonlinear also empirically
control laws, as be seen in the MuJoCo experiments Section 4.2 Additionally, show that even simple features, such as disturbance histories, can be sufficient for even nonlincar control tasks (Sn 4.1 Appendix B.3h. Furthermore, for lasks requiring universal approximation capabilities. the proposed framework can be extended to use kernel methods (e.g., Gaussian kernels (Long et al., 2024) or Neural Tangent Kernels (Jacot et al. 2018)). This allows the algorithm to operate high-dimensional implicit feature spaces without manual feature engincering, all while preserving the convexity of the training objective. In fact, in Section 4.2 successfully employ the use of Gaussian kernels for certain experiments.

---

Published in Transactions on Machine Learning Research (03/2026)
3 Robust DisturbanceAAwrr Aware MPC
3.1 Robustification around disturbance trajectory
The non-causal MPC expert oplimizes directly against the noisy disturbance trajectory. However, due to stochasticity and/or potential distribution shifts in the data, performance might be degraded, and we may even observe instabilities. Therefore, we opt for policy that is robust to such issues. To this end, let us introduce the robust counterpart to the non-causal MPC 7 described as
ync-rmpe
(x, w) = min
max WEW(()
Σ
dak uz) + cy(2N)
(8)
s.L.
11 E UN
ne--mpp (x, w).
where W(w)= C --Na= i is the disturbance uncertainty set centered around the trajectory w, and
UN ucc-rmpc (x, w) = {u E N=. u UN mne--mpe (z, w), Vw E W(w))
(9)
A problem like {8 can easily be computationally intractable, even if its non-robust version is not. When dealing with such problems, it is therefore common for conservative approximations to be used even when the nominal model so is linear. Here, we propose an uncertainty set W for which ( is tractable under linear dynamics and constraints and quadratic costs. Before we proceed, let us introduce a useful preparatory Lemma.
Lemma 3.1 (Vectorized MPC formulation for linear dynamics). Under linear nominal dynamics Jo(E, u) Ar + Bu, and quadratic costs c(r, = ||=1 + +lullle.. and cf(2, u) = || where Qz Qf to and Qu > 0, the objective of 6 con be equivalently expressed by
Ax + Bu Ewll2. + llul22..
with Qx blkdiag(IN-1 & Qy) Qu IN Qu A blkcol(A, AN) B = Tv(A, E Tx(A E Moreover, when the stage constraints are polytopic U {u R G.u S hut and X {or E Ra= Gx S hz), the constraint set of is also polylopic in the form of
uno-mee (x, w) {u Nu. Fr Gu S h(w)}, UN
with FT = [G2 0J. GT = [x,B Gu h(w) = [(hx GxEw) bul], Gx = IN 8 Gz, Gu = IN & Gu, h = Ix 8 hr, h, = N & ha-
Thanks to Lemma 3.1 the MPC problem
6 can be simplified to the convex quadratic program
min
|
Bu + Ewllg, + +lulla
(10)
s.t. Fx. + Gu h(w)
The uncertainty set W consider here is ball centered on the N-length disturbance trajectory
W(w) { w E Now |[ w|[ s e²]
(11)
where P a desired geometry on the uncertainty trajectories. With this choice of uncertainty set, the robust constraints Uh mmmp (x., w), as defined in (9) enjoy an exact polytopic representation.
B
0
AB
B
Denotes matrix (A,
L
B
7

---

Published in Transactions on Machine Learning Research (03/2026)
Lemma 3.2 (Exact polytopic representation of robust constraint set). Under the hypotheses of Lemma 3.1 with uncertainly set (11 and P > 0, the construints
have the following polylopic represenlalion
Fx + Gu h(w)
where h(w)T [(xx - - hn T(w) The vectors 9, are such that [w], 9. w.
71 (w) F2(w) ...], and g,(w) e e 1/p-1/2g:1 gil + +g.., w, Vi.
-ellp-
The proof is provided Appendix A We are now in a position to state our main result.
Theorem 3.3 (Exact SDP reformulation). Under the hypotheses of Lemmas 3.1 and 3.2 the robust non-causal MPC problem is expressed as the min-mar problem
nn--mmpe (I, w) min
A + Bu + Ew 2 Q + Iulla.
max (w
(12)
s.t.
Fx
Gu S h(w)
Furthermore, let us denote X(x, u) Ar Bu. Then, the optimization problem 12 reformulalion
admits the conver
min 7,
722
A 0, Fr + Gu S h(w),
E ETQUE - XP (TQxX(r: u) + APw
- X (I e²)
50.
IN
B + Q.)
1/2 )²²
50.
Az u) + A Qx - 7/2
The proof is relegated to Appendix A
Remark 3.4 (Uncertainty set). The uncertainty set [1 is not necessarily uniform in time as it is ball din-diinnsional space, i.c., not all いh components of w need to be distanced equally from wk. For instance, considering the case when P = IN we then have
T E IR New
E ||w w S e²)
The above uncertainty set includes disturbances with similar mcasures of energy to w. Other similar approaches Löfberg 2003) aim to mitigate this by considering uncertainty sets such as {maxs ||v - - wx s e²), where each realization bounded uniformly in time. However, since multiple quadratic inequalities are introduced as constraints, this necessitates the use of the inexact S-Lemma (Boyd al., 1994). which inserts conservativeness. We use its exact version since only one quadratic incquality is involved in the constraints, thus allowing for exact reformulation.
3.2 Approximating with Inverse Optimization
The non-causal policy 12 can be expressed the form
n ne-rmpp (x, w)
arg min Que-rupe (r, u, w) (z.w)
(13)
with 4TT unermpp (z. w) and Quc-rmpo(x: u, w) are defined accordingly, as in the previous sections. The procedure
to approximate [3] with Inverse Optimization is identical to that used for the non-robust disturbance--ware
MPC of Section
2.3 and outlined by Algorithm [ The only difference lies in the expert policy used; in
this context, policy 13) used instcad of 77). One key difference with ( that (13) requires solving a
semidefinite program -instead of a quadratic one- so we can expect greater computational improvement, albeit potentially at the expense of reducing the quality of the approximation.
8

---

Published in Transactions on Machine Learning Research (03/2026)
By combining Q-function
(3)
and
7
arrive at the convex optimization program whose solution is the fitted
min 0
Σ (o 0(a) min uEl((a))
Qa u)
(14)
s.t. 07 x " wi+1),
where the labels úh are the in-hindsight optimal inputs computed by the min-max problem S parallel can be drawn between the the labels 0
An interesting
dilemma and the robustification when computing
Remark 3.5 (Exploration vs exploitation). When looking at the exploration/exploitaaie dilemma competitive game between two conflicting objectives, nole that a similar trade-off exists in the min-max MPC (8) that is controlled by the uncertainty radius This trade-off allows us to take into account disturbance trajectorics different than the ones observed, a key feature that is addressed by exploration RL and hence helps with generalization. This hypothesis is also confirmed by our numerical results in Section and with additional experiments in the Appendix.
Remark 3.6 (Computational ional considerations). While the proposed Robust MPC is semidefinite program (SDP), its complexity is only dependent on the planning horizon N and the nominal model dimensions (Na, nu), and does not scale with the dataset size T. Furthermore, the action improvement step can be computed independently for each data point, and is therefore trivially parallelizable. However, the IO distillation problem is also an SDP, with an LMT constraint for each sample; thus, the problem TO problem complexity does scale with the dataset size T. While modern solvers can handle large SDPs quite efficiently, we recognize that after a certain size, the IO problem might become intractable; this case, iterative optimization methods may be employed (e.g., stochastic gradients (Zattoni Scroccaro et al. 2025), or block coordinate descent (Long et al. 2024 ), which scale graccfully while retaining the thcoretical guarantees of the convex landscape.
4 Numerical Experiments
In our numerical analysis, we focus on two domains, the quadrotor environment from safe-control-gym (Brunke et al. 2 and the MuJoCo control benchmark (Todorov et al. 2012). Additionally, we include detailed ablation studics for two more experiments found in Appendix Section the control of the lincarized dynamics of a fighter jet (Safonov al. 1981] and a noulinear temperature control problem.
4.1 Quadrotor environment
We conduct experiments in a nonlinear quadrotor enviromment from safe-control-gym (Brunke el al. 2021 evaluate our approach in comparison with two RT algorithms: Proximal Policy Optimization (PPO) (Schulman et al.) 2017) and Conservative Q-Learning (CQL) (Kumar al., 2020p. Both are model-free, with CQL falling under the offline RL paradigm and PPO being an on-policy algorithm.
1 and
Environment specifications: The quadrotor environment. consists of 6-dimensional state space and two control inputs. The objective is to reach a fixed goal state starting from a randomly sampled starting position while keeping the quadrotor stable under an unknown external force that acts as a disturbance. This force consists of sinusoidal signal of random phase with additive Gaussian noise applied to the body of the quadrotor. The environment has a nonlincar dynamical system, assumed be known, with the minimum and maximum episodic cost being and 300, respectively.
Experimental Setup: before giving it to the MPC policies. In the following experiments, we denote an MPC policy that is oblivious to the external sinusoidal urstuee by MPC (obl), and with MPC (f-dst), we refer to an MPC that has the full information of the future disturbance trajectory. In our evaluations, we trained the PPO agent with 3M environment steps, which we refer to as PPO-3M, and the CQL agent for 50k iterations, where we observed convergence in performance. Both 10 and CQL agents are trained with the same dataset generated by an MPC (obl) policy with 25-step horizon.
We linearize the dynamics around equilibrium point to form nominal model
9

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: Distribution (mean + standard deviation over 4 seeds) of last-epoch scores of different baselines in the MuJoCo benchmark. All datasets are "medium". The "Datasct" column represents the performance of the trajectories used for training. The sullix in the environment name (e.g., 5K, 10K, 1M) denotes the amount of data used, obtained by taking the first N chunks of the dataset.
Environment
Dataset
10
IQL
CQL
COMBO
MОPO
TD3BC
hopper 5K
47.2
45.9
46.8+3.6
49.0+4.3
50.9+1.1
4.1+4.0
12.549.8
hopper 1M
44.3+11.6
51.7++0.9
65.7+8.1
59.1+4.11
84.7+9.3
62.8+38.1
60.8+3.4
walker2d 10K
65.9+17.9
42.2
54.447.0
51.027.0
40.7+22.7
5.019.1
1.1110
walker2d 1M
62.1+23.9
71881.9
81.1+2.6
83.6+0.5
83.9+1.9
85.4+2.9
84.422.1
Table 2: Distribution (mean + standard deviation) of best-epoch scores of different baselines in the MuJoCo benchmark. All datascts are "modium". The "Datasct" column represents the performance of the trajectories used for training. The sulfix in the environment name (e.g., 5K, 10K, 1M) denotes the amount of data used, obtained by taking the first N chunks of the dataset.
Environment
Datasct
10
IQL
CQL
COMBO
MOPO
TD3BC
hopper 5K
47.2+19.2
83.8
90.517.1
72.3#1.5
68.427.1
26.218.4
32.6#0.9
hopper 1M
44.3111.6
60.813.2
95.412.1
89.312.8
100.210.4
93.1127.9
77.0
walker2d 10K
65.9+17.9
72.1+1.4
73.9+1.6
67.8+2.2
76.3+3.0
16.9+4.6
5.440.9
walker2d 1M
62.1423.9
77.5
87.4
87.7
87.440.6
92.510.8
87.4
Figures 1 and show our comparisons and ablation studies in the quadrotor environment. Tn all six figures, T denotes the dataset length, N is the MPC horizon, and is the lookback horizon. 10-RMPC* is the p-tuned policy. Unless stated otherwise, the default values of N and H are set 25 and 2, respectively, and each evaluation of agent is performed with 20 different starting points. To normalize the effect of the randomized initial starting points, in both figures, only report the steady-stat costs. The dashed lines indicate the median values, and the tubes contain the range between the 20th to 80th percentiles of the costs, if not stated otherwise.
Comparisons: In the left plot of Fig. 1 we compare the episodic cost histograms of four agents evaluated with 20 different initial conditions. Our evaluations show that IO-RMPC yields significantly lower costs even with a limited dataset of T = 3, 000 samples. The center plot of Fig. D shows a comparison of the IO-RMPC policy against various CQL agents. Although CQL converges to IO-RMPC performance, it requires an order of magnitude more samples. Finally, compare the MPC performances with the PPO agent. Although MPC policies are only given linear nominal model of the environment, starting with the 15-step horizon, they surpass the PPO performance.
Additionally, we analyze the effect of the uncertainty radius p and lookback horizon H. The center plot of Fig. [ indicates that robustification of the TO-MPC policy, up to some value of p, improves performance even in the absence of disturbance bias. This behavior also present in the left plot of Fig. p where the IO-RMPC policy even surpasses the MPC (f-dst) policy. We posit that this is due to the fact that robustifying also helps with model mismatch between the actual dynamics and the nominal one. Finally, we perform an ablation on the lookback horizon of the IO-MPC policy shown in the right plot of We observe that when the parameter H is set to 2, 10-MPC almost recovers the performance of the inllinformation MPC (f-dst) policy, whereas a further increase in H degrades performance.
Ablation studies:
Fig. 2
4.2 MuJoCo benchmark
Next, we compare TO agents with widely used model-based and model-free offline RL algorithms within the MuJoCo control benchmark Todorov et al.) 2012). In these experiments, we employ a model-free version of the 10 agent, where the actions कr in Algorithm are directly taken from the dataset. The augmented
3 Defined as the last 40% of data points of trajectory.
10

---

Published in Transactions on Machine Learning Research (03/2026)
COL (T=3K) 10-RMPC* (T=3K)
PPO-3M
COL
RMPCY (T=3K)
MPC (1-dst)
PPO-3M
CQL(T=3K)
episodic cost (ss)
episodic cost (ss)
1.0
log density
150
10
100
50
101
150 200
episodic cast
dataset size T
MPC horizon N
Figure 1: Comparisons of several agents in the quadrotor environment. Left: The cost histogram of the offline IO and CQL agents and online model-based MPC and model-free PPO-3M (trained with 3M environment steps) agents. Center:
The cost. distributions of CQL agents trained with 4 seeds on various dalaset. lengths compared to a single 10-RMPC policy trained with 3000 samples. Right: Comparison of the cost distributions between oblivious and full disturbance MPC policies against the model-free PPO agent.
(T=3K)
MPC (abl)
10-MPC
MPC (f-dst)
I0 MPC MPC
MPC (tdst)
IM-R
10-MPC
MPC (ebl)
episodic cost (ss)
episodic cost (5s)
40
0.03
log density
35
10a
30
25
30
50 60
70
10
16
episodic cost (ss)
Uncertainty radius p
lookback horizon
The cost histogram of IO and MPC agents with 25-step horizon. Center: The cost distribution of IO-RMPC policy with different values and IO-MPC policy with the same horizon N. The tube contains the range from the 40th to the 60th percentiles of the costs. Right: The steady-state cost distributions of the 10-MPC policy with various look-back horizons (H) against MPC policics. The tube contains a narrower range from the 45th to the 55th percentiles of the costs.
Figure/ 2: Ablation studies of MPC and 10 policies in the quadrotor environment. Left:
Table 3: Comparison of parameter counts and per-epoch wall-clock time (elapsed time per training epoch) in walker2d across algorithms, Wall-clock time is measured
NVIDIA GeForce RTX 3090 GPU.
10
IQL
CQL
COMBO
MОPO
TD3BC
Parameters
9,390
286,214
414,732
1,320,672
1,123,296
215,814
Train time (1 epoch) 1.24s
160.97s
418.11s
479.86s
191.55s
96.07s
state 6(2;-44; u,-4x) includes the last four state-action pairs, the cross-products of state features, constant. bias term, Radial Basis Function (RBF) features over the state, and the state sinusoidal terms. The latter augmentation is motivated by the periodic nature of the largeted tasks in robolics.
We use the dataset from the D4RL repository (Fu et al. 2020 to train 10 agents and offine RL algorithms. We employ an itcrative version of the IO algorithm, using gradient- basod optimization to minimize the objective function in Equation (2). We trained each algorithm with four different. seeds and evaluated the agents after each epoch using 40 different seeds throughout the training process. In addition to experiments with the full datasct (1M samples), we also evaluate low-data regimes by restricting the training set to the first 10K samples for walker2d and the first 5K samples for hopper. We report the evaluation scores at last-epoch of the training in Table 1 and best-epoch evaluation scores" P across training in Table 2
Experiment setup:
We excluded the RBF features the low-data regime runs the TO agent. See Table Appendix for details.
11

---

Published in Transactions on Machine Learning Research (03/2026)
We note that selecting the first 5K or 10K samples is arbitrary, and that the average dataset score is largely unchanged relative to the full dataset. We obtained the scores for the offline RT, algorithms by running the implementations provided in the OffineRL-Kit repository (Sum 2023), which match the originally reported scores when the algorithms are executed on the full datasct. See Appendix B.1 for a detailed study with the 10 agent across varying sizes of uniformly sampled training sets.
The best-epoch scores provided in Table demonstrate the expressiveness of the 10 hypothesis class in the low-data regime of both the hopper and walker2d environments. In that regime, the IO agent achieves competitive scores compared to widely used model- and model-free baselines. Using the last-epoch scores in Table 1 we show that the IO agent attains dataset-level performance when trained on the full dataset while using an order of magnitude fewer parameters; Table 3 reports parameter counts and per-epoch wall-clock time (measured in walker2d) across algorithms. This outcome is expected, since the 10 agent does not employ the action improvement step proposed in Section 3 in the D4RL experiments.
We argue that the successful performance of the 10 algorithm with such a low number of parameters is due to the inherent richness of the I0 hypothesis class, combined with a convex optimization loss function that allows us to provably reach the (in-sample) global optimizer during the training phase. Furthermore, due to the inherent simplicity of the proposed policy class, the IO algorithm is able to generalize with significantly fewer samples.
In these experiments, we refrain from running our proposed 1O-RMPC agent that employs the action improvement step, since constructing a nominal model for MuJoCo tasks, required for the MPC experts, is a task that is inherently difficult and beyond the scope of this work. Nevertheless, our experiments with the plain IO agent revcal promising and competitive results in MuJoCo control tasks. These results underscore the substantial potential of IO-based algorithms within the RL or IL contexts, especially in scenarios with limited data. Extending the RMPC-based action improvement step to deal with more complicated dynamics remains an avenue for future research.
5 Concluding Remarks, Limitation, and Future Directions
In this work, we presented a convex and robust offline RL framework that utilizes a nominal model and in-hindsight information to learn an optimal policy. Through empirical evaluations, we showcased that our proposed algorithm can recover the performance of non-causal agents with complete environmental knowledge, while at the same time significantly outperforming RL algorithms in the low-sample data regimes (both online and ofline). We further remonstrated that the IO framework, leveraging its expressivity and convexity properties, effectively recovers teacher-level performance in challenging MuJoCo offline control tasks. Our results show that IO yields performance comparable to established reward-driven basclines, particularly in low-data regimes, while employing orders of magnitude fewer parameters than its competitors.
We also find essential to mention some of the inherent limitations of our approach. While the proposed quadratic hypothesis class, when paired with appropriate features, has demonstrated sufficient expressiveness in the control enviromments examined within our numerical studies, for more sophisticated lasks, additional steps can be required, such as applying kernel tricks, as was done for some of the MuJoCo experiments, or employing nonlinear state embedding. Another drawback of our approach is the reliance of our robust MPC formulation on a nominal model. This requirement can become impractical for complex environments where approximating a nominal model is challenging. However, these limitations are not inherent and can be potential avenuos for future rescarch, including topics such as:
(i) approximating non-causal policies by utilizing in-hindsight information in real-time, using tools from Online Convex Optimization; and
(ii) extending the robust min-max optimization (RMPC) framework to off-policy and offine RL settings
As we conclude, we position our approach as a step towards bridging the gap between robust control and offline RL, offering a particular applicability in continuous control tasks with substantial distribution shifts from training to test and also in environments where the availability of training data is limited.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Acknowledgements
The authors would like to thank the reviewers for their useful comments. This work was partially supported by the Horizon Europe Pathfinder Open project REL1EVE-101099481 and by the European Research Council (ERC) project TRUST-949796.
A Technical Proofs
A.1 Proof of Lemma 3.2
The original constraint expresses a row-wise inequality. With the parameterization [GyEw], = g. w, the incquality Fr + Gu h(w), vw W(w) is equivalent to solving the following optimization program for every i:
(w) = max w
{
w.: |w w w iipse?
To that end, let w= p-1p1/?(w w). Then the above becomes
9 (w) max { (eP-1/2 + w) || S <1}
The maximization of linear function on the unit disk has an analytical solution and that is g (w) = e p 2/2 gi + g w
By putting everything together we conclude the proof.
A.2 Proof of Theorem 3.3
The program
12 follows directly by combining the results of Lemmas 3.1 and 3.2
Let us denote the inner
maximization as
j(u)= = max wEW(w]
|Ar + Bu Ewl Q~ + |
and its corresponding Lagrangian
()(A.u,w) = || Bu + Ew| Q- 1|u1/2 ~((lW w| g²)
A After some manipulations and rearrangements, we have
= (w, (ETQUE (W. AP) w) +2 (Ax + Bu) + APw, w)
| + Bu 2 Q. + |lull2.. - (1w113 e²)
Let us introduce the following notation
A(A) = ETQ CTQ AP
M(A, u) ETQ (Ar + Bu) + XPw
(A) = À (1 [w| e²
v2(u) | + Bul 2 Qx
+ ||u|| Q.
v(A u) = (A) + (1)
The dual of this problem is then
M(AA u) +
d u) = max C,(A,
if A(A) 0 and u)) (T (T-A())A(A))- =0 +00, otherwise
13

---

Published in Transactions on Machine Learning Research (03/2026)
Strong duality holds due to the S-Lemma Boyd & Vandenberghe (004). Therefore, J(u) minx20 (A, u).
Now consider the following epigraph reformulation
J(u)
min Am
Y1 + u2(
s.1.
A O,
A(A) x
M(), T)TT (I - A(A)A(A)) =0,
M(A u) ((A) u) (1)
The last three constraints can be cast as an LMI using the non-strict Schur complement Boyd et al. 1994 and we have
(() = min A,71
"1 + v2(
s.t.
X A20 O, A((A)
(A, u) (A) - 71
x 0
Therefore the overall robust MPC problem can now be written as mina {J(u): : Fr + Gu S h(w)). In order to write this the standard SDP form, we will have to use another epigraph reformulation, that of uy))):
min A,11.72
?1 + 72
8.1.
O,
A(A)
M(,, u)
x 0,
* (A) - 71
Fr + Gu & h(w).
AA Bull 2 Q.
||u| 1/
The last constraint now be written 72 (u, (BTQ.. Q.) u) + 2BTQQx u) + || which can be expressed as the LMI:
IN
(BQ + Q.) 1/2
0
2 (BT u) A Q. -
72
Hence, by putting everything together we arrive that the original problem
12 is equivalent to:
min
71 + 72
8.b.
A2 0,
Fr + Gu < h(w),
ETQ - XP
ETQ, (Ax + Bu) + APw
71 ||w||2 o²)
0,
IN
BBQ - Q.) 1/2 u
$ 0.
2 ABTQ, u) + || 12_
-?2
We have arrived at the formulation in the Theorem statement, and as such, we conclude the proof.
Additional Numerical Experiments
Besides the numerical experiments in Section here we include more details and numerical results for the MuJoCo experiments of Section 4.2 and we further include two more examples that enable us to study our approach in more detail.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Table 4: TO hyperparameters. The suffix in the environment name (e.g., last, best) denotes the runs
produce the respective scores.
Environment
epoch
Ir
Ir decay
RBF
walker2d 10K (last)
400
0.05
0.975
Yes
walker2d 10K (best)
100
0.05
0.985
No
walker2d 1M
400
0.05
0.975
Yes
hopper 5K
100
0.05
0.9625
No
hopper 1M
100
0.05
0.95
Yes
Table 5: The IO agent performance in walker2d-medium using a subset of uniformly sampled datasets.
Datasize
Last
Last. 5%
Best
Best 5%
10K
40.7#2.0
41.5++0.7
55.0=1144
49.9+0.55
50K
50.2:1..
48.910.7
63.2=1.9
57.6+0.8
250K
53.0=1..6
52.4+1.2
68.0=3.2
62.4+0.6
1M
71.8+1199
70.8++0.6
77.5+0.8
75.6+0..3
B.1 MuJoCo additional results
Here, we provide details of the runs reported in Section 4.2 and ablation studies on the D4RL benchmark.
We obtain the scores in Table 2 and Table 1 by running each algorithm for 100 epochs with 10,000 gradient update steps per epoch, except for MOPO (300 epochs) and the full-dataset (IM samples) IO experiments (400 epochs). Table T reports the number of epochs used for the 10 experiments, along with other hyperparameters, across all dataset configurations.
In the experiments reported in Section 4.2 we use a fixed chunk from each dataset: the first. 5K samples for hopper and the first 10K samples for walker2d. Additionally, Table 5 shows how 10 performance scales with the size of a uniformly sampled training set in walker2d. We observe that both last-epoch and best-epoch performance improve as the dataset size increases. Moreover, the walker2d score obtained using the first 10K samples (Table
close to the score obtained using uniformly sampled 10K subset..
In addition to the ablation on dataset size, we compare the performance of the IO agent across two different quality levels of the walker2d dataset, namely "medium" and "expert". Table reports the scores for both settings. Comparing the "expert" and "medium" results, we observe that the performance of the IO agent scales with the quality of the input data. The best-epoch scores show that the agent is capable of exceeding the teacher in both cases (110 for "expert", 77 for "medium"). However, the "medium" dataset shows significantly higher stability at the end of training (a smaller gap between best and last), suggesting that the "medium" distribution may be casier for the quadratic hypothesis class represent robustly without. overfitting.
In Figure 3 show the evaluation scores at each epoch for all algorithms, along with the dataset average, for both environments (i.c., hopper and walker2d) in the low-data and full-data regimes. The figure corresponds to the runs reported in Table 1 For ease of visualization and comparison, although we run MOPO (3M steps) and TO (see Table 4 for more than 1M steps, we plot only the first 1M steps (100 epochs). Overall, the results suggest that, in the low-data. regime, the 10 agent performs competitively against the best offline RL baselines, while in the full-data regime it can exceed the teacher (dataset average) score.
B.2 Linear fighter jet
We consider the regulation the unstable dynamics a six- eix-dienniionl fighter jet (Safonov et al., 1981 with additive unknown disturbances W = (t; wp) + U141, where fm has sinusoidal component with random phase Wp ~ 20 x/2) and bias term, and ii N/ E.). As the dynamics are given and linear, the nominal model fo(2, u, w) Ax Bu + Fw coincides with the true dynamics f. Initial conditions are sampled
15

---

Published in Transactions on Machine Learning Research (03/2026)
Table The 1O agent performance walker2d with full dataset (1M samples) across "medium" and "exper" data qualities.
Experiment
Last
Last 5%
Best
Best 5%
walker2d-expert
77.018.2
77.5
109. 77.5:10.8
108.5=0.2
walker2d-medium
71.81:199
70.8:0..6
75.6:10.3
Hopper
Hopper 1M
Detnset 10
COMBO
MORO TO38C
Datase
COMBO
MORD TD3RC
200
100
Normal laad Syplocde Rarward
Noraalze Epicods
Step
Step
Walker2d 10k
Walker2d 1M
Normmal Reward
Normalted Spwicaee Bonsard
2.24
2.40 0.54 0.31
0.2 0.61 3.00 0.8M
Figure 3: Normalized episodic reward of all agents, including the dataset average (gray dashed line), across 1M steps (100 epochs): hopper using (top left) the first 5K samples and (top right) the full dataset, and walker2d using (bottom left) the first 10K samples and (bottom right)) the full dataset. Thick curves represent the mcan reward, and the transparent curves show the reward of individual sceds, averaged over 40 evaluations al each epoch. The curves are smoothed for clarily.
randomly as Fo (00 0.146). Further, we impose that the state be constrained in {a R | S 1} and
the input in {u R² |u'| 2, |u2| < 3}. We select the 10 features as 6(Xin ui:)) = (It, 1, Wt-1: w.).
The dynamics of the fighter jet Safonov al. (1981) have been discretized with sampling time of 0.035 s, resulting in the following discrete-time system matrices:
0.9991
-1.3736
-0.6730
-1.1226
0.3420
-0.2069
0.1457
-0.0819
0
0.0000
0.9422
0.0319
-0.0000
0.0166
0.0091
0.0072
0.0035
0
0.0004
0.3795
0.9184
-0.0002
0..5188
0.4612
0.4085
0.2893
1
A
B
E
0.0000
0.0068
0.0335
1.0000
-0.0136
0.0096
0. 0.0052
0.0037
0
0
0
0
0
0.3499
0
0.6501
0
0
0
0
0
0.3499
0
0.6501
16

---

Published in Transactions on Machine Learning Research (03/2026)
As mentioned in the main body, the disturbances are w1+1 fw wo) + 0141, where is Eu), wo 410, //2,, with
fw wo) = [
0.55 sin + wo) and En = 0
0..01
0
0.01
0.001
The cost parameters are selected as Q = Q. diag(1, 1, 10 102, 10 1,1) and Qn = 12, and the MPC horizon is N = 20.
Approximating NC-MPC with IO: First, we want to validate that hindsight can be used to mitigate unknown disturbances. As such, we will compare the following policies: MPC (obl), an MPC that can measure only 2e at time and does not know fur as described in 4 MPC (dst), an MPC that can measure both zy and We+1 at time I, and also knows fw: and IO-MPC, the policy resulting from applying Algorithm to a dataset of trajectories obtained from MPC (obl). All 10-derived policies described in this paragraph and the next are trained with datasct containing 10 trajectories induced by MPC (obl) of length 51 cach. In the left plot of Figure 4 we have the cost histogram of c(1r, u) for each tested policy during steady state We can see that in both plots the IO-MPC policy recovers significant part of the performance of MPC (dst), both in terms of median and of variance.
A Appoxxmating NC-RMPC with IO: Using the same setup and data, we impose a distribution shift. in the disturbances during evaluation by adding a constant bias to wy; specifically, we apply th instead of wi where पे = 1o + [0.1 0.05]". We therefore compare the following: MPC (obl), as before; MPC (p-dst), as MPC (dst) of the previous section only measures W MPC (f-dst), similar to MPC (p-dst), except that it has access to W++1 instead of W1+1; IO-MPC, as before; IO-RMPC, robust MPC of the form (12) trained with the same data as 1O-MPC and equipped with P = INng and o= 10-2 It is immediately obvious from the middle and most. cost distributions of Figure 4 that imitating the robust expert yields performance benefits when faced with distribution shift, as the median performance of 10-RMPC is better than that of IO-MPC. Not only that, but IO-RMPC manages to recover the median performance of MPC (f-dst), albeit with larger variance.
Effect of uncertainty radius: We further explore the impact of the robustness parameter (uncertainty radius e) on the steady-state cost distribution across different training datasets. In the left plot of Fig. 5 we observe that increasing 2 until g" yields a consistent reduction in the time-averaged steady-state cost across different training sets. What is surprisingly interesting is that there are some datasets which, when trained with properly tuned e, can match the performance of the full-information agent MPC (f-dst). We also looked into the performance of such controllers on the entire distribution of the steady-state cost in the middle plot of Fig. 5 @ has positive impact on the entire steady-state cost distribution (and not only the median average). We also note that the non-robust controller 10-MPC coincides with the robust one (IO-RMPC) for sufficiently small g. In the right plot of Fig. 5 we freeze & = g and look at the ent steady-state cost. distributions of the three policies involved in the middle plot. We observe that even though the median performance of IO-RMPC surpasses that of MPC (f-dst), its variance across the test set is much more spread, making it more high-risk than MPC (f-dst). However, as the variance of the non-robust. TO policy is similarly wide, the takeaway message here is that robustification combats distribution shift during policy evaluation.
B.3 Nonlinear temperature control
Here, we consider a nonlinear 4-th order dynamical system that describes the heat transfer equations of two coupled heating elements (inputs) and two temperature sensors (outputs), akin to that of Park et al. (2020). Specifically, the nonlinear differential equations describing the heat-transfer dynamics are the following:
τ = a2 (Tx 21) + (TTT -=a)= + az(2z - 21) + an - -=t) + u1
π = a1 (Tx - x2) + +zz(Th - x²) + az(x1 - x2) + ar(xt - =²) + bzuz
(16)
T = Ii 33
T = x2- IA
Defined as the last 40% of data points of each trajectory.
17

---

Published in Transactions on Machine Learning Research (03/2026)
log prob (55)
log prob (ss)
log prob (ss)
10
2
10
10-PP
10 MPC
MPC (7-d50)
MP C (dst)
MPC (p-dst)
Crob))
20
60
50
100
cost
cost
cost
Figure 4: Steady-state cost distributions (log-log scale) over 100 trials of the experiments described in
Section B.2 Dashed lines represent the median values. Left:: MPC policies vs TO-MPC .Center: Difference in performance between the robust and non-robust version of 10 policies when faced with distribution shift. Right: Performance IO-RMPC vs MPC policies when faced with distribution shift.
200
cost
140
120
MPC (f-50
ID-RMPC
(ost/ss)
150
10-RMPC
log prob (55)
10
100
80
ID-R MPC (29)
MPC (f-d5i)
10-M
10
10
10
40
80
100
120
140
Uncertainty radius
Uncertainty radius
cost
Figure Additional experiments as described in Section B.2 Left: Time-averaged steady-state cost for different controllers trained with 50 different datasets and for varying @ solid lines indicate the median values, and the tube indicates the range from the 5th the 95th percentiles. Center: Steady-state cost distribution for different, controllers trained with 1 dataset and for varying @; the tubes consist of the 20th to the 80th percentile range from 100 trials, while the dashed lines represent the median values. Right: Steady-state cost histograms for optimal o" over 100 trials of a single controller realization; dashed lines indicate the median values.
250
250
I0-AMPC
10"
wil
200
cost(ss)
200
(ss)
log prob (55)
150
20
10-MPC (1)
10-MPC (2)
10
20
40
60
80
cost
B.3 Left: Steady-state cost distribution for different controllers trained with dataset over 100 trials: we vary the size of H and the effect the bias term has. Center: Steady-state cost distribution for different controllers trained with dataset for varying & over 100 trials. Right: Stcady-stato cost histograms for the policies described Section B.3 over 200 trials a single controller realization. all three figures, dashed lines indicate medians, and in the first two, the tubes consist of the range between the 20th and 80th percentiles.
Figure 6: Experiments of Section
18

---

Published in Transactions on Machine Learning Research (03/2026)
Q1
02
03
0.4
b
b2
Tc
Th
4 10-3 3
5.1
10-11
7.3
10-3
10
0.011
0.006
18.3
2
Table 7: lumped-paramette coefficients of system
15
with outputs y1 = Iz and U/ = r4. The parameters d1, 02, 03,04, bj, b2, To, Th, are lumped-parammterr coefficients that can be summariied Table 7 We assune full state feedback. The ambient temperature Too is constant throughout each trial, but randomly sampled from a uniform distribution Txc ~ 4[18 28), and is subjected control objective is for the outputs y to track the temperature setpoints T 55 and r2 = 45"C, with Q. = Q, = 12 and Q. diag(1, 0.5). To obtain the nominal model fo, we linearize (15) around (æ, ū) which corresponds to the steady-state solution of y r, and then discretize with a sampling rate of 10s. As such, here the resulting nominal model fo used for the MPC controllers differs from the true nonlincar dynamics f. Due to this, the in-hindsight disturbance trajectorics contain terms that stem from model mismatch:
additional Gaussian noise U++1 (O 1) before entering the nonlinear dynamics. The
Wt+1 = E (f( Ue Too + m(+1) - 3 Adx, - Bou)
where ōz;; = 2z Iz, ou, u are its zero coordinates, on which our policies operate.
Similarly to before, we want to evaluate the performance Inverse Optimization derived policies, in both the robust and non-robust settings. Specifically, we will investigate the performance of the following policies: MPC, naive MPC with the assumption that Txo = E[Tx]] = C3" I0-MPC (1), an 10-derived policy akin to (6) with feature map ((1 u1:t) (ōz,, 1, We- 12); 10-MPC (2), like 10-MPC (1), but with no bias term and H = 8, thus ((x1,, u1:1) = (ōx,, Wt-ra); IO-RMPC (1), the robust counterpart to IO-MPC (1), equipped with P = IN and e == 70. All TO-derived policies resulted from the same dataset, containing 10 trajectories of length 51 each.
Firstly, we performed an ablation on the features: whether or not to inchude a bias term and what is the best value of H (lookback horizon). The results of this are present in the leftmost plot of Figure k Tt is evident that the optimal combination of features no bias term and // (IO-MPC (2)). When evaluating the robust counterpart of TO-MPC (2), we found that for small values of g, there was little to no performance improvement, and for larger values the performance deteriorated. We posit that given our experimental setting, IO-MPC (2) has enough expressivity that it can generalize well to unscen disturbances and capture most of the available performance, and thereby robustification has little benefit to add. On the other hand, when performing the same procedure on the worse porse-performing policy 1O-MPC (1) with a bias term in the features and H = 2, we saw that robustification led to better generalization, as the performance improved when compared with its non-robust counterpart, as can be depicted in the middle plot of Figure
Finally, in the rightmost plot of Figure D we can clearly that each TO policy surpasses the performance of the naive approach (MPC), but that is to be expected per our previous experimental discussions. The takeaway message from this figure is that robustifying can help in better generalization capabilities, and that our framework has the potential to deal with disturbance sequences that are correlated with the state, such as in cases where there is model mismatch.
References
Naman Agarwal, Brian Bullins, Elad Hazan, Sham Kakade, and Karan Singh. Online control with adversarial disturbances. In International Conference on Machine Learning, 2019.
Syed Adnan Akhtar, Arman Sharifi Kolarijani, and Peyman Mohajerin Esfahani. Learning for Control: An Inverse Optimization Approach. IEEE Control Systems Letters, 2021.
Dimitri Bertsekas. Abstract Dynamic Programming. Athena Scientific, 3rd edition, 2021.
Mohak Bhardwaj. Sanjiban Choudhury, and Byron Boots Blending MPC & Value Function Approximation for Efficient. Reinforcement Learning. In International Conference on Learning Representations, 2020.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Stephen Boyd, Laurent El Ghaoui, Eric Feron, and Venkataramanan Balakrishnan. Linear Matrir Inequalities in System and Control Theory. STAM, 1994.
Stephen P. Boyd and Lieven Vandenberghe. Conver Optimization. Cambridge University Press, 2004.
Lukas Brunke, Melissa Greeff, Adam W. Hall, Zhaocong Yuan, Siqi Zhou, Jacopo Panerati, and Angela P. Schoellig. Safe learning in robotics: From learning-bart based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 2021.
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch doep reinforcement learning. In International Conference on Neural Information Processing Systems, 2020.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rys: What is essential for offinn rl via supervised learning? In International Conference Learning Representations, 2021.
Dylan Foster and Max Simchowitz. Logarithmic regret for adversarial online control. In International Conference on Machine Learning, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasels for deep data-driven reinforcement learning. arXiv preprint #xXi:2004.002211 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning, 2018.
Gautam Goel and Babak Hassibi. Regret-optimal measurement-fecbacck control. In Learning for Dynamics and Control, 2021.
Elad Hazan, Sham Kakade, and Karan Singh. The Nonstochastic Control Problem. In International Conference on Algorithmic Learning Theory, 2020.
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning; A survey of learning methods. ACM Computing Surveys, 2017.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in noural networks. Advances in neural information processing systems, 2018.
Πya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement lcarning with implicit q-lcarning. International Conference on Learning Representations, 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. Advances in neurol information processing systems, 2020.
Jaewoo Lee, Sujin Yun, Taeyoung Yun, and Jinkyoo Park. Gta: Generative trajectory augmentation with guidance for offline reinforcement learning. Advances in Neural Information Processing Systems, 2024.
Guanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long, and Weinan Zhang. DiffStitch: Boosting offline reinforcement learning with diffusion-based trajectory stitching, In International Conference on Machine Learning, 2024.
Johan Löfberg, Minimax Approaches to Robust Model Predictive Control, Linköping Studies in Science and Technology Disscrtations. Univ, 2003.
Youyuan Long, Tolga Ok, Pedro Zattoni Scroccaro, and Peyman Mohajerin Esfahani. Scalable kernel inverse optimization. Tn Advances in Neural Information Processing Systems, 2024.
20

---

Published in Transactions on Machine Learning Research (03/2026)
Kendall Lowrey, Aravind Rajeswaran, Sham Kakade, Emanuel Todorov, and Igor Mordatch. Plan Online, Learn Offline: Efficientt Learning and Exploration via Model-Based Control. In International Conference on Learning Representations, 2018.
Jiafei Lyu, Xiu Lí, and Zongqing Lu. Double check your state before trusting it: Confidence-aware bidirectional offline model-based imagination. Advances in Neural Information Processing Systems, 2022.
Peyman Mohajerin Esfahani, Soroosh Shafieezadeh- Abadeh, Grani A. Hanasusanto, and Daniel Kuhn. Data-driven inverse optimization with imperfect information. Mathematical Programming, 2018.
Junho Park, R. Abraham Martin, Jeffrey D. Kelly, and John D. Hedengren. Benchmark temperature micrec for process dynamics and control. Compulers & Chemical Engineering, 2020.
Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offiine reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 2021.
M. Safonov, A. Laub, and G. Hartmann. Feedback properties of multivariable systems: The role and use of the return difference matrix. IEEE Transactions Automatic Control, 1981.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xrxiv preprint ar uxXiv:1707.0634/7 2017.
Yihao Sun. Offlinerl-kit: An elegant pytorch offline reinforcement learning library. bttps://github.com 2023.
Emanuel Todorov, Tom Erez, and Yuval Tassa, Mujoco: A physics engine for model-based control. In IROS, 2012.
Jianhao Wang, Wenzhe Li, Haozhe Jiang, Guangxiang Zhu, Siyuan Li, and Chongjie Zhang. Offline reinforcement learning with reverse model-based imagination. Advances in Newral Information Processing Systems, 2021.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint Xiv::11911. 11361, 2019.
Tianhe Yu, Aviral Kumar, Rafacl Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsca Finn. Combo: Conservative offline model-based policy optimization. In Advances in Neural Information Processing Systems, 2021.
Pedro Zattoni Scroccaro, Bilge Atasoy, and Peyman Mohajerin Esfahani. Learning in inverse optimization: Incenter cost, augmented suboptimality loss, and algorithms. Operations Research, 2025.
Junjie Zhang, Jiafei Lyu, Xiaoteng Ma, Jiangpeng Yan, Jun Yang, Le Wan, and Xiu Li. Uncertainty-driven trajectory truncation for data augmentation in offline reinforcement learning. In European Conference on Artificial Intelligence, 2023.
Mingyuan Zhong, Mikala Johnson, Yuval Tassa, Tom Erez, and Emanuel Todorov. Value function ap- proximation and model predictive control. In JEEE Symposium on Adaptive Dynamic Programmingg and Reinforcement Learning, 2013.
21


Reviews:
[{"review_id": "Df5zgV8aQS", "reviewer": "Reviewer_aZkk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new offline reinforcement learning approach combining a non-causal MPC control formulation and a reduction to the space of parameterized causal policies, solving for a sub-optimality loss. The optimization problems are formulated such that they remain convex under some popular assumptions on the dynamics and constraints, making them easy to solve and allowing learning policies with much fewer parameters than standard ORL methods.\n\n## Strengths:\n\n- Interesting teacher-student setup where a non-causal MPC is used as a data augmentation tool to learn a better causal policy.\nThe conclusion is transparent about the limitations of the work. This is very appreciated.\n\n## Weaknesses:\n\n- Instead of applying traditional ORL algorithms, this paper proposes to first solve an MPC problem, then fit a Q function/policy using a sub-optimality loss. As we assume nominal dynamics and a model for the disturbances, it would be useful from the beginning of the paper to have an explanation of why traditional MPC methods cannot be applied, and why using the sub-optimality loss is a good idea.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The results of the paper are formally proven under a specific set of assumptions. These assumptions would be too restrictive in some practical settings, but the work clearly states them.", "audience_yn": "Yes", "audience_explanation": "The paper presents a novel method for offline reinforcement learning with direct applications, a topic of particular interest to TMLR’s audience.", "requested_changes": "### Experiments\n\nThe main weakness is the lack of comparison with other recent offline RL works in the experiments. Although the authors use CQL, IQL and COMBO as baseline, it would be useful to use other more recent baselines, for example:\n\n- Adaptive Neighborhood-Constrained Q Learning for Offline Reinforcement Learning\n\n- MOReL: Model-based offline reinforcement learning\n\n-  MOPO: Model-based offline policy optimization. \n\n- Supported policy optimization for offline reinforcement learning.\n\nIt would also be useful to show an ablation study with different behavioral policies.\n\n### Minor:\n\n- LaTeX: ``sub-optimality loss\" should be ``sub-optimality loss'' throughout the paper\n\n- Notation for Remark 2.1, shouldn’t \\mathcal{X} and \\mathcal{U} be X and U instead?\n\n- Please clarify the definition of $\\mathcal{T}_N(A, B)$. For example, the following would be easier to understand:\n$$ \\mathcal{T}_N(A, B) = \\begin{bmatrix} B & 0 & \\cdots & 0 & 0\\\\ AB & B & \\cdots & 0 & 0\\\\ \\vdots & \\ddots & \\ddots & 0 & 0\\\\ A^{N-2}B & & \\ddots & B & 0\\\\ A^{N-1}B & A^{N-2}B & \\cdots & AB & B\\\\ \\end{bmatrix}. $$\n\n- There seems to be a missing $\\lVert u \\rVert_{Q_u}^2$ term in Equation 12", "broader_impact_concerns": "There are no ethical implications associated with this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "meWtIjvzbr", "reviewer": "Reviewer_c9t9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to improve learning RL policies from offline data by correcting for policy shift in the dataset via an MPC expert. This expert can then be imitated via IO, which the authors adapt to this offline RL setting. This two-step process is validated on a Quadrotor environment as well as the MuJoCo benchmark with different dataset sizes where is outperforms standard offline RL algorithms when only little data is available.", "claims_and_evidence_yn": "No", "claims_explanation": "Generally, the paper is well-written, the motivation is clear and the proposed method solves the problem posed in an elegant way. I also like that several ablations are included in the experiments. In fact, I believe the methodology and the Quadrotor experiments are well done and demonstrate the efficacy of this idea nicely. \n\n\nI have some concerns about the D4RL experiments, however, that currently prevent me from recommending acceptance. Mainly, my doubts center around the fact, that the IO workflow clearly outperforms the offline RL baselines on the small datasets but perform noticably worse than the RL baselines on the full dataset. In the dataset scaling results on Quadrotor, the proposed IO approach did not improve meaningfully with dataset size. Do these results mean that this method is only useful on quite small datasets? If that is the conclusion, I believe the D4RL experiments do not show sufficiently where the capabilities end and offline RL algorithms become more desirable. I assume this was done due to the comparatively higher computational cost of training the RL baselines, but in my opinion, it is vital information where the ideal application domain for this approach ends. Right now, these experiments show a siginificant limitation without exploring its dynamics. Is it possible to train at least one of the RL baselines on a few in-between dataset sizes to show at which point it outperforms the proposed IO method? Or conversely train the IO version on larger dataset sizes?\n\n\nFurthermore, I have a few smaller questions about the D4RL setup I did not quite understand:\n- How were the smaller versions of the D4RL datasets obtained? Random sampling, or was there a specific sampling procedure? \n- Why are only the top 5% of evaluation scores reported? This is an extensive filtering of the actual performance data. I believe in this case the actual scores should be included somewhere in the paper, even if it’s the appendix.\n- What is the uncertainty in Table 1? This should ideally be included in the table heading.\n- What is the reasoning behind including only 3 seeds on the D4RL experiments? The deviation in the results for the RL baselines is still very large, is it computationally possible to add more runs to stabilize the results?", "audience_yn": "Yes", "audience_explanation": "I believe efficient policy learning from a dataset is an important topic and the proposed method is an interesting solution, even if I believe the paper currently does not demonstrate the actual usecase in enough detail to support the claims. These issues do not take away from the overall interest in the approach, however.", "requested_changes": "- Include the full D4RL scores, at least in the appendix\n- Discuss the dataset scaling behavior on D4RL in some way. \n- Add a link to a public code repository (at the very latest at submission). Uploading the code with the paper on OpenReview without a link in the paper is only very technically an open source implementation.", "broader_impact_concerns": "I don’t have any concerns about the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VoPl4x0XSU", "reviewer": "Reviewer_qhfe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary:\n\nThis paper propose a novel offline Reinforcement Learning algorithm that generates better actions from old data based on MPC, and imitates expert by solving a convex inverse optimization problem. \n\n\nStrengths\n\nI think this paper is worth recommending for its clear presentation and derivations, and also for its theoretical novelty and concise framework.\n\n1. The authors give a clear description for two-step solution with convex sub-optimality loss to bridge offline RL with Imitation Learning, distilling the non-causal MPC expert \n2. The authors successfully derive the robust non-causal MPC (which optimizes against worst-case model-mismatch trajectories within a prescribed uncertainty set) as a convex optimization problem. \n3. The authors provide experimental evidence that the learned policy (which is based on recovering hidden disturbances from data and a MPC that plans for worst-case variations) is reliable under distribution shift between training and deployment.\n\nWeaknesses\n\nThis paper presents a new method clearly with both sufficient theoretical proof and experimental support. I think\nthis paper is worth recommending, but still have some concerns about its practical application: \n\n1. Rely on the assumption of linear nominal dynamics:\n\nIn order to make MPC become convex quadratic, the authors request to prepare a nominal dynamics and limit it to be linear. However, the true dynamics in many real world RL use cases is nonlinear and highly. \n\nRegarding heavily rely on  \"linear dynamics, quadratic stage/terminal costs, and polytopic constraints\", authors admit that the nominal model is challenging to handle the complex use case like MuJoCo and do not run IO-RMPC on MuJoCo (see 4.2 \"In these experiments, we refrain from running our proposed IO-RMPC agent that employs the action\nimprovement step, since constructing a nominal model for MuJoCo tasks, required for the MPC experts, is a task that is inherently difficult and beyond the scope of this work.\").\n\n2. Q-function affine in state may require heavy feature engineering for ϕ\n\nEquation (1) limits Q-function to be linear in state feature vector. In section 2.3, state s is defined as st=ϕ(x1:t, u1:t)\nand authors mentioned \"ϕ is a feature-engineering problem and lies outside the scope of this paper\". \n\nThe concern is designing feature map ϕ could be difficult and complex in practice:\nTo approximate real world true complex Q-function, ϕ may need to be very complex and high-dimensional since Equation (1) only shows Q affine in s and pushes all nonlinearity into feature map ϕ.\n\nIn section 5. authors also mentioned that \"for more sophisticated tasks, additional steps can be required, such as applying kernel tricks or employing a nonlinear state embedding\"\n\n3. Computational cost for robust MPC and IO training could be very high\n\na. In Theorem 3.3, authors write the MPC min-max problem as an SDP so that can directly solve it using convex optimization.\nSDP can introduce cubic time complexity O(n^3) in matrix dimension n when use Interior Point Methods. For each time step, an action is produced by solving an SDP. So for a long dataset with large T or high dimension in states, it will become very expensive.\n\nb. IO traning which is a convex optimization problem over data samples and parameters also easily faces high computational cost for large size of data samples and dimension. Compare to neural net SGD steps, solving such large convex program could be much slower.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper supports its claims with clear theoretical derivations and proofs. The authors also give experiments on quadrotor control and MuJoCo benchmarks that consistently show the proposed methods performing as described. (Although the authors do not run the full IO-RMPC on MuJoCo because building a nominal model is too difficult, their MuJoCo experiments could be potentially improved in the further work.)", "audience_yn": "Yes", "audience_explanation": "Readers who work on reinforcement learning, control, and optimization would be interested in this new offline RL algo.", "requested_changes": "1. Please discuss more clearly the practical limits of assuming a linear nominal model with quadratic costs and polytopic constraints. If possible, please add a small experiment or simulation to show how IO-RMPC behaves when the true dynamics are noticeably nonlinear, especially since IO-RMPC is not run on MuJoCo for this reason. (refer to weakness 1. Rely on the assumption of linear nominal dynamics)\n\n2. Please expand the study/discussion for the feature map ϕ in more complex tasks by showing a simple nonlinear / kernelized variant. (refer to weakness 2. Q-function affine in state could cause heavy feature engineering for ϕ)\n\n3. Please report/analyze the computational cost of both the robust MPC (SDP) step and the IO training step in terms of a. number of time steps and b. dimension and dataset size. If possible, please compare this overhead to neural net offline RL methods trained with SGD. (refer to weakness 3. computational cost for robust MPC and IO training could be very high)", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)