Offline Hierarchical Reinforcement Learning via Inverse Optimization
Authors: Carolin Schmidt
Daniele Gammelli
James Harrison
Marco Pavone
Filipe Rodrigues
Venue: ICLR 2025 Poster (2025)
Keywords: Offline Reinforcement Learning
Hierarchical Reinforcement Learning
Primary area: reinforcement learning
TL;DR: In this work, we propose a unified framework for offline reinforcement learning of hierarchical policies.

Abstract: Hierarchical policies enable strong performance in many sequential decision-making problems, such as those with high-dimensional action spaces, those requiring long-horizon planning, and settings with sparse rewards. 
However, learning hierarchical policies from static offline datasets presents a significant challenge.
Crucially, actions taken by higher-level policies may not be directly observable within hierarchical controllers, and the offline dataset might have been generated using a different policy structure, hindering the use of standard offline learning algorithms.
In this work, we propose $\textit{OHIO}$: a framework for offline reinforcement learning (RL) of hierarchical policies. 
Our framework leverages knowledge of the policy structure to solve the $\textit{inverse problem}$, recovering the unobservable high-level actions that likely generated the observed data under our hierarchical policy.
This approach constructs a dataset suitable for off-the-shelf offline training.
We demonstrate our framework on robotic and network optimization problems and show that it substantially outperforms end-to-end RL methods and improves robustness.  We investigate a variety of instantiations of our framework, both in direct deployment of policies trained offline and when online fine-tuning is performed. Code and data are available at https://ohio-offline-hierarchical-rl.github.io.

Published as a conference paper at ICLR 2025
OFFLINE HIERARCHICAL REINFORCEMENT LEARNING VIA INVERSE OPTIMIZATION
Carolin Schmidt' Daniele Gammelli", .James James Harrison', Marco Pavone?, Filipe Rodrigues' 'Technical University of Denmark, SStanford University," Gooogle DeepMind desasc,rodi } jdtu.. ck, {gamme pa e
ee ford edu,
ABSTRACT
Hierarchical policies enable strong performance in many sequential decision-making problems, such as those with high-dimensional action spaces, those requiring long-horizon planning, and settings with sparse rewards. However, learning hierarchical policies from static offline datasets presents a significant challenge. Crucially, actions taken by higher-level policies may not be directly observable within hierarchical controllers, and the offline dataset might have been generated using different policy structure, hindering the use of standard offline learning algorithms. In this work, we propose OHIO: framework for offline reinforcement learning (RL) of hierarchical policies. Our framework leverages knowledge of the policy structure to solve the inverse problem, recovering the unobservable high-level actions that likely generated the observed data under our hierarchical policy. This approach constructs a dataset suitable for the-shelf offline training. We demon- strate our framework on robotic and network optimization problems and show that it substantially outperforms end-to-end RL methods and improves robustness. We investigate a variety of instantiations our framework, both in direct deploymen policies trained offline and when online fine-tuning is performed. Code and data are available at thttp:: /hhio- www.ine-hiierrcceeeeeeee www-hhheerrchhcace -rl. github. io
INTRODUCTION
Deep reinforcement learning (RI.) and optimal control (OC) have made significant progress within a broad range of continuous control tasks, such as locomotion skills (Lillierap et al. 2015), dexterous manipulation (Zhu et al. 2019), and robotic navigation (Long et al.; 2018). However, most of these tasks are inherently atomic, as they can be completed by performing basic skills episodically rather than through complex multi-level reasoning. Hierarchical policy decompositions, in which multiple sub- policies are composed to perform control at successively higher levels of temporal and representational abstraction, have long held the promise to help solve such complex tasks (Barto & Mahadevan [0033]
Nachum et al.] 2018). Specifically, by defining a hierarchy of policies where higher levels influence the behavior of the lower levels, it becomes easier to train high-level policies plan over longer time scales. Morcover, approaches for OC often leverage problem-specifie structure by constructing hierarchical policies over convenient state representations. c.g., planning operational space as opposed to direct joint space control for robot manipulation (Khatib 1987 Peters & Schaal 2008).
Similarly, sequential decision-making systems operating in the real world-such vehicle routing and traffic control (Rasheed et al. 2020 Zardini et al. (022), supply chain management (Rolf et al. 2023). and power grid optimization (Duan al., 2019). among many
historically benefited
from hierarchical abstractions. Hierarchically structured policies are commonly used to (i) decompose a large optimization problem into smaller, tractable ones (Fluri et al., (209),, and (ii) combine the benefits of differently-structee optimization (Delarue et al., 2020 Gammelli et al. 2023).
sub-policies, such integrating shariing-bsse methods with direct
Nevertheless, there remains significant gap between the theoretical promise of hierarchically structured policies and their practical application to complex, real-world decision-makinn problems: previous work often relies costly online data collection, which is impractical for real-world, safety-critical systems. To address this issue, offline RL has gained attention for its ability to train policies from static offline datasets, thus avoiding the need for expensive or unsafe online exploration.

---

Published as a conference paper at ICLR 2025
Training
Offine Trajectory Data (8p, og,, Ro, 81, ox (80, Ro, Rl....)
Probiem
High-livel Dalassel
Compute high--woee action
(#oy Bay Ri: (1y---)
le max log mi ae, we) L(we)
Offline Al
84 max logpitsst *r, Rd) L(ue)
s.1. 6 (A 14)
Low lovel Action
Polly
High loss Action
m 8, u)
fa e) arg min (s,
u Tu
acA
Inference
Figure 1: We propose OHIO, framework to learn hierarchical policies from offline data. By exploiting structural knowledge of the low-level policy, we solve an inverse problem (top center) transform low-level trajectory data (top left) into dataset amenable to offline RL (top right), regardless of the nature the policy used for data collection. At inference time, the RL-trained policy provides inputs to the low-level policy (bottom).
However, offline policy learning has had a limited impact within hierarchical formulations due to two fundamental issues. First, unless assume that offline data collection is performed using the same hicrarchical policy that we intend learn, actions across hicrarchy levels may not be observable, thus hindering the direct application of standard offline RL algorithms. Second, even if the offline dataset is collected using a hierarchical policy, any modifications to the hierarchical controller or its components lead to ill- plo-posed offline RL formulations.
In this work, we propose a framework to learn hierarchical behavior policies from offline data, OHIO: Offline Hierarchical reinforcement learning via Inverse Optimization. Specifically, we solve the inverse problem, mapping state transitions (and, optionally, low-level actions) to the high-level actions that likely generated those transitions. This approach allows us to create dataset that can be used by standard offline RL algorithms within any hierarchical policy scheme, regardless of the nature of the policy used for data collection (Figure
Contributions.
Concretely, the contributions of this work are:
We present novel framework for hierarchical offline RL that leverages structural knowledge of the hierarchical policy to construct dataset amenable to off off-the-shelf offline RL algorithms.
We derive principled inverse optimization objectives solve the inverse problem both analytically and numerically, thus making our method amenable to generic policy structures and behavior policies used for data collection.
We investigate design decisions and learning strategies within our framework, such as the impact of model learning, the choice of inverse optimization algorithm, dataset characteristics, and their impact on system performance and fine-tuning capabilities.
Through experiments on robotic tasks, supply chain inventory control, and dynamic vehicle routing, we show how our framework substantially improves the performance of off-the-shelf offline learning al algorithms across a diverse set embodiments and policy structures, while providing the safety guarantees needed for safe, real- world deployment.
PROBLEM STATEMENT
We consider discounted infinite horizon Markovian problem setting in which agent interacts with a Markov decision process, M=(S,A,P.r.)) We denote the state and action at time f as 8t, and or, and S S..A are the state and action spaces, respectively. Additionally, P(s s,,a,) denotes the (prob- abilistic) state transition dynamics, r(s,,a,) denotes the reward function, and y is the discount factor. For brevity, will also refer single state transitions using and s' for 8, and 8t4 respectively.
2

---

Published as a conference paper at ICLR 2025
We will consider learning hierarchical policy consisting of two components,
(. a~
(I)
where Tu denotes a learned upper policy, and denotes fixed lower-level policy, and u is the output of the upper policy that an input the lower-level policy. Our approach generally agnostic the form of the lower-level policy, although we will identify important example cases in the next section.
We focus on the case where the dataset comprises state-only trajectories, specifically considering In this context, we assume access to approximate
(80,8) (17,8))).)877) or - (so, (9, dynamics P and reward models. These assumptions scenarios where certain elements of domain knowledge are inherently available. For instance, in robotic manipulation, having access robot arm's 's dynamics model not only standard practice but essential for operating any low-level controller. Similarly, in network decision systems, real-world algorithms often depend on deterministic approximations of system dynamics. For example, the analysis of transportation systems frequently employs simple macroscopic models derived from traffic flow theory,
reasonable, our work targets common
3 METHODOL.OGY
In this section, we provide comprehensive overview of the OHIO framework. We begin by presenting each component in general setting, followed by specific examples of their implementations. The section begins with discussion of the lower-level policies used and then moves on address the inverse optimization problem and its corresponding solution methods. Lastly, we present the overall framework.
3.1 LOWER-LEVEL POLICIES
We will broadly consider two classes of lower-level policies: explicit and implicit policies. We distin- guish between these two classes of policies, which are necessarily treated differently in policy inversion.
Explicit policies.
(Stochastic) explicit policies require only a single function evaluation, or a (s,u,e),
where is generic random variable (enabling policy stochasticity through reparameterizattoon) and f is a generic function. This class of policies is broad. It includes policies that range from simple feedback suntrollers--uchh as PID, or linear state-space controllers such as those computed via LQR-to more complex policies such as those parameterized by neural networks and learned by RL or imitation learning (IL).
Implicit policies.
Implicit policies, on the other hand, are defined as arg min f(sa,,u,
a aEA
or approximate solutions thereof. This class includes optimization-bann based policies such Model Predictive Control (MPC) (Rawlings & Mayne 2013), and implicit learning based policies such as diffusion-baan based or other implicit methods (Chi et al.) al.] 2023 Florence et al. 2022).
Let us consider a quadratic cost (negative reward) function with linearized Gaussian dynamics, where we assume the output of the upper policy, denoted uu. is a goal post-transition state. Specifically, by Taylor-expanding the reward function (yielding terms m and M) and approximate dynamics around the current state and zero action, the optimization problem becomes:
Example 3.1 pnnear--uadratii-Gaasssss
max a
E. 2
|aa m Il + I|s' 2
s.t.
s' N'~N(As -a...).
where u is the goal state, V is an arbitrary matrix that weighs satisfaction of achieving this goal state ver- satisfying other reward terms]] and A,B,c are the terms resulting from Taylor-expanding the known dynamics. The solution to this problem is computationally tractable. In particular, if the action is uncon- strained, this corresponds
variant of the linear quadratic regulator (LQR) problem, where the optimal action is a = K + with policy parameters K,k depending on the reward function and dynamics. ?
As has been noted by Gammelli ct al. 2023), this term may be learned alongside the RI, policy. We include the detailed derivation in A Appendix A.3
3

---

Published as a conference paper at ICLR 2025
3.2 POLICY INVERSION
Our framework aims to map available data to high-level action representations. In particular, we aim compute high-level trajectories 7; =
=; (00:t00/0::..987) given a dataset of lower-level trajectories 7Ü.
Lower-level policy inversion. The core of our approach is to exploit the known optimization structure of the inner problem to approximately compute the most likely high-level action inducing
each state transition and use these for offline training. We treat this problem as a probabilistic inference problem and aim to solve the regularized maximum likelihood estimation problem:
log P(S1:T [Q0:T-1.00:--- + Σ L(u))
uo:T
arg max U0:T-1
(5)
s.t.
a₂ ( Vt,
which is constructed jointly across each trajectory, and where L denotes a regularization term. We require an approximate dynamics model P. In many applications, a simple approximate dynamics model, which only needs be reasonably accurate for one timestep, is already known. This is often the case in robotics, where approximate models of the robot's 's dynamics are used for trajectory planning and motion control. other scenarios, a dynamics model can be learned from the dataset. The use of an approximate dynamics model makes OHIO novel combination of model-based and model-free RL.. leverages model information locally in time, while the RL-based component. This ensures the overall framework is not susceptible to compounding emrors from multi-step prediction. When low-level actions are observed, as in the classical offline RL setting, can bypass the need for approximate dynamics model, as discussed in Appendix A.
performance is achieved through
In practice, we will decompose this joint likelihood across time and, assuming that the policy is stochastic, solve the one-timestep problem:
ig arg max Ut
log P(s [a,,s)+++
(6)
s.t.
a (.,
In cases where the policy is deterministic and the likelihood under the policy is not well-defined, a simple alternative loss function (e.g., MSE loss) is used instead. This decomposition across time is sub-optimal, and the original joint-across-time problem has strong similarities to classical filtering and smoothing in partially- observed systems. However, we find that is effective for our experiments due to the structure of many decision decision-making problems, leaving the consideration of the full joint likelihood for future work.
Solving the policy inversion problem analytically. How is this inverse problem solved? We will first illustrate one possible approach by building on the previous example.
Example 3.2 (Analytical inverse: solving the inverse linear-quadratic problem). Here, we will continue Example 3.1 and discuss the solution of the inverse problem. Specifically, given the linearized Gaussian dynamics derived in Example 3.1:
P(s' [s,a") +c,2),
fora"=m(s.>"
whereby a" = Ku+k. k. The goal of the inverse problem to compute the most likely high-level actions u by solving Problem 6 By substituting a" into P(s' s, a²), we obtain the following likelihood function describing the objective function for our inverse problem:
(As + BKu+ Bc,
(8)
By expressing the likelihood in log terms and leveraging the fact that the fog-likelihood is concave in u, we can easily derive maximum value as:
t û= (BK)ttt (s'-))sssss (AA+++AA+BB) Bk)),
where (-) denotes the Moore-Penrose inverse. We can then use û, a", and s to compute the reward f. Ultimately, this leads an analytical solution the inverse problem via Equation 9
This unconstrained linear-quadratic setting is one the few that can be solved exactly. However, we emphasize important distinction with work on differentiable optimization (Agrawal et al. 2019) and prior work on structured policies (Amos et al. 2018): our RL-based outer policy training does not require gradient propagation through the optimization problem, and thus any method may be used to solve the inverse problem. Indeed, because the inverse problem only needs to be solved to construct dataset for offline training, comparatively expensive methods can be used.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 OHIO: Offline Hierarchical Reinforcement Learning via Inverse Optimization
Require: State transition dataset D; Optionally: approximate dynamics P. reward function r.
De 0
> Initialize high-level dataset
for e do
Compute & via 6
Low-level policy inversion
D+-DU(f) reward information is available, include in f, otherwise compute ft =r()),ā;))
end for
Solve offline RI problem using D to yield high-level policy
Solving the policy inversion problem numerically. Several methods exist beyond analytical solutions, and our framework is agnostic to the method used. For discrete high-level actions, the inverse problem can be solved by exhaustive search over u. Similarly, we can employ sampling techniques for u or zeroth-order optimization such as CEM (Rubinstein] [1997] (1977 Boer et al.] (005)) In certain cases, exact gradients may be computed through the inner problem, enabling the use of gradient-baser optimization -such as gradient descent- to solve Problem Thus, the numerical solution of the inverse problem is a considerably more general approach. More details are provided in Appendix A.4
3.3 FULL METHODOLOGY
Algorithm Thighlights the relative simplicity of our approach: it focuses on leveraging approximate knowledge of the system dynamics and reward function to derive likely high-level actions and rewards. Subsequently, we apply off-the-shelf offline RL algorithms- -including behavior cloning as a special case-on this dataset.
4 RELATED WORK
Our work is closely related to previous approaches to learning control within hicrarchical poli- cies (Ichter et al. 2018 Bansal et al. 2020: Xia et al. 2021 Lew et al. (023) and offline RL within
these settings (Le et al., 2018 Gupta et al. 2019 Zhou et al. 2021 Ajay et al. [202] Rosete-Beaa
ct al. 2023), providing a way to train general general-purponn purrpose hierarchical policies from offline data.
Hierarchical and Bi-Level RL. Hierarchical IL jointly learns high-level policies and low-level con- trollers from optimal demonstrations (Le 2018 Gupta et al. 2019]. These methods have two main drawbacks: (i) they typically learn high-level actions in the form of sub-goals, thus, in the raw observa- tion space, and (ii) they require oracle trajectory data. Our method alleviates both of these drawbacks by (i) learning high-level policies in intermediate (potentially lower-dimensiona)) representation space, and (ii) leveraging offline RL methods to learn from sub-optimal data. Another class methods uses offline RL to train the high-level policy in learned latent spaces (Zhou al. 202 Ajay et 2021I. However, the policy used to generate the offline datasets may not match the hierarchical structure that interested in learning. Therefore, prior work typically formulates potentially complex, multi-step training schemes for the individual policy components, e.g., unsupervised trajectory autoencoders combined with hindsight relabeling collect dataset with the inferred high-level latent action and the respective reward (Rosete-Beat et al. 2023). To address these limitations, in our framework, we leverage approximate knowledge of the system dy dynamics compute the most likely high-level action from raw trajectory data, thus avoiding the misalignment caused by intermediate objectives that do not necessarily correlate with the downstream task (e.g., reconstruction losses within generative models).
In robotics, numerous strategies have been developed for learning control with bi-level formulations that leverage traditional planning methods as inner components. For instance, prior work focuses on decomposing the overall policy into a high-level learned policy that generates waypoint-lik like represen- tations for a low-level motion planner, e.g., based on sampling- based search (Ichter et al. 2018 Xia et al. 1021), model-based planning (Bansal et al. 2020), or trajectory optimization (Lew et al. (023). Within this context, the high-level policy is typically ly learned through either imitation oracle waypoint selection strategies online RL. Analogously to previous methods, our approach uses the output of higher-level, learned policy in a hierarchical structure. Crucially, however, we focus on solving complex control tasks from offiine data by constructing datasets amenable to tff-the-shelff offline RL algorithms.
5

---

Published as a conference paper at ICLR 2025
Offline RL and Learning from State-Only Demonstrations. Lastly, work is closely related to methods for learning from observations (LfO), by introducing a framework for offline RL from (potentially) state-only demonstrations. Distribution matching methods represent a principled approach to LfO (Boborzi et al. 2022). (Kim et al. 2022) by interactively estimating and minimizing the discrepancy between two stationary distributions: one generated by the expert and another by the learning agent. However, traditional approaches based on distribution matching typically require online interactions with the environment, with limited applications tasks where exploration is expensive or unsafe. Moreover, methods that focus on learning from offline data typically cast LfO as an imitation learning problem, whereby the goal is to imitate the behavior of expert policy and, thus, the overall performance can be limited by the quality of the data collection policy (Zhu et al., 2020b), (Qin et al. 2023), (Bewley al. 200T). To address these limitations, our work introduces a new offline LTO approach to recover optimal policies from (potentially sub-optimal) state-only demonstrations.
EXPERIMENTS
In this section, we demonstrate the performance and broad applicability of our framework, OHIO, on two robotics scenarios (Section 3.1) and two real-world network optimization problems (Section 5.2) In particular, in the robotics scenarios, we evaluate a practical application of OHHIO, where the high-level policy is learned through RL, and the low-level policy is an explicit policy, i.c. traditional, non- non-learned controller. In the network optimization scenarios, we demonstrate the performance of a particularly relevant instantiation of our framework, in which the low-level policy is optimization-based.
The goal of our experiments is to address the following key questions: (1) Can OHIO successfully recover hierarchical policies from datasets collected by arbitrary (i.e., non-hierarchica. behavior policies? 5.2) (2) How do different inverse methods compare in performance? (5.1.I) (3) Does
OHIO enable effective offline RL even when the dataset is collected with different or unknown low-level configurations? 5.1.2) (4) How does OHIO compare to traditional hierarchical RL? (51.2)
(5) Does OHIO improve scalability and robustness relative to end-to-ent approaches? (5.2)
Benchmarking, To isolate the contributions of OHIO, our analyses include the following comparisons: (i) OHIO with known low-level policy, (ii) a traditional hierarchical RL approach with known low-level policy but without the dataset reconstruction provided by OHIO (i.e., "Observed State " baseline, which selects the next observed state as the high-level action), (iii) a hierarchical RL formulation (i.e.. "HRL") in which both high-level and low-level policies learned, and (iv) "flat" end-to-end approaches (i.e.. -End-to-Endd "" with minimal architectural differences in the RL policy.
Experimental design. The learning algorithms used in this section include both off-the-shelf offline IL RL approaches (e.g., IQL (Kostrikov et al. 2021), CQL (Kumar et al. (020 and behavioral cloning (BC) algorithms. Dataset collection follows standard practices specific to each environment, such as (pre-trained) RL policies in robotics scenarios and dorain-driven optimization or heuristic-based policies in network optimization. For consistent comparisons across datasets, we normalize scores to range of 0 to 100, calculated as normali ized score 100 online RL formance (robotics) and normal zzd score
score
100 oraale performance (network-optimiization)
score
5.1 ROBOTICS
In this section, we focus on two distinct robotics scenarios: the first, traditionally solved using an end-to-end approach, is detailed in Section 3.1.1 the second scenario typically addressed with a hierarchical reinforcement learning framework that includes non-leamnd lower-level controllers (i.c.. RL policies guiding operational-space controllers for manipulation tasks), as discussed in Section 5.1.2
5.1.1 GOAL-DIRECTED CONTROL
We evaluate OHIO in a non-linear system using the Reacher task (Tunyasuvunakool et al., 2020), where the objective is to control two-jointed robotic arm to move its end-effector to randomly positioned target. This task is particularly suitable because it allows us to derive low-level policies with analytical solution to the inverse problem, facilitating the comparison of different inverse methods methods--specffcaall numerical approaches like gradient-bart based optimization versus analytical methods. Additionally, since this task is typically solved using end-to-end approaches, it serves as

---

Published as a conference paper at ICLR 2025
Table 1: Normalized score the reacher task comparing BC performance within End-to-End,
"Observed State" and OHIO formulations, including the choice of algorithm for the inverse problem
OHIO
OBSERVED STATE
DATASET
NUMERICAL
ANALYTICAL
REG. ANALYTICAL
END-TO-ENT TODDD
HR DATASET
97.1110.2
98.2. 3.2
97.1
0.05 0.04
95.41114.2
E2E DATASET
99.2.1550
94.8=22.0
95.4:123.4
99.3116.0
E2E- 10C DATASET
95.3+24.5
91.8=27.1
94.6+25.3 93.3+26.9
E2E-105 10S DATASET
98.4#17.8
93.5=23.5
an effective demonstration of OHIO's capability to transform "flat" (non-hierarchical) offline dataset into one suitable for offline hierarchical reinforcement learning.
Datasets. Our objective is to learn policy from dataset of non non-hierarchhcal demonstrations (i.e. collected by an expert RL policy) using behavioral cloning (BC).
The inverse problem. We formulate the low-level policy a linear feedback policy, specifically a finite-horizon LQR, where the high-level action a goal state (position and velocity of robot joints). The detailed derivation of the inverse problem provided in Appendix B.I.I
Choice of inverse algorithm. We first evaluate OHIO under ideal conditions, on low-level demonstra- tion data collected within the same hierarchical framework, i.c. trained higher-level RL policy coupled with lower-level LQR controller, and by assuming access to approximate dynamics for the inverse method. We refer to this case as "HR Dataset". Results in Table Tdemonntee that OHIO can learn policy closely matching the performance of the dataset. On the other hand, the baseline that selects the observed next state as the high-level action (i.e., "Observed State") results an ineffective policy,
Access to the approximate (linearized) dynamics of the robotic arm is common practice and essential for operating the lower-level controller. However, we also investigate a more challenging scenario involving dataset derived from demonstrations by end-to-end agent (i.e.. "E2E Dataset"). In this setting, do not assume access to a dynamics model for policy inversion; instead, we learn an approximate model directly from the dataset. The results demonstrate that, despite the data being sourced from a different (i.e. flat) policy structure, OHIO achieves performance comparable to that of the end-to-end policy.
To evaluate OHIO's robustness to model misspecifications in the lower-level controller, perturb the LQR parameters by increasing either the state or control cost by a factor of 10. resulting in datasets 102E=10SS and "F2F-1OC", respectively. The results in [ that while the analytical inverse fast and exact, it more susceptible to model misspecifications. Conversely, the numerical inverse maintains consistent performance across scenarios.
5.1.2 ROBOTIC MANIPULATION
Robotic manipulation tasks often benefit from integrating learning- based and non-learming-baat based components, making them an ideal domain for evaluating our proposed method. RoboSuite (Zhu et al.) 2020a) is widely used robotic manipulation environment that closely aligns with standard practices woal robotic implementations. In this setup, an RL agent interacts with a lower-level controller, abstracting away the direct control of joint torques.
Datasets. We generate datasets for two tasks within the RoboSuite environment: Block Lifting (i.e., "Lift") and Door Opening (i.e., "Door"). We follow popular approach for offline RL data collection and utilize the replay buffers collected during online RL training. The default RoboSuite environment operates within hierarchical framework, allowing for the collection of both high-level and low-level actions. This configuration enables us to evaluate OHIO's 's performance in reconstructing high-level actions in comparison to training on the original actions. Furthermore, we can assess OHIO's 's potential to facilitate effective offline RL under modified controller settings.
The inverse problem. We utilize the operational space controller of RoboSuite as our lowelevel policy, which computes joint torques required to minimize the error between the current and goal pose (both position and orientation) the end-effector. In this experiment, we use numerical inversion, showcasing the broad applicability of OHIO even in the absence of a closed-form solution for policy inversion.
Robustness to low-level controller configurations. The results presented in Table highhlgtt a fundamental advantage of OHIO over standard offline RI. ipplemetnttton (IQL.. Traditional offline
7

---

Published as a conference paper at ICLR 2025
Table 2: Normalized score comparing offline (IQL) to OHIO robotic manipulation scenarios.
LIFT
DOOR
DATASET
IQL
OHIO
IQL
OHIO
ORIGINAL CONTROLLER
88.51:20.7
89.6.11944
91.4116.8
94.11:14.1
MODIFIED STIFENESS
86.8=16.4
98.94 4.4
18.6#14.3
92.7+15.8
MODIFIED DAMPING
24.1=11.1
75.8
2.9 2.2
76.7+28.2
Table 3: Normalized score comparing OHIO to offline hicrarchical RL (HRL) with high-level goal in either (i) directly on the joint space, or (ii) same representation used by OHIO on robotic manipulation scenarios.
TASK
OHIO
OBSERVED STATE
HRL-JOINE SPACE
HRL REDUCED GOAL
LIFT
89.6 94.1+14.1
0.1
77.4=25.4
0.02
DOOR
0.4
84.3
0.07=0.1
DOOR RED. DATA
88.2
72.7=34.8
RL tends to perform well when paired with the original controller used for data collection; however, its performance deteriorates significantly when the lower-level controller is configured with different parameters (IQL performance with modified stiffness and damping). In contrast, OHIO demonstrates strong performance across a wide range parameters and tasks. Importantly, these findings demon- strate OHIO's 's potential to facilitate effective offline RL, even when data is collected using varied or unknown low-level configurations challenge that is exceedingly common in practical applications.
Moreover, results in Table indicate that despite utilizing the same subgoal representation and lower-level policy as OHIO, the "Observed State" baseline struggles to learn the task effectively. This outcome clearly highlights the importance of the policy inversion method in enhancing performance.
Choice of low-level policy. Additionally, the results in Table indicate that OHIO can improve the performance over HRL while allowing RL-based policies be combined with standard low-level controllers. Decreasing the dataset size ("Door Red. Data") reveals another advantage of OHIO: the performance of low-level action reconstruction independent of dataset quality and coverage, whereas shows a clear performance drop when not provided with an extensive dataset.
5.2 NETWORK OPTIMIZATION
In this section, we examine two real-world examples of societally-critical systems: vehicle routing and supply chain management. These problems represent real- world systems characterized by key features: (i) high-dimensional action spaces, such nodes and edges in a large transportation network, (ii) complex system-level constraints that need to be strictly satisfied at all times (c.g., capacity limitations in warchouses), and (ii) readily available offline datasets of state transitions from system operators.
Problem settings. Vehicle routing problems are central to a wide range of mobility and logistics applications. The primary objective identify the least-cost routes for fleet of vehicles while meeting the demands of geographically dispersed customers. In a similar vein, supply chain inventory management involves the strategic ordering and distribution of products within network of interconnected warehouses and stores. The goal is to satisfy customer demand while minimizing system costs, which may include storage, transportation, and out-of-stoce penalties, all while adhering to operational constraints like storage capacities. Comprehensive descriptions of the environments can be found Appendices B.3 and
Datasets. To generate offline datasets, we simulate the operation of mobility -on-iit-----d services and supply chains using both optimization and heuristic-basse policies. the vehicle routing scenario, collect eight datasets across real-world urban settings NaCssNNC-.. and Shwnzhennutilleeee uuilizzin four different behavior policies: informed rebalancing ("INF") (Wallar et 2018). dynamic trip-vehicle assignment (DTV) MAlonso-Mora et al.) (017), a demand- propoottppooooiinaall heuristic ("PROP"), and random dispersion heuristic ("DISP"), For the supply chain scenario, we collect four datasets across two systems: one warehouse with three ("1 ("133)") and ten stores ("1 ("1W10S"), respectively. These datasets are generated using an optimization- based policy ("MPC") and heuristic ("HEU") policy. We record the low-level actions, which represent the flows of vehicles or goods across the network.
8

---

Published as a conference paper at ICLR 2025
Table 4: Normalized score comparing online (SAC) and offline (BC, IQI., CQL. algorithms within both End-to-End and OHIO formulations on the dynamic vehicle routing scenario.
END-TO-END
OHIO
DATASET
BЕR. PoL.
SAC
BC
IQL
CQL
SAC
BC
IQL
CQL
NYC-INE
91.
-8.
88.7 11.5
48.
48.1 #1.5
1.9
97.6
98.1 #2.8
93.0
NYC-DTV
89.4 #2.1
-35.2 48.
67.0 =.6
48.9 1:55
69.2 #2.3
98.0=1.9
89.2 #2.3
91.1 +2.8
43.5
NYC-PROP
-35.248.3 15.
83.1 =1.7
42.2
68.
98.0-1.9
85.7 +2.5
85.8 42.2
88.0 +2.4
NYC-DISP
45.8 10.7
-35.2 #8.3
44.1 =2.7
57.4
92.9
9.9
16.5
82.8 2222
94.1
SHZ-INF
90,7
-3.7
90.1 =1.5
90.4
42.2
95
90,4 87.0
81.
SHZ-DTV
92.8
+3.3
89.7 11.4
84,9 1.14
60.5
95.5 1.0
92.5 1:3
90.7 +1.7
90.8
SHZ-PROP
14.5
+3.7
85.5
86.5
59.8+1.6
83.3 95.5=10
87,4 83.6
SHZ-DISP
73.5
+3.3
83.2=1.4
92.5
11.
98.0=1.0
89.0
92.5
91.8
Table 5: Normalized score comparing online (SAC) and offline (BC, CQL) algorithms within E2E and OHIO formulations the supply chain management scenario. J. refers to transfer performance be- tween two environments (in this case, policies trained on 1W10S-MPC, tested on 11S--MP--CAL
END-TU-END
OHIO
BER.
IWES-HEUR
95.9 1.5
S0L3
808 1.7
2203. 5,9
96.1 2.0
79.4
81.5
79.0=
IW2S MPC
9.
95.9 1.3
97.1 2.4
97.6 97.6116
2.
96.1 2.0
9..4
96.01.166
1.
IWIOS-HEUR
15.3
87.5 + 1.7
-199. 1 146.7
4.54 4.8
4.2
90.7 + 1.1
10.9
112244.1
1..4
IWIOS-MPC
87.5 9..1+114
94.8
95.1
-1677.8
90.7
91.8+
6.0333
9.38
39.9 33.3
45.8 1 13.6
8..9.10.99
89.9 1.4
32.2 1.8
The inverse problem. We formulate the low-level optimization policies as linear programs (LPs), which allows us to exploit the fact that the inverse optimization problem of an LP can itself be formulated as LP (Chan C. Y. et al. 2022). In practice, this results in an L1 L1-norm minimization problem, thus projecting the low-level action onto the space of high-level actions within a feasible set of solutions. Specifically, the higher-level action commands goal state representing distribution of the commodities to be controlled (i.e. vehicles or goods). Please refer to Appendices B.3.5 and B.4.5 for a detailed derivation of the inverse.
Scalability and robustness in direct deployment. Results in Table 4 highlight significant advantage of OHIO, which consistently outperforms E2E approaches. As observed in previous studies(Fluri et al.) 2019; Gammelli et al. 2021; Skordilis et al. 2022 Singhal et al., 2024 E2E
policies struggle with the high- high-dimensional action space inherent in large networks. Specifically, real-world transportation networks exhibit dense graph structures that result in exponential growth of the (low-level) action spaces-1 -eeees--96--- and 289-dimensional for NYC and SHZ. In contrast, OHIO effectively capitalizes the dimensionality reduction induced by the hierarchical decomposition, leading to 14- and 17-dimensional high-level action spaces for NYC and SHZ, respectively.
Similarly, Results in TableeShighligh several insortantinsights, First, OHIO enhances the performance of offline learning algorithms that require querying the value function on unseen actions during training (e.g., CQL). As noted in (Kumar et al. 2020), sample-based value estimation in high-dimensional action spaces poses significant challenges due to high variance and the curse of dimensionality. In this context, the hierarchical decomposition introduced by OHIO allows for more accurate value function estimation through dimensionality reduction, resulting in more stable offline learning process.
Second, at first glance, there may not appear to be clear advantage OHIO when employing BC and IQL. in the context of moderately sized graphs. However, the results in rebee that E2E policies are extremely brittle, even when subjected to minimal variations in the scenario. Specifically, we evaluate both OHIO and E2E policies (trained on IW WIOS-MPC data) in minimally modified version the same environment (i.e., IW (OS-MPC-CAP) where all state elements remain unchanged except for reduction of storage capacity store facilities from 15 to 10. Results in Table fillustteletttrtte the advantages of OHIO, with E2E policies experiencing a performance drop of at least 50%, whereas OHIO's performance only deteriorates by up to 5%.

---

Published as a conference paper at ICLR 2025
100
100
75
75-
Scote
Score
50-
Beh. Pol.
50-
FT-OHB0
25-
FT-OHIO
FT-E2E
FT-E2E
2
8
10
12
14
26
18
20
10
20
30
40
50
60
70
80
90
100
Training (10*)
Training steps
(a)
(b)
Figure Supply chain fine-tuning performance of OHIO (FT-OHIO) and end-to-ende (FT-F2F) policies pre-trained on (a) sub-optimal (i.e., HEUR) and (b) biased data (i.e., MPC with biased forecast).
Robustness in online fine-tuning. We further examine various scenarios of practical relevance, including fine-tuning policies that (i) are trained on sub-optimal data (FigureZb). and (ii) must adapt to out-of-distributiin bias (Figure 2p). Although both policies yield similar results during direct deployment (in the case of IQL), Figure 2 iillustrates that OHIO policies demonstrate significantly greater stability and robustness during fine-tuning. The OHIO policy consistently improves upon the performance of the sub- sub-optimal and biased policy it trained on, whereas the E2E policy rapidly declines to a score below zero. Crucially, this degradation coincides with the E2E policy violating constraints during online interaction with the environment. This issue likely arises from two factors: (i) constraint violations are rarely included in offiin datasets, as current operators must adhere to critical constraints during operations, and (ii) achieving hard guarantees within E2E architectures is challenging. In contrast, the OHIO policy can effectively encode domain-speciff constraints through its low-level optimization- baseed policy, thereby avoiding infeasible -fddsiiiiitiitnoonnn states by design.
Alongside the safety guarantees provided, this framework enables current system operators to train policies using offline data until they achieve a satisfactory level performance. Subsequently, they can deploy these policies while safely enhancing performance through online interactions with the system.
6 DISCUSSION AND CONCLUSIONS
RL within large-scale, complex real-worke systems has so far been limited by issues such as lack of robustness, sensitivity to distribution shifts, and expensive training processes. Hierarchical policy struc- tures and offline RL are both promising strategies to tackle these issues, yet their integration remains an open challenge. overcome the difficulties combining offline RL with hierarchical policies, we pro- pose an approach that leverages the structure of low-level policies along with approximate knowledge of the system dynamics and reward function. This approach formulates an inverse problem that transforms low-level state (and possibly action) information into datasets suitable for standard offline RL tools. OHIO not only successfully recovers hierarchical policies from datasets generated by arbitrary, i.e. flat behavior policies, but it also effectively utilizes datasets collected under varying unknown low-level controller configurations
common challenge in practice that often hinders the efficient use of available data (e.g., data collected across multiple robotic embodiments). Our approach demonstrates strong performance across all problem settings we evaluate, substantially outperforming end-to-end RL and other hierarchical approaches in terms of both performance and, crucially, robustness. While standard offline RL struggles to avoid constraint violations that are not present in the dataset, OHIO addresses this by directly encoding domain domain-specific constraints. infeasible out-of-distribiee
result, OHIO inherently avoids states, facilitating more robust deployment and safer online fine-tuning.
While our approach demonstrates considerable strengths, it also has certain limitations. Since OHIO integrates elements of both model-based and model-free reinforcement learning, its performance sensitive to the accuracy of the dynamics approximation. Although we have not explored the robustness of our framework against model errors in this study, this represents highly promising avenue for future research. Moreover, solving the inverse problem can be computationally intensive, though this process is conducted entirely offline. In our current implementation, simplified action reconstruction by neglecting temporal information for computational feasibility; however, more sophisticated estimation methods, such cross-timestey losses, present compelling direction for future exploration. More generally, we believe this research opens several promising directions for the extension of these concepts wider range of large-scale, real-world applications.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
A. Agrawal, B. Amos, S. Barratt, S. Boyd, S. Diamond, and J. Z. Kolter. Differentiable convex optimization layers. In Conf. on Neural Information Processing Systems, 2019.
A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. In Int. Conf. on Learning Representations, 2021.
Javier Alonso-Mora, Samitha Samaranayake, Alex Wallar, Emilio Frazzoli, and Daniela Rus. On-demand high-capacity ride-sharing via dynamic trip-vehicle assignment. Proceedings of the National Academy of Sciences, 14, 2017.
B. Amos, I. D. J. Jimenez, J. Sacks, B. Boots, and 7. Kolter. Differentiable MPC for end-to-end planning and control. In Conf. on Neural Information Processing Systems, 2018.
Bansal, Tolani, S. Gupta, J. Malik, and C. Tomlin. Combining optimal control and learning for visual navigation in novel environments. In Conf. Robot Learning, 2020.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13:341-379, 2003.
T. R. Bewley, P. Moin, and R. Temam. bass-baedd predictive control of turbulence: an optimal benchmark for feedback algorithms. Journal of Fluid Mechanics, 447:179-225, 2001.
Damian Boborzi, Soristophh-Nkollss Strachle. Jens Stefan Buchner, and Lars Mikelsons. State-only imitation learning by trajectory distribution matching, 2022.
Timothy C.Y. Chan and Neal Kaw. Inverse optimization for the recovery of constraint parameters.
European Journal of Operational Research, 282(2):415 427, 2020. ISSN 0377-2217. doi: https://doi.org org:://ooi.oog/IOe org/ 10. (or.0016.ccor.221....0000
Timothy Chan C. Y., Rafid Mahmood, and I. Yihang Zhu. Inverse optimization: Theory and applications. Operations Research, 23, 2022.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusson policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems, 2023.
De Boer, D. P. Kroese, S. Mannor, and Y. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134, 2005,
Arthur Delarue, Ross Anderson, and Christian Tjandraatmadja. Reinforcement learning with
combinatorial actions: An application to vehicle routing. In Conf. on Neural Information Processing Systems, 2020.
Jiajun Duan, Di Shi, Ruisheng Diao, Haifeng Li, Zhiwei Wang, Bei Zhang, Desong Bian, and Zhehan Yi. pxee-reinforcmmeeellamnnnnnnnnie autonomous voltage control for power grid operations. IEEE Transactions on Power Systems, 35(1):814-887, 2019.
Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Conf. on Robot Learning, pp. 158-168. PMLR, 2022.
C. Fluri, C. Ruch, Zilly, J. Hakenberg, and Frazzoli. Learning to operate fleet of cars. In Proc. IEEE Int. Conf. Intelligent Transportation Systems, 2019.
D. Gammelli, K. Yang. J. Harrison, F. Rodrigues, F. C. Pereira, and M. Pavone. Graph neural network reinforcement learning for autonomous mobility-on-dentt demandd systems. In Proc. IEEE Conf. on Decision and Control, 2021.
D. Gammelli, K. Yang. J. Harrison, F. Rodrigues. F. Pereira, and M. Pavone. Graph meta reinfrrrenfor learning for transferable autonomous mobility-on-dent -nddeaadd.d. ACM lnt. Conf. on Knowledge Discovery and Data Mining, 2022.
11

---

Published as a conference paper at ICLR 2025
D. Gammelli, J. Harrison, K. Yang, M. Pavone, Rodrigues, and Pereira Franciscc. Graph reinforce- ment learning for network control via bi-level optimizztioo In Int. Conf. on Machine Learning. 2023.
J. Gilmer, S. Schoenholz, P. Riley, O. Vinyals, and G. Dahl. Neural message passing for quantum chemistry. In Int. Conf. Machine Learning, 2017.
A. Gupta, V. Kumar, C. Lynch, S. Levine, and Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Conf. on Robot Learning, 2019.
Tuomas Haarnjja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018.
Hado Hasselt, Arthur Guez, and David Silver. Double q-learming. In Conf. on Neural Information Processing Systems, 2010.
Taylor A Howell, Simon Le Cleac'h ', Sumeet Singh, Pete Florence, Zachary Manchester, and Vikas Sindhwani. Trajectory optimization with optimization-based dynamics. IEEE Robotics and Automation Letters, 7(3)x6700-67575 67570 2022.
B. Ichter, J. Harrison, and M. Pavone. Learning sampling distributions for robot motion planning. In Proc. IEEE Conf. on Robotics and Automation, Brisbane, Australia, May 2018.
Oussama Khatib. A unified approach for motion and force control of robot manipulators: The operational space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, 1987.
Geon-Hyeong Kim, Jongmin Lee, Youngsoo Jang, Hongseok Yang. and Kee-Eung Kim. Lobsdice: Offline learning from observation via stationary distribution correction estimation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.). Advances Neural Information Processing Systems, volume 35, pp. 8252-8264. Curran Associates, Inc., 2022.
T.-N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. In Int. Conf. on Learning Representations, 2017.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. CoRR, abs/21 10.06169, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Conf. on Neural Information Processing Systems, 2020.
H. Le, N. Jiang, A. Agarwal, M. Dudik, Y. Yue, and H. Daumé. Hierarchical imitation and reinforcement learning. In Int. Conf. on Machine Learning, 2018.
T. Lew, S. Singh, M. Prats, J. Bingham, J. Weisz, et al. Robotic table wiping via reinforcement learning and whole- body trajectory optimization. In Proc. IEEE Conf. Robotics and Automation, 2023.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 02971, 2015.
Pinxin Long, Tingxiang Fan, Xinyi Liao, Wenxi Liu, Hao Zhang, and Jia Pan. Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In Proc. IEEE Conf. on Robotics and Automation, 2018.
Mnnh, K. Kavukcuoglu, D. Silver, A. A. Rusu, et al. Human-level control through deep reinforcement learning, Nature, 5 51887540):529-533, 2015.
O. Nachum, Gu, Lee, and Levine. Data-efficient hierarchical reinforcement learning. In Conf. on Neural Information Processing Systems, 2018.
Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-traiiing for efficient online fine- tne-tuning. In Conf. Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2025
NYC Taxi & Limousine Commission.
Trip Record Data, 2013.
See https
// //www..ncc http//////e/e//////////////--/-/
page
Jan Peters and Stefan Schaal. Learning to control in operational space. Int. Journal of Robotics Research, 27(2): 197-2127 2008
Aoyang Qin, Feng Gao, Qing Li, Song-Chun Zhu, and Sirui Xie. Learning non-markovian decision-making from state-only sequences. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, Curran Associates, Inc., 2023.
pp. 65966696666 6618.
Faizan Rasheed, Kok-Lim Alvin Yau, Rafidah Md Noor, Celimuge Wu, and Yeh- Low. Deep reinforcement learning for traffic signal control: review. IEEE Access, 8:208016-208044, 2020.
J. Rawlings and D. Mayne. Model predictive control: Theory and design. Nob Hill Publishing, 2013.
Benjamin Rolf, Ilya Jackson, Marcel Müller, Sebastian Lang, Tobias Reggelin, and Dmitry Ivanov. A review on reinforcement learning algorithms and applications in supply chain management. International Journal of Production Research, 61 1151777999
E. Rosete-Beas, O. Mees, G. Kalweit, J. Boedecker, and W. Burgard. Latent plans for task-agnostic offline reinforcement learning. In Conf. on Robot Learning, 2023.
R. Rubinstein. Optimization of computer simulation models with rare events. European Journal
of Operational Research, 99, 1997.
A. Singhal, D. Gammelli, Luke, K. Gopalakrishnan, D.. Helmreich, and M. Pavone. Real-time control of electric autonomous mobility-on-demand systems via graph reinforcement learning, 2024.
Erotokritos Skordilis, Yi Hou, Charles Tripp. Matthew Moniot, Peter Graf, and David Biagioni. A modular and transferable reinforcement learning framework for the fleet rebalancing problem. IEEE Transactions on Intelligent Transportation Systems, 23, 2022.
Francesco Stranieri, Edoardo Fadda, and Fabio Stella. Combining deep reinforcement learning and multi-stage stochastic programming to address the supply chain inventory management problem. International Journal of Production Economics, 268, 2024.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm dm control: Software and
tasks for continuous control. Soflware Impacts, 6:100022, 2020. ISSN 2665-9638. doi: http://doi.org/I0 h00l://..imp..22222....2
Alex Wallar, Menno Van Der Zee, Javier Alonso-Mora, and Daniela Rus. Vehicle rebalancing for Sability-on-demanee systems with ride-sharing. IEEE/RSJ Int. Conf. on Intelligent Robots & Systems, 2018.
F. Xia, C. Li, R. Martín-Martin O. Litany, A. Toshev, and S. Savarese. Relmogen: Integrating motion generation reinforcement learning for mobile manipulation. In Proc. IEEE Conf. on Robotics and Automation, 2021.
Gioele Zardini, Nicolas Lanzetti, Marco Pavone, and Emilio Frazzoli. Analysis and control of autonomous coobility-on-demanee systems. Annual Review of Control, Robotics, and Autonomous Systems, 5:633-658, 2022.
D. Zhang, Zhao, Zhang, and He. Urbaneps: cyberrppysical system based on multi-source big infrastructure data for heterogeneous model integration. In Int. Conf. on Sabss-lhysccal Systems, pp. 338-247,11 247,20115 2015.
W. Zhou, Bajracharya, and D. Held. Plas: Latent action space for offline reinforcement learning. In Conf. on Robot Learning, 2021.
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In Proc. IEEE Conf. Robotics and Automation, 2019.
13

---

Published as a conference paper at ICLR 2025
Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martin- Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint wrXfv:2009.122999 2020a.
Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12402- 22402-12413. Curran Associates, Inc., 2020b.
14

---

Published as a conference paper at ICLR 2025
Algorithmic Details
16
A.1
Policy Inversion in the Observed-action Case
16
A.2
Policy Inversion Problem Statement
16
A.3
Inverse Problem: Sunear-Quadratic Gaussian
17
A.4
Solving the Inverse Problem Numerically
18
A.5
Assumptions on Data Quality and Property
18
B
Experimental Details
19
Gire-directed Control
19
B.1.1
Analytical Inverse
19
I.2
Numerical Inverse
20
B.2
Robotic Manipulation
20
B.2.J
Numerical Inverse
B.2.2
Hierarchical (
21
Vehicle Routing
B.3.1
Environment Detail
21
B.3.2
MDP Details
22
B.3.3
Model Implementation
22
B.3.4
Optimization Policy Formulation
24
B.3.5
Analytical Inverse
24
B.4
Supply Chain Inventory Management
25
B B.4.1
Environment Details
25
B.4.2
MDP Details
25
................................
B.4.3
Model Implementation
26
B.4.4
Optimization Policy Formulation
26
B.4.5
Analytical Inverse
27
B.5
Learning Components for Network Optimization
27
B.5.1
Network Architectures
27
B.5.2
Online fine-tuning
28
B.5.3
Hyperparameters
28
Further Experimental Results
29
Analytical Inverse on Linear State Space Model
29
C22
Visualization of Policies in Robotic Manipulation
29
................................
Leveraging Different Data Sources for Robotic Manipulation
................................
30
Comparison of Sample-efficiency during Online Training
30
Comparison Runtime at Inference
30
C.6
Online tne-ttunig of OHIO Policy in Network Optimization Scenarios
31
15

---

Published as a conference paper at ICLR 2025
Ut
at
ut+1
a++1
Ft
T++1
St
St+1
Figure 3: A graphical model for the system evolution, assuming Markovian dynamics and policies.
A
ALGORITHMIC DETAILS
In this section, provide further algorithmic details, and discuss practicalities of the policy inversion problem.
A.1 POLICY INVERSION IN THE OBSERVED-ACTION CASE
We consider the case in which we assume access dataset D = { = (5,,00. note that, critically, This matches the information available in the classical offline RL setting; not assume access to upper-level actions u.
J= consisting of trajectories
Lower-level policy inversion. Our approach aims to recover the high-level actions that generated the observed low-level actions. We treat this problem probabilistic inference problem, and aim to solve the (regularized) maximum likelihood estimation problem
uo.r = arg max
log p( [o: (Fo.. El(u)
Vo:T-1
(10)
ar a= Vt,
which is constructed jointly across each trajectory, and where L denotes regularization term, e.g.. Le regularization the high-level actions. practice, we will decompose this joint likelihood across time and (assuming that the policy is stochastic) solve the one-timestep problem,
arg max Uy
log (a +
(11)
A.2 POLICY INVERSION PROBLEM STATEMENT
The policy inversion problem is to recover ug.r from available data, provided in 7; (corresponding to episode i), which includes either states, possibly actions, and possibly rewards. We include graphical model for the system evolution in Figure in the case where only the high-level actions are unobserved. In general, we may assume the low-level actions and/or the rewards are also unobserved.
There are numerous inferential procedures for the graphical model specified in Figure 3 In particular, EM-based methods or variational inference methods are possible characterize uncertainty. and are well- selllestallised in e.g. hidden Markov models. In our settings, however, the mapping from high-level action low-level action is typically underdetermined, and thus the inverse mapping is typically overdetermined. For example, in network control tasks, the high-level action corresponding to goal state may be satisfied by many low-level (edge flow) actions.
Thus, we turn to regularized maximum likelihood approach.
uo = arg max log p(T; [uorr)++ + [(wor) u0:77
(12)
which we decompose across time as previously mentioned, yielding inferential procedures shown in Figure
In the action-observed case, it sufficient to directly invert the policy. Typically, this takes the form of = arg max logp(ar s4,4) I(u))-
(13)
16

---

Published as a conference paper at ICLR 2025
Ut Qt
Ut
at
St
St
S1+1
Figure 4: A diagram showing our inference procedure. Left: the observed-action case. Right: the case in which low-level actions are not observed.
In the case of explicit policies, this corresponds standard inverse problem, in which one aims to recover the input from an output. In some cases this is analytically tractable (as in the LQR example), but more commonly one must turn to numerical optimization, which the objective corresponds to minimizing predictive error.
For implicit policies-especially optimization- based policies- this takes the form of inverse optimization (Chan C. Y. et al., 2022). Inverse optimization is analytically tractable in limited cases, but more generally one must turn to numerical methods.
When actions are not observed, the inversion procedure is similar to the action observed case. In contrast to e-g. EM procedures (which may be a natural inferential scheme to recover a, jointly) we first compute point estimate of based on state transitions, and in turn use this to compute point estimate u. While this may induce error in the inverse problem, we have found this scheme to work effectively in the settings consider. Typically, systems dynamics to optimization based dynamics. which can occur for example in robotics with contact or multi-agent mecision-making (Howell al. 2022). Thus, we define dynamics s' = =g(s,a) logp(s s' s,u) straightforwardly written in terms of 0,M,,, leading to similar objective to the action-observed case.
explicit-
opposed
A.3 INVERSE PROBLEM: LINEAR LINEAR-QUADRATE QUADRATIC GAUSSIAN
In this section, we provide the details on the linear-quadratic example from the body of the paper. Recall that the optimization policy is
2 E. |la ml 11a-mllir+ +|ss -1ss'=2211 -
arg max aEA A s.t.
(14)
s' N'~N NNAs --((s-BBtcc.))
where, by substituting, we have
arg max
Eg
||a m 11 -l
(15)
AEA
(0, For the case where A = (A), this problem is concave, and thus for maximizer a*,
O=E[M(a - Bd (As Ba +c+e-u))
(16)
= -
(17)
which yields
a=
(18)
which corresponds to a" = Kutkfor
K= <=MMBB"BBB k=
(19)
V (As+c)).
(20)
To compute the next state density, we can substitute this action, to yield policy density
8' (As+ B(Ku+k)
(21)
17

---

Published as a conference paper at ICLR 2025
which has concave log-density. Again, the maximizer u² is achieved when
0 0= DB(KK E- ' 3.''-Asta Astcc++(Kuutt
(22)
which is satisfied by
-'= ((Bs)((s((AstctBB))
(23)
A.4 SOLVING THE INVERSE PROBLEM NUMERICALLY
We have shown that for a particular choice of inner policy and dynamics models, an analytical solution of the policy inversion problem is possible. While this approach is numerically efficient due to recursive sensitivity calculation and our exploitation of problem convexity, we can instead turn to numerical solutions.
First, using automatic differentiation, the sensitivity the low-level action (or post-transition state) to may be automatically computed. This enables a partially structured approach, in which we still exploit problem convexity, but automate computations that are potentially error-prone.
An alternative approach is simply turning to non-convex optimization methods-sus gradient descent-to compute an approximate minimizing u. Indeed, the only requirement for gradient descent the differentiability of the policy with respect lower level is not differentiable we can resort to gradient-free methods, e.g. CEM. Thus, the numerical solution of the inverse problem is considerably more general approach, and will typically favor this approach. Algorithm 2 and Algorithm Billustrate the numerical inverse depending on available information the dataset. Note, that Algorithm [ we could also calculate the loss over each state tuples in the lower-level (e.g. ((r.,8), instead of only the resulting state.
we have discussed previously. Finally, if the
Algorithm 2 Numerical inverse state-only trajectories
1: Given approximate dynamics A,
2: Given 8,57
3: u'=s'
4: for each step do
5: Set
6:
ior
7:
Get action (=
8:
Unroll system dynamics: r++1 + Ax, BB
9:
end for
10:
Compute loss L(Er,ST)
Update using Lusal, either using gradient descent or CEM
12: end for
Algorithm 3 Numerical inverse with low-level actions
1: Given, $1.7.C1..T
2: u+0
3: for each step do
4: Initialize cumulative loss Liotal +-
5: i= do
6:
Get action ú;= 2. (8i,u) 7: Compute loss Lumal L( ((agaa))
8:
end for
9: Update using Crotal, either using gradient descent or CEM
10:
end for
A.5 ASSUMPTIONS ON DATA QUALITY AND PROPERTY
OHIO is agnostic the learning signal used for training the high-level policy and, therefore. does not impose additional data quality assumptions beyond standard offline RL requirements, We demonstrate OHIO's effectiveness with both expert data -collected from trained RL agents and
18

---

Published as a conference paper at ICLR 2025
traditional MPC policies- and suboptimal data gatthered from partially trained RL agents, heuristic- and optimization- based policies. We note that, when no approximate dynamics model is available, we require that low-level actions are observed to either learn approximate dynamics model or directly perform policy inversion with low-level actions.
B EXPERIMENTAL DETAILS
In this section, we provide further detail about experiment details for the goal-directed control (Appendix B.11) and manipulation experiments (Appendix B.2). Further, we provide details on environment specifics relating to vehicle routing (Appendix B.3 and supply chain control (Appendix(B.4) experiments, respectively and on learning components (Appendix [B.5 for the network optimization tasks. The training of our models was executed on a Tesla V100 16 GB GPU.
B.1 GOAL-DIRECTED CONTROL
For the first robotic experiment, we evaluate OHIO on the Reacher-hard task from the DeepMind Control Suite (Tunyasuvunakkol et al. 2020). The end-to-end policy directly learns the low-level environment actions, whereas our hierarchical framework learns a desired goal state (position and velocity of robot joints), which serves as input to a goal-conditioned Regulator (LQR) with a horizon of = 5. Specifically, the system is linearized the current state using finite differences, yielding constant linear dynamics À and B. The cost function penalizes deviations from the goal state, and we perform the Riccati recursion over finite horizon to compute the time-varying feedback gains for control law [(s u). At each step of the recursion, the optimal feedback gain is computed as
heiteehorizzo Linear Quadratic
K, = (R+ P1 PB. P==Q+A P4t(A+BKi))
Pt+1A),
To generate the datasets, we train both an E2E and hierarchical SAC policy (Haarnoja al. 2018) online and use the final checkpoint collect the demonstration data. All policy and value function networks are MLPs with two hidden layers, each containing 256 units. Similarly, the learned dynamics model consists of two MLP layers with 256 units each and two output layers that map to the and B dynamics matrices.
All datasets used for this experiment consist of 250 episodes of interactions (each consisting of 1000 timesteps). To Iearn the dynamics model, use train/val split of 0.9/0.1.
The SAC and BC algorithms use the following hyperparameters indicated in Table 6
As lower-level policy we use an LQR policy with
10.0
0.0
0.0
0.0
0.0
10.0
0.0
0.0
Q
0.0
0.0
1.0
0.0
0.0
0.0
0.0
1.0
and
0.1
0.0
R
0.0
0.1
B.1.1 ANALYTICAL INVERSE
Our goal is compute the inverse analytically for an LQR policy that tracks goal state with temporal abstraction T to the higher-level policy. First, without temporal abstraction, compute the next state density, substitute the action a= K (s-u)) to yield the policy density:
*~N(As+
+c,E),
(24)
which has concave log-density, The maximizer u² is achieved when
0 0= (B(K))
(
- (s'- As- (s)(s)
(25)
19

---

Published as a conference paper at ICLR 2025
This is satisfied by
" (B+)((AA
(26)
We define two recursive terms _2 and 42 that evolve time, allowing us to generalize our low-level policy across temporal horizon T.
Initialization:
i = Ko,
(Ao+
Recursive computation /= - T:
1 = (Ai + B, #1
4 (Ar + B₁ 1 P2
Final Solution for u::
4= 1 -2))
Regularised analytical inverse. We show that using this analytical inverse formulation recovers the high-level action exactly in linear state space model Appendix C.I However, in the main body, we apply this method to more challenging, non-linear system using approximate linearized dynamics. When solving this inverse problem exactly, we observe large magnitudes in the recovered actions, the solution attempts perfectly fit the data under these approximate dynamics, which reduces generalizability and makes learning harder. To address this, we employ an implicit regularization technique by using the analytical gradient in a gradient descent algorithm, iteratively updating the solution with early stopping to prevent overfitting.
B.1.2 NUMERICAL INVERSE
As an alternative to the analytical solution, we can solve the inverse numerically (see Algorithm 2). We use the Adam optimizer with learning rate of 0.01 and run 10000 steps per data point with early stopping if the difference between the previous and current loss is below le". To avoid that solutions from bad local minima impact learning, only include transitions with a loss 0.2 in the dataset. Further, for all of the datasets we scale the action to within 1,1].
Table 6: Hyperparameters of SAC,
Parameter
Value
Optimizer
Adam
Learning rate
x 10-3
Discount (?)
0.97
Batch size
100
Entropy coefficient
0.3
Target smoothing coefficient (r)
0.005
Target update interval
Gradient step/env.interan interaction
B.2 ROBOTIC MANIPULATION
In Robosuite, we selected two tasks Door Opening and Lift -tha can be solved using online RL to collect datasets, both using standard environment configurations. We use the recommended lower-level controller, the Operational Space Controller, and set the temporal abstraction between the high-level RL and low-level controller 55
For data collection, trained standard online SAC (Haarnoja et al. 2018) algorithm for 1.500 episodes for Door Opening and 2,000 episodes for Lift to convergence. This process yielded datasets containing 750,000 transitions for Door Opening and 1,000,000 transitions for Lift, respectively. Data collection is done under standard controller settings with original stiffness (kp = [150, 1550,150,155 1550,100, 150, 150) and damping (kd 11,1,1,1,1,11, For the modified controller scenarios, we either change to kp = [150,155 150,150,50 50,500,, or kd - (3,,,
For offline training of IQL., the parameters indicated in Table
20

---

Published as a conference paper at ICLR 2025
Table 7: Hyperparameters of IQI..
Parameter
Value
Optimizer
Adam
Learning rate
1 1010-
Discount (7)
0.97
Batch size
256
Target smoothing coefficient (T)
0.005
Target update interval
Gradient step/env.interaction interaction
Temperature Quantile
(Lift), (Door)
0.7 (Lift), 0.9 (Door)
B.2.1 NUMERICAL INVERSE
For this lower-level controller, we resort numerical methods. Fortunately, it remains differentiable, allowing us to utilize both gradient- based and gradient-free free methods, such as the Cross- Cross-Entropy Method (CEM). In general we observe, that gradient- based approaches require fewer controller runs thus re- ducing computational time-to converge to a solution. However, depending on the initialization of u. the higher exploration of CEM can help to escape local minima and lead better results. (see Algorithm 3
Consequently, in scenarios where we aim to restore the high-level action for the given controller, we initialize u as a zero vector and run gradient descent. For transitions where gradient descent gets stuck in local minima, indicated by high final loss, we rerun CEM to improve the results.
In casesExec where we want to modify controller settings compared to the data collection, we can use the original high-level action the initialization. In this case, gradient-based methods tend to perform well and deliver satisfactory results.
For the gradient descent algorithm, use the Adam optimizer with learning rate of 0.01, running for up to 10,000 steps with early stopping triggered if the loss falls below 1e For the Cross-Entropy Method (CEM), we generate 50 samples per iteration, retaining the top 20% of them based on performance in each step. Early stopping is applied the loss does not improve over the course of 4 consecutive steps.
B.2.2 HIERARCHICAL RL ("HRL")
For the "HRL" baseline, where both levels are learned, we evaluate two scenarios: (i) the upper level predicts the same reduced goal state as in the case of an operation space controller (position and orientation of end-effectors) (ii) the upper level predicts the goal state directly in the low-level joint space (position and velocity of the robot joints. The lower-level network then outputs the required torques to reach the desired goal state. As the network architecture, we employ a recurrent neural network consisting of two LSTT layers and a fully connected MLP layer, followed by a Tanh activation function. The Tanh output is scaled to match the feasible range for torque control, ensuring the actions remain within valid limits. All hidden dimensions are set 256, we use train/val split of 09./01. and train for 100 epochs, where we save the model with the best validation loss.
B.3 VEHICLE ROUTING
B.3.1 ENVIRONMENT DETAILS
In our experiments, we focus on two case studies generated from trip record data, which we provide with our codebase, from the cities of New York, USA Taxi & Limousine Commission] 2013). and Shenzhen, China (Zhang et al. 2015). We are looking at a taxi-like system serving commute demand in the areas of Brooklyn and Shenzhen, respectively. In each scenario, the road network is segmented into geographical areas, representing stations. The trip record data are converted to demand, travel times, and trip prices between stations. As in (Gammelli et al. (022), the arrival of passengers assumed to be -dpendeen Poisson process, where the Poisson rate is aggregated from the trip record data every 3 minutes.
21

---

Published as a conference paper at ICLR 2025
An on-demand service provider coordinates M single-occupancy autonomous vehicles transportation network represented by a complete graph 9 (V,E) where V= {or {o) (i=1:N.) and (=(e)) represent the set of vertices and edges of G. Specifically, defines the set of stations (e.g., pick-up drop-off locations), and & defines the shortest paths between stations. The time horizon is discretized into a set of time steps I = 1.2 of length T. At any time step t, vehicles are controlled to travel along the shortest path between station i and j fi E V with travel time of rf, E Z and travel cost Cij. as a function of travel time. At each time step t. passengers submit trip requests for desired origin-destination pair (i.j) E VxV, which is characterized by demand di,, and price pl,y- The operator matches passengers to vehicles, and the vehicles will transport the passengers to their destinations. For idle vehicles that not matched with any passengers, the operator controls them to stay at the same station or rebalance to other stations. We denote fig. e & <di as the passenger flow, i.e., the number of passengers traveling from station i to station at time l and fig,n n E N as the rebalancing flow, i.e., the number of vehicles rebalancing from station to station j at time L.
B.3.2 MDP DETAILS
Reward (r(s;, we choose the reward be the operator's profit, which we define as the difference between the revenue from serving passengers and the cost of operations:
r(st,a:)= = Σ MP
- 4 -
Σ Sey.ecy.
(K,))E
(LJ)EE
State space (S): the state space describes the current status of the transportation network via node features. Specifically, given a planning horizon K, we consider: (1) the current and projected availability of idle vehicles in each station m E [0,M] Vi V and ) E (2) provider-level
transportation information trip price vl., and cost ci.g. (3) current d' and estimated (d) v ++ demand between all stations.
High-level action u: Given the number of idle vehicles and their current spatial distribution, we consider the problem of determining the desired idle vehicle distribution o;
B.3.3 MODEL IMPLEMENTATION
In what follows, we provide details for the implemented data collection methods and models.
Optimization- and heuristic-baree -based approaches:
1. Random Dispersion: at each time step, the desired distribution is sampled from Dirichlet
prior with a concentration parameter c= (1.11...... 1].
2. Informed rebalancing (INF) (Wallar et al. | 20188: This model assigns idle vehicles Vale to rebalance to regions R. that are reachable within a pre-defined time horizon H. By including the forecasted demand rate in each region Ay. it maximizes the expected number of requests each vehicle would observe in its assigned rebalancing regions. This formulation is extremely sensitive to the hyperparameters H and p. In our experiments, we tune them through an exhaustive grid search (Table 8 Table D). Variable Vhy indicates assignment for vehicle e Videe to rebalance to region center j R and Ti,y gives the travel time from vehicle i E Videe to region center j E R.
max Σ
Σ
(27a)
BE
JER
s.t. Σ VnSI $1
Vie Vidle
(27b)
jER
Vij (H1-) 20
ViE Ville jER
(27c)
Σ Boj (H (Ax H2-pp
VjER
(27d)
VEVia
3. Dynamic trip-vehicle assignment (DTV) (Alonso-Mora et al. 2017): This model assigns vehicles to unassigned requests by minimizing travel time and either assigning all idle vehicles
22

---

Published as a conference paper at ICLR 2025
Vidle or all open requests (Rto). Ye,r indicates assignment of vehicle e Vidie requestr e Rko, while Te,r gives the shortest travel time for vehicle E Valee to pickuup uprequest E Rko-
min y
Turryu,r
(28a)
s.t. Σ
Σ Norr = l.1RRol))
(28b)
VEVa.
Rio
0 yuur 1 Verrey.
(28c)
4. Proportional Heuristic (PROP): This heuristic distributes excess vehicles according the forecasted demand X;- rebalances proportional to the averaged forecasted demand over the next K =6 timesteps in region q: Σgev
min Jis.8
Σ dyfis.n
(29a)
s.t.
Σ A
(29b)
fu.RS S S9,iEV
(29c)
Table Hyperparameter tuning of SHZ-INE.
H
4
5
6
60.1
60.2
60.1
60.1
60.1
60.0
3
60.9
59.6
59.6
59.8
60.0
60.1
52.97
50.65
51.50
51.79
52.00
60.10
5
50.83
44.62
44.19
44.19
44.19
52.10
48.61
43.13
43.44
43.44
43.44
44.19
Table 9: Hyperparameter tuning of NYC-INE.
H
p
2
3
2 27.25
27.20
27.18
27.09
26.95
26.89
26.89
3
39.70
38.30
39.17
39.30
29.30
39.30
39.30
4
49.50
50.08
50.54
50.55
50.55
50.55
50.55
5
52.88
54.21
54.23
54.23
54.23
54.23
54.20
6
56.22
57.24
57.25
57.25
57.25
57.25
57.25
7
56.68
56.04
56.04
56.22
56.22
56.22
56.20
Learning-based approaches:
1. End-to-end RL: for the end-to-end RL implementation, the flow action is defined along the edges opposed the desired distribution nodes. We achieve this through minimal changes with respect to the OHIO network architecture. Specifically, this results in an edge convolution (consisting of linear layers of 256 units) that outputs the mean and standard deviation parameters of a Gaussian policy for each edge in the graph.
2. OHIO: for all networks, we use one layer of GCN with 256 hidden units with a sum aggregation function, followed by linear layers of 256 hidden units and a final layer mapping to the respective output's is support.
23

---

Published as a conference paper at ICLR 2025
B.3.4 OPTIMIZATION POLICY FORMULATION
Given desired next state described by the desired number ifle vehicles across stations of vie we define the following linear control problem according to Gammelli et al. (2233 as follows:
min Σ EdySER Fi.. (1.))E
(30a)
s.t.
- R
iEV
(30b)
141
Σ Fi.. ftL
iEV
(30c)
20
(i,))
(30d)
where the objective function (3aa represents the rebalancing cost, constraint (30b) ensures that the resulting number of vehicles close the desired number of vehicles, and with constraints (30c).
(30d) ensuring that the total rebalancing flow from a region upper-bounded by the number of idle vehicles in that region and non-negative.
B.3.5 ANALYTICAL INVERSE
We formulate the inverse problem using data-driven inverse optimization approach Chan C. Y. et al. (2022). The general primal optimization problem models network flow system that minimizes rebalancing costs and penalties for deviations from the desired target distribution (in cases where not all desired distributions are feasible and reachable). The primal problem formulated as follows:
Primal Objective:
Minimize e x+p
s.t.: Ar B2
x x20, =20.
where in our example, x denotes rebalancing flows, and is slack variable penalizing deviations from the target distribution, with respective costs and p. The constraint matrix A encodes the flow and supply constraints, while B accounts for the deviations from the goal state.
In the inverse optimization problem, the goal is to reconstruct the unknown desired target distribution by adjusting the right-hand side (b) such that the observed solution (x*) remains feasible and maximizes the fit of the forward model. This is achieved by minimizing absolute sub-optimality. The absolute sub-opte uuttoptialy problem can be reformulated and efficiently solved using strong duality Chan
C. Y. al. (022)..
Minimize:
s.t.: Ax" +B Bb A A
B X
A A50.e20,20.bEA.
==(<_r=-p-z)-0
where specific components of b are fixed to the current vehicle distribution. This ensures that the reconstructed aligns with the observed flows (a:") while enforcing feasibility and minimizing the optimality gap under the adjusted model. The desired target distribution can then be inferred from the optimized b. We note, that this inverse model is bilinear, but Chan & Kaw (2020) show that this problem has analytical solution. For increased exploration and robustness the higher level, we could reconstruct different (potentially infeasible) target distributions that lead to the same observed flows. For this, could add small regularizer to the objective function, such as E,a, b /r), which encourages ficxibility in reconstructing the target distribution, which is left as an interesting direction for future work.
24

---

Published as a conference paper at ICLR 2025
On closer examination, in the current hierarchical formulation with action definitions fully connected graph where all actions are feasible and reachable- and since all the data collection strategies are central policies and we, therefore, observe optimal x*, i.c. minimal cost flows, the inverse problem simplifies. Specifically, the constraint parameters that satisfy equation (30b) with equality Ax* = provide one direct solution the inverse problem. The reconstructed target distribution d is then computed as: q. = +z Lfjj Vij..).
B.4 SUPPLY CHAIN INVENTORY MANAGEMENT
In what follows, we describe environment specifics, MDP definitions, baseline implementation, and the low-level, optimization formulation.
B.4.1 ENVIRONMENT DETAILS
In our scenario, we consider a distribution network a supply chain consisting of interconnected ware- houses and stores aiming meet customer demand while minimizing storage and transportation costs. We define the supply chain as a graph G {V,E), where V= UV is the set of warehouse Vw and distribution V/2 nodes respectively and & the set of edges connecting warehouses stores. If sufficient inventory is available, demand dā is fulfilled in stores s Va and sold at price p. Unsatisfied orders are back ordered cost, At each time step warehouse i orders additional units of inventory wi bounded by production capacity Cp and stores available ones bounded by storage capacity C.. Simultaneously, each store orders additional inventory from the warehouses bounded by storage capacity c; Ordered units get delayed by production t and travel times 442 respectively. During operations, production m² transportation m storage m² and backorder costs m occur. All stores are assumed to have an independent demand -eeaannngnnrtiing process. We simulate seasonal demand behavior by representing demand d, Va a co-sinusoidal function with stochastic component defined as follows:
d'max 2
f2 T
1+00s
(0,
(31)
where d'nxx the maximum demand value, (0 is uniformly distributed random variable, T the episode length, f and controlling the frequency and shift respectively.
Environment parameters defined Tables
Table 10: Parameters for the IF1OS environment.
Parameter
Explanation
Value
Parameter
Explanation
Value
d
Maximum demand
[5, 15, 20]
m
Storage cost
[0.1, 0.5, 0.5, 0.5]
dr
Demand variance
2, 21
m
Production cost
f
Demand frequency
[2, 6]
Cp
Production capacity
25
r
Demand shift
[1.
m"
Transportation cost
0.5
ty
Travel time
[1,
P
Price
Production time
m
R
Backorder cost
1.5
Storage capacity
[50, 15, 15, 15]
T
Episode length
30
Table 11: Parameters for the IF10S environment.
Param.
Explanation
Value
Param.
Explanation
Value
Maximum demand
m
Storage cost
[0.005.2....2
yur
Demand variance
[2, 2,
m
O
Production cost
5
Demand frequency
6, 4, 2, 6,
Cp
Production capacity
60
Demand shift
1,
m
Transportation cost
0.5
by
Travel time
[1,
p
Price
15
Production time
m P
Backorder cost
1.5
Storage capacity
880.15..5..55.555.........5
T Episode length
30
B.4.2 MDP DETAILS
Reward (r(s,a,)) J): we select the reward function in the MDP as the profit of the inventory manager, computed as the difference between sales revenues and the sum of storage, production, transportation,
25

---

Published as a conference paper at ICLR 2025
penalties for capacity violations, and backorder cost:
r()),at)=
Σ
p- p-min(d,.9.)
Σ
m
Σ mi
in--vi w
Σ
myj
S'
iEVW
LEV
iEVi
(i,)EE
1.xxp-max(O.jf -c,))
Σ iEVa
m mm..maax00
(32)
BEV2
where of is the inventory level at node at time I, wa the production order at warehouse i Vw at time t and f the shipment flow from warehouse i E Va to store at time t.
State Space (S): the state describes the current state of the supply chain network by defining node and edge features. Node features contain (i) current and back-ordered demand, (ii) current inventory, (iii) storage and production cost, sales price, and storage and production capacities, (iv) incoming flow or orders for the next T timesteps EjeyJjät++ 1.7 or Waa+ 1.7. Edge features are represented by the concatenation of (i) travel time lij and (ii) transportation cost.
Output of the RL RL policy u: we define u by two elements: (i) goal production in warehouse nodes Vi V. and (ii) goal inventory over nodes d,. vie Va.
B.4.3 MODEL IMPLEMENTATION
In what follows, we provide additional details for the implemented dataset collection strategies and baselines.
Domain-drived heuristics:
S-type Policy:
commonly known as the "order-up-to" policy, operates on the basis of the order-up-to level for the warehouses and stores. Essentially, at every time step the inventory manager places orders an amount that will bring the total inventory on hand and transit up to their respective order-up-to levels. In practice, the optimal order-up-to levels for each environment are determined through an exhaustive grid search.
MPC-based: Within this class of methods, measure the performance of traditional optimization- based approaches using an MPC approach.
1. MPC-Oracle: this benchmark serves the purpose of quantifying the performance of an "oracle" controller. We provide this model with perfect foresight information on future demand and sys- tem dynamics. By providing the optimization model with Oracle knowledge of the realization of stochastic elements, we are able to quantify the optimality gap for the presented methods. 2. MPC: We relax the assumption of perfect foresight information MPC-Oracle and substitute it with noisy and unbiased estimate of demand.
Learning-based
1. End-to-end RL: this benchmark does not approach the problem via the proposed hierarchical formulation of OHIO, but rather through more traditional end-to-end (E2E) architectures. Specifically, the flow action is defined along the edges opposed to over the nodes. We achieve this through minimal changes to the architecture by edge convolution (consisting of 2 linear layers of 32 hidden units) that outputs and | parameters of a Beta distribution for each edge the graph. We adopt an individual upper bound for each action respective to the storage capacities/production capacities of the previous stage (Stranieri et al. 2024).
2. OH1O: for networks, we use two layers of message-passing neural network of 256 hidden units with sum aggregation function, followed by linear layer mapping to the respective output's support.
B.4.4 OPTIMIZATION POLICY FORMULATION
Given the output of the high-level RL-based policy defined as (i) the desired production of at warehouse nodes i E Vw and (ii) the desired distribution of available inventory over distribution nodes
26

---

Published as a conference paper at ICLR 2025
% Vi Va, we define the following optimization-bant based policy Gammelli al. (2023)
min Ry
Σ
K
Eleg
(33a)
NEW
iEVa
s.t.
Σ
fj.
-6 +1 +67,00
+
ViEVa
(33b)
JEN
(1)
Σ
-dk 5Cs,.
ViEVa
(33c)
JEN
Σ fig Sdi
ViEV
(33d)
JEN+()
df +w
Σ
Cp.s
ViEVw
(33e)
(EN-())
m/ = wi +
ViEVw
(33f)
1 20,
(i.))
(33g)
where w/ is the realised production at node i E Vw at time I, füg the commodity flows from node i node at time t, dī the demand in node i E Va at time of the available inventory each node i E V, and C..; the storage and Cp.; the production capacity at node i V. The objective function (33 represents the distance metric that penalizes the deviation from the desired next states. Constraint 33b ensures that the total incoming flow in distribution nodes is as close as possible to the desired inventory, constraint 33 ensures that the inventory after demand realization, and incoming shipments do not exceed the storage capacity, constraint 33d guarantees that the combined shipment quantity is upper bounded by the warehouse inventory. Constraint 33e ensures capacity adherence in the warehouse, and constraint 33f ensures that orders from manufacturers are close to the desired orders quantity and, lastly, that commodity flows are defined as non-negative
B.4.5 ANALYTICAL INVERSE
The flow dynamics in this problem are deterministic, and the cost terms minimize the one-norm of error terms. These error terms measure disagreement between the realized next state under the system dynamics and the goal next state. In the context of inverse optimization, our goal is to infer the values of constraint parameters o Vi E Vs (which correspond high-level action u) that make the observed inventory flows optimal for the original LP. Similar to the vehicle routing scenario, we can achieve this through data-driven inverse optimization Chan C. Y. et al. (2022). When assuming that the observed flows do not violate capacity constraints, the direct solution to the inverse problem can be derived as q. +++ (j) Vi e Va. Note, that this way we only generate feasible high-level actions for the offline dataset. The inclusion of error term e'y,: in the inverse formulation to create non-feasible high-level action as means of exploration for the offline RL agent is left as future work.
B.5 LEARNING COMPONENTS FOR NETWORK OPTIMIZATION
In this section, we provide details about the learning component in the network optimization experiments.
B.5.1 NETWORK ARCHITECTURES
We parameterize policy, Q- and value function estimators through graph neural network encoders. The specific network architectures are problem-specific and can be summarized as follows:
1. Vehicle Routing: to define valid vehicle distribution, the output of the policy network is sampled from a Dirichlet distribution u, ~ Dir u, c). More precisely, we use a Graph convolutional neural network (GCN) (Kipf & Welling 2017) with aggregation function, followed by three linear layers that compute the concentration parameters E R over N, regions, where the positivity of is ensured by a Softplus nonlinearity. The Q- and value functions have the same backbone architecture. In the f-uuccioon architecture, the node
27

---

Published as a conference paper at ICLR 2025
Table 12: Hyperparameters of SAC.
Parameter
Value
Optimizer
Adam
Learning rate
1 1+103
Discount (?)
0.97
Batch size
100
Entropy coefficient
0.3
Target smoothing coefficient (r)
0.005
Target update interval
Gradient seep/envinteraction
embeddings concatenated with the action before being fed into the linear layers. The value function maps from node embeddings to the final value estimate through a sum aggregation. message- meessage-assing neural network (MPNN) (Gilmer et al. 2017) with sum aggregation. The output of the policy network is defined as (i) concentration parameters CE I of Dirichlet distribution over warehouses and stores for computing the shipment flows, and (ii) E [R!
2. Supply Chain Inventory Management:
and B E [gvvv of Beta distri- bution, where the output is scaled by the production capacity to define the desired production. The Q- and value functions share the same encoder architecture. The action is concatenated with the node embeddings before the linear layers to achieve Q-value estimate, while the node embeddings are aggregated through summation compute the value function estimate.
B.5.2 ONLINE FINE-TUNING
In this section, state the specifics of our online fine-tuning procedure.
With offline RL., we obtain a policy initialization, which is intended for sample-efficient online fine- tuning. Offline learned policy and value functions via IQL or BC can be fine-tuned directly. However, conservative methods such as CQL tend to learn smaller Q-values than their true values. Consequently, initial interactions during online fine-tuning
spent adjusting the Q-function, leading to unintentional unlearning of the initial policy. To address this, we calibrate the Q-function during offline learning to match the range of the ground-truth Qavauuss via Cal-CQL, as proposed in Nakamoto al. (2023)
Further, to mitigate large gradient updates during initial fine-tuning that potentially lead to unstable policy behavior, (i) freeze the weights of the policy network and only train the Q- and/or value function for the first 200 episodes (ii) we start by sampling 50% of the batch from the offline dataset and gradually decrease this proportion to Over 3000 episodes.
B.5.3
HYPERPARAMETERS
SAC: The hyperparameters used to train online baseline can be found in Table 12 To improve learning stability, we implement (i) a double estimator for the Q-function (Hasselt et al. 2010) and (ii) target Q-networks (Mnih al. 2015)
CQL: For offline experiments, train the CQL(H) version of CQL with trade-off factor a= 1. We use a policy learning rate of + 10-4 and a critic learning rate of 3 10-4, The remaining hyperparameters kept identical to the online SAC version in Table 12
IQL: We use 0.9 and 3 for all implementations and policy learning rate of 1 10-4 and a critic learning rate of 3 + 10 General hyperparameters are kept the same as in the online SAC version in Table
28

---

Published as a conference paper at ICLR 2025
C FURTHER EXPERIMENTAL RESULTS
C.I ANALYTICAL INVERSE ON LINEAR STATE SPACE MODEL
Given linear state space model with state transition matrix A, and control input matrix B,: -t=
At and B
Ar
A,
we recover several high-level actions with Equation (33) for different
&
LQR parameter settings that exactly lead to the next observed state, which are illustrated in Figure Specifically, we test RE (0,0.2), and Q1 E (3.5.4.0.455,55005553 and Q22 = 1, Q Q2 =0
Recovered goal states under LQR with varying Q and R
5.50
X
No Control Cost
5.25
1.5
Initial State
Next State
X
5.00
1.4
4.75
Cost
Velocty
1.3
X
4.50
1.2
4.25
1.1
3.75
1.0
3.50
1.0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
Position
Figure 5: Visualistion of high-level actions recovered by the analytical inverse for different LQR parameter settings
C.2 VISUALIZATION OF POLICIES IN ROBOTIC MANIPULATION
We provide visualization of the policies obtained by standard offline RL in Figure which fails to perform the task of opening the door, and OHIO Figure 6b) performing effective offline RL and completing the task.
(a)
(b)
Figure 6: Door opening task with modified stiffness (a) standard offline RL (No Sucess), (b) OHIO (Succes)
29

---

Published as a conference paper at ICLR 2025
C.3 LEVERAGING DIFFERENT DATA SOURCES FOR ROBOTIC MANIPULATION
We introduce "Expert Dataset" comprising 250 episodes collected using an expert policy for the Door Opening task. Using this dataset, we compare:
Hierarchical imitation learning (HILL. This approach applies an imitation learning objective to both the high- and low-level policies. The high-level policy predicts a goal state in the joint space (position and velocity of the robot joints), whereas the low-level learns the robot torques to reach this goal state.
ii) OHIO-based Imitation Learning ("OHIO-IIL") This method employs an imitation learning objective for the high-level policy, combined with the inverse optimization process derived by an operational space controller for the low-level policy. The high-level policy predicts goal state in the operational space (position of the robot end effectors).
As shown Table 13 (left) learning solely from expert demonstrations proves challenging due to limited data coverage, which impacts both HIL and OHIO-based methods and prevents them from consistently solving the task. To address this limitation, we leverage a broader set of demonstrations to enhance offline learning. Specifically, demonstrations from three distinct expert policies, each collected under a different controller setup (200 episodes per policy, totaling 600 episodes) setup commonly encountered in practice. As shown in Table 13 (right), OHIO demonstrates its effectiveness in integrating diverse data sources, outperforming non-O non-OHIO-Bee OHIO-based methods. Furthermore, offline RL objectives enable learning from datasets created by multiple expert policies, consistently outperforming imitation learning approaches (e.g., HIL vs. HRL and OHIO-IL vs. OHIO-IQL).
construct "Combined Expert Dataset" by merging
Table 13: Normalized scores on the door opening task
Expert Dataset
Combined Expert Dataset
HIL
OHIO IL
HIL
HRL
OHIO
OHIO IQL
57.73
+:39.2
26.44 +:37.9
29.89 +34.8
64.75 +:37.7
81.85 +-33.1
91.1 +:3.2
C.4
COMPARISON SAMPLE-EFHCIEENY DURING ONLINE TRAINING
We provide the training curves in Figure 7 to demonstrate the significant improvement in sample-efficiency and learning stability of hierarchical RL over traditional E2E RL.
10000
Newors
110000
20000
30000
Bi-level RL E2E RL
40000
54 60
Training steps (10')
Figure 7: Training curve for online hierarchical (Bi-level) RL and E2E RL in the supply chain W 10S experiment.
C.5 COMPARISON OF RUNTIME AT INFERENCE
We perform additional analyses times for the DVR problem (Table 14). We show how, despite the substantially increased graph sizes, the computation time of our method remains tractable across different real-world scale.
30

---

Published as a conference paper at ICLR 2025
Table 14: Runtimes of OHIO inference the Dynamic Vehicle Routing environment on different graph sizes.
No. Edges
16,000
10,000
40,000
90,000
E2E
s
s.04
(..00))
OHIO
0.09 s (+0.01
0.73
s
14.90 (=225)
MPC-Oracle (T=6)
0.47 (.00))
(
2
44.13 (.76)
MPC-Oracle (T=12)
1.69 s ( 0.28)
4
85.23
163.06 s (# 11.29)
C.6 ONLINE HINE-TUNING OF OHIO POLICY NETWORK OPTIMIZATION SCENARIOS
We further evaluate the performance of the policy learned by OHIO during online fine-tuning, both in-distribution (i.e., within the same city) and in a transfer learning setting (i.e., requiring adaptation to different city, with unseen topology, demand patterns, travel times, etc.). Results in Figure @ show how, both cases, OHIO policies are able to reliably improve upon the starting performance learned from offline data. Crucially, the policy learned by OHIO is consistently above the performance of the behavior policy during the entire fine-tuning process, thus avoiding prohibitively expensive low reward interactions during the initial phases of training and potentially alleviating critical bottleneck for the deployment of RI. within real-world systems. In Figure 9 we show how the E2E policy is unable adhere to constraint violations during online fine-tuning.
100
75
Beh. Pol.
Score
Online-RL
Spore
25
FT-E2E
50
OHIO
25
Online
SHZI
0
Training steps (10")
Training steps (10')
(h)
Figure 8: Vehicle routing fine-tuning performance (y-axis) of pre-trained hierarchical policies (FT-OHIO) compared with training from scratch (Online-BLy as function gradient steps deriving from online interaction (X-axis) with either (a) same-city scenario (NYC NYC) (b) in transfer learning setting (NYC-SHH) "Beh. Pol." indicates the performance of the behavior policy.
100
75
Excoss Capachty
6000
Beh. Pol. FT-OHIO
50
Beh, Pol. FT-OHIO
a000
FTEEZ
25
FT-E2E
12
16
20
12
14
18
20
Training steps (10')
Training stops (10")
(a)
(b)
Figure 9: Supply chain fine-tuning performance (a) and constraint violation (b) of OHIO (FT-OHIO) and end-to-end (FT-E2E) policies pre- trained on near-optimal data (i.e., MPC).
31


Reviews:
[{"review_id": "y2W4eIV3mR", "reviewer": "Reviewer_a4kN", "summary": "Authors introduce a new offline HRL approach that can recover high level actions given trajectories of state transitions.  To find the most likely high level actions, a new simple maximum likelihood approach is used.  They show their approach is able to solve a series of robotics and network optimization tasks.", "questions": "1. Can you clarify what the observed state baseline is?  It seems that it is a high-level policy that always selects the next state from a trajectory as the subgoal?  Why was the next state chosen instead of e.g., 10 timesteps later?  Can you provide some intuition on why is performed poorly?\n2.  Can you provide some data on the time horizon (i.e., number of primitive actions) to complete each task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The approach to discovering the high level actions given a set of state transitions is simple.  Select the high level action with the highest likelihood.\n- Significant empirical outperformance versus baseslines\n- Empirical performance was robust to different controllers and modeling errors\n- Applied algorithm to domains with very high dimensional action spaces (i.e., the network optimization problems)", "weaknesses": "The approach does appear to have significant limitations.\n1.  The approach assumes access to an approximate model of the transition dynamics.  The authors argue this can be relatively simple to learn because it only requires a single step, but this can be difficult in high-dimensional and stochastic settings.\n2.  The approach often assumes a pre-trained low-level controller.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sgK7mU9zs5", "reviewer": "Reviewer_7JuN", "summary": "The paper presents a method for obtaining high-level actions from an offline (state only) dataset, by solving the lower level policy inverse problem. The motivation is very clear: High-level, temporal abstractions and hierarchical decomposition is crucial for solving many complex, long horizon problems, therefore, a method for extracting high-level options from an offline dataset that does NOT contain those high-level actions is crucial. In addition, the paper propose to exploit the extracted high-level actions for performing offline reinforcement learning with the resulting high-level dataset, such that the found high-level policy can parameterize the low-level policy at inference time. The proposed method (OHIO) is evaluated on diverse environments (multiple robotics problems as well as network optimization), and performs favorably in comparison with numerous online and offline RL methods.", "questions": "How much is this method affected by the quality of the approximate transition model? It would be extremely interesting to see how well the method performs when the ground truth transition model is used, compared with a model learned from the offline data (I guess this would only be possible when low-level actions are part of the trajectories). Did one of the experiments somehow access this and I missed it?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Very well written, clearly motivated, thoroughly validated both empirically and analytically.\nTo the best of my knowledge this work addresses an important gap in the literature, the possibility to learn high-level actions that parameterize arbitrary low-level policies from state-only trajectories (and assuming approximate knowledge about transition dynamics).", "weaknesses": "Not much here to be honest. The only thing that comes to mind is that it was not immediately clear to me how to solve Eq (6) in practice and in the general case, but this becomes very clear with Appendix A.4 and algorithm 2 and 3. Perhaps the content of A.4 could be more integrated into the main method section. \n\nIt seems to me like method mainly deals with the implicit policy case, therefore, I am not sure how much value is being added by discussing the explicit policy case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PbKW8DdKX4", "reviewer": "Reviewer_dYhu", "summary": "This paper presents OHIO, a framework for offline reinforcement learning (RL) using hierarchical policies. The framework uses inverse optimization to recover high-level actions that generated the observed data. The authors perform experiments in robotic manipulation and network optimization environments, and show improvements against end-to-end and hierarchical policies.", "questions": "Can you provide comparisons of OHIO with prior hierarchical approaches that leverage offline RL, e.g OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning [1] and approaches that leverage expert demonstrations like Relay Policy learning [2]. \n\nCan you discuss how would the method compare against prior methods that leverage expert demonstrations for learning the inherent hierarchical structure, e.g. Relay Policy learning [2].\n\n[1]. OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, Ofir Nachum)\n[2]. Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, Karol Hausman)", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents an interesting approach for effectively leveraging the inherent hierarchical structure for downstream learning.\n\n2. The authors show impressive results on roobotic and network optimization environments, with many ablations pertaining the hierarchical structure.\n\n3. It covers good related work section and the limitations of this method are clearly mentioned in the discussion section.", "weaknesses": "1. The main weaknesses in this paper are the experiments. Although the authors mention prior related hierarchical approaches for offline learning, the comparisons with such sophisticated approaches are missing in the experiment section. This makes it hard to judge the efficacy of the method against prior approaches. A more detailed comparison with such baselines on complex multi-level reasoning tasks is thus necessary.\n\n2. Further, Algorithm 1 provides minimal information for the hierarchical formulation, and can be significantly improved by adding appropriate details of the training procedure. Extensive details on how the hierarchical levels interact with each other during training is also missing. The paper would benefit by adding pseudocode for how the high-level and low-level policies are trained and interact, with details on how the inverse optimization is integrated into the training loop.\n\n3. Minor: Typo in Line 201.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)