ICLR 2025

Published as a conference paper at ICLR 2025
SCALABLE DECISION-MAKING IN STOCHASTIC EN-
VIRONMENTS THROUGH LEARNED TEMPORAL AB-
STRACTION
Baiting Luo1, Ava Pettet3, Aron Laszka2, Abhishek Dubey1, Ayan Mukhopadhyay1
1Vanderbilt University, 2Pennsylvania State University, 3Nissan Advanced Technology Center
{baiting.luo, abhishek.dubey, ayan.mukhopadhyay}@vanderbilt.edu
alaszka@psu.edu, ava.pettet@nissan-usa.com
ABSTRACT
Sequential decision-making in high-dimensional continuous action spaces, partic-
ularly in stochastic environments, faces significant computational challenges. We
explore this challenge in the traditional offline RL setting, where an agent must
learn how to make decisions based on data collected through a stochastic behav-
ior policy. We present Latent Macro Action Planner (L-MAP), which addresses
this challenge by learning a set of temporally extended macro-actions through
a state-conditional Vector Quantized Variational Autoencoder (VQ-VAE), effec-
tively reducing action dimensionality. L-MAP employs a (separate) learned prior
model that acts as a latent transition model and allows efficient sampling of plau-
sible actions. During planning, our approach accounts for stochasticity in both the
environment and the behavior policy by using Monte Carlo tree search (MCTS).
In offline RL settings, including stochastic continuous control tasks, L-MAP ef-
ficiently searches over discrete latent actions to yield high expected returns. Em-
pirical results demonstrate that L-MAP maintains low decision latency despite
increased action dimensionality. Notably, across tasks ranging from continuous
control with inherently stochastic dynamics to high-dimensional robotic hand ma-
nipulation, L-MAP significantly outperforms existing model-based methods and
performs on-par with strong model-free actor-critic baselines, highlighting the ef-
fectiveness of the proposed approach in planning in complex and stochastic envi-
ronments with high-dimensional action spaces.
1
INT...