ICML 2025

Reinforcement Learning with Random Time Horizons
Enric Ribera Borrell * 1 2 Lorenz Richter * 1 3 Christof Sch¨utte 1 2
Abstract
We extend the standard reinforcement learning
framework to random time horizons. While the
classical setting typically assumes finite and deter-
ministic or infinite runtimes of trajectories, we ar-
gue that multiple real-world applications naturally
exhibit random (potentially trajectory-dependent)
stopping times. Since those stopping times typi-
cally depend on the policy, their randomness has
an effect on policy gradient formulas, which we
(mostly for the first time) derive rigorously in this
work both for stochastic and deterministic poli-
cies. We present two complementary perspectives,
trajectory or state-space based, and establish con-
nections to optimal control theory. Our numerical
experiments demonstrate that using the proposed
formulas can significantly improve optimization
convergence compared to traditional approaches.
1. Introduction
The goal in reinforcement learning is to identify policies π
that maximize the expected return
J(π) = Eπ
" N
X
n=0
rn(Sn, An)
#
,
(1)
where the states Sn and the actions An run until a time N.
While the classical framework considers the case where N
is either finite and deterministic or infinite, runtimes can as
well be random. To name two examples, one could consider
robots that aim to reach a target in a noisy environment or
one could play a game whose terminating state depends on
reaching a certain level. Furthermore, considering stochastic
policies (as commonly done in practice in order to balance
exploration and exploitation) leads to the fact that runtimes
*Equal contribution 1Zuse Institute Berlin, 14195 Berlin, Ger-
many 2Institute of Mathematics, Free University Berlin, 14195
Berlin, Germany 3dida Datenschmiede GmbH, 10827 Berlin,
Germany.
Correspondence to:
Enric Ribera Borrell <rib-
era.borrell@zib.de>, Lorenz Richter <richter@zib.de>.
Proceedings of the 42 nd International Conference ...