ICML 2025

Partially Observable Reinforcement Learning with Memory Traces
Onno Eberhard 1 2 Michael Muehlebach 1 Claire Vernade 2
Abstract
Partially observable environments present a con-
siderable computational challenge in reinforce-
ment learning due to the need to consider long his-
tories. Learning with a finite window of observa-
tions quickly becomes intractable as the window
length grows. In this work, we introduce memory
traces. Inspired by eligibility traces, these are
compact representations of the history of observa-
tions in the form of exponential moving averages.
We prove sample complexity bounds for the prob-
lem of offline on-policy evaluation that quantify
the return errors achieved with memory traces
for the class of Lipschitz continuous value esti-
mates. We establish a close connection to the
window approach, and demonstrate that, in cer-
tain environments, learning with memory traces
is significantly more sample efficient. Finally,
we underline the effectiveness of memory traces
empirically in online reinforcement learning ex-
periments for both value prediction and control.
1. Introduction
Learning and acting in partially observable environments
requires memory. It is generally not enough to act simply
according to the most recent observation, and an agent must
therefore keep track of the past, as every piece of informa-
tion is likely to be relevant for future actions. In the absence
of additional assumptions, optimal behavior requires the
agent to reason over its entire history of observations. This
space quickly becomes intractable, and just like with contin-
uous state spaces in fully observed systems, it is therefore
necessary to resort to approximation. This article inves-
tigates the fundamental limitations and trade-offs arising
from these approximations. We focus on extracting features
from the history of observations and analyze the complexity
of learning functions of these features. In contrast to meth-
1Max Planck Institute for Intelligent Sy...