Use the Online Network If You Can: Towards Fast and Stable Reinforcement
  Learning
Authors: name: Ahmed Hendawy, name: Henrik Metternich, name: Théo Vincent, name: Mahdi Kallel, name: Jan Peters, name: Carlo D'Eramo

The use of target networks is a popular approach for estimating value
functions in deep Reinforcement Learning (RL). While effective, the target
network remains a compromise solution that preserves stability at the cost of
slowly moving targets, thus delaying learning. Conversely, using the online
network as a bootstrapped target is intuitively appealing, albeit well-known to
lead to unstable learning. In this work, we aim to obtain the best out of both
worlds by introducing a novel update rule that computes the target using the
MINimum estimate between the Target and Online network, giving rise to our
method, MINTO. Through this simple, yet effective modification, we show that
MINTO enables faster and stable value function learning, by mitigating the
potential overestimation bias of using the online network for bootstrapping.
Notably, MINTO can be seamlessly integrated into a wide range of value-based
and actor-critic algorithms with a negligible cost. We evaluate MINTO
extensively across diverse benchmarks, spanning online and offline RL, as well
as discrete and continuous action spaces. Across all benchmarks, MINTO
consistently improves performance, demonstrating its broad applicability and
effectiveness.