ICLR 2025

Published as a conference paper at ICLR 2025
EXPECTED RETURN SYMMETRIES
Darius Muglich∗
University of Oxford
darius@robots.ox.ac.uk
Johannes Forkel∗
University of Oxford
johannes.forkel@eng.ox.ac.uk
Elise van der Pol
Microsoft Research AI for Science
evanderpol@microsoft.com
Jakob Foerster
University of Oxford
jakob.foerster@eng.ox.ac.uk
ABSTRACT
Symmetry is an important inductive bias that can improve model robustness and
generalization across many deep learning domains. In multi-agent settings, a pri-
ori known symmetries have been shown to address a fundamental coordination
failure mode known as mutually incompatible symmetry breaking; e.g. in a game
where two independent agents can choose to move “left” or “right”, and where a
reward of +1 or −1 is received when the agents choose the same action or differ-
ent actions, respectively. However, the efficient and automatic discovery of envi-
ronment symmetries, in particular for decentralized partially observable Markov
decision processes, remains an open problem. Furthermore, environmental sym-
metry breaking constitutes only one type of coordination failure, which motivates
the search for a more accessible and broader symmetry class. In this paper, we
introduce such a broader group of previously unexplored symmetries, which we
call expected return symmetries, which contains environment symmetries as a sub-
group. We show that agents trained to be compatible under the group of expected
return symmetries achieve better zero-shot coordination results than those using
environment symmetries. As an additional benefit, our method makes minimal a
priori assumptions about the structure of their environment and does not require
access to ground truth symmetries.
1
INTRODUCTION
Incorporating the symmetries of an underlying problem into models has had demonstrable success in
improving generalization and accuracy across many different machine learning domains (Bronstein
et al., 2021; Krizhevsky et al., 2017; Cohen et al., 201...