ICLR 2025

Published as a conference paper at ICLR 2025
A GENERALIST HANABI AGENT
Arjun Vaithilingam Sudhakar∗1,2,3 Hadi Nekoei∗1,2,4 Mathieu Reymond1,2,3
Janarthanan Rajendran6 Miao Liu5 Sarath Chandar1,2,3,7
1Chandar Research Lab
2Mila – Quebec AI Institute
3Polytechnique Montr´eal
4Universit´e de Montr´eal
5IBM Research
6Dalhousie University
7Canada CIFAR AI Chair
ABSTRACT
Traditional multi-agent reinforcement learning (MARL) systems can develop co-
operative strategies through repeated interactions. However, these systems are
unable to perform well on any other setting than the one they have been trained
on, and struggle to successfully cooperate with unfamiliar collaborators. This is
particularly visible in the Hanabi benchmark, a popular 2-to-5 player coopera-
tive card-game which requires complex reasoning and precise assistance to other
agents. Current MARL agents for Hanabi can only learn one specific game-setting
(e.g., 2-player games), and play with the same algorithmic agents. This is in
stark contrast to humans, who can quickly adjust their strategies to work with
unfamiliar partners or situations. In this paper, we introduce Recurrent Replay
Relevance Distributed DQN (R3D2), a generalist agent for Hanabi, designed to
overcome these limitations. We reformulate the task using text, as language has
been shown to improve transfer. We then propose a distributed MARL algorithm
that copes with the resulting dynamic observation- and action-space. In doing
so, our agent is the first that can play all game settings concurrently, and extend
strategies learned from one setting to other ones. As a consequence, our agent
also demonstrates the ability to collaborate with different algorithmic agents —
agents that are themselves unable to do so. The implementation code is available
at: R3D2-A-Generalist-Hanabi-Agent
1
INTRODUCTION
Humans were able to thrive as a society through their ability to cooperate. Interactions among multi-
ple people or agents are essential components of...