ICML 2025

Exploring Large Action Sets with Hyperspherical Embeddings
using von Mises-Fisher Sampling
Walid Bendada 1 2 Guillaume Salha-Galvan 3 Romain Hennequin 1
Th´eo Bontempelli 1 Thomas Bouabc¸a 1 Tristan Cazenave 2
Abstract
This paper introduces von Mises-Fisher explo-
ration (vMF-exp), a scalable method for explor-
ing large action sets in reinforcement learning
problems where hyperspherical embedding vec-
tors represent these actions. vMF-exp involves
initially sampling a state embedding representa-
tion using a von Mises-Fisher distribution, then
exploring this representation’s nearest neighbors,
which scales to virtually unlimited numbers of
candidate actions. We show that, under theoretical
assumptions, vMF-exp asymptotically maintains
the same probability of exploring each action as
Boltzmann Exploration (B-exp), a popular alter-
native that, nonetheless, suffers from scalability
issues as it requires computing softmax values
for each action. Consequently, vMF-exp serves
as a scalable alternative to B-exp for exploring
large action sets with hyperspherical embeddings.
Experiments on simulated data, real-world public
data, and the successful large-scale deployment of
vMF-exp on the recommender system of a global
music streaming service empirically validate the
key properties of the proposed method.
1. Introduction
Exploration is a fundamental component of the reinforce-
ment learning (RL) paradigm (Amin et al., 2021; McFar-
lane, 2018; Sutton & Barto, 2018). It allows RL agents to
gather valuable information about their environment and
identify optimal actions that maximize rewards (Amin et al.,
2021; Chiappa et al., 2023; Dulac-Arnold et al., 2015; Jin
et al., 2020; Ladosz et al., 2022; McFarlane, 2018; Reynolds,
1Deezer Research, Paris, France.
2LAMSADE, Universit´e
Paris Dauphine, PSL, Paris, France. 3SPEIT, Shanghai Jiao Tong
University, Shanghai, China. Correspondence to: Walid Bendada
<bendadaw@gmail.com>.
Proceedings of the 42 nd International Conference on M...