<figure><img src="image_1.png" /></figure>

HAL
open science

# Buzz, Choose, Forget: A Meta-Bandit Framework for
Bee-Like Decision Making

Emmanuelle Claeys, Elena Kerjean, Jean-Michel Loubes

<figure><img src="image_2.png" /></figure>

## To cite this version:

Emmanuelle Claeys, Elena Kerjean, Jean-Michel Loubes. Buzz, Choose, Forget: A Meta-Bandit Framework for Bee-Like Decision Making. 2025. hal-05374841

HAL Id: hal-05374841

https://ut3-toulouseinp.hal.science/hal-05374841v1

Preprint submitted on 20 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
BUZZ, CHOOSE, FORGET: A META-BANDIT FRAMEWORK FOR BEE-LIKE DECISION MAKING

**Emmanuelle Claeys,**
University of Toulouse, IRIT, Toulouse, France
*emanuelle.claeys@irit.fr*

**Elena Kerjean**
University of Toulouse, CBI, Toulouse, France
*elena.kerjean@utoulouse.fr*

**Jean-Michel Loubes**
Regalia Team, INRIA
University of Toulouse, France *jean-michel.a.loubes@inria.fr*

ABSTRACT

We introduce a sequential reinforcement learning framework for imitation learning designed to model heterogeneous cognitive strategies in pollinators. Focusing on honeybees, our approach leverages trajectory similarity to capture and forecast behavior across individuals that rely on distinct strategies: some exploiting numerical cues, others drawing on memory, or being influenced by environmental factors such as weather. Through empirical evaluation, we show that state-of-the-art imitation learning methods often fail in this setting: when expert policies shift across memory windows or deviate from optimality, these models overlook both fast and slow learning behaviors and cannot faithfully reproduce key decision patterns. Moreover, they offer limited interpretability, hindering biological insight. Our contribution addresses these challenges by (i) introducing a model that minimizes predictive loss while identifying the effective memory horizon most consistent with behavioral data, and (ii) ensuring full interpretability to enable biologists to analyze underlying decision-making strategies and finally (iii) providing a mathematical framework linking bee policy search with bandit formulations under varying exploration-exploitation dynamics, and releasing a novel dataset of 80 tracked bees observed under diverse weather conditions. This benchmark facilitates research on pollinator cognition and supports ecological governance by improving simulations of insect behavior in agroecosystems. Our findings shed new light on the learning strategies and memory interplay shaping pollinator decision-making.

# 1 INTRODUCTION

Over the past decade, researchers have increasingly turned to artificial intelligence (AI) and computational modeling to replicate or simulate animals’ decision processes, referred to as imitation learning (Cully et al., 2015). In this case, the goal is to train an agent to learn by observing and reproducing the animal’s behavior in the same way as if the animals were experts. In particular, reinforcement learning (RL) frameworks have gained increasing attention as a way to describe how animals learn from trial and error, as an alternative to statistical models or simple heuristic rules. These RL models serve a dual purpose: they help biologists to understand how these animals learn to facilitate rule discovery (Wason, 1960) (i.e. policy modelisation) from real animal data experiments, and they can also be used to test ecological decisions in simulation. However, the state-of-the-art models have some difficulties in representing bee behaviors for some reasons: (1) some of them exclude the balance between contextual and non-contextual strategies in the decision process modeling. (2) These models overlook the archetypical mechanism in limited memory observable reinforcement learning (RL): we define here the *memory* of the animal by a parameter, $\tau$, that truncates the observation history to the $\tau$ most recent observations. This parameter needs to be optimized in the imitation learning. (3) These models assume homogeneity among bees, although individuals may exhibit distinct behaviors and no explainability is given for each individual. In fact, some bees
are able to understand the context information to limit the regret in their strategies, and some others do not Gurfa et al. (2022). The goal is then to provide a model that can explain and forecast the policy of each bee. This paper proposes a new algorithm to model bees behaviors focusing on contextual binary foraging tasks (scenarios with two alternatives in a Y-maze, with left vs. right choices, where the reward depends on a one-dimensional contextual information). We summarize key methodologies and show (1) how to identify the best τ window size, (2) how individuals vary according to their strategies, and (3) that by combining bandit RL algorithms with several similarity measures, we can forecast any individual's policy regardless of their specific skills. Our method is summarized in Fig. 1. Our code is open-source and our data are openly available.<fnref n="1" />

**A new imitation learning framework** Imitation learning (IL) enables agents to acquire behavior from expert demonstrations in order to limit costly or unsafe exploration Zhao et al. (2020). By grounding policy optimization in expert trajectories, IL offers a sample-efficient framework to capture adaptive strategies. However, when the goal is to imitate different experts rather than efficiently learning the optimal policy, many IL methods fail when the expert has a non-optimal strategy. This is mostly explained by the fact that they prioritize policy optimization over expert imitation. Unlike classical IL, biological experts (such as bees) often follow non-optimal policies. In nature, several policies may coexist, and a single expert can change its strategy over time. This is largely due to the limited memory of insects, which restricts decisions to a short history of past actions and rewards. Therefore, attention must be paid to how the expert guides the agent, and how the agent adapts to multiple experts. It requires not only defining what should be imitated, but also handling this memory limitation. Our contribution is MAYA (Multi Agent Y-maze Allocation) which enables bee imitation learning on a sequential two-choice learning (Y-maze). MAYA combines several multi armed bandit (MAB) policies (including random and contextual variants) with a fixed memory setting for similarity evaluation. Similarity evaluation can be based on probability of success (with Kullback-Leibler or Wasserstein distance) or on trajectory (with Dynamic Time Warping DTW similarity). The best choice of similarity is made according to the ability to imitate the expert and limit the cumulative cost of wrongly replicated actions over trials. Then, our paper studies the similarity that should be used.

**Understanding the learning skill** MAYA models bee policies as mixtures of multiple agents, thereby providing a quantitative framework for analyzing behavioral variability. Since bees possess limited memory of past experiences, their decision policies may shift over time. To capture this, MAYA decomposes the observed trajectories into segments that align with distinct agent models, each defined by a specific MAB. These MAB vary according to their strategies: pure exploration, deterministic or stochastic reward-based choice between left and right arms of the Y-maze, and context-dependent strategies where external cues guide decisions (see App. 13). By structuring bee behavior as a combination of such agents, MAYA not only reproduces expert trajectories but also yields an interpretable description of policy shifts and memory constraints.

**Window-size discovering** MAYA requires the specification of a sliding window $τ \in T$ in order to select the importance of the past information used to align the behavior of the bee and the MAB. Consequently, similarity evaluations are restricted to a finite history of previous trials. In our experiments, we assess how this setting influences the ability to imitate the bee. We find that the optimal window length decreases under adverse weather conditions, but generally stabilizes around seven past trials across all datasets. As a complementary analysis, we also include experiments with mice, where a similar optimal setting emerges. This suggests that this memory-related hyperparameter reflects a biologically grounded constraint observed across different species.

**Open dataset and ecological insights** We release to the community a new open dataset recording experiments on 80 bees (with [22 – 40] sequential trials per bee) across 5 diverse situations (favorable and adverse weather, in Oceania and Europe). More details about the experiment are given in App 6.1. In each experiment, a bee enters a Y-maze where it is exposed to a number of visual stimuli presented on both the left and right arms. The reward is consistently located on the side displaying the greater number of stimuli. During a session, each bee performs between 22 and 40 trials, where the rewarded side has been randomly assigned at each trial.

<footnote n="1">¹https://github.com/emmanuelleclaeys/maya</footnote>
<figure><img src="image_4.png" /><figcaption>Figure 1: MAYA (Multi-Agent Y-maze Allocation) is an imitation learning framework for policy selection via windowed regret matching. Leveraging logged bee trajectories and three similarity metrics (Wasserstein, KL, DTW), MAYA maps learning dynamics onto 2-armed bandit strategies (UCB, Epsilon-greedy, LinUCB, Uniform). Beyond performance alignment, MAYA provides interpretability of bee behaviors by revealing differences in memory span and learning aptitude, thereby distinguishing “good learners” from “poor learners” in contextual experiments.</figcaption></figure>

## 2 PRELIMINARIES

**Problem formulation.** We model the bee prediction task (forecast the decision left or right) as an RL problem. At each trial $t \in 1, \dots, T$ the environment reveals a state $s_t \in S$ described by the number of trial $t$ and available contextual information : $x_t \in \mathbb{R}^2$ with $x_t = (\text{stimuli on Left and Right side, weather,})$. The bee selects an action (i.e. chose a side) : $a_t \in A := \{L, R\}$, corresponding to Left and Right. Then, the bee receives a reward $r_t = r(s_t, a_t) \in \{0, 1\}$, which captures whether the choice is correct or incorrect (e.g., sugar or quinine). This model is actually a Markov Decision Process (MDP) Sutton & Barto (2018). It is defined as a tuple $(S, A, P, \mathcal{R})$ with a state space $S$, an action space $A$. In our setting, $S = |S|$ and $A = |A|$ are finite (i.e $S, A < \infty$). The quantity $P = (P_a : a \in A)$ is called the transition function with $P_a : S \times S \to [0, 1]$ and so $P_a(s, s')$ is the probability that the agent moves from state $s$ in state $s'$ according to action $a$. The set space $\mathcal{R}$ is defined by all outputs of reward functions $r_a$ according an action $a : \mathcal{R} = (r_a : a \in A)$. We are on a discrete-time series system such as the initial state is defined by $S_1$. In each round $t$ the agent observes the state $S_t \in S$, chooses an action $A_t \in A$ and receives the reward $r_{A_t}(S_t)$. The environment then samples $S_{t+1}$ from the probability vector $P_{A_t}(S_t) \in P$. The **history** $H_t = (S_1, A_1, r(S_1, A_1), \dots, S_{t-1}, A_{t-1}, r(S_{t-1}, A_{t-1}), S_t)$ or more simply $H_t = (S_1, A_1, r_1, \dots, S_{t-1}, A_{t-1}, r_{t-1}, S_t)$, contains the information available before the action for the round $t$ is to be chosen. A **policy** is a (possibly randomised) map from the set of possible histories to actions. The set of such policies is denoted by $\Pi$ and its elements are identified with maps $\pi : A \times S \to [0, 1]$ with $\sum_{a \in A} \pi(a|s) = 1$ for any $s \in S$ so that $\pi(a|s)$ is interpreted as the probability that policy $\pi$ takes action $a$ in state $s$. We are on a finite-trial experiment i.e. $t \in \{1, \dots, T\}$ where $T$ is the total number of trials. We consider here that all rewards are equivalent whatever the future, then an **optimal policy** $\pi^*$ for a discrete time $T$ system is a policy that satisfies, for any state $s : \pi^* = \arg\max_{\pi \in \Pi} \sum_{t=1}^T \gamma^t r(A_t, S_t)$ with $\gamma = 1$. Finally, let $N_t(a)$ denote the total number of times action $a$ has been selected up to round $t$. We define $Q_t(a) = \frac{1}{N_t(a)} \sum_{j=1}^{t-1} r_j 1_{\{a_j=a\}}$ as the simple average of rewards which have been observed.

**Regret.** Let $\pi^*$ denote the (unknown) optimal policy. The instantaneous regret at trial $t$ is defined as $\Delta_t = r(s_t, a_t^*) - r(s_t, a_t)$, where $a_t^* := \pi^*(s_t) = \underset{a \in A}{\argmax} r(s_t, a)$ is the optimal action under the state $s_t$. The cumulative simple regret after $T$ trials is the sum of instantaneous regrets $R(\pi, 1, T) = \sum_{t=1}^T \Delta_{\pi, t}$.

In the experiment, the reward given to the bee at each state $s_t$ does not depend on the state $s_{t-1}$. Hence our model can be seen as a 2-armed bandit problem and not a classical reinforcement learning
problem. Bees differ in solving such learning task. Based on biology literature Capela et al. (2024), their different behaviors can be modeled by four different two-armed bandit strategies (MAB) :

1. **Epsilon Greedy** Sutton & Barto (1998) : exploits current action that maximize observed average reward (i.e. .) and explore the other action according a small probability (ε).

$$A_t = \begin{cases} \operatorname{Argmax}_a[Q_t(a)] & \text{with probability } 1 - \epsilon \\ a \sim \operatorname{Uniform}(\mathcal{A} \setminus \{\operatorname{Argmax}_a[Q_t(a)]\}) & \text{with probability } \epsilon \end{cases}$$

2. **Optimistic stategy UCB style** Auer et al. (2002) : construct an adaptative upper confidence bound around $Q_t(a)$. In this case UCB1 chose according :

$$A_t = \operatorname{Argmax}_a[Q_t(a) + \sqrt{\frac{\ln t}{N_t(a)}}].$$

The number of trials on $N_a(t)$ and empirical observed reward on each arms are considered.

3. **Contextual-multi-armed bandits (CMAB) LINUCB style** Li et al. (2010) : At the beginning of trial $t$, the agent observes a context $x_t$. It's redefine the choice of an action according the context information $x_t$. Let $G_a = X_a^\top X_a + \lambda I$ where $X_a$ is the matrix with the context vectors of action $a$ as rows, $I$ the identity matrix and $\lambda \in \mathbb{R}$ is a regularization parameter. LINUCB1 chose according :

$$A_t = \operatorname{Argmax}_a[x_t^\top \hat{\Theta}_{a,t} + \sqrt{x_t^\top G_a^{-1} x_t}].$$

where $\hat{\Theta}_{a,t} \in \mathbb{R}^2$ are estimated parameter of action $a$ at $t$.

4. **Random choice strategy UNIFORM style** : At each trial, the agent chooses an action uniformly at random, independently of past observations or contexts. This baseline strategy does not exploit reward or contextual informations, and serves as a comparison.

$$A_t \sim \operatorname{Uniform}(\mathcal{A}).$$

Among the strategies considered above, LINUCB1 (ref as LINUCB) is the only bandit algorithm here that explicitly incorporates contextual information. Consequently, it is the sole approach capable of asymptotically converging to the optimal policy in our Y-maze experimental setting. Regardless of its ability to adopt the optimal strategy (i.e., to use contextual information), the bee selects an action $A_t$ based on memory history. This memory reflects the history of past actions, rewards, and contexts. However, learning and memory of honeybees can be impacted by a large amount environmental conditions, like the weather variation Gérard et al. (2022). Additionally, the learning process in itself may be reflected by the succession of sub-optimal strategy (based on $Q_a(t)$ or based on a random choice) to the optimal strategy (based on the contextual information) with potential transitive states. Therefore, comparing bee strategies with these four policies must be carried out in a **non-stationary** framework. Unfortunately, the effective history length is difficult to anticipate, as it may evolve in different ways: piecewise-constant with abrupt changes at unknown breakpoints, smoothly varying with gradual trends, monotonically increasing or decreasing, or within bounded variation, where the total change over time remains limited Fiandri et al. (2024).

Then, to incorporate this bee's memory concept, defined in psychology as the recency effect Glanzer & Cunitz (1966), we introduce the concept of a sliding window $\tau \in \mathcal{T}$ to lay the stress on recent data. The history becomes $H_{\tau, \tau} = (S_{\tau-\tau}, A_{\tau-\tau}, r_{\tau-\tau}, \dots, S_{\tau-1}, A_{\tau-1}, r_{\tau-1}, S_{\tau})$ and the policy becomes $\pi : \mathcal{A} \times \mathcal{S} \times \mathcal{T} \to [0, 1]$ with $\sum_{a \in \mathcal{A}} \pi(a|s, \tau) = 1$. The simple regret according $\tau$ is : $R(\pi, \tau, 1, T) = \sum_{t=\tau}^{T} \Delta_{\pi,t}$.

**Imitation learning to approximate a bee's behaviour** Our goal is to learn a policy $\pi_\theta$ which is close to $\pi_{\text{bee}}$. The selection of the best MAB algorithm that mimics a bee's behaviour will be achieved by looking at the trajectories of their regrets. For this, for a well chosen similarity distance $d$, we define for two policies $\pi_1$ and $\pi_2$, their distance $\delta(\pi_1, \pi_2, 1, T) := d((\Delta_{\pi_1,t})_{t=1}^T, (\Delta_{\pi_2,t})_{t=1}^T)$.

Note that, if we take into account the memory effect, $\delta$ becomes $t$ and $\tau$ adapted as follows $\delta(\pi_1, \pi_2, \tau, T) := d((\Delta_{\pi_1,t})_{t=\tau}^T, (\Delta_{\pi_2,t})_{t=\tau}^T)$. In the following, we will consider for $d$ three different distances: the sequence of regrets is on the one hand considered as a sequence of random
variables and the natural distances between the sequence of regrets are distributional distances such as Kullback-Leibler (KL) divergence (related to similarity for probabilistic inference) or Wasserstein distance (capturing geometric information between distributions); on the other hand, the sequence of regrets is considered as a trajectory of positions and a suitable choice is the Dynamic Time Warping (DTW) similarity, which focuses on temporal alignment.

Finally, the success of the imitation learning algorithm will be quantified here using the following cost of a wrong **reproduced action**. Let :

$$c(s_t|a_t) = \begin{cases} 1 & \text{if } a_t \neq \pi_{\text{bee}}(s_t) \\ 0 & \text{otherwise} \end{cases}$$

Assume that $\pi_{\theta}(a \neq \pi_{\text{bee}}(s)|s) \le \epsilon$, with $\epsilon \in [0, 1]$ then Ross et al. (2010) shows that $\mathbb{E}[\sum_{t=1}^{T} c(s_t, a_t)] \le \epsilon T$. If $\pi_{\theta}$ is learned by minimizing previous distances, success is measured by considering this cost.

## 3 CONTRIBUTION

Our contribution, the MAYA (Multi-Agent Y-maze Allocation) algorithm addresses the challenge in biology of inverse reinforcement learning when expert demonstrations are heterogeneous and not necessarily optimal. Rather than assuming a single “perfect” expert, MAYA models bee trajectories by dynamically aligning them with candidate MAB policies, thereby capturing both successful and sub-optimal learning behaviors. We present here a condensed version of MAYA; the complete procedure is provided in Appendix (App. 11).

**Inputs.** The algorithm takes as input the logged regret trajectory of a bee policy $R(\pi_{\text{bee}}, 1, T)$, a finite set $\mathcal{P} = \{\pi_1, \dots, \pi_N\}$ of $N$ candidate bandit policies, and a window size $\tau$. The window size controls how much historical regret information is used at each step: for $t < \tau$ the algorithm uses all past data, while for $t \ge \tau$ it only considers the most recent $\tau$ steps.

**Initialization.** The algorithm initializes a placeholder policy $\pi_{\theta}$ and an agent buffer $\xi$.

**Warm-up Phase ($t < \tau$).** For each time step $t \in \{2, \dots, \tau - 1\}$:

1. The algorithm observes the bee regret $R(\pi_{\text{bee}}, 1, t - 1)$ with the context information $x_t$.

2. For each candidate policy $\pi_i \in \mathcal{P}$, the algorithm simulates its action distribution $\pi_i(s_{t-1}|x_t)$ and computes the cumulative regret $R(\pi_i, 1, t - 1)$. w

3. A distance $d(\cdot, \cdot)$ is computed between the bee regret trajectory and the simulated regret of $\pi_i$, then we compute $\xi_t = \text{argmin}_{\pi \in \mathcal{P}} \delta(\pi_{\text{bee}}, \pi, t)$ according to the choice of $d(\cdot, \cdot)$. In case of a tie, $\xi_t$ is sampled from the set of best candidates.

4. The algorithm updates $\pi_{\theta}$ to imitate $\pi_{\xi_t}$, i.e. $\pi_{\theta}(a_t|s_{t-1}) \leftarrow \pi_{\xi_t}(a_t|s_{t-1})$, and we store $\xi_t$.

The chosen policy $\pi_{\theta}$ is then used to sample an action $A_t$, a reward $r_t$ is received, and all candidate policies are updated.

**Windowed Phase ($t \ge \tau$).** For subsequent steps $t \in \{\tau, \dots, T\}$, the procedure is analogous, except that only the most recent observations $\tau$ are used when computing regret and $\xi_t = \text{argmin}_{\pi \in \mathcal{P}} \delta(\pi_{\text{bee}}, \pi, \tau, t)$. Specifically, regret and policy regrets are evaluated over the interval $[t - \tau, t - 1]$ rather than the full trajectory. Again, the best match $\xi_t$ is calculated between each policy $\pi \in \mathcal{P}$, and $\pi_{\theta}$ is updated according to the best match.

**Output.** After $T$ steps, the algorithm returns the policy $\pi_{\theta}$, which best matches the bee's regret profile, while adapting online to the context and rewards.
## 3.1 SIMILARITY EVALUATION

The algorithm depends on the choice of the distance $d$ between the trajectories of the regrets. We will consider three distinct distances. For a review of different distances see for instance in Besse et al. (2015).

1. **Dynamic Time Warping (DTW).** One of the most used similarity measures between two paths is given by the so-called DTW. It is defined as follows. Given two temporal sequences $X = (x_1, \dots, x_{T_1})$ and $Y = (y_1, \dots, y_{T_2})$ over $E \subset \mathbb{R}^d$ with $d \in \mathbb{N}^*$. DTW aligns them by finding an admissible path $\psi = \{(i_k, j_k)\}_{k=1}^K$ that respects temporal ordering. Formally, the DTW is defined as $DTW(X, Y) = \min_{\psi} \sum_{k=1}^K \|x_{i_k} - y_{j_k}\|$, with $K \in \mathbb{N}^*$ where the minimization runs over all monotone alignment paths $\psi$ between the indices of $X$ and $Y$. This distance enables comparison of sequences with different lengths or temporal distortions, by optimally stretching or compressing the time axis.

2. **KL-distance.** In this case, the sequence of the regrets is considered as a realization at each step of a Bernoulli distribution. Hence we can define the Kullback Leibler distance between each trajectory by a proper normalization. Set $Q$ a probability measure on $E$. If $P$ is another probability measure on $(E, \mathcal{B}(E))$, then the KL divergence is $D_{KL}(P||Q) = \int_E \log \frac{dP}{dQ} dP$, if $P \ll Q$ and $\log \frac{dP}{dQ} \in L^1(P)$, and $+\infty$ otherwise.

3. **Wasserstein-distance.** We consider again the distributional point of view. The 1-Wasserstein distance is defined as follows. For two distributions $\pi_1$ and $\pi_2$ over $E \subset \mathbb{R}^d$ a compact subset, endowed with the norm $\|\cdot\|$, recall that their 1-Wasserstein distance is defined as $W_1(\pi_1, \pi_2) = \min_{\pi \in \Pi(\pi_1, \pi_2)} \int_{x \in E, y \in E} \|x - y\| d\pi(x, y)$, where $\Pi(\pi_1, \pi_2)$ denotes the set of distributions on $E \times E$ with marginals $\pi_1$ and $\pi_2$.

## 3.2 THEORETICAL ANALYSIS

We provide in App 14 worst-case upper bounds on the cumulative regret gap between $\pi_{MAYA}$ and $\pi_{bee}$ across stationary and cyclic regimes, expressed in terms of $T$ and the memory window $\tau$. We inform the choice of $\tau$ to control the error in non-stationary settings.

# 4 EXPERIMENTAL EVALUATION

Our experiments aim to address the following questions: i/ What is the best window size and similarity metric to approximate bee learning? ii/ What information can MAYA provide about the exploratory and contextual process of bees? iii/ How MAYA can adapt itself to contextual, none contextual, fast and slow learners. iv/ How external information (here the weather) can impact the window size?

**Experiment description** The datasets vary according to location (three from France and two from Australia) and weather conditions (two cold, one moderate, and two hot). Each dataset contains the trajectories of 16 bees with 22 or 40 trials (depending on the dataset, see App 6.1 for more details). We also include a complementary experiment in the App 12, adapted from Ashwood et al. (2020b), using data from mice performing perceptual decision-making tasks.

**Metrics** MAYA and comparative methods are evaluated based on their ability to minimize the cost of incorrectly reproduced actions over the sequence of trials. We then report, for our five datasets, the $MSE(\sum_{t=1}^T c(s_t | a_t))$ and $MAE(\sum_{t=1}^T c(s_t | a_t))$, computed across all bees within the same dataset. We first observe how these metrics evolve in Sec.4.1. Then, in Sec. 4.2, we show how MAYA generates trajectories that closely match those of bees across all datasets. Sec 4.3 provides an individual analysis. We also observe how trajectories are clustered in a similar manner in Sec.4.4. This can be considered as an additional performance metric: the ability to assign trajectories to the same cluster.
## 4.1 BEST WINDOW SIZE AND DISTANCE METRICS

Figure 2 reports the average MSE and MAE results according to $\tau$ for the five datasets. When several MAB agents are candidates for $\xi_t$, the selection is random, which can introduce some variability. For readability, we report only the average MSE/MAE over 1,000 simulations, and we provide the standard deviation for several $\tau$ in the App 6.2. However, the recorded standard deviations are small and nearly constant. This is easily explained: if agents follow the same action sequence, their costs are identical. Therefore, the effect of randomness is limited. This can be seen in Fig.3, for example, where UCB and UNIFORM act identically at the beginning of the experiment.

Across all datasets, the results confirm the trend that for $\tau \in [5, 10]$ the losses decrease. However, weather influences the optimal $\tau$. Cold weather requires to choose $\tau \in [5, 7]$, moderate weather $\tau \in [6, 8]$, and hot weather $\tau \in [7, 10]$. According to this observation, we suggest fixing $\tau = 7$ to handle multiple weather conditions. This choice is also the best parameter in complementary experiments with mice (App 12). Then, we fix $\tau = 7$ for the rest of the paper. With this window, MAYA-Wass provides the best results across all datasets.

<figure><img src="image_5.png" /><figcaption>Figure 2: Comparative study of the best window size $\tau$ by average MSE and MAE; weather and location for each dataset are provided. The maximum window value corresponds to using the full sequence (i.e., no window). \* symbol refers as best performance according standard deviation and average reward (see Tab.8. in App 9 for the full results)</figcaption></figure>

## 4.2 COMPARATIVE STUDY OF REPRODUCTIVE BEHAVIOR

We compare the performance of MAYA-Wasserstein, MAYA-KL and MAYA-DTW with all IRL algorithms implemented in the imitation library of Gleave et al. (2022). It includes implementations of Generative Adversarial Imitation Learning (GAIL), Behavioral Cloning (BC), Dataset Aggregation (DAgger), Adversarial Inverse Reinforcement Learning (AIRL), Density-based reward modeling (DBR), Reward Learning through Preference Comparisons (Pref-Comp), Maximum Causal Entropy Inverse Reinforcement Learning (MCE) and Soft Q Imitation learning (SQIL). These methods are the baseline references of IRL methods. We provide a full explanation of these methods in App 8. We also compare our results with a generalized linear model (GLM) applied to the full trajectory. In this case, the GLM captures each bee trajectory through a response transformation, while allowing the variance of each measurement to depend on its predicted value. We further introduce a variant that incorporates contextual information $x_t$ as covariates (GLM-Context).

The reported results are in Tab 1 for the MSE. As the best performances are almost identical for the MAE we provide MAE results in App 8.1. Methods such as Pref-comp, MCE, and DBR tend to overshoot bee trajectories and focus mainly on minimizing regret (policy optimization), as the context provides all the necessary information to choose correctly. These methods fail to reproduce bee behavior: the loss between the bee regret trajectory and the policy increases over time, while the learned policy achieves only a small cumulative regret. In fact, these methods generally act like
<table><caption>Table 1: MSE comparison of methods across the five datasets. Values are reported as mean ± standard deviation. We fix τ = 7 for all MAYA variant. Best performance of comparative methods are reported here.</caption><thead><tr><th>Dataset</th><th>GAIL</th><th>BC</th><th>AIRL</th><th>Dagger</th><th>DBR</th><th>MCE</th><th>Pref-Comp</th><th>SQIL</th><th>GLM (no ctx)</th><th>GLM (ctx)</th><th>MAYA-KL</th><th>MAYA-Wass</th><th>MAYA-DTW</th></tr></thead><tbody><tr><td>1</td><td>29.6 ± 41</td><td><b>5.16 ± 3</b></td><td><b>0 ± 0</b></td><td>22.8 ± 32</td><td>43.1 ± 54</td><td>148.83 ± 38</td><td>104.57 ± 57</td><td>26.2 ± 19</td><td><b>3.0 ± 1</b></td><td><b>3.0 ± 1</b></td><td>4.2 ± 3</td><td><b>2.5 ± 1</b></td><td>6.7 ± 7</td></tr><tr><td>2</td><td>23.2 ± 17</td><td><b>2.86 ± 2</b></td><td><b>0 ± 0</b></td><td>9.67 ± 12</td><td>15.26 ± 16</td><td>49.5 ± 14</td><td>24.54 ± 18</td><td>9.8 ± 6</td><td><b>1.4 ± 2</b></td><td><b>1.4 ± 2</b></td><td>1.6 ± 1</td><td><b>1.5 ± 1</b></td><td><b>1.3 ± 3</b></td></tr><tr><td>3</td><td>27.5 ± 40</td><td><b>5.5 ± 4</b></td><td><b>0 ± 0</b></td><td>21.6 ± 46</td><td>41.38 ± 51</td><td>140.3 ± 34</td><td>125.7 ± 44</td><td>22.6 ± 15</td><td><b>3.1 ± 1</b></td><td><b>3.1 ± 1</b></td><td>3.7 ± 3</td><td><b>2.6 ± 1</b></td><td>5.7 ± 5</td></tr><tr><td>4</td><td>25.3 ± 39</td><td><b>5.35 ± 4</b></td><td><b>0 ± 0</b></td><td>22.9 ± 34</td><td>46.06 ± 55</td><td>148.2 ± 39</td><td>124.1 ± 52</td><td>25.3 ± 20</td><td><b>3.0 ± 1</b></td><td><b>3.0 ± 1</b></td><td>3.7 ± 3</td><td><b>3.6 ± 2</b></td><td>8.3 ± 10</td></tr><tr><td>5</td><td>47.7 ± 45</td><td>26.7 ± 42</td><td><b>0 ± 0</b></td><td>25.8 ± 47</td><td>115.7 ± 242</td><td>374 ± 311</td><td>284 ± 254</td><td><b>25.0 ± 16</b></td><td>8.0 ± 8</td><td><b>7.9 ± 8</b></td><td><b>3.4 ± 3</b></td><td>4.5 ± 5</td><td>10.3 ± 11</td></tr></tbody></table>

LinUCB. GAIL, Dagger and SQIL fail to capture the full range of behaviors, instead tending to mimic the most frequently represented populations in the dataset. AIRL reproduces bee trajectories identically, and can therefore be seen as a full copy-paste of the dataset without any real capacity for generalization. This leads to overfitting, where the model memorizes observed trajectories rather than capturing the underlying decision-making mechanisms. When we fine-tune the parameters of these methods, we reinforce the influence of the expert on the learning process (see App 9). However, this requires more computation time, and the resulting MSE/MAE values are higher than those obtained with MAYA (considering a large set of τ). The GLM can be considered the most challenging baseline to outperform in terms of MSE/MAE, since it is explicitly designed to fit the bee trajectory rather than to learn a policy. Adding contextual covariates has only a minor influence, which is expected because the GLM primarily captures direct statistical dependencies rather than adaptive decision-making. Among all variants, MAYA with the Wasserstein distance (MAYA-Wass) consistently achieves the best performance across datasets, highlighting its robustness in capturing trajectory similarity. We report in Fig 3a and Fig3b MAYA's fitting for two bees.

<figure><img src="image_6.png" /><figcaption>Figure 3: For bee 1 (fast learner, low regret) and bee 15 (slow learner, high regret) from dataset 2. **Left:** Cumulative regret of 4 MABs, GLM and MAYA ($\tau = 7$) for two bees from Dataset 2. **Right:** Choice interpretability with MAYA-Wass ($\tau = 7$).</figcaption></figure>

## 4.3 UNDERSTANDING THE LEARNING PROCESS

MAYA provides post-hoc behavioral explainability at both the individual-bee and dataset levels. Concretely, our explainability metric is the alignment rate: the proportion of time MAYA's chosen action $a_t$ matches the action prescribed by a reference MAB policy (e.g., LinUCB, UCB, Epsilon-greedy, or Uniform). Empirically, when focusing on low-regret bees, MAYA's decisions align predominantly with LinUCB-like choices (see Fig3c). In contrast, for high-regret bees, decisions align more often with Epsilon greedy (see Fig3d) as a context-agnostic heuristic tends to over-exploit early, locking onto arms that were temporarily lucky and thus yielding higher cumulative regret (exploited 80% of the time). Table 2b reports the MAB-policy alignment proportions aggregated across all bees from the five datasets. Across MAYA variants, alignment profiles are broadly similar, though MAYA-DTW exhibits greater variability. The strong performance of MAYA-Wass appears consistent with its higher alignment to LinUCB-style trajectories, suggesting better identification of context-sensitive decision patterns. Additional per-$\tau$ and per-dataset analyses are provided in App 7. There we also show that using a smaller temporal window ($\tau \le 5$) can increase divergence in explainability (i.e., alignment instability), especially for slow learners, whose behavior requires longer horizons to disambiguate. These empirical proportions are useful for biologists as priors for forward ecological simulations: one can sample actions from the observed mixture over reference policies to synthesize realistic behavioral decisions.
## 4.4 SIMULATE AND FORECAST THE BEE TRAJECTORY

We study MAYA's ability to forecast realistic trajectories that capture bee behavior. We consider two behavioral types (slow and fast learners) defined by their cumulative regret according to the number of trials. We jointly cluster real bee trajectories and MAYA-generated trajectories separately, and we assess alignment by checking whether real and simulated samples fall into the same clusters via a confusion matrix. We define a clustering function $κ$ that assigns each trajectory $R(.)$ to a cluster: $κ: R(.) \mapsto \{1, ..., K\}$, where K is the number of clusters. As an additional performance evaluation, we compute the proportion of cases where the model and bee trajectories fall into the same cluster: ClusterAcc = $\frac{1}{J} \sum_{i=1}^{J} 1[\kappa(R(\pi_{bee}^{j}, 1, T) = R(\pi_{θ}^{j}, 1, T))$, where J denotes the total number of bees across all datasets (here, 80), $\pi_{bee}^{j}$ is the policy of the jth bee, and $\pi_{θ}^{j}$ is the policy generated by MAYA for the jth bee. We use $K = 2$ clusters to mirror the two archetypes. Clustering I (Euclidean) uses Euclidean distance on time series and requires equal lengths. We pool all bees across datasets. Lengths differ, so we truncate each series to the minimum common length (22). We apply a second clustering with Dynamic Barycenter Averaging (DBA); a DTW-based clustering method. It handles unequal lengths and local time shifts, so no truncation is needed. We report DBA clustering of real bee trajectories in Fig4, and in Fig5 MAYA-Wass trajectories clustering. We report the confusion matrix (real vs. simulated labels) in Tab 2a. According to Fig 6 and Fig 7 the MAYA-Wass error fluctuations remain bounded with Gaussian amplitude. It shows that MAYA-Wass's dynamics are stable, almost 0-centered with a maximal standard deviation error equal to 3. We provide additional figures of Euclidean Clustering in App 10.

<table><caption>Table 2: **Left:** For all bees in the five datasets, we report average ClusterAcc (%) under two prototype aggregation regimes: (i) Euclidean averaging with a maximum sequence length of 22, and (ii) DBA with a maximum sequence length of 40. Across both regimes, MAYA-Wass achieves the highest accuracy (79% and 91%), followed by MAYA-KL and MAYA-DTW. Standard errors are $\le 1\%$ for all entries and are omitted for readability. **Right:** Proportion of $a_t$ according to all trials for all dataset (5). We fix $\tau = 7$ for all MAYA variants.</caption><thead><tr><th></th><th>MAYA-KL</th><th>MAYA-Wass</th><th>MAYA-DTW</th></tr></thead><tbody><tr><td>ClusterAcc (Euclidean, Max L = 22)</td><td>77%</td><td>79%</td><td>70%</td></tr><tr><td>ClusterAcc (DBA, Max L = 40)</td><td>84%</td><td>91%</td><td>80%</td></tr></tbody></table><table><thead><tr><th></th><th>Epsilon-Greedy</th><th>Lin-UCB</th><th>UCB</th><th>Uniform</th></tr></thead><tbody><tr><td>MAYA-KL</td><td>34.4%±2</td><td>10.5%±1</td><td>22.6%±1</td><td>32.5%±2</td></tr><tr><td>MAYA-W</td><td>31.1%±1.5</td><td>16.2%±0.8</td><td>22.2%±0.9</td><td>30.5%±1.4</td></tr><tr><td>MAYA-DTW</td><td>36.5%±2.5</td><td>10.8%±1</td><td>17.5%±1</td><td>35.2%±2.3</td></tr></tbody></table>

(a) ClusterAcc (%)

(b) MAYA explainability for all bees choices

<figure><img src="image_7.png" /><figcaption>Figure 4: Bees trajectories</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 5: MAYA-Wass</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 6: Cluster 0</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 7: Cluster 1</figcaption></figure>

Figure 8: **Left:** Centroids of two clustering of 80 bees trajectories (in Fig4) and 80 MAYA-Wass simulated trajectories (with $\tau = 7$ in Fig.5). Clustering is done with DBA method (Clustering II). The two centroids estimated from MAYA-generated trajectories closely match those from real trajectories, indicating that MAYA preserves the underlying behavioral structure. **Right:** average difference between MAYA predictions and real trajectories ($R(\pi_{\theta}, 1, t) - R(\pi_{bee}, 1, t)$) (z-axis) for Clustering 0 and 1. Red range corresponds to $\pm\sigma$ (standard deviation). Average cost evolves according trial $t$ (time step) and average trajectory values (cumulative regret, in $y$ axis).
# 5 DISCUSSION

We introduced MAYA, a sequential imitation-learning model that forecasts individual bee trajectories across heterogeneous cognitive strategies. Across datasets and weather conditions, a memory window of ($\tau = 7$) consistently yielded the best fit; this choice is corroborated by complementary experiments and corresponds roughly to 15–30 minutes in our protocol (≈ seven trials, depending on the bee). Among variants, MAYA-Wass achieved the strongest overall performance, while MAYA-KL and MAYA-DTW remained competitive. Beyond accuracy, MAYA provides interpretable, per-trial explanations of choice, enables the generation of “artificial bees,” and supports forward simulation for ecological what-if scenarios. These results position MAYA as a viable alternative to IRL baselines and traditional statistical models. Future work will deploy MAYA in large-scale ecological simulations to assess its predictive value for ecological management decisions.

# REFERENCES

* Zoe C. Ashwood, Nicholas A. Roy, Ji Hyun Bak, The International Brain Laboratory, and Jonathan W. Pillow. Inferring learning rules from animal decision-making. In *Proceedings of the 34th International Conference on Neural Information Processing Systems*, NIPS '20, Red Hook, NY, USA, 2020a. Curran Associates Inc. ISBN 9781713829546.

* Zoe C. Ashwood, Nicholas A. Roy, Ji Hyun Bak, Jonathan W. Pillow, and International Brain Laboratory. Inferring learning rules from animal decision-making. *Advances in Neural Information Processing Systems*, 33:3442–3453, 2020b. ISSN 1049-5258. URL https://pubmed.ncbi.nlm.nih.gov/36177341/.

* Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. *Machine Learning*, 47(2-3):235–256, 2002. doi: 10.1023/A:1013689704352.

* Philippe Besse, Brendan Guillouet, Jean-Michel Loubes, and Royer François. Review and perspective for distance based trajectory clustering, 2015. URL https://arxiv.org/abs/1508.04904.

* Jay M. Biernaskie, Steven C. Walker, Robert J. Gegear, Associate Editor: Marc Mangel, and Editor: Michael C. Whitlock. Bumblebees learn to forage like bayesians. *The American Naturalist*, 174 (3):413–423, 2009. ISSN 00030147, 15375323. URL http://www.jstor.org/stable/10.1086/603629.

* Jessica F. Cantlon and Elizabeth M. Brannon. Shared system for ordering small and large numbers in monkeys and humans. *Psychological Science*, 17(5):401–406, 2006. doi: 10.1111/j.1467-9280.2006.01719.x. PMID: 16683927.

* Nuno Capela, Xiaodong Duan, Elżbieta M. Ziółkowska, and Christopher John Topping. Modelling foraging strategies of honey bees as agents in a dynamic landscape representation. *Food and Ecological Systems Modelling Journal*, 5:e99103, 2024. doi: 10.3897/fmj.5.99103. URL https://doi.org/10.3897/fmj.5.99103.

* Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. URL https://arxiv.org/abs/1706.03741.

* Antoine Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret. Robots that can adapt like animals. *Nature*, 521(7553):503–507, May 2015. ISSN 1476-4687. doi: 10.1038/nature14422. URL http://dx.doi.org/10.1038/nature14422.

* Marie Dacke and Mandyam V Srinivasan. Evidence for counting in insects. *Animal cognition*, 11: 683–689, 2008. Publisher: Springer.

* Stanislas Dehaene. *The Number Sense: How the Mind Creates Mathematics*. Oxford University Press, Oxford, UK, revised and updated edition edition, 2011.
Vincent Dumoulin, Daniel D. Johnson, Pablo Samuel Castro, Hugo Larochelle, and Yann Dauphin. A density estimation perspective on learning from pairwise human preferences, 2024. URL https://arxiv.org/abs/2311.14115.

Marco Fiandri, Alberto Maria Metelli, and Francesco Trovò. Sliding-window thompson sampling for non-stationary settings. CoRR, abs/2409.05181, 2024. doi: 10.48550/ARXIV.2409.05181. URL https://doi.org/10.48550/arXiv.2409.05181.

Dylan J. Foster, Adam Block, and Dipendra Misra. Is behavior cloning all you need? understanding horizon in imitation learning, 2024. URL https://arxiv.org/abs/2407.15007.

Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforcement learning, 2018. URL https://arxiv.org/abs/1710.11248.

Maxence Gérard, Anahit Amiri, Bérénice Cariou, and Emily Baird. Short-term exposure to heatwave-like temperatures affects learning and memory in bumblebees. Global Change Biology, 28(14):4251–4259, 2022.

Martin Giurfa, Claire Marcout, Peter Hilpert, Catherine Thevenot, and Rosa Rugani. An insect brain organizes numbers on a left-to-right mental number line. Proceedings of the National Academy of Sciences, 119(44):e2203584119, 2022.

Murray Glanzer and Anita R Cunitz. Two storage mechanisms in free recall. Journal of verbal learning and verbal behavior, 5(4):351–360, 1966.

Adam Gleave, Mohammad Taufeeque, Juan Rocamonde, Erik Jenner, Steven H. Wang, Sam Toyer, Maximilian Ernestus, Nora Belrose, Scott Emmons, and Stuart Russell. Imitation: Clean imitation learning implementations. arXiv:2211.11972v1 [cs.LG], 2022. URL https://arxiv.org/abs/2211.11972.

Hans J Gross, Mario Pahl, Aung Si, Hong Zhu, Jürgen Tautz, and Shaowu Zhang. Number-based visual generalisation in the honeybee. PloS one, 4(1):e4263, 2009. Publisher: Public Library of Science San Francisco, USA.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning, 2016. URL https://arxiv.org/abs/1606.03476.

Scarlett R Howard, Aurore Avarguès-Weber, Jair E Garcia, Andrew D Greentree, and Adrian G Dyer. Numerical ordering of zero in honey bees. Science, 360(6393):1124–1126, 2018. Publisher: American Association for the Advancement of Science.

Scarlett R Howard, Aurore Avarguès-Weber, Jair E Garcia, Andrew D Greentree, and Adrian G Dyer. Symbolic representation of numerosity by honeybees (Apis mellifera): matching characters to small quantities. Proceedings of the Royal Society B, 286(1904):20190238, 2019. Publisher: The Royal Society.

Véronique Izard, Coralie Sann, Elizabeth S Spelke, and Arlette Streri. Newborn infants perceive abstract numbers. Proceedings of the National Academy of Sciences, 106(25):10382–10385, 2009.

Jackelyn M Kembro, Mathieu Lihoreau, Joan Garriga, Ernesto P Raposo, and Frederic Bartumeus. Bumblebees learn foraging routes through exploitation-exploration cycles. Journal of the Royal Society Interface, 16(156):20190103, 2019.

Ann-Katrin Kraeuter, Paul C Guest, and Zoltán Sarnyai. The y-maze for assessment of spatial working and reference memory in mice. In Pre-clinical models: Techniques and protocols, pp. 105–111. Springer, 2018.

Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web (WWW), pp. 661–670. ACM, 2010. doi: 10.1145/1772690.1772758.

Stephan Lochner, Daniel Honerkamp, Abhinav Valada, and Andrew D. Straw. Reinforcement learning as a robotics-inspired framework for insect navigation: From spatial representations to neural implementation, 2024. URL https://arxiv.org/abs/2406.01501.
Peter McCullagh and John Ashworth Nelder. *Generalized Linear Models*. Chapman and Hall/CRC, 2nd edition, 1989. ISBN 9780412317606.

Randolf Menzel, Uwe Greggers, Alan Smith, Sandra Berger, Robert Brandt, Sascha Brunke, Gesine Bundrock, Sandra Hülse, Tobias Plümpe, Frank Schaupp, et al. Honey bees navigate according to a map-like spatial memory. *Proceedings of the National Academy of Sciences*, 102(8):3040–3045, 2005.

John Ashworth Nelder and Robert WM Wedderburn. Generalized linear models. *Journal of the Royal Statistical Society: Series A (General)*, 135(3):370–384, 1972. doi: 10.2307/2344614.

Andreas Nieder. The neuronal code for number. *Nature Reviews Neuroscience*, 17(6):366–382, 2016.

Andreas Nieder. The adaptive value of numerical competence. *Trends in Ecology & Evolution*, 35 (7):605–617, 2020.

Siddharth Reddy, Anca D. Dragan, and Sergey Levine. {SQIL}: Imitation learning via reinforcement learning with sparse rewards. In *International Conference on Learning Representations*, 2020. URL https://openreview.net/forum?id=S1xKd24twB.

Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. No-regret reductions for imitation learning and structured prediction. *CoRR*, abs/1011.0686, 2010. URL http://arxiv.org/abs/1011.0686.

Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2011. URL https://arxiv.org/abs/1011.0686.

Richard S. Sutton and Andrew G. Barto. *Reinforcement Learning: An Introduction*. MIT Press, Cambridge, MA, 1998.

Richard S. Sutton and Andrew G. Barto. *Reinforcement Learning: An Introduction*. The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the-book-2nd.html.

Catarina Vila Pouca, Connor Gervais, Joshua Reed, Jade Michard, and Culum Brown. Quantity discrimination in port jackson sharks incubated under elevated temperatures. *Behavioral Ecology and Sociobiology*, 73:1–9, 2019.

P. C. Wason. On the failure to eliminate hypotheses in a conceptual task. *Quarterly Journal of Experimental Psychology*, 12(3):129–140, 1960. doi: 10.1080/17470216008416717.

Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In *2020 IEEE Symposium Series on Computational Intelligence (SSCI)*, pp. 737–744, 2020. doi: 10.1109/SSCI47803.2020.9308468.
# 6 APPENDIX

## 6.1 DATASET DESCRIPTION

In this dataset, bees are confronted to a numerical discrimination task. Bees first enter the maze in an entrance chamber before flying in a hole and facing two images located at the end of each arm. The image has different number of dots : for example in dataset 1 and 2, one of the image has two dots while the other have four dots. If the bee chooses the correct image (i.e. the side with the highest number of dots), it will be rewarded by a sugar reward (50% sugar/water) placed in pipette in the middle of the image, alternatively if it chooses the incorrect image then it will be punished finding a bitter tasting solution (quinine solution) within the pipette. Bees cannot detect (neither visually nor by odor) which solution is located where. Then, they are only able to know image on each side before to choose. Between each trials the bee will go back to the hive to deliver the collected sugar, before reaching back the maze for another trial (typically lasting a few minutes). During this time the experimenter randomly changes the images or not, and varying the position of the dots. The localization of the correct image alternate between the right and left arm according to a pseudo-random sequence. Each dataset include 16 bees.

<table><caption>Table 3: Datasets summary</caption><thead><tr><td>Dataset</td><td>nb indiv</td><td>T</td><td>Location</td><td>Weather</td></tr></thead><tbody><tr><td>Dataset 1</td><td>16</td><td>40</td><td>France</td><td>Cold</td></tr><tr><td>Dataset 2</td><td>16</td><td>22</td><td>France</td><td>Hot</td></tr><tr><td>Dataset 3</td><td>16</td><td>40</td><td>France</td><td>Moderate</td></tr><tr><td>Dataset 4</td><td>16</td><td>40</td><td>Australia</td><td>Cold</td></tr><tr><td>Dataset 5</td><td>16</td><td>30</td><td>Australia</td><td>Hot</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Figure 9: Proportion of cumulative regret for the five datasets, per bees</figcaption></figure>
## 6.2 MSE AND MAE OF MAYA ACCORDING $\tau$

<table><caption>Table 4: Dataset 1 (Cold weather, France)</caption><thead><tr><th>side, window</th><th>MAYA KL, mean MSE</th><th>MAYA KL, mean MAE</th><th>MAYA Wass, mean MSE</th><th>MAYA Wass, mean MAE</th><th>MAYA DTW, mean MSE</th><th>MAYA DTW, mean MAE</th></tr></thead><tbody><tr><td>3.0</td><td>4.2 ± 4</td><td>1.5 ± 0.8</td><td>3.0 ± 2</td><td>1.3 ± 0.6</td><td>8.0 ± 13</td><td>1.2 ± 0.6</td></tr><tr><td>4.0</td><td>4.0 ± 4</td><td>1.4 ± 0.9</td><td>3.8 ± 3</td><td>1.5 ± 0.6</td><td>6.8 ± 7</td><td>1.9 ± 1</td></tr><tr><td>5.0</td><td>4.1 ± 2</td><td>1.6 ± 0.6</td><td>2.8 ± 2</td><td>1.2 ± 0.5</td><td>7.5 ± 7</td><td>2.0 ± 1</td></tr><tr><td>6.0</td><td>4.2 ± 3</td><td>1.5 ± 0.7</td><td>2.8 ± 1</td><td>1.2 ± 0.5</td><td>6.7 ± 7</td><td>1.9 ± 1</td></tr><tr><td>7.0</td><td>5.5 ± 5</td><td>1.7 ± 0.9</td><td>3.7 ± 3</td><td>1.4 ± 0.7</td><td>7.2 ± 7.8</td><td>2.0 ± 1</td></tr><tr><td>8.0</td><td>3.8 ± 3</td><td>1.4 ± 0.7</td><td>2.9 ± 2</td><td>1.2 ± 0.6</td><td>8.8 ± 9</td><td>2.2 ± 1</td></tr><tr><td>9.0</td><td>5.5 ± 5</td><td>1.7 ± 0.9</td><td>4.1 ± 4</td><td>1.5 ± 0.8</td><td>8.7 ± 10</td><td>2.0 ± 1</td></tr><tr><td>10.0</td><td>5.4 ± 5</td><td>1.6 ± 0.8</td><td>4.8 ± 5</td><td>1.5 ± 0.9</td><td>8.7 ± 10</td><td>2.1 ± 1</td></tr><tr><td>20.0</td><td>4.3 ± 3</td><td>1.5 ± 0.6</td><td>4.4 ± 3</td><td>1.5 ± 0.7</td><td>8.4 ± 10</td><td>2.0 ± 1</td></tr><tr><td>30.0</td><td>5.1 ± 5</td><td>1.6 ± 1</td><td>4.8 ± 6</td><td>1.5 ± 0.9</td><td>9.7 ± 11</td><td>2.2 ± 1</td></tr></tbody></table>

<table><caption>Table 6: Dataset 3 (Moderate weather, France)</caption><thead><tr><th>side, window</th><th>MAYA KL, mean MSE</th><th>MAYA KL, mean MAE</th><th>MAYA Wass, mean MSE</th><th>MAYA Wass, mean MAE</th><th>MAYA DTW, mean MSE</th><th>MAYA DTW, mean MAE</th></tr></thead><tbody><tr><td>3.0</td><td>3.8 ± 2</td><td>1.3 ± 0.6</td><td>4.0 ± 4</td><td>1.4 ± 0.8</td><td>8.4 ± 12</td><td>2.0 ± 1</td></tr><tr><td>4.0</td><td>4.4 ± 4</td><td>1.5 ± 0.9</td><td>3.9 ± 4</td><td>1.4 ± 0.8</td><td>7.4 ± 11</td><td>1.9 ± 1</td></tr><tr><td>5.0</td><td>4.3 ± 4</td><td>1.5 ± 0.7</td><td>3.0 ± 3</td><td>1.2 ± 0.7</td><td>7.3 ± 11</td><td>3.9 ± 1</td></tr><tr><td>6.0</td><td>4.4 ± 4</td><td>1.5 ± 0.8</td><td>3.1 ± 2</td><td>1.3 ± 0.6</td><td>7.7 ± 9</td><td>2.0 ± 1</td></tr><tr><td>7.0</td><td>3.7 ± 3</td><td>1.4 ± 0.6</td><td>2.6 ± 1</td><td>1.2 ± 0.5</td><td>5.7 ± 5</td><td>1.8 ± 0.8</td></tr><tr><td>8.0</td><td>4.1 ± 3</td><td>1.5 ± 0.7</td><td>2.5 ± 1</td><td>1.1 ± 0.4</td><td>8.3 ± 9</td><td>2.1 ± 1</td></tr><tr><td>9.0</td><td>5.8 ± 5</td><td>1.8 ± 0.8</td><td>4.2 ± 2</td><td>1.6 ± 0.6</td><td>8.1 ± 8</td><td>2.1 ± 1</td></tr><tr><td>10.0</td><td>6.4 ± 3</td><td>1.4 ± 0.7</td><td>4.9 ± 5</td><td>1.6 ± 1</td><td>7.1 ± 9</td><td>1.9 ± 1</td></tr><tr><td>20.0</td><td>5.3 ± 4</td><td>1.7 ± 0.8</td><td>5.2 ± 5</td><td>1.7 ± 0.7</td><td>6.5 ± 8</td><td>1.9 ± 1</td></tr><tr><td>30.0</td><td>3.8 ± 2</td><td>1.4 ± 0.5</td><td>4.4 ± 3</td><td>1.6 ± 0.7</td><td>8.7 ± 9</td><td>2.2 ± 1</td></tr><tr><td>T = 40</td><td>4.2 ± 4</td><td>1.5 ± 0.8</td><td>3.45 ± 3</td><td>1.3 ± 0.6</td><td>9.3 ± 11</td><td>2.2 ± 1</td></tr></tbody></table>

<table><caption>Table 8: Dataset 5 (Hot weather, Australia)</caption><thead><tr><th>side, window</th><th>MAYA KL, mean MSE</th><th>MAYA KL, mean MAE</th><th>MAYA Wass, mean MSE</th><th>MAYA Wass, mean MAE</th><th>MAYA DTW, mean MSE</th><th>MAYA DTW, mean MAE</th></tr></thead><tbody><tr><td>3</td><td>6.6 ± 9</td><td>1.6 ± 1</td><td>6.3 ± 9</td><td>1.6 ± 1</td><td>8.6 ± 10</td><td>1.9 ± 1</td></tr><tr><td>4</td><td>8.1 ± 8</td><td>2.0 ± 1</td><td>10.4 ± 12</td><td>2.2 ± 1</td><td>9.4 ± 8</td><td>2.1 ± 1</td></tr><tr><td>5</td><td>4.3 ± 5</td><td>1.4 ± 0.9</td><td>8.4 ± 10</td><td>2.0 ± 1</td><td>10.4 ± 12</td><td>2.2 ± 1</td></tr><tr><td>6</td><td>3.6 ± 3</td><td>1.4 ± 0.7</td><td>3.9 ± 8</td><td>1.2 ± 1</td><td>12.0 ± 11</td><td>2.3 ± 1</td></tr><tr><td>7</td><td>3.4 ± 3</td><td>1.2 ± 0.9</td><td>4.5 ± 5</td><td>1.5 ± 1</td><td>10.3 ± 11</td><td>2.1 ± 1</td></tr><tr><td>8</td><td>4.1 ± 3</td><td>1.5 ± 0.6</td><td>4.4 ± 5</td><td>1.5 ± 0.9</td><td>10.3 ± 12</td><td>2.2 ± 1</td></tr><tr><td>9</td><td>5.5 ± 8</td><td>1.6 ± 1</td><td>5.7 ± 6</td><td>1.7 ± 1</td><td>12.9 ± 16</td><td>2.4 ± 1</td></tr><tr><td>10</td><td>3.3 ± 3</td><td>1.3 ± 0.6</td><td>3.3 ± 3</td><td>1.3 ± 0.7</td><td>9.6 ± 10</td><td>2.1 ± 1</td></tr><tr><td>20</td><td>6.4 ± 6</td><td>1.8 ± 1</td><td>4.7 ± 5</td><td>1.5 ± 0.8</td><td>11.8 ± 13</td><td>2.3 ± 1</td></tr><tr><td>T = 30</td><td>6.1 ± 5</td><td>1.8 ± 0.9</td><td>6.1 ± 5</td><td>1.8 ± 0.9</td><td>9.2 ± 10</td><td>2.1 ± 1</td></tr></tbody></table>

Table 9: MSE and MAE of MAYA as a function of the window size $\tau$. The *T* row denotes the no-window setting ($\tau = T$), where at each trial the full trajectory up to time *t* is used.

<table><caption>Table 5: Dataset 2 (Hot weather, France)</caption><thead><tr><th>side, window</th><th>MAYA KL, mean MSE</th><th>MAYA KL, mean MAE</th><th>MAYA Wass, mean MSE</th><th>MAYA Wass, mean MAE</th><th>MAYA DTW, mean MSE</th><th>MAYA DTW, mean MAE</th></tr></thead><tbody><tr><td>3.0</td><td>1.2 ± 1</td><td>0.7 ± 0.4</td><td>1.5 ± 1</td><td>0.8 ± 0.4</td><td>2.4 ± 1</td><td>1.1 ± 0.4</td></tr><tr><td>4.0</td><td>1.3 ± 0.8</td><td>0.8 ± 0.3</td><td>1.5 ± 1</td><td>0.8 ± 0.4</td><td>2.7 ± 2</td><td>1.1 ± 0.6</td></tr><tr><td>5.0</td><td>2.1 ± 1</td><td>1.0 ± 0.4</td><td>1.9 ± 2</td><td>1.0 ± 0.5</td><td>2.4 ± 2.7</td><td>1.0 ± 0.6</td></tr><tr><td>6.0</td><td>2.1 ± 1</td><td>1.0 ± 0.5</td><td>1.5 ± 1</td><td>0.8 ± 0.4</td><td>3.5 ± 3</td><td>1.3 ± 0.7</td></tr><tr><td>7.0</td><td>1.6 ± 1</td><td>0.9 ± 0.4</td><td>1.5 ± 1</td><td>0.8 ± 0.4</td><td>3.0 ± 3</td><td>1.2 ± 0.6</td></tr><tr><td>8.0</td><td>1.9 ± 1</td><td>1.0 ± 0.3</td><td>1.8 ± 1</td><td>0.9 ± 0.4</td><td>2.8 ± 2</td><td>1.2 ± 0.6</td></tr><tr><td>9.0</td><td>1.8 ± 1</td><td>0.9 ± 0.4</td><td>2.2 ± 2</td><td>1.0 ± 0.6</td><td>2.5 ± 2</td><td>1.1 ± 0.6</td></tr><tr><td>10.0</td><td>2.3 ± 2</td><td>1.0 ± 0.5</td><td>2.1 ± 2</td><td>1.0 ± 0.6</td><td>2.7 ± 1</td><td>1.2 ± 0.6</td></tr><tr><td>20.0</td><td>2.3 ± 1</td><td>1.0 ± 0.4</td><td>2.6 ± 1</td><td>1.1 ± 0.3</td><td>2.0 ± 1</td><td>1.0 ± 0.4</td></tr><tr><td>T = 22</td><td>3.2 ± 3</td><td>1.2 ± 0.6</td><td>2.8 ± 1</td><td>1.2 ± 0.4</td><td>2.1 ± 1</td><td>1.0 ± 0.5</td></tr></tbody></table>

<table><caption>Table 7: Dataset 4 (Cold weather, Australia)</caption><thead><tr><th>side, window</th><th>MAYA KL, mean MSE</th><th>MAYA KL, mean MAE</th><th>MAYA Wass, mean MSE</th><th>MAYA Wass, mean MAE</th><th>MAYA DTW, mean MSE</th><th>MAYA DTW, mean MAE</th></tr></thead><tbody><tr><td>3.0</td><td>4.5 ± 4</td><td>1.6 ± 0.9</td><td>3.0 ± 4</td><td>1.2 ± 0.6</td><td>7.1 ± 10</td><td>1.8 ± 13</td></tr><tr><td>4.0</td><td>3.8 ± 3</td><td>1.5 ± 0.7</td><td>3.6 ± 3</td><td>1.5 ± 0.6</td><td>7.1 ± 9</td><td>1.9 ± 1</td></tr><tr><td>5.0</td><td>4.9 ± 3</td><td>1.7 ± 0.7</td><td>2.6 ± 3</td><td>1.2 ± 0.7</td><td>7.6 ± 11</td><td>1.9 ± 1</td></tr><tr><td>6.0</td><td>4.1 ± 3</td><td>1.5 ± 0.7</td><td>2.6 ± 1</td><td>1.2 ± 0.4</td><td>7.8 ± 9</td><td>2.0 ± 1</td></tr><tr><td>7.0</td><td>3.7 ± 3</td><td>1.4 ± 0.6</td><td>3.6 ± 2</td><td>1.5 ± 0.5</td><td>8.3 ± 10</td><td>2.1 ± 1</td></tr><tr><td>8.0</td><td>6.2 ± 8</td><td>1.7 ± 1</td><td>3.4 ± 2</td><td>3.4 ± 2</td><td>6.2 ± 7</td><td>1.8 ± 1</td></tr><tr><td>9.0</td><td>4.6 ± 3</td><td>1.6 ± 0.7</td><td>3.1 ± 2</td><td>1.3 ± 0.5</td><td>8.1 ± 7</td><td>2.1 ± 1</td></tr><tr><td>10.0</td><td>7.7 ± 7</td><td>2.0 ± 1</td><td>4.8 ± 4</td><td>1.6 ± 0.8</td><td>8.4 ± 10</td><td>2.0 ± 1</td></tr><tr><td>20.0</td><td>5.4 ± 4</td><td>1.7 ± 0.8</td><td>4.4 ± 2</td><td>1.6 ± 0.5</td><td>8.5 ± 11</td><td>2.1 ± 1.2</td></tr><tr><td>30.0</td><td>5.5 ± 4</td><td>1.7 ± 0.7</td><td>6.7 ± 7</td><td>1.9 ± 0.9</td><td>9.0 ± 12</td><td>2.1 ± 1</td></tr><tr><td>T = 40</td><td>4.2 ± 5</td><td>1.4 ± 0.8</td><td>3.3 ± 2</td><td>1.3 ± 0.6</td><td>9.0 ± 10</td><td>2.2 ± 1</td></tr></tbody></table>
# 7 UNDERSTANDING THE LEARNING PROCESS

## 7.1 MAYA EXPLAINABILITY WITH τ = 7

<figure><img src="image_12.png" /><figcaption>Figure 10: MAYA-KL</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 11: MAYA-Wass</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Figure 12: MAYA-DTW</figcaption></figure>

Figure 13: For bee 1 (fast learner, low regret) from dataset 2 we report choice interpretability for MAYA-variants ($\tau = 7$).

<figure><img src="image_15.png" /><figcaption>Figure 14: MAYA-KL</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 15: MAYA-Wass</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>Figure 16: MAYA-DTW</figcaption></figure>

Figure 17: For bee 15 (slow learner, high regret) from dataset 2 we report choice interpretability for MAYA-variants ($\tau = 7$).
## 7.2 MAYA EXPLAINABILITY WITH τ = 3

<figure><img src="image_18.png" /><figcaption>Figure 18: MAYA-KL</figcaption></figure>

<figure><img src="image_19.png" /><figcaption>Figure 19: MAYA-Wass</figcaption></figure>

<figure><img src="image_20.png" /><figcaption>Figure 20: MAYA-DTW</figcaption></figure>

Figure 21: For bee 1 (fast learner, low regret) from dataset 2 we report choice interpretability for MAYA-variants ($\tau = 3$).

<figure><img src="image_21.png" /><figcaption>Figure 22: MAYA-KL</figcaption></figure>

<figure><img src="image_22.png" /><figcaption>Figure 23: MAYA-Wass</figcaption></figure>

<figure><img src="image_23.png" /><figcaption>Figure 24: MAYA-DTW</figcaption></figure>

Figure 25: For bee 15 (slow learner, high regret) from Dataset 2 we report choice interpretability for MAYA-variants ($\tau = 3$).

<figure><img src="image_24.png" /><figcaption>Figure 26: Bee 1</figcaption></figure>

<figure><img src="image_25.png" /><figcaption>Figure 27: Bee 15</figcaption></figure>

Figure 28: Regret modelization for bee 1 (lower cumulative regret) and bee 15 (higher cumulative regret) of Dataset 2, with $\tau = 3$.
8 COMPARATIVE METHODS DESCRIPTION

* Generative Adversarial Imitation Learning (GAIL) GAIL learns a policy by simultaneously training it with a discriminator that aims to distinguish expert trajectories against trajectories from the learned policy. Ho & Ermon (2016)

* Behavioral Cloning (BC) Behavioral cloning directly learns a policy by using supervised learning on observation-action pairs from expert demonstrations. It is a simple approach to learning a policy, but the policy often generalizes poorly and does not recover well from errors. Foster et al. (2024).

* AIRL, similar to GAIL, adversarially trains a policy against a discriminator that aims to distinguish the expert demonstrations from the learned policy. Unlike GAIL, AIRL recovers a reward function that is more generalizable to changes in environment dynamics. Fu et al. (2018).

* DAgger (Dataset Aggregation) iteratively trains a policy using supervised learning on a dataset of observation-action pairs from expert demonstrations (like behavioral cloning), runs the policy to gather observations, queries the expert for good actions on those observations, and adds the newly labeled observations to the dataset. DAgger improves on behavioral cloning by training on a dataset that better resembles the observations the trained policy is likely to encounter, but it requires querying the expert online Ross et al. (2011).

* Density-based reward modeling is an inverse reinforcement learning (IRL) technique that assigns higher rewards to states or state-action pairs that occur more frequently in an expert's demonstrations. The key intuition behind this method is to incentivize the agent to take actions that resemble the expert's actions in similar states Dumoulin et al. (2024).

* Maximum Causal Entropy Inverse Reinforcement Learning (MCE IRL) : The principle of maximum causal entropy is a method that extends the classical maximum entropy idea to sequential settings. Instead of considering probabilities in isolation, it uses causally conditioned probabilities, which means that the model explicitly accounts for the fact that information is revealed step by step over time. This allows us to properly capture how side information becomes available and how it influences decisions at each stage Biernaskie et al. (2009).

* Preference Comparisons : The preference comparison algorithm learns a reward function from preferences between pairs of trajectories. The comparisons are modeled as being generated from a Bradley-Terry (or Boltzmann rational) model, where the probability of preferring trajectory A over B is proportional to the exponential of the difference between the return of trajectory A minus B. In other words, the difference in returns forms a logit for a binary classification problem, and accordingly the reward function is trained using a cross-entropy loss to predict the preference comparison. Christiano et al. (2023).

* Soft Q Imitation Learning (SQIL) : Soft Q Imitation learning learns to imitate a policy from demonstrations by using the DQN algorithm with modified rewards. During each policy update, half of the batch is sampled from the demonstrations and half is sampled from the environment. Expert demonstrations are assigned a reward of 1, and the environment is assigned a reward of 0. This encourages the policy to imitate the demonstrations, and to simultaneously avoid states not seen in the demonstrations Reddy et al. (2020).

* GLM : A Generalized Linear Model (GLM) is a statistical framework that extends linear regression to response variables with non-Gaussian distributions. In our setting, the regret trajectory $R(\pi, 1, T)$ is modeled as a function of time, $R(\pi, 1, T) \sim f(t)$, where $f$ is linked to a linear predictor through a canonical link function. A Poisson GLM is employed when the noise structure is count-like, while a Gamma GLM is used to capture multiplicative noise. This allows us to statistically frame the evolution of regret as a stochastic process, while accounting for heterogeneous variability across agents. Nelder & Wedderburn (1972).

* Contextual GLM : The contextual variant incorporates side information (e.g., environmental or experimental conditions) into the predictor, enabling the model to capture how context modulates regret dynamics. Then $R(\pi, 1, T) \sim f(t, x_t)$ McCullagh & Nelder (1989).
## 8.1 MAE COMPARISON OF METHODS

<table><caption>Table 10: MAE comparison of methods across the five datasets. Values are reported as mean ± standard deviation. We fix τ = 7 for all MAYA variant</caption><thead><tr><th>Dataset</th><th>GAIL</th><th>BC</th><th>AIRL</th><th>Dagger</th><th>DBR</th><th>MCE</th><th>Pref-Comp</th><th>SQIL</th><th>GLM (no ctx)</th><th>GLM (ctx)</th><th>MAYA-KL</th><th>MAYA-Wass</th><th>MAYA-DTW</th></tr></thead><tbody><tr><td>1</td><td>3.75±2.5</td><td><strong>1.61 ± 0.79</strong></td><td><strong>0 ± 0</strong></td><td>2.9 ± 2.8</td><td>4.3 ± 3.8</td><td>10.38 ± 1.60</td><td>8.35±3.25</td><td>3.71±1</td><td><strong>1.4 ± 0.3</strong></td><td><strong>1.4 ± 0.3</strong></td><td>1.5 ± 0.7</td><td><strong>1.2 ± 0.5</strong></td><td>1.9 ± 1</td></tr><tr><td>2</td><td>3.69±1.8</td><td><strong>1.24 ± 0.72</strong></td><td><strong>0 ± 0</strong></td><td>1.93 ± 1.7</td><td>2.72 ± 1.89</td><td>6.04 ± 1.0</td><td>3.7 ± 1.9</td><td>2.18±0.9</td><td><strong>0.8 ± 0.5</strong></td><td><strong>0.8 ± 0.5</strong></td><td>1.4 ± 0.6</td><td>1.5 ± 0.5</td><td>2.1 ± 1</td></tr><tr><td>3</td><td>3.62±2.4</td><td><strong>1.79 ± 0.98</strong></td><td><strong>0 ± 0</strong></td><td>2.6 ± 3.1</td><td>3.4 ± 4.1</td><td>8.13 ± 1.10</td><td>9.76 ± 1.75</td><td>3.2±1</td><td><strong>1.4 ± 0.4</strong></td><td><strong>1.4 ± 0.4</strong></td><td>3.7 ± 3</td><td><strong>2.6 ± 1</strong></td><td>1.8 ± 0.8</td></tr><tr><td>4</td><td>3.1±2.8</td><td><strong>1.65 ± 0.86</strong></td><td><strong>0 ± 0</strong></td><td>3.0 ± 2.7</td><td>4.60 ± 4.8</td><td>10 ± 1.6</td><td>9.7 ± 1.7</td><td>3.2±1</td><td><strong>2.1 ± 1</strong></td><td><strong>2.1 ± 1</strong></td><td>1.4 ± 0.6</td><td>1.5 ± 0.5</td><td>2.1 ± 1</td></tr><tr><td>5</td><td>4.9 ± 2.8</td><td><strong>3.23 ± 3</strong></td><td><strong>0 ± 0</strong></td><td>6.5 ± 5.1</td><td>5.5 ± 7.8</td><td>15.0 ± 7.6</td><td>14.3 ± 6.92</td><td>4.52±2</td><td>8.0 ± 8</td><td><strong>2.2 ± 1</strong></td><td><strong>1.2 ± 0.9</strong></td><td>1.3 ± 0.7</td><td>2.1 ± 1</td></tr></tbody></table>

## 9 FINETUNING IMITATION LEARNING

We present ablations over the fine-tuning budget of the IRL methods. As the tuning knobs differ across methods, we use the unified notation *b* for the method-specific budget (see Tab 11). The best results are summarized in the main text.

<table><caption>Table 11: Hyperparameters of each comparative methods.</caption><thead><tr><th>b<sup>(GAIL)</sup></th><th>b<sup>(BC)</sup></th><th>b<sup>(Dagger)</sup></th><th>b<sup>(DBR)</sup></th><th>b<sup>(MCE)</sup></th><th>b<sup>(PrefComp)</sup></th><th>b<sup>(PrefComp)</sup></th></tr></thead><tbody><tr><td>epochs</td><td>epochs</td><td>env. steps</td><td>epochs</td><td>epochs</td><td># envs</td><td>eval episodes</td></tr></tbody></table>

<table><caption>Table 12: Dataset 1 (Cold weather, France)</caption><thead><tr><th></th><th>MSE (b=1)</th><th>MAE (b=1)</th><th>MSE (b=10)</th><th>MAE (b=10)</th><th>MSE (b=50)</th><th>MAE (b=50)</th></tr></thead><tbody><tr><th>GAIL</th><td>29.6 +/- 41</td><td>3.75+/-2.5</td><td>29.6 +/- 41</td><td>3.75+/-2.5</td><td>29.6 +/- 41</td><td>3.75+/-2.5</td></tr><tr><th>BC</th><td>23.2 +/- 30.8</td><td>3.26 +/- 2.74</td><td>19.8 +/- 26.5</td><td>3.1+/-2.3</td><td>5.16+/-3.94</td><td>1.61+/-0.79</td></tr><tr><th>AIRL</th><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td></tr><tr><th>Dagger</th><td>22.8+/- 32.9</td><td>2.9+/-2.8</td><td>36.9+/-52.0</td><td>3.7 +/- 3.8</td><td>32.5 +/- 50.6</td><td>3.7+/- 3.3</td></tr><tr><th>Density based reward</th><td>43.1 +/- 54.81</td><td>4.3+/-3.8</td><td>43.1 +/- 54.8</td><td>4.3+/-3.8</td><td>43.1 +/- 54.8</td><td>4.3+/-3.8</td></tr><tr><th>MCE</th><td>148.83 +/- 38.47</td><td>10.38 +/- 1.60</td><td>148.83 +/- 38.47</td><td>10.38 +/- 1.60</td><td>148.83 +/- 38.47</td><td>10.38 +/- 1.60</td></tr><tr><th>Pref-Comp</th><td>120.25 +/- 52.1</td><td>9.17 +/- 2.99</td><td>114 +/- 53</td><td>8.9 +/- 2.9</td><td>104.5 +/- 57</td><td>8.35 +/- 3.25</td></tr><tr><th>SQIL</th><td>26.2 +/-19</td><td>3.75 +/- 1</td><td>26.2 +/-19</td><td>3.75 +/- 1</td><td>26.2 +/-19</td><td>3.75 +/- 1</td></tr></tbody></table>

<table><caption>Table 13: Dataset 2 (Hot weather, France)</caption><thead><tr><th></th><th>MSE (b=1)</th><th>MAE (b=1)</th><th>MSE (b=10)</th><th>MAE (b=10)</th><th>MSE (b=50)</th><th>MAE (b=50)</th></tr></thead><tbody><tr><th>GAIL</th><td>23.2 +/- 17</td><td>3.69 +/- 1.8</td><td>23.2 +/- 17</td><td>3.69 +/- 1.8</td><td>23.2 +/- 17</td><td>3.69 +/- 1.8</td></tr><tr><th>BC</th><td>12.1+/-12.1</td><td>2.54+/-1.74</td><td>7.3+/-7.7</td><td>1.99+/-1.3</td><td>2.86 +/- 2.95</td><td>1.24 +/- 0.72</td></tr><tr><th>AIRL</th><td>0</td><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td></tr><tr><th>Dagger</th><td>15.63 +/- 19.2</td><td>2.54+/-2.2</td><td>11.8 +/- 16.5</td><td>2.1+/-2.0</td><td>9.67+/- 12.6</td><td>1.93 +/-1.7</td></tr><tr><th>Density based reward</th><td>15.26 +/- 16.43</td><td>2.72 +/- 1.89</td><td>15.26 +/- 16.43</td><td>2.72 +/- 1.89</td><td>15.26 +/- 16.43</td><td>2.72 +/- 1.89</td></tr><tr><th>MCE</th><td>49.5 +/- 14.2</td><td>6.04 +/- 1.0</td><td>49.5 +/- 14.2</td><td>6.04 +/- 1.0</td><td>49.5 +/- 14.2</td><td>6.04 +/- 1.0</td></tr><tr><th>Pref-Comp</th><td>24.54+/-18.3</td><td>3.7 +/-1.9</td><td>30.15 +/-17.3</td><td>4.49 +/- 1.53</td><td>28.84 +/- 16.13</td><td>4.46 +/- 1.30</td></tr><tr><th>SQIL</th><td>9.80 +/-6</td><td>2.18+/-0.9</td><td>9.80 +/-6</td><td>2.18+/-0.9</td><td>9.80 +/-6</td><td>2.18+/-0.9</td></tr></tbody></table>

<table><caption>Table 14: Dataset 3 (Moderate weather, France)</caption><thead><tr><th></th><th>MSE (b=1)</th><th>MAE (b=1)</th><th>MSE (b=10)</th><th>MAE (b=10)</th><th>MSE (b=50)</th><th>MAE (b=50)</th></tr></thead><tbody><tr><th>GAIL</th><td>27.5 +/- 40</td><td>3.62 +/-2.5</td><td>27.5 +/- 40</td><td>3.62 +/-2.5</td><td>27.5 +/- 40</td><td>3.62 +/-2.5</td></tr><tr><th>BC</th><td>15.9+/-24</td><td>2.67 +/- 2.26</td><td>22.0+/-25</td><td>3.55+/-2.1</td><td>5.5+/-4.1</td><td>1.79+/-0.98</td></tr><tr><th>AIRL</th><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td><td>0 +/- 0</td></tr><tr><th>Dagger</th><td>35.4+/- 61.8</td><td>3.3 +/-3.7</td><td>34.5+/-48.2</td><td>3.5 +/- 3.4</td><td>21.6+/-46.0</td><td>2.6 +/-3.1</td></tr><tr><th>Density based reward</th><td>41.38 +/- 51.1</td><td>3.4+/-4.1</td><td>41.38 +/- 51.1</td><td>3.4+/-4.1</td><td>41.38 +/- 51.1</td><td>3.4+/-4.1</td></tr><tr><th>MCE</th><td>140.3 +/-34.7</td><td>8.13 +/-1.10</td><td>140.3 +/-34.7</td><td>8.13 +/-1.10</td><td>140.3 +/-34.7</td><td>8.13 +/-1.10</td></tr><tr><th>Pref-Comp</th><td>130.98 +/-44.7</td><td>9.98 +/-1.98</td><td>134.12+/-3.7</td><td>10.12 +/-1.39</td><td>125.70 +/- 44.1</td><td>9.76 +/- 1.75</td></tr><tr><th>SQIL</th><td>22.65+/-15</td><td>3.2+/-1</td><td>22.65+/-15</td><td>3.2+/-1</td><td>22.65+/-15</td><td>3.2+/-1</td></tr></tbody></table>
<table><caption>Table 15: Dataset 4 (Cold weather, Australia)</caption><thead><tr><th></th><th>MSE (b=1)</th><th>MAE (b=1)</th><th>MSE (b=10)</th><th>MAE (b=10)</th><th>MSE (b=50)</th><th>MAE (b=50)</th></tr></thead><tbody><tr><td>GAIL</td><td>25.3 +/-39</td><td>3.1 +/- 2.8</td><td>25.3 +/-39</td><td>3.1 +/- 2.8</td><td>25.3 +/-39</td><td>3.1 +/- 2.8</td></tr><tr><td>BC</td><td>23.2 +/- 28.6</td><td>3.4 +/-2.4</td><td>22.3 +/- 26.1</td><td>3.5 +/-2.2</td><td>5.35 +/-4.17</td><td>1.65 +/-0.86</td></tr><tr><td>AIRL</td><td>0+/-0</td><td>0+/-0</td><td>0+/-0</td><td>0+/-0</td><td>0+/-0</td><td>0+/-0</td></tr><tr><td>Dagger</td><td>22.9 +/-34.0</td><td>3.0 +/- 2.7</td><td>45.3 +/- 52.8</td><td>4.6 +/- 3.6</td><td>24.4 +/- 24.2</td><td>3.2 +/- 2.7</td></tr><tr><td>Density based reward</td><td>46.06 +/-55</td><td>4.60 +/-4.8</td><td>46.06 +/-55</td><td>4.60 +/-4.8</td><td>46.06 +/-55</td><td>4.60 +/-4.8</td></tr><tr><td>MCE</td><td>148.2 +/- 39.6</td><td>10.3 +/-1.6</td><td>148.2 +/- 39.6</td><td>10.3 +/-1.6</td><td>148.2 +/- 39.6</td><td>10.3 +/-1.6</td></tr><tr><td>Pref-Comp</td><td>124.1 +/-52</td><td>9.4 +/- 2.78</td><td>128.29 +/- 42.7</td><td>9.86 +/- 1.68</td><td>125.68 +/- 44.19</td><td>9.7 +/- 1.7</td></tr><tr><td>SQL</td><td>25.3 +/-20</td><td>3.2 +/- 1</td><td>25.3 +/-20</td><td>3.2 +/- 1</td><td>25.3 +/-20</td><td>3.2 +/- 1</td></tr></tbody></table>

<table><caption>Table 16: Dataset 5 (Hot weather, Australia)</caption><thead><tr><th></th><th>MSE (b=1)</th><th>MAE (b=1)</th><th>MSE (b=10)</th><th>MAE (b=10)</th><th>MSE (b=50)</th><th>MAE (b=50)</th></tr></thead><tbody><tr><td>GAIL</td><td>45.71 +/- 45.7</td><td>4.9 +/- 2.8</td><td>45.71 +/- 45.7</td><td>4.9 +/- 2.8</td><td>45.71 +/- 45.7</td><td>4.9 +/- 2.8</td></tr><tr><td>BC</td><td>124.4 +/- 186.46</td><td>6.94 +/- 7.05</td><td>39.7 +/- 70</td><td>3.91 +/- 3</td><td>26.7 +/- 42.7</td><td>3.23 +/- 3.17</td></tr><tr><td>AIRL</td><td>0+/-0</td><td>0+/-0</td><td>0+/-0</td><td>0+/-0</td><td>0+/-0</td><td>0+/-0</td></tr><tr><td>Dagger</td><td>113.4 +/-247.5</td><td>6.0 +/-7.1</td><td>93.2 +/-115.9</td><td>6.5 +/- 5.1</td><td>25.8 +/- 47.1</td><td>6.5 +/- 5.1</td></tr><tr><td>Density based reward</td><td>115.7 +/- 242.51</td><td>5.5 +/-7.8</td><td>115.7 +/- 242.51</td><td>5.5 +/-7.8</td><td>115.7 +/- 242.51</td><td>5.5 +/-7.8</td></tr><tr><td>MCE</td><td>374 +/-311.9</td><td>15.0 +/-7.6</td><td>374 +/-311.9</td><td>15.0 +/-7.6</td><td>374 +/-311.9</td><td>15.0 +/-7.6</td></tr><tr><td>Pref-Comp</td><td>284 +/-254</td><td>12.9 +/- 7</td><td>335.6 +/-271</td><td>14.5 +/-</td><td>332.8 +/- 272.29</td><td>14.3 +/-6.92</td></tr><tr><td>SQL</td><td>25 +/- 16</td><td>4.52 +/-2</td><td>25 +/- 16</td><td>4.52 +/-2</td><td>25 +/- 16</td><td>4.52 +/-2</td></tr></tbody></table>
10 CLUSTERING : OTHER VARIANTS

<figure><img src="image_26.png" /><figcaption>Figure 29: Bees trajectories</figcaption></figure>

<figure><img src="image_27.png" /><figcaption>Figure 30: MAYA-Wass</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>Figure 31: MAYA-KL</figcaption></figure>

<figure><img src="image_29.png" /><figcaption>Figure 32: MAYA-DTW</figcaption></figure>

Figure 33: Centroides of two clustering of 80 bees trajectories (in Fig29) and 80 MAYA-variant (Fig30, Fig31 and Fig32) simulated trajectories (with τ = 7). Clustering are done with Euclidean method (Clustering I).

<figure><img src="image_30.png" /><figcaption>Figure 34: Bees trajectories</figcaption></figure>

<figure><img src="image_31.png" /><figcaption>Figure 35: MAYA-Wass</figcaption></figure>

<figure><img src="image_32.png" /><figcaption>Figure 36: MAYA-KL</figcaption></figure>

<figure><img src="image_33.png" /><figcaption>Figure 37: MAYA-DTW</figcaption></figure>

Figure 38: Centroides of two clustering of 80 bees trajectories (in Fig34) and 80 MAYA-variant (Fig35, Fig36 and Fig37) simulated trajectories (with τ = 7). Clustering are done with DBA method (Clustering II).

<figure><img src="image_34.png" /><figcaption>Figure 39: Cluster 0 (I)</figcaption></figure>

<figure><img src="image_35.png" /><figcaption>Figure 40: Cluster 1 (I)</figcaption></figure>

<figure><img src="image_36.png" /><figcaption>Figure 41: Cluster 0 (II)</figcaption></figure>

<figure><img src="image_37.png" /><figcaption>Figure 42: Cluster 1 (II)</figcaption></figure>

Figure 43: Average difference between MAYA-Wass (τ = 7) predictions and real trajectories (R(π MAYA, 1, t) - R(π bee, 1, t)) (z-axis) for Euclidean (I) and DBA (II) Clustering according 0 and 1 Cluster. Red range correspond to ±σ (standard deviation).
<figure><img src="image_38.png" /><figcaption>Figure 44: Cluster 0 (I)</figcaption></figure>

<figure><img src="image_39.png" /><figcaption>Figure 45: Cluster 1 (I)</figcaption></figure>

<figure><img src="image_40.png" /><figcaption>Figure 46: Cluster 0 (II)</figcaption></figure>

<figure><img src="image_41.png" /><figcaption>Figure 47: Cluster 1 (II)</figcaption></figure>

Figure 48: Average difference between MAYA-KL ($\tau = 7$) predictions and real trajectories ($R(\pi_{\text{MAYA}}, 1, t) - R(\pi_{\text{bee}}, 1, t)$) (z-axis) for Euclidean (I) and DBA (II) Clustering according to 0 and 1 Cluster. Red range correspond to $\pm\sigma$ (standard deviation).

<figure><img src="image_42.png" /><figcaption>Figure 49: Cluster 0 (I)</figcaption></figure>

<figure><img src="image_43.png" /><figcaption>Figure 50: Cluster 1 (I)</figcaption></figure>

<figure><img src="image_44.png" /><figcaption>Figure 51: Cluster 0 (II)</figcaption></figure>

<figure><img src="image_45.png" /><figcaption>Figure 52: Cluster 1 (II)</figcaption></figure>

Figure 53: Average difference between MAYA-DTW ($\tau = 7$) predictions and real trajectories ($R(\pi_{\text{MAYA}}, 1, t) - R(\pi_{\text{bee}}, 1, t)$) (z-axis) for Euclidean (I) and DBA (II) Clustering according to 0 and 1 Cluster. Red range correspond to $\pm\sigma$ (standard deviation).
11 MAYA ALGORITHM

<caption>**Algorithm 1 MAYA : Multi Agent Y-maze Allocation**</caption>

**Require:** Logged bee regret trajectory $R(\pi_{\text{bee}}, 1, T)$
**Require:** Set $\mathcal{P}$ of $N$ bandit policies $\{\pi_1, \dots, \pi_N\}$
**Require:** Window size $\tau$ such that $t \ge \tau$
**Require:** A similarity metric $\delta$
1: $\xi = ()^T_{t=1}$
2: Init $\pi_0$
3: **for** $t \in \{2, \dots, \tau - 1\}$ **do**
4: &nbsp;&nbsp;&nbsp;&nbsp;Observe $R(\pi_{\text{bee}}, 1, t - 1)$
5: &nbsp;&nbsp;&nbsp;&nbsp;Observe a context information $x_t$
6: &nbsp;&nbsp;&nbsp;&nbsp;**for** $i = 1$ to $N$ **do**
7: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Simulate policy agent $\pi_i(s_{t-1}|x_t)$
8: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute cumulative regret $R(\pi_i, 1, t - 1)$
9: &nbsp;&nbsp;&nbsp;&nbsp;**end for**
10: $\xi_t = \operatorname*{argmin}_{\pi \in \mathcal{P}} \delta(\pi_{\text{bee}}, \pi, t)$
11: $\pi_\theta(a_t|s_{t-1}) \leftarrow \pi_\xi(a_t|s_{t-1})$
12: Select $A_t \sim \pi_\theta(a_t|s_{t-1})$
13: Receive reward $r_t$
14: Update $\pi_i \quad \forall \pi_i \in \mathcal{P}$
15: $\xi[t] \leftarrow \xi_t$
16: **end for**
17: **for** $t \in \{\tau, \dots, T\}$ **do**
18: &nbsp;&nbsp;&nbsp;&nbsp;Observe $R(\pi_{\text{bee}}, \tau, 1, t - 1)$
19: &nbsp;&nbsp;&nbsp;&nbsp;Observe a context information $x_t$
20: &nbsp;&nbsp;&nbsp;&nbsp;**for** $i = 1$ to $N$ **do**
21: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Simulate policy agent $\pi_i(s_{t-1}|x_t)$
22: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute cumulative regret $R(\pi_i, \tau, 1, t - 1)$
23: &nbsp;&nbsp;&nbsp;&nbsp;**end for**
24: $\xi_t = \operatorname*{argmin}_{\pi \in \mathcal{P}} \delta(\pi_{\text{bee}}, \pi, \tau, t)$
25: $\pi_\theta(a_t|s_{t-1}) \leftarrow \pi_\xi(a_t|s_{t-1})$
26: Select $A_t \sim \pi_\theta(a_t|s_{t-1})$
27: Receive reward $r_t$
28: Update $\pi_i \quad \forall \pi_i \in \mathcal{P}$
29: $\xi[t] \leftarrow \xi_t$
30: **end for**
31: **return** $\pi_\theta$
12 MICE DATASET EXPERIMENT

**Dataset and setup.** We use the dataset of Ashwood et al. (2020a), which reports trial-by-trial changes in mice policy and decomposes those updates into a learning component and a noise component (see Fig. 54a). Unlike their original analysis, which simulates an average trajectory across individuals, our method (MAYA) simulates one trajectory per individual. The dataset contains 19 rats with between 1500 and 6000 trials each. To control the computational cost of DTW and to align with our bee experiments, we reduce the number of individual at 100.

**Selecting the memory horizon $\tau$.** According with Tab 17, Fig 54b shows MAE and MSE as a function of the memory window $\tau$. MAYA-KL clearly identifies an optimal range around $\tau \in [6, 7]$, whereas MAYA-Wass suggests $\tau \in [8, 10]$ when balancing MAE and MSE. For consistency with previous experiments, we set $\tau = 7$ in all subsequent analyses.

**Explanations and performance.** With $\tau = 7$, Fig. 63 and Fig. 59 provides MAYA explanations for the rats with the lowest and highest cumulative regret (see Fig. 55). For slow learners, all MAYA variants behave similarly (Fig. 65); for fast learners, MAYA-KL achieves the best fit, capturing rapid policy changes better than MAYA-Wass (Fig. 64). A plausible explanation is that, under KL similarity, MAYA acts more often from LinUCB-like behavior than with Wasserstein similarity (see Tab18b). As in previous datasets, MAYA-DTW tends to act more like Epsilon-Greedy, likely due to DTW's alignment properties. Overall, all MAYA variants outperform GLM baselines (Table 18a).

<table><caption>Table 17: MSE and MAE of MAYA as a function of the window size $\tau$ for Mice Dataset.</caption><thead><tr><th rowspan="2">side_windows</th><th colspan="2">MSE MAYA-KL</th><th colspan="2">MAE MAYA-KL</th><th colspan="2">MSE MAYA-Wass</th><th colspan="2">MAE MAYA-Wass</th><th colspan="2">MSE MAYA-DTW</th><th colspan="2">MAE MAYA-DTW</th></tr><tr><th>mean</th><th>std</th><th>mean</th><th>std</th><th>mean</th><th>std</th><th>mean</th><th>std</th><th>mean</th><th>std</th><th>mean</th><th>std</th></tr></thead><tbody><tr><td>3</td><td>5760</td><td>3894</td><td>59</td><td>24</td><td>8083</td><td>5012</td><td>72</td><td>25</td><td>5790</td><td>5683</td><td>55</td><td>29</td></tr><tr><td>4</td><td>3868</td><td>3493</td><td>46</td><td>25</td><td>6547</td><td>3672</td><td>64</td><td>23</td><td>5815</td><td>5770</td><td>55</td><td>30</td></tr><tr><td>5</td><td>3046</td><td>3307</td><td>40</td><td>24</td><td>5724</td><td>3803</td><td>59</td><td>23</td><td>5819</td><td>5788</td><td>55</td><td>29</td></tr><tr><td>6</td><td>2763</td><td>3090</td><td>37</td><td>23</td><td>5276</td><td>3511</td><td>57</td><td>21</td><td>5830</td><td>5758</td><td>55</td><td>29</td></tr><tr><td>7</td><td>2786</td><td>3161</td><td>38</td><td>23</td><td>4640</td><td>3382</td><td>53</td><td>22</td><td>5822</td><td>5747</td><td>55</td><td>29</td></tr><tr><td>8</td><td>2974</td><td>3197</td><td>39</td><td>23</td><td>4728</td><td>3722</td><td>53</td><td>23</td><td>5851</td><td>5777</td><td>55</td><td>29</td></tr><tr><td>9</td><td>3114</td><td>3424</td><td>40</td><td>24</td><td>4231</td><td>3403</td><td>50</td><td>22</td><td>5819</td><td>5740</td><td>55</td><td>29</td></tr><tr><td>10</td><td>3223</td><td>3378</td><td>41</td><td>25</td><td>4197</td><td>3576</td><td>49</td><td>24</td><td>5810</td><td>5701</td><td>54</td><td>29</td></tr><tr><td>20</td><td>4710</td><td>6689</td><td>47</td><td>33</td><td>3491</td><td>3515</td><td>43</td><td>25</td><td>5771</td><td>5725</td><td>54</td><td>29</td></tr><tr><td>30</td><td>5618</td><td>8543</td><td>50</td><td>38</td><td>3453</td><td>3896</td><td>41</td><td>27</td><td>5760</td><td>5724</td><td>54</td><td>29</td></tr></tbody></table>

<figure><img src="image_46.png" /></figure>

(a) According Ashwood et al. (2020a), on each trial, a sinusoidal grating (with contrast values between 0 and 100%) appears on either the left or right side of a screen. Mice must report the side of the grating by turning a wheel (left or right) in order to receive a water reward.

(b) Comparative study of the best window size $\tau$ by average MSE and MAE. * symbol refers as best performance according standard deviation and average reward (see Tab.17 for the full results). MSE is displayed as $\times 10^2$.

<caption>Figure 54: Left: experimental description of the Mice Dataset. Right: Comparative study of the best window size $\tau$ for Mice Dataset.</caption>
<figure><img src="image_47.png" /><figcaption>Figure 55: Proportion of cumulative regret for the Mice dataset, per mice</figcaption></figure>

<table><thead><tr><th rowspan="2"></th><th colspan="2">MSE</th><th colspan="2">MAE</th></tr><tr><th>Mean</th><th>Std</th><th>Mean</th><th>Std</th></tr></thead><tbody><tr><td>MAYA KL</td><td>2786</td><td>3161</td><td>38</td><td>23</td></tr><tr><td>MAYA-Wass</td><td>4640</td><td>3382</td><td>53</td><td>22</td></tr><tr><td>MAYA-DTW</td><td>5822</td><td>5777</td><td>55</td><td>29</td></tr><tr><td>GLM</td><td>6427</td><td>4137</td><td>63</td><td>21</td></tr><tr><td>GLM Contextual</td><td>6416</td><td>4133</td><td>63</td><td>21</td></tr></tbody></table>

(a)

<table><thead><tr><th></th><th>Epsilon-Greedy</th><th>Lin-UCB</th><th>UCB</th><th>Uniform</th></tr></thead><tbody><tr><td>MAYA-KL</td><td>30%±2.5</td><td>2%±1.1</td><td>29%±1.3</td><td>36%±2.2</td></tr><tr><td>MAYA-W</td><td>27%±1.8</td><td>10%±1</td><td>28%±1</td><td>33%±1.5</td></tr><tr><td>MAYA-DTW</td><td>28%±3</td><td>0.5%±1</td><td>56%±4</td><td>15%±3</td></tr></tbody></table>

(b)

<table><caption>Table 18: **Left**: MSE and MAE comparison of MAYA (with $\tau = 7$) and GLM variants. **Right**: MAYA explainability for all MAYA choices ($\tau = 7$)</caption><thead><tr><th></th><th>MAYA-KL</th><th>MAYA-Wass</th><th>MAYA-DTW</th></tr></thead><tbody><tr><td>ClusterAcc (Euclidean, Max L = 1400)</td><td>90%</td><td>85%</td><td>75%</td></tr><tr><td>ClusterAcc (DBA, Max L = 6000)</td><td>80%</td><td>75%</td><td>65%</td></tr></tbody></table>

<figure><img src="image_48.png" /><figcaption>Figure 56: MAYA-KL</figcaption></figure>

<figure><img src="image_49.png" /><figcaption>Figure 57: MAYA-Wass</figcaption></figure>

<figure><img src="image_50.png" /><figcaption>Figure 58: MAYA-DTW</figcaption></figure>

Figure 59: MAYA explainability for mouse 20 (fast learner, low regret) from Mice dataset. We report choice interpretability for MAYA-variants ($\tau = 7$).

<figure><img src="image_51.png" /><figcaption>Figure 60: MAYA-KL</figcaption></figure>

<figure><img src="image_52.png" /><figcaption>Figure 61: MAYA-Wass</figcaption></figure>

<figure><img src="image_53.png" /><figcaption>Figure 62: MAYA-DTW</figcaption></figure>

Figure 63: MAYA explainability for mouse 2 (slow learner, high regret) from Mice dataset. We report choice interpretability for MAYA-variants ($\tau = 7$).
<figure><img src="image_54.png" /><figcaption>Figure 64: Mouse 20</figcaption></figure>

<figure><img src="image_55.png" /><figcaption>Figure 65: Mouse 2</figcaption></figure>

Figure 66: Regret modelization for mouse 20 (best) and mice 2 (worst) from Mice 2, with $\tau = 7$

<figure><img src="image_56.png" /><figcaption>Figure 67: Mouse' trajectories</figcaption></figure>

<figure><img src="image_57.png" /><figcaption>Figure 68: MAYA-KL trajectories</figcaption></figure>

Figure 69: Centroides of Clustering (I) of 100 mice' (Left) and MAYA-KL ($\tau = 7$) (Right) trajectories.

<figure><img src="image_58.png" /><figcaption>Figure 70: Cluster 0</figcaption></figure>

<figure><img src="image_59.png" /><figcaption>Figure 71: Cluster 1</figcaption></figure>

Figure 72: Average difference between MAYA-KL ($\tau = 7$) predictions and real trajectories ($R(\pi_{MAYA}, 1, t) - R(\pi_{mice}, 1, t)$) (z-axis) for Euclidean (I) Clustering according to 0 and 1 Cluster. Red range correspond to $\pm\sigma$ (standard deviation).
13 COMPLEMENTARY INFORMATION ABOUT THE BIOLOGY INTEREST

We share with other vertebrates a basic ability for abstract number representation, the *number sense* Dehaene (2011). As early as two days postnatally Izard et al. (2009), this ability enables us to evaluate numbers as concepts: three books are perceived as similar to three cups, even though they differ completely in their visual features (i.e., sensory information). To evaluate quantity, both numerical and sensory information can be used. For example, when visually comparing two quantities, the larger set will often contain more items (i.e., numerosity), but may also exhibit greater density, a larger total surface area, or a wider convex hull encompassing all elements. Neuronal encoding of sensory information occurs early in the primary cortex, whereas numbers are computed in higher integrative areas by what Nieder et al. identified as *number neurons* Nieder (2016).

Quantity discrimination is necessary in contexts as diverse as evaluating food patches, regulating social attraction, or competing for resources Nieder (2020). From sharks to mammals, all major vertebrate clades appear capable of discriminating between different quantities, either spontaneously or in learning tasks Vila Pouca et al. (2019). By carefully designing protocols that control for sensory cues, researchers have demonstrated that several non-human species are capable of performing quantity discrimination based on the abstract evaluation of numbers Cantlon & Brannon (2006). Among them is an insect: the honeybee (Apis mellifera). Beyond discriminating numerosities of up to eight items, these insects, with brains of fewer than one million neurons, can also manipulate numbers, performing simple addition, subtraction, and symbolic tasks Dacke & Srinivasan (2008); Gross et al. (2009); Howard et al. (2018; 2019); Giurfa et al. (2022).

Later experiments required a Y-maze: a three-armed apparatus shaped like the letter Y, commonly used to study memory, learning, and decision-making in rodents Kraeuter et al. (2018) (see Fig. 73). These mazes required bees to inhibit their spatial memory Menzel et al. (2005) (e.g., recalling that the last reward was in the left arm) and to focus instead on the visual stimuli displayed at the end of each arm. The balance between exploring new options and exploiting previously rewarded ones is key to their foraging behavior and likely plays a crucial role in their learning performance within these devices Kembro et al. (2019); Lochner et al. (2024).

<figure><img src="image_60.png" /><figcaption>Figure 73: Y-maze for bees experiments</figcaption></figure>
14 MATHEMATICAL PROOF OF MAYA ACCORDING $\tau$

**Stationary case (1) : upper bound of MAYA error** Consider the case of two policies $\pi_1$ that achieves the highest regret i.e. $R(\pi_1, 1, T) = T$ and $\pi_0$ that achieves a zero regret i.e. $R(\pi_0, 1, T)$. In this case

$$ \Delta_{\pi_1,t} - \Delta_{\pi_0,t} \le 1 \quad \forall t $$

as the reward is in $\{0, 1\}$. The maximal bound of $R(\pi_{MAYA}, 1, T) - R(\pi_{bee}, 1, T)$ corresponds to the case where $R(\pi_{bee}, 1, T)$ is always centered between $R(\pi_1, 1, T)$ and $R(\pi_0, 1, T)$ (see Fig74a). Let's define $\epsilon_t^*$ the agent who acts the closest of the bee at $t$ and $\epsilon_t$ the agent chosen by MAYA at $t$. Then

$$ \mathbb{P}[\epsilon_t = \epsilon_t^*] = 0.5 \quad \forall t $$

as no best agent are better from the other one. This case corresponds to an equality between the two possible agent (with extreme regret values) and leads to the worst scenario of a stationary case when the similarity distance $d(t)$ are when defined. Then the maximal cumulative gap between MAYA-regret and Bee-regret in stationary case are :

$$ \begin{aligned} \sum_{t=1}^{T} |\Delta_{MAYA,t} - \Delta_{Bee,t}| &\le \frac{1}{2} \sum_{t=1}^{T} |\Delta_{\pi_1,t} - \Delta_{Bee,t}| + \frac{1}{2} \sum_{t=1}^{T} |\Delta_{\pi_0,t} - \Delta_{Bee,t}| \\ &\le \sum_{t=1}^{T} \frac{t}{2} \\ &\le \frac{\frac{T}{2}(\frac{T}{2} + 1)}{2} \\ &\le \frac{1}{8}(T(T + 2)) \end{aligned} \quad (1) $$

**Stationary case (2) : upper bound of the worst policy** Consider the case where $\pi_{MAYA}$ always chose like $\pi_1$ and $\pi_{bee}$ always chose like $\pi_0$ (see Fig 74b). Then the similarity distance $d(t)$ fails to provide a correct measure and MAYA chose the agent with the largest regret gap relative to the bee's regret. Then for all $t$

$$ \mathbb{P}[\epsilon_t \neq \epsilon_t^*] = 1. $$

Then the maximal cumulative gap between MAYA-regret and Bee-regret in the worst policy in stationary case are :

$$ \begin{aligned} \sum_{t=1}^{T} |\Delta_{MAYA,t} - \Delta_{Bee,t}| &\le \sum_{t=1}^{T} |\Delta_{\pi_1,t} - \Delta_{\pi_0,t}| \\ &\le \frac{T \cdot (T + 1)}{2} \end{aligned} \quad (2) $$

The alternative case where $\pi_{MAYA}$ always chooses as $\pi_0$ and $\pi_{bee}$ always chooses as $\pi_1$ is equivalent.

**Cyclic case : upper bound of MAYA error with no windows ($\tau = T$) policy** Consider that after $S$ trials the bee moves from $\pi_1$ to $\pi_0$ (alternative cases are equivalent, see Fig 75a). Consider that the distances are well defined, as in the stationary case (1). Then :

$$ \sum_{t=1}^{S} |\Delta_{MAYA,t} - \Delta_{Bee,t}| \le \frac{1}{8}(S \times (S + 2)) \quad (3) $$

The time required for MAYA to act like $\pi_0$ is $2S + 1$ but at $t = 2S + 1$, the bee changes from $\pi_0$ to $\pi_1$ and MAYA continues to act like $\pi_1$ (see Fig.75a). Recursively, MAYA always act like $\pi_1$ from $t = 1$ until $t = T$. Then

$$ \mathbb{P}[\epsilon_t = \pi_1] = 1 \quad \forall t $$

and

$$ \mathbb{P}[\epsilon_t = \epsilon_t^*] = \frac{N_*(T)}{T}, \quad \forall t $$
<figure><img src="image_61.png" /><figcaption>Figure 74: Maximal cumulative gap between MAYA-regret and Bee-regret in stationary case according the distance d(·) abilities to provide a correct measure</figcaption></figure>

Where

$$ 
\begin{aligned}
N_*(T) &= qS + \min(S, r), \\
q &= \left\lfloor \frac{T}{2S} \right\rfloor, \\
r &= T - 2Sq \in [0, 2S).
\end{aligned}
$$

A minimal bound of $N_*$ are :

$$ N_*(T) \geq \frac{T}{2} $$

Then the maximal cumulative gap between MAYA-regret and Bee-regret in a cyclic case with no windows is :

$$ 
\begin{aligned}
\sum_{t=1}^{T} |\Delta_{\text{MAYA},t} - \Delta_{\text{Bee},t}| &\leq \frac{N_*(T)}{T} \frac{1}{8}(T(T+2)) + \left(1 - \frac{N_*(T)}{T}\right) \frac{T(T+1)}{2} \\
&\leq \frac{T}{2} \frac{1}{T} \frac{1}{8}(T(T+2)) + \left(1 - \frac{T}{2}\right) \frac{T(T+1)}{2} \\
&= \frac{T(5T+6)}{16}
\end{aligned}
\quad (4) $$

**Cyclic case : upper bound of MAYA error with windows $\tau = S$** Assume that $S$ are even. Consider that after $S$ trials, the bee moves from $\pi_1$ to $\pi_0$ (alternative cases are equivalent, see Fig75b). Consider that the distance is well defined like in the stationary case (1). From time $t = 1$ until $S$, MAYA acts as the best agent :

$$ \sum_{t=1}^{S} |\Delta_{\text{MAYA},t} - \Delta_{\text{Bee},t}| \leq \frac{1}{8}(S \times (S+2)) \quad (5) $$

and

$$ \mathbb{P}[\varepsilon_t = \varepsilon_t^*] = 1 \quad \forall t \in \{1, \dots, S\}. $$

From time $S+1$ until $S + \frac{S}{2}$, MAYA acts as the worst policy (start cycle)

$$ \sum_{t=S+1}^{S+\frac{S}{2}} |\Delta_{\text{MAYA},t} - \Delta_{\text{Bee},t}| \leq \sum_{t=S+1}^{S+\frac{S}{2}} t \quad (6) $$

$$ \leq \frac{S(5S + 2)}{8} \quad (7) $$
and

$$ \mathbb{P}[\varepsilon_t \neq \varepsilon_t^*] = 1 \quad \forall t \in \{S+1, \dots, S+\frac{S}{2}\}. $$

And from $t = S + \frac{S}{2} + 1$ until $t = 2S$ MAYA acts with the best policy (end cycle):

$$ \sum_{t=S+\frac{S}{2}+1}^{2S} |\Delta_{\text{MAYA},t} - \Delta_{\text{Bee},t}| \le \sum_{t=S+\frac{S}{2}+1}^{2S} \frac{t}{2} \le \frac{S(7S+2)}{16} \quad (8) $$

and

$$ \mathbb{P}[\varepsilon_t = \varepsilon_t^*] = 1 \quad \forall t \in \{S + \frac{S}{2} + 1, \dots, 2S\}. $$

Consider a full cycle, the event $\varepsilon_t = \varepsilon_t^*$ appears $S - \frac{S}{2}$ times. Let's set

$$ q = \left\lfloor \frac{\max(0, T - S)}{S} \right\rfloor, \quad r = \max(0, T - S) - qS \in [0, S). $$

Here $q$ is the number of full cycle $S$ in $t > S$, and $r$ is the rest of a potential unfinished tail segment of the started cycle. Let $N_*(T) = \sum_{t=1}^T 1_{\varepsilon_t = \varepsilon^*}$ with $N_*(T) \le T$ equal to

$$ N_*(T) = \min(T, S) + q \cdot \frac{S}{2} + \max(0, r - \frac{S}{2}) $$

If $S$ is even and $T > S$ then

$$ N_*(T) \ge \frac{T}{2} + \frac{S}{4} \quad (9) $$

### Proof:

With $T = S + qS + r$:

$$ N_*(T) - \left(\frac{T}{2} + \frac{S}{4}\right) = \frac{S}{2} - \frac{r}{2} + \max\left(0, r - \frac{S}{2}\right) \ge 0, $$

where the minimum are archived with $r = \frac{S}{2}$.

$$ \mathbb{P}[\varepsilon_t = \varepsilon_t^*] = \frac{N_*(T)}{T} \ge \frac{1}{2} + \frac{S}{4T} \quad (10) $$

In the cases where $S$ is not not even

$$ q = \left\lfloor \frac{T-S}{S} \right\rfloor, \quad r = T - S - qS \in [0, S). $$

then

$$ N_*(T) = S + \frac{q(S+1)}{2} + \max\left(0, r - \frac{S-1}{2}\right). $$

As $T = S + qS + r$, we have

$$ N_*(T) - \frac{T}{2} = \frac{S}{2} + \frac{q}{2} + \max\left(0, r - \frac{S-1}{2}\right) - \frac{r}{2}. $$

and for any $r \in [0, S)$,

$$ \min_r \left( \max\left(0, r - \frac{S-1}{2}\right) - \frac{r}{2} \right) = -\frac{S-1}{4}. $$

Then

$$ \begin{aligned} N_*(T) \ge \frac{S}{2} + \frac{q}{2} - \frac{S-1}{4} + \frac{T}{2} &= \frac{S+1}{4} + \frac{q}{2} + \frac{T}{2} \ge \frac{S+1}{4} + \frac{T}{2}. \\ N_*(T) \ge \frac{T}{2} + \frac{S+1}{4} \ge \frac{T}{2} + \frac{S}{4}. \end{aligned} \quad (11) $$
Which are better to the S parity case.

Then the maximal cumulative gap between MAYA-regret and Bee-regret with windows $\tau = S$ is

$$ 
\begin{aligned}
\sum_{t=1}^{T} |\Delta_{\text{MAYA},t} - \Delta_{\text{Bee},t}| &\le \frac{N_*(T)}{T} \frac{T(T+2)}{8} + \left(1 - \frac{N_*(T)}{T}\right) \frac{T(T+1)}{2} \\
&\le \left(\frac{T}{2} + \frac{S}{4}\right) \cdot \frac{1}{T} \cdot \frac{T(T+2)}{8} + \left(1 - \left(\frac{T}{2} + \frac{S}{4}\right) \cdot \frac{1}{T}\right) \frac{T(T+1)}{2} \\
&\le \frac{10T^2 + 12T - 3ST - 2ST}{32}
\end{aligned}
\quad (12) $$

<figure><img src="image_62.png" /><figcaption>Figure 75: Maximal cumulative gap between MAYA regret and bee regret in a non-stationary case, measured with respect to window $\tau$. The purple arrow highlights the period during which MAYA chooses actions in accordance with the agent whose behavior is most distant from that of the bee.</figcaption></figure>

**Cyclic case: upper bound of MAYA error with windows $\tau \in \{\frac{S}{2}+1; \dots, S-1\}$**. We consider the case where $\frac{S}{2}+1 \le \tau < S$ (see Fig75c). Assume that $S$ are even. From time $t=1$ until $S$, MAYA acts as the best agent (stationary case 1):

$$ \sum_{t=1}^{S} |\Delta_{\text{MAYA},t} - \Delta_{\text{Bee},t}| \le \frac{1}{8}(S \times (S+2)) \quad (13) $$

and

$$ \mathbb{P}[\varepsilon_t = \varepsilon^*] = 1 \quad \forall t \in \{1, \dots, S\}. $$

From time $S+1$ until $S+\frac{\tau}{2}$, MAYA acts as the worst policy (start cycle)

$$ 
\begin{aligned}
\sum_{t=S+1}^{S+\frac{\tau}{2}} |\Delta_{\text{MAYA},t} - \Delta_{\text{Bee},t}| &\le \sum_{t=S+1}^{S+\frac{\tau}{2}} t \\
&\le \frac{\tau}{4}(2S+1+\frac{\tau}{2}) \\
&\le \frac{\tau^2}{8} + \frac{S\tau}{2} + \frac{\tau}{4}
\end{aligned}
\quad (14) $$

and

$$ \mathbb{P}[\varepsilon_t \neq \varepsilon^*] = 1 \quad \forall t \in \{S+1, \dots, S+\frac{\tau}{2}\}. $$

And from $t = S + \frac{\tau}{2} + 1$ until $t = 2S$, MAYA acts as the best policy (end cycle) with:

$$ 
\begin{aligned}
\sum_{t=S+\frac{\tau}{2}+1}^{2S} |\Delta_{\text{MAYA},t} - \Delta_{\text{Bee},t}| &\le \sum_{t=S+\frac{\tau}{2}+1}^{2S} \frac{t}{2} \\
&\le \frac{(3S + \frac{\tau}{2} + 1)(S - \frac{\tau}{2})}{4}
\end{aligned}
\quad (15) $$
and

$$ \mathbb{P}[\varepsilon_t = \varepsilon_t^*] = 1 \quad \forall t \in \{S + \frac{\tau}{2} + 1, \dots, 2S\}. $$

Consider a full cycle, the event $\varepsilon_t = \varepsilon_t^*$ appears $S - \frac{\tau}{2}$ times. Let's set

$$ q = \lfloor \frac{T-S}{S} \rfloor \quad r = (T-S) - qS \in [0, S). $$

Let $N_*(T) = \sum_{t=1}^{T} 1_{\varepsilon_t = \varepsilon^*}$ with $N_*(T) \le T$ equal to

$$ N_*(T) = S + q(S - \frac{\tau}{2}) + \max(0, r - \frac{\tau}{2}). $$

and

$$ \mathbb{P}[\varepsilon_t = \varepsilon_t^*] = \frac{N_*(T)}{T} \qquad (16) $$

The maximal cumulative gap between MAYA-regret and Bee-regret with windows $\tau \in \{\frac{S}{2} + 1; \dots, S-1\}$ with $S$ parity is

$$ \begin{aligned} \sum_{t=1}^{T} |\Delta_{\text{MAYA},t} - \Delta_{\text{Bee},t}| &\le \frac{N_*(T)}{T} \cdot \frac{T(T+2)}{8} + \left(1 - \frac{N_*(T)}{T}\right) \cdot \frac{T(T+1)}{2} \\ &\le \frac{S + q(S - \frac{\tau}{2}) + \max(0, r - \frac{\tau}{2})}{T} \cdot \frac{T(T+2)}{8} \\ &\quad + \left(1 - \frac{S + q(S - \frac{\tau}{2}) + \max(0, r - \frac{\tau}{2})}{T}\right) \cdot \frac{T(T+1)}{2} \end{aligned} $$

As $N_*(T) \ge T(1 - \frac{\tau}{2S})$ without any condition on $S$ parity, the maximal cumulative gap between the MAYA-regret and the Bee-regret with windows $\tau \in \{\frac{S}{2} + 1; \dots, S-1\}$ is

$$ \sum_{t=1}^{T} |\Delta_{\text{MAYA},t} - \Delta_{\text{Bee},t}| \le \frac{T(T+2)}{8} + \frac{(3T+2)T}{16} \frac{\tau}{S} \qquad (17) $$

**Cyclic case: upper bound of MAYA with windows $\tau < \frac{S}{2} + 1$** In this case, there is no way to be sure that the distance $d()$ do not fails to identify the best agent. It's equivalent to choose randomly and the worst case corresponds to the upper bound of the worst policy. Then the maximal cumulative gap between MAYA regret and Bee-regret with $\tau < \frac{S}{2} + 1$ in cyclic case are equivalent to Eq. 2.

**Cyclic case: upper bound of MAYA with windows $\tau > S$** In this case, the time required to change the policy is over a cycle $S > 1$. Then, the bee switch two times in $\tau$ and MAYA allows it to act as the same agent. Then it is equivalent to act as a cyclic case with no windows ($\tau = T$) Then the maximal cumulative gap between MAYA regret and Bee-regret with $\tau > S$ in cyclic case are equivalent to Eq. 4.
15 DISCLOSURE OF LLM USE

Large Language Models (LLMs) were used in a limited capacity during the preparation of this paper. Their use was restricted to (i) spelling and phrasing assistance (to support a dyslexic co-author), and (ii) suggesting improvements to Python scripts for graph generation and visualization. No part of the scientific content, analyses, or conclusions was produced by LLMs.

16 ACKNOWLEDGMENTS

We thank Rémi Airiau for his contribution to the initial prototype during his internship. We would like to acknowledge the CBI for their support and for providing the experimental facilities that made this work possible.