Rethinking Teacher-Student Curriculum Learning through the Cooperative Mechanics of Experience
Authors: Manfred Diaz
Liam Paull
Andrea Tacchetti
Venue: Accepted by TMLR ()

Abstract: Teacher-Student Curriculum Learning (TSCL) is a curriculum learning framework that draws inspiration from human cultural transmission and learning. It involves a teacher algorithm shaping the learning process of a learner algorithm by exposing it to controlled experiences. Despite its success, understanding the conditions under which TSCL is effective remains challenging. In this paper, we propose a data-centric perspective to analyze the underlying mechanics of the teacher-student interactions in TSCL. We leverage cooperative game theory to describe how the composition of the set of experiences presented by the teacher to the learner, as well as their order, influences the performance of the curriculum that is found by TSCL approaches. To do so, we demonstrate that for every TSCL problem, an equivalent cooperative game exists, and several key components of the TSCL framework can be reinterpreted using game-theoretic principles. Through experiments covering supervised learning, reinforcement learning, and classical games, we estimate the cooperative values of experiences and use value-proportional curriculum mechanisms to construct curricula, even in cases where TSCL struggles. The framework and experimental setup we present in this work represents a novel foundation for a deeper exploration of TSCL, shedding light on its underlying mechanisms and providing insights into its broader applicability in machine learning.

Published in Transactions on Machine Learning Research (09/2024)
Rethinking Teacher-Student Curriculum Learning through the Cooperative Mechanics of Experience
Manfred Diaz
diazcabm mila quila.
Mila, University of Montreal
Liam Paull
paulll @ mila miila.quebec
Mila, University of Montreal Canada CIFAR Al Chair
Andrea Tacchetti
attacher@ google.com
Google DeepMind
Reviewed on http://:wwww///////////////////a
Abstract
Teacher-Studen Curriculum Learning (TSCL) is a curriculum learning framework that draws inspi- ration from human cultural transmission and learning. involves a teacher algorithm shaping the learning process learner algorithm by exposing it to controlled experiences. Despite its success, understanding the conditions under which TSCL, effective remains challenging. In this paper, we propose a data-centric perspective to analyze the underlying mechanics of the teacher-student interactions in TSCL. We leverage cooperative game theory describe how the composition of the set of experiences presented by the teacher to the learner, well as their order, influences the performance of the curriculum that is found by TSCL approaches. To do so, we demonstrate that for every TSCL problem, an equivalent cooperative game exists, and several key components of the TSCL framework can be reinterpreted using game-theoretic principles. Through experiments covering supervised learning, reinforcement learning, and classical games, we estimate the coopera- tive values of experiences and use value-proportional curriculum mechanisms to construct curricula, in cases where TSCL struggles. The framework and experimental setup we present in this work represents novel foundation for deeper exploration of TSCL, shedding light its underlying mechanisms and providing insights into its broader applicability in machine learning.
Introduction
Controlling the sequence of tasks that a learning algorithm is exposed to through curriculum has been shown to potentially enhance learning efficiency (Elman, 1993; Krueger & Dayan 2009, Bengio et al. 2009). One widely used curriculum framework, known as Teacher- Staueetttt Curriculum Learning (TSCL) (Graves et al. 2017 Matisen et al. 2020), specifically gives teacher algorithm the ability to control this sequence. While it commonly understood that presenting tasks with increasing difficulty can improve learning, the underlying dynamics and structure of teacher- student interaction in this context are still relatively unexplored. Very few works have attempted to understand when, and how TSCL works (Lee al. the problem (Portelas et al. 2019a Turchetta et al. 2020 Liu et al.) 2020 Feng et al.) (021). In this paper, we propose novel data-centric perspective
2021: Wu et al. 2020) while most have focused providing algorithmic improvements to
2021 to understand and analyze TSCL algorithms.
We begin by formalizing a general notion of units of experience to describe the teacher algorithm's 's control objects (consumed by the learner). Subsequently, our approach draws inspiration from work on feature attribution (Patel al., 2021], data valuation (Ghorbani Zou, [019 Yan & Procaccia 2021) and explainability (Lundberg & Lee 2017), and leverages tools from cooperative game theory (Von Neumann & Morgenstern 1944. (Shapley 1952) analyze how the

---

Published in Transactions on Machine Learning Research (09/2024)
compositions of these units impact teacher-student interactions. We show that, for every TSCL problem, there exists equivalent cooperative game where units of experience are players and teacher-student interactions approximate sequential coalition formation process SSec. 4p. As a result, the learning progression objective (Schmidhuberi 1991 Oudeyer et al. 2007 Graves et al. 2017) and the teacher bandit policy (Gittins) 1979; Matiisen et al. 2020), two essential components of TSCL, have alternative interpretations as an approximation of player (unit) marginal contribution (Weber) 1988) and fair allocation mechanism, respectively (Sec. 4.2 & 4.3). Furthermore, because the order matters in the case of curriculum learning (Krueger & Dayan 2009 Bengio et al., 2009), traditional cooperative game-theoretic arguments produce unintuitive results (Nowak & Radzik 1994). Thus, leverage generalized cooperative games and their solution concepts (Nowak & Radzik 1994, Sanchez & Bergantiños, 1997) to overcome these limitations and formally extend these data-centric game-theoretic formulations to the curriculum learning setting.
To demonstrate the predictive power and range of problems where this nume-theoretic and data-centric interpretation of TSCL. applies, we build an experimental setting that evaluates the prospect of cooperation among units of experience in problems spanning supervised learning (SL), reinforcement learning (RL), and classical games (Sec. 5). These experiments simulate ordered and unordered coalition formation processes and approximate the cooperative games we developed to describe TSCL. For every problem,
estimate units priori value (e.g.. Shapleyy or Nowak & Radzik values) and demonstrate that these priori values, although expensive to compute (Deng & Papadimitrion 1994]. are useful proxies to find curricula. To this end, we design unordered and ordered value- Ialue-proportional curriculum mechanisms inspired by value-proportional allocations (Bachrach et al., 2020). In most settings, the unordered mechanism fails to find a reasonable curriculum, demonstrating the unsuitability of traditional game-theoretic tools for the TSCL problem. However, the ordered mechanism consistently finds an optimal or near-optimal ordering (i.e., curriculum) even when TSCL fails (Sec. 5.2p. To understand what impacts the ability of TSCL in those settings, leverage another cooperative game-theoretic tool, namely, measures of interactions (Grabisch & Roubens 1999 Procaccia et al., 2014), and in particular the Value of Player to other Player (vPoP) (Hausken & Mohr, 2001), to quantify positive, neutral, or negative pairwise interactions among units. We show that in settings with considerable unit interference, as characterized by their negative pairwise interactions, TSCL cannot produce useful curricula.
2 Preliminaries
2.1 Cooperative Game Theory
Cooperative Games. Cooperative games model problems where players interact maximize collective gain (Roth 1988). In (traditional) cooperative game in characteristic function form among set of players U, denoted by G = (U, v), the characteristic function v 2 + IR associates to each coalition C E 2 belonging to the powerset 2 a real number that represents the benefits produced by the players in C acting jointly. In a cooperative game, a solution concept represents a mechanism that produces allocation vectors ø E R!U| (Shubik 1981). Particularly, Shapley's value (Shapley 1952) allocates to each player u its average marginal contribution vi C + u) v(C) to coalitions
C U, where u U-C
ICI((IU
|C| T
1)! [u(C + u) x(C)
Σ
Cugc
and uniquely satisfies the axioms of efficiency, null-player, symmetry, and linearity, which are generally considered to be properties of a fair allocation mechanism (van den Brink van der Laan 1998).
Generalized Cooperative Games. When the order in which players join determines coalitional worth, traditional cooperative games and their solution concepts (e.g., Shapley's value) may produce unintuitive allocations (Nowak P(2 - R assigns to every ordered
& Radzik 1994). In these games, the generalized characteristic function coalition C E U2 in the powerset of permutations P(2U) its worth if members join in the permutation order. Nowak & Radzzkk 1994) and Sanchez & Bergantiños (1997) extended Shapley's 's work and proposed solution concepts for these generalized cooperative games. We focus on the former due to its intuitive formulation
ØNR (u)
1
Σ
[e(C: -Cc()
(2)
JUl!
CEP(2") Cugc
2

---

Published in Transactions on Machine Learning Research (09/2024)
that averages, for all ordered coalitions C P(2)) where the unit u E U appended last, its marginal contribution to the newly formed ordered coalition C: u.
Measures of Interactions. In a cooperative game, a measure of interaction (Grabisch & Roubens, 1999, Procaccia et al., 2014) computes players' influences on other players' outcomes. In particular, we leverage the value of a player to another player (vPoP) (Hausken & Mohr 2001). For the games above, vPoP constructs a matrix whose entries (u, u,) E measure the influence player u, exerts over player uj. measures how the Shapley value of unit changes in the absence of another. More precisely,
(IU) - [C)(C) 1)!
Σ CCU U.
[ō(u;, C) o(u;, C - u:)]
(3)
U|!
where o(u C) is the Shapley value of unit u (Eq. 1 in the cooperative game restricted to players in C. This matrix marginal o(u,) Σj (u; u,) corresponds to each player's Shapley value. We extend vPoP to games in generalized characteristic function form by applying Eqq. 3 mutatis mutandis using Nowak Radzik (1994) value to provide an
ordered pairwise interaction metric cNR (U; U;).
2.2 Bandit Algorithms
Multi-armed bandit algorithms provide solution to problems of decision- decision-making under uncertainty (Gittins 1979 Lattimore & Szepesvári, (020)) where, at each interaction, a decision must be made about which arm u E U must be pulled. We are particularly interested in action-value-base algorithms that maintain empirical value estimates qu (u)
computed
T (u)
No Σ r(u))Lo,.
(4)
and that estimate the average reward received by the algorithm in the iterations Ng S k where the u-arm has been pulled. Bandit algorithms, like the ones Graves et al. (2017) and Matiisen al. (2020) in their work, transform the estimated average contributions into arms interactions by deriving from estimated values Boltzmann policy Tыe E A(U) such that the probability of interaction is proportional to the value estimates:
Th (u) x (u))
(5)
More sophisticated approaches (e.g., the ExP3 (Auer et al. 2003) used in our experiments) account for other factors, like recency, bias, stochasticity, or non-stationarity (Lattimore Szepesvári 2020).
3 Experience to Control
The TSCL framework commonly operates under the assumption that tasks presented to a learning algorithm can influence its learning dynamics. Modern iterative learning algorithms process tasks in discrete units. For instance, SL and RL algorithms operate over instances and transitions, respectively. But also, collections of these elementary units, such as batches or episodes, datasets or environments, or more generally benchmarks or environment suites, describe a hierarchy of aggregations experience. Henceforth, we utilize the term unit of experience for referring to any collection of discrete units that teacher algorithm can use to control the dynamics of the learner algorithm.
Example 3.1. For analysis, we may define unit of experience the set of instances of class in SL classification problem. For example, in the MNIST dataset (LeCun & Cortes 2010), there may be ten units of experience, namely, classes ZERO, ONE, TWO, NINE.
The units of experience abstraction indistinctly applies supervised or reinforcement learning problems. On either paradigm, any iterative learning algorithm is controllable system whose control inputs units of experience.
3

---

Published in Transactions on Machine Learning Research (09/2024)
Algorithm Generalized Teacher-Student Curriculum Learning.
1: procedure GENTSCL
2: inputs policy: algorithm: L, units: U. metric: J
3: teacher: To E AUU), targets: U, budget: K
4: for k = 1...K do
us (u)
TT [(TR-1,, Us)
Tk [[KK U) I
8: Th+1 UPDATERULB(TR: Tk)
9: end for
10: output: тĄ
11: end procedure
Example 3.2. There are four control inputs in mini-batch gradient descent (Goodfellow et al., 2016): the mini-batee
{a1, }B), the loss function i, the parameters 0, and the learning rate v such that:
B
0x+1 02 Vo,
((0,, x;)
A styl- algorithm, as presented Alg. 1 solves a data-centric control problem. The learner algorithm C(T 1, ug) is a black-box system (line 6) controlled by teacher algorithm through units drawn with probability Uj The (u). The learner output, each iteration k, is policy or model тT whose performance is measured by metric function J that quantifies the model's 's performance cumulative learning progression reward (line 7p. For the teacher's UPDATERULEF, we focus on multi-armed bandit learning (see Sec. 2.2). We adopt a data-centric perspective to perform a systematic investigation of its components.
of evaluation units U. The teacher aims maximize the
4 The Cooperative Mechanics of Experience
The ideal teacher-student interaction mechanics assume that the learner monotonically increases its performance on the target task. We conjecture that prerequisite for this idealistic curriculum learning dynamics (Matiisen et al. 2020) to occur within TSCL-style algorithms is that experience (or data) presented to the learner should not interfere with each other. In other words, units of experience should interact cooperatively. From a game-theoretic perspective, we explain how these cooperative mechanies may emerge among units by examining the history of teacher-student interactions, the reward function, and the bandit selection policy.
4.1 The Mechanics of Coalition Formation
We establish a cooperative game where each unit of experience u E U is a player. Next, we interpret the history of
k < teacher-student interactions H, {u,,... uu through their empirical frequencies ps (u) E Au which form unit vectors that lie in the |U]-probability simplex (UU) The effective support (i.e., non-zero probabilities) determines an unordered coalition (i.e., set) C. C U (see Faigle (2022). Chapter 8), formed by the units presented to the learner up interaction < K. We study this interpretation through cooperative game in characteristic function form Sec. 2.
Example 4.1. (Example 3.1 cont'd) In the class-as-unir equivalence MNIST, unordered training coalition, e.g., the two-unit coalition C. = [ZERO, NINE), describes teacher-student interactions limited to instances from those classes.
Next, we note that the outcome of a coalition's work is the policy or model me. Thus, estimating the performance of the policy Ty through the metric function J akin to approximating the characteristic function v(Cx) (Alg | line 7). Moreover, these approximations are conditioned on an evaluation (or target) unit U. We model the target-task and multiple-tase task settings (Graves et al. 2017) where units of experience should increase learner performance on an evaluation unit (e-g., task, or environment) multiple evaluation units (e.g., set of tasks environments).
4

---

Published in Transactions on Machine Learning Research (09/2024)
TSCL
Cooperative Game
Learning
Example
Units
Players
U1, U2: Ug
Sequence
Coalition
(U2, U1)
Policy Value
Coalitional Worth
" ((u2 U1
Learning Progression
Marginal Contribution
J (TT2 - (mT)
u(U2 U1 v (1)) ((1))
Unit Value
Player Value
9k u1
Interactions
Allocations
Table 1: The mechanies of Teachhe-Studeet Curriculum Learning equivalent cooperative game among units of experience.
Consequently, every notion of coalitional worth is conditional the evaluation units, thus generating space of cooperative games.
Definition 4.1. (TSCL Cooperative Games) Let U denote set of units of experience E and U set of evaluation wnits U. Every evaluation coalition C 2 induces parameterized characteristic function ve (Cx) whose value measures the worth of coalition C₁ when the members of C are the evaluation units. Therefore, the TSCL-family of algorithms operate over parameterized space of cooperative games:
g (u,-) (( [C
comprising 2| 2|I possible games and where the target-task (i.e., C 11) and the multiple-tasks (i.e., C settings are special cases.
Example 4.2. If a learner algorithm is presented with units from C = {ZERO, on MNIST, the following condition is expected to hold:
CC= NINR, (C) > CC (EER(ZER,,D (C)
4.2 Marginal Contributions to Learning
The notions of coalitions and coalitional worth above induce a game-theoretic interpretation of the learning progression reward. At any itcration S K, this reward r(us) E R (Alg. [ line measures the improvement in policy performance after the teacher presents a unit uy to the learner algorithm that produces a new policy n [( Uc). Thus, we can restate this reward terms of game in characteristic function form:
r r(u)) = v(C,) v(Cx-1) = v(Cr-1 + uk) ( Ck-1)
(6)
and note its equivalence to computing the marginal contribution (see Sec. 2 and Eq. of aggregating the unit of experience Ua = u to the existing coalition Cs-1-
4.3 Fair Allocation Mechanism
A principle of fair attribution in cooperative games is that players get assigned values proportional to their expected marginal contribution. We note that under the learning progression objective, a bandit action-value estimate (u)
Eq. 4) approximates every unit's (or arm's 's) average marginal contribution after & interactions. Morcover, as discussed in Sec. 2.2] multi-arm bandit algorithms may transform action-value through Bolızmann projection (Eq. 5) that converts the value estimates into units' probabilities of interaction with the learner (i.e., the (stochastic) policy тı (u)). Consequently, the units that, up interaction & < K, have produced more significant increases learner performance and would be allocated larger fractions of the remaining K interactions.
Thus, a multi-armed bandit teacher implements a fair allocation mechanism that computes units' values by approximating their average marginal contributions and converts these approximations into the currency-like utility of the TSCL games, namely, interactions with the learner.
Table summarizes the equivalences between TSCL components and those of a cooperative game we have established throughout this section.
5

---

Published in Transactions on Machine Learning Research (09/2024)
5 An Experiment on The Prospect of Cooperation
We design an experimental setting to empirically verify the equivalences draw between TSCL components and cooperative game-theoretic concepts (e.g., do cooperative solution concepts capture some notion of curriculum?) and to highlight the utility of this data-centric approach to understanding TSCL failure modes. To this end, we incrementally build different parts of the equivalence in supervised learning, reinforcement learning, and classical game settings as follows:
For any set of units of experience given to the teacher algorithm, we simulate their interactions and compute each a priori Shapley or Nowak & Radzik value (Sec. 5.1.
We build two paalopprrpooiional mechanisms (i.e., pre-computed teacher policies) leveraging the prior units' values validate whether curriculum exists and to show that cooperative solution concepts retrieve such notion.
We leverage units' estimated pairwise interactions to understand the value- and TSCL failure from data-centric perspective (i.e., the composition of the of units).
value-propone proportional mechanism success
5.1 A Simulation of Cooperation
We simulate two coalition formation processes where units of experience (e.g., classes, environments, opponents) in each coalition fairly share finite interaction budget K E N and connect the value of the resulting learner's policies with coalitional's 's worth.
5.1.1 Coalitional Mechanics and Worth
Cooperative (Non-Ordered) Game. To simulate a traditional cooperative game (Sec. 2). we design coalition
formation process that draws at each interaction A < K a unit Uμ E C. with uniform probability rC(u:) x |C|-1 from a coalition C, and present it to learner algorithm. We compute this procedure for every coalition of units C E 2 The uniform distribution reflects an a priori ignorance of units' importance before measuring their effect on the learner.
Generalized (Ordered) Game. To test whether our formulation captures order for curriculum, we build a coalition formation process that simulates a generalized cooperative game. In this setting, a unit E C is continually presented to the learner, for K /|C[ interactions, in its permutation order on ordered coalition C. We repeat this procedure for every C E P22)). As before, the ordered equipartition of interactions reflects our ignorance about units priori effect.
Coalitional Worth & Characteristic Function. For every coalition, we obtain model =e after K interactions with the units in C through either the traditional or generalized mechanics described above. Therefore, by the equivalence we established between policy performance and (conditional) coalitional worth (Sec. 4.2). the evaluation of each policy determines the characteristic function (C). More importantly, because the resulting policy = is unbiased for the evaluation units or coalitions (c.g., the uninformed priors in the mechanics), we estimate, using the same policy T the coalitions worth for every characteristic function (C) parameterized by every evaluation target coalition C.
Example 5.1. (Example 4.1 cont'd) Assume a budget of K 100 interactions and a subset (coalition) of classes from MNIST, for instance, units (classes) ZERO and NINE. In the simulation of traditional game, for coalition C {ZERO, NINE), we uniformly draw instances from each unit with probability r(u) }. For coalition C = İZERO, NINE] in a generalized game, instances from unit ZERO are presented for the first k = 50 iterations followed by k = 50 instances from NINE. The resulting policy Th performance is equivalent to the value of the characteristic function v(C), evaluated at coalition C.
The players marginal contributions are the central quantity of cooperative solution concepts. What do marginal contributions capture in our experiments? First, in both coalition formation processes, adding a unit u to a coalition C while keeping the number of interactions K constant reduces the learner algorithm's 's interactions with the existing units. For instance, the traditional cooperative game simulation, adding unit u' reduces the probability of drawing any unit already in u E C from Pc(u) C to PC+ (u) (IC) + 1)- while in the the generalized game reduces units' interactions from [K/|C|] to [K/([C) + 1)]. Therefore,
Marginal Contributions & Solution Concepts.

---

Published in Transactions on Machine Learning Research (09/2024)
in either simulation, unit u marginal contribution v(C+ + u) - e((C) measures the change in performance produced by increasing learner's interactions with u while reducing those with the existing units in C.
Example 5.2. (Example 5.1 cont'd 'd In a traditional cooperative game simulation on MNIST, a marginal contribution such as v({ {ZERO, NINE}) } D( ((ZERO)) measures the change in learner performance produced by exchanging approxi- mately K/2 interactions with unit ZERO for interactions with NINE. However, in the generalized game simulation, the same expression measures the change in performance produced by exchanging K interactions with unit {ZBRO} for K/2 with ZERO first (pre-training) followed by K/2 with NINE (fine-tuning).
In consequence, both solution concepts leveraging marginal contributions, namely, the Shapley value for traditional games (Fq. T) and the Nowak & Razik's 's value for generalized games (Fq.. 2) estimate, in our simulations, a unit's average marginal contribution to learning, and thus capture their helpfulness or cooperativeness for curriculum.
5.1.2 A Sanity Check Through Supervised Classification
Our rumning examples on MNIST inspire the first setting we examine empirically validate the connections have established between TSCL and cooperative games. In this experiment, we considered instances aggregated in classes as units of experience and benefited from trained classifier's s confusion matrix to extract ground- truth information from unit interactions. For both MNIST and CIFAR 10 (Krizhevsky 2009), we trained a model on the complete dataset (c.g., for 200 epochs), extracted the confusion matrix on validation, and identify the top-k most confused pairs of classes. On MNIST, we selected the five classes TWO, THREE, FOUR, FIVE and SEVEN belonging to the top-three most confused pairs (see Appenddxxx A Fig. 4a), grouped their instances into five units of experience, and conduct the approximations described in [Sec. 5.1 for traditional cooperative game (i.e., not considering order). Next, we followed the same approach for CIFARIO six classes with more significant pairwise confusion ertors on validation, namely, CAR, CAT, DEER, DOG, FROG and HORSE (see Appendix A Fig. 4b).
If the equivalences we drew are correct, one may expect a unit Shapley value to be higher when the evaluation target is the same unit. Moreover, the pairwise interactions between units should approximately reflect the negative interactions we extracted from the confusion matrices of each trained classifier.
Units' Values. In effect, for either MNIST CIFAR1O, each unit's Shapley value estimated from the traditional cooperative game simulations correctly matches the ground truth information. In the target-unit setting, where each unit of experience is also used as evaluation unit, each unit of experience (or class) matching the evaluation unit (or class) has the largest Shapley value, as depicted in Fig. la (first five targets) for MNIST and Fig. 1c (first six targets) for CIFAR1O. 10. For instance, on MNIST, unit u = TWO has the largest Shapley value @(TWO) = 0.995 when the evaluation unit is û = TWO. We observed a similar effect on CIFAR 10. Furthermore, for the all-units setting, every
unit's Shapley value is approximately equal on both MNIST and CIFARIO (Fig. 1a and Tc. all column), matching the intuition that, conditional on an all units evaluation, every unit of experience should be equally valuable.
Measures of Interactions. We also computed the vPoP measure (see (see/Sec. 2.1 Eq. 3) to verify whether its decom- position of Shapley values into pairwise interaction values correctly identifies the most confused pairs of classes For both MNIST and CIFARI0 10, their respective vPoP matrices, displayed in Fig. 1b [d provide a reasonable approximation to the ground-trutt pairwise interactions extracted from the confusion matrices. For instance, on MNIST the wnits TWO and SEVEN have the lowest interaction value d(TWo, SEVEN) = ((SEVVN, TWO) = -0.007 which corresponds to largest entry (2, 7) 20 of the confusion matrix (see Appendix A Fig. 4ap. Also, in CIFAR 10 the -0.0164 coinciding with the most confused classes M (DOG, CAT) 66 on validation Appendix A Fig. 4bp. However, note that, for instance, on MNIST confusion matrix M(2, 7) * (7, 2) and, similarly, on CIFARIO'S 10's M (DOG, CAT) A (CAT, DOG). Nevertheless, we interpret these values as reasonable proxies for units negative, positive, or neutral pairwise interactions.
units DOG and CAT have the lowest interaction value d(DOG, CAT)
A valuable aspect of our cooperative game-theoretic analysis of TSCL is that the equivalences we introduced are not limited to the supervised classification setting where ground truth information is available. Next, we demonstrate the broad applicability of these ideas through problems in RL and classical games, where finding ground-truth notions of curriculum is non-trivial. Details reproduce these experiments are provided in inppppdiixx A
7

---

Published in Transactions on Machine Learning Research (09/2024)
Leggete [4)
Lista [ciees)
Thrae
Garen
0.05G
0.00L
0200
Tea-
LUNVY
GUOOZ
CUOO7
CL0G 0.9
OLSAA
O 3000
40002 - COLS
0,007
S.DCLL
Thrnk CU0O0
0.003
4.000
3334
CO00OL
GL000
4.003
0.0011
Ligg-0.000
-3.044
0.000
2006 QURe
01005
6.0002
PAND-
0.000
US0
Phe--
QUAGC
0,006
0.797
CU0G2
Pogg 700-00000000 01.9 -.000
0.00
#00--0000
0.000
4.002
40200
0.000
0.99
0.00
-00007
30000
-0.0022
0.0000
020033
900
63
60
600
Figure I: We validated the prospect prior using the class-as a a-umit analogy on MNIST and CIFARIO. In Figures (a) and (c), cach column represents units' Shapley values d(u) in cach cooperative game parameterized by a target-unit and the target coalition of all units. In Figures (b) and (d). we present the vPoP decomposition matrix (E.. 3 measuring the pairwise interaction values d(u u,) among units the all-units target.
5.1.3 Reinforcement Learning
We investigate the MINIGRID- -RORII (Chevalier-Boissertee Boisvert et al. 2018) set of three environments, namely, TwoRoOMS, FOURROOMS, and SIXROOMS for which it is folk knowledge that an optimal curriculum exists]] We apply the prospect prior simulation of generalized game where we consider each environment a unit of experience. As learner algorithm, we used PPO (Schulman et al., 2017) with interaction budget of K" 500, 000 steps, and estimated, from the outcome of these simulations, the Nowak & Radzik values (See. 2 Eq. 2}. with every environment and uniform distribution over all, as evaluation targets. In Fig. 2a] we show that the Nowak & Radzik values we estimate match folk knowledge. First, there is no requirement for environments other than TWORoOMS as the only positive value da TWOROOMS) = 0.423 correctly measures. Then, for FOURROOMS, the values of Φu (TwoRooMs) = 0.041 and du (FOURROOMS) = 0.107 indicate that both environments are required. And finally, environments values of da (FOURRROMS) = 0.03 and du (SIXROOMs) 0.03 indicate that both are needed for solving SIXRooMS.
5.1.4 Classical Games
We introduce an experimental setting, the Adversurial Sparse Iterated Prisoner's Dilemma (A-SIPD), that utilizes the Prisioner's s Dilemma (Flood 1952, Axelrod & Hamilton 1981) classical two-player game as a base but in a more challenging sparse and iterated version where at the end of a finite number of interactions (e.g., 200 steps), a win- draw-loss reward is given to the learner if it accrues more cumulative payoff than its opponent. Opponents are drawn from a population of five well-known strategies: ALWAYSCOOPERATTE, ALWAYSDEFECT,, WINSTAYL.OSE TITFORTAT (Axelrod [98T) and a ZERODETERMINAN strategy (Hilbe et al., 2013). We apply the prospect prior simulation of generalized game where consider each opponent unit of experience. As a learner algorithm, we used PPO (Shhulman et al. 2017) with a budget of K 100, 000 interactions.
We estimated the Nowak & Radzik values Sec.2 2 Eq. 2). conditioned on each opponent and a uniform mixture over all, as evaluation targets. The results we present in Fig. 2c show that playing uniquely against TITFORTAT is sufficient across all evaluation targets, including the most challenging opponents, ALWAYSDEFECT and ZERODETERMINANT. This result contrasts with folk knowledge in population- based training (e.g., playing against the population's Nash more in-depth discussion of this finding.
strategy (Nash 1950)). We defer to Sec. 5.3
5.2 Curriculum from A Priori Values
For both the RL and classical games settings, the Nowak & Radzik value correctly ordered tasks and opponents, respectively, by their contribution to learning. Nevertheless, we also investigated whether these values' magnitude indicated the proportion of interactions (i.e., a fraction of the K interactions budget) that should be allocated to each unit and whether the combination of order and magnitude retrieved an approximate curriculum.
Therefore, inspired by calle-proportionall allocations (Bachrach et al. 2020), developed two mechanisms that turn units' values into interactions with the learner by projecting any value vector (u) g onto vectors T r(u) E Au in U simplex. While Bachrach et al. (202)) use a linear projection unable to handle negative marginal contributions, we
'The curriculum order between these three environments RL is an intuition that, to knowledge, has not heen quantified before.
8

---

Published in Transactions on Machine Learning Research (09/2024)
terpert [+]
Teatocma
Shipars
06-
fad copis
041
Techocma-
4442
6D41
0000
0083
ronkk oroggi
Cospessrr
6:00
033
4:20
02
TUNGR di strpic
Suloon
-0342
130
20
6:11
00
12300
2000
30d000
40000
30000
0:200
0.186
0.00
-04-
200
43000
60/00
molon
1200
(0) Rudaa
0) Loxning Curves (all)
60) Novak
Radok Vaaaes
Louning Curves (all)
MINICIED-
A-SIPD
Figure 2: Nowak & Radzik values (a, c) conditional on each single-umit and the multiple-umits (all) evaluations, and the learning curves (b,d) for our mechanisms and TSCL.
01/91
0:0091
COOL
0120
CALOT
5.099
DAGS
EOGG
AL508
6-A-13
(b) Stuptee--PPPP
( Nowakdittddttoole
G0 Staplay-shott
(d0 Newokkiittttkesaff
Figure 3: The vPoP decomposition of Shapley's 's and Nowak & Radzik values, conditioned on the all all-units evaluation target, for the MINIGRID-ROOD (a. b) and A-SIPD (c. d) problem settings.
investigate the Boltzmann or softmax projection, commonly used in multi-arm bandit algorithms (see Sec. 2.2 Eq. 5) and an Euclidean projection (Blondel et al. 2014) that projects to zero any unit with negative value.
When values (u) are Shapley values, the projected vectors used as pre-computed teacher policies (i.e., probability distributions), mimicking TSCL's interactions but fixed a-priori knowledge of units' value. However, when values Φ(u) are Nowak & Radzik, vectors E Au are used as ordered compositional vectors (Aitchison, 1982) that represent ordered fractions of K interactions. Thus, we construct a pre-computed teacher policy that, first orders units by their Nowak & Radzík values, projects the ordered values onto т E A and presents the i-th ranked unit u, E to the learner for the number of interactions indicated by т; E N. This mechanism preserves the ordered values captured by Nowak & Radzik's solution concept.
Fig. shows that for both MINIGRID-ROOME and A-SIPD the teacher policies obtained from these value caluee-roppottonall mechanisms, in particular the Euclidean projection of Nowak & Radzik values (nowak-al) soowak-all-sinpe all-simplex), consistently produce learner-induced policies that solve the target tasks, and empirically validate our hypothesis that a priori values, in particular the generalized Nowak & Radzik value we introduced, retrieve both the order of and the proportions of interactions that should be allocated to each unit to produce curriculum.
TSCL Data-Centric Failures. Moreover, Fig. 2 also highlights the multi-arm bandit approach (ExP3 (Auer al. 2003:
Graves al. 2017)) TSCL (i.e., tscl-all-exp3s) failure produce an effective curriculum. We found that in the presence of units with non-cooperative interactions, bandit-driven TSCL fails. Fig. 3 presents the vPoP decomposition of Shapley and Nowak & Radzik values conditioned on the all-units evaluation. In both settings, the interactions measured from the Shapley-based decomposition produce lower (negative) values than those obtained with Nowak & Radzik. As we show through Seee. 4| (Sec. 5| we take the Shapley-based interaction values as fair approximations of TSCL interactions, and thus they provide data-centric explanation to TSCL failures (see Ssee/Appedix B). On the other hand, the Nowak elimination, through the Éuclidean projection, of negatively-vatus valued units. These pruning strategies are employed effectively by data valuation techniques (Yan & Procaccia, 2021).
Radze values explain the success of nowak-all-simples mechanism, along with the
5.3 Single and Population-based Curriculum
These results offer an alternative approach to what population- based training approaches prescribe as curriculum (Lanctot et al. 2017, Balduzzi al.) 2019) Garnelo et al. 2021 [iu al. 2022). Generally, meta-strategy solvers for population- based training leverage tools from non-cooperative game theory (Von Neumann & Morgenstern 1944) to find for
9

---

Published in Transactions on Machine Learning Research (09/2024)
instance, the mixed Nash equilibrium (Nash 1950) of the empirical meta-game (Wellman, 2006) played by the population of opponents. In this sense, we may also understand TSCI. as a cooperative mela-strategy solver that prioritizes among a fixed population of (non-lcarning) opponents (units of experience) those that improve the learning progression of a single learning player against one or more opponents of the same population.
In the sparse and iterated version of Prisoner's Dilemma that we introduced, the Defector strategy remains the (empirical) game Nash equilibrium. However, the ordered prospect prior results in Fig. scoh that when evaluated the population Nash = Defector, the largest Nowak & Radzik value corresponded to the Tat strategy du (TiiFoorTa)) 0.34. Playing against the TitForTat strategy remains the optimal solution across all evaluation targets, meaning that TitForTat is the best proxy opponent to learn from and reach the Nash equilibrium strategy. We can interpret this result from two perspectives. First, it could indicate that the sparse, iterated, and overtly adversarial version of the game we constructed is a more complicated problem than the original, and the Nash equilibrium. the Defector strategy, a stronger opponent. However, these results may also indicate that a cooperative approach to meta-strategy selection may improve performance in some scenarios. We believe that this connection warrants further investigation.
6 Limitations
The simulations we computed in our experiments are computationally expensive. It is well-established that cooperative solution concepts are NP-hard (Deng & Papadimitriou 1994 Elkind al., 2009). However, better approximations are possible (Yan & Procaccia 2021, Mitchell et al.) 2022) although we do not explore them in this work. Consequently, we do not consider the prospeci prior experiments and the value-proportional curriculum mechanism as algorithmic innovations to replace TSCL. They represent a data data-centric approach to study the limits imposed on TSCL-style algorithms by the (non)cooperatiie mechanics among units of experience. However, we acknowledge that the mechanics of units' interactions also affect other aspects of TSCL. These aspects may include, for instance, the teacher's credit- assignment problem (Gittins 1979) or neural networks learning and forgetting dynamics (e.g., [ee al. (202T)). We control for these factors by keeping them constant in our experiments (see (spenndp A) but do not undertake their analysis here. Our work is starting point for more thorough explorations of TSCL and curriculum learning, their underlying mechanisms and broader applicability in machine learning.
7 Related Work
Curriculum Learning. Since the seminal works of Elman (1993); Krueger& & Dayan (2009) and Bengio et al. (2009),
large body of literature has been produced on curricula for machine learning algorithms. Excellent surveys presented
by Narvekar et al. (2022); Wang et al. 2022), Soviany et al. (2022) offer a comprehensive state of recent advances in the field. Our work closely inspects the TSCL. framework concurrently introduced by GGraves et al. (2017) and Matiisen
et al. (2020). While follow-up works have generally focused on either algorithmic innovations (Weinshall et al. 2018
Portelas et al., 2020, Feng et al. 2021: Liu al., 20200 Portelas et al. 2019b Racaniere et al. 2020 Florensa al.
2017 Campero al., 2021 Du et al. 2022 Florensa et al., 2018) or evaluation benchmarks (Chevalier-ooisvert
et al. 2019. Romac et al. 2021). ), the data-centric perspective (Ng. 2021 Karpathy & Abbeel 2021 proposed here is less explored. The work by Wu et al. (2020) empirically verifies the same fundamental questions on why, when, and how curriculum learning works. However, while our work is more specifically focused on TSCL, we provide a thee-the grounding that could be further extended to analyze Wu et al.) (2020) setting. On the other hand, Lee al. (202T) specifically explore TSCL but with a focus on the deep neural network dynamics leading to catastrophic forgetting McCloskey& & Cohen (1989). The evaluation mechanisms that we propose in the prospect prior simulation and the nume-theoretic characterization of experience interference we introduce here provide a more general grounding for the problems studied there.
Machine Learning & Game Theory. Our work draws much of its inspiration from the tradition of cross-po rross-pollination between machine learning and game theory research (Cesa- Bianchi & Lugosi 2006). In recent years, game theory
research has fuelled work in optimization (Daskalakis & Panageas 2018 Jin et al. 2019). generative modelling (Good-
fellow et al.) 2014; Farnia & Ozdaglar 2020 Mohebbi Moghaddam et al. 2023). ) or robustness (Madry et al.) 2018 Huang et al., 2022a). Also, game-theoreti arguments have been used to revitalize mta-siin problems in ma- chine learning. For instance. (Gemp et al. (2021) reframed the century-old problem of Principal Component Analysis
(PCA) (FR.S.. 1901 Hotelling 1933) as the Nash equilibrium (Nash 1950) of multiplayer game. Recently, Chang
10

---

Published in Transactions on Machine Learning Research (09/2024)
et al. (2020) reformulated sequential prcision-making problems (i.e., RL (Sutton & Barto 2018)) as local economic transactions among specialized self-interester agents. Moreover, and closer to our data-centric approach, Lundberg & Lee (2017) pioneered work on feature attribution and explainable machine learning models through cooperative game theory. In particular, they approximate Shapley's 's value (Shapley 1952) (Eq. 1p to compute the influence input features have on model predictions. In way, our work extends Lundberg & Lee (2017) input-feature-ung leceattrr-sssaaapae -s-a--llaye analogy through the notion of units of experience and introduces substantial elements to discuss how the cooperative mechanics of experience affects TSCL at different levels of abstraction (i.e., from features to tasks) and varied learning paradigms.
Active Learning. As Graves et al. (2117) and Matiisen et al. (2020) noted in their works, TSCL is an experience- prioritization mechanism holding a solid connection to algorithms that actively select experience. The central problem in active learning is to decide which experience to present to a learning algorithm by querying an expert or oracle (Settles) 2010). Areas of research that have also adopted this paradigm of actively sampling experience include experience replay mechanisms in RL. [Schaul et al. (2016); Andrychowiez et al. (2017); Fedus al. (2020) where transitions stored in a replay buffer (Lin 1992) are sampled according to some prioritization mechanism. This category of active methods fits perfectly under the TU-game among units of experience introduced in |Sec. 4
Multitask Learning. The cooperative game we designed also provides grounding the problem of determining which tasks should be learned together and what their optimal order is, originally from the literature of multitask learning (Caruana 1997). The discussion on experience interference in (Sec. 4] the prospect prior simulation and our computation of ordered and unordered Shapley values, and the vPoP metric are some novel tools we introduce that could shed new light on this problem. Recent work has explored the different avenues of how task interference and ordering affect learning outcomes (Standley et al. 2019) Shamsian et al. 2023 Zhang et al. 2021 Fifty et al.) 2021 Lin et al. 2019). Our work provides a novel grounding that relates easily to these efforts.
Continual Learning. Finally, another area of interest to our work is continual learning Parisi et al. (2019); De Lange et al. (2022); Mundt al. (2023). We note that the ordered prospect prior simulates a continual learning setting. Units of experience are presented the learning algorithm in order, and their importance (i.e., value in the cooperative game) is estimated using Nowak & Radzik (1994) extension (Eq. 2) to Shapley's 's value. This perspective offers principled approach to rank and order tasks by their importance, critical aspect of the continual learning setting. We believe that future extensions to the ordered prospect prior experiment may be part of toolbox to understand continual learning benchmarks (Lomonaco& & Maltoni 2017 Lin et al. 2021 Srinivasan et al., 2022), although we do not undertake that task here.
8 Conclusions & Future Work
We reexamined TSCL through the lens of cooperative game theory. By drawing inspiration from work on data valuation, feature attribution and explainability, we provide a novel data-centric perspective that re- re-frames several of its components through alternative cooperative game- gameetheoretic interpretations. Our experiments confirmed the appropriateness of studying TSCL-style under this framework and highlighted the impact of units' cooperative mechanics on this problem. However, we only began to unveil the potential of allocation mechanisms, solution concepts, and measures of interactions to explain some fundamental aspects of TSCL and hope our work inspires an influx of novel geme-thhereee approaches to the problem. Future work would explore more decoretialll-ggoundee between convex games (Shapley, 1971) and moduullattt in discrete combinatorial optimization (Dughmi) 2009 Bach, 2011 Krause & Guestrin 2011) and the extension to continuous set of units through values of non-atomic games (Aumann & Shapley 1974).
analysis of this problem through the connection
Acknowledgements
MD would like to thank Eugene Vinitsky for insightful feedback on earlier versions of this manuscript. The NSERC Discovery Grant and the Canada CIFAR AI Chair program supported MD and LP. Computing resources provided by Mila, Québec AI Institute, partly enabled this research.
11

---

Published in Transactions on Machine Learning Research (09/2024)
References
Aitchison. The statistical analysis of compositional data. Journal of the Royal Statistical Society, 44(2):139-160,
January 1982. 9
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Openai Pieter Abbeel, and Wojeiech Zaremba. Hindsight experience replay. In I Guyon, U Von Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, and R Gareett (eds.), Advances in Neural Information Processing
Systems, volume 30. Curran Associates, Inc., 2017. 11
Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM J. Comput., 32(1) 77, jan 2003. ISSN 0097-5397. doi: 10. 10.1137/50997553003398777 URL
3
20
21
R. J. Aumann and Shapley. Values of Non-Atomic Games.
Princeton University Press, 1974. URL http:
7/www. http:///st.toooot/tttt/e//......
R Axelrod and W Hamilton. The evolution of cooperation. Science, 21 1418 March 1981.
8
Robert Axelrod. The emergence of cooperation among egoists. The American political science review, 75(2):306-318,
1981.
8 21
Francis Bach. Learning with submodular functions: convex optimization perspective. November 2011.
Yoram Bachrach, Richard Everett, Edward Hughes, Angeliki Lazaridou, Joel Leibo, Marc Lanctot, Michael Johanson, Wojeiech M Czarnecki, and Thore Graepel. Negotiating team formation using deep reinforcement learning. Artificial intelligence, (a0((1333)):03333555) November 2020. g
David Balduzzi, Marta Gamelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel. Open-ended learning in symmetric zero-sum games. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 Proceedings of Machine Learning Research, pp. 434 443. PMLR, 2019. 9
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pp. 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. URL
1553380
I0
Lilian Besson. SMPyBandits: an Open-Source Research Framework for Single and Multi-Players Multi-Arms Bandits (MAB) Algorithms in Python. Online at: git com/ SMMPPBaandtt///EMPaaaaddd...... 2018. URL httts:
//github. com Seoom//MPyaandiitee StMPPBaanditst/SMPPPannnrr /
Code http://gghhiiicooooyyyyyyyaattte
documentation at http:://mpyyyaandd..iiit.b.....e 20
Mathieu Blondel, Akinori Fujino, and Naonori Ueda. Large-Scale multiclass support vector machine training via
euclidean projection onto the simplex. In 2014 22nd International Conference on Pattern Recognition, pp. 1294,
August 2014. 9
Andres Campero, Roberta Raileanu, Heinrich Kuttler, Joshua B. Tenenbaum, Tim Rocktäschel, and Edward Grefen- stette. Learning with [amig)o: Adversarially motivated intrinsic goals. In International Conference on Learning Representations, 2021. URL httpsss//openeerie..cettt
EEBe. MIMgoX
10
Rich Caruana. Multitask learning. Machine learning. 28:41-75, 1997.
Nicolo Cesa-Biancha and Gabor Lugosi. Prediction, Learning, and Games. Cambridge University Press, 2006. doi:
10. 10.1077/C80970051155550 T0
Michael Chang, Sid Kaushik, S Matthew Weinberg, Tom Griffiths, and Sergey Levine. Decentralized reinforcement learning: Global Decision-Makiig via local economic transactions. Hal Daumé lii and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1437-1447. PMLR, 2020. TO
12

---

Published in Transactions on Machine Learning Research (09/2024)
Maxime Chevalier-Boiswe Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium, 2018. URL httph::t//// http//tttt.......
onn Miniiiiiidd 20
Maxime Chevalier-Boinze Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. BabyAI: First steps towards grounded language learning with a human in the loop. International Conference on Learning Representations, 2019. URL httph:t//tppteeeee
IJeXCoDeYX 10
Constantinos Daskalakis and Ioannis Panageas. The limit points of (optimistic) gradient descent in min-max optimization. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Biana cisaaBianchh,, and R. Gareett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso-
ciates, Inc., 2018. URL httpps:/pproceedngg..nt www.:eeeerrrrppseeees..e http:///.....ppppp...aaaa/////// Files//paperr/201//EiDe 139c3c1 b7ca 6a 9d4 fd6d1 63d98a f635- pddf
10
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on patter analysis and machine intelligence, 44(7):3366-335, July 2022.
Xiaotie Deng and Christos H. Papadimitriou. On the complexity of cooperative solution concepts. 19:257-266, 1994. ISSN 0364-765X.
2
T0
Yuqing Du, P Abbeel, and Aditya Grover. It takes four to tango: Multiagent selfplay for automatic curriculum generation.
International Conference on Learning Representations, 2022. TO
Shaddin Dughmi. Submodular functions:
Extensions, distributions, and algorithms. survey. December 2009,
11
Edith Elkind, Leslie Ann Goldberg, Paul W Goldberg, and Michael Wooldridge. On the computational complexity of weighted voting games. Annals of mathematics and artificial intelligence, 16(2):099-311 June 2009.
I0
L. Elman. Learning and development in neural networks: the importance of starting small. Cognition, 48(1):71-99, July 1993.
10
UIrich Faigle. Mathematical Game Theory. 2022.
Farzan Farnia and Asuman Ozdaglar. GANs may have no nash equilibria. arXiv e-prints, pp. arXiv:2002.09124. February 2020. I0
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney, Revisiting fundamentals of experience replay. In Hal Daumé III and Aarti Singh (cds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3061-3071. PMLR, 13-18 Jul 2020. URL
mll (treas/vl19/feddae
html
Dieqiao Feng, Carla P Gomes, and Bart Selman. A novel automated curriculum strategy to solve hard sokoban planning instances. October 2021. I0
Christopher Fifty, E Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for Multi-Task learning. Advances in neural information processing systems, 2021.
Merrill M. Flood. Some Experimental Games. RAND Corporation, Santa Monica, CA, 1952.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp. 482-495, PMLR, 2017. 10
Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for reinforcement learning agents. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1515-1528. PMLR, 2018. 10
13

---

Published in Transactions on Machine Learning Research (09/2024)
Karl Pearson F.R.S. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-5777 1901. doi: 10. 10.110001148864401194467777
10
Marta Garnelo, Wojeiech Marian Czarnecki, Siqi Liu, Dhruva Tirumala, Junhyuk Oh, Gauthier Gidel, Hado van Hasselt, and David Balduzzi. Pick your battles: Interaction graphs as population-leve objectives for strategic diversity. In Proceedings of the 20th International Conference on Aulonomous Agents and MultiAgent Systems, AAMAS '21, pp. 1501-1503, Richland, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450383073.
Ian Gemp. Brian McWilliams, Claire Vermade, and Thore Graepel. Eigengame: (PCA) as a nash equilibrium. In International Conference on Learning Representations, 2021. URL
net wttp://wwwee//oo////// forum?id=
NzTU59SYbNq
10
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. April 2019,
C Gittins. Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society, 41(2):148-164, January 1979.
2 3 10
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville,
and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27, Curran Asso- ciates, Inc., 2014. URL https::
neur wwwr.ps.cc/ppppeeeeeeeeee Filess/paeer2211/fiiee .pdf
10
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Michel Grabisch and Marc Roubens. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of Game Theory, 28(4):547-565, Nov 1999. ISSN 0020-7276. 2
Alex Graves, Mare Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning. volume 70 of Proceedings of Machine Learning Research, pp. 1311-1320 PMLR, 2017.
20 21
Kjell Hausken and Matthias Mohr. The value of player in n-person games. Social choice and welfare, 18(3):465-483, 2001. ISSN 1176-1714, 3
Christian Hilbe, Martin A. Nowak, and Karl Sigmund. Evolution of extortion iterated prisoner's dilemma games. Proceedings of the National Academy of Sciences of the United States of America, 110(17):6913-69) 69119991116111- Apr 2013.
ISSN 0027-8424,
Harold Hotelling. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology, 24:498-521 520. 1933. 10
Peide Huang, Mengdi Xu, Fei Fang, and Ding Zhao. Robust reinforcement learning as a stackelberg game via adaativeely-reuularizedd adversarial training. February 2022a.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022b.
html
20
21
Chi Jin, Praneeth Netrapalli, and Michael Jordan. Minmax optimization: Stable limit points of gradient descent ascent are locally optimal. arXiv.org, 2019. 10
Andrej Karpathy and Picter Abbeel. The robot brains podcast: Andrej karpathy on the vi-
sionary ai in tesla's autonomous driving. https::/podcastee apple wttpappppp..ooooppeeeeeeeeeeeeee
1000513993723 2021. 10
14

---

Published in Transactions on Machine Learning Research (09/2024)
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representatiins, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
wttp://.rrggabs//11111/ 6980
19 20 21
Vince Knight, Owen Campbell, Marc, J Gaffney, Eric Shaw, Vsn Reddy Janga, Nikoleta Glynatsi, James Campbell, Karol M Langner, Sourav Singh, Julie Rymer, Thomas Campbell, Jason Young, MHakem, Geraint Palmer, Kristian Glass, Daniel Mancia, edouardArgenson Martin Jones, kjurgielajtis, Yohsuke Murase, Sudarshan Parvatikar, Melanie Beck, Cameron Davidson- Pilonn, Marios Zoulias, Adam Pohl, Paul Slavin, Timothy Standen, Aaron Kratz, and Areeb Ahmed. AxelroddPyyhhon/Axelndd v4. 12.0, October 2021. 21
Andreas Krause and Carlos Guestrin. Submodularity and its applications in optimized information gathering. ACM Trans. Intell. Syst. Technol. 2(4):1-20, July 2011.
II
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, April 2009.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009,
19
Kai A Krueger and Peter Dayan. Flexible shaping: how learning in small steps helps. Cognition, 110(3):380-394
March 2009.
2 10
Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel. A unified game--hhoree approach to multiagent reinforcement learning. Advances in neural information processing systems, 30, 2017. 2
Tor Lattimore and Csaba Szepesvári. Bandit Algorithms. Cambridge University Press, July 2020. 3
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: lecun. com/
exdb/ axcb/mnist// mnist//
B 19
Sebastian Lee, Sebastian Goldt, and Andrew M. Saxe. Continual learning in the teacher-student setup: Impact of task
similarity. International Conference on Machine Learning, 2021. URL htttps::/www.... http:///wwwwssmmttcccccoooooooo
10
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning. 8(3):233-321, May 1992.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and S Kwong. Pareto Multi-Tase learning. Advances in neural information processing systems, 2019. 11
Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. In Thirty-ffth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Rui Liu, Tianyi Wu, and Barzan Mozafari. Adam with bandit sampling for deep learning. October 2020.
Sigi Liu, Luke Marris, Daniel Hennes, Josh Merel, Nicolas Heess, and Thore Graepel. NeuPL: Neural population learning. In International Conference on Learning Representations, 2022. URL htt.tttss//oggeeereee..et net.
forum?id=MIX3EN
Vincenzo Lomonaco and Davide Maltoni. Core50: new dataset and benchmark for continuous object recognition. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pp. 17-26. PMLR, 13-15 Nov 2017. URI.
attps://procedinng. mlr. wwwss/pessssv. v /tt.r/cceddnnne.......ess/////// http:///////mmmmmm1111...a.
II
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pp. 4768-4777 4777, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 97815 10860964.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL
https://openrevii ew net. 2tt.::/rrrrmmm.m..1
d=rJzIRfZAb
15

---

Published in Transactions on Machine Learning Research (09/2024)
Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Student curriculum learning. transactions on neural networks and learning 3ystems, 310))3333--3740, September 2020.
JEEE
TO
Michael McCloskey and Neal J Cohen. Catastrophie interference connectionist networks: The sequential learning problem. In Gordon Bower (ed.), Psychology of Learning and Motivation, volume 24, pp. 109-165. Academic Press, January 1989. T0
Rory Mitchell, Joshua Cooper, Eibe Frank, and Geoffrey Holmes. Sampling permutations for shapley value estimation. Journal of machine learning research: JMLR, 23(43):1-46, 2022.
Monireh Mohebbi Moghaddam, Bahar Boroomand, Mohammad Jalali, Arman Zareian, Alireza Daeijavad, Moham- mad Hossein Manshaei, and Marwan Krunz. Games of GANs: game-theoretical models for generative adversarial
networks. Artificial Intelligence Review, February 2023. TO
Martin Mundt, Yongwon Hong. Iuliia Pliushch, and Visvanathan Ramesh. wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning. Neural networks: the official journal of the International Neural Network Society, 160:306-323 336, March 2023.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: framework and survey. Journal of machine learning research: JMIR, 21(1): 7382-7431, June 2022. 10
JF Nash. Equilibrium points in N-Person games. Proceedings of the National Academy of Sciences of the United States of America, 36(1):48-49, January 1950. 8 TO
Andrew Ng. Mlops: From model-centric data-centric ai.
hetpss//www..youuuuss.coo//aaaaa
06 AZXmwH fo 2021. I0
Andrzej S Nowak and Tadeusz Radzik. The shapley value for n n-person games in generalized characteristic function
form. Games and economic behavior, 6(1):150-161, January 1994. 3
Pierre-Yvee Oudeyer, Frédéric Kaplan, and Verena V Hafner. Intrinsic motivation systems for autonomous mental
development. IEEE transactions evolutionary computation, 11(2):265-286 2007. 2
German Parisi, Ronald Kemker, Jose I. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural networks: the official journal of the International Neural Network Society, 113:
54-71, May 2019. 11
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http:::/ttaapppee nww.ss.nuuuggeee www.ips.ucpcccmmmmm wttpppp.ccppppp...ppppp 9015 pytorchoh-hhho.h
imperativee....mmmmmm style high www.pppproooommeeeeeeemmmme deep Learninannnn wibrrriibbrrr..... -earaing-
19
20
21
Roma Patel, Marta Garnelo, Ian Gemp, Chris Dyer, and Yoram Bachrach. Game-theoretic vocabulary selection via the shapley value and banzhaf index. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2799-2798, Online, June 2021. Association for Computational Linguistics.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer. Teacher algorithms for curriculum learning of
deep RL in continuously parameterized environments. October 2019a. [
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning of deep RL in continuously parameterized environments. October 2019b.
10
Rémy Portelas, Clément Romac, Katja Hofmann, and Pierre-Y Yves Oudeyer. Meta automatic curriculum learning. November 2020. 10
16

---

Published in Transactions on Machine Learning Research (09/2024)
Ariel Procaccia, Nisarg Shah, and Max Tucker. On the structure of synergies in cooperative games. Proceedings of the
AAAI Conference on Artificial Intelligence, 28(1), Jun 2014. ISSN 2374-3468. doi: 10.1609/aaii.
URL
2
Sebastien Racaniere, Andrew Lampinen, Adam Santoro, David Reichert, Vlad Firoiu, and Timothy Lillicrap. Automated
curriculum generation through setter-solver interactions. In International Conference on Learning Representations, 2020. 10
Clément Romac, Rémy Portelas, Katja Hofmann, and Pierre-Yves Oudeyer. TeachMyAgent: a benchmark for automatic curriculum learning in deep RL. March 2021. T0
Alvin E Roth (ed.). The Shapley value: essays in honor of Lloyd S. Shapley. Cambridge University Press, October 1988. 2
Estela Sanchez and Gustavo Bergantiñoos. On values for generalized characteristic functions. Operations-Research Spektrum, 19(3):229-22) 234, September 1997.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In 4th International Conference on Learning Representations, 2016. 11
Schmidhuber. possibility for implementing curiosity and boredom in model-building neural controllers. The MIT
Press, 1991. ISBN 9780262256674. 2
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization
algorithms. arXiv preprint arXiv: 1XXi::777.0033 2017. g 20 21
Burr Settles. Active learning literature survey. Machine learning, 15(2):201-221, 2010.
Aviv Shamsian, Aviv Navon, Neta Glazer, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Auxiliary learning as
asymmetric bargaining game. ArXiv, 2023.
11
Lloyd S Shapley, A value for N-Person games. Technical report, RAND Corporation, 1952.
Lloyd S Shapley. Cores of convex games. International Journal of Game Theory, 1(1):11-26, December 1971, 11
Martin Shubik. Game theory models and methods in political economy. In Kenneth Arrow and Michael Intriligator (eds.), Handbook of Mathematical Economics, volume 1, pp. 285-330. Elsevier, 1981. 2
Petru Soviany, Radu Tudor lonescu, Paolo Rota, and Nicu Sebe. Curriculum learning: A survey. International journal of computer vision, 13066):1526-166; June 2022. 10
Tejas Srinivasan, Ting-You Yun Chang, Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for vision-and -an-a-an--- tasks. volume 35, pp. 19440-29453, 2022. [T
Trevor Standley, Amir R 7amir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? May 2019. II
Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. The MIT Press, 2nd edition, 2018.
11
Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal. Safe reinforcement learning via curriculum induction. Advances in neural information processing systems, 13:12151- 2020.
René van den Brink and Gerard van der Laan. Axiomatizations of the normalized banzhaf value and the shapley value. Social Choice and Welfare, 15(4):567-582 1998. ISSN 01761714, 1432217X. URL http::/www/ istor org/
stabl mtable//411022
2
John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. Princeton University Press, Princeton, NJ, 1944. 9
Xin Wang. Yudong Chen, and Wenwu Zhu. survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, ((9):455545576 September 2022. 10
17

---

Published in Transactions on Machine Learning Research (09/2024)
Robert J Weber. Probabilistic values for games. In A E Roth (ed.), Essays the Shapley Value and Its Applications, pp. 101-119. Cambridge University Press, 1988. 2
Daphna Weinshall, Gad Cohen, and Dan Amir. Curriculum learning by transfer leariigg: Theory and experiments with deep networks. Technical report, 2018. TO
Michael P. Wellman. Methods for empirical game-theoretic analysis. In Proceedings of the 2 National Conference on Artificial Intelligence Volume 2, AAA[06, pp. 1552-1555. AAAI Press, 2006. ISBN 9781577352815, I0
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? International Conference on Learning Representations, 2020.
Omry Yadan. Hydra framework for elegantly configuring complex applications. Github, 2019. URL https:
19
Tom Yan and Ariel D Procaccia. you like shapley then you'll love the core. Proceedings of the AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 35())5771-5575, May 2021. L
Yipeng Zhang, Tyler L Hayes, and Christopher Kanan. Disentangling transfer and interference in learning. ArXiv, 2021.
18

---

Published in Transactions on Machine Learning Research (09/2024)
A Prospect Prior Experiments Details
We provide for all problems, models or policies architectures, algorithm hyperparameters, and other reproducibility details All models and architectures implemented with PYTORCH (Paszke et al. 2019), configured using HYDRA (adann 2019), and fit on workstation equipped with a 16 GB NVIDIA RYX A4000 GPU, 32 GB of RAM, and 32 CPU cores.
A.1 Supervised Learning
MNIST. We trained a model on the MNIST (LeCun & Cortes, 2010) supervised 10-digits classification task. Specifica- tion of the model architecture and hlperr-paameete selection are provided in Table
Hyperparameter
Value
Model
optimizer learning-rand rate betas eps batch-size
ADAM
Kingma & Ba 10
2015
CONv2D(32, RELUO
(0.9, 0.999) 10
CONv2D (64, 1)
RELUO
MAXPOOL2D (2,
epochs
200
DROPOUT(0.25)
shuffle
Yes
FLATTENO LINEAR(9216, 128)
RELUO DROPOUT(0.5)
LINEAR(128, 10)
Table 2: Details on the learning algorithm hyperparameters (left) and model architecture (right) used in the MNIST (LeCun & Cortes (010) experiments. Model components and the optimizer provided PYTORCH (Paszke et al. 2019). These details remained constant throughout the rest the experiments with MNIST.
CIFAR10. The experiments on CIFAR10 Krizhevsky et al. 2009) follow the same setting as those on MNIST. We similarly trained a model on the supervised 10-classes task. Specification of the model architecture and hyperparameter
selection are provided in Table
Hyperparameter
Value
Model
optimizer learning-ratt -aate betas
ADAM Kingma Ba 2015
CONv2D (3, 6, 5) RELUO
10
(0.9, 0.999)
MAXPOOL2D (2,
10
CONv2D (6, 16, 5)
batch-size epochs
RELUO MAXPOOL.2 2 (2, 2) FLATTEN(
200
shuffle
Yes
LINEAR(400, 120) RELUO LINEAR 84)
RELUO
LINEAR(84, 10)
Table 3: Details learning algorithm hyperparameters (left) and model architecture (right) used in the CIFAR10 Krizhevsky ct al. 2009) experiments. Model components and the optimizer are provided by PYTORCH (Paszke et al. 2019) These details remained constant throughout the the experiments with CIFAR10.
Regardlesss plan release the complete source code of all experiments.
19

---

Published in Transactions on Machine Learning Research (09/2024)
oight
Flig
Dgg
24
20
DUSS
[UO01
40000
0J177
0,0001
000022
30X27
Dyo
L2
17
40001
4L003
0000
4,000
0.198
Four-
byc
25
FrAe
-00000
40.003
0001
0.977
0.000
0.77
FIx--
0.0006
0L000S
00022
Dgg
Seare
<0001
<0000
0.000
00000
0.995
CUNDS
Soort-
0.0003
0,0002
0,208
Cal MNISS
Sarguee [4)
QAR
DOG
CAT
INS
<UOD1
-00017
4,0011
11
27
9G
0,055
00020
01000
0,0000
00007
0C004
(L0054
0,0053
23
Des 1000-000 0.012
0,09
0027
004
00040
0.003
0,0012
0000
0.:20
ELO057
4L0054
020077
Dog 0100
4L003
0772
4109
4U03S
CURSS
Dog- 0.00
00164
0.057
0:233
0,0029
0.0062
hog-00 D08
0.04
0.036
00:5
0.1
00009
0.121
hag-
20007
0.004
0.054
0,0023
0.1266
0,0000
4U234
4000
4.0
201
0113
IEm OIIILL QU00S 4U0371
4L0003
4L0000
0.344
conPustons
(b) CIFARIO
Figure 4: The class-as-a-unit analogy applied to MNIST (a) and CIFARIO (b) served our ground truth. For each problem, we derived the Shapley's value from the precomputed priors (left) [Eq. on each cooperative game (Sec. 5). Our results verify that units values on the targer-unit settings approximately ordered the most confused pairs of classes. For instance, digits 2 & 7 in MNIST, or dog & cat in CI When the target is all classes, the vPoP decomposition (right) also (Sec. 2.I) identifies interfering pairs.
A.2 Reinforcement Learning
MINIGRID ROOMS. We utilized a sequence of TWOROOMS, FOUR and SIXROOMS gridworlds provided on
MINIGRID CChealler-Booisee et al. (2118) as units of experience. As the learning algorithm, we trained for 500, 000 steps a PPO Schulman et al. (2117) agent, whose implementation we derived from CLEANRL
al. (2022b).
Policy and actor-critic architecture, with shared backbone, as well as other PPO hyperparameters details are presented in Table 4. For the TSCL experiments, we leveraged Exp3S (Auer et al. 2003) implementation from Besson (2018)
with default hyperparameters a 10-5 and 0.05, as defined in Graves et al. (2117).
Hyperparameter
Value
Actor
Critic
optimizer learning-rate
ADAM Kingma & Ba 2015)
CONv2D(16, 2, 2) RELUO
0.0025
annealing num-steps
Yes
MAXPOOL2D(2, 2)
128
CONV2D(16, 32, 2, 2) RELUO CONV2D(16, 64, 2, 2) RELUO
total-timesteps
500, 000
seeds gamma
5
0.99
GAF-lambda
0.95
LINEAR(64, 64) TANH() LINEAR(61, 7)
LINEAR 64) TANH()
num-minibatches
update- epotces
4
LINEAR 1)
Yes
clip-value-loss
Yes
clip-coeff
0.2
entropy-coeff
0.01
vf-coeff
0.5
max- max-grad -rad-rra--ormm target-kl
0.5
No
Table 4: Details on the PPO hyperparameters (left) and actor-critic architecture (right) used in the MINIGRIDROOMS Chevalier
Boisvert et al. 2018) experiments. Policy and critic components, and optimizer, are provided by PYTORCH (Paszke et al. 2019).
Implementation and default hyperparameters are derived from CLEANRI (Huang et al. 2022b). These details remained constant throughout the rest of the experiments with MINIGRIDROOMS.
20

---

Published in Transactions on Machine Learning Research (09/2024)
A.3 Populations & Games.
Adversarial SIPD. In our more challenging sparse and iterated version of Prisoner's Dilemma, the end of 200
interactions, inspired by Axelrod's competition (Axelrod 1981). A win-draw-loss reward r {,1,0,1} is given to learning player if it beats fixed opponent. Opponents are drawn from a population of five well-known strategies: always cooperate, always defect, wiin-stay-lose-sviich tit-for-tat, and zero-determinant strategy (Axelrod 1981 Hilbe
0,
et al. 2013) Knight al. 202T). We trained for 500 episodes (or 100, 000 steps) PPO (Schulman et 2017) agent
adapted from CLEANRL Huang et (2022b) default implementation. Policy and actor-critic architecture, without shared backbone, well other PPO hyperparameters details are presented in Table For the TSCL experiments. leveraged Exp3S (Auer et al., 2003 implementation from Besson (2018) with default hyperparameters 10
and y 0.05, as defined in Graves et al. (2017).
Hyperparameter
Value
Actor
Critic
optimizer learning-rad rate annealing nun-steps
ADAM
Kingma & Ba 2015)
LINEAR(2, 64) ORTHOINIT() TANH() LINEAR(64, 64) ORTHOINIT() TANH()
LINEAR(2, 64)
0.0025
ORTHOINIT()
Yes
TANH()
128
LINEAR(64, 64)
timesteps
100, 000
ORTHOINIT
seeds gamma
5
TANH()
0.99
LINEAR 2)
LINEAR 1)
GAF-lombda
0.95
minibatches
epochs
udvantage-nore
Yes
clip-value-loss
Yes
clip-coeff
0.2
entropy-coeff
0.01
vf-coeff max-gru max-grad-norm
0.5
target-kl
No
Table 5: Details on the PPO hyperparameters (lefi) and actor-critic architecture (right) used in the ADVERARIAL-SIPD experiments. Policy and critic components, and the optimizer, are provided by PYTORCH (Paszke et al, 2019). Implementation and default hyperparameters are derived from CLEANRI. (Huang al. 20226). These details remained constant throughout the rest of the experiments.
21

---

Published in Transactions on Machine Learning Research (09/2024)
B Extended Experiments Results
B.1 Value-Proportional Curriculum
TwoRooms
FourRooms
Six SoxRooms
projection
projection
nowel taclirros
INtUrs
0.2-
5,03
nonaka irplax
rownle-dree dimpter
0.3-
5031
2020
100300
200000
400000
220000
330009
400000
1000c0
200300
2000c0
400000
502000
DO MINtCRIDRROwS
Cooperator
TitFortat
3.0
profection
3.0
stapley Singlex stapley sofmax fotrk
-0.2
0.8
projetiinn
recplt
retums
-0.1
roweles
shspleycmples
nowaksingg
-03
-1.
nowat-iimpter
100000
20300
00000
103000
60000 St.pp
10000
180000
Stop
WinLaoseSwitchh
ZD-Extortioner
projection
nonaki syere-solima
retums
shuploy sirra--iipp
projection
shubloy
tholey-dmppes
mnassuttrox
-1.0
mosecammple
-1.0
20000
43030
snom
20565
60:03
acoco
100020
(U ATVERSSARIALSSPP
Figure 5: We also investigated the prior-proportion curriculum in the targer-unit setting. For each target unit, we allocate to each training unit interactions proportional to their pre-computed values for each target. For the ADVERSARIAL-SID and ROOMS controlled their learning dynamics by presenting the units according to anordered and ordered mechanisms in Sec. 5] On each task, the paote-proportional curriculum derived from the prospect priot outperforms TSCL (Iscl- *xxx3s)). We further investigate the reason for TSCL failures on this scenario.
MINGRRID-ROOMS
22

---

Published in Transactions on Machine Learning Research (09/2024)
B.2 TSCL Failures
Conperator
Defector
0.24
0.05
D:04
0.00
0-02
0.52
0.00
-6:22
-0.56
144
-0.4
50000
50002
22000
$0000
500000
150000
20200
50000
102000
WinLoaseSwitch
Extortioner
M
0.050
3.25
W
1
2.00
50022
220002
03000
100000
(3) Incisidted Kum
SEVE
3405
E:12-6
Debittr
0.038
3,005S
Lekiaan
33332
-002
60112
002222
2AU5
40220
En-Lor
Doser
3770
14X4
Teaam 4,2346
E3338
00544
32785
- 0133
6.3223
35435
23354
0,122
Zi-berioonr 4:337T
00531
00000
3389
1337
0:290
53324
03133
6/%.9/
(b) XPOP METRIC
Figure 6: To understand the failure modes of TSCL on ADVERSARIAL-SIPD we represented the individual runs (i.e., each of five seeds) on every target unit. TSCL (tscl- 3sxx33) (top row) extremely brittle, unstable, and generally not robust units interference. We surmise that these failures are related to the exploration negatively-valued unit are hard to overcome (forgetting dynamics). This issue requires further investigation, and we defer it to future work.
dilemma. Exploratory steps presenting
23

---

Published in Transactions on Machine Learning Research (09/2024)
TwoRooms
FourRooms
SixRooms
5.010
0,022
0.04
0,013
5.005
0.02
5,000
5eed
0.00
0.02
-0.4
200000
400000
100500
t00bec
402003
502000
units (aria]
units
Syhonms
Sybowi
01022
-2.817
9:6929
6.0091
-0.535
Taa-Roomo
0.0002
0.9277
+ 01111
wnkk 5999
Feutherrs
.0,050L
0.0309
5A217
+3,0110
6,0091
10 0.00899
0.0000
0.0277
00013
Sofoerra
-0.136
0.0011
-S.817
Siatcoraa
+3 330010
.5.0733
-0.0535
Stuhooma
0.0000
-0.023 0.0223
-3 0131
(6) MINiCHIRRoOsS
Figure 7: We found that TSCL presents a similar problem in MINIGRIDROOMS. When actions (units) need to be almost deterministi- cally drawn several steps, and other actions (units) have negative interference target, TSCL unable find a stable and robust solution to the roblem.
24


Reviews:
[{"review_id": "Lgr7mXiw3i", "reviewer": "Reviewer_TvC9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The key strength of the work is that this appears to be a novel application of game-theoretic techniques to problems in machine learning, demonstrating a way to interpret concepts from machine learning using the language and formalisms of game theory.\n\nThe primary weakness is that the framework appears to be too broad, and insufficiently specific, for deriving actionable insights for analyzing the domains presented here. It would be very useful for the authors to formulate specific hypotheses or claims that can then be subsequently proven or disproven empirically. As currently formulated, it is not clear what insights can be drawn from the experiments.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper describes an approach for using game theory to study Teacher-Student Curriculum Learning (TSCL) and leveraging tools from game theory, as well as incorporating game-theoretic considerations, to understand various problems in machine learning, ranging from supervised learning to reinforcement learning. A number of empirical computational experiments with these domains are conducted in applying the game-theoretic approach.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The empirical evaluation is difficult to follow. As described in the Strengths and Weaknesses section, it is not clear what the hypothesis that is being tested in this paper is, and how it could be proven or disproven. For example, the Reinforcement Learning section refers to folk knowledge about the existence of an optimal curriculum, and appears to show estimated allocation values that \"match folk knowledge\". It's not clear what this means concretely and what insights one can draw from these experiments. Improving the clarity of the experimental evaluation is critical for acceptance.\n\nThere are a number of typographical errors, for example:\n- beginning of section 2.2: \"Multi-armed bandit algorithms...uncertainty\" is missing a word, perhaps something like \"involving\"; \"about with\" should be \"about which\"\n- bottom of page 7: \"we We\" should be \"we\"", "broader_impact_concerns": "There do not appear to be any broader impact concerns that would need to be addressed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E1zlN8by08", "reviewer": "Reviewer_wosX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper makes an interesting and natural connection between two different lines of work: Teacher-Student Curriculum Learning (TSCL) and cooperative game theory. To my knowledge, this connection is new, and certainly brings a number of interesting insights to TSCL. It is notable that the results in the paper are not just based on standard ideas from cooperative game theory, such as the Shapley value, but they also use more advanced ideas. For example, the paper of Nowak and Radzik which addresses the problem of order in the coalition formation, not present in the traditional formulation of Shapley; as the authors demonstrate experimentally, that refinement turns out to be particularly crucial in the context of TSCL, in which the order (in terms of units of experience controller by the teacher algorithm) is expected to be make a difference. Another interesting concept leveraged is the measure of interaction, which is used to quantify the pairwise interaction between the units. I found that measure particularly informative in the experiments. So, the paper makes a concrete contribution and provides a number of new insights to an important problem. I also found that the experiments were very interesting. covering a broad range of applications. The paper is also very well-written and organized, and the related work section is quite thorough, discussing many related lines of work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of Teacher-Student Curriculum Learning (TSCL) using tools and ideas from cooperative game theory, which have been used in the past for feature attribution and explainability. In particular, the paper provides a game-theoretic interpretation to many important concepts of TSCL using the notion of units of experience, which are controller by the teacher algorithm; the cooperation between the units of experience is precisely what drives the connection with cooperative game theory. The cooperation between the units of experience is evaluated experimentally in different problems, ranging from supervised learning, reinforcement learning, and game theory.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some minor comments for the authors, which do not have a bearing in the evaluation of the paper:\n\n- Last paragraph in Page 7, there is a typo: \"We we\"\n- Footnotes should come after punctuation marks\n- In the concluding section, the authors discuss about the possibility of using super(sub)modularity in discrete combinatorial optimization. I am not able to follow what the authors mean here. Perhaps they can elaborate further.\n- How do the authors bypass the fact that many of the solution concepts used in the experiments are NP-hard to compute? Is some approximation used, or do they somehow solve the problem to optimality?", "broader_impact_concerns": "No concerns regarding ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "C0cn2iv2Jr", "reviewer": "Reviewer_UkMA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The authors provide helpful illustrations in the figures that support understanding the concepts.\n\n2. This paper extensively reexamines TSCL through the lens of cooperative game theory. The experiments covering supervised learning, reinforcement learning, and classical games provide insights how TSCL can be applied in machine learning.\n\n**Weaknesses**\n\n1. In general, the conducted experiments are restricted to toy datasets (e.g., MINIST and CIFAR10 in the supervised classification setting). The backbones are limited to 2-3 layers convolutional neural networks. It is not clear if TSCL still remains effective for complicated tasks.\n\n2. For each problem, the proposed framework relies on a priori value to find a curricula (e.g., the model's confusion matrix for MINIST). While such priori value is effective, the computation cost can be very expensive with large-scale training samples.\n\n3. Based on the above observations, the TSCL seems to be a post-training learning algorithm for well trained models. It may contain issues such as forgetting dynamics. I am wondering whether authors have any ideas to address such issues.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "To understand the conditions under which teacher-student curriculum learning (TSCL) is effective, this paper proposes a data-centric perspective to analyze the underlying mechanics of the teacher-student interactions in TSCL. Authors leverage cooperative game theory to describe how the composition of the set of experiences and orders presented by the teacher to the learner influences the performance of the curriculum. Authors further demonstrate that for every TSCL problem, there exists an equivalent cooperative game. Extensive experiments demonstrates the cooperative values of experiences and use value-proportional curriculum mechanisms to construct curricula.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses above, especially empirically or theoretically justifying the proposed concepts. More details and results on advanced tasks are needed.", "broader_impact_concerns": "The reviewer does not see any major ethical or moral concerns regarding the method or dataset used in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that the paper has met the criteria for acceptance and state that their questions and concerns regarding the initial submission have been addressed in the revised manuscript.
Decision: Accept (poster)