Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks
Authors: Luise Ge
Michael Lanier
Anindya Sarkar
Bengisu Guresti
Chongjie Zhang
Yevgeniy Vorobeychik
Venue: ICML 2025 poster (2025)
Keywords: Policy Committee; Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: We build small committees of policies to handle diverse tasks. Each task is matched with a policy that performs nearly optimally. This ensures fairness by helping most users do well.

Abstract: Many dynamic decision problems, such as robotic control, involve a series of tasks, many of which are unknown at training time.
Typical approaches for these problems, such as multi-task and meta reinforcement learning, do not generalize well when the tasks are diverse. On the other hand, approaches that aim to tackle task diversity, such as using task embedding as policy context and task clustering, typically lack performance guarantees and require a large number of training tasks. To address these challenges, we propose a novel approach for learning a policy committee that includes at least one near-optimal policy with high probability for tasks encountered during execution. While we show that this problem is in general inapproximable, we present two practical algorithmic solutions.
The first yields provable approximation and task sample complexity guarantees when tasks are low-dimensional (the best we can do due to inapproximability), whereas the second is a general and practical gradient-based approach. In addition, we provide a provable sample complexity bound for few-shot learning. Our experiments on MuJoCo and Meta-World show that the proposed approach outperforms state-of-the-art multi-task, meta-, and task clustering baselines in training, generalization, and few-shot learning, often by a large margin. Our code is available at https://github.com/CERL-WUSTL/PACMAN.

Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks
Luise Ge
Michael Lanier
Anindya Sarkar! Bengisu Guresti Chongjie Zhang'
Yevgeniy Vorobeychik
Abstract
crap, 2015) to game playing (Xu et al., 2018). However, many real-world applications involve highly diverse sets of tasks, making it impractical to rely on single, fixed policy. In these settings, both the reward structures and the transition dynamics can vary significantly across tasks, Existing approaches, such as multi-task RL (MTRL) and meta-reinforcemeet learning (meta-RL), struggle to general- ize effectively for diverse and previously unseen tasks.
Many dynamic decision problems, such as robotic control, involve series of tasks, many of which are unknown at training time. Typical approaches for these problems, such as multi-task and meta- reinforcement learning, do not generalize well when the tasks are diverse. On the other hand, approaches that aim to tackle task diversity, such as using task embedding as policy context and task clustering, typically lack performance guar- antees and require a large number of training tasks. To address these challenges, we propose novel approach for learning policy committee that includes at least one near-optimal policy with high probability for tasks encountered during ex- ecution. While we show that this problem is in general inapproximable, we present two practical algorithmic solutions. The first yields provable approximation and task sample complexity guar- antees when tasks are low-dimensional (the best we can do due to inapproximability) whereas the second is a general and practical gradient-bar based approach. addition, we provide a provable sample complexity bound for few-shot learning. Our experiments on MuJoCo and Meta-World show that the proposed approach outperforms state-of-the-art multi-task, meta-, and task clus- tering baselines in training, generalization, and few-shot learning, often by a large margin. Our code is available at https: / /t////// /ggtttb..cccoo/// com/ SESL-WUUSL/PACMAAA
Multi-task RL methods typically train single policy or shared representation across tasks (Vithayathil Varghese & Mahmoud, 2020). However, they often face negative trans- fer, where optimizing for one task degrades performance on others (Zhang et al., 2022). On the other hand, meta- RL approaches, such as Model-Agnostic Meta-Learning (MAML) (Finn al., 2017b) and PEARL (Rakelly et al., 2019), aim to enable fast adaptation new tasks but rely heavily on fine-tuning at test time, which can be computa- tionally expensive and ineffective in environments with high variability in both rewards and transitions, like Meta-Worke benchmark tasks (Yu et al., 2020b).
Several lines of work attempt to address the problem of diverse tasks and negative transfer. The first is to learn poli- cies through MTRL or meta-RL. that explicitly take task representation as input (Lan et al., 2024; Grigsby et al., 2024; Sodhani et al., 2021; Zintgraf et al., 2020). However, such approaches rely on shared parameters, which limit the model's flexibility. When tasks are highly diverse, using a single neural network- even with multi-head architec- ture- may not generalize effectively. Moreover, the greater the task variation, the more data is typically required to train policies that can effectively condition on task embeddings, and the model will often generalize poorly when the number of training tasks is small.
1. Introduction
Several alternative methods have thus emerged that propose clustering tasks and training distinct policies for each clus- ter (Ackermann et al., 2021; Ivanov & Ben-Porat 2024). The associated algorithmic approaches commonly leverage EM-style methods that interleave RL and task clustering. In practice, however, they also require large number of train- ing tasks to be effective and consequently perform poorly when the number of training tasks small. In addition, if the number of clusters too small, some clusters may still
Reinforcement learning (RL) has achieved remarkable suc- cess in variety of domains, from robotic control (Lilli-
'Deparmeett of Computer Science & Engineering, Wash- ington University St. Louis. Correspondence to: Luise Ge <g.luise@ wustl.edu>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Learning Policy Committees with Diverse Tasks
Theoretically Grounded Framework for Learning Pol- icy Committees: We present the first approach for learn- ing policy committees in MTRL and meta-RL domains that offers provable performance guarantees, in addition to practical algorithms. In particular, we provide sample efficient generalization (Theorem 3.7) and few-shot adap- tation guarantees that are independent of the size of the state or action space (Theorem 3.8).
Task: Drawer Close
Task: Door Open
Average Reward
Simple and Effective Policy Committees: PACMAN is design to handle high task diversity while using a small number of policies to achieve high efficacy. Additionally, our approach leverages LLM-based task embeddings for non-parametric tasks, which provides highly general and scalable solution for broad array of environments.
SOTA peliy
Commisee memberr
Commitse members
Commitee members
Policy acccomplish Doer Open taok Best Policy to accomplish Drawer Close task
Figure 1. Performance on a single task across committee members compared to .TRLL policy.
We demonstrate the efficacy of PACMAN through extensive experiments on challenging multi-task benchmarks, including MuJoCo and Meta- World. Our policy committee framework consistently outperforms tfate-of-the-ar multi-task RL, meta-RL, and task clustering baselines in both zero-shot (MTRL) and few-shot (meta-RL) settings, achieving better generaliza- tion and faster adaptation across diverse tasks.
Empirical Validation:
contain too diverse a set of tasks, with concomitant negative transfer remaining a significant issue.
We propose PACMAN, novel framework and algorithmic approach for learning policy committees that enables (a) sample efficient generalization (Theorem 3.7), (b) few-shot adaptation guarantees that are independent of the dimension of the state or action space (Theorem 3.8), and (c) signif- icant improvements in performance compared to 11 state- of-the-art baselines on both MTRL and few-shot adaptation metrics (Section 4). The key idea behind PACMAN is to lever- age parametric task representation which allows clustering tasks in the parameter space first, followed by reinforcement learning for each cluster. The benefit of our approach is illustrated in Figure 1: even though the single-policy base- line uses a -fxture-of-expprrt approach (Hendawy et al., 2024), its performance is poor since tasks require fundamen- tally distinct skills. In contrast, learning policy committee allows learning custom policies for distinct classes of tasks.
Our work is closely related to three key areas within the broader reinforcement learning literature: multi-task RL, meta-RL, and personalized RL.
Related Work:
Multi-Tass RI. (MTRL)): The broad goal of MTRL ap- proaches is to leverage inter-task relationships to learn poli- cies that are effective across multiple tasks (Yang et al., 2020; Sodhani et al., 2021; Sun et al., 2022). An important challenge in MTRL is task interference. One class of ap- proaches aims to mitigate this issue through gradient align- ment techniques (Hessel al., 2019; Yu et al., 2020a). An alternative series of approaches addresses this issue from representation learning perspective, enabling the learning of policies that explicitly condition on task embeddings (Hen- dawy et al., 2024; Lan et al., 2024; Sodhani et al., 2021).
A crucial insight in our approach is to define the objective of clustering as obtaining high coverage of the (unknown) task distribution, as opposed to insisting on a full coverage- that is, achieving near-optimal performance on randomly drawn task with high probability. This enables to devise efficient algorithms with provable performance guarantees and polynomial task sample complexity, as well as obtain few-shot adaptation guarantees that depends only on the number of clusters, but not on the size of state and action space. Moreover, while many MTRL and meta-RL prob- lems of interest are non-parametric, we show that we can leverage high-level natural language task descriptions to ob- tain highly effective embeddings of tasks, akin to Sodhani et al. (2021), and Hendawy et al. (2024), but without requir- ing subtask decomposition. Consequently, our approach can be effectively applied to broad class of non-parametric MTRL and meta-RL RL domains as well.
However, most MTRL methods still face challenges when tasks are diverse, particularly when it comes to generalizing to previously unseen tasks.
Meta-RL: The goal of meta-RL is an ability to quickly adapt to unseen tasks-what we refer to as few-shot learning. Meta-RL methods can be categorized broadly into two cat- egories: (i) gradient-based and (ii) context-based (where context may include task-specific features). Gradient-based approaches focus on learning a shared initialization of a model across tasks that is explicitly trained to facilitate few-shot learning (Finn et al., 201 7b; Stadie et al., 2018 8; Mendonca et al., 2019; Zintgraf et al., 2019). However, they perform poorly in zero-shot generalization, and tend to require a large number of adaptation steps. Context- based methods learn a context representation and use as a policy
In summary, we make the following contributions:
2

---

Learning Policy Committees with Diverse Tasks
input (Bing et al., 2023; Gupta al., 2018; Duan et al., 2017; Lee et al., 2020a;b; 2023; Rakelly al., 2019). How- ever, these approaches often exhibit mode-seeking behavior and struggle to generalize, particularly when the number of
t E II that is effective.' ! Next, we formalize this problem.
Let V be the value of policy ι for a given task i.c.,
h Σ 7'r, (st, ai)lar = T((i)
training tasks is small. While some recent approaches, such
VT = E
as Bing et al. (2023), attempt to improve performance by using natural language task embedding, they still require a large number of training tasks to succeed.
where the expectation is with respect to To and p. Let P be a (possibly restricted, e.g., parametric) space of policies that we take to be exogenously specified. We define Vi = maX V as the optimal value for a task T, that is, the value of optimal policy for T.
Personalized RL: Ackermann et al. (2021) and Ivanov & Ben-Porat (2024) proposed addressing task diversity by clustering RL tasks and training a distinct policy for each cluster. Both use EM-style approaches to jointly cluster the tasks and learn a set of cluster- sluster-specific policies. Our key contribution is to leverage an explicit parametric task representation, and reformulate the objective as flexible coverage problem for unknown distribution of tasks. This enables us to achieve task sample efficiency both in theory and practice. In particular, we empirically show that our PACMAN method significantly outperforms the state-of-the- art personalized RL approach.
Define V = max=en V/", that is, we let the value of a committee II to a task T be the value of the best policy in the committee for this task. There are a number of reasons why this evaluation of a committee is reasonable. As an example, if a policy implements responses to prompts for conversational agents and II is small, we can present multi- ple responses if there is significant semantic disagreement among them, and let the user choose the most appropriate. In control settings, we can rely on domain experts who can use additional semantic information associated with each т E II and the tasks, such as the descriptions of tasks N was effective for at training time, and similar descriptions to rest-time tasks, to choose a policy. Moreover, as we show in Section 3.4, this framework leads naturally to effective few-shot adaptation, which requires neither user nor expert imput to determine the best policy.
Reward-Free RL: Another related line of research is reward- free RL where the agent's goal is to first explore the MDP without pre-specified reward function and then plan near- optimal policies for set of given reward functions (Agarwal
et al., 2023; Cheng et al., 2022: Jin et al., 2020). However, results in this space make strong assumptions, such as as- suming that the action space is finite or linear dynamics. Our results, on the other hand, make no assumptions on the dynamics and do not depend on the size or dimension of the state action space.
One way to define the value of a policy committee II with respect to given MT-MDP and FS-MT-MDP is, respec- tively, as VM = Erer [v:"] and VM, A En= V.!! The key problem with these learning goals is that when the set of tasks is highly diverse, different tasks can confound learning efficacy for one another. For example, suppose that we have five tasks corresponding to target velocities of 10, 12, 20, 22, 100, and the task succeeds if a policy imple- ments its target velocity sufficiently closely (say, within 1). If we either train single policy for all tasks, or divide them into two clusters, the outlier target velocity of 100 will con- found training for the others. More generally, if any policy is trained on set of tasks that require different skills (c.g.. cluster of tasks that includes outliers), the conflicting reward signals will cause negative transfer and poor performance.
2. Model
We consider the following general model of multi-task MDPs (MT-MDP). Suppose we have a dynamic environ- ment E (S,A, h, Y, p) where S is a state space, A an action space, h the decision horizon, y the discount factor, (T,r) in
and p the initial state distribution. Let a task which T is the transition model where asx.a) is proba- bility distribution over next state s' as function of current state-action pair (s, a) and r(s, a) the reward function. A Markov decision process (MDP) is thus a composition of the dynamic environment and task, (E,
We address this limitation by defining the goal of policy committee learning differently. First, we formalize what it means for a committee II have a good policy for most of the tasks.
Let I be distribution over tasks т. We define a MT- MDP M as the tuple (E, (), as in typical meta-RL mod- els (Beck et al., 2023; Wang et al., 2024). Additionally, we define a simite-sample variant of MT-MDP, FS-MT-MDP, as M₁ = (E, ,,.,,,.., ), where T. ~ r. An FS-MT-MDP thus corresponds to multi-task& RL (Zhang& & Yang, 2021).
Definition 2.1. A policy committee IT is an (e, 1 - -cover for a task distribution I' if v > V - e with probability at least - 8 with respect to I. Il is an (e, 1 -)-ovr for
At the high level, our goal is to learn a committee of policies II such that for most tasks, there exists least one policy
Note that this also admits policies that can further depend on task embeddings, with this dependence heing different in different clusters.
3

---

Learning Policy Committees with Diverse Tasks
a set of lasks (fiy....TTl Tnl if V 2 V for least fraction 1 - 8 of tasks.
To this end, we propose the following high-level algorithmic framework in which we only need K independent (and, thus, entirely parallelizable) RL runs. This framework involves three steps:
Clearly, an (c, 1 5) cover need not exist for an arbitrary committee II (if the committee is too small to cover enough tasks sufficiently well). There are, however, three knobs that we can adjust: K, e, and 8. Next, we fix as exogenous, treating it effectively and suppose that K' is a pre-specified bound on the maxi- mum size of the committee.
1. SAMPLE n. tasks i.i.d. from r, obtaining T = {01,...,0,, On} (parameters of associated tasks
domain-specifie hyperparameter,
(T.....TT]). In MTRL settings, T is given.
2. CLUSTER the task set T into K subsets, each with an associated representative O2, and
Problem 1. Fix the maximum committee size K and c. Our goal is to find IT which is a (e, 1 - -coevr for the smallest ò E [0, 1].
3. TRAIN committee member &o E TT for each cluster l: represented by 0μ-
As we shall see presently (and demonstrate experimentally in both Subsection 4.3 and Appendix H.2), conventional clustering approaches are not ideally suited for our problem. We thus propose several alternative approaches which yield theoretical guarantees on the quality of II, in the special case that each committee member represents one policy.
As a corollary to Theorem 2.6 in Skalse & Abbte (2023), we now present sufficient conditions under which a policy committee can yield higher expected rewards than a single policy. The precise definition of these conditions is provided in Appendix A.
Corollary 2.2 (Theorem 2.6, (Skalse & Abate, 2023)). Sup- pose K > 1. Unless two tasks T1. T2 have only their reward functions r1 and 12 differ by potential shaping, positive linear scaling, and Si-redistribution, we have Vii 2 V V N V.S and V + +V-I -V, + Vo, for any single policy π.
Empirically, we show that the proposed framework outper- forms state of the art even when tasks also have distinct transition distributions.
T2
F2
3.2. Clustering
The key aspect of our algorithmic design is clustering. We are motivated by connection between the clustering step (step (2) of the framework above) and efficacy of optimal policies learned for each cluster (step (3) of the framework), as a variant of simulation lemma (Lobel & Parr, 2024), whose formal implications are discussed in the next section. By leveraging a strong representation space, we can avoid the overhead of evaluating and clustering tasks on-policy and achieve high efficacy with minimal computational cost. Thus, we focus on the following problem instead:
Next, we present algorithmic approaches for this problem. Subsequently, Section 3.4, as well as our experimental re- sults, vindicate our choice of this objective.
3. Algorithms for Learning a Policy Committee
In this section, we present algorithmic approaches for train committee members to solve Problems 1. We consider the special case of the problem in which the tasks have structure representation. Specifically,
assume that each task can be represented using a parametric model v'e(s, a). where the parameters 0 E I comprise both of the param- eters of the transition distribution T and reward function r. Often, parametric task representation is given or direct; in cases when tasks are non-parametric, such as the Meta- World (Yu et al., 2020b), we can often use approaches for task embedding, such as LLM-based task representations (see Section 3.6). Consequently. we identify tasks with their representation parameters 0 throughout, and overload I' to mean the distribution over task parameters, i.e., 0 ~ I.
Definition 3.1. A set of representatives C = {(1..... ,0k is an (c, 5)-parameter-cove for task distribution I if mingreo |0 0' =0 S e with probability at least 1 6 with respect to 0 ~ T.
Problem 2. Fix K and e. Our goal is to find C with |C| S
K which is a (e, - poreeeeeeetrrrrr for the smallest 8 [0, 1].
Notably, while conventional clustering techniques, such as k-means, can be viewed as proxies for these objectives, there are clear differences insofar as the typical goal to minimize sum of shortest distances of all vectors from cluster representatives, whereas our goal, essentially, is to "cover" as many vectors as we can. In Appendix H.2, we provide histogram of individual task returns to illustrate the different impact the two clustering methods make.
3.1. A High-Level Algorithmic Framework
Even conventional RL presents a practical challenge in com- plex problems, as learning is typically time consuming and requires extensive hyperparameter tuning. Consequently, a crucial consideration in algorithm design is to minimize the number of RL runs we need obtain policy committee.
We show next that our problem is strongly inapproximable, even if we restrict attention to K
4

---

Learning Policy Committees with Diverse Tasks
Definition 3.2 (MAX-I-COVER) Let = (01.-.., 0n) S Rd. Find O Rd which maximizes the size of S C T with
follows. The first stage of the algorithm is to create an intersection tree for each dimension independently. For s-th dimension, we sort the datapoints' s-th coordinates in ascending order. We refer to the sorted coordinates as 21 < I22 ... < In. and create a list for each point x; to remember how many other points can be covered together with it with initialization being [xx]] itself.
maxerss |10 00 S.
Theorem 3.3. For any constant > O. MAX-I-COVER does not admit an n -approximation unless P NP.
We prove this in Appendix B via an approximation- preserving reduction from the Maximum Clique prob- lem (Engebretsen & Holmerin, 2000). Despite this strong negative result, we next design two effective algorithmic approaches. The first method runs in polynomial time with a constant d, and provides a constant factor approximation. The second is general gradient-based approach.
Starting from the second smallest datapoint 22. we check if 22 S x1 +c, L.. if 22 21 + 2c. Since x2- 21 - due to our sorting, any point inside [12 e, +e] can cover both I1, 22. Therefore if this interval is valid, we add æ1 to the list [22] to indicate the existence of a simultaneous coverage for 21, 22. In general, for Ii, we check if x; S ky++2r with a descending j = i-1 to 1 or until the condition no longer holds. If the inequality is satisfied, we add xy to 24 's list. Then since have ordered the set, for every index j' less than j. 2, > 2i + 2e > Iyy + 2e. The coverage for all the x in v, 's list would be the interval [a:; C, 2y + c, where j is the smallest index in x, 's list. There are 1 + 2 + + n - = O(?)) comparisons in total. We form a set of these lists, and call it A, for the s-th dimension. The figure above illustrates how the algorithm works find out A = [[n)), [x2), [Ir 22, wa), [114, #5]]-
Greedy Elimination Algorithm (GEA) Before we dis- cuss our main algorithmic approaches, we begin with GEA, which provides a useful building block, but not theoretical guarantees. Consider set 7' of task parameter vectors, fix K, and suppose we wish to identify an (c, 1 - 8) parameeer- cover with the smallest 8 (Problem 2). but restrict attention to 6 E T in constructing such a cover. This problem is an instance of MAX-K-COVER problem (where subsets correspond sets covered by each 9 E 7), and can be approximated using& greedy algorithm which iteratively adds one 0 E T to C' that maximizes the most uncovered vectors in T. Its fixed-ō variant, on the other hand, is a set cover problem if 8 = 0, and a similar greedy algorithm approximates the minimum-A K cover C for any 6. However, neither of these algorithms achieves a reasonable approxi- mation guarantee (as we can anticipate from Theorem 3.3), although our experiments show that greedy elimination is nevertheless an effective heuristic. But, as we show next, we can do better if we allow cluster covers 0 to be unrestricted.
The second stage is to find a hypercube covering the most points, consisting of an axis from each dimension. By the ge- ometry of the Euclidean space, two points 0,, 02 are within in loo -distance iff they appear inside one's list together for each dimension. Therefore, in order to find the maxi- mum coverage with one hypereube such that its center is within loo -is-ttnne the most points, we wonder which combination of lists, 41..44 la each from the sets AI... Aa produces an intersection of the maximum cardinality. In our example, we can conclude that [#1, 32, æa), [It,, ærs] need to be covered separately by two points between the blue or red vertical lines.
Greedy Intersection Algorithm (GIA) We now present our algorithm which yields provable approximation guaran- tees when task dimension is low. The key intuition behind GIA is that for any 6, c-hybercube centered at 0 char- acterizes all possible 0' that can cover 0 in the sense of Definition 3.1. Thus, if any pair of e-hypercubes centered at f and 8 intersects, any point at the intersection covers both. To illustrate, consider the following simple example:
The full algorithm is provided in Appendix C. Next, we show that GIA yields provable coverage guarantees. We defer the proof to Appendix D. For these results we use GIA(K) to refer to the solution (set C' = [(,....,0k] Okl) returned by GIA.
Theorem 3.4. Suppose T contains
2log(5/a)
i.i.d. sam-
[13, [a] [21, 22, 23] [11. T2]
6
X
I4 +
from I. Let 1 - 8* (K, e) be the optimal coverage of a c-parameter-covel of T under fixed K, c. Then with proba- bility at least a-, GIA(K) is (c, (1-1111))) -2(( 8= (K,e)- KB3) parameteer-ccoeerrrrrr of
2
F3
x
U3
+E
F2
22
4
[2]]
31
x
He
Each cross represents parameter we aim to cover, while each line segment indicates the possible locations of the e-close representative for that parameter. By selecting a point within the overlapping region of these intervals, we can effectively cover their parameters simultaneously.
The key limitation of GIA is that it is exponential in d, and thus requires the dimension to be constant. This is reason- able assumption in some settings, such as low-dimensional control. However, in many other settings, both n and d can be large. Our next algorithm addresses this issue.
The proposed GIA algorithm generalizes this intuition as
Gradient- Based Coverage Consider Problem 2. For a
5

---

Learning Policy Committees with Diverse Tasks
finite set T, we can formalize this as the following optimiza- tion problem:
See Appendix F for proof. Lipschitz continuity is mild assumption; for example, it is satisfied by ReLU neural networks.
max (01...,Okc
Σ
(imm
min |10, kE[K]
~, S sc),
(L)
Next, we combine Theorem 3.4 and Lemma 3.6 to conclude that we can compute a policy committee IT which provides provable coverage guarantees for a task distribution I' with polynomial sample complexity.
BET
where 1()) is if the condition is true and 0 otherwise. However, objective (1) is non-convex and discontinuous. To address this, we propose the following differentiable proxy:
Theorem 3.7. Suppose T contains 2 9log(5/a) i.i.d. sam- 232
ples from I and re(s,a) is L-Lipschitz in Loo- Let 1 - 8K, e) be the optimal coverage of a e-parameter- cover of T under fixed K, and suppose that for any task T we can find a policy π with V 2 V 17. Then we can compute a committee II which is a (2L 1-1
- )
min (9 0x. wER n
RELU
o (wsx))) -0ill00
(2)
1-- +
n, (1 (((1-)(C (K,,) K3)) for I' when Y < 1 and
where is the assignment matrix mapping each 0g to 8% with unbounded weights and (.) is a softmax function to normalize the assignment. Next, we demonstrate that this is a principled proxy by showing that when full coverage of T is possible, solutions of (1) and (2) coincide. The proof is in Appendix E.
(2Lhc + (1)))))(1,,,, (K, KB)) ()))).oo for I Thhen y = 1 with probability at least 1 a.
3.4. Few-Shot Adaptation
A particularly useful consequence of learning policy com- mittee II that is a (e, - 5)-cover is that we can leverage it in meta-learning for few-shot adaptation. The algorithmic idea is straightforward: evaluate each of K policies in 11 by computing a sample average sum of rewards over N randomly initialized episodes, and choose the best policy E II in terms of empirical average reward.
Theorem 3.5. Fix K and suppose 30 E (Oi...., 0x} such that ||e - 0|ll I e for all i. Then the sets of optimal solutions to (1) and (2) are equivalent.
Thus, we can use gradient-based methods with objective in (2) to approximate solutions to Problem 2. Because the objective is still non-convex, we can improve performance by initializing with the solution obtained using GEA or GIA when d is low.
In particular, suppose that 7 1. We now show that this translates into a few-shot sample complexity on a previously unseen task 7 that is linear in K (the size of the committee). Details of the proof are in Appendix G.
3.3. Theoretical Analysis for MTRL
Theorem 3.8. Suppose II is a 5)-vver for I and T ~ Т. Under some mild conditions, if we run p 2 32M(H +1) 11))) episodes for each policy II, where (8 22122
We now put things together by showing that we have a sam- ple efficient approach for learning a policy committee II which achieves (c, 1 -cooer for I. For this result, we focus on the special case where each committee member is one fixed policy and assume that each task has a shared dynamics, and parametric reward function ra a) where 9 identifies a task-specific reward. While this is a theoretical limitation, we note that our subsequent clustering and train- ing algorithms do not in themselves require this assumption, and our experimental results demonstrate that the overall approach is effective generally.
H is a constant, the policy π that maximizes the empirical return yields V > V c 3 with probability at least - 8 8- a, where Vi is the optimal reward for T.
3.5. Training
The output of the CLUSTERING step above is set of repre- sentative task parameters C' = (O....., 0x) ). The simplest way to use these to obtain a policy committee II is to train a policy Ty optimized for each 0g E C. However, this ignores the set of tasks that comprise each cluster l: associated with a representative 0₂ (i.e., the set of tasks closest to 0s). As demonstrated empirically in the multi-task RL literature, us- ing multiple tasks to learn a shared representation facilitates generalization (effectively enabling the model to learn fea- tures that are beneficial to all tasks in the cluster) (Sodhani et al., 2021; Sun et al., 2022; Yang et al., 2020).
Let oं denote the optimal policy for task 7;. We use V." to denote the value of task T; using a policy that is optimal for task Tj.
Lemma 3.6. Suppose that ae(s,a) is L-Lipschitz in Loo norm, that is, for all 0,0', supsa (ro(s,a) a) - (s, 1 LI10- - 0'Iloo. Then, for any two tasks t, and Tú with re-
spective O2 and 03 that satisfy 110, - 0,11x S V.3 >
V. - -2 -- I-n e if < 1 and V 2 V 2Lhe if
To address this, we propose an alternative which trains a policy ne to maximize the sum of rewards of the tasks in
7= =
6

---

Learning Policy Committees with Diverse Tasks
cluster k. Notably, our approach can use any RL algorithm to learn a policy associated with a cluster of tasks; in the experiments below, we use the most effective MTRL or meta-RL baseline for this purpose.
to run different velocities, and rewards are based on the distance to target velocity. The second is HumanoidDir where the agent has to move along the preferred direction, and the reward is the distance to the target direction. In both, we generate diverse rewards by randomly generating target velocity and direction, respectively, and use 100 tasks for training and another 100 for testing (in both zero-shot and few-shot settings), with parameters generated from a Gaus- sian mixture model with 5 Gaussians. In few-shot cases, we draw a single task for fine-tuning, and average the result over 10 tasks. For clustering, we use K = 3, = .6. and use the gradient-based approach initialized with the result of the Greedy Intersection algorithm. For few-shot learning, we fine-tune all methods for 100 epochs.
Furthermore, we emphasize that the additional overhead in- troduced by our method (due to clustering) is small, with the computational complexity in practice being predominantly determined by the RL problem. To illustrate, our Meta- World experiments show that training a single policy for million steps necessitates approximately 40 hours using A40 GPU. Conversely, the clustering process completes in roughly 1 second within a Google Colab notebook, and ob- taining task embeddings takes approximately 2 minutes on an A40 GPU. Importantly, in cases where efficient RI. learn- ing is achievable (Brafman & Tennenholtz, 2002; Kearns & Singh, 2002) and the dimension d remains constant, our approach additionally boasts polynomial algorithmic com- plexity.
Meta- World We focus on the set of robotic manipulation tasks in MT50, of which use 30 for training and 20 for testing. This makes the learning problem significantly more challenging than typical in prior MTRL and meta-RL work, where training sets are much larger compared to test sets (5 tests and 40 trains in the traditional MT45 setting). We lever- age an LLM to generate a parameterization (Section 3.6) of the task. Specifically, text descriptions (see Appendix are fed "Phi-3 Mini-128k Instruct" (Microsoft, 2024) and we compute the channel-wist mean over the features of penultimate layer as a 50 dimensional parameterization for each task. We use K- = 3 and e = .7.
3.6. Dealing with Non-Parametric Tasks
Our approach assumes that tasks are parametric, so that we can reason (particularly in the clustering step) about param- eter similarity. Many practical multi-task settings, however, are non- non-oonnnann pora-mmettrrc, so that our algorithmic framework can- such cases, our approach can
not be applied directly. make use of any available method for extracting a paramet- ric representation of an arbitrary task 7. For example, it is often the case that tasks can be either described in natural language. We propose to leverage this property and use text embedding (e.g., from pretrained LLMs) as the parametric representation of otherwise non-pann parrameetrii tasks, where this is feasible. Our hypothesis is that this embedding captures the most relevant semantic aspects of many tasks in practice, a hypothesis that our results below validate in the context of the Meta-War World benchmark. This is analogous to what was done by Bing et al. (2023), with the main difference being that our task descriptions are with respect to higher-level goals, whereas Bing et al. (2023) describe tasks in terms of associated plans. We provide the full list of task descriptions for the Meta- World environment in Appendix I.
Additionally, we highlight that we use success rate instead of returns as our evaluation metric in Meta- Morad, as has been the standard metric across different papers.
4.1. Baselines and Evaluation
We compare our approaches to 11 state-of of-the-rtt baselines. Five of these are designed for MTRL: 1) CMTA (Lan et al., 2024), 2) MOORE (Hendawy et al., 2024), 3) CARE (Sod- hani et al., 2021), 4) soft modularization (Soft) (Yang et al., 2020), and 5) Multi task SAC (Yu et al., 2020a). Five more are meta-RL algorithms: 1) MAML (Finn et al., 2017a), 2) RL2 (Duan et al., 2017), 3) PEARL (Rakelly et al., 2019), 4) VariBAD (Zintgraf et al., 2020), and 5) AMAGO (Grigsby et al., 2024). Finally, we also compare to the sate-of-thh--a approach using inrectation-maiiiiaatt policy committee (EM) (Ivanov & Ben-Porat, 2024).
(EM) to learn
4. Experiments
We study the effectiveness of our approach-PACMMYY PACMAN in two environments, MuJoCo (Todorov et al., 2012) and Woold (Yu et al., 2020b). In the former, the tasks are low-dimensional and parametric, and we only vary the reward functions, whereas the latter has non-parametric robotic manipulation tasks with varying reward and transi- tion dynamics.
Our evaluation involves three settings: training, test, and few-shoz. The training evaluation corresponds to standard MTRL. The test evaluation uses a test set to evaluate all approaches with no fine-tuning. Finally, our few-shot test evaluation allows a short round of fine-tuning on the test data. For PACMAN select the hest-performing policy for training and test, and use the proposed few-shot approach to learn the best policy through empirical policy evaluation for the few-shot test setting (see Section 3.4). In all figures, error bounds are sample standard deviation.
MuJoCo We selected two commonly used MuJoCo envi- ronments. The first is HalfCheetah Vel where the agent has
7

---

Learning Policy Committees with Diverse Tasks
0.0
CVTL
PRCORE
Succed Ratto
NITE
ANDOC
NAN.
0.0.2
Verbed
Number Trames
number frames
0.0
0.2
0.6
0.6 Stops
0.3
1e5
1.0
0.0
0.2
0.4
Steps
0.5
Figure 2. HalfCheetah Vel train (left) and test (right) comparisons.
Figure 3. MetaWorld train (left) and test (right) comparisons.
4.2. Results
Table 2. Few-Shot Learning Results.
Method
6k Updates
Updates
MuJoCo In the MuloCo environment, focus person- alization, varying only reward functions and focusing the ability to generalize to a diverse set of rewards. Conse- quently, our baselines here include meta-RL approaches (RL2, VariBAD, AMAGO) and EM (personalized RL, which requires the dynamics to be shared across tasks, there- fore not applicable for Meta- Ssta-World), and PACMAN uses VariBAD the within-cluster RL method.
MAML
0.0025 + 0.006
0.01 = 0.03
PEARL
0.03 0.03
0.27 0.07
RL2
0.007 + 0.01
0.02 000::0002
VariBAD
0,025 + 0.06
0.027 + 0.07
AMAGO
0.08
.033
Soft
0.27 0.07
0.26 0.08
MTTE
0.37 - 0.08
0.40 0.10
CARE
0.39 1: 0.05
0.40 0.06
CMTA
0.45 1 0.07
0.34 0.08
MOORE
0.41 + 0.08
0.44 0.11
PACMAN
0.53 = 0.02
0,60 = 0.02
Figure 2 presents the MuJoCo results for the training and test evaluations in HalfCheetah We can see that PACMAN consistently outperforms the baselines in both evaluations, with VariBAD the only competitive baseline. The advantage of PACMAN also pronounced in HumanoidDir, whose results are deferred to Appendix H.
2000 evaluation episodes over 3 seeds. Here, the advantage of PACMAN over all baselines is particularly notable. First, somewhat surprisingly, the meta-RL baselines, with the exception of PEARL, underperform MTRL baselines in this setting. This is because our evaluation is significantly more challenging, with only 30 training tasks but with 20 diverse test tasks, and the adaptation phase has a very short (6- 66-12K updates) time horizon for few-shot training, than typical in prior work. In contrast, MTRL methods fare reasonably well. The proposed PACMAN approach, however, significantly outperforms all the baselines. For example, only 12K updates suffice to reliably identify the best policy (comparing with zero-shot results in Table 2), with the result outperforming the best baseline by >36%.
Table Few-shot learning effectiveness (MuJoCo).
Halfcheetah
Humanoid
RL,
-314.37 + 1.15
946.17 0.73
VariBAD
-137.99 + 1.14
1706.38 + 0.75
EM
-325.29 + 1.84
947.06 + 0.84
Amago
-279.13 0.67
1533.45 0.49
PACMAN
-54.03 + 1.34
2086.50 + 0.89
The few-shot comparison is provided in Table 1, where the advantage of PACMAN is especially notable. In Halfchee- tah Vel, the improvement over the best baseline is by a factor of more than 2.5 while in HumanoidDir it is over 22%.
4.3. Further Empirical Investigation of Our Algorithm
We investigate our algorithmic contribution in two ways. First, we compare our method with three common clustering methods in Meta-Worlde zero-shot setting: KMeans++ (Arthur & Vassilvitskii, 2006), DBScan (Khan et al., 2014), GMM (Bishop, 2006), well as with random clustering.
Meta-World Next, turn to the complex multi-task Meta- World environment. In this environment, our approach uses MOORE for within-cluster training. Figure 3 presents the results for training and test cvaluations, where we compare to the MTRL baselines (all meta-RL. baselines are signifi- cantly worse on these metrics, likely because the goals of these algorithms are primarily efficacy in seot settings).
Method
125K Steps
250K Steps
KMeans++
0.22 = 0.06
0.30 + 0.07
DBScan
0.19 + 0.04
0.28 + 0.06
GMM
0.29 + 0.07
0.33 + 0.08
Random
0.22 + 0.04
0.25 + 0.04
We observe that PACMAN significantly outperforms all base- lines in both train and test cases ~55% improvement over the best baseline after 500K steps).
PACMAN (Ours)
0,36 1 0.07
0.48 1 0.09
Table 3. Performance Comparison for different clustering methods at 125K and 250K Steps.
The results for few-shot learning are provided in Table 2. Performance is a moving average success rate for the last
Notably, PACMAN exhibits 45% performance improve-
8

---

Learning Policy Committees with Diverse Tasks
ment over the next best clustering method for K = 3. We have also conducted the same ablations over the MuloCo set- ting, where the advantage of our method is also significant; see Table 4 in Appendix H.2 for more details. In addition, Figure 5 in Appendix H.2 compares a histogram of rewards for PACMAN and KMeans++ in the MuJoCo Halfcheetah environment, showing a consistent distributional improve- ment (not merely in expectation), excepting small number of outliers.
Acknowledgements
This work was supported by the National Science Founda- tion (NSF) under grants IIS-221414] and CCF-2403758, the Office of Naval Research (ONR) under award N00014- 24- 1-2663, the Army Research Office (ARO) under grant 199-5-225--000,,, and with the generous support of NVIDIA Corporation.
Impact Statement
Additionally, we consider the impact of varying budget K. We observe that efficacy of PACMAN is non-monotonic in K. The reason is that once K is sufficiently large to cover the entire set of training tasks, increasing it further reduces the number of training tasks in individual clusters and thereby hurts generalization, both within clusters and to previously unseen tasks. Finally, we also consider the impact of changing the hyperparameter e, and observe that the results are relatively robust to small changes in
This paper provides a theoretically grounded and empiri- cally validated framework for personalization in Markov Decision Processes (MDPs) through the construction of policy committees. The simplicity and modularity of our approach enable wide applicability across domains such as robotics, adaptive systems, and user-centered decision- making frameworks. Furthermore, our method inherently promotes faimess by ensuring that a diverse range of tasks are represented by at least one effective policy in the com- mittee. The clustering strategy may also extend beyond reinforcement learning to other domains where balancing efficiency and individualization is critical. By offering a scalable and provably efficient solution, our work lays a foundation for future advances in equitable and adaptive Al systems.
Overall, our method stands out by introducing only a single hyperparameter, e, which proves casy to tune. This simplic- ity is a significant advantage, especially when contrasted with the myriad of hyperparameters often encountered in typical deep RL methods. A practical approach to setting c is to first compute the distances between all task embed- dings and subsequently fine-tune it to ensure it remains small while still providing adequate coverage. Notably, our ablation studies further indicate that the performance of our method far less sensitive to compared to the typical hyperparameter sensitivity observed in RL. See Appendix H for further details these ablations.
References
Ackermann, J., Richter, O., and Wattenhofer, R. Unsuper- vised task clustering for multi-task reinforcement learn- ing. In European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 222-237, 2021.
5. Conclusion and Limitations
Agarwal, A., Song, Y., Sun, W., Wang, K., Wang, M., and Zhang, X. Provable benefits of representational transfer in reinforcement learning. In Conference on Learning Theory, pp. 21 2023.
We developed a general algorithmic framework for learning policy committees for effective generalization and few-shot learning in multi-task settings with diverse tasks that may be unknown training time. We showed that our approach is theoretically grounded, and outperforms MTRL, meta-Ra RL, and personalized RI. baselines in both training, and zero- shot and few-shot test evaluations, often by a large margin. Nevertheless, our approach exhibits several important limi- tations. First, it requires tasks to be parametric, and while we demonstrate how L.MMs can be used to effectively obtain task embeddings in the Meta-World environments, is not clear how do generally. Second, it includes scalar hyperparameter, which determines how we evaluate the quality of task coverage and needs to be adjusted separately for each environment, although this hyperparameter is easily tunable in practice.
Arthur, D. and Vassilvitskii, S. k-means++: The advantages of careful seeding. In Symposium on Discrete Algorithmis, 2006.
Azar, M. G., Lazaric, A., and Brunskill, E. Regret bounds for reinforcement learning with policy advice. In Joint Eu- ropean Conference on Machine Learning and Knowledge Discovery in Databases, pp. 97-1 Springer, 2013.
Beck, J., Vuorio, R., Liu, E, Z., Xiong, Z., Zintgraf, L., Finn, C., and Whiteson, S. A survey of reiforreinoree learning, arXiv preprint arXiv:2301.080288 2023.
Bing, Z., Koch, A., Yao, X., Huang, K., and Knoll, A. rentorreinf learning via language instructions. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5985-5991. IEFF, 2023.

---

Learning Policy Committees with Diverse Tasks
Bishop, C. M. Pattern recognition and machine learning, volume 4. Springer, 2006.
Jin, C., Krishnamurthy, A., Simchowitz, M., and Yu, T. Reward-free exploration for reinforcement learning. In International Conference on Machine Learning, pp. 4870- 4879, 2020.
Brafmann, R. I. and Tennenholtz, M. R-max-a general poly- nomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research, 3(Oct): 213-231, 2002.
Kearns, M. and Singh, S. Near-optimal reinforcement learn- ing in polynomial time. Machine learning, 49:209-232, 2002.
Cheng. Y., Feng. S., Yang, J., Zhang, H., and Liang, Y. Provable benefit of multitask representation learning in reinforcement learning. In Neural Information Processing 2022.
Khan, K., Rehman, S. U., Aziz, K.,, Fong. S., and Sarasvady, S. Dbscan: Past, present and future. In International Conference on the Applications of Digital Information and Web Technologies, pp. 232-238, 2014.
Systems, pp.
Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I,, and Abbeel, P. RI2: Fast reinforcement learning via slow reinforcement learning. In International Conference on Learning Representations. 2017.
Lan, S., Zhang, R., Yi, Q., Guo, J., Peng, S., Gao, Y., Wu, F., Chen, R., Du, Z., Hu, X., et al. Contrastive modules with temporal attention for multi-task reinforcement learning. Neural Information Processing Systems, 2024.
Engebretsen, L. and Holmerin, Clique is hard to ap- proximate within n! !--o(1) In International Colloquium Automata, Languages, and Programming, pp. 2-12. Springer, 2000.
Lee, A. X., Nagabandi, A., Abbeel, P., and Levine, S. Stochastic latent actor-critic: Deep reinforcement learn- ing with latent variable model. In Neural Information Processing Systems, pp. 741-752, 2020a.
Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks. CoRR, abs/1703.034400, 2017a. URL httphtttt//axi..orttttt
Lee, K., Seo, Y., Lee, S., Lee, H., and Shin, Context- aware dynamics model for generalization in model-based reinforcement learning. In International Conference on Machine Learning, pp- 5757-5766. PMLR, 2020b.
Lee, S., Cho, M., and Sung, Y. Parameterizing non- parametric meta-reinforcemeen learning tasks via sub- task decomposition. In Neural Information Processing Systems, pp. 43356 43383, 2023.
Finn, Abbeel, P., and Levine, S. Model-agnostic meta- learning for fast adaptation of deep networks, In Interna- tional Conference on Machine Learning, pp. 1126-1135, 2017b.
Lillicrap, T. Continuous control with deep reinforcement learning, arXiv preprint arXiv:1509.0200 2015.
Grigsby, J., Fan, L., and Zhu, Y. Amago: Scalable in-context reinforcement learning for adaptive agents, 2024. URL https: htaps:///axiiv org/ abs/ 2210 0 0 09911.
Lobel, S. and Parr, An optimal tightness bound for the simulation lemma. arXiv preprint arXiv:2406.1624 16249, 2024.
Gupta, A., Mendonca, R., Liu, Y., Abbeel, P., and Levine, S. Meta-reinforreement learning of structured exploration strategies. Advances in Neural Information Processing Systems, 31, 2018.
Mendonca, R., Gupta, A., Kralev, R., Abbeel, P., Levine, S., and Finn, C. Guided meta-policy search. Advances in Neural Information Processing Systems, 32, 2019.
Hendawy, A., Peters, J., and D' D'Eramo, C. Multi-task rein- forcement learning with mixture of orthogonal experts. In International Conference on Learning Representations, 2024.
Microsoft.
Phi-3 mini-128k instruct.
https:
wtt.c/ooocc..o..eoooe.oooooooooe 12k-kk2kkkknnttttt 2024. Accessed:
2024-09-23.
Hessel, M., Soyer, H., Espeholt, L.. Czarnecki, W., Schmitt, S., and Van Hasselt, H. Multi-task deep reinforcement learning with popart. In AAAI Conference on Artificial Intelligence, pp. 3796-3803, 2019.
Nemhauser, G. L., Wolsey, L. A., and Fisher, M. L. An analysis of approximations for maximizing submodular set functions Mathematical programming 14:265- 294, 1978.
Ivanov, D. and Pen-Po O. Personalized reinforcement learning with a budget of policies. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 12735- 12743, 2024.
Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. Efficient off-policy retf-reinforeemmet learning via prob- abilistic context variables. In International Conference on Machine Learning, pp. 5331-5340, 2019.
10

---

Learning Policy Committees with Diverse Tasks
Skalse, J. and Abate, A. Misspecification in inverse rein- forcement learning. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 37, pp. 15136- 15143, 2023.
Zhang, Y. and Yang, Q. A survey multi-task learning.
IEEE Transactions Knowledge and Data Engineering, 34(12)5586-5609, 2021.
Zintgraf, L., Shiarli, K., Kurin, V., Hofmann, K., and White- son, S. Fast context adaptation via meta-learning. In International Conference on Machine Learning, pp. 7693- 7702. PMLR, 2019.
Sodhani, S., Zhang, A., and Pineau, J. Multi-task rein- forcement learning with context-based representations. In International Conference on Machine Learning, pp- 9767-9779, 2021.
Zintgraf, I.., Shiarlis, K., Igl. M., Schulze, S., Gal, Y., Hof- mann, K., and Whiteson, S. Varibad: A very good method for bayes-adaptive deep via meta-learning In Interna- tional Conference Learning Representation, 2020.
Stadie, B.. Yang, G., Houthooft, R., Chen, P., Duan, Y.,
Wu, Y., Abbeel, P., and Sutskever, I. The importance of sampling in meta-reinforcement learning, Advances in Neural Information Processing Systems, 31, 2018.
Sun, L.,, Zhang, H., Xu, W., and Tomizuka, cormmteee-commposttonalll multi-task reinforcement learn- ing. In Neural Information Processing Systems, pp- 21495-21507, 2022.
Paco:
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-hased hased control. In 2012 IEEE/RS Inter- national Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Vithayathil Varghese, N. and Mahmoud, Q. H. A survey of multi-task deep reinforcement learning, Electronics, 9()): 1363, 2020.
Wang, C., Sui, M., Sun, D., Zhang, Z., and Zhou, Y. The- oretical analysis meta reinforcement learning: Gen- eralization bounds and convergence guarantees. arXiv preprint arXiv:2405. 2024.
Xu, Z., Hasselt, H. P., and Silver, D. Meta-gradient reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Yang, R., Xu, H., WU, Y., and Wang, X. Multi-task rein- forcement learning with soft modularization. Advances in Neural Information Processing Systems, 33:4767-4777 2020.
Yu, T., Kumar, S., Gupta,
A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020a.
Yu, T., Quillen, D., He, Z.,, Julian, R., Hausman, K., Finn, C.. and Levine, S. Meta-world: benchmark and evalua- tion for multi-task and meta reinforcement learning. In Conference robot learning, pp. 1094-1100, PMLR, 2020b.
Zhang, W., Deng, L., Zhang, L., and Wu, D. A survey on negative transfer. IEEE/CAA Journal of Automatica Sinica, 10(2):305 -399, 2022.

---

Learning Policy Committees with Diverse Tasks
A. Proof of Corollary 2.2
Definition A.1. A potential function is function & + R. Given a discount 7,71 and r2 differ by potential shaping for some potential Q, have that r2(s, a, s') ri (s, a, s') + ? ((s') ((s).
Definition A.2. Given transition function T, r1 and r2 differ by S' s'diriiittibution if ( s']] (ssTT((sa)) (s, s").
Definition A.3. 71 and differ by positive linear scaling if (2 e. ri(ss,, for some positive constant
Proof. Theorem 2.6 from (Skalse & Abate, 2023) says that two tasks 71, 72 have the same ordering of policies if and only if ri and 22 differ by potential shaping, positive linear scaling, and s'-redistribution. Therefore, if we could find optimal policies for these two tasks separately; they necessarily differ. And by forming policy committee of these two optimal policies, we have V 2 V V 2 V and V V V + V for any policy n.
B. Proof of Theorem 3.3
Definition B.1 (Gap preserving reduction for maximization problem). Assume IT and IT2 maximization problems. gap-preserving reduction from II, Ilg comes with four parameters (functions) f1, a, fz and B. Given instance æ of 11, the reduction computes in polynomial time an instance y of II, such that: OPT(x) fi(x) = OPT(μ) 2 f2( and (P)(w) < (x) = OP)(v) < (ly)
Proof. Let G = (V, E) be an undirected graph with 5 vertices and edges as follows:
4
5
3
2
We create instance of Max-coverage for set of Os in R" by filling out their coordinate matrix Ag
0 ifi=)
1.5c
i, adjacent
2.5c i, are not adjacent
dim
0
02
0g
01
03
2.5
2.5
2.5
2.5
2
2.5
2.5
2.5
2.5
3
2.5
2.5
2.5
1.5
4
2.5
2.5
2.5
0
1.5
5
2.5
2.5
1.5
1.5
0
Let 0, 20, 2.5), 02 = [2.5, 0, 2.5, 2.5), 0g = [2.5, 2,5,2,5,,, 1.5)) 04 = -2..5, 2.5, 2,5, 0,2.5, 0s [2.5, 2.5, 1.5, 1..
Projected onto the fifth axis, our thetas look like:
355
35
C
x
+
02 6
x
32 e
33
x
F3
C
x
+ C
And similarly, onto the third axis:
12

---

Learning Policy Committees with Diverse Tasks
23
33
31 6
I1 + 6
22 6
32 + E
24
I4 + E
35 C X 23 + c
We claim that we have constructed gap-preserving reduction for any t 0
OPT(A) = OPT(B) OPT(A) < <n = OPT(B) < nn'
To begin with, if the Max lax-Clique instance consists of a complete graph, then the As we created have coordinates equal to 1.5c everywhere except i-th coordinate, which is zero. So they can all be covered by one 0 [0.7e, 0.7c, 0.7c]. the coverage size is n. Therefore, the first implication is true.
Then for the second statement, we argue with the contrapositive: assume that one of the maximum coverage sets is S = [ii...... } and & 2 n²-t, We have to prove that the maximum clique has size greater than or equal to k 2 n--t.
Specifically, we prove that the vertices corresponding to the elements from form a clique.
If 0.,, f are from the set S, then they should be covered on each dimension since the 110, 10-00ylee l/c = max 10 05 S c. So 8,, fg have to be adjacent, because otherwise their corresponding coordinates the i-th and j-th dimension are more than away. For example, we have theta 63 0 and 05 O, so og and 0 must be 1.5c rather than 2.5c, which indicates that 3, 5 are neighbors in the graph.
Therefore, the points in S correspond to clique of size k 2 n1- in the graph. Thus, if the graph G' has a clique of size less than n² -t then the maximum coverage set has size less than n !-
C. Pseudocode of Greedy Intersection Algorithm
The full pseudocode for the Greedy Intersection Algorithm (GIA) algorithm is provided as Algorithm 1.
D. Proof of Theorem 3.4
Based on the proof of maxmizing monotone submodular functions by (Nemhauser et al., 1978).
Lemma D.1. Suppose 1 - 8* (K) is the optimal (e, 1 111-60 (..1-68)parammttrr po-mpraameter-oveer of achievable with fixed K. With probability at least a, , the probability of 0 from getting covered by the first representatives generated by Algorithm is greater than 1-8 (K)
K2 -K)-Ka (1-
Proof. We will prove the lemma through induction. We begin by defining the coverage region of each of the K committee member in the optimal parameter-cover as S; Furthermore, let IT denote the region covered by this optimal parameter-cover. Thus, II* = US$ St. Next, let A; denote the region covered by the representative selected on the i-th iteration. And let C, denote the set of Os from the dataset T that are covered after i-th iteration.
First of all, we want to show at i 1, the probability for 0 ~ I getting covered is greater than 1-5* K3 1/K)0 1-5" KB
K
By Hoeffding's theorem, Proer Ex (1 E - 4,) E. 10 probability at least 1 - g. Pre-r 10 E
2 $1 exp( 2xP(-2N33/9))
= Hence, with
N
U A = Ee-r (100 E U A S
Σ. 1(0:EU A₂) =
Cr-1l N
+ +/.
N
Now the union bound first gives that Præ-r(0 E II 0 0 A,J > Pre-r 0 E 11 Pre-r 10
i-1 Ajl =
8t (K) P000-Prr--e 10 U Ay). Applying union bound again, obtain that with probability at least 1 - 01, Σ - Pro-i
8y A0 & U An 2 (f II A & U A,} 2 1 8 (K) - ( (0--1 N .) Hence, max Pro-r E S A0
U 1 (-) 16 ) Let us call this maximising S S.
13

---

Learning Policy Committees with Diverse Tasks
Algorithm 1 Greedy Intersection
Input: T = [0 > 0, K 2
Output: Parameter cover C
C+
2: for round k= 1 to K do
1 to d do 4: Sort 7' in ascending order based on their m-th coordinates
3: for dimension
5: listsm +
6: for indiviual i = to N do
7:
S; + [0:]
8:
for j = 1 to do
9:
if 0,'s m-th coordinate < 03's 's m-tth coordinate +2c then
10:
Add 8, S; else
11:
12:
if listsm |-1] 1 C S then listsm| [1] t S:
13:
14:
else
Add S; listsm
end if break
17:
18:
end if
19: end for
20: end for
21:
end for
51
Sm +- argmax elisttt...... 15t 0 ..... 5m
23: covered -- S1* n... n Simx
24:
0g- f average of the covered
T covered
26:
C.adds(0.))
27:
end for
28: return C
According to our Algorithm A; covers the most As from T that were not covered in the previous rounds by U3 Aj. In particular, |C& - |C;-1/ is greater or equal to the number of As from T covered $ but not U Aj. Let us denote the latter as 81, and the former as 82, then 81 82 S
Hoeffding's s theorem gives Pre-r (Eg~r(1je SA0 U -1ll-s//N) 2 ( s (g)* and (re
[1[0 [1 SAO $ U A -
A & U A,]] ) (g) Hence with probability least - Bo
A 0 & U A,]] = (Be-r(1(0 e SAO & - A,]] s1/N) + (81 - + (s2/N E [1 (0 E A A 0 $
< 0 3. U A
Applying the result obtained at the beginning of the proof, we have with probability at least 1 1 -2())
Pr D~r
AAD &
U
UAJ
j=1
i-1
9
2 Pr E E & U
A,
3
Ø=1
1- ( - C
)
8
14

---

Learning Policy Committees with Diverse Tasks
Since nothing is covered before the first iteration, use step (3) with |Col prove the base condition for the claim. Because 2 1, we have 1-6-(K)-
1-8-(K) (1108/3 K
(K)
K
K
The induction hypothesis is that for all i K 1, we have Pro-r U₁=1 A, 2
1-8 (K]KK Cj=o(1-1/K))
By Hoeffding, Præ-r] Pr
1 1- A,} [C, N 1 2 8/3] - 2 expp( - 22 /9)). In other words, with probability at
least 1 1 2 Pre-r - ر,
Cc il
B/3 and N > Pra-r 0 -ز=14, A,] -3/3.
N
Then at the step K, since for :hB E (0, 1), (g)* e. we have with probability least 1-2 - - -2g)) 2 1 - 1 a,
8 1r UA,
0-1 j=1
i-1 U
Pr d-1
Aj] + Pr 0 AAA & A
j=1
C--1
8
1 (K) - 1(
+ +1/33
N
3
K
3
(K)
C--1l
(2K + -1)8
+(1 1/K)
K
N
3K
1 - (K)
i-1
(2K + 1).
+ 1/K) (/K)(
Pr 0
U
A,] 8/3)
K
A-T
3K
j=1
(K)
+ (1-1//H) 1/K) - (K)
K3 i-1
(2K + 1)8
E (1 - 1/KY²) - -1- 1/K)3/3 -1111/8)8//
(1-1/KY))
K
K
3K
1=0
1 (K)
(2K 1 + K - 1)8
1 8 (K) K3
i
Σ(1 1/KY
K
3K
K
j=1
1 (K) - KB
1/KY
K
1=0
Proof of Theorem 3.4. We can directly apply lemma D.1 to i K. Call the region defined by the cover generated by
Algorithm I = U Ay. Using the inequality (1 1/K) 2 1/e for all 2 0, we have
1- (K) - K3 K
Pr 0 E 1xl 2 d-T
Σ (1 - 1/KY/
K
j=
8 8(K)) K3 1 - (1 - 1/K)K
K
1- - (1-1/K) 1/K)
=(1 - (*K)) K1) (1 (1-1/K) K)
2(1 - 1/e) - -()(K) K39).
E. Proof of Theorem 3.5
Proof. Let us call the optimal solutions set to (1) A₁, and the optimal solutions set to (2) A2.
15

---

Learning Policy Committees with Diverse Tasks
We first show A1 Ag. Pick any {01,.... 0k} ) E A1. Due to the premise, for each i, since min 110 0s llo 0. there exists 0g such that |10g- 0: < 0. Thus,
have wik 1, and Wik 0 for all the other o k* Then
ReLU ( ({Zeetk) (wwa)))0 0:11 0:1100 - = (eLU 0:11x0 c) 0. By setting w this way, we could achieve the zero loss for the relaxation problem. Hence (01..... 0x} E Ag.
Now to show A2 C A1, suppose {0,,.... 0, w is a optimal solution. Due to the premise, we must have that ReLU (ςInete) o (wa))10. - 0e :) = for each Now fix i, since each a (wax) nonegative and summing them over k yields 1, there must be some positive coordinate o (wwz). Hence for all such k', ReLU(| 0. - -c) i.., ||0g - Ox Thus, min ||0x l. |10k 0o clso holds, and (01....... 0x) E, [minnE][A] ||0A 0: S e) Consequently, (0...... 0x) E Ar.
F. Proof of Lemma 3.6
E
ro, (0t, 0z) [m]]
E
7 (ro, (82, az) ra, (s,, at) + ro, (52, az)) [T]]]
E1
(re, (st, az) -r, (s,, az)) |- + EE
Aro, (St, [T]]
t=
E
o (16, 0z) ra, (sc, ae) |_m]] Vi"
T
LL10 - 01i100 + +v (-V + vi7))
+ (V22 - -V,"² )- + V.T.
T41
T
+ E
7 (ro ac) - -ro (82, ax) [mz] + V,
T+1 1 7
22,
6
If the discount factor 7 1, the argument is follows:
T
E
ro sc, ue) |T[?]
E
ro at) (s,,()), ro at) |m²l
ro(s 0t) - ro, at) |mm] + E
(ra, az) [m]]]
T
E
To, (8,, at) - -ro, (+)
at) |mر + Viji
T
L110; 0,1100 + V/ (-v;") + v.1))
0=
16

---

Learning Policy Committees with Diverse Tasks
ST + (V - (( + V."
=T +
ra. (st, ro, a+) |m]] + v,"
+2,
G. Proof of Theorem 3.8
We prove this by leveraging the following lemma by Azar et al. (2013).
Definition G.1. The average expected reward for a given policy is measured per time step as
h Σ
10
(0((s)
r(s,, π (8))) 80 =
h
And the empirical average return after n episodes is
h
A
nh
Assumption G.2. There exists a policy π. E II, which induces a unichain Markov process on the MDP M, such that the average reward n² 2 (s) Vs E S and any policy E II. The span of the bias function is sp(A*") max, x=* x=)(s) -mmn, x= (s) S H for some constant H, where À the bias is defined as X (s) ((s(s with s' being the state after the interaction (s, π (s)).
induces on the MDP M a single recurrent class with some additional transient states, i.e., 1tt (s) 1x for all S, and sp(x") & H for some finite H.
Assumption G.3. Suppose that each
Lemma G.4. (Azar et al., 2013, Lemma 1) Under Assumption G.2 and G.3, 1i 18 S 2(H + 1)V probability at least
2log(2/a) ph
+
H h
with
Proof. Let 32k 108(++1)+ log(4/o) Denote the average rewards of the best and second best policy in the committee as (3 2H-2H)
rt* 1 II pt p > B/h, by ensuring the difference between the estimation and the true average reward is small than
3/2h. We can make sure we have picked the best policy. From Lemma G.4, we know Príj S 1 +2HH+1)) 2 1og(4/o] +
ph
1 =rla- م +3/2h| 2 1-a/2. And Pr[a+ 2 /+ 2(H V 2log(4/a) + H Prfa < Hence with probability at least - a, f > / - 3/2h 2 μ + 3/hh - - > A Thus the empirically best policy we have picked is also the best expectation. Now if pr p < B/h, no matter which one we pick, we have the difference bound by 3/h. h. The same holds for all pairs of policies ordered based on their expected values. Either way, with probability 1 a, could find the best policy in the committee. Since our committee is (c, 8) cover, able to pick the policy with suboptimality B + with probability 1 8
2 1-a/2.
H. Additional Empirical Results
H.I. Results on Humanoid Direction
We present the learning curves for both the training and zero-shot testing case as Figure H.I. The few shot result has been listed in Table 1.
H.2. Additional results for empirical investigation our method
1. CLUSTERING ABLATIONS
We also obtained different clusters using PACMAN than using Kmeans++ and, as a result, much better performance for Halfcheetah- Halfcheetah-Velocityy as shown in the table below:
17

---

Learning Policy Committees with Diverse Tasks
Average returns of onee oploods
Number of frames
Number frames
Figure 4. Humanoid Direcctin Train and Zero Shot
Table 4. Comparison of clustering algorithms in HalfCheetah- ValfCheetah--ellcity Zero-Shot.
Method
Million Frames
12 Million Frames
MMeans++
1005.91 + 3.44
-89.50 + 3.10
DBScan
-234.05 + 9.56
-213.42 + 4.09
GMM
-239.32 + 11.27
-199.86 1 9.54
Random
-274.13 1 16.76
-258.07 + 13.62
PACMAN
-97.42 3.70
-74.20 6.76
As shown in Table ??, the proposed approach outperforms all baselines.
While the performance the KMeans++ algorithm appears relatively close to our method due to the significant gap between and the other three clustering methods (DBSCAN, GMM, and Random), we emphasize that this result considers one hundred percent of the population.
The advantage of our algorithm becomes
apparent when focusing the welfare of the majority. To illustrate this, present histogram of rewards for individual test tasks during zero-shot testing using policies trained with our algorithm versus KManns++ on the Half-Cheetah- Velocity benchmark:
Halfcheetah-Vel Zero Shot
Number of tasks (total: 100)
PACMANIkmeans)
PACMAN
25
20
15.
10-
individual return of one episode
Figure 5. Histogram comparison of two clustering methods for zero-shot individual task rewards in Half-Cheetah (velocity).
The results vividly highlight significantly greater density of high-performinn tasks (red regions on the right) with our method. This suggests that our approach effectively promotes superior task performance while minimizing underperformance. In contrast, KMeans++ yields a more uniform but mediocre distribution of task performance. There ideological difference between these clustering methods.
H.2.2. HYPERPARAMETER ABLATIONS
We consider here additional ablations varying K and omitted from the main body.
First, we present the results of ablations on K on both Mujoco (Halfcheetah- (Halfcheetah-Vellciitt) and World.
Both the ablation results for Meta- World and Mujoco demonstrate clear advantage of utilizing a policy committee.
18

---

Learning Policy Committees with Diverse Tasks
Hulfchettah-Ve Thais
Halfcheetah-Vis Zero Shot
Surcess
Bats
Success Batto
Number framess
Stepe
Steps
Figure 6. Varying K from to for Halfcheetah- Halfocectah-Velocity Velocity (left two) and to 4 for Meta-Worke (right two)
Table 5. Few-shot in Meta- World, varying K.
Method
6K Updates
12K Updates
MOORE
0.42 + 0.06
0.43 + 0.05
PACMAN (K =1)
0.32 + 0.05
0.31 0.0.4
PACMAN (K
0.50 0.05
0.50 0.05
PACMAN (K 0.61 0.04
0.62 0.05
PACMAN (K = 0.32 1 0.05
0.35 0.05
Especially for Meta-World, our method beats the baseline for every K greater than Here, in few-shot settings, even using K 2 already results in considerable improvement over the best baseline (MOORE), with K 3 significant further boost. Another thing to note is that increasing K is not always better. The results in both the figure and the table show that as the number of tasks becomes increasingly partitioned, the generalization ability of each committee member may weaken. Hence the performance for K = 4 is worse than K = 3.
Finally, we show the effect the e hyperparameter in the Meta-World zero-shot setting. These results are reported as the success rate across all tasks for
500K Steps
0.05
0.28
0.29
1M Steps
0.05
0.31
0.40
Table 6. Success rate for K
2.
We present also results for K = 3. The main results in the paper are for .6. Results are reported as the success rate over
all tasks for 3 seeds.
6 .
.8
500K Steps
0.23 + 0.08
0.54 0.08
0.56 0.09
0.60 + 0.07
1M Steps
0.25 + 0.07
0.60 0.10
0.58 0.14
0.58 + 0.07
Table 7. Success for 3.
We find that increasing c to cover more tasks can also improve performance (for similar reason that increasing K may not, as higher can ensure that we do not end up with clusters with too few tasks). Of course, for sufficiently high only a single cluster will emerge, so this, too induces an interesting tradcoff.
19

---

Learning Policy Committees with Diverse Tasks
I. Meta-Wooe Task Embeddings
First, we manually generated the following task descriptions by referencing the metaworld documentation.
Task Name
Objective
Environment Details
Reach-v1
Move the robot's end-effector to a target position.
The task is set on flat surface with random goal positions. The target position marked by small sphere or point in space.
Push-vl
Push a puck to specified goal posi- tion.
The puck starts in a random position on flat surface. The goal position is marked the surface.
Pick-Place-vl
Pick up puck and place des- ignated goal position.
The puck placed randomly the surface. The goal position is marked by target area.
Door-Open-vl
Open door with a revolving joint.
The door can be opened by rotating it around the joint. Door positions are randomized.
Drawer-Open-vi
Open a drawer by pulling it.
The drawer is initially closed and can slide out on rails.
Drawer-Close-vl
Close an open drawer by pushing it.
The drawer starts in an open position.
Button-Press-
Press button from the top.
The button mounted on a panel or flat surface.
Topdown-vl
Peg-Insert Lng-Innert-Side-
Insert a peg into a hole from the side.
The peg and hole are aligned hori- zontally.
vl
Window-Open-vl
Slide window open.
The window is set within a frame and can slide horizontally.
Window-Close-
Slide window closed.
The window starts in an open posi- tion.
v1
Door-Close-vl
Close a door with revolving joint.
The door can be closed by rotating it around the joint.
Reach-Wall-vl
Bypass a wall and reach a goal posi- tion.
The goal is positioned behind a wall.
Pick-Place-Wii Wall-
Pick a puck, bypass a wall, and place it at goal position.
The puck and goal are positioned with wall in between.
vl
Bypass a wall and push a puck to a goal position.
The puck and goal are positioned with wall in between.
Button-Press-vl
Press button.
The button is mounted on a panel surface.
Mutton--Pesss
Bypass wall and press button
The button is positioned behind wall on panel.
Toplown-Wall-vl
Wall-v1
from the top.
Button-Press-
Bypass a wall and press button.
The button is positioned behind wall.
Wall-v1
Peg- Unplug-Sia Siiee
Unplug peg sideways.
The peg is inserted horizontally and needs to be unplugged.
v1
Disassemble-vl
Pick a nut out of a peg.
The nut is attached to a peg.
Hammer-vl
Hammer a nail on the wall.
The robot must use a hammer to drive nail into the wall.
Plate-Slide-vi
Slide a plate from cabinet.
The plate is located within cabinet. The plate is within a cabinet and must be removed sideways.
Slide a plate from a cabinet side- ways.
vl
Plate-Slide
Slide a plate into cabinet.
The robot must place the plate back into a cabinet.
vl
20

---

Learning Policy Committees with Diverse Tasks
Plate-Slide-Back-
Slide a plate into a cabinet sideways.
The plate is positioned for sideways entry into the cabinet.
Side-vl
Handle-Presse Press-vl
Press handle down.
The handle is positioned above the robot's end-effector.
Handle-Pull-vl
Pull a handle up.
The handle is positioned above the robot's end-effector.
Handle-Press-
Press a handle down sideways.
The handle is positioned for side- ways pressing.
Side-vl
Handle-Pull-Side-
Pull a handle up sideways.
The handle is positioned for side- ways pulling.
vI
Stick-Push-vl
Grasp stick and push a box using the stick.
The stick and box are positioned ran- domly.
Stick-Pull-vl
Grasp stick and pull box with the stick.
The stick and box are positioned ran- domly.
Basketball-vl
Dunk the basketball into the basket.
The basketball and basket are posi- tioned randomly.
Soccer-v1
Kick a soccer ball into the goal.
The soccer ball and goal are posi- tioned randomly.
Faucet- Faucet-Open-vl
Rotate the faucet counter-clockwiss
The faucet is positioned randomly.
Paucet-Close-v
Rotate the faucet clockwise.
The faucet is positioned randomly.
Coffee-Push-vl
Push a mug under a coffee machine.
The mug and coffee machine are po- sitioned randomly.
Coffce-Pull-vl
Pull a mug from a coffee machine.
The mug and coffee machine are po- sitioned randomly.
Push a button on the coffee machine.
The coffee machine's button is posi- tioned randomly.
Sweep-vl
Sweep a puck off the table.
The puck positioned randomly on the table.
Sweep-Into-vl
Sweep puck into hole.
The puck is positioned randomly the table near a hole.
Pick-Out-Of-
Pick up puck from a hole.
The puck is positioned within hole.
Hole-vl
Assembly-v1
Pick up a nut and place it onto a peg.
The nut and peg are positioned ran- domly.
Shelf-Place-vl
Pick and place a puck onto a shelf.
The puck and shelf are positioned randomly.
Push-Back-vl
Pull a puck to goal.
The puck and goal are positioned ran- domly.
Lever-Pull-vl
Pull a lever down 90 degrees.
The lever positioned randomly.
Dial-Turn-vl
Rotate a dial 180 degrees.
The dial is positioned randomly. The puck and bins are positioned ran- domly.
Bin-Picking-vl Grasp the puck from one bin and place it into another bin.
Box-Close-vl
Grasp the cover and close the box with it.
The box cover is positioned ran- domly. The hole is positioned randomly. The lock is positioned randomly.
-asd-
Insert the gripper into hole.
Door-Lock-vl
Lock the door by rotating the lock clockwise
Door-Unlock-y
Unlock the door by rotating the lock counter-llokwwise.
The lock is positioned randomly.
Our test tasks are the following: assembly, basketball, bin picking, box close, button press topdown, button press topdown- wall, button press, button press wall, coffee button, coffee pull, coffee push, dial turn, disassemble, door close, door lock,
21

---

Learning Policy Committees with Diverse Tasks
door open, door unlock, drawer close, drawer open and faucet close.
Then for each task, we fed the following prompt to an LLM:
"<Task Name>": "Objeciive: <Objective description>
ronment Details: <Environment details>"
For example:
"Reach-vi"; "Objective: Move the robot's s end-effector to a target position.
Environment Details: The task is set on flat surface with random goal positions.
The target position is marked by a small sphere or point in space."
Next, we extracted the penultimate layer's s activations of the LLM and computed the channel-wise mean. The result was a single 50-dimensional vector that represented the task.
J. Meta-World Clustering Analysis and Discussion
Simply put, method works by having committee members which are innately specialized to specific tasks, illustrated below. Here committee member 2 is specialized to door open and committee member 3 is specialized to door close. At the same time, committee member 2 performs door close poorly and committee member 2 performs door open poorly. A MTRL policy in trying to perform all tasks doesn't it perform any particular task well. Our method will select committee member for door open and committee member for door close.
To understand the parametrization discussed in section 3.4 produces suitable clusters we have applied PCA PCA to clustering of 10 tasks. We note that the window tasks and drawer tasks close in task space. Additionally, the dynamics and goals of the push and pick-place tasks are nearly identical. Window close is close to door open both these tasks have the agent needing to move to the horizontally to begin the task.
PCA Tank Wacler
component
2
PCA Componen:1
Figure 7. PCA for our parametrization described in 3.4.
When K = 3, the three Meta-World clusters (for one of the random subsamples of 30 tasks) are provided below.
On the high level, the clustering can be described as corresponding to three categorics of manipulation tasks. The first cluster involves pulling and sliding manipulations (e.g., pull puck, move end-effector, slide plate to and from the cabinet). The second cluster includes manipulations that involve pressing and pushing (e.g., rotate the faucet, press a handle down, insert a peg into a hole, kick soccer ball, sweep puck). The third cluster consists of tasks involving indirect constrained manipulation and which thus require better precision and control (e.g., bypass a wall, grasp stick and pull push box with it).
22

---

Learning Policy Committees with Diverse Tasks
Cluster
Cluster 2
Cluster
handle-press-side-.
faucet-open-vl
hand-insert-vl
peg-unplug-side-vi
hammer-vl
handle-pull-vi
pick-out-of-hole-vl
handle-press-siee-vl
lever-pull-vl
pick-place-vl
handle-press-vl
peg-insert-side-vl
plate-slide-backe back-v1
handle-pulle
suddle-pullsideevl
push-wall-vl
plate-slide-side-vl
peg-insert-side-vl
reach-wall-vl
plate-slide-vl
peg-upplug-side-vl
shelf-place-vi
push-back-vl
pick-out-of-hole--
stick-pull-vl
push-vl
pick-place-vl
stick-push-vl
reach-v1
pick-place-wall-v/
sweep-into-vl
stick-push-vl
window-close-vl
sade--llde-side-vl
window-open-vl
plate-slide-vl
soccer-v1
sweep-vl
Table 9. Meta- World Task Clusters
23


Reviews:
[{"review_id": "LZkRfV1ceG", "reviewer": "Reviewer_fviP", "summary": "This paper introduces PACMAN, a novel framework and algorithmic approach for learning policy committees in multi-task reinforcement learning (MTRL) and meta-reinforcement learning (meta-RL) settings with diverse tasks. The key challenge addressed is the difficulty of generalizing effectively across diverse tasks, where traditional MTRL and meta-RL approaches often struggle due to negative transfer or computational expense.\n\nPACMAN introduces a theoretically grounded approach for learning policy committees. It frames the problem as finding an (ε, 1-δ)-cover of the task distribution, ensuring that with high probability, there's at least one near-optimal policy in the committee for any encountered task.\n\nThe paper presents two practical algorithms: Greedy Intersection Algorithm (GIA), which provides provable approximation and task sample complexity guarantees when task dimensions are low and Gradient-Based Coverage, which is a general, practical, gradient-based approach suitable for higher-dimensional tasks.\n\nThe paper provides a provable sample complexity bound for few-shot learning that depends only on the number of clusters (committee size) and not on the state or action space size.\n\nThe authors performed extensive experiments on MuJoCo and Meta-World benchmarks demonstrate that PACMAN outperforms state-of-the-art MTRL, meta-RL, and task clustering baselines in training, generalization, and few-shot learning, often by a large margin.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments!\n\n\n>**Comment:** The reliance on a parametric task representation (or the ability to obtain one using LLMs) is a limitation, although a common one.\n\n**Response**: Indeed, our method does require access to a parameterization for the tasks. We note, however, that our results with embeddings based on language descriptions suggest that our approach can nevertheless be very effectively applied to a very broad array of problems. Indeed, the continuing improvement of both the quality and scope of LLMs suggests that the space of problems to which our approach can be effectively applied will likely continue to expand.\n\n>**Comment:** The need to tune the hyperparameter ε is a practical consideration.\n\n**Response**: Our method introduces only one hyperparameter, $\\epsilon$, which is easy to tune, particularly in comparison to myriad of hyperparameters that typical deep RL methods introduce. For example, our ablation studies in Appendix H suggest that choosing the smallest $\\epsilon$ that ensures full coverage for the largest $K$ a practitioner can afford to train works well in practice.\n\n>**Comment:** The GIA algorithm's exponential time complexity in d makes it only efficient for constant d.\n\n**Response**: This is true; indeed, as we show in Theorem 3.3, the problem is inapproximable in the worst case.  We address this issue by presenting a practical gradient-based coverage algorithm, which is the main workhorse of the Meta-World experiments, and which outperforms SOTA baselines.\n\n>**Comment:** The paper mentions that the RL step will typically dominate computational complexity. Could you provide some empirical evidence or a more detailed analysis comparing the time spent on clustering versus RL training, particularly for the Meta-World experiments? This would affect evaluation by giving insight into computational cost.\n\n\n**Response**: For the Meta-World experiments, training a single policy requires approximately 40 hours using an A40 GPU for 1M steps. In comparison, clustering takes about 1 second in a Google Colab notebook, and obtaining task embeddings takes around 2 minutes on an A40 GPU. We will include these details along with the corresponding timings in the revised version of the paper. \n\n\n>**Comment:** In the Meta-World experiments, you use MOORE for within-cluster training. Have you experimented with other MTRL or meta-RL algorithms for this step, and if so, how did they compare? This would clarify the extent to which the gains are attributable to the overall framework versus the specific choice of within-cluster RL method.\n\n**Response:** While we have not extensively evaluated this issue, we did run experiments with alternative within-cluster training to MOORE. We have found that PACMAN typically outperforms the approach it wraps by a considerable margin. For example, PACMAN trained using CMTA produced significant improvement as well:\n\n|       |   CMTA   | PACMAN (CMTA)  |\n|-------|--------|--------|\n|125K Steps| .20 $\\pm$ .08 | .28 $\\pm$ .08| \n|250K Steps| .25 $\\pm$ .09 | .41 $\\pm$ .10| \n\n\n>**Comment:** The (ε, 1-δ)-cover concept is central to the theoretical analysis. Could you elaborate on the practical implications of choosing different values of ε? How does one typically set this hyperparameter in a new environment? Clarifying how users could tune this parameter.\n\n**Response**:  In Appendix H.2.2, Table 5, we show that when the coverage is not complete due to a small ε, performance significantly degrades.  Practically, the choice of ε depends on the cost of training a policy. A user should aim to select the largest committee size they can afford and then choose an ε that ensures maximum coverage. One approach is to first compute the distances between all tasks' embeddings. This could provide an initial estimate for setting ε. From there, we can fine-tune ε to ensure it is small while still guaranteeing adequate coverage. From what we have observed, ε is far less sensitive than typical RL hyperparameters.", "claims_and_evidence": "The claims are generally well-supported. The theoretical results (Theorems 3.3, 3.4, 3.5, 3.7, 3.8, Lemma 3.6) have provided proofs (although some are deferred to the appendix). The empirical results are comprehensive, covering multiple environments, baselines, and evaluation settings. The ablation studies (varying K and ε) provide further evidence for the effectiveness of the proposed approach.", "methods_and_evaluation_criteria": "The methods and evaluation criteria are appropriate. The use of MuJoCo and Meta-World benchmarks is standard in the field. The evaluation metrics (training performance, zero-shot generalization, few-shot adaptation) are relevant to the problem. The comparison to a wide range of baselines strengthens the evaluation.", "theoretical_claims": "I have roughly checked the proofs provided in the main text and skimmed through proofs in the appendix, and they appear to be logically sound, given the assumptions made.", "experimental_designs_or_analyses": "The experimental design is generally sound. The choice of environments, baselines, and evaluation metrics is appropriate. The ablation studies help to understand the impact of key hyperparameters. The use of multiple random seeds and reporting of standard deviations provides statistical confidence in the results.", "relation_to_broader_scientific_literature": "The key contributions build on MTRL and meta-RL literature, improving over baselines like RL2 and VariBAD. The policy committee idea relates to ensemble methods or mixture of experts, and clustering is common in ML, but applying it to RL policy learning is novel. Theoretical bounds are crucial for RL efficiency.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n1. Strong theoretical grounding with provable guarantees.\n\n2. Novel algorithmic approach combining clustering and policy committee learning.\n\n3. Empirical results demonstrating significant improvements over strong baselines.\n\n4. Applicability to both parametric and non-parametric task settings.\n\n5. Clear and well-written presentation.\n\nWeaknesses:\n\n1. The reliance on a parametric task representation (or the ability to obtain one using LLMs) is a limitation, although a common one.\n\n2. The need to tune the hyperparameter ε is a practical consideration.\n\n3. The computational cost of training multiple policies could be a concern in some settings, although the paper argues that the RL step dominates.\n\n4. The GIA algorithm's exponential time complexity in d makes it only efficient for constant d.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. The paper mentions that the RL step will typically dominate computational complexity. Could you provide some empirical evidence or a more detailed analysis comparing the time spent on clustering versus RL training, particularly for the Meta-World experiments? This would affect evaluation by giving insight into computational cost.\n\n2. In the Meta-World experiments, you use MOORE for within-cluster training. Have you experimented with other MTRL or meta-RL algorithms for this step, and if so, how did they compare? This would clarify the extent to which the gains are attributable to the overall framework versus the specific choice of within-cluster RL method.\n\n3. The (ε, 1-δ)-cover concept is central to the theoretical analysis. Could you elaborate on the practical implications of choosing different values of ε? How does one typically set this hyperparameter in a new environment? Clarifying how users could tune this parameter.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BO1qda6Lba", "reviewer": "Reviewer_zPNZ", "summary": "The paper introduces a new learning paradigm, called policy committee learning effectively for solving multi-task RL. More specifically, a policy committee targets learning a set of policies maximizing the best-performing policy's discounted return for any given task sampled from a predefined task distribution. The paper assumes that the tasks differ by their transition dynamics and reward functions, but not state and action spaces. Also, the paper assumes access to a parameterization of these tasks.\n\nFor this new learning paradigm, the paper introduces three algorithms, GEA, GIA, and, Gradient-based Coverage. These methods centers around finding a clustering that achieves $(\\epsilon, 1 − \\delta)$-parameter-cover following Definition 3.1. Further theoretical analysis indicates that learning individual policy for each cluster leads to a solution to policy committee learning (Definition 2.1) with bounded optimality gap.\n\nThe paper conducts the experiments in MetaWorld, Halfcheetah, and Humanoid domains, and demonstrating improved asymptotic performance and few-shot learning performance compared to recent meta-learning and multi-task RL approaches.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments and suggestions!\n\n>**Comment:** Meta-World domain breaks the assumption made by the theoretical analysis that all the tasks share the same transition dynamics, as different tasks involve interacting with different objects, and therefore uses different transition dynamics.\n\n\n**Response**: This was by design. Our key finding is that while our analysis (like most worst-case analysis) makes restrictive assumptions, its value is exhibited by the Meta-World experiments that demonstrate robustness of the proposed approach to the assumptions. This is common in theoretical analysis, which of necessity must make assumptions that are inevitably violated in practice; what ultimately demonstrates value is empirical performance. The theoretical analysis, in turn, serves both to explain why we should expect strong performance in practice, and to ensure that the approach is principled (and, therefore, less likely to be fragile as we change domains). \n\nThus, we view the combination of (a) superior empirical performance on the mainstream benchmarks and (b) provable worst-case guarantees in an important and broad array of settings (diverse rewards) makes our approach a particularly significant advance over prior art.\n\n\n>**Comment:** It is hard to believe simpler clustering strategies, like KMeans++, do not give the same clustering result... Could author also conduct comparison in Meta-World domain?\n\n**Response**: We indeed obtained different clusters using PACMAN than using Kmeans++ and, as a result, much better performance for Meta-World, as shown in the table below:\n\n|       | Kmeans++    |  PACMAN (Ours)     |\n|-------|--------|--------|\n|125K Steps| 0.22 $\\pm$ 0.06 | 0.36 $\\pm$ 0.07| \n|250K Steps| 0.30 $\\pm$ 0.07 | 0.48 $\\pm$ 0.09| \n\nNotably, PACMAN exhibits $\\sim 60$\\% performance impovement over Kmeans++.\n\n>**Comment**: It would be interesting to visualize the clustering results in HalfCheetahVel and compare it with KMean++.\n\n**Response**: In Appendix H.2.1., we provided a histogram of the final clustering results in terms of individual task returns. We will also include a visualization of our and KMeans++ clusters in the revision.\n\n>**Comment**: Differentiating the work from *Onur et. al., ICML 2024*\n\n**Response:** While our approach shares the key insight of addressing task diversity with MoE methods like the one proposed by Onur et al., we take it a step further, generating a small number of clusters upfront and using RL for each cluster independently thereafter. This facilitates more efficient and effective clustering (since no RL is involved for that) with provable guarantees, and is particularly valuable for few-shot adaptation (also with theoretical guarantees). Notably, our baselines include MoE-based methods (e.g., MOORE from ICLR 2024).\n\n>**Comment:**  Could the author give an rough estimation of the optimality bound for HalfCheetch domain?\n\n**Response:** $2L\\frac{1-\\gamma^{h+1}}{1-\\gamma} \\epsilon \\approx 100$, which is quite tight in this setting. When $K=3$, the committee generated by GIA yields near-perfect coverage $1-\\delta^* \\approx 1$, whereas for $K=1$, $1-\\delta^* \\approx 0.4$. The second bound is thus $\\sim 0.63$ for the former and $\\sim 0.25$ for the latter, so that the advantage of $K=3$ is a factor of 2.5. In practice, we find that these **worst-case** bounds are quite conservative, and empirical performance is far better.\n\n>**Comment:** I checked Appendix J, but I would be interested to see the actual assignment of the 50 tasks to the three policies in the policy committee. Could the author also provide the actual assignments of the 50 tasks to the three policies in the policy committee? \n\n**Response:** We will add full details of task-to-cluster mappings in the revision. At the high level, the three Meta-World clusters (for one of the random subsamples of 30 tasks) can be described as corresponding to three categories of manipulation tasks.  The first cluster involves pulling and sliding manipulations (e.g., pull a puck, move end-effector, slide a plate to and from the cabinet). The second cluster includes manipulations that involve pressing and pushing (e.g., rotate the faucet, press a handle down, insert a peg into a hole, kick a soccer ball, sweep a puck). The third cluster consists of tasks involving indirect or constrained manipulation and which thus require better precision and control (e.g., bypass a wall, grasp a stick and pull or push a box with it). \n\n>**Comment:** There are a couple of missing citations: (1) line 336 SAC; (2) line 349 MetaWorld. In Theorem 3.4, 3.7 missing defintions\n\n**Response:** Thank you for pointing this out. We will rectify this in the revision.", "claims_and_evidence": "The paper claims a theoretically-grounded framework for learning policy committees, however the theory is rather limited in that it assumes all the tasks share the same transition dynamics. Also, I am not sure whether the theoretical bound is tight enough can explain the performance gain in the experiments.", "methods_and_evaluation_criteria": "Yes, learning a policy committee makes sense for solving multi-task RL problems. Learning different policies may mitigate the gradient conflict and also by limiting the total number of policies, the learning may become more efficient benefit from the share similarity between tasks.", "theoretical_claims": "I did not check the proofs.", "experimental_designs_or_analyses": "1. MetaWorld domain breaks the assumption made by the theoretical analysis that all the tasks share the same transition dynamics, as different tasks involve interacting with different objects, and therefore uses different transition dynamics.\n\n2. It is also a bit suspicious that although the language embedding provide a continuous parameterization of the tasks, the real task space is discrete and only have 50 tasks in total. In that, it is hard to believe simpler clustering strategies, like KMean++, do not give the same clustering result. I saw that comparison with other clustering algorithms is only conducted in HalfCheetahVel. Could author also conduct comparison in MetaWorld domain? \n\n3. It would be interesting to visualize the clustering results in HalfCheetahVel and compare it with KMean++. HalfCheetahVel uses a simple one-dimensional task parameter, I wonder in what degree the clustering result of PACMAN differs from conventional goal of minimizing sum of shortest distances.", "relation_to_broader_scientific_literature": "The paper introduces a new learning paradigm called policy committee learning. I think the idea is close to mixture of experts.", "essential_references_not_discussed": "The idea is pretty similar to mixture of expert for RL:\nCelik, Onur, Aleksandar Taranovic, and Gerhard Neumann. \"Acquiring diverse skills using curriculum reinforcement learning with mixture of experts.\" arXiv preprint arXiv:2403.06966 (2024).\n\nIn Onur et. al., the clustering (partition of the task space) is guided by the experts performance, e.g., if the expert shows promises in solving the task. The paper instead considers the $(\\eplison, 1 − \\delta)$-parameter-cover. I think the author should consider differentiating with Onur et. al.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "1. There are a couple of missing citations: (1) line 336 SAC; (2) line 349 MetaWorld. \n2. In Theorem 3.4, 3.7, $1/e$ is not defined. In Theorem 3.8, $H$ is not defined. The theoretical analysis could be presented for clearly.", "questions_for_authors": "1. In theorem 3.7, could the author give an rough estimation of the optimality bound for HalfCheetch domain? Specifically, what are the magnitudes for  \n$2L\\frac{1−\\gamma^{h+1}}{1−\\gamma} \\epsilon + \\eta$ and $(1 − \\frac{1}{e}) )(1 − \\delta^∗(K, \\epsilon) − K\\beta))$, using $K=3$ and $\\epsilon = 0.6$. How does the bound compared to $K=1$ using one single policy to learn all the tasks?\n2. It is also a bit suspicious that although the language embedding provide a continuous parameterization of the tasks, the real task space is discrete and only have 50 tasks in total. In that, it is hard to believe simpler clustering strategies, like KMean++, do not give the same clustering result. I saw that comparison with other clustering algorithms is only conducted in HalfCheetahVel. Could author also conduct comparison in MetaWorld domain?\n3. Could the author also provide the actual assignments of the 50 tasks to the three policies in the policy committee?\n4.  Could the authors also provide the visualization of the clustering results in HalfCheetahVel and compare it with KMean++. HalfCheetahVel uses a simple one-dimensional task parameter, I wonder in what degree the clustering result of PACMAN differs from conventional goal of minimizing sum of shortest distances.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KSi6vymsn3", "reviewer": "Reviewer_sYFo", "summary": "The paper introduces PACMAN, a novel framework for learning policy committees in multi-task Markov decision processes (MDPs) with diverse tasks. The approach includes clustering-based approach to create a representative cover over tasks and a gradient-based alternative to group tasks based on parametric representations and then trains a separate policy per cluster. The authors support their method with theoretical guarantees and demonstrate significant empirical improvements on benchmarks such as MuJoCo and Meta-World.\n\n**Update after rebuttal**--I appreciate the authors’ thorough revision and the additional experiments addressing my concerns. I hope those will be implemented in the revised draft. I have decided to keep my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and suggestions!\n\n>**Comment:** The assumption of parametric task structure may limit generalizability.\n\nResponse: We agree that assuming that tasks are parametric is a limitation. However, we show that we can often use LLM embedding of natural language task descriptions to enable parametric representations of non-parametric tasks. Our results on the Meta-World benchmark (in which tasks are non-parametric) demonstrate that this is quite effective, and suggests that our approach is nevertheless very broadly applicable. Indeed, as capabilities of LLMs continue improving, we expect the scope of applicability to continue to expand.\n\n>**Comment:**  Sensitivity to the hyperparameters and the challenge of tuning it in different domains.\n\n**Response**: Our method adds only a single hyperparameter, $\\epsilon$, which is easy to tune without running RL. Our ablation studies in Appendix H suggest that choosing a small $\\epsilon$ that ensures full coverage for the largest $K$ a practitioner can afford to train works well in practice. As RL algorithms (particularly in multi-task settings) come with a large number of hyperparameters that are difficult to tune in practice, we view the addition of the $\\epsilon$ hyperparameter to be well worth the performance improvement and theoretical guarantees.\n\n>**Comment:**  Scalability concerns for very high-dimensional tasks or a massive number of tasks.\n\n**Response**: While theoretically our framework is hard in high-dimensional settings, in practice the computational cost is dominated by RL (seconds or minutes for clustering compared to hours for RL). Moreover, we show that the proposed gradient-based approach works well in higher dimensions. Moreover, both our greedy and gradient-based approaches run in polynomial time in the number of tasks. Nevertheless, consideration of extremely high dimensions and massive numbers of tasks remain important research challenges in multi-task and meta-RL.\n\n>**Comment:**  ...A more detailed discussion of contextual MDPs might further enrich the context... The notion of “policy committee” feels similar to “multi-policy training” proposed in “Model-Based Transfer Learning for Contextual Reinforcement Learning (NeurIPS 2024).”\n\n**Response**: Thank you for pointing this out! We will add a discussion of contextual MDPs in the revision. While our notion of a “policy committee” is similar to “multi-policy training”, the focus of Cho et al. is on a sequential task selection problem, with a task-specific policy trained on the selected task. Our setting, in contrast, begins with a given set of tasks (we cannot choose tasks for training) and entails clustering, with the policy committee associated with task clusters. We will add this discussion in the revision.\n\n>**Comment:** How can I relate PACMAN results in Table 1 and Table 3?\n\n**Response**: As Table 1 is for few-shot and Table 3 for zero-shot generalization, they are not directly comparable. The former shows that PACMAN outperforms SOTA in few-shot learning, while the latter shows that the novel clustering approach we propose for multi-task learning is better than using conventional clustering methods such as Kmeans++ for task clustering in this setting.\n\n>**Comment** Table 4 shows that larger K could decrease the performance. Then how should we choose the optimal K in practice? any recommendations or thoughts? Similarly, I’m also curious about the tradeoff for higher epsilon mentioned in line 958\n\n\n**Response**: We discuss this in detail in section H.2.2 (the paragraph under the table in question). In practice, our guideline is to use $K$ no larger than what is sufficient to provide a $(\\epsilon,1)$ cover (i.e., cover all training tasks); moreover, $K=3$ has worked well across distinct domains and in many practical settings may thus be a good starting point. As for $\\epsilon$, we provide some guidance on its selection in our response to Reviewers fviP and zPNZ given a fixed $K$.\n\n\n>**Comment:** Curious how this approach relates to the special case of MoE with hard gate.\n\n\n**Response**: In MoE, the gating mechanism typically needs to be trained as part of the network, and with hard gating, selecting the right expert is a challenging task. In contrast, our method alleviates this computational burden by directly producing a set of policies, and the best can be quickly identified through policy evaluation for any task.\n\n>**Comment:** Citation error and figure readability. \n\n**Response**: Thank you for pointing this out! We will rectify the citation errors and improve the readability of the figures in the revised version.", "claims_and_evidence": "The paper makes several key claims:\n- It offers provable guarantees on generalization and few-shot adaptation.\n- It achieves improved performance over state-of-the-art baselines in diverse task settings.\nThe theoretical results (e.g., Theorems 3.7 and 3.8) are well-supported by proofs in the appendices, and the experimental results consistently back up the claims.\n- However, some assumptions (e.g., tasks being parametric, Lipschitz continuity of rewards) may not hold in real-world settings.\n- The proposed approaches are validated with multi-task benchmarks.", "methods_and_evaluation_criteria": "The methodology is thoughtfully designed:\n\n- The clustering approach effectively handles task diversity by creating a representative cover over task parameters.\n- The integration of gradient-based optimization to refine cluster representatives is innovative.\nThe evaluation on standard benchmarks is thorough, with comparisons to 11 state-of-the-art baselines.\n- Evaluation criteria (average return of one episode or success ratio) from benchmarks seems to be reasonable.", "theoretical_claims": "The paper presents several rigorous theoretical contributions including the derivation of sample complexity bounds and few-shot guarantees. While the proofs appear sound, some assumptions (such as the parametric form of tasks and the specific Lipschitz conditions) might limit applicability in settings where such conditions are not met.", "experimental_designs_or_analyses": "Experiments are extensive and well-structured:\n\n- The use of both zero-shot and few-shot evaluations provides a comprehensive picture of performance.\n- Comparisons with multiple baselines help in understanding the relative strengths of PACMAN.\n- The scalability of the clustering method in higher-dimensional settings and its sensitivity to the number of tasks are areas that may benefit from further clarification.\n- In addition, there are many other things remained questionable in experiment settings. For instance, I’m also curious how the choice of 30 training tasks and 20 test tasks in MT-50 (Meta-World) affects the other baseline’s performance. The effect of task description and language embedding can affect to some extent.", "relation_to_broader_scientific_literature": "The paper is well-positioned within the broader literature on multi-task and meta-reinforcement learning. It clearly distinguishes itself from prior works such as MAML, PEARL, MOORE, and task clustering approaches by addressing the challenges of task diversity and negative transfer.", "essential_references_not_discussed": "While the paper cites many relevant works, it might benefit from a discussion of more recent approaches to policy ensembles and adaptive task embeddings in non-parametric settings. This could help in understanding how PACMAN compares to or complements emerging methodologies. A more detailed discussion of contextual MDPs from related areas might further enrich the context. Specifically, the notion of “policy committee” feels similar to “multi-policy training” proposed in “Model-Based Transfer Learning for Contextual Reinforcement Learning (NeurIPS 2024).”", "other_strengths_and_weaknesses": "Strengths:\n- Strong theoretical foundation with rigorous proofs.\n- Extensive and convincing empirical validation across challenging benchmarks.\n- Innovative combination of clustering with gradient-based refinement.\n\nWeaknesses:\n- Dependence on the assumption of parametric task structure may limit generalizability.\n- Sensitivity to the hyperparameters and the challenge of tuning it in different domains.\n- Scalability concerns for very high-dimensional tasks or a massive number of tasks.", "other_comments_or_suggestions": "- Section 4 → citation error in line 349\n- Section 4.1 → citation error in line 335 and (EM) in line 342\n- Curious how this approach relates to the special case of MoE with hard gate.\n- Font size in figures are sometimes too small.\n- The readability of the figures can be improved. It’s not as friendly for now.", "questions_for_authors": "- How can I relate PACMAN results in Table 1 and Table 3?\n- Table 4 shows that larger K could decrease the performance. Then how should we choose the optimal K in practice? any recommendations or thoughts? Similarly, I’m also curious about the tradeoff for higher epsilon mentioned in line 958.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jIApQ7sMcC", "reviewer": "Reviewer_avxU", "summary": "The paper proposes PACMAN, a novel method that learns a policy committee, ensuring that at least one near-optimal policy is included for each task with high probability. The proposed approach is evaluated on Half-Cheetah Velocity, Humanoid, and MetaWorld tasks, demonstrating superior performance compared to multiple baselines.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments and clarification questions!\n\n>**Comment:** The method, especially the algorithm is not clear to me.\n\n**Response**: The details of the algorithm are provided in Section 3.1. In summary, we cluster tasks into groups, where each group contains similar tasks (in a way that yields provable in terms of performance and generalization) and then train a policy for each group. We call the resulting set of policies a *policy committee*. This committee can then be used for both in-sample and out-of-sample multi-task RL settings, as well as for few-shot meta-RL problems.\n\n>**Comment:** Can you provide detailed analysis on what tasks are clustered together to verify the clustering mechanism works?\n\n**Response**: Certainly. At the high level, the three Meta-World clusters (for one of the random subsamples of 30 tasks) can be described as corresponding to three categories of manipulation tasks.  The first cluster involves pulling and sliding manipulations (e.g., pull a puck, move end-effector, slide a plate to and from the cabinet). The second cluster includes manipulations that involve pressing and pushing (e.g., rotate the faucet, press a handle down, insert a peg into a hole, kick a soccer ball, sweep a puck). The third cluster consists of tasks involving indirect or constrained manipulation and which thus require better precision and control (e.g., bypass a wall, grasp a stick and pull or push a box with it). We will add full details of task-to-cluster mappings in the revision.\n\n>**Comment:** How many clusters are there in the committees for both experiments? \n\n**Response**: For the main experiments, $K=3$ in all domains, but we also consider ablations with $K \\in \\{1,2,4\\}$ (see the Appendix for details).\n\n>**Comment:** Is the number of clusters predefined or learned automatically?\n\n**Response**: The number of clusters is predefined in our experiments, but can also be obtained automatically if we have a coverage target (e.g., if we wish to cover 100% of training tasks). \n\n>**Comment:**  Will new committee member be added when learning new tasks? Or will it reuse existing member?\n\n**Response**: We reuse existing members for new tasks; in zero-shot settings, we apply these directly on out-of-sample tasks, while in few-shot settings we fine-tune policies in the committee.\n\n>**Comment:**  How is the number of parameters for PACKMAN compared with other baselines?\n\n**Response**: If an architecture used for a policy has $p$ parameters, the committee of $K$ policies will train $Kp$ parameters. Different baseline approaches vary significantly in the number of parameters $p$, as this is not typically a primary consideration in prior literature, particularly in the context of the challenging Meta-World benchmark.\n\n>**Comment:** What is the optimal performance for each set of experiments? \n\n**Response**: For meta-learning mujoco experiments, the theoretical upper bound on performance is 0. For Meta-World, the theoretical upper bound is the success rate of 1. However, we are unaware of any SOTA method that comes close to achieving either of these, and it is unknown what is a feasible optimal performance on these benchmarks with respect to out-of-sample tasks.\n\n\n>**Comment:** Is the policy state-based or image-based?\n\n**Response**: The policy is state-based.\n\n>**Comment:** What are the 30 and 20 tasks mentioned in the paper? How are they constructed? Are they randomly sampled?\n\n**Response**: We uniformly randomly partition the 50 total tasks in Meta-World into 30 trainig and 20 test tasks.\n\n>**Comment:** Is the return equivalent to the success rate? Clarifying this would improve result interpretation.\n\n**Response**:  While high returns generally correlate with a high success rate, return and success rate are not the same. Return refers to the total reward over a trajectory, which is important for the theoretical analysis as well as how RL is trained in practice (since it yields a dense reward). Success rate is the standard evaluation metric in Meta-World and is used in our experiments. We will clarify this distinction in the revised version.", "claims_and_evidence": "The claims are primarily supported by empirical experiments.", "methods_and_evaluation_criteria": "The method, especially the algorithm is not clear to me.\n\nRegarding the clustering mechanism:\n - How many clusters are there in the committees for both experiments? Is the number of clusters predefined or learned automatically?\n - Can you provide an ablation study to analyze the impact of the number of clusters on performance?\n - Additionally, can you conduct an ablation study on the selection of key hyperparameters for the clustering mechanism?\n - Can you provide detailed analysis on what tasks are clustered together to verify the clustering mechanism works? \n\nRegarding few-shot generalization:\n - Will new committee member be added when learning new tasks? Or will it reuse existing member?", "theoretical_claims": "No I am not falimiar with the theoretical analysis.", "experimental_designs_or_analyses": "For comparison with baselines:\n - How is the number of parameters for PACKMAN compared with other baselines?\n - What is the optimal performance for each set of experiments? Providing this information would help in better understanding the reported performance metrics.\n\nFor experiments in metaworld, \n - Is the policy state-based or image-based?\n - What are the 30 and 20 tasks mentioned in the paper? How are they constructed? Are they randomly sampled?\n - Is the return equivalent to the success rate? Clarifying this would improve result interpretation.", "relation_to_broader_scientific_literature": "The problem is highly relevant to the field. It is a new method for multi-task learning", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "There are some references missing in line 335 and line 346.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers generally agreed that the paper provided strong theoretical contributions that support solid empirical results on two benchmarks (MuJoCo and Meta-World). Overall, the paper is clear, well-motivated, and offers impressive improvements over existing methods. There were some concerns regarding the method relying on parametric task representations (a common issue) and sensitivity to hyperparameters. After reading the paper myself, I also agree with one reviewer’s comment that the paper would benefit from a more explicit algorithm summarizing the entire approach even in the appendix — that would clarify exactly how all of the parts developed over the paper fit together into the final approach.
Decision: Accept (poster)