Policy Aggregation

Authors: parand a alamdari
soroush  ebadian
ariel d procaccia
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 19:38:37

Abstract: We consider the challenge of AI value alignment with multiple individuals that have different reward functions and optimal policies in an underlying Markov decision process. We formalize this problem as one of *policy aggregation*, where the goal is to identify a desirable collective policy. We argue that an approach informed by social choice theory is especially suitable. Our key insight is that social choice methods can be reinterpreted by identifying ordinal preferences with volumes of subsets of the *state-action occupancy polytope*. Building on this insight, we demonstrate that a variety of methods — including approval voting, Borda count, the proportional veto core, and quantile fairness — can be practically applied to policy aggregation.

Main paper: 
 introduction: 
 Early discussion of AI value alignment had often focused on learning desirable behavior from an individual teacher, for example, through inverse reinforcement learning [27, 1] . But, in recent years, the conversation has shifted towards aligning AI models with large groups of people or even entire societies. This shift is exemplified at a policy level by OpenAI's "democratic inputs to AI" program [41] and Meta's citizens' assembly on AI governance [8] , and at a technical level by the ubiquity of reinforcement learning from human feedback [30] as a method for fine-tuning large language models. We formalize the challenge of value alignment with multiple individuals as a problem that we view as fundamental -policy aggregation. Our starting point is the common assumption that the environment can be represented as a Markov decision process (MDP). While the states, actions and transition functions are shared by all agents, their reward functions -which incorporate values, priorities or subjective beliefs -may be different. In particular, each agent has its own optimal policy in the underlying MDP. Our question is this: How should we aggregate the individual policies into a desirable collective policy?A naïve answer is to define a new reward function that is the sum of the agents' reward functions (for each state-action pair separately) and compute an optimal policy for this aggregate reward function; such a policy would guarantee maximum utilitarian social welfare. This approach has a major shortcoming, however, in that it is sensitive to affine transformations of rewards, so, for example, if we doubled one of the reward functions, the aggregate optimal policy may change. This is an issue because each agent's individual optimal policy is invariant to (positive) affine transformations of rewards, so while it is possible to recover a reward function that induces an agent's optimal policy by observing their actions over time, 2 it is impossible to distinguish between reward functions that are affine transformations of each other. More broadly, economists and moral philosophers have long been skeptical about interpersonal comparisons of utility [19] due to the lack of universal scale -an issue that is especially pertinent in our context. Therefore, aggregation methods that are invariant to affine transformations are strongly preferred.Our approach. To develop such aggregation methods, we look to social choice theory, which typically deals with the aggregation of ordinal preferences. To take a canonical example, suppose agents report rankings over m alternatives. Under the Borda count rule, each voter gives m -k points to the alternative they rank in the k'th position, and the alternative with most points overall is selected.The voting approach can be directly applied to our setting. For each agent, it is (in theory) possible to compute the value of every possible (deterministic) policy, and rank them all by value. Then, any standard voting rule, such as Borda count, can be used to aggregate the rankings over policies and single out a desirable policy. The caveat, of course, is that this method is patently impractical, because the number of policies is exponential in the number of states of the MDP.The main insight underlying our approach is that ordinal preferences over policies have a much more practical volumetric interpretation in the state-action occupancy polytope O. Roughly speaking, a point in the state-action occupancy polytope represents a (stochastic) policy through the frequency it is expected to visit different state-action pairs. If a policy is preferred by an agent to a subset of policies O ′ , its "rank" is the volume of O ′ as a fraction of the volume of O. The "score" of a policy under Borda count, for example, can be interpreted as the sum of these "ranks" over all agents.Our results. We investigate two classes of rules from social choice theory, those that guarantee a notion of fairness and voting rules. By mapping ordinal preferences to the state-action occupancy polytope, we adapt the different rules to the policy aggregation problem.The former class is examined in Section 5. As a warm-up we start from the notion of proportional veto core; it follows from recent work by Chaudhury et al. [7] that a volumetric interpretation of this notion is nonempty and can be computed efficiently. We then turn to quantile fairness, which was recently introduced by Babichenko et al. [4] ; we prove that the volumetric interpretation of this notion yields guarantees that are far better than those known for the original, discrete setting, and we design a computationally efficient algorithm to optimize those guarantees.The latter class is examined in Section 6; we focus on volumetric interpretations of α-approval (including the ubiquitous plurality rule, which is the special case of α = 1) and the aforementioned Borda count. In contrast to the rules studied in Section 5, existence is a nonissue for these voting rules, but computation is a challenge, and indeed we establish several computational hardness results. To overcome this obstacle, we implement voting rules for policy aggregation through mixed integer linear programming, which leads to practical solutions. Finally, our experiments in Section 7 evaluate the policies returned by different rules based on their fairness; the results identify quantile fairness as especially appealing. The experiments also illustrate the advantage of our approach over rules that optimize measures of social welfare (which are sensitive to affine transformations of the rewards). 
related work: 
 Noothigattu et al. [28] consider a setting related to ours, in that different agents have different reward functions and different policies that must be aggregated. However, they assume that the agents' reward functions are noisy perturbations of a ground-truth reward function, and the goal is to learn an optimal policy according to the ground-truth rewards. In social choice terms, our work is akin to the typical setting where subjective preferences must be aggregated, whereas the work of Noothigattu et al. [28] is conceptually similar to the setting where votes are seen as noisy estimates of a ground-truth ranking [39, 9, 6] .Chaudhury et al. [7] study a problem completely different from ours: fairness in federated learning. However, their technical approach served as an inspiration for ours. Specifically, they consider the proportional veto core and transfer it to the federated learning setting using volumetric arguments, by considering volumes of subsets in the space of models. Their proof that the proportional veto core is nonempty carries over to our setting, as we explain in Section 5.There is a body of work on multi-objective reinforcement learning (MORL) and planning that uses a scalarization function to reduce the problem to a single-objective one [32, 21] . Other solutions to MORL focus on developing algorithms to identify a set of policies approximating the problem's Pareto frontier [38] . A line of work more closely related to ours focuses on fairness in sequential decision making, often taking the scalarization approach to aggregate agents' preferences by maximizing a (cardinal) social welfare function, which maps the vector of agent utilities to a single value. Ogryczak et al. [29] and Siddique et al. [34] investigate generalized Gini social welfare, and Mandal and Gan [25] , Fan et al. [13] and Ju et al. [23] focus on Nash and egalitarian social welfare. Alamdari et al. [3] study this problem in a non-Markovian setting, where fairness depends on the history of actions over time, and introduce concepts to assess different fairness criteria at varying time intervals. A shortcoming of these solutions is that they are not invariant to affine transformations of rewards -a property that is crucial in our setting, as discussed earlier.Our work is closely related to the pluralistic alignment literature, aiming to develop AI systems that reflect the values and preferences of diverse individuals [35, 10] . Alamdari et al. [2] propose a framework in reinforcement learning in which an agent learns to act in a way that is considerate of the values and perspectives of humans within a particular environment. Concurrent work explores reinforcement learning from human feedback (RLHF) from a social choice perspective, where the reward model is based on pairwise human preferences, often constructed using the Bradley-Terry model [5] . Zhong et al. [42] consider the maximum Nash and egalitarian welfare solutions, and Swamy et al. [36] propose a method based on maximal lotteries due to Fishburn [14] . 
preliminaries: 
 For t ∈ N, let [t] = {1, 2, . . . , t}. For a closed set S, let ∆(S) denote the probability simplex over the set S. We denote the dot product of two vectors as ⟨x, y⟩ = d i=1 x i • y i for x, y ∈ R d . A halfspace in R d determined by w ∈ R d and b ∈ R is the set of points satisfying {x ∈ R d | ⟨x, w⟩ ⩽ b}. A polytope O ⊆ R d isA multi-objective Markov decision process (MOMDP) is a tuple defined as M = ⟨S, A, P, R 1 , . . . , R n ⟩ for the average-reward case and ⟨S, d init , A, P, R 1 , . . . , R n , γ⟩ for the discounted-reward case, where S is a finite set of states, A is a finite set of actions, and P : (S × A) → ∆(S) is the transition probability distribution. P(s t , a t , s t+1 ) is the probability of transitioning to state s t+1 by taking action a t in s t . For i ∈ [n], R i : S × A → R is the reward function of the ith agent, the initial state is sampled from d init ∈ ∆(S), and γ ∈ (0, 1] is the discount factor. A (Markovian) policy π(a|s) is a probability distribution over the actions a ∈ A given the state s ∈ S. A policy is deterministic if at each state s one action is selected with probability of 1, and otherwise it is stochastic. The expected average return of agent i for a policy π and the expected discounted return of agent i for a policy π are defined over an infinite time horizon asJ avg i (π) = lim T →∞ 1 T E π,P T t=1 R i (s t , a t ) , J γ i (π) = (1-γ) E π,P ∞ t=1 γ t R i (s t , a t )| s1∼dinitwhere the expectation is over the state-action pairs at time t based on both the policy π and the transition function P. Definition 1 (state-action occupancy measure). Let P t π be the probability measure over states at time t under policy π. The state-action occupancy measure for state s and action a is defined asd avg π (s, a) = lim T →∞ 1 T E T t=1 P t π (s)π(a|s) , d γ π (s, a) = (1 -γ) E ∞ t=1γ t P t π (s)π(a|s) .For both the average and discounted cases, we can rewrite the expected return as the dot product of the state-action occupancy measures and rewards, that is,J i (π) = (s,a) d π (s, a) • R i (s, a) = ⟨d π , R i ⟩.In addition, the policy can be calculated given the occupancy measure as π(a|s) = d π (s, a)/( a d π (s, a)) if a d π (s, a) > 0, and π(a|s) = 1/|A| otherwise. Definition 2 (state-action occupancy polytope [31, 40] ). For a MOMDP M in the average-reward case, the space of valid state-action occupancies is the polytopeO avg = d avg π | d avg π ⩾ 0, s,a d avg π (s, a) = 1, a d avg π (s, a) = s ′ ,a ′ P(s ′ , a ′ , s)d avg π (s ′ , a ′ ), ∀s ∈ S .We similarly define this polytope for the discounted-reward case in Appendix A.A mechanism receives a MOMDP and aggregates the agents' preferences into a policy. An economical efficiency axiom in the social choice literature is that of Pareto optimality. Definition 3 (Pareto optimality). For a MOMDP M and ϵ ⩾ 0, a policy π is ϵ-Pareto optimal if there does not exist another policy π ′ such that J i (π ′ ) ⩾ J i (π) + ϵ for all i ∈ N , with strict inequality for at least one agent. For ϵ = 0, we simply call such policies Pareto optimal.We call a mechanism Pareto optimal if it always returns a Pareto optimal policy. In special cases where all agents unanimously agree on an optimal policy, Pareto optimality implies that the mechanism will return one such policy. We discuss Pareto optimal implementations of all mechanisms in this work.In the classical social choice setting, we have a set of n agents and a set C of m alternatives. The preferences of voter i ∈ [n] is represented as a strict ordering over the alternativesσ i : [m] → C equal to σ i (1) ≻ i σ i (2) ≻ i . . . ≻ i σ i (m), where c 1 ≻ i c 2 denotes agent i prefers c 1 over c 2 for c 1 , c 2 ∈ C.A (possibly randomized) voting rule aggregates agents' preferences and returns an alternative or a distribution over the alternatives as the collective decision.Positional Scoring Rules. A positional scoring rule with scoring vector ⃗ s = (s 1 , . . . , s m ) such that s 1 ⩾ s 2 ⩾ . . . ⩾ s m works as follows. Each agent gives a score of s 1 to their top choice, a score of s 2 to their second choice, and so on. The votes are tallied and an alternative with the maximum total score is selected. A few of the well-known positional scoring rules are: Plurality: (1, 0, . . . , 0), Borda: (m -1, m -2, . . . , 1, 0), k-approval: (1, . . . , 1, 0, . . . , 0) with k ones. 
occupancy polytope as the space of alternatives: 
 In a MOMDP M , each agent i incorporates their values and preferences into their respective reward function R i . Agent i prefers π over π ′ if and only if π achieves higher expected return, J i (π) > J i (π ′ ), and is indifferent between two policies π and π ′ if and only if J i (π) = J i (π ′ ). As discussed before, given a state-action occupancy measure d π in the state-action occupancy polytope O, we can recover the corresponding policy π. Therefore, we can interpret O as the domain of all possible alternatives over which the n agents have heterogeneous weak preferences (with ties). Agent i prefers d π to d π ′ in O if and only if they prefer π to π ′ . We study the policy aggregation problem through this lens; specifically, we design or adapt voting mechanisms where the (continuous) space of alternatives is O and agents have weak preferences over them determined by their reward functions R i .Affine transformation and reward normalization. A particular benefit of this interpretation, as mentioned before, is that all positive affine transformations of the reward functions, i.e., aR i + b for all a ∈ R ⩾0 and b ∈ R, yield the same weak ordering over the polytope. Hence, we can assume without loss of generality that J i (π) ∈ [0, 1]. Further, we can ignore agents that are indifferent between all policies, i.e., min π J i (π) = max π J i (π), and normalize reward functionsR i ← Ri-minπ Ji(π)maxπ Ji(π)-minπ Ji(π) such that min π J i (π) = 0 and max π J i (π) = 1. The relative ordering of the policies does not change since for all points d π ∈ O we have s,a d π (s, a) = 1. Volumetric definitions. A major difference between voting over a continuous space of alternatives and the classical voting setting is that the domain of alternatives is infinite and not all voting mechanisms can be directly applied to the policy aggregation problem. In particular, various voting rules require reasoning about the rank of an alternative or the size of some subset of alternatives. For instance, the Borda count of an alternative c over a finite set of alternatives is defined as the number (or fraction) of candidates ranked below c. In the continuous setting, for almost all of the mechanisms that we discuss later, we use the measure or volume of a subset of alternatives to refer to their size. For a measurable subset O ′ ⊆ O, let vol(O ′ ) denote its measure. The ratio vol(O ′ )/vol(O) is the fraction of alternatives that lie in O ′ . A probabilistic interpretation is that for a uniform distribution over the polytope O, vol(O ′ )/vol(O) denotes the probability that a policy uniformly sampled from O lies in O ′ . We can also define the expected return distribution of an agent over O as a random variable that maps a policy to its expected return, i.e., one that maps d π ∈ O to ⟨d π , R i ⟩. The pdf and cdf of this r.v. is defined below. Definition 4 (expected return distribution). For a MOMDP M and v ∈ R, the expected return distribution of agent i ∈ [n] is defined asf i (v) = 1 vol(O) x∈O δ(v -⟨x, R i ⟩) dx, F i (v) = v x=-∞ f i (x) dx = vol(O ∩ {⟨x, R i ⟩ ⩽ v}) vol(O) ,where f i and F i are the pdf and cdf of the expected return distribution and δ(•) is the Dirac deltafunction indicating v = ⟨x, R i ⟩.A useful observation about f i , the pdf, is that it is unimodal, i.e., increasing up to its mode mode(f i ) ∈ arg max v∈R f i (v) and decreasing afterwards, which follows from the Brunn-Minkowski inequality [17] . Since f i (pdf) is the derivative of F i (cdf), the unimodality off i implies that F i is a convex function in (-∞, mode(f i )] and concave in [mode(f i ), ∞).In our algorithms, we use a subroutine that measures the volume of a polytope, which we denote by vol-comp({Ax ⩽ b}). Dyer et al. [12] designed a fully polynomial time randomized approximation scheme (FPRAS) for computing the volume of polytopes. We report the running time of algorithms in terms of the number of calls to this oracle. 
fairness in policy aggregation: 
 In this section, we utilize the volumetric interpretation of the state-action occupancy polytope to extend fairness notions from social choice to policy aggregation, and we develop algorithms to compute stochastic policies provably satisfying these notions.The proportional veto core was first proposed by Moulin [26] in the classical social choice setting with a finite set of alternatives where agents have full (strict) rankings over the alternatives. For simplicity, suppose the number of alternatives m is a multiple of n. The idea of the proportional veto core is that x% of the agents should be able to veto x% of the alternatives. More precisely, for an alternative c to be in the proportional veto core, there should not exist a coalition S that can "block" c using their proportional veto power of |S| /n. S blocks c if they can unanimously suggest m(1 -|S| /n) candidates that they prefer to c. For instance, if c is in the proportional veto core, it cannot be the case that a coalition of 60% of the agents unanimously prefer 40% of the alternatives to c.Chaudhury et al. [7] extended this notion to a continuous domain of alternatives in the federated learning setting. We show that such an extension also applies to policy aggregation. Definition 5(proportional veto core). Let ϵ ∈ (0, 1 /n). For a coalition of agents S ⊆ [n], let veto(S) = |S| /n be their veto power. A point d π ∈ O is blocked by a coalition S if there exists a subset O ′ ⊆ O of measure vol(O ′ )/vol(O) ⩾ 1 -veto(S) + ϵ such that all agents in S prefer all points in O ′ to d π , i.e., d π ′ ≻ i d π for all d π ′ ∈ O ′ and i ∈ S. A point d π is in the ϵ-proportional veto core if it is not blocked by any coalition.A candidate in the proportional veto core satisfies desirable properties that are extensively discussed in prior work [26, 7, 24, 22] . It is worth mentioning that any candidate in the ϵ-proportional veto core, besides the fairness aspect, is also economically efficient as it satisfies ϵ-Pareto optimality. This holds since the grand coalition S = [n] can veto any ϵ-Pareto dominated alternative.Moulin [26] proved that the proportional veto core is nonempty in the discrete setting and Chaudhury et al. [7] proved it for the continuous setting. The following result is a corollary of Theorem 1 of Chaudhury et al. [7] .ALGORITHM 1: Seq. ϵ-Prop. Veto Core [7] O0 ← O, δ ← 1 n -ϵ n+1 for i = 1 to n do Using binary search find v * i ∈ [0, 1] s.t. vol(Oi-1 ∩ {⟨x, Ri⟩ ⩽ v * i }) ≈ δ • vol(O) Oi ← Oi-1 ∩ {⟨x, Ri⟩ ⩾ v * i } return a welfare maximizing point dπ ∈ On ALGORITHM 2: ϵ-Max Quantile Fairness Procedure q-quantile-feasible(q): Oq ← {x ∈ O | ⟨x, Ri⟩ ⩾ F -1 i (q), i ∈ [n]} if Oq is a feasible linear program then returnTrue else return False Using binary search find maximum q ∈ [0, 1] s.t.q-quantile-feasible() is feasible return a welfare maximizing point dπ ∈ Oq Theorem 1. Let ϵ ∈ (0, 1/n). For a policy aggregation problem, the ϵ-proportional veto core is nonempty. Furthermore, such policies can be found in polynomial time using O(log(1/ϵ)) many calls per agent to vol-comp.We refer the reader to the paper of Chaudhury et al. [7] for the complete proof, and provide a high-level description of Algorithm 1, which finds a point in the proportional veto core. Algorithm 1 iterates over the agents and lets the ith agent "eliminate" roughly 1 /n • vol(O) of the remaining space of alternatives. That is, agent i finds the hyperplaneH i = {⟨x, R i ⟩ ⩽ v *i } such that its intersection with O i-1 (the remaining part of O at the ith iteration) has a volume of approximately vol(O)/n. This value, for each agent, can be found by doing a binary search over v * i to a precision of [v * i -ϵ, v * i ] by O(log(1/ϵ)) calls to the volume estimating subroutine vol-comp. 3Pareto optimality. We briefly discuss why Algorithm 1 is Pareto optimal. During the ith iteration, the space of policies is contracted by adding a linear constraint of the form J i (π) ⩾ v * i . If the returned welfare-maximizing policy π (derived from d π ) is Pareto dominated by another policy π ′ , then π ′ would satisfy all these linear constraints asJ i (π ′ ) ⩾ J i (π) ⩾ v *i with the earlier inequality being strict for at least one agent. Therefore, d π ′ ∈ O n and π ′ achieves a higher social welfare, which is a contradiction. The same argument can be used to establish the Pareto optimality of other mechanisms discussed later; each of these mechanisms searches for a policy that satisfies certain lower bounds on agents' utilities, from which a welfare-maximizing, Pareto optimal policy can be selected.Next, we consider an egalitarian type of fairness based on the occupancy polytope, building on very recent work by Babichenko et al. [4] in the discrete setting; surprisingly, we show that it is possible to obtain stronger guarantees in the continuous setting. Babichenko et al. [4] focus on the fair allocation of a set of m indivisible items among n agents, where each item must be fully allocated to a single agent. They quantify the extent an allocation A is fair to an agent i by the fraction of allocations over which i prefers A (note that the number of all discrete allocations is n m ). In other words, if one randomly samples an allocation, the fairness is measured by the probability that A is preferred to the random allocation. An allocations is q-quantile fair for q ∈ [0, 1] if all agents consider this allocation among their top q-quantile allocations. Babichenko et al. [4] aim to find a universal value of q such that for any fair division instance, a q-quantile fair allocation exists. They make an interesting connection between q-quantile fair allocations and the Erdős Matching Conjecture, and show under the assumption that the conjecture holds, (1/2e)-quantile fair allocations exist for all instances. 4We ask the same question for policy aggregation, and again, a key difference is that our domain of alternatives is continuous. The notion of q-quantile fairness extends well to our setting. Agents assess the fairness of a policy π based on the fraction of the occupancy polytope (i.e., the set of all policies) to which they prefer π, or equivalently, the probability that they prefer the chosen policy to a randomly sampled policy. Definition 6 (q-quantile fairness). For a MOMDP M and q ∈ [0, 1], a policy π is q-quantile fair if for every agent i ∈[n], π is among i's top q-fraction of policies, vol(O∩{⟨x, R i ⟩ ⩽ J i (π)}) ⩾ q•vol(O).We show that a (1/e)-quantile fair policy always exist and that this ratio is tight; note that this bound is twice as good as that of Babichenko et al. [4] , and it is unconditional. We prove this by making a connection to an inequality due to Grünbaum [17] . The centroid of a polytope P is defined as centroid(P ) = x∈P x dx x∈P 1 dx . Lemma 1 (Grunbaum's Inequality). Let P be a polytope and w a direction in R d . For the halfspace H = {x | ⟨w, x -centroid(P )⟩ ⩾ 0}, it holds that1 e ⩽ d d + 1 d ⩽ vol(P ∩ H) vol(P ) ⩽ 1 - d d + 1 d ⩽ 1 - 1 e ,Furthermore, this is tight for the n-dimensional simplex. Theorem 2. For every MOMDP M , there always exist q-quantile fair policy where q = ( ℓ-1 ℓ ) ℓ-1 and ℓ = |S| • |A|. Note that q ⩾ 1 e ≈ 36.7%. Furthermore, this bound is tight: For any ℓ, there is an instance with a single state and ℓ actions where no q-quantile fair policy exists for any q > ( ℓ-1 ℓ ) ℓ-1 .Proof. First, we show that the centroid of the occupancy polytope c = centroid(O) is q-quantile fair policy for the aformentioned q. Since O is a subset of the (n -1)-simplex (see Definition 2), O has a nonzero volume in some lower dimensional space ℓ ′ ⩽ |S| • |A| -1. By invoking Grunbaum's inequality (Lemma 1) with w i being equal to R i projected to the ℓ ′ -dimensional subspace for allagents i ∈ [n], we have that vol(O ∩ H i ) ⩾ ( ℓ ′ ℓ ′ +1 ) ℓ ′ • vol(O) where H i = {⟨x -c, w i ⟩ ⩾ 0} = {⟨x, R i ⟩ ⩾ J i (c)}. Since ℓ ′ ⩽ ℓ -1, we have ( ℓ ′ ℓ ′ +1 ) ℓ ′ ⩾ ( ℓ-1 ℓ ) ℓ-1, which completes the proof. To show tightness, take a MOMDP with a single state S = {s} -hence a constant transition function P(•) = s -and ℓ actions A = {a 1 , . . . , a ℓ } and ℓ agents. The reward function of agent i is R i (s, a i ) = 1 and 0 otherwise. The state-action occupancy polytope of this MOMDP is the(ℓ -1)-dimensional simplex O = { a d π (s, a) = 1, d π ∈ R 1×ℓ ⩾0 }. Take any point in d π ∈ O.There exists at least one agent i that hasJ i (d π ) = d π (s, a i ) ⩽ 1 ℓ . Take the halfspace H i = {⟨x, R i ⟩ ⩾ 1 ℓ }. Observe that H i ∩ O is equivalent to a smaller (ℓ -1)-dimensional simplex { a d π (s, a) = 1 -1 ℓ } which has volume of vol (1 -1 ℓ )O = ℓ-1 ℓ ℓ-1 vol(O). Therefore, vol(O∩Hi) vol(O) = ℓ-1 ℓ ℓ-1 .The centroid of the occupancy polytope, as per Theorem 2, attains a worst-case measure of quantilefairness. However, the centroid policy can be highly suboptimal as it disregards the preferences of the agents involved. For instance, there could exist a 99%-quantile fair policy. To this end, we take an egalitarian approach and aim to find a q-quantile fair policy with the maximum q.Max quantile fair algorithm. Algorithm 2 searches for the optimal value q * , for which a q * -quantile fair policy exists, and gets close to q * by a binary search. To perform the search, we need a subroutine that checks, for a given value of q, if a q-quantile fair policy exists. Suppose we have the q-quantile expected return F -1 i (q) for all i, that is, the expected return amount v i such that F i (v i ) = q. Then, the problem of existence of a q-quantile fair policy is equivalent to the feasibility of the linearprogram {x ∈ O | ⟨x, R i ⟩ ⩾ F -1 i (q), i ∈ [n]}, which can be solved in polynomial time. Importantly, after finding a good approximation of q, there can be infinitely many policies that are q-quantile fair and there are various ways to select the final policy after finding the value q. As mentioned earlier, a desirable efficiency property is Pareto optimality; to satisfy it, one can return the policy that maximizes the sum of agents' expected returns among the ones that are q-quantile fair. Finally, to calculate F -1 i (q), we can again use binary search to get δ close to the value v i for which F i (v i ) = q using O(log 1/δ) calls to vol-comp. The discussion above is summarized below. Proposition 3. Assuming an optimal oracle for F -1 i , Algorithm 2 finds a q-quantile fair policy that is ϵ close to the optimal value in polynomial time with O(log(1/ϵ)) per agent calls to the oracle. A δ-approximation to F -1 i (q) can be computed using O(log(1/δ)) calls to vol-comp. 
policy aggregation with voting rules: 
 In this section, we adapt existing voting rules from the discrete setting to policy aggregation and discuss their computational complexity.ALGORITHM 3: α-Approvals MILP compute F -1 i (α) for all i ∈ [n] solve the mixed integer linear program max i∈[n] ai s.t. ai • F -1 i (α) ⩽ ⟨dπ, Ri⟩ ∀i ∈ [n] dπ ∈ O, ai ∈ {0, 1} ∀i ∈ [n]return a Pareto optimal policy subject to max α-approval {i | ai = 1}ALGORITHM 4: ϵ-Borda count MILP compute F -1 i (kϵ) for all i ∈ [n] and k ∈ [ 1 ϵ ] solve the mixed integer linear program max i∈[n] k∈[1/ϵ] a i,k • (Fi(kϵ) -Fi((k -1)ϵ)) s.t. a i,k • kϵ ⩽ ⟨dπ, Ri⟩ ∀i ∈ [n] dπ ∈ O, a i,k ∈ {0, 1}, ∀i ∈ [n], k ∈ [1/ϵ]return a Pareto optimal policy subject to max Borda scorePlurality. The plurality winner is the policy that achieves the maximum number of plurality votes or "approvals," where agent i approves a policy π if it achieves their maximum expected returnJ i (π) = max π ′ J i (π ′ ) = 1.Hence the plurality winner is a policy in arg maxπ i∈[n] I[J i (π) = 1].This formulation does not require the volumetric interpretation. However, in contrast to the discrete setting where one can easily count the approvals for all candidates, we show that solving this problem in the context of policy aggregation is not only NP-hard, but hard to approximate up to factor of a 1/n 1/2-ϵ . We establish the hardness of approximation by a reduction from the maximum independent set problem [20] ; we defer the proof to Appendix B. Theorem 4. For any fixed ϵ ∈ (0, 1), there is no polynomial-time 1 n 1/2-ϵ -approximation algorithm for the maximum plurality score unless P = NP.Nevertheless, we can compute plurality in practice, as we discuss below. α-approval. We extend the k-approval rule using the volumetric interpretation of the occupancy polytope, similarly to the q-quantile fairness definition. For some α ∈ [0, 1], agents approve a policy π if its return is among their top α fraction of O, i.e., F i (J i (π)) ⩾ α. The α-approval winner is a policy that has the highest number of α-approvals, so it is in arg maxπ i∈[n] I[F i (J i (π)) ⩾ α].Note that plurality is equivalent to 1-approval. It is worth mentioning that there can be infinitely many policies that have the maximum approval score and, to avoid a suboptimal decision, one can return a Pareto optimal solution among the set of α-approval winner policies.Theorem 2 shows that for α ⩽ 1/e, there always exists a policy that all agents approve, and by Proposition 3 such policies can be found in polynomial time, assuming access to an oracle for volumetric computations. Therefore, the problem of finding an α-approval winner is "easy" for α ∈ (0, 1/e). In sharp contrast, for α = 1 -namely, plurality -Theorem 4 gives a hardness of approximation. The next theorem shows the hardness of computing α-approval for α ∈ (7/8, 1] via a reduction from the MAX-2SAT problem. We defer the proof to Appendix B. Theorem 5. For α ∈ (7/8, 1], computing a policy with the highest α-approval score is NP-hard. This even holds for binary reward vectors and when every F i has a closed form.Given the above hardness result, to compute the α-approval rule, we turn to mixed-integer linear programming (MILP). Algorithm 3 simply creates n binary variables for each agent i indicating whether i α-approves the policy, i.e., F i (J i (π)) ⩾ α which is equivalent to J i (π) ⩾ F -1 i (α). To encode the expected return requirement for agent i to approve a policy as a linear constraint, we precompute F -1 i (α). This can be done by a binary search similar to Algorithm 2. Importantly, Algorithm 3 has one binary variable per agent and only n constraints which is key to its practicability.Borda count. The Borda count rule also has a natural definition in the continuous setting. In the discrete setting, the Borda score of agent i for alternative c is the number of alternatives c ′ such that c ≻ i c ′ . In the continuous setting, F i (J i (π)) indicates the volume of the occupancy polytope to which agent i prefers π. The Borda count rule then selects a policy among arg max π i∈[n] F i (J i (π)).The computational complexity of the Borda count rule remains an interesting open question, though we make progress on two fronts. 5 First, we identify a sufficient condition under which we can find an approximate max Borda count policy using convex optimization in polynomial time. Second, similar to Algorithm 3, we present a MILP to approximate the Borda count rule in Algorithm 4.The first is based on the observation in Section 4 that F i is concave in range [mode(f i ), ∞). We assume that the max Borda count policy π appears in the concave portion of all agents, i.e., J i (π) ⩾ mode(f i ) for all i ∈ [n]. Then, the problem becomes a maximization of the concave objectivemax i F i (⟨d π , R i ⟩) over the convex domain {d π ∈ O | ⟨d π , R i ⟩ ⩾ mode(f i ), ∀i ∈ [n]}.Second, Algorithm 4 is a MILP that finds an approximate max Borda count policy. As a preprocessing step, we estimate F i for each agent i separately. We measure F i for the fixed expected return values of {ϵ, 2ϵ, . . . , 1 -ϵ, 1}. This accounts for 1/ϵ oracle calls to vol-comp per agent. Then, for the MILP, we introduce 1/ϵ binary variables for each agent indicating their ϵ-rounded return levels, i.e., a i,k = 1 iff ⟨d π , R i ⟩ ⩾ kϵ for k ∈ [1/ϵ]. The MILP then searches for an occupancy measure d π ∈ O with maximum total Borda score among the ϵ-rounded expected return vectors (see Appendix C for more details).Finally, we make a novel connection between q-quantile fairness and Borda count in Theorem 6. We defer the proof to Appendix B. A corollary of Theorems 2 and 6 is that the policy returned by ϵ-max quantile fair algorithm (Algorithm 2) achieves a (1/e -ϵ) multiplicative approximation of the maximum Borda score. Theorem 6. A q-quantile fair policy is a q-approximation of the maximum Borda score.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We describe all the essential details for reproducing the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiments: 
 Environment. We adapt the dynamic attention allocation environment introduced by D'Amour et al. [11] . We aim to monitor several sites and prevent potential incidents, but limited resources prevent us from monitoring all sites at all times; this is inspired by applications such as food inspection and pest control 6 . There are m = 5 warehouses and each can be in 3 different stages: normal (norm), risky (risk) and incident (inc). There are |S| = 3 m states containing all possible stages of all warehouses. In each step, we can monitor at most one site, so there are m + 1 actions, where action m + 1 is no operation and action i ⩽ m is monitoring warehouse i. There are n agents; each agent i has a list ℓ i of warehouses that they consider valuable and a reward function R i . In each step t, R i (s t , a t ) = -I[a t ⩽ m] -j∈ℓi ρ i w j • I[s t,j = inc ∧ a t ̸ = j], where w j ∈ {100, 150, • • • , 250} denotes the penalty of an incident occurring in warehouse j, ρ i is the scale of penalties for agent i which is sampled from {0.25, 0.5, • • • , n}, and -1 is the cost of monitoring. In each step, if we monitor warehouse j, its stage becomes normal. If not, it changes from norm to risk and from risk to inc with probabilities p j,risk and p j,inc , and stays the same otherwise. Probabilities are sampled i.i.d. uniformly from [0.5, 0.8]. The state transitions P is the product of the warehouses' stage transitions.Rules. We compare the outcomes of policy aggregation with different rules: max-quantile, Borda, α-approval (α = 0.9, 0.8), egalitarian (maximize minimum return) and utilitarian (maximize sum of returns). We sample 5 • 10 5 random policies based on which we fit a generalized logistic function to estimate the cdf of the expected return distribution F i (Definition 4) for every agent. The policies for α-approval voting rules are optimized with respect to maximum utilitarian welfare. The egalitarian rule finds a policy that maximizes the expected return of the worst-off agent, then optimizes for the second worst-off agent, and so on. The implementation details of Borda count are in Appendix D.In Figure 1 , we report the normalized expected return of agents asJi(π)-minπ Ji(π) max π ′ Ji(π ′ )-min π ′′ Ji(π ′′ )(sorted from lowest to highest) which are averaged over 10 different environment and agents instances. We observe that the utilitarian and egalitarian rules are sensitive to the different agents' reward scales and tend to perform unfairly. The utilitarian rule achieves the highest utilitarian welfare by almost ignoring one agent. The egalitarian rule achieves higher return for the worst-off agents compared to the utilitarian rule, but still yields an inequitable outcome. The max-quantile rule tends to return the fairest outcomes with similar normalized returns for the agents. The Borda rule, while not a fair rule by design, tends to find fair outcomes which are slightly worse than the max-quantile rule. The α-approval rule with max utilitarian completion tends to the utilitarian rule as α → 0 and to plurality as α → 1. Importantly, although not shown in the plots, the plurality rule ignores almost all agents and performs optimally for a randomly selected agent.In addition to the fine-grained utility distributions, in Table 1 , we report two aggregate measures based on agents' utilities: (i) the Gini index, a statistical measure of dispersion defined max-q borda 90%-a 80%-a egal -where a lower Gini index indicates a more equitable distribution, and(ii) the Nash welfare, defined as the geometric mean of agents' utilities i∈N J i (π) 1/n -where a higher Nash welfare is preferable. We observe a similar trend as above, where utilitarian and egalitarian rules perform worse across both metrics. For the other four rules, the Nash welfare scores are comparable in both scenarios, with Borda showing slightly better performance. The Gini index, however, highlights a clearer distinction among the rules, with max-quantile performing better.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: Yes. They are included in the plots, and the results are reported based on the average of multiple runs.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We discuss the potential for positive societal impact through better AI alignment. While it is conceivable that our techniques would lead to worse AI alignment, this scenario seems implausible. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
discussion: 
 We conclude by discussing some of the limitations of our approach. A first potential limitation is computation. When we started our investigation of the policy aggregation problem, we were skeptical that ordinal solutions from social choice could be practically applied. We believe that our results successfully lay this initial concern to rest. However, additional algorithmic advances are needed to scale our approach beyond thousands of agents, states, and actions. Additionally, an interesting future direction is to apply these rules within continuous state or action spaces, as well as in online reinforcement learning setting where the environment remains unknown.A second limitation is the possibility of strategic behavior. The Gibbard-Satterthwaite Theorem [16, 33] precludes the existence of "reasonable" voting rules that are strategyproof, in the sense that agents cannot gain from misreporting their ordinal preferences; we conjecture that a similar result holds for policy aggregation in our framework. However, if reward functions are obtained through inverse reinforcement learning, successful manipulation would be difficult: an agent would have to act in a way that the learned reward function induces ordinal (volumetric) preferences leading to a higher-return aggregate stochastic policy. This separation between the actions taken by an agent and the preferences they induce would likely alleviate the theoretical susceptibility of our methods to strategic behavior.A State-action Occupancy Polytope for Discounted Reward Definition 7 (State-action Occupancy Polytope for discounted-reward [31, 40] ). For a MOMDP M in the discounted-reward case, the space of valid state-action occupancies is the polytopeO γ = d γ π | d γ π ⩾ 0, a d γ π (s, a) = (1 -γ)d init (s) + γ s ′ ,a ′ P(s ′ , a ′ , s)d γ π (s ′ , a ′ ), ∀s ∈ SWe define a fully connected MOMDP as a MOMDP M where for every pair of states s, s ′ ∈ S and any action a ∈ A, P(s, a, s ′ ) = 1 |S| . In all the hardness proofs, we create a fully connected MOMDP. For such MOMDPs, it is not difficult to observe that the state-action occupancy polytope, for both the average and discounted reward, is equivalent to{d π | a d π (s, a) = 1|S| , ∀s ∈ S} for both the discounted and average reward case. Furthermore,EQUATIONB.1 Proof of Theorem 4Proof of Theorem 4. We show hardness of approximation by a approximation-preserving reduction from the maximum independent set (MIS) problem.Definition 8 (maximum independent set (MIS)). For a graph G = (V, E) with vertex set V and edge set E ⊆ 2 ( V 2 ) , the maximum independent set (MIS) of G is the maximum subset of vertices V ′ ⊆ V such that there are no edges between any pair of vertices v 1 , v 2 ∈ V ′ .Theorem 7 (Håstad [20] ). For any fixed ϵ ∈ (0, 1), there is no polynomial-time 1 n 1/2-ϵ -approximation algorithm for the MIS problem unless P = NP, and no 1 n 1-ϵ -approximation algorithm unless ZPP = NP.Let G = ([n], E) be a graph for which we want to find the maximum independent set. Create a fully connected MOMDP M with |E| states {s e }, one per edge e ∈ E. There are only two actions A = {a 1 , a 2 }. In state s e of edge e = (e 1 , e 2 ), performing action a 1 and a 2 correspond to e 1 and e 2 respectively. We create n agents where agent i corresponds to vertex i. The reward function of agent i for state s e and action a k for k ∈ {1, 2} is defined asEQUATIONIn other words, the reward functions encode the set of edges incident to vertex i.A policy π is optimal for agent i iff for all the edges incident to i the action corresponding to i is selected with probability 1. If a policy π is considered optimal by two agents i and i ′ , then e = (i, i ′ ) / ∈ E, since at state s e either π(a 1 |s) = 1 or π(a 2 |s) = 1. Therefore, the set of agents that consider a policy optimal corresponds to an independent set in G. Furthermore, take any independent set V ′ ⊆ [n]. Let π be the policy that for each edge e selects the action corresponding to the vertex in V ′ and, if no such vertices exist, select one arbitrarily. This policy is well defined since V ′ is an independent set and there are no edges with both vertices from V ′ . Policy π is optimal for agents of V ′ since at each state their favourite action is selected. Thus, we have an equivalence between the maximum independent set of G and the plurality winner policy of M . Therefore, the hardness of approximation follows from Theorem 7.Proof of Theorem 5. We reduce the MAX-2SAT problem to finding an α-approval policy for α ∈ (7/8, 1].For two Boolean variables x 1 and x 2 , we denote the disjunction (i.e., logical or) of two variables by x 1 ∨ x 2 and the negation of x 1 by ¬x 1 . Garey et al. [15] showed that the MAX-2SAT problem is NP-hard.For an instance of the MAX-2SAT problem, let {C 1 , . . . , C n } be a set of m 2-literal disjunction clauses over n variables {x 1 , . . . x m }. We create a fully connected MOMDP M with m states -state s j representing variable x j . There are only two actions, a True and a False which at state s j correspond to setting variable x j to True and False respectively. This way, a policy π(a True |s j ) can be interpreted as the probability of setting the variable x j to True, subject to π(a True |s j ) = 1 -π(a False |s j ).Agents and reward function. We introduce some notation before introducing the agents and their reward function. Take a clauseC i = c i,1 ∨ c i,2 where c i,k ∈ {x j , ¬x j } j∈[m] for k ∈ [2]. By combining the former relation with the mapping of x j and ¬x j to state-action pairs (s j , a True ) and (s j , a False ) respectively, we define λ : {c i,1 , c i,2 } → S × A. For every c i,k , λ(c i,k ) is the state-action pair that evaluates c i,k to True when selected with probability one. Now, we are ready to introduce the agents. For each clause C i , we create 3 agents per clause as follows:• Agent ag i,1 with a reward function that is 1 for λ(c i,1 ) and λ(c i,2 ) and zero otherwise. An optimal policy of this agent selects the two state-action pairs with probability one, which can be interpreted as c i,1 ← True and c i,2 ← True that implies C i = True.• Agent ag i,2 with a reward function that is 1 for λ(c i,1 ) and λ(¬c i,2 ) -note the negation. This is another assignment of variables, c i,1 ← True and c i,2 ← False, that implies C i ← True.• Similarly, the reward function of ag i,3 is 1 for λ(¬c i,1 ) and λ(c i,2 ) (which again impliesC i ← True).For ease of exposition, and by a slight abuse of notation, in the rest of the proof we use π to refer to the occupancy measure. From Equation (1) we have |S| • d π (s, a) = π(a|s) and the α-approval rule is invariant to affine transformation. Therefore, we let J i (π) = ⟨π, R⟩.Expected return distribution. The expected return of agent ag i,1 for a policy π is π(λ(c i,1 )) + π(λ(c i,2 )). For conciseness, let v 1 = π(λ(c i,1 )), v 2 = π(λ(c i,2 )). Then, the cdf isF ag i,1 (v) = v v1=0 v v2=0 I[v 1 + v 2 ⩽ v] • dv 1 dv 2 = v 2 2 for v ∈ [0, 1], 1 -(2-v) 2 2 for v ∈ [1, 2].See Figure 2 for a visualization. The cdf of all other agents have the same form. For each of the 3n agents above, their maximum expected return is 2.Rounding a policy. Observe that F i (3/2) = 7/8. For agent ag i,1 to α-approve a policy for α ∈ (7/8, 1], they require a utility (strictly) more than 3/2 = F -1 ag i,1 (7/8). The fact that v 1 , v 2 ∈ [0, 1] in addition to v 1 + v 2 > 3/2, implies that v 1 > 1/2 and v 2 > 1/2. Therefore, if a policy π is α-approved by agent ag i,1 , we have π(λ(c i,1 )) > 1/2 and π(λ(c i,2 )) > 1/2. Further, observe that for at most one of the three agents {ag i,k } k∈ [3] the condition of J ag i,k (π) > 3/2 may hold as every pair of the agent disagree on one literal.We round such a policy π to an assignment ϕ by letting ϕ(x j ) ← True if π(a True |s j ) > 1 2 and ϕ(x j ) ← False otherwise. If an agent in {ag i,k } k∈ [3] α-approves π, then C i is satisfied by the assignment ϕ. Therefore, we have that OPT α ⩽ OPT MAX-2SAT , where OPT α is the maximum feasible number of α-approvals among all policies and OPT MAX-2SAT is the maximum number of clauses that can be satisfied among all assignments.Next, we show OPT MAX-2SAT ⩽ OPT α by deriving a policy π that gets OPT MAX-2SAT αapprovals based on the optimal assignment for OPT MAX-2SAT by simply letting π(a True |s j ) = 1 iff ϕ(x i ) = True. If ϕ satisfies a clause C i , then for the agent ag ∈ {ag i,k } k∈ [3] that matches the literal assignments of ϕ for C i , we have J ag (π) = 2, which is an optimal, i.e., 1-approval, policy for ag. For the other two agents for C i , J ag (π) = 1 which gets a return less than their required utility of 3/2 for an α-approval. Therefore, we have that OPT MAX-2SAT = OPT α and the hardness of computation follows from our reduction.Proof of Theorem 6. The Borda count of a policy is defined asBorda(π) = i∈[n] F i (J i (π)). Since F i (v) ∈ [0, 1], max π i∈[n] F i (J i (π)) ⩽ n.From Definition 6, for a q-quantile fair policy π q , we have F i (J i (π q )) ⩾ q, for all i ∈ [n]. Therefore, i∈[n] F i (J i (π q )) ⩾ qn and we haveBorda(π q ) max π Borda(π) ⩾ qn n ⩾ q.C A Note on Algorithm 4Here, we expand on Algorithm 4. As mentioned before, in the pre-processing step, for every agent i, we measure F i (kϵ) for k ∈ [1/ϵ] via 1/ϵ calls to vol-comp. Let J(π) = (J 1 (π), . . . , J n (π)) be the (expected) return vector. Importantly, our MILP approximates the Borda score of a return vector r by its ϵ-rounded-down return vector r ϵ = (⌊r i /ϵ⌋• ϵ | i ∈ [n]). Therefore, the MILP finds a policy that has a Borda count which is at least as high as the maximum Borda count among ϵ-rounded return vectors. This is not necessarily equivalent to a (1 -ϵ) approximation of the max Borda count.Experiments are all done on an AMD EPYC 7502 32-Core Processor with 258GiB system memory. We use Gurobi [18] to solve LPs and MILPs.Running time. All our voting rules has a running time of less than ten minutes on the constructed MOMDPs of 3 5 = 243 states, 6 actions, and 10 agents. The most resource extensive task of our experiments was sampling 5 • 10 5 random policies and computing the expected return for every agent which is a standard task. For each instance, we did this in parallel with 20 processes in a total running time of less than 2 hours per instance.Implementation details of Borda count rule. After fitting a generalized logistic function for F i based on the expected return of sampled policies, we find the value mode(f i ), and check the existence of a policy by solving a linear program that achieves a expected return of mode(f i ) for all agents. Next, to utilize Gurobi LP solvers, we approximate the concave function F i by a set of linear constraints that form a piecewise linear concave approximation of F i . Therefore, our final program for an approximate Borda count rule is simply an LP.Justification: The full code with the running instructions are attached.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We do not use any types of data or models that have the potential for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the optimization tools utilized in our code.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.And we assume this is done accurately, in order to focus on the essence of the policy aggregation problem.It is important to check the non-emptyness of the search space Oi as an over-estimation of v * i could lead to an empty feasible region.This result is for arbitrary monotone valuations. For additive valuations, they show 0.14 e -quantile fair allocations always exist without any assumptions.We suspect the problem to be NP-hard since the objective resembles a summation of "sigmoidal" functions over a convex domain, which is known to be NP-hard[37].The code for the experiments is available at https://github.com/praal/policy-aggregation. 
 
 Appendix:  


Decision: Reviewers are all positive about the paper. They consider the paper to be well-motivated from aligning AI systems and they find the paper to be well-written. It will be good to address reviewers' comments on the empirical section, and it might be good to add the discussion about normalizing utilities in the rebuttal to the paper if there's space.

Reviews: review: paper_summary: paper_summary: This paper studies aggregating multiple policies–which can be seen as a formalization of the task of aligning an AI system to the values of multiple individuals. When the number of states is small (such as, when multiple individuals have to select one out of a few candidates), this problem has been widely studied in voting and social choice theory, and there are many efficient aggregation rules (such as the Borda count). This paper, however, considers the other extreme: where the state-action space is huge, and it is not obvious how to design efficient methods to aggregate policies. 

The main insight of this paper is that preferences over policies have a volumetric interpretation in the state-action policy space that, in some cases, leads to efficient aggregation algorithms. 

Concretely, the authors examine two types of aggregation rules: (1) two aggregation rules that are known to have desirable fairness properties (namely, proportional veto code and, the recently introduced, quantile fairness) and (2) voting or score-based rules such as Borda count and $\alpha$-approval voting rule. 

Building on their insight the authors prove several results, including 
1. an algorithm which finds the policy wrt an $\epsilon$-approximation of the proportional veto core using $O(log(1/\epsilon))$ queries to a volume computation oracle, 
2. the existence of $q$-quantile fair policies for all $q\geq 1/e$ (which is tight and stronger than the best possible bound in the discrete case),
3. NP-hardness and inapproximability results for $\alpha$-approval score.
strength_weakness: strength_weakness: I am not sure if the empirical results section is adding any value to this paper: it evaluates different aggregation rules, but I think this is not the focus of this work–I think the focus is to design efficient algorithms and/or prove existential results. If other reviews and the area chairs agree, my suggestion is to drop the empirical results section and use the additional space to add more exposition on the proofs. To be clear, this is no a significant concern for me.
questions: questions: I do not have any specific questions for the authors.
limitations: limitations: I do not see significant limitations.
paper_summary: paper_summary: The paper solves the problem which arises in preference aggregation of individual policies to a collective policy – (1) summation based aggregation are sensitive to affine transformations and (2) voting rule based aggregation faces problem of policies being exponential in S. Towards solving this, the paper proposes voting over continuous space of alternatives (which eliminates affine sensitivity) and a volumetric definition of preference ordering. The paper next proposes efficient algorithms to (1) find approximate volumetric veto core and (2) approximate q-Quantile Fairness. They also show complexity of existing voting rules, notably plurality voting and borda count. They show that problem is computationally hard for plurality and open for broad count. I am inclined towards accepting the paper.
strength_weakness: strength_weakness: In the experimental section, using a common metric to quantify the “level of fairness” guaranteed by different algorithms would be beneficial for a more learned comparison.  

In Def. 4 should the expression be vol(O’)/vol(O) >= 1 – veto(S) + epsilon instead of vol(O’) >= 1 – veto(S) + epsilon as currently stated?
questions: questions: see weakness.

Why can't yours be a special case of Noothigattu et al. [27]?
limitations: limitations: Authors adequately address the problem statement and discussed limitations/candidate improvements of their work which is left to future work.
paper_summary: paper_summary: This paper joins a long list of recent work that studies how to aggregate the preferences of several agents (e.g., humans) in a reinforcement learning framework inspired by social choice theory. The problem is modeled as a multi-objective MDP with $n$ different reward functions. The authors propose to use the state-action occupancy measure instead of each agent's most preferred policy or reward function directly. Popular concepts from social choice theory, such as the Borda count rule and approval voting, are then studied from this perspective.
strength_weakness: strength_weakness: - In my opinion, the contributions of this work are limited. E.g., only the full information case is studied
- The primary justification of this work (which is repeatedly mentioned in the paper) is that prior work on policy aggregation and fair RL is not invariant to affine transformations of the reward function. Essentially, agents can have differently scaled reward functions, which makes, e.g., maximizing for social welfare a bad objective. However, I don't understand why we cannot simply normalize the reward function of each agent, so that the reward functions are directly "comparable". I find the concern about affine transformations quite weak.
questions: questions: - I'm a bit surprised about the title of the paper since you're not aggregating policies, but reward functions. In fact, the policies play a minor role in the paper, since you look at the preference relation induced by the reward function (which you then express in term of occupancy measures). Could you explain why what you're doing is policy aggregation and not just preference aggregation? 

Typo: "Policy Aggergation" in title of Section 5
limitations: limitations: Limitations are adequately addressed in my opinion.

Comments: title: Response to authors
comment: Thank you for your response and sorry for my late reply. My concerns are addressed and I can align with the other reviewers on a score of 6. Regarding the title, in my opinion, something along the lines of “socially fair reward aggregation” would better capture the point of the paper. I think, it’s fine either way, and it seems that I was the only one who expected slightly different content given the title (I was expecting direct policy aggregation).
I thank the authors for their response. I will keep my previous score and think this paper should be accepted.