Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards
Authors: Silviu Pitis
Venue: NeurIPS 2023 poster (2023)
Keywords: Normative Agency Design
Reward Design
Sequential Decision Making
Reinforcement Learning
Intertemporal Fairness
Multi-Objective Decision Making
TL;DR: It is shown that Markovian aggregation of Markovian reward functions is not possible when the time preference for each objective may vary; a practical non-Markovian aggregation scheme is proposed.

Abstract: As the capabilities of artificial agents improve, they are being increasingly deployed to service multiple diverse objectives and stakeholders. However, the composition of these objectives is often performed ad hoc, with no clear justification. This paper takes a normative approach to multi-objective agency: from a set of intuitively appealing axioms, it is shown that Markovian aggregation of Markovian reward functions is not possible when the time preference (discount factor) for each objective may vary. It follows that optimal multi-objective agents must admit rewards that are non-Markovian with respect to the individual objectives. To this end, a practical non-Markovian aggregation scheme is proposed, which overcomes the impossibility with only one additional parameter for each objective. This work offers new insights into sequential, multi-objective agency and intertemporal choice, and has practical implications for the design of AI systems deployed to serve multiple generations of principals with varying time preference.

Consistent Aggregation of Objectives with Diverse
Time Preferences Requires Non-Markovian Rewards
Silviu Pitis
University of Toronto and Vector Institute spitisecs. toronto. edu
Abstract
As the capabilities of artificial agents improve, they are being increasingly deployed to service multiple diverse objectives and stakeholders. However, the composition of these objectives is often performed ad hoc, with no clear justification. This paper takes normative approach to multi-objective agency: from set of intuitively appealing axioms, it shown that Markovian aggregation of Markovian reward functions is not possible when the time preference (discount factor) for each objective may vary. It follows that optimal multi-objective agents must admit rewards that non-Markovian with respect the individual objectives. To this end, a practical non-Markovian aggregation scheme proposed, which overcomes the impossibility with only one additional parameter for each objective. This work offers new insights into sequential, multi-objective agency and intertemporal choice, and has practical implications for the design of AI systems deployed serve multiple generations of principals with varying time preference.
Introduction
The idea that we can associate human preferences with scalar utility values traces back hundreds of years and has found usage in numerous applications 71 perhaps most important, is the design of artificial agents. In the field of reinforcement learning 10, which lets us define objectives in terms of a discounted sum of Markovian rewards. While foundational results from decision theory
28 49 One of the most recent, and
(RL), this idea shows up as the reward hypothesis 774
67
81 62 and inverse RL 52 54 justify the reward hypothesis when single objective or principal is considered, complexities arise in multi-objective scenarios [7]. The literature on social choice largely defined by impossibilities (5), and multi nulti-objective composition in the RL and machine learning literature is typically restrictive 688 51, applied without clear justification [29], or based on subjective evaluations of empirical efficacy 211. Addressing these limitations is crucial for the development of artificial agents capable of effectively serving the needs of diverse stakeholders.
This paper extends previous normative work in RL by adopting an axiomatic approach to the aggregation of objectives. The approach is based on set of intuitively appealing axioms: the von Neumann- MorggensteMeenn (VNM) axioms, which provide foundation for rational choice under uncertainty; Pareto indifference, which efficiently incorporates individual preferences; and dynamic consistency, which ensures cimecconsistent decision-makeen From these axioms, an impossibility is derived, leading to the conclusion that optimal multi-objectie agents with diverse time preferences must have rewards that are non-Markovian with respect to the individual objectives. To address this challenge, a practical state space expansion is proposed, which allows for the Markovian aggregation of objectives requiring only one parameter per objective. The results prompt an interesting discussion on dynamic preferences and intertemporal choice, leading to novel "historical discounting" strategy that trades off dynamic consistency for intergenerational fairness. Finally, is shown how both our results can be extended (albeit non-normatively) to stochastic policies.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

The remainder of this paper is organized as follows: Section motivates the problem by modeling human procrastination behavior as an aggregation of two objectives, work and play, and showing how a plan that appears optimal today may lead to the worst possible future outcome. Section 3 presents the axiomatic background and the key impossibility result. Section 4 presents the corresponding possibility result and a practical state expansion to implement it. Section S relates the results to intertemporal choice, proposes N --step commitment and historical discounting strategies for managing intergenerational tradeoffs, extends the results to stochastic policies, and discusses related topics in RL. Section concludes with some final thoughts and potential future research directions.
Motivation: The Procrastinator's Peril
We begin with numerical example of how the naive aggregation of otherwise rational preferences lead to undesirable behavior. The example, which will be referred to throughout as the "Procastinator's Peril", involves repeated procrastination, phenomenon to which the reader might relate. An agent aggregates two competing objectives: work and play. At each time step the agent can choose to either work play. The pleasure of play is mostly from today, and the agent doesn't value future play nearly as much as present play. On the other hand, the consequences of work are delayed, so that work tomorrow is valued approximately as much work today.
Let us model the agent's preferences for work and play as two separate Markov Decision Processes (MDP), each with state space S 0 and action space A = {w. p}. In the play MDP, we have rewards R(p) = 0.5, R(x) = 0 and a discount factor of "play = 0.5. In the work MDP, we have rewards R(p) = O, R(w) 0.3 and discount factor of Toork 0.9. One way to combine the preferences for work and play is to value each trajectory under both MDPs and then add up the values. Not only does this method of aggregation seem reasonable, but it is actually implied by some mild and appealing assumptions about preferences (Axioms and in the sequel). Using this approach, the agent assigns values trajectories follows:
T
P. P. P...
V(n) (..))) 0.5
-1.00
T2
N, W. W...
((12) (0.)) 0.3
3.00
T3
P.W, W. W...
((Ta) 0.9 (T2)
3.20
T4
P.P,W, W.W. W...
(Ta) =0.75 0.92 (122)
3.18
We see that the agent most prefers 73: one period (and one period only!) of procrastination is optimal. Thus, the agent procrastinates and chooses to play today, planning work from tomorrow onward. Come tomorrow, however, the agent is faced with the same choice, and once again puts off work in favor of play. The process repeats and the agent ends up with the least preferred alternative TΙ.
This plainly irrational behavior illustrates the impossibility theorem. Observe that the optimal policy 73 is non-Markovian--it must remember that the agent has previously chosen play in order work forever. But any MDP has a stationary optimal policy (55), so it follows that we need rewards that are mon-Markovian with respect to the original state-action space. Alternatively, will see in Subsection 4.2 that we can expand the state space to make the optimal policy Markovian.
3 Impossibility of Dynamically Consistent, Pareto Indifferent Aggregation
Notation We assume familiarity with Markov Decision Processes (MDPs) (55) and reinforcement learning (RL. [74). We denote an MDP by M = A, 7) where ? S x A + R+ is state-action dependent discount function. This generalizes the usual "fixed" ? E and covers both the episodic and continuing settings [85]. We use lowercase letters for generic instances, e.g. S, and denote distributions using tilde, e.g. In contrast to standard notation we write both state- and state-action value functions using a unified notation that emphasizes the dependence of each on the future policy: we write V(s, T) and V(s, ar) instead of V*(s) (s) and Q" a). We extend V to operate on probability distributions of states, V3 II) = E II), and we allow for non-stationary, history dependent policies (denoted by uppercase IT, Ω). With this notation, we can understand V as an expected utility function defined over prospects of the form (5, IT). We use the letter h to denote histories (trajectories of states and actions) these may terminate on either a state or action, may be inferred from the context. For convenience, sometimes directly concatenate

---

histories, states, actions and/or policies (e.g., hs, so, sIl, alI) to represent trajectory segments and/or the associated stochastic processes. For simplicity, we assume finite |S], |A].
3.1 Representing rational preferences
This paper is concerned with the representation of aggregated preferences, where both the aggregation and its individual components satisfy certain axioms of rationality. We define the objects of preference to be the stochastic processes ("prospects") generated by following (potentially non-stationary and stochastic) policy II from state Distributions or "lotteries" over these prospects may be represented by (not necessarily unique) tuples of state lottery and policy (3, II) E ((S) x II =: C(P). We write ($1, MT (52, 22) if (51, IT) is strictly preferred to ($2, S2) under preference relation >-
To be "rational", we require > to satisfy the "VNM axioms"
which capture in Axiom
Axiom 1 (VNM). For all p. i. F E ((P) we have:
Asymmetry: p > q. then not g > p;
Negative Transitivity: If not p> q and not ğ. not p>>;;
Independence: If p 33-0) ÿ. then ap + (1 a) > aq + (1 a) Va E (0;
Continuity: If p q > i, then 3 a, (0, 1) such that op + a)f > 3p + (1- B);;
where ap + (1 q denotes the mixture lottery with a% chance of p and (1 a 2% chance of q.
Asymmetry and negative transitivity together form the basic requirements of a strict preference relation-eque equivalent to completeness and transitivity of the corresponding weak preference relation, t (defined as p ž q + q o p). Independence can be understood as an irrelevance of unrealized alter- natives, consequentialist, axiom: given that the a% branch of the mixture is realized, preference between 5 and ç is independent of the rest of the mixture (i.e., what could have happened on the (1) branch). Finally, continuity is a natural assumption given that probabilities continuous.
We further require > to be dynamically consistent 70
44
Axiom 2 (Dynamic consistency). (s, all) > (s, asl) if and only (T a), II) > (T) where T(s, a) is the distribution over next states after taking action a in state s.
This axiom rules out the irrational behavior in the Procrastinator's Peril, by requiring today's prefer- ences for tomorrow's actions to be the same tomorrow's 's preferences. While some of these axioms (particularly independence and dynamic consistency) have been the subject of debate (see, e.g., note that the standard RL model is more restrictive than they require [54)
The axioms produce two key results that we rely on (see Kreps (39 and Pitis (54] for proofs):
Theorem 1 (Expected utility representation). The relation > defined on the set L(P) satisfies Axiom I if and only if there exists a function V
such that, V P. q E L(P):
Σ Excompi (=))((=) ExEmptor ((z)V(z).
b
Another function V gives this representation iff V is positive affine transformation of V.
Using Theorem we extend the domain of value function V to L(P) V (())V():..
Theorem (Generalized Bellman representation). If> satisfies Axioms 1-2 and V is an expected utility representation of >, there exist R : 5 x A R. 7 S S A + R+ such that V o, II, V(s, all) = R(s, a) + (s, (T(,
Remark 3.1.1 Instead of preferences stochastic processes of potential futures, could begin with preferences over trajectories [86] 64 10]. The author takes issue with this approach, however, as it's unclear that such preferences should satisfy Asymmetry Independence without additional assumptions (humans often consider counterfactual outcomes when evaluating the desirability of a trajectory) (54). By using Theorem 2 to unroll prospects, can extend preferences over prospects to define preferences over trajectories according to their discounted reward.
Remark 3.1.2 Theorem 2, as it appeared in Pitis 54 required an additional, explicit "Irrelevance of unrealizable actions" axiom, since prospects were defined as tuples (6, IT). This property implicit in our redefinition of prospects stochastic processes.
3

---

Remark 3.1.3 In this line of reasoning only the preference relation > is primitive; V and its Bellman form (R) are simply representations of whose existence is guaranteed by the axioms. Not all numerical representations of > have these forms 84 In particular, (strictly) monotonically increasing transforms preserve ordering, so that any increasing transform Vt of a Theorem I representation V is itself a valid numerical representation of > (although lotteries will no longer be valued by the expectation over their atoms unless the transform is affine, per Theorem
3.2 Representing rational aggregation
Let us now consider the aggregation of several preferences. These may be the preferences of an agent's several principals or preferences representing a single individual's competing interests. Note at the outset that it is quite natural for different objectives or principals to have differing time preference. We saw one example in the Procrastinator's Peril, but we can also consider a household robot that seeks to aggregate the preferences of Alice and her husband Bob, for whom there is no reason to assume equal time preference [26].
An intuitively appealing axiom for aggregation is Pareto indifference, which says that if each individual preference indifferent between two alternatives, then too is the aggregate preference.
Axiom 3 (Pareto indifference). i mi 7 (Vi E 2), "x q.
Here, 5 = q means indifference (not relation, I is a finite index set over the individuals, and mg indicates the aggregate relation. There exist stronger variants of Pareto property that require monotonic aggregation (e.g., if all individuals prefer ÿ. so too does the aggregate; see Axiom 3' in Subsection 5.1). We opt for Pareto indifference to accommodate potentially deviant individual preferences (e.g., g., if all individuals are indifferent but for a sociopath, the aggregate preference may be opposite of the sociopath's)
and not p), so that R3; is the ith individual indifference
If require individual and aggregate preferences to satisfy Axiom and, jointly, Axiom 3, we obtain a third key result due to Harsanyi 332. (See Hammond 31 for proof).
Theorem 3 (Harsanyi's representation). Consider individual preference relations {= E I and aggregated preference relation > y, each defined on the set L(P), that individually satisfy Axiom and jointly satisfy Axiom 3. If {V;i E I} and Vy expected utility representations of of {> E I} and > respectively, then there exist real-valued constant c. and weights {w,;i E EI} such that:
Vz(p) = Eier w.VV(().
That is, the aggregate value can be expressed as a weighted sum of individual values (plus a constant).
According to Harsanyi's representation theorem, the aggregated value function is function of the individual value functions, and nothing else. other words, Pareto indifferent aggregation of VNM preferences that results in VNM preference is necessarily context-free- the same weights {w;} apply regardless of state and policy.
We will also make use of two technical conditions to eliminate certain edge cases. Though sometimes left implicit, these are both common requirements for aggregation functions (5]
Axiom 4 (Technical conditions on >>)).
Unrestricted Domain: > is defined for all valid individual preference sets {>ıi E Z}. Sensitivity: Vi E Z, holding j. f i constant, there exist > > resulting in different >z.
The first condition allows us consider conflicting objectives with different time preference. The second condition implies that the weights w, in Theorem 3 are nonzero.
Remark 3.2.1 It is worth clarifying here the relation between Harsanyi's theorem and related class of aggregation theorems, occasionally cited within the machine learning literature (c.g., (6), based on axioms originating with Debreu [1). One instance of this class states that any aggregate preference satisfying six reasonable axioms can be represented the form: V=( = Eier ((V( where m a strictly increasing monotonic function from the family {x* 0 < S 1) U (log(z) Ip = (-2 < 0} 47 If we (1) drop the symmetry axiom from this theorem to obtain variable weights wi, and (2) add monotonicity axiom to Harsanyi's theorem to ensure positive weights w; 32]. then the only difference between the theorems is the presence of monotonic function But

---

note that applying to each Vú or to Vi individually does not change the preference ranking they represent; it does, however, decide whether V, and V/ are VNM representations of >. Thus, we can understand Harsanyi's i theorem as saying: if V,, VE are VNM representations of > then m must be linear. Or conversely, if m is non-linear (p # 1), V; and Vy are not VNM representations.
Remark 3.2.2 A caveat of Harsanyi's theorem is that it implicitly assumes that all individual preferences, and the aggregate preference, use the same set of agreed upon, "objective" probabilities. This is normatively justifiable if we use the same probability distribution (e.g.. that of the aggregating agent) to impose "ideal" preferences > on each individual, which may differ from their implicit subjective or revealed preferences [62]. As noted by Desai et al. [20], Harsanyi's theorem fails if the preferences being aggregated use subjective probabilities. Note, however, that the outcome of the "bargaining" construction in Desai et al. (20) is socially suboptimal when the aggregrating agent has better information than the principals, which suggests that effort should be made to unify subjective probabilities. We leave exploration of this to future work.
3.3 Impossibility result
None of Theorems assume all Axioms Doing leads to key result, follows.
Theorem 4 (Impossibillity). Assume there exist distinct policies, II, A, none of which a mixture (i.e., convex combination) of the other two, and consider the aggregation of arbitrary individual preference relations {>; E I defined on L(P) that individually satisfy Axioms 1-2. There does not exist aggregaled preference relation >x satisfying Axioms 1-4.
Sketch of Proof. The full proof is Appendix B Briefly, we consider [ 2 and use Axiom (Unrestricted Domain) to construct mixtures of II and & so that each mixture is considered indifferent A by one of the individual preference relations. Then, by applying Theorem 2 and Theorem 3 in alternating orders to the difference between the value of the mixture policy and the value of A, and doing some algebra, we arrive at the equations
wi a) = wi (x, and W272 a) = ( a),
from which conclude that (y a) Y1 (s,ss) a) (2 a). But this contradicts assumption that individual preferences {> E I) may be chosen arbitrarily, completing the proof.
find two policies (a II/S2 mixture, and A) between which individual preference > indifferent, but individual preference > is not. Then for mixtures of this II/ mixture and A, >- remains indifferent, but the strength of >2 changes, that > must have the same time preference as >2- By an analogous argument, > must have the same time preference as > leading
Intuition of Proof. Under mild conditions,
violation of Unrestricted Domain.
The closest results from the economics literature consider consumption streams (S IR) 88 L5
Within reinforcement learning, equal time preference has been assumed, without justification, when merging MDPs 688 and value functions for different tasks (2)].
Remark 3.3.1 A consequence of Unrestricted Domain, critical to the impossibility result, is that individual preferences may have diverse time preferences (i.c., different discount functions). If discounts are equal, Markovian aggregation is possible.
Remark 3.3.2 Per Remark 3.2.2, by applying Harsanyi's 's theorem, we implicitly assuming that all preferences are formed using the same "objective" probabilities over prospects; is there a notion of "oojjective" time preference that should be used? If so, this would resolve the impossibility (once again, requiring that we impose a notion of ideal preference on individuals that differs from their expressed preference). We leave this consideration for future work.
Remark 3.3.3 Theorem appliess to the standard RI. setup, where 71, 72, TE are constants.
4 Escaping Impossibility with Non-Markovian Aggregation
An immediate consequence of Theorem 4 is that any scalarized approach to multi-objective RL 779 generally insufficient represent composed preferences. But the implications run deeper: insofar as general tasks consist of several objectives, Theorem 4 pushes Sutton's 's reward hypothesis to its
5

---

limits. To escape impossibility, the Procrastinator's Peril is suggestive: to be productive, repeat play should not be rewarded. And for this to happen, we must keep track of past play, which suggests that reward must be non-Markovian even when all relevant objectíves are Markovian. That is, even if we have settled on some non-exhaustive state representation that is "sufficiently" Markovian, an extra aggregation step could render it no longer sufficient.
Relaxing Markov Preference The way in which non-Markovian rewards (or equivalently, non- Markovian utilities) can be used to escape Theorem 4 is quite subtle. Nowhere in the proofs of Theorems 4 is the Markov assumption explicitly used. Nor does it obviously appear in any of the Axioms. The Markov assumption is, however, invoked in two places. First, to establish history- prosspcctsss (s, ll)-and second,
independent comparability between the basic objects of to extend that comparison set to include "prospects" of the form (T a), IT). To achieve initial comparability, Pitis [54] applied "Markov preference" assumption (preferences over prospects are independent of history) together with an "original position" construction that is worth repeating here:
[]]t admittedly difficult to express empirical preference over prospects ... an agent only ever chooses between prospects originating in the same state ... [Neverthclcss. we imagine a hypothetical state from which an agent chooses between [lotteries] of prospects, denoted by LPP). We might think of this choice being made from behind a "veil of ignorance" [8
(2)
In other words, to allow for comparisons between prospect ($1, I1) and (s2, II), we prepend some pseudo-state, 50, and compare prospects (5081, I1) and (5052, II). Markov preference then lets us cut off the history, so that our preferences between (81, II) and (#2, II) are cardinal.
The impossibility result suggests, however, that aggregate preference is not independent of history, so that construction [ cannot be applied. Without this construction, there is no reason to require relative differences between ((s1; +) and ((s2, *) be meaningful,
even think about lotteries/mixtures of the two prospects (as done in Axiom T. Letting go of this ability to compare prospects starting in different states means that Theorem I is applicable only sets of prospects with matching initial states, unless we shift our definition of "prospect" to include the history; i.e., letting h represent the history, we now compare "historical prospects" with form (h, I1).
Though this does not directly change the conclusion of Theorem 2 T(s, a) in (TT(s, a), II) includes piece of history, (s, a). and can no longer be computed as E. (Ts,a)) V(s', II.. Instead, since the agent is not choosing between prospects of form (s', IT) but rather (abusing notation) prospects of form (sas', IT), the expectation should be computed as E.g/T((s V(sas', IT).
The inability to compare prospects starting in different states also changes the conclusion of Theorem 3 which implicitly uses such comparisons to find constant coefficients w; that apply everywhere in the original L(P). Relaxing the application of Harsanyi's 's theorem to not make inter-state comparisons results in weights w;(h) that are history dependent when aggregating the historical prospects.
4.1 Possibility Result
Allowing the use of history dependent coefficients in the aggregation of Vi(T a). II) resolves the impossibility. The following result shows that given some initial state dependent coefficients (x), we can always construct history dependent coefficients (h(h) that allow for dynamically consistent aggregation satisfying all axioms. In the statement of the theorem, [(Px) is used to denote the set of lotteries over prospects starting with history h of arbitrary but finite length (not to be confused with the set of lotteries over all historical prospects, of which there is only one). Note that a preference relation satisfies Axioms 1-2 with respect to L(P), the natural extension to ((Phe), (hs, II,) (hs, I12) = (s, [I, (s. II2), satisfies Axioms 1-2 with respect to C(Phs). Here, we are using hs to denote a history terminating in state s.
Theorem 5 (Possibility). Consider the aggregation of arbitrary individual preference relations {ret E I} defined on L(P), and consequently ((Pr), wh, that individually satisfy Axioms 1-2. There exists aggregated preference relations (>b),, defined on L(Pr), Wh, that satisfy Axioms 1-4.
In particular, given 8, Vg. [V;). {w:(s)}, where (A) each V: satisfies Axioms 1-2 on L(P), and Ve satisfies Axioms 1-2 on L(Ph), ), wh, and (B) Vg(s, all E, (x((s) all)), then, choosing
wi wi(ss'') == wi (sa) (s) for all i,8'
(3)

---

implies that Vs(sas',11) x E, w,(sa)Va(s(, (iss, II) so that the aggregated preferences {>gnd) satisfy
Axiom 3 on s(Psas). Unrolling this woilts- == w (,(hs)) produuces a set of constructive, history dependent weights [w;(h)) such that Axiom 3 is satisfied for all histories {h.)
Sketch of Proof. The full proof is in Appendix B Following the proof of Theorem we arrive at and w2(s))a(ss a) w> w=(ss))x(s.u).
wi (s) (s, a) wi (sa)nz(s,
(4)
from which we conclude that: wa (20)
w2(s))a(s,
(5)
(sa) wi (1)))ma
This shows the existence of weights wy (y(ss), unique up to a constant scaling factor, for which Vs(T(ss, a), II) x Ej (; a), II), that apply regardless of how individual preferences are chosen or aggregated at s. Unrolling the result completes the proof.
From Theorem we obtain a rather elegant result: rational aggregation over time discounts the aggregation weights assigned to each individual value function proportionally to its respective discount factor. In the Procrastinator's 's Peril, for instance, where we started with wy wy(s) w,(ss 1, at the initial (and only) state s, we might define wp (sps) = 0.5 and (2 = 0.9. With these non- Markovian aggregation weights and (x(sp) = you can verify that (1) the irrational procrastination behavior is solved, and (2) the aggregated rewards for work and play are now non-Markovian.
Remark 4.1 (Important!) The discount "I is left undetermined by Theorem One might determine several ways: by appealing to construction 2 with respect to historical prospects in order to establish inter-state comparability, by setting it to be the highest individual discount (0.9 in the Procrastinator's Peril), by normalizing the aggregation to weights to sum to at each step, or perhaps by another method. In any case, determining z would also determine the aggregation weights, per equation 4 (and vice versa). We leave the consideration of different methods for setting " and establishing inter-state comparability of Vs future work. (NB: This is normative question, which we leave unanswered. While one can make assumptions, as we will for our numerical example in Subsection future research should be wary of accepting solution just because it seems work.)
5.2
4.2
A Practical State Space Expansion
The basic approach to dealing with non- Mank-Mvann rewards is to expand the state space in such a way that rewards becomes Markovian 27 14 2] However, naively expanding S to history of length H could have O((SS + [AA)) complexity. Fortunately, the weight update in equation 4 allows us to expand the state using single parameter per objective. In particular, for history hsa and objective i, we append to the state the factors y (i(sa)) := yı (h)ssss ()/1x(x, a), which are defined for every history, and can be accumulated online while executing trajectory. Then, given composition with any set of initial weights {w:) J. we can compute the weights of augmented state Sang = (8, yi (hs)) as (sg(sm)) ys (hs) we Letting L(P(v)) be the set of prospects on the augmented state set, we get the following corollary to Theorem
Corollary 1. Consider the aggregation of arbitrary individual preference relations {=i I}
defined on ((P), and consequently ((P(v)), that individually satisfy Axioms 1-2. There exists aggregated preference relation {> defined on L(P(s)), that satisfies Axioms 1-4.
5
Discussion and Related Work
5.1 A Fundamental Tension in Intertemporal Choice
The state space expansion of Subsection 4.2 allows to represent the (now Markovian) values of originally Mar-Mvrko policies in a dynamically consistent way. While this allows us to design agents that implement these policies, it doesn't 't quite solve the intertemporal choice problem.
In particular, is known that dynamic consistency (in the form of Koopmans' Stationarity 38 together with certain mild axioms, implies a first period dictatorship: the preferences at time ..- = 1 are
decisive for all time 255 (in a sense, this is the very definition of dynamic consistency!). Generally speaking, however, preferences time t a are not the same preferences at time f = 1 (this is
7

---

what got us into our Procrastinator's ' Peril to begin with!) and we would like to care about the value all time steps, not just the first.
A typical approach is to treat each time step as a different generation (decision maker), and then consider different methods of aggregating preferences between the generations [3) Note that (1) this aggregation assumes intergencrational comparability of utilities (see Remark 4.1), and (2) each generation is expressing their personal preferences about what happens in all generations. not just their own, Since this is a single aggregation, it will be dynamically consistent (we can consider the first period dictator as a benevolent third party who represents the aggregate preference instead of their own). A sensible approach might be to assert a stronger version of Axiom that uses preference:
Axiom 3' (Strong Pareto Preference) If p E (Vi I), then m q; and if, furthermore. 3j I such that p } (. then P > Q-
Using Axiom 3' in place of Axiom for purposes of Theorem gives representation that assigns strictly positive weights to each generation's 's utility. Given infinite periods, follows (e.g., by the Borel-Cantelli Lemma for general measure spaces) that if utilities are bounded, some finite prefix of the infinite stream decides the future and we have a "finite prefix dictatorship". which is not much better than first period one.
The above discussion presents a strong case against using dynamic consistency to determine a long horizon policy. Intuitively, this makes sense: preferences change over time, and our current generation should not be stuck implementing the preferences of our ancestors. One way to do this by taking time out of the equation, and optimizing the expected individual utility of the stationary state-action distribution, dx (cf. Sutton and Barto (74 (Section 10.4) and Naik et al. (50):
( = - (s)
(6)
Unfortunately, as should be clear from the use of a stationary policy n, this neutrllural approach falls short for our purposes, which suggests the use of non-Markovian policy. While optimizing equation O would find the optimal stationary policy in the Procrastinator's Peril (work forever with ((22) 3.0), it seems clear that we should play at least once (V(T3) = 3.2) as this hurts no one and makes the current decision maker better iff simply optimizing G violates the Pareto principle.
This discussion exemplifies a known tension in intertemporal choice between Pareto optimality and the requirement treat every generation equally-it is impossible to have both [6 42). In a similar spirit to Chichilnisky [16]. who has proposed an axiomatic approach requiring both finite prefixes of generations and infinite averages of generations to have a say in the social preference, we will examine compromise (to the author's ' knowledge novel) between present and future decision makers in next Subsection.
5.2 N-Step Commitment and Historical Discounting
We now consider two solutions to the intertemporal choice problem that deviate just enough from dynamic consistency to overcome finite period dictatorships, while capturing almost all value for each decision maker. In other words, they are "almost" dynamically consistent. We leverage the following observation: due to discounting, the first step decision maker cares very little about the far off future. If we play once, then work for 30 steps in the Procrastinator's Peril, we are already better off than the best stationary policy, regardless of what happens afterward.
This suggests that, rather than following the current decision maker forever, we allow them commit to a non-Markovian policy for some N < co steps that brings them within some c of their optimal policy, without letting them exert complete control over the far future. implement this, we could reset the accumulated factors y to I every N steps. This approach is the same as grouping every consecutive N step window into a single, dynamically consistent generation with a first period dictator.
A more fluid and arguably better approach is to discount the past when making current decisions ("historical discounting").. We can implement historical discounting with factor 77 E [0. 1] by changing the update rule for factors y: to
7: (sss a) y (hsa) = (1)
+ + (s),
where w/' denotes the initial weight for objective at the with generation (if preferences do not change over time, w/' =w:). This performs exponential moving average of preferences over time-
8

---

high 7) < initially gives the first planner full control, but eventually discounts their preferences until they are negligible. This obtains a qualitatively different effect from N-step commitment, since the preferences of all past time steps have positive weight (by contrast, on the Nth step, N-step commitment gives no weight to the N past preferences). As result, in the Procrastinator's Peril, any sufficiently high returns policy T3. whereas N- N-step commitment would play every N steps.
Historical discounting is an attractive compromise because it is both Pareto efficient, and also anonymous with respect to tail preferences-il both optimizes equation O in the limit, and plays on the first step. Another attractive quality is that historical discounting allows changes in preference to slowly deviate from dynamic consistency over timethe first period is not a dictator.
As numerical example, we consider what happens in the Procrastinator's Peril if start to realize that we value occasional play, and preferences shift away from the play MDP to playn MDP. The playn MDP has fixed Yplayn 0.9 and non-Markovian reward R(p no play in last N-1 steps) = 0.5, R(anything else) = 0. We assume preferences shift lin- early over the first 10 timesteps, from Uplay 1, Uplaya at t 0 Wplay 0, Wplaya at t= 10 (and Woork = I throughout). Then, the optimal trajectories T" for different 7), and resulting discounted reward for the first time step, (7*),, are follows:
V'(+*)
7
0.00
play for steps, then play every steps
2.635
0.30
play for 5 steps, then play every 10 steps
2.635
0.50
play for 3 steps, then play every 10 steps
2.932
0.90
play, then work for 14 steps, then play every 10 steps
3.105
0.95
play, then work for 23 steps, then play every 10 steps
3.163
0.98
play, then work for 50 steps, then play every 10 steps
3.198
1.00
play, then work forever (same as T3)
3.200
When y = 0, each time step acts independently, and since the play MDP has high weight at the start, we experience a brief period of consistent play in line with the original Procrastinator's Peril, before preferences fully shift to occasional play. With high < 1, we capture almost all value for the first time step, while also eventually transitioning to the equilibrium "play every 10 steps" policy.
5.3 Extension to Boltzmann Policies
The preference relation > is deterministic, but the associated policy does not have be. In many cases dases-partialll-oon partially multi-agent, or even fully-observed settings [74] 330 stochastic policies outperform deterministic ones. And for generative sequence models such LLMs [33, stochastic policies are inherent. We can extend our analysis impoossiiiiissiiiiiii...... possibility, state space expansion, and intertemporal choice rules-to such cases by adopting stochastic choice rule.
To formalize this, we will take the stochastic choice rule as primitive, and use it to define a (deter- ministic) relation > that, for practical purposes, satisfies Axioms 1-4 [8]. We assume our choice rule satisfies Luce's Choice Axiom [33, which says that the relative rate of choosing between two alternatives in choice set of alternatives is constant regardless of the choice set. This can be implemented numerically with a Bradley-Terry choice model [1 by associating each alternative a, with a scalar P(a;), that given choice set fa,, a, p(a;) ((a:)/(Q(a)) + M(a,)). (An interesting interpretation for 2(as) that connects to both probability matching [82 65 and statistical mechanics 45 is the number of "outcomes" (microstates) for which o; is the best choice.)
We then simply "define" preference as a, > aj + M(a.) > M(az). and utility as V(a,) := A log 2 (a), so that the policy is softmax of the utilitics. The way these utilities are used in practice (e.g., [30 (7]) respects Axioms 2 And summing utilities is a common approach to composition 21 (2]. which is consistent with Harsanyi's representation (Theorem 3). For practical purposes then, the impossibility result applies whenever composed objectives may have different time preference.
Remark 5.3 Unlike our main results, this extension to Boltzmann policies is motivated by practical, rather than normative, considerations. Simple counterexamples to Luce's Choice Axiom exist [87 and probability matching behavior is evidently irrational in certain circumstances (6) We note, however, that certain theoretical works tease the existence a normative justification for Boltzmann policies [18 8 73 23); given the practice, clear justification would be of great value.

---

5.4 Related Work in RL
Task Definition Tasks are usually defined as the maximization of expected cumulative reward in an MDP [74] 66]] Preference-basee RL [86] avoids rewards, operating directly with preferences (but note that preference aggregation invokes Arrow's impossibility theorem [5] 48). while other works translate preferences into rewards 17
72
12 35 This paper joins growing list of work [54 [ 76
77 that challenges the implicit assumption that "MDPs are enough" in many reward learning papers,
particularly those that disaggregate trajectory returns into stepwise rewards [22 56 [5]].
Discounting Time preference or discounting can be understood part of the RL task definition. Traditionally, a constant 7 has been used, although several works have considered other approaches,
such as state-action dependent discounting [85 66 and non-Markovian discounting 24 63 Several
works have considered discounting as tool for optimization [80] or regularization
13 57
Task Compositionality Multi-objective RL 61 represents or optimizes over multiple objectives general value functions (75), which are often aggregated with a linear scalarization function 37.
Rather than scalarizing obtain single solution to a multi-objective problem, one can also seek sets of solutions, such as the set of Pareto optimal policies 788 or the set of acceptable policies [46].
Several works have also considered formal task decompositions [1 [5]] where simple addition of
MDPs is insufficient [68] More broadly, in machine learning, composition can be done via mixtures
of experts and/or energy-based modeling [34 211, which have also been applied RL (29 41]. Our
results provide normative justification for linear scalarization when time preference is the same for all objectives, but call for non-Markovian adjustments when time preferences differ.
Non-Markovian Rewards The necessity of non- Mar-Maaoov rewards was demonstrated in other settings by Abel et al. [ and more recently, in concurrent work by Skalse and Abate [69] Though several papers explicitly consider RI. with non-Markovian rewards (27 60), this is usually motivated by task compression rather than necessity, and the majority of the RL literature restricts itself to Markovian models. Many popular exploration strategies implicitly use non-Markovian rewards Our work is unique in that non-Markovian rewards arise from aggregating strictly Markovian quantities, rather non-Markovian quantities present in the task definition or algorithm.
37 53
6 Conclusion and Future Work
The main contribution of this work is an impossibility result from which one concludes that non- Markovian rewards (or an equivalent state expansion) are likely necessary for agents that pursue multiple objectives or serve multiple principals. It's possible that this will be the case for any advanced agent whose actions impact multiple human stakeholders. To accurately align such agents with diverse human preferences we need to endow them with the capacity to solve problems requiring non-Markovian reward, for which this paper has proposed an efficient state space expansion that uses one new parameter per aggregated objective. While the proposed state space expansion allows multi-objective agents to have dynamically consistent preferences for future prospects, it does not, in itself, solve the intertemporal choice problem. To that end, we have proposed "historical discounting", novel compromise between dynamic consistency and fair consideration of future generations.
Interesting avenues for future work include quantifying the inefficiency of Markovian representations, investigating normative approaches to aggregating preferences based on subjective world models (Remark 3.2.2), considering the existence of an "objective" time preference (Remark 3.3.2), improv- ing methods for determining subjective time preference (e.g.. (63 ), implementing and comparing approaches to determining "2 (Remark 1.1,, investigating historical discounting more detail (Sub- section 5.2). and considering the existence of a normative justification for Boltzmann policies and their composition (Subsection 5.3)
Acknowledgments and Disclosure of Funding
I thank Elliot Creager, for fruitful discussion that prompted Subsection 5.1 and motivated turn this into a full paper; Duncan Bailey, who assisted with an earlier workshop version; the anonymous reviewers, who provided detailed reviews and suggestions that helped improve the final manuscript: and Jimmy Ba and the Ba group for early discussions. This work was supported by an NSERC CGS D Award and a Vector Research Grant.
10

---

References
[1] David Abel, Will Dabney, Anma Harutyunyan. Mark K Ho, Michael Littman, Doina Precup. and Satinder Singh. the expressivity markov reward. Advances in Neural Information Prucessing Systems. 34: 7799-7812, 2021.
[2] David Abel, André Barreto, Michael Bowling, Will Dabney, Steven Hansen, Anna Harutyunyan, Mark Ho, Ramana Kumar, Michael Littman, Doina Precup, and Satinder Singh. Expressing non-man non-markov reward to markov agent. In RLDM, pages 1-5, 2022.
[3] Axel Abels, Diederik M Roijers, Tom Lenaerts, Ann Nowé, and Denis Stookelmacher. Dynamic weights in multi-objective deep reinforcement learning. arXiv preprint
2018.
Ron Amit, Ron Meir, and Kamil Ciosek. Discount factor regularizer in reinforcement learning. In International conference on machine learning, pages 369-278. PMLR, 2020.
[5] Kenneth Arrow, Amartya Sen, and Kotaro Suzumura. Handbook of social choice and welfare, volume Elsevier, 2010.
[6] Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell- Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 15538876-38189, 2022.
[7] André Barreto, Will Dabney, Rémi Munos, Jonathan Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances neural information processing systems, 30, 2017.
[8] Sergio methodologies. group value function problem, and applications of a measure of information to class of multiattribute problems. Rice University, 1979.
On decisions with multiple objectives: Review and classification of preseriptive
[9] D. Berouuli. Specimen thcoriac novac de mensura sortis, Commentari Academiac Scientiarum Imperialis Petropolitanae (5, 175-192, 1738). Econometrica, 22:23-36, 1954. Translated by L. Sommer.
[10] Michael Bowling, John D Martin, David Abel, and Will Dabney. Settling the reward hypothesis. arXiv preprint arXiv:2212.10420, 2022.
[11] Ralph Allan Bradley and Milton E Terry. Rank analysis incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324333 345, 1952.
[12] Daniel S Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. arXiv preprint arXiv: 000..990440000 06387, 2019.
[13] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
[14] Alberto Camacho, Rodrigo Toro Iearte, Toryn Q Klassen, Richard Anthony Valenzano, and Sheila Mcllraith. Ltl and beyond: Formal languages for reward function specification in reinforcement learning. In IJCAI, volume 19, pages 6065-6073, 2019.
[15] Christopher P Chambers and Federico Echenique. On multiple discount rates. Econometrica, 86(4): 2018.
[16] Graciela Chichilnisky, An axiomatic approach to sustainable development. Social choice and welfare, 13: 231-257, 1996.
[17] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. In Advances Neural Information Processing Systems, 2017.
[18] Gerard Debreu. Stochastic choice and cardinal utility. Econometrica: Journal of the Econometric Society. pages 440-444, 1958.
[19] Gerard Debreu. Topological methods in cardinal utility theory, mathematical methods in the social sciences (kj arrow, karlin, and p. suppes, eds.), 1960.

---

[20] Nishant Desai, Andrew Critch, and Stuart J Russell. Negotiable reinforcement learning for pareto optimal sequential decision-making. Advances in Neural Information Processing Systems, 3,, 2018.
[21] Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems, 33:6637 6647, 2020.
[22] Yonathan Efroni, Nadav Mcrlis, and Shie Mannor. Reinforeement learning with trajectory feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7288-7295, 2021.
[23] Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. In International Conference on Learning Representations, 2022.
[24] William Fedus, Carles Gelada, Yoshua Bengio, Marc G Bellemare, and Hugo Larochelle. Hyperbolic discounting and learning over multiple horizons. arXiv preprint (90i:::9900006666 2019.
[25] John Ferejohn and Talbot Page. On the foundations of intertemporal choice. American Journal of Agricultural Economics, 6012):269 275, 1978.
[26] Shane Frederick, George Loewenstein, and Ted O' O'Donoghue. Time discounting and time preference: A critical review. Journal of economic literature, 40(2):351-40.1 2002.
[27] Maor Gaon and Ronen Brafman. Reinforcement learning with non non-n markovian rewards. Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3980-3987, 3987, 2020.
[28] Paul W Glimcher. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proceedings of the National Academy of Sciences, 2011.
[29] Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Com- posable deep reinforcement learning for rohotic manipulation. In 2018 IEEE international conference on robotics and automation (ICRA), pages 6244 6251. JEEE, 2018.
[30] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870, PMLR, 2018.
[31] Peter Hammond. Harsanyi's utilitarian theorem: A simpler proof and some ethical connotations, In Rational Interaction, pages 305-319. Springer, 1992.
[32] John C Harsanyi. Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility. Journal of political economy, 3304)3309-321, 1955.
[33] Geoffrey Heal. Intertemporal welfare economics and the environment. Handbook environmental economics, 3:1105- 1145. 2005.
[34] Robert Jacobs, Michael Jordan. Steven Nowlan, and Geoffrey E Hinton, Adaptive mixtures of local experts. Neural computation, 3(1):79-87, 1991.
[35] Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415 43:4415-4426, 2020.
[36] Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference Autonomous Agents and Multiagent Systems, pages 1181-11888 2015.
[37] Zico Kolter and Andrew Ng. Near-bayesian exploration in polynomial time. In Proceedings of the 26th annual international conference on machine learning, pages 513-520, 2009.
[38] Tjalling Koopmans. Stationary ordinal utility and impatience. Econometrica: Journal of Econometric Society, pages 287-309, 1960.
[39] David Kreps. Notes on the Theory of Choice. Westview press, 1988
[40] David M Kreps and Evan L Porteus. Temporal resolution of uncertainty and dynamic choice theory. Econometrica: journal of the Econometric Society, pages 185-200, 1978.
[41] Romain Laroche, Mehdi Fatemi. Joshua Romoff, and Harm van Seijen. Multi-advisor reinforcement learning. arXiv preprint arXiv:1704.00755 2017.
12

---

[42] Lue Lauwers. Intertemporal objective functions: Strong pareto versus anonymity. Mathemutical Social Sciences, 35(1):37-55, 1998.
[43] R Duncan Luce. Individual choice behavior. 1959.
[44] Mark Machina. Dynamic consistency and non-expected utility models choice under uncertainty. Journal of Economic Literature, 27(4):1622 1668, 1989.
[45] Franz Mandl, Statistical physics, volume 14, John Wiley & Sons, 1991.
[46] Shuwa Miura. On the expressivity of multidimennional markov reward. arXiv preprint 1rXiv:307.12188. 2023.
[47] Hervé Moulin. Fair division and collective welfare. MIT press, 2004.
[48] Krikamol Muandet. Impossibility of collective intelligence. arXiv preprint urXiv:2206.02788. 2022.
[49] Sendhil Mullainathan and Richard Thaler. Behavioral economies, 2000.
[50] Abhishek Naik, Roshan Shariff, Niko Yasui, Hengshuai Yao, and Richard S Sutton. Discounted reinforce- ment learning is not an optimization problem. arXiv preprint 1wX.v:1910.02111 2019.
[51] Geraud Nangue Tasse, Steven James, and Benjamin Rosman. A boolean task algebra for reinforcement lcarning. Advances in Neural Information Processing Systems, 33:9497-9507, 2020.
[52] Andrew Y Ng and Stuart Russell. Algorithmss inverse reinforcement learning. In The Seventeenth International Conference on Machine Learning, pages 663-670, 2000.
[53] Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning. PMLR, 2017.
[54] Silviu Pitis. Rethinking the discount factor in reinforcement learning: A decision theoretic approach. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7949-7956, 2019.
[55] Martin I. Puterman, Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[56] David Raposo, Sam Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matt Botvinick, Hado van Has- selt, and Francis Song. Synthetic returns for long-tenn term credit assignment. arXiv preprint 22X1v:2102.12425, 2021.
[57] Sarah Rathnam, Sonali Parbhoo, Weiwei Pan, Susan Murphy, and Finale Doshi-Velez. The unintended consequences of discount regularization: Improving regularization in certainty equivalence reinforcement learning. International Conference on Machine Learning, pages 28746 28767. PMLR, 2023.
[58] John Rawls. A theory of justice: Revised edition. Harvard university press, 2009.
[59] Zhizhou Ren, Ruihan Guo, Yuan Zhou, and Jian Peng.
Learning long neg-ttemm reward redistribution via randomized return decomposition. In International Conference on Learning Representations, 2023.
[60] Gavin Rens, Jean-François Raskin, Raphaël Reynoaud, and Giuseppe Marra. Online learning of non- markovian reward models. In Proceedings of the Thirteenth International Conference on Agents and Artificial Intelligence. Scitepress, 2020.
[61] Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making, Journal of Artificial Intelligence Research, 48:67-113, 2013.
[62] Leonard Savage. The foundations of statistics. Wiley, 1954.
[63] Matthias Schultheis, Constantin A Rothkopf, and Heinz Koeppl. Reinforcement learning with non- exponential discounting. In Advances neural information processing systems, 2022.
[64] Mehran Shakerinava and Siamak Ravanbakhsh. Utility theory for sequential decision making. In Interna- tional Conference Machine Learning, pages 19616-19625. PMLR, 2022.
[65] David R Shanks, Richard J Tunney, and John D McCarthy. A re-examination of probability matching and rational choice. Journal of Behavioral Decision Making, 15(3):233-250, 2002.
[66] David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel Dulac- Arnold, David Reichert, Neil Rabinowitz, André Barreto, and Thomas Degris. The predictron: End-to-end learning and planning. In The foirty-fourth International Conference Machine Learning, 2017.
13

---

[67] David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence. 299; 199515, 2021.
[68] Satinder P Singh and David Cohn. How to dynamically merge markov decision processes. In Advances in neural information processing systems, pages 1057-1063, 1998.
[69] Joar Skalse and Alessandro Abale. On the limitations of markovian rewards to express multi-objective, risk-sensitive, and modal tasks. In Uncertainty in Artificial Intelligence, pages 1974-1984, PMLR, 2023.
[70] Matthew J Sohel. Ordinal dynamic programming. Management science, 21(9):967-775, 1975.
[71] Robert M Solow et al. Growth theory. an exposition. In Growth theory. An exposition. Oxford: Clarendon Press., 1970.
[72] Nisan Stiemnon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodci, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021. 2020.
[73] Patrick Suppes. Behavioristic foundations of utility. Econometrica: Journal of The Econometric Society, pages 186-202, 1961.
[74] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[75] Richard S Sutton, Joseph Modayil, Michael Delp. Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Voluee Volume 2, pages 761-768. International Foundation for Autonomous Agents and Multiagent Systems, 2011.
[76] Csaba Szepesvári. Constrained mdps and the reward hypothesis. Musings about machine learning and other things (blog), 2020.
[77] Peter Vamplew, Benjamin Smith, Johan Källström, Gabriel Ramos, Roxana Rădulescu, Diederik M Roijers, Conor F Hayes, Fredrik Heintz, Patrick Mannion, Pieter JK Libin, et al. Scalar reward is not enough: A response to silver, singh, precup and sutton (2021). Autonomous Agents and Multi-Agent Systems, 36(2):41, 2022.
[78] Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominat- ing policies. The Journal of Machine Learning Research, 15(1 15(43333333351,, 2014.
[79] Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-ohjective reinforcement learning: Novel design techniques. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). pages 191-199. JEEE, 2013.
[80] Harm Van Seijen, Mehdi Fatemi, and Arash Tavakoli. Using logarithmic mapping to enable lower discount factors in reinforcement learning. Advances Neural Information Processing Systems, 32, 2019.
[81] John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior. Princeton university press, 1953.
[82] Nir Vulkan. An economist's perspective on probability matching. Journal of economic surveys, 14(1): 101-118, 2000.
[83] Martin Weizzman. Gamma discounting. American Economic Review, 91(1):200-271,
[84] John A Weymark. Measurement theory and the foundations of utilitarianism. Social Choice and Welfare, 25(2-3):527-555, 2005.
[85] Martha White. Unifying task specification in reinforeement leariig. In The Thirty-fourth International Conference on Machine Learning, 2017.
[86] Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fümkranz. A survey of preference-based reinforcement leamning methods. Journal of Machine Learning Research, 18( 8116):11-46 2017.
[87] John I Yellott Jr. The relationship between luce's choice axiom, thurstone's theory of comparative judgment, and the double exponential distribution. Journal of Mathematical Psychology, 15(2):109-144, 1977.
[88] Stephane Zuber. The aggregation of preferences: can we ignore the past? Theory and decision, 70(3): 367-384, 2011.
14

---

A
Notation Glossary
Table I: Notation summary.
Basic notation
C()
the set of probability distributions its argument set of its argument
M
generalized MDP T, To, 7) the state space S with generic state E S
S,s
A, a
the action space A with generic action a A transition function 5 A
((S)
the reward function 8 A
Y
the generalized discount factor 7 S A R
trajectory [[So, (n), ($1, o.)....] which may may not terminate
h
history (trajectory up to time !)
m
stationary policies т,, W S ((A)
11
non-stationary policies (TT, Tó. TT41 To 1+ ....)
policy space, so n, w; 11, Ω E ПI; note C(II) = II space
of prospects over which preferences are expressed; note
C(P) = L(S) x II
Pn
set historical prospects starting with h; NB: (h2, IT) & Phş hı # h2. set prospects on augmented state space S {y:}
plu)
[I, E
prospect (stochastic process representing the controlled future)
II)
historical prospect (state space has been expanded to include h)
hs, sa, aTI, aTT,
concatenations of subcomponents of trajectorieshhssooree/ppiiii for example, the non-stationary policy sll (a, Io, .I.....)
V
value function V L(S) II - R, which is equal to the discounted sum of future
rewards: V(5, II) = E.g-s.n E [I[ 7(5-11 az- 1)] r(s;, az)
supp(f)
the support distribution ø index
individuals (assumed finite)
wy, (x(s), (h) (h) da
the ith aggregation weight, either constant, function of state history
the ith aggregation factor for history h
the stationary state-action distribution for policy
Generic modifiers
index of sequence, vector, collection of agents
the slice from to j of sequence or vector; e.g., of k is the trajectory slice [(s,, a.).... (#t+R, d++x]]
indicates an aggregated item (c.g., >5 is social/aggrogated preference)
indicates next timestep when time implicit (e.g.. s')
indicates probability distribution (e.g., E L(S)) indicates preference relation on Ps.
Operators
defined (is the definition
strict preference
weak preference (p 2 +> q X
indifference (p y and q
15

---

B Proofs
Theorem 4 (Impossibilitty). Assume there exist distinct policies, 11, A. none of which a mixture (ie., convex combination) of the other two, and consider the aggregation of arbitrary individual preference relations {> E I} defined on L(P) that individually satisfy Axioms 1-2. There does not exist aggregated preference relation >g satisfying Axioms 1-4.
Proof. Fix 8, a. Using Theorem 2, choose {ri, 71 rE, 7E to represent individual and aggregate preferences over ((P). Define mixture policy a == 31I + (1 - 3)92. W.l.o.g. assume [Z] 2.
We Axiom 4 (Unrestricted Domain) to set
(T(s, a) II >1 (T(s, A) > (T a2, (T(s, >2 (T(s, A) >2 (T(s,a), IT),
and, using Theorem I to shift V1, 1/2 we have
Vi(T(s, A) = (2, A),
(I(T(s,)),I) > 0 > (i,T(s,))).) Q) s.t. Vi(T(s,,)) a), ll.,) = 0, s.t. V3(T(s, a), pp,) = 0,
(8)
(2(T(s, Q) > 0 > (s,
for some 31, 32 E (0, 1) with 31 + B2 (again appealing to Unrestricted Domain). Intuitively, equation requures individual preferences to conflict, and the 31 * 32 condition requires them to be non-symmetric about A. Equation iis merely a convenient choice of numerical representation.
We now apply Theorem followed by Theorem the expression (I alls) VE(s, aA). We again invoke Theorem to shift Vi2 and eliminate the constant term, so that by Theorem 3 3 {w;} for which,
Vs(s, alls) - (s(s, = Σ w; VisVis,, alls) - Σ w; as) EEI
iEI
=wrnss, a) [Vi a), [I3 Vi(T(s,a),
(9)
w272(8, a) [V2 8,, a) II,) (2(T(ss,), A)]
= (V a), II3) + w272(s, a) V/2 (T 1s).
where the second line applies Theorem with rewards cancelling out.
Altemattvely, applying Theorem 2 followed by Theorem 3 to the same expression yields,
VT(8) Vi(s, aA) = 7x ()) (T)) (V)(,, a), wrnx(s a) - -)) w2n2(s, [
(10)
Combining equations and 10
obtain:
w1 (s (T(s, + (ttp::/lll__ttto.....////.//,/,
(11)
a a), + wrrz(s a)
Finally, setting 3 to be 31 or B2 in equation we obtain the equalities:
(x) a), [T,,) wn rr a) (T(,,)) lT,),
(12)
w272 (2T a), Hl,))
The w; are non-zero (Axiom 4 Sensitivity) and the remaining values are non-zero (else 31 -2). so conclude that (x a) (s, a) 72(s, a). But this contradicts assumption that individual preferences {- E Σ} may be chosen arbitrarily, completing the proof.
16

---

the Theorem 5 (Possibility). Consider the aggregation of arbitrary individual preference relations {> ]} defined on L(P). and consequently ((Pa), wh, that individually satisfy Axioms 1-2. There exists aggregated preference relations {>E), defined on ((Pr), Yh, that satisfy Axioms 1-4.
In particular, given 8, a, V5, {Vi). {w;(s)}, where (A) each Vi satisfies Axioms 1-2 on [(P), and Vr. satisfies Axioms 1-2 (PPh), Vh, and (B) Vx(s,al) E; (us(s)V((ssall) then, choosing
wi((as') == ws(sa) == (=))))((), for all i,6'
(13)
implies that Vy(sas', II) x E, (;(ss) (I(,, that the aggregated preferences {>gndt satisfy Axiom 3 on sas)) Unrolling this sosultwy(hsaa == w. w,(hs));(s, a) p)pruucccces a set of constructive, history dependent weights {w;(h)} such that Axiom 3 is satisfied for all histories {1e).
Proof. We follow the proof of Theorem 4 Fix s, a. Using Theorem 2, choose (res wh rE, TE to represent individual and aggregate preferences over ((Ps) and C(Panst). Define mixture policy 11 = 311 + (1 - 3)92. W.l.o.g. assume [Z] 2.
We use Axiom 4 (Unrestricted Domain) to set
(T(s,a), IT) >1 (T(s, A) ~1 (T(s, ((s,a), (T(),a) >2 (T(s,a), A) >2 (T(s.a),I))
(14)
and, using Theorem I to shift V1, 1/2 we have a))
Vi(T(s,)), II) > 0 > (T, s.t. (s, H1,)) 0,
(15)
V2(T(s,a),,) > 0 > V2(T((s, a), II) s.t. V2(T(s,, a), l1,)) = 0,
for some 31, 12 E (0 with 31 + B2 (again appealing to Unrestricted Domain).
We now apply Theorem 3 followed by Theorem to the expression (s(s,, allل) Vz(ss, aA). We again invoke Theorem to shift V/ and eliminate the constant term, so that by Theorem 3 3 (s) for which,
alla) - Vs(s, = Σ (ss, Σ Wi(e Vi(s, aA)
w.(()))(s..) IIs) V)) (V (n(())(ss.))i((((ss))) 11)) (=))))a VV2
(16)
(a(
where the second line applies Theorem 2, with rewards cancelling out.
It suffices to find one set of satisfactory w;;h) (h), the same for all s'. This will allow us to factor out below. Then, applying Theorem 2 followed by Theorem to the same expression yields,
assume that, given 8, a, w; (sas')) == w w(s))
Vi(s, allp) Vi(s, aA)
72(s, V) a), Ms) 1. wi (*a))((s (VT(( [T)) a() ())
(17)
- A|
wi (V) [3)) (3)
Combining equations 16 and 17 setting 3 to be 31 or B2 in equation, and rearranging, we obtain equalities:
(s. wi (sa))1(ss a) and w2(s))a(ss w=)s))x((s,0)
(18)
from which we conclude that:
wz (2(s))
u2 (2()))2(8,0)
(19)
wi (sa)
wi (s)
This shows the existence of weights w=(s)), unique up to a constant scaling factor, for which VE(T(ss, a), II) x En Wi(sa)V.(T(ss a), II), that apply regardless of how individual preferences chosen or aggregated at s. Unrolling the result completes the proof.
17


Reviews:
[{"review_id": "zmjlYX5ti3", "reviewer": "Reviewer_WDhN", "summary": "This paper examines multi-objective reinforcement learning---the setting in which multiple distinct objectives are desired, and often combined, to form a composite objective. Concretely, the paper explores the limits of aggregating different objectives by appealing to three main pools of ideas. First, to the von Neumann-Morgenstern expected utility, axioms; Second, to Pareto indifference; and Third, to dynamic consistency. The main result is an impossibility result, illustrating that objectives with different time-based objectives (that is, different discount factors), cannot be aggregated in a way that yields a Markovian reward function even if the individual objectives themselves are Markovian. The paper then explores what lies beyond this impossibility result, exploring a mechanism for expanding the state-space to collapse the non-Markovian aggregated objective down to a Markovian one. This results in a \"historical\" discount factor, a hindsight view of the discount factor. ", "questions": "Primary recommendation: My strongest recommendation is to move Section 4 to the Appendix, and to use the remaining space to provide additional clarity and exposition around key ideas around results. I believe all of the Axioms need more careful and simpler introductions and discussions (and especially Axiom 3), as well as the main results. I believe this switch will strengthen the paper considerably.\n\nMain Comments/Questions:\n- \"Our main contribution is an impossibility result from which one concludes that non-Markovian rewards are likely necessary for RL agents that pursue multiple objectives or serve multiple principals\". This seems slightly too strong, by my reading of the main result---it really only applies when the discounts across the objectives differ. Is this the role that \"likely\" is playing in this statement? If so I might suggest adjusting the language to be more precise.\n- Separately, I wonder if the paper can comment on when the discount should be associated with an _agent_ rather than an _objective_ in isolation.\n- Two pieces of related work come to mind. First, the expressivity of \"multi-dimensional\" reward was explored by Miura in 2022: On the Expressivity of Multidimensional Markov Reward at the RLDM workshop on RL as Agency. I do wonder about the connections between these two sets of results. Second, Tasse et al. propose an algebra on tasks in RL in \"A boolean task algebra for reinforcement learning\". I wonder about how this style of composition bears on the findings and perspectives from the present paper.\n\nSmall Questions:\n- \"While this allows us to design agents that implement these policies, it doesn’t quite solve the intertemporal choice problem\": what is meant by this statement? Actually designing agents that implement these policies does not seem to be in the purview of this paper. Instead, we can aggregate objectives in a way that allows us to _incentivize_ agents according to an appropriate objective, but this is different from the agent design process.\n- I do not understand Axiom 3 as stated and the explanation of the notation below is too brief to elucidate. Given the importance of the Axiom, I encourage expanding the explanation.\n\nTypos and writing suggestions:\n- As mentioned above, I think the phrase \"multi-objective agency\" is not the appropriate way to describe the main concept of the paper. I suggest replacing \"multi-objective agency\" with \"multi-criteria objectives\", or \"multiple objectives\", or some variant thereof.\n- I don't believe \"(history dependent)\" is needed in the abstract following \"non-Markovian\"\n- It looks as thought reward functions are defined as both $R$ (in the definition of the MDP) and $r$ (in, say, Theorem 2). I would encourage picking one use throughout.\n\n", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "**STRENGTHS**\n\nThe paper possesses many strengths:\n1) The aspirations of the work are ambitious and important. Clearly establishing when certain kinds of objectives can and cannot be captured is important.\n2) The work is rigorous, and well-connected to classical results in decision theory.\n3) The examples are clear and help to communicate the main ideas.\n4) The impossibility result on its own is interesting. Once I understood the details, it is not ultimately surprising, but I do not believe the result needs to be surprising to be useful.\n5) Historical discounting is a new and interesting idea.\n", "weaknesses": "\n**WEAKNESSES**\n\nAt the same time, the paper has several weaknesses:\n1) Language. First, and my biggest critique, the work commonly makes use of unusual and vague language surrounding some of the main concepts. For instance, the title, and central idea of the work---multi-objective agency---is not well-defined, and by my reading is not an appropriate choice of description for the content of the work. I would recommend moving away from \"agency\" as a term, and certainly \"multi-objective agency\", as neither are well defined in this paper. Instead, I would suggest using \"multi-critieria objectives\", or \"multi-objective RL\", as is used in prior literature. It is much more clear and precise, and more well connected with the work.\n2) Clarity, and detail of exposition. Ideas are often introduced abruptly and not explained in much detail. For instance, the axioms in section 3.1 are simply stated without any added context or explanation. While vNM is quite common, dynamic consistency is less so, and likely deserves a more thorough, careful, and simple explanation, given its central role in the work. Similarly, after some theorems are introduced, they are sometimes not discussed. Other details are often left unexplained, such as \"...none of which is a mixture of the other two\" in Theorem 4 (the main result). From a quick reading it is possible to understand this, but it would be worthwhile to spell this out carefully.\n3) Notation is sometimes defined quite precisely, but it is often overly complex or not defined. For instance, in Theorem 5, it is unclear what {$\\succ_\\Sigma^{sas'}$} is intended to mean. Conventions tend to deviate quite a lot from typical work in reinforcement learning as well (getting rid of the Q function in favor of two uses of V, using $\\Pi$ instead of $\\pi$). \n4) Unsurprising results. Lastly, I do believe most of the results are unsurprising. When discount factors vary across objectives, it is perhaps expected that their aggregation will not be representable in the same form (and that we can augment the state space to remedy this). Still, it  is useful to make these arguments carefully and rigorously.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and actionable suggestions. Please see the general response for the most important points. Otherwise:\n\n**Primary Rec:** Regarding your suggestion to move Section 4 to the Appendix, we assume you are referring to just the “Relaxing Markov Preference” section (up to Subsection 4.1). We think perhaps Subsection 5.3 is an alternative candidate for the Appendix. In any case, we will do our best to improve the exposition per your suggestion.\n\n**Comment 1:** That was indeed the role that “likely” was meant to play (see ** in Global discount factor in General Response); we will adjust.\n\n**Comment 2:** Our bias is toward viewing discount factors in the economic sense, as representations of “time preference”, which arises naturally when considering preferences over temporal processes / trajectories / streams of consumption. This view would make the discount factor a property of “preferences”, which could either be expressed globally, as the preferences of a complex agent (e.g. a household robot), or individually with respect to a particular objective (e.g. the paperclip making agent). We are not sure if this is responsive to your comment (we see agent and objective as being entangled), but it is in contrast to a rather common view in RL that discounting is either a mathematical convenience, or done for purposes of optimization or regularization. We will consider adding some commentary on this in Subsection 5.4 (Related work in RL). \n\n**Comment 3:**\n    \n- *Miura 2023*: Thank you, we were not aware of this work. Based on our read, Miura shows that any \"set of acceptable policies\" (as previously defined in Abel et al. 2021) can be represented as the solution set to a constraint-based multi-objective problem (i.e., all acceptable policies, and only acceptable policies, perform better than some lower bound with respect to all objectives). This type of constraint-based composition therefore reduces to a *set* of (equally preferred) policies, whereas the scalarization-based composition in our work reduces to a single social reward function (preference ranking). The fact that Markovian scalarization is insufficient to represent a \"set of acceptable policies\" was shown in Abel et al. 2021. \n\n- *Tasse et al. 2020*: In the Boolean Task Algebra, Tasse et al. consider a particular family of \"goal achievement\" tasks that make it possible to precisely compose tasks using boolean operations (min/max). Like our work, it is a scalarization-based approach to value function composition. Although it appears as non-linear scalarization (via min/max), we think it can be understood as taking the extreme of the linearly scalarized soft-value function composition approach taken by Haarnoja et al. 2018 and Van Niekerk et al 2019, which makes it related to the discussion in our Subsection 5.3. Unlike our work, it is restricted to a particular family of goal achievement tasks; furthermore they assume their tasks are terminating/undiscounted, so our results would not apply. \n\n**Question 1:** You are correct, thank you. We will adjust.\n\n**Question 2:** As applied to voting / social choice, Axiom 3 says that if all members in a society (indexed by $\\mathcal{I}$) are indifferent between two (lotteries of) alternatives ($\\tilde p$ and $\\tilde q$), then so too is society. We will improve the exposition here. "}, {"review_id": "tpk31EZP05", "reviewer": "Reviewer_j2th", "summary": "The authors analyze the implications of preference aggregation within a Markov Decision Process Framework. They show that it is not possible to ensure dynamic consistency in an aggregated MDP if one also wants to be able to accommodate arbitrary preference criteria, even if the criteria are individually dynamically consistent. They further show that by relaxing the Markov condition incrementally, dynamic consistency can be recovered.\n", "questions": "Technical Comments\n\nC1. The whole concept of a discount factor in MDPs is to represent a global time preference. Why should anyone have thought it could be coherent to have different time preferences on different criteria? To the extent that discounting is really capturing dynamics in the world (e.g., consequences of work persisting), then arguably that would more properly be represented in the state space to begin with. That is, using the discount factor for this is a hack and one should not be surprised it is fragile.\n\nC2. Axiom 3 is really strong. It basically entails strong independence of criteria--what is needed to get an additive representation. Many readers will not realize that. Moreover, I suspect it is much stronger than needed to get your key impossibility result. That is, even forms with many more interaction terms will probably run into problems with temporal consistency.\n\nC3. On reflection, it seems to me that the key technical point here could be restated as saying that given Axiom 3, the *only* thing that can go wrong is mismatch of discount factors. Do you agree with that characterization?\n\nQuibbles\n\nQ1. About the procrastination story. Line 67 says that the policy must remember it had previously chosen \"play\" in order to work forever. Not true (it actually does not matter what was executed in the first step): it just needs to know it is beyond the first step.\n\nQ2. Title. \"Multi-objective agency\" is a very ill-defined term, so hard to buy the assertion it requires anything in particular. What is actually proved here is about what is required for dynamically consistent preference aggregation. \n\nQ3. Exposition had a few minor gaps, for example \\cal{P} never formally defined (clear from context, though).", "limitations": "\nThe paper adequately discusses assumptions and limitations.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Addresses a fundamental representational issue in MDPs. The authors clearly have a deep understanding of the temporal consistency literature in economics and decision theory, and bring it to bear here. The technical exposition is clear, and the reasoning is sound. The example of procrastination is instructive. The construction of a patch to deal with different discount rates is perhaps the most directly useful contribution. There is also some interesting extended discussion about intemporal preferences that could be quite relevant in an AI context.\n", "weaknesses": "The authors attempt to motivate the contribution in the context of the current \"Reward is Enough\" debate in RL. This is tantalizing, but seems to me a bit forced. Really we have a basic technical question in representation of intertemporal preferences, and the paper underlines a common lesson that aggregating across separate preferences is never as straightforward as one might think. The technical points are perhaps connected to some arguments debate participants have brought up (and they should indeed be better versed in intertemporal choice), but ultimately we face the same questions we always do about how much and what kind of state should we incorporate to keep things approximately enough Markovian.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and helpful commentary/questions.\n\n> The authors attempt to motivate the contribution in the context of the current \"Reward is Enough\" debate in RL. This is tantalizing, but seems to me a bit forced. …\n\nWe think it is relevant for the following reason:\n\n- Suppose we have decided “how much and what kind of state” keeps things “approximately enough Markovian” for Alice and Bob, who have different preferences. Now suppose Alice and Bob purchase an LLM-based personal assistant Carl, who inherits (i.e. aggregates) their preferences. *The same state that was sufficiently Markovian for Alice and Bob may not be sufficiently Markovian for Carl.*\n\nIf Carl now becomes a principal (e.g. it delegates some work to household robot Dave), we may need to again adjust our definition of \"sufficiently Markovian\" for Dave. And so on. \n\n--- \n\n**C1:** Please see general response. To the extent that different humans can have different time preferences, then agents serving multiple humans will face this problem. We think discounting captures more than just the “dynamics in the world” (and can actually be considered entirely apart from dynamics, as is the case in work that analyzes discounting of consumption streams (e.g. Koopmans 1960/Diamond 1965, etc.)). From a representational standpoint, we agree that the state space can be used to absorb everything, but such a representation may give up the compression advantages of Markovian rewards/discounts. \n\n**C2:** “I suspect it is much stronger than needed to get your key impossibility result.” <- this is a really interesting idea, which we will consider (at least for future work). That being said, while the consequence of Axiom 3 (together with VNM) is strong, we do not think the Axiom itself is strong: we think it is difficult to come up with a reasonable case where society (the agent) should prefer A to B if all members of society (the principals) are indifferent between A and B. Variations of the pareto axiom are commonly assumed to be desirable (e.g. for Arrow’s Impossibility Theorem and other work on social welfare functions, and the works cited in Subsection 5.1). \n\n**C3:** Yes. Mismatch of discount factors is the only way Theorem 4 takes effect. That said, per our general response, we argue that mismatch of time preference is the general case whenever there are multiple principals. \n\n**Q1:** Good point, thank you. This could definitely be repaired by a more complex example, but TBD if we can maintain the simplicity while keeping the original claim. We will either correct the claim or tweak the example slightly. \n\n**Q2-Q3:** See general response. \n"}, {"review_id": "oVC8NIBvRI", "reviewer": "Reviewer_qGsM", "summary": "The paper considers a general multi-objective sequential decision-making setting where each objective may use a different discount factor. Using an axiomatic approach, the authors prove that under some axioms (vNM axioms + dynamic consistency for the relation on each objective), an aggregated preference relation cannot simultaneously satisfy the vNM axioms, dynamic consistency, Pareto indifference, and some technical conditions. In addition, the authors discuss some ways out to this impossibility result, notably via state augmentation or relaxing dynamic consistency.", "questions": "None", "limitations": "Not applicable, this is a theoretical paper", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 2, "strengths": "Impossibility result, although the result is actually not very surprising when different discount factors are allowed\n\nProposition and discussion about different solutions to this impossibility result", "weaknesses": "The presentation and organization of the paper could be improved. Notably:\n\nThe results could be presented in a more accessible way to a more general audience. The authors seems to know well the related literature in decision theory and economics, which may not necessarily be the case for the NeurIPS audience. For instance, the second and third paragraphs of Section 5.1 are quite hard to follow for a non-expert.\n\nSection 4.2 should be checked. Some notations (e.g., h_{:-1}, y_i, or y) are not explained properly or are not used in a rigorous way.\n\nIt is not clear to me why Section 5 combines a presentation of other solutions to the impossibility theorem and a discussion of related work.\n\nAs far as I know, most work in multiobjective reinforcement learning applies an identical discount factor on all the objectives. Since the impossibiity theorem doesn't apply in this case, the results of this paper don't apply to most such work. Therefore, I believe researchers may be mislead by the title of this paper. I suggest the authors to use a more precise one.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful commentary. Please see the General Response above re: most points. \n\nOtherwise, the reason Section 5 is somewhat of a hybrid is because (1) the parts about intertemporal choice and stochastic preference necessarily introduce \"related work\" that wouldn't fit, or would be duplicative in a standalone related work section, and (2) having a standalone related work section for just reinforcement learning work seems odd. We're not sure how to improve this, but welcome any suggestions."}]

Author rebuttal:
We thank the reviewers for their time and detailed reviews. We find the reviewers have understood our work and have provided helpful suggestions. We are acting on several of the suggestions, as noted below, and agree the changes will improve the paper. We welcome any additional feedback and questions during the discussion period.

General responses / planned changes:

- **Title / “Multi-Objective Agency” (WDhN, j2th, qGsM):** Upon reflection we agree this was not a good choice, and will change the title. We are considering alternatives that move away from the word “agency” and are explicit about “diverse time preference”, along the lines of:
    - Aggregating Objectives with Diverse Time Horizons Requires Non-Markovian Rewards
    - Non-Markovian Aggregation of Objectives with Diverse Time Horizons
    - Where Markovian Scalarization Fails: Aggregating Objectives with Diverse Time Preferences

	We will make similar tweaks throughout to improve precision.
	
- **Global discount factor vs per objective discounting (WDhN, j2th in C1):** If an “Actor” possesses a single global discount function, and there are two such Actors (e.g. Mom and Dad), jointly acting as principals for a third Actor that we will call the “Agent” (e.g. a household Robot), then the Agent has inherited objectives/rewards from its principals that may** have conflicting time preference. 
    
    (**actually, we believe this to be the general case, not merely a rare occurrence; see, e.g., Frederick, Loewenstein, and O’Donoghue. 2002, Section 8 Paragraph 2, “Thus, there is no reason to expect that discount rates should be consistent across different choices”). 

- **Unsurprising main result (WDhN, qGsM):** A number of recent works have recognized or used (Markovian) transition dependent discounting (e.g. Bowling et al. 2023), and the initial question that led to this work was actually, “*How might such a transition dependent discount factor naturally arise?*”. 

    Our initial hypothesis was that it might arise through aggregation of objectives with different fixed-discount factors (i.e., our initial hypothesis was a possibility). So while we agree that the impossibility of linearly aggregating MDPs with different discounts into an MDP with a fixed discount and the same state space is not surprising (as recognized by, e.g., Singh & Cohn 1998), and we understand why others might view our result as unsurprising, we were, in fact, surprised by the impossibility.

- **Exposition (WDhN, qGsM):** Per your suggestion, we will do our best to make room for some additional exposition around the Axioms/Theorems to make it more accessible to a NeurIPS audience, while maintaining the current content and structure. This may involve moving certain bits to the Appendix, per Reviewer WDhN’s suggestion. 

- **Notation (WDhN, j2th, qGsM):** We will double check usage for precision / rigor / consistency and add a notation glossary to the Appendix. 


Meta-review: The reviewers unanimously praised the paper for its soundness, and agreed that the core technical contributions will be of interest to the community, and thus I am recommending this paper be accepted. Reviewers further agreed that the title, and one of the primary terms of the work ("Multi-objective agency"), is ill-defined, and best changed to something less contentious. I agree with this recommendation, and from the discussion it sounds as though the authors are willing to make this change. Of the proposed titles, I am most inclined towards "Aggregating Objectives with Diverse Time Horizons Requires Non-Markovian Rewards", but I leave this to the authors' discretion. I otherwise encourage the authors to take the reviewers' other suggestions into account.
Decision: Accept (poster)