Title: Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards
Venue: NeurIPS 2023
Accepted: yes
Keywords: Normative Agency Design

Reward Design

Sequential Decision Making

Reinforcement Learning

Intertemporal Fairness

Multi-Objective Decision Making
Abstract: As the capabilities of artificial agents improve, they are being increasingly deployed to service multiple diverse objectives and stakeholders. However, the composition of these objectives is often performed ad hoc, with no clear justification. This paper takes a normative approach to multi-objective agency: from a set of intuitively appealing axioms, it is shown that Markovian aggregation of Markovian reward functions is not possible when the time preference (discount factor) for each objective may vary. It follows that optimal multi-objective agents must admit rewards that are non-Markovian with respect to the individual objectives. To this end, a practical non-Markovian aggregation scheme is proposed, which overcomes the impossibility with only one additional parameter for each objective. This work offers new insights into sequential, multi-objective agency and intertemporal choice, and has practical implications for the design of AI systems deployed to serve multiple generations of principals with varying time preference.

Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards

Abstract

As the capabilities of artificial agents improve, they are being increasingly deployed to service multiple diverse objectives and stakeholders. However, the composition of these objectives is often performed ad hoc, with no clear justification. This paper takes a normative approach to multi-objective agency: from a set of intuitively appealing axioms, it is shown that Markovian aggregation of Markovian reward functions is not possible when the time preference (discount factor) for each objective may vary. It follows that optimal multi-objective agents must admit rewards that are non-Markovian with respect to the individual objectives. To this end, a practical non-Markovian aggregation scheme is proposed, which overcomes the impossibility with only one additional parameter for each objective. This work offers new insights into sequential, multi-objective agency and intertemporal choice, and has practical implications for the design of AI systems deployed to serve multiple generations of principals with varying time preference.

1 Introduction

The idea that we can associate human preferences with scalar utility values traces back hundreds of years and has found usage in numerous applications [9, 71, 28, 49]. One of the most recent, and perhaps most important, is the design of artificial agents. In the field of reinforcement learning (RL), this idea shows up as the reward hypothesis [74, 67, 10], which lets us define objectives in terms of a discounted sum of Markovian rewards. While foundational results from decision theory [81, 62] and inverse RL [52, 54] justify the reward hypothesis when a single objective or principal is considered, complexities arise in multi-objective scenarios [61, 77]. The literature on social choice is largely defined by impossibilities [5], and multi-objective composition in the RL and machine learning literature is typically restrictive [68, 51], applied without clear justification [29], or based on subjective evaluations of empirical efficacy [21]. Addressing these limitations is crucial for the development of artificial agents capable of effectively serving the needs of diverse stakeholders.

This paper extends previous normative work in RL by adopting an axiomatic approach to the aggregation of objectives. The approach is based on a set of intuitively appealing axioms: the von Neumann-Morgenstern (VNM) axioms, which provide a foundation for rational choice under uncertainty; Pareto indifference, which efficiently incorporates individual preferences; and dynamic consistency, which ensures time-consistent decision-making. From these axioms, an impossibility is derived, leading to the conclusion that optimal multi-objective agents with diverse time preferences must have rewards that are non-Markovian with respect to the individual objectives. To address this challenge, a practical state space expansion is proposed, which allows for the Markovian aggregation of objectives requiring only one parameter per objective. The results prompt an interesting discussion on dynamic preferences and intertemporal choice, leading to a novel 'historical discounting' strategy that trades off dynamic consistency for intergenerational fairness. Finally, it is shown how both our results can be extended (albeit non-normatively) to stochastic policies.

The remainder of this paper is organized as follows: Section 2 motivates the problem by modeling human procrastination behavior as an aggregation of two objectives, work and play, and showing how a plan that appears optimal today may lead to the worst possible future outcome. Section 3 presents the axiomatic background and the key impossibility result. Section 4 presents the corresponding possibility result and a practical state expansion to implement it. Section 5 relates the results to intertemporal choice, proposes N -step commitment and historical discounting strategies for managing intergenerational tradeoffs, extends the results to stochastic policies, and discusses related topics in RL. Section 6 concludes with some final thoughts and potential future research directions.

2 Motivation: The Procrastinator's Peril

Webegin with a numerical example of how the naive aggregation of otherwise rational preferences can lead to undesirable behavior. The example, which will be referred to throughout as the 'Procastinator's Peril', involves repeated procrastination, a phenomenon to which the reader might relate. An agent aggregates two competing objectives: work and play . At each time step the agent can choose to either work or play . The pleasure of play is mostly from today, and the agent doesn't value future play nearly as much as present play . On the other hand, the consequences of work are delayed, so that work tomorrow is valued approximately as much as work today.

Let us model the agent's preferences for work and play as two separate Markov Decision Processes (MDP), each with state space S = ∅ and action space A = { w , p } . In the play MDP, we have rewards R ( p ) = 0 . 5 , R ( w ) = 0 and a discount factor of γ play = 0 . 5 . In the work MDP, we have rewards R ( p ) = 0 , R ( w ) = 0 . 3 and a discount factor of γ work = 0 . 9 . One way to combine the preferences for work and play is to value each trajectory under both MDPs and then add up the values. Not only does this method of aggregation seem reasonable, but it is actually implied by some mild and appealing assumptions about preferences (Axioms 1 and 3 in the sequel). Using this approach, the agent assigns values to trajectories as follows:

We see that the agent most prefers τ 3 : one period (and one period only!) of procrastination is optimal. Thus, the agent procrastinates and chooses to play today, planning to work from tomorrow onward. Come tomorrow, however, the agent is faced with the same choice, and once again puts off work in favor of play . The process repeats and the agent ends up with the least preferred alternative τ 1 .

This plainly irrational behavior illustrates the impossibility theorem. Observe that the optimal policy τ 3 is non-Markovian-it must remember that the agent has previously chosen play in order to work forever. But any MDP has a stationary optimal policy [55], so it follows that we need rewards that are non-Markovian with respect to the original state-action space. Alternatively, we will see in Subsection 4.2 that we can expand the state space to make the optimal policy Markovian.

3 Impossibility of Dynamically Consistent, Pareto Indifferent Aggregation

Notation We assume familiarity with Markov Decision Processes (MDPs) [55] and reinforcement learning (RL) [74]. We denote an MDP by M = ⟨S , A , T, R, γ ⟩ , where γ : S × A → R + is a state-action dependent discount function. This generalizes the usual 'fixed' γ ∈ R and covers both the episodic and continuing settings [85]. We use lowercase letters for generic instances, e.g. s ∈ S , and denote distributions using a tilde, e.g. ˜ s . In contrast to standard notation we write both state- and state-action value functions using a unified notation that emphasizes the dependence of each on the future policy: we write V ( s, π ) and V ( s, aπ ) instead of V π ( s ) and Q π ( s, a ) . We extend V to operate on probability distributions of states, V (˜ s, Π) = E s ∼ ˜ s V ( s, Π) , and we allow for non-stationary, history dependent policies (denoted by uppercase Π , Ω ). With this notation, we can understand V as an expected utility function defined over prospects of the form (˜ s, Π) . We use the letter h to denote histories (trajectories of states and actions)-these may terminate on either a state or action, as may be inferred from the context. For convenience, we sometimes directly concatenate

histories, states, actions and/or policies (e.g., hs , sa , s Π , a Π ) to represent trajectory segments and/or the associated stochastic processes. For simplicity, we assume finite |S| , |A| .

3.1 Representing rational preferences

This paper is concerned with the representation of aggregated preferences, where both the aggregation and its individual components satisfy certain axioms of rationality. We define the objects of preference to be the stochastic processes ('prospects') generated by following (potentially non-stationary and stochastic) policy Π from state s . Distributions or 'lotteries' over these prospects may be represented by (not necessarily unique) tuples of state lottery and policy (˜ s, Π) ∈ L ( S ) × Π =: L ( P ) . We write (˜ s 1 , Π) ≻ (˜ s 2 , Ω) if (˜ s 1 , Π) is strictly preferred to (˜ s 2 , Ω) under preference relation ≻ .

To be 'rational', we require ≻ to satisfy the 'VNM axioms' [81], which is capture in Axiom 1:

Axiom 1 (VNM) . For all ˜ p, ˜ q, ˜ r ∈ L ( P ) we have:

Asymmetry : If ˜ p ≻ ˜ q , then not ˜ q ≻ ˜ p ;

Negative Transitivity : If not ˜ p ≻ ˜ q and not ˜ q ≻ ˜ r , not ˜ p ≻ ˜ r ;

Independence : If ˜ p ≻ ˜ q , then α ˜ p +(1 -α )˜ r ≻ α ˜ q +(1 -α )˜ r , ∀ α ∈ (0 , 1] ;

Continuity

: If

˜

p

≻

˜

q

≻

˜

r

, then

∃

α, β

∈

(0

,

1)

such that

α

˜

p

+(1

-

α

r

≻

˜

q

≻

β

˜

p

+(1

-

β

r

;

where α ˜ p +(1 -α )˜ q denotes the mixture lottery with α % chance of ˜ p and (1 -α )% chance of ˜ q .

Asymmetry and negative transitivity together form the basic requirements of a strict preference relation-equivalent to completeness and transitivity of the corresponding weak preference relation, ⪰ (defined as p ⪰ q ⇔ q ̸≻ p ). Independence can be understood as an irrelevance of unrealized alternatives, or consequentialist, axiom: given that the α % branch of the mixture is realized, preference between ˜ p and ˜ q is independent of the rest of the mixture (i.e., what could have happened on the (1 -α )% branch). Finally, continuity is a natural assumption given that probabilities are continuous.

We further require ≻ to be dynamically consistent [70, 40, 44]:

Axiom 2 (Dynamic consistency) . ( s, a Π) ≻ ( s, a Ω) if and only if ( T ( s, a ) , Π) ≻ ( T ( s, a ) , Ω) where T ( s, a ) is the distribution over next states after taking action a in state s .

This axiom rules out the irrational behavior in the Procrastinator's Peril, by requiring today's preferences for tomorrow's actions to be the same as tomorrow's preferences. While some of these axioms (particularly independence and dynamic consistency) have been the subject of debate (see, e.g., [44]), note that the standard RL model is more restrictive than they require [54].

The axioms produce two key results that we rely on (see Kreps [39] and Pitis [54] for proofs):

Theorem 1 (Expected utility representation) . The relation ≻ defined on the set L ( P ) satisfies Axiom 1 if and only if there exists a function V : P → R such that, ∀ ˜ p, ˜ q ∈ L ( P ) :

Another function V † gives this representation iff V † is a positive affine transformation of V .

Using Theorem 1, we extend the domain of value function V to L ( P ) as V (˜ p ) = ∑ z ˜ p ( z ) V ( z ) .

Theorem 2 (Generalized Bellman representation) . If ≻ satisfies Axioms 1-2 and V is an expected utility representation of ≻ , there exist R : S × A → R , γ : S × A → R + such that ∀ s, a, Π ,

Remark 3.1.1 Instead of preferences over stochastic processes of potential futures, one could begin with preferences over trajectories [86, 64, 10]. The author takes issue with this approach, however, as it's unclear that such preferences should satisfy Asymmetry or Independence without additional assumptions (humans often consider counterfactual outcomes when evaluating the desirability of a trajectory) [54]. By using Theorem 2 to unroll prospects, one can extend preferences over prospects to define preferences over trajectories according to their discounted reward.

Remark 3.1.2 Theorem 2, as it appeared in Pitis [54], required an additional, explicit 'Irrelevance of unrealizable actions' axiom, since prospects were defined as tuples (˜ s, Π) . This property is implicit in our redefinition of prospects as stochastic processes.

Remark 3.1.3 In this line of reasoning only the preference relation ≻ is primitive; V and its Bellman form ( R,γ ) are simply representations of ≻ whose existence is guaranteed by the axioms. Not all numerical representations of ≻ have these forms [84]. In particular, (strictly) monotonically increasing transforms preserve ordering, so that any increasing transform V † of a Theorem 1 representation V is itself a valid numerical representation of ≻ (although lotteries will no longer be valued by the expectation over their atoms unless the transform is affine, per Theorem 1).

3.2 Representing rational aggregation

Let us now consider the aggregation of several preferences. These may be the preferences of an agent's several principals or preferences representing a single individual's competing interests. Note at the outset that it is quite natural for different objectives or principals to have differing time preference. We saw one example in the Procrastinator's Peril, but we can also consider a household robot that seeks to aggregate the preferences of Alice and her husband Bob, for whom there is no reason to assume equal time preference [26].

An intuitively appealing axiom for aggregation is Pareto indifference, which says that if each individual preference is indifferent between two alternatives, then so too is the aggregate preference.

Axiom 3 (Pareto indifference) . If ˜ p ≈ i ˜ q ( ∀ i ∈ I ) , ˜ p ≈ Σ ˜ q .

Here, ˜ p ≈ ˜ q means indifference (not ˜ p ≻ ˜ q and not ˜ q ≻ ˜ p ), so that ≈ i is the i th individual indifference relation, I is a finite index set over the individuals, and ≈ Σ indicates the aggregate relation. There exist stronger variants of Pareto property that require monotonic aggregation (e.g., if all individuals prefer ˜ p , so too does the aggregate; see Axiom 3 ′ in Subsection 5.1). We opt for Pareto indifference to accommodate potentially deviant individual preferences (e.g., if all individuals are indifferent but for a sociopath, the aggregate preference may be opposite of the sociopath's).

If we require individual and aggregate preferences to satisfy Axiom 1 and, jointly, Axiom 3, we obtain a third key result due to Harsanyi [32]. (See Hammond [31] for proof).

Theorem 3 (Harsanyi's representation) . Consider individual preference relations {≻ i ; i ∈ I} and aggregated preference relation ≻ Σ , each defined on the set L ( P ) , that individually satisfy Axiom 1 and jointly satisfy Axiom 3. If { V i ; i ∈ I} and V Σ are expected utility representations of {≻ i ; i ∈ I} and ≻ Σ , respectively, then there exist real-valued constant c and weights { w i ; i ∈ I} such that:

That is, the aggregate value can be expressed as a weighted sum of individual values (plus a constant).

According to Harsanyi's representation theorem, the aggregated value function is a function of the individual value functions, and nothing else . In other words, Pareto indifferent aggregation of VNM preferences that results in VNM preference is necessarily context-free -the same weights { w i } apply regardless of state and policy.

We will also make use of two technical conditions to eliminate certain edge cases. Though sometimes left implicit, these are both common requirements for aggregation functions [5].

Axiom 4 (Technical conditions on ≻ Σ ) .

̸

Unrestricted Domain : ≻ Σ is defined for all valid individual preference sets {≻ i ; i ∈ I} . Sensitivity : ∀ i ∈ I , holding ≻ j , j = i constant, there exist ≻ 1 i , ≻ 2 i resulting in different ≻ Σ .

The first condition allows us to consider conflicting objectives with different time preference. The second condition implies that the weights w i in Theorem 3 are nonzero.

Remark 3.2.1 It is worth clarifying here the relation between Harsanyi's theorem and a related class of aggregation theorems, occasionally cited within the machine learning literature (e.g., [6]), based on axioms originating with Debreu [19]. One instance of this class states that any aggregate preference satisfying six reasonable axioms can be represented in the form: V Σ (˜ p ) = ∑ i ∈I m ( V i (˜ p )) , where m is a strictly increasing monotonic function from the family { x p | 0 < p ≤ 1 } ∪ { log( x ) | p = 0 } ∪ {-x p | p < 0 } [47]. If we (1) drop the symmetry axiom from this theorem to obtain variable weights w i , and (2) add a monotonicity axiom to Harsanyi's theorem to ensure positive weights w i [32], then the only difference between the theorems is the presence of monotonic function m . But

̸

note that applying m to each V i or to V Σ individually does not change the preference ranking they represent; it does, however, decide whether V i and V Σ are VNM representations of ≻ . Thus, we can understand Harsanyi's theorem as saying: if V i , V Σ are VNM representations of ≻ , then m must be linear. Or conversely, if m is non-linear ( p = 1 ), V i and V Σ are not VNM representations.

Remark 3.2.2 A caveat of Harsanyi's theorem is that it implicitly assumes that all individual preferences, and the aggregate preference, use the same set of agreed upon, 'objective' probabilities. This is normatively justifiable if we use the same probability distribution (e.g., that of the aggregating agent) to impose 'ideal' preferences ≻ i on each individual, which may differ from their implicit subjective or revealed preferences [62]. As noted by Desai et al. [20], Harsanyi's theorem fails if the preferences being aggregated use subjective probabilities. Note, however, that the outcome of the 'bargaining' construction in Desai et al. [20] is socially suboptimal when the aggregrating agent has better information than the principals, which suggests that effort should be made to unify subjective probabilities. We leave exploration of this to future work.

3.3 Impossibility result

None of Theorems 1-3 assume all Axioms 1-4. Doing so leads to our key result, as follows.

Theorem 4 (Impossibility) . Assume there exist distinct policies, Π , Ω , Λ , none of which is a mixture (i.e., convex combination) of the other two, and consider the aggregation of arbitrary individual preference relations {≻ i ; i ∈ I} defined on L ( P ) that individually satisfy Axioms 1-2. There does not exist aggregated preference relation ≻ Σ satisfying Axioms 1-4.

Sketch of Proof. The full proof is in Appendix B. Briefly, we consider |I| = 2 and use Axiom 4 (Unrestricted Domain) to construct mixtures of Π and Ω so that each mixture is considered indifferent to Λ by one of the individual preference relations. Then, by applying Theorem 2 and Theorem 3 in alternating orders to the difference between the value of the mixture policy and the value of Λ , and doing some algebra, we arrive at the equations

from which we conclude that γ Σ ( s, a ) = γ 1 ( s, a ) = γ 2 ( s, a ) . But this contradicts our assumption that individual preferences {≻ i ; i ∈ I} may be chosen arbitrarily, completing the proof.

Intuition of Proof. Under mild conditions, we can find two policies (a Π / Ω mixture, and Λ ) between which individual preference ≻ 1 is indifferent, but individual preference ≻ 2 is not. Then for mixtures of this Π / Ω mixture and Λ , ≻ 1 remains indifferent, but the strength of ≻ 2 changes, so that ≻ Σ must have the same time preference as ≻ 2 . By an analogous argument, ≻ Σ must have the same time preference as ≻ 1 , leading to a violation of Unrestricted Domain.

The closest results from the economics literature consider consumption streams ( S = R ) [88, 15, 83]. Within reinforcement learning, equal time preference has been assumed, without justification, when merging MDPs [68, 41] and value functions for different tasks [29].

Remark 3.3.1 A consequence of Unrestricted Domain, critical to the impossibility result, is that individual preferences may have diverse time preferences (i.e., different discount functions). If discounts are equal, Markovian aggregation is possible.

Remark 3.3.2 Per Remark 3.2.2, by applying Harsanyi's theorem, we are implicitly assuming that all preferences are formed using the same 'objective' probabilities over prospects; is there a notion of 'objective' time preference that should be used? If so, this would resolve the impossibility (once again, requiring that we impose a notion of ideal preference on individuals that differs from their expressed preference). We leave this consideration for future work.

Remark 3.3.3 Theorem 4 applies to the standard RL setup, where γ 1 , γ 2 , γ Σ are constants.

4 Escaping Impossibility with Non-Markovian Aggregation

An immediate consequence of Theorem 4 is that any scalarized approach to multi-objective RL [79] is generally insufficient to represent composed preferences. But the implications run deeper: insofar as general tasks consist of several objectives, Theorem 4 pushes Sutton's reward hypothesis to its

limits. To escape impossibility, the Procrastinator's Peril is suggestive: to be productive, repeat play should not be rewarded. And for this to happen, we must keep track of past play, which suggests that reward must be non-Markovian, even when all relevant objectives are Markovian . That is, even if we have settled on some non-exhaustive state representation that is 'sufficiently' Markovian, an extra aggregation step could render it no longer sufficient.

Relaxing Markov Preference The way in which non-Markovian rewards (or equivalently, nonMarkovian utilities) can be used to escape Theorem 4 is quite subtle. Nowhere in the proofs of Theorems 1-4 is the Markov assumption explicitly used. Nor does it obviously appear in any of the Axioms. The Markov assumption is , however, invoked in two places. First, to establish historyindependent comparability between the basic objects of preference-prospects ( s, Π) -and second, to extend that comparison set to include 'prospects' of the form ( T ( s, a ) , Π) . To achieve initial comparability, Pitis [54] applied a 'Markov preference' assumption (preferences over prospects are independent of history) together with an 'original position' construction that is worth repeating here:

[I]t is admittedly difficult to express empirical preference over prospects . . . an agent only ever chooses between prospects originating in the same state . . . [Nevertheless,] we imagine a hypothetical state from which an agent chooses between [lotteries] of prospects, denoted by L ( P ) . We might think of this choice being made from behind a 'veil of ignorance' [58].

(2)

In other words, to allow for comparisons between prospect ( s 1 , Π) and ( s 2 , Π) , we prepend some pseudo-state, s 0 , and compare prospects ( s 0 s 1 , Π) and ( s 0 s 2 , Π) . Markov preference then lets us cut off the history, so that our preferences between ( s 1 , Π) and ( s 2 , Π) are cardinal.

The impossibility result suggests, however, that aggregate preference is not independent of history, so that construction 2 cannot be applied. Without this construction, there is no reason to require relative differences between V ( s 1 , ∗ ) and V ( s 2 , ∗ ) to be meaningful, or to even think about lotteries/mixtures of the two prospects (as done in Axiom 1). Letting go of this ability to compare prospects starting in different states means that Theorem 1 is applicable only to sets of prospects with matching initial states, unless we shift our definition of 'prospect' to include the history; i.e., letting h represent the history, we now compare 'historical prospects' with form ( h, Π) .

Though this does not directly change the conclusion of Theorem 2, T ( s, a ) in V ( T ( s, a ) , Π) includes a piece of history, ( s, a ) , and can no longer be computed as E s ′ ∼ T ( s,a ) V ( s ′ , Π) . Instead, since the agent is not choosing between prospects of form ( s ′ , Π) but rather (abusing notation) prospects of form ( sas ′ , Π) , the expectation should be computed as E s ′ ∼ T ( s,a ) V ( sas ′ , Π) .

The inability to compare prospects starting in different states also changes the conclusion of Theorem 3, which implicitly uses such comparisons to find constant coefficients w i that apply everywhere in the original L ( P ) . Relaxing the application of Harsanyi's theorem to not make inter-state comparisons results in weights w i ( h ) that are history dependent when aggregating the historical prospects.

4.1 Possibility Result

Allowing the use of history dependent coefficients in the aggregation of V i ( T ( s, a ) , Π) resolves the impossibility. The following result shows that given some initial state dependent coefficients w i ( s ) , we can always construct history dependent coefficients w i ( h ) that allow for dynamically consistent aggregation satisfying all axioms. In the statement of the theorem, L ( P h ) is used to denote the set of lotteries over prospects starting with history h of arbitrary but finite length (not to be confused with the set of lotteries over all historical prospects, of which there is only one). Note that if a preference relation satisfies Axioms 1-2 with respect to L ( P ) , the natural extension to L ( P hs ) , ( hs, Π 1 ) ≻ ( hs, Π 2 ) ⇐⇒ ( s, Π 1 ) ≻ ( s, Π 2 ) , satisfies Axioms 1-2 with respect to L ( P hs ) . Here, we are using hs to denote a history terminating in state s .

Theorem 5 (Possibility) . Consider the aggregation of arbitrary individual preference relations {≻ i ; i ∈ I} defined on L ( P ) , and consequently L ( P h ) , ∀ h , that individually satisfy Axioms 1-2. There exists aggregated preference relations {≻ h Σ } , defined on L ( P h ) , ∀ h , that satisfy Axioms 1-4.

In particular, given s, a , V Σ , { V i } , { w i ( s ) } , where (A) each V i satisfies Axioms 1-2 on L ( P ) , and V Σ satisfies Axioms 1-2 on L ( P h ) , ∀ h , and (B) V Σ ( s, a Π) = ∑ i w i ( s ) V i ( s, a Π)) , then , choosing

implies that V Σ ( sas ′ , Π) ∝ ∑ i w i ( sa ) V i ( s ′ , Π) so that the aggregated preferences {≻ sas ′ Σ } satisfy Axiom 3 on L ( P sas ′ ) . Unrolling this resultw i ( hsas ′ ) := w i ( hs ) γ i ( s, a ) -produces a set of constructive, history dependent weights { w i ( h ) } such that Axiom 3 is satisfied for all histories { h } .

Sketch of Proof. The full proof is in Appendix B. Following the proof of Theorem 4, we arrive at

from which we conclude that:

This shows the existence of weights w i ( sa ) , unique up to a constant scaling factor, for which V Σ ( T ( s, a ) , Π) ∝ ∑ i w i ( sa ) V i ( T ( s, a ) , Π) , that apply regardless of how individual preferences are chosen or aggregated at s . Unrolling the result completes the proof.

From Theorem 5 we obtain a rather elegant result: rational aggregation over time discounts the aggregation weights assigned to each individual value function proportionally to its respective discount factor. In the Procrastinator's Peril, for instance, where we started with w p ( s ) = w w ( s ) = 1 , at the initial (and only) state s , we might define w p ( s p s ) = 0 . 5 and w w ( s p s ) = 0 . 9 . With these nonMarkovian aggregation weights and γ Σ ( s p ) = 1 , you can verify that (1) the irrational procrastination behavior is solved, and (2) the aggregated rewards for work and play are now non-Markovian.

Remark 4.1 (Important!) The discount γ Σ is left undetermined by Theorem 5. One might determine it several ways: by appealing to construction 2 with respect to historical prospects in order to establish inter-state comparability, by setting it to be the highest individual discount (0.9 in the Procrastinator's Peril), by normalizing the aggregation to weights to sum to 1 at each step, or perhaps by another method. In any case, determining γ Σ would also determine the aggregation weights, per equation 4 (and vice versa). We leave the consideration of different methods for setting γ Σ and establishing inter-state comparability of V Σ to future work. (NB: This is a normative question , which we leave unanswered. While one can make assumptions, as we will for our numerical example in Subsection 5.2, future research should be wary of accepting a solution just because it seems to work.)

4.2 A Practical State Space Expansion

The basic approach to dealing with non-Markovian rewards is to expand the state space in such a way that rewards becomes Markovian [27, 14, 2]. However, naively expanding S to a history of length H could have O (( |S| + |A| ) H ) complexity. Fortunately, the weight update in equation 4 allows us to expand the state using a single parameter per objective. In particular, for history hsa and objective i , we append to the state the factors y i ( hsa ) := y i ( h ) γ i ( s, a ) /γ Σ ( s, a ) , which are defined for every history, and can be accumulated online while executing a trajectory. Then, given a composition with any set of initial weights { w i } , we can compute the weights of augmented state s aug = ( s, y i ( hs )) as w i ( s aug ) = y i ( hs ) · w i . Letting L ( P ( y ) ) be the set of prospects on the augmented state set, we get the following corollary to Theorem 5:

Corollary 1. Consider the aggregation of arbitrary individual preference relations {≻ i ; i ∈ I} defined on L ( P ) , and consequently L ( P ( y ) ) , that individually satisfy Axioms 1-2. There exists aggregated preference relation {≻ Σ } , defined on L ( P ( y ) ) , that satisfies Axioms 1-4.

5 Discussion and Related Work

5.1 A Fundamental Tension in Intertemporal Choice

The state space expansion of Subsection 4.2 allows us to represent the (now Markovian) values of originally non-Markovian policies in a dynamically consistent way. While this allows us to design agents that implement these policies, it doesn't quite solve the intertemporal choice problem.

̸

In particular, it is known that dynamic consistency (in the form of Koopmans' Stationarity [38]), together with certain mild axioms, implies a first period dictatorship: the preferences at time t = 1 are decisive for all time [25] (in a sense, this is the very definition of dynamic consistency!). Generally speaking, however, preferences at time t = 1 are not the same as preferences at time t = 1 (this is

what got us into our Procrastinator's Peril to begin with!) and we would like to care about the value at all time steps, not just the first.

A typical approach is to treat each time step as a different generation (decision maker), and then consider different methods of aggregating preferences between the generations [33]. Note that (1) this aggregation assumes intergenerational comparability of utilities (see Remark 4.1), and (2) each generation is expressing their personal preferences about what happens in all generations, not just their own. Since this is a single aggregation, it will be dynamically consistent (we can consider the first period dictator as a benevolent third party who represents the aggregate preference instead of their own). A sensible approach might be to assert a stronger version of Axiom 3 that uses preference:

Axiom 3 ′ (Strong Pareto Preference) If ˜ p ⪰ i ˜ q ( ∀ i ∈ I ) , then ˜ p ⪰ Σ ˜ q ; and if, furthermore, ∃ j ∈ I such that ˜ p ≻ j ˜ q , then ˜ p ≻ Σ ˜ q .

Using Axiom 3 ′ in place of Axiom 3 for purposes of Theorem 3 gives a representation that assigns strictly positive weights to each generation's utility. Given infinite periods, it follows (e.g., by the Borel-Cantelli Lemma for general measure spaces) that if utilities are bounded, some finite prefix of the infinite stream decides the future and we have a 'finite prefix dictatorship', which is not much better than a first period one.

The above discussion presents a strong case against using dynamic consistency to determine a long horizon policy. Intuitively, this makes sense: preferences change over time, and our current generation should not be stuck implementing the preferences of our ancestors. One way to do this is by taking time out of the equation, and optimizing the expected individual utility of the stationary state-action distribution, d π (cf. Sutton and Barto [74] (Section 10.4) and Naik et al. [50]):

Unfortunately, as should be clear from the use of a stationary policy π , this time-neutral approach falls short for our purposes, which suggests the use of a non-Markovian policy. While optimizing equation 6 would find the optimal stationary policy in the Procrastinator's Peril ( work forever with V ( τ 2 ) = 3 . 0 ), it seems clear that we should play at least once ( V ( τ 3 ) = 3 . 2 ) as this hurts no one and makes the current decision maker better off-i.e., simply optimizing (6) violates the Pareto principle.

This discussion exemplifies a known tension in intertemporal choice between Pareto optimality and the requirement to treat every generation equally-it is impossible to have both [16, 42]. In a similar spirit to Chichilnisky [16], who has proposed an axiomatic approach requiring both finite prefixes of generations and infinite averages of generations to have a say in the social preference, we will examine a compromise (to the author's knowledge novel) between present and future decision makers in next Subsection.

5.2 N-Step Commitment and Historical Discounting

We now consider two solutions to the intertemporal choice problem that deviate just enough from dynamic consistency to overcome finite period dictatorships, while capturing almost all value for each decision maker. In other words, they are 'almost' dynamically consistent. We leverage the following observation: due to discounting, the first step decision maker cares very little about the far off future. If we play once, then work for 30 steps in the Procrastinator's Peril, we are already better off than the best stationary policy, regardless of what happens afterward.

This suggests that, rather than following the current decision maker forever, we allow them commit to a non-Markovian policy for some N < ∞ steps that brings them within some ϵ of their optimal policy, without letting them exert complete control over the far future. To implement this, we could reset the accumulated factors y i to 1 every N steps. This approach is the same as grouping every consecutive N step window into a single, dynamically consistent generation with a first period dictator.

A more fluid and arguably better approach is to discount the past when making current decisions ('historical discounting'). We can implement historical discounting with factor η ∈ [0 , 1] by changing the update rule for factors y i to

where w n i ( s ) denotes the initial weight for objective i at the n th generation (if preferences do not change over time, w n i = w i ). This performs an exponential moving average of preferences over time-

high η < 1 initially gives the first planner full control, but eventually discounts their preferences until they are negligible. This obtains a qualitatively different effect from N -step commitment, since the preferences of all past time steps have positive weight (by contrast, on the N th step, N -step commitment gives no weight to the N -1 past preferences). As a result, in the Procrastinator's Peril, any sufficiently high η returns policy τ 3 , whereas N -step commitment would play every N steps.

Historical discounting is an attractive compromise because it is both Pareto efficient, and also anonymous with respect to tail preferences-it both optimizes equation 6 in the limit, and plays on the first step. Another attractive quality is that historical discounting allows changes in preference to slowly deviate from dynamic consistency over time-the first period is not a dictator.

As a numerical example, we can consider what happens in the Procrastinator's Peril if we start to realize that we value occasional play, and preferences shift away from the play MDP to a playn MDP. The playn MDP has fixed γ playn = 0 . 9 and non-Markovian reward R ( p | no play in last N-1 steps ) = 0 . 5 , R ( anything else ) = 0 . We assume preferences shift linearly over the first 10 timesteps, from w play = 1 , w playn = 0 at t = 0 to w play = 0 , w playn = 1 at t = 10 (and w work = 1 throughout). Then, the optimal trajectories τ ∗ for different η , and resulting discounted reward for the first time step, V 1 ( τ ∗ ) , are as follows:

When η = 0 , each time step acts independently, and since the play MDP has high weight at the start, we experience a brief period of consistent play in line with the original Procrastinator's Peril, before preferences fully shift to occasional play. With high η < 1 , we capture almost all value for the first time step, while also eventually transitioning to the equilibrium ' play every 10 steps' policy.

5.3 Extension to Boltzmann Policies

The preference relation ≻ is deterministic, but the associated policy does not have to be. In many cases-partially-observed, multi-agent, or even fully-observed settings [74, 30]-stochastic policies outperform deterministic ones. And for generative sequence models such as LLMs [13], stochastic policies are inherent. We can extend our analysis-impossibility, possibility, state space expansion, and intertemporal choice rules-to such cases by adopting a stochastic choice rule.

To formalize this, we will take the stochastic choice rule as primitive, and use it to define a (deterministic) relation ≻ that, for practical purposes, satisfies Axioms 1-4 [18]. We assume our choice rule satisfies Luce's Choice Axiom [43], which says that the relative rate of choosing between two alternatives in a choice set of n alternatives is constant regardless of the choice set. This can be implemented numerically with a Bradley-Terry choice model [11] by associating each alternative a i with a scalar Ω( a i ) , so that given choice set { a i , a j } , p ( a i ) = Ω( a i ) / (Ω( a i ) + Ω( a j )) . (An interesting interpretation for Ω( a i ) that connects to both probability matching [82, 65] and statistical mechanics [45] is as the number of 'outcomes' (microstates) for which a i is the best choice.)

We then simply 'define' preference as a i ≻ a j ⇐⇒ Ω( a i ) > Ω( a j ) , and utility as V ( a i ) := k log Ω( a i ) , so that the policy is a softmax of the utilities. The way these utilities are used in practice (e.g., [30, 17]) respects Axioms 1-2. And summing utilities is a common approach to composition [21, 29], which is consistent with Harsanyi's representation (Theorem 3). For practical purposes then, the impossibility result applies whenever composed objectives may have different time preference.

Remark 5.3 Unlike our main results, this extension to Boltzmann policies is motivated by practical, rather than normative, considerations. Simple counterexamples to Luce's Choice Axiom exist [87] and probability matching behavior is evidently irrational in certain circumstances [65]. We note, however, that certain theoretical works tease at the existence of a normative justification for Boltzmann policies [18, 8, 73, 23]; given the practice, a clear justification would be of great value.

5.4 Related Work in RL

Task Definition Tasks are usually defined as the maximization of expected cumulative reward in an MDP [74, 55, 67]. Preference-based RL [86] avoids rewards, operating directly with preferences (but note that preference aggregation invokes Arrow's impossibility theorem [5, 48]), while other works translate preferences into rewards [17, 72, 12, 35]. This paper joins a growing list of work [54, 1, 76, 77] that challenges the implicit assumption that 'MDPs are enough' in many reward learning papers, particularly those that disaggregate trajectory returns into stepwise rewards [22, 56, 59].

Discounting Time preference or discounting can be understood as part of the RL task definition. Traditionally, a constant γ has been used, although several works have considered other approaches, such as state-action dependent discounting [85, 66] and non-Markovian discounting [24, 63]. Several works have considered discounting as a tool for optimization [80] or regularization [36, 4, 57].

Task Compositionality Multi-objective RL [61] represents or optimizes over multiple objectives or general value functions [75], which are often aggregated with a linear scalarization function [7, 3]. Rather than scalarizing to obtain a single solution to a multi-objective problem, one can also seek out sets of solutions, such as the set of Pareto optimal policies [78] or the set of acceptable policies [46]. Several works have also considered formal task decompositions [14, 51] where simple addition of MDPs is insufficient [68]. More broadly, in machine learning, composition can be done via mixtures of experts and/or energy-based modeling [34, 21], which have also been applied to RL [29, 41]. Our results provide normative justification for linear scalarization when time preference is the same for all objectives, but call for non-Markovian adjustments when time preferences differ.

Non-Markovian Rewards The necessity of non-Markovian rewards was demonstrated in other settings by Abel et al. [1] and more recently, in a concurrent work by Skalse and Abate [69]. Though several papers explicitly consider RL with non-Markovian rewards [27, 60], this is usually motivated by task compression rather than necessity, and the majority of the RL literature restricts itself to Markovian models. Many popular exploration strategies implicitly use non-Markovian rewards [37, 53]. Our work is unique in that non-Markovian rewards arise from aggregating strictly Markovian quantities, rather non-Markovian quantities present in the task definition or algorithm.

6 Conclusion and Future Work

The main contribution of this work is an impossibility result from which one concludes that nonMarkovian rewards (or an equivalent state expansion) are likely necessary for agents that pursue multiple objectives or serve multiple principals. It's possible that this will be the case for any advanced agent whose actions impact multiple human stakeholders. To accurately align such agents with diverse human preferences we need to endow them with the capacity to solve problems requiring non-Markovian reward, for which this paper has proposed an efficient state space expansion that uses one new parameter per aggregated objective. While the proposed state space expansion allows multi-objective agents to have dynamically consistent preferences for future prospects, it does not, in itself, solve the intertemporal choice problem. To that end, we have proposed 'historical discounting', a novel compromise between dynamic consistency and fair consideration of future generations.

Interesting avenues for future work include quantifying the inefficiency of Markovian representations, investigating normative approaches to aggregating preferences based on subjective world models (Remark 3.2.2), considering the existence of an 'objective' time preference (Remark 3.3.2), improving methods for determining subjective time preference (e.g., [63]), implementing and comparing approaches to determining γ Σ (Remark 4.1), investigating historical discounting in more detail (Subsection 5.2), and considering the existence of a normative justification for Boltzmann policies and their composition (Subsection 5.3).

Acknowledgments and Disclosure of Funding

I thank Elliot Creager, for a fruitful discussion that prompted Subsection 5.1 and motivated me to turn this into a full paper; Duncan Bailey, who assisted with an earlier workshop version; the anonymous reviewers, who provided detailed reviews and suggestions that helped improve the final manuscript; and Jimmy Ba and the Ba group for early discussions. This work was supported by an NSERC CGS D Award and a Vector Research Grant.

References

David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of markov reward. Advances in Neural Information Processing Systems , 34: 7799-7812, 2021.

David Abel, André Barreto, Michael Bowling, Will Dabney, Steven Hansen, Anna Harutyunyan, Mark Ho, Ramana Kumar, Michael Littman, Doina Precup, and Satinder Singh. Expressing non-markov reward to a markov agent. In RLDM , pages 1-5, 2022.

Axel Abels, Diederik M Roijers, Tom Lenaerts, Ann Nowé, and Denis Steckelmacher. Dynamic weights in multi-objective deep reinforcement learning. arXiv preprint arXiv:1809.07803 , 2018.

Ron Amit, Ron Meir, and Kamil Ciosek. Discount factor as a regularizer in reinforcement learning. In International conference on machine learning , pages 269-278. PMLR, 2020.

Kenneth J Arrow, Amartya Sen, and Kotaro Suzumura. Handbook of social choice and welfare , volume 2. Elsevier, 2010.

Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et al. Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems , 35:38176-38189, 2022.

André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems , 30, 2017.

Sergio Batiz-Solorzano. On decisions with multiple objectives: Review and classification of prescriptive methodologies, a group value function problem, and applications of a measure of information to a class of multiattribute problems. Rice University, 1979.

D. Bernoulli. Specimen theoriae novae de mensura sortis, Commentarii Academiae Scientiarum Imperialis Petropolitanae (5, 175-192, 1738). Econometrica , 22:23-36, 1954. Translated by L. Sommer.

Michael Bowling, John D Martin, David Abel, and Will Dabney. Settling the reward hypothesis. arXiv preprint arXiv:2212.10420 , 2022.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Daniel S Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. arXiv preprint arXiv:1904.06387 , 2019.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Alberto Camacho, Rodrigo Toro Icarte, Toryn Q Klassen, Richard Anthony Valenzano, and Sheila A McIlraith. Ltl and beyond: Formal languages for reward function specification in reinforcement learning. In IJCAI , volume 19, pages 6065-6073, 2019.

Christopher P Chambers and Federico Echenique. On multiple discount rates. Econometrica , 86(4): 1325-1346, 2018.

Graciela Chichilnisky. An axiomatic approach to sustainable development. Social choice and welfare , 13: 231-257, 1996.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems , 2017.

Gerard Debreu. Stochastic choice and cardinal utility. Econometrica: Journal of the Econometric Society , pages 440-444, 1958.

Gerard Debreu. Topological methods in cardinal utility theory, mathematical methods in the social sciences (kj arrow, s. karlin, and p. suppes, eds.), 1960.

Nishant Desai, Andrew Critch, and Stuart J Russell. Negotiable reinforcement learning for pareto optimal sequential decision-making. Advances in Neural Information Processing Systems , 31, 2018.

Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. Advances in Neural Information Processing Systems , 33:6637-6647, 2020.

Yonathan Efroni, Nadav Merlis, and Shie Mannor. Reinforcement learning with trajectory feedback. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 7288-7295, 2021.

Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. In International Conference on Learning Representations , 2022.

William Fedus, Carles Gelada, Yoshua Bengio, Marc G Bellemare, and Hugo Larochelle. Hyperbolic discounting and learning over multiple horizons. arXiv preprint arXiv:1902.06865 , 2019.

John Ferejohn and Talbot Page. On the foundations of intertemporal choice. American Journal of Agricultural Economics , 60(2):269-275, 1978.

Shane Frederick, George Loewenstein, and Ted O'Donoghue. Time discounting and time preference: A critical review. Journal of economic literature , 40(2):351-401, 2002.

Maor Gaon and Ronen Brafman. Reinforcement learning with non-markovian rewards. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 3980-3987, 2020.

Paul W Glimcher. Understanding dopamine and reinforcement learning: the dopamine reward prediction error hypothesis. Proceedings of the National Academy of Sciences , 108(supplement_3):15647-15654, 2011.

Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforcement learning for robotic manipulation. In 2018 IEEE international conference on robotics and automation (ICRA) , pages 6244-6251. IEEE, 2018.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Peter J Hammond. Harsanyi's utilitarian theorem: A simpler proof and some ethical connotations. In Rational Interaction , pages 305-319. Springer, 1992.

John C Harsanyi. Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility. Journal of political economy , 63(4):309-321, 1955.

Geoffrey Heal. Intertemporal welfare economics and the environment. Handbook of environmental economics , 3:1105-1145, 2005.

Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts. Neural computation , 3(1):79-87, 1991.

Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems , 33:4415-4426, 2020.

Nan Jiang, Alex Kulesza, Satinder Singh, and Richard Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems , pages 1181-1189, 2015.

J Zico Kolter and Andrew Y Ng. Near-bayesian exploration in polynomial time. In Proceedings of the 26th annual international conference on machine learning , pages 513-520, 2009.

Tjalling C Koopmans. Stationary ordinal utility and impatience. Econometrica: Journal of the Econometric Society , pages 287-309, 1960.

David Kreps. Notes on the Theory of Choice . Westview press, 1988.

David M Kreps and Evan L Porteus. Temporal resolution of uncertainty and dynamic choice theory. Econometrica: journal of the Econometric Society , pages 185-200, 1978.

Romain Laroche, Mehdi Fatemi, Joshua Romoff, and Harm van Seijen. Multi-advisor reinforcement learning. arXiv preprint arXiv:1704.00756 , 2017.

Luc Lauwers. Intertemporal objective functions: Strong pareto versus anonymity. Mathematical Social Sciences , 35(1):37-55, 1998.

R Duncan Luce. Individual choice behavior. 1959.

Mark J Machina. Dynamic consistency and non-expected utility models of choice under uncertainty. Journal of Economic Literature , 27(4):1622-1668, 1989.

Franz Mandl. Statistical physics , volume 14. John Wiley & Sons, 1991.

Shuwa Miura. On the expressivity of multidimensional markov reward. arXiv preprint arXiv:2307.12184 , 2023.

Hervé Moulin. Fair division and collective welfare . MIT press, 2004.

Krikamol Muandet. Impossibility of collective intelligence. arXiv preprint arXiv:2206.02786 , 2022.

Sendhil Mullainathan and Richard H Thaler. Behavioral economics, 2000.

Abhishek Naik, Roshan Shariff, Niko Yasui, Hengshuai Yao, and Richard S Sutton. Discounted reinforcement learning is not an optimization problem. arXiv preprint arXiv:1910.02140 , 2019.

Geraud Nangue Tasse, Steven James, and Benjamin Rosman. A boolean task algebra for reinforcement learning. Advances in Neural Information Processing Systems , 33:9497-9507, 2020.

Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In The Seventeenth International Conference on Machine Learning , pages 663-670, 2000.

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning . PMLR, 2017.

Silviu Pitis. Rethinking the discount factor in reinforcement learning: A decision theoretic approach. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 7949-7956, 2019.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

David Raposo, Sam Ritter, Adam Santoro, Greg Wayne, Theophane Weber, Matt Botvinick, Hado van Hasselt, and Francis Song. Synthetic returns for long-term credit assignment. arXiv preprint arXiv:2102.12425 , 2021.

Sarah Rathnam, Sonali Parbhoo, Weiwei Pan, Susan Murphy, and Finale Doshi-Velez. The unintended consequences of discount regularization: Improving regularization in certainty equivalence reinforcement learning. In International Conference on Machine Learning , pages 28746-28767. PMLR, 2023.

John Rawls. A theory of justice: Revised edition . Harvard university press, 2009.

Zhizhou Ren, Ruihan Guo, Yuan Zhou, and Jian Peng. Learning long-term reward redistribution via randomized return decomposition. In International Conference on Learning Representations , 2023.

Gavin Rens, Jean-François Raskin, Raphaël Reynoaud, and Giuseppe Marra. Online learning of nonmarkovian reward models. In Proceedings of the Thirteenth International Conference on Agents and Artificial Intelligence . Scitepress, 2020.

Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research , 48:67-113, 2013.

Leonard J Savage. The foundations of statistics . Wiley, 1954.

Matthias Schultheis, Constantin A Rothkopf, and Heinz Koeppl. Reinforcement learning with nonexponential discounting. In Advances in neural information processing systems , 2022.

Mehran Shakerinava and Siamak Ravanbakhsh. Utility theory for sequential decision making. In International Conference on Machine Learning , pages 19616-19625. PMLR, 2022.

David R Shanks, Richard J Tunney, and John D McCarthy. A re-examination of probability matching and rational choice. Journal of Behavioral Decision Making , 15(3):233-250, 2002.

David Silver, Hado van Hasselt, Matteo Hessel, Tom Schaul, Arthur Guez, Tim Harley, Gabriel DulacArnold, David Reichert, Neil Rabinowitz, André Barreto, and Thomas Degris. The predictron: End-to-end learning and planning. In The Thirty-fourth International Conference on Machine Learning , 2017.

David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence , 299:103535, 2021.

Satinder P Singh and David Cohn. How to dynamically merge markov decision processes. In Advances in neural information processing systems , pages 1057-1063, 1998.

Joar Skalse and Alessandro Abate. On the limitations of markovian rewards to express multi-objective, risk-sensitive, and modal tasks. In Uncertainty in Artificial Intelligence , pages 1974-1984. PMLR, 2023.

Matthew J Sobel. Ordinal dynamic programming. Management science , 21(9):967-975, 1975.

Robert M Solow et al. Growth theory. an exposition. In Growth theory. An exposition. Oxford: Clarendon Press., 1970.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Patrick Suppes. Behavioristic foundations of utility. Econometrica: Journal of The Econometric Society , pages 186-202, 1961.

Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction . MIT Press, 2018.

Richard S Sutton, Joseph Modayil, Michael Delp, Thomas Degris, Patrick M Pilarski, Adam White, and Doina Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2 , pages 761-768. International Foundation for Autonomous Agents and Multiagent Systems, 2011.

Csaba Szepesvári. Constrained mdps and the reward hypothesis. Musings about machine learning and other things (blog) , 2020.

Peter Vamplew, Benjamin J Smith, Johan Källström, Gabriel Ramos, Roxana R˘ adulescu, Diederik M Roijers, Conor F Hayes, Fredrik Heintz, Patrick Mannion, Pieter JK Libin, et al. Scalar reward is not enough: A response to silver, singh, precup and sutton (2021). Autonomous Agents and Multi-Agent Systems , 36(2):41, 2022.

Kristof Van Moffaert and Ann Nowé. Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research , 15(1):3483-3512, 2014.

Kristof Van Moffaert, Madalina M Drugan, and Ann Nowé. Scalarized multi-objective reinforcement learning: Novel design techniques. In 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL) , pages 191-199. IEEE, 2013.

Harm Van Seijen, Mehdi Fatemi, and Arash Tavakoli. Using a logarithmic mapping to enable lower discount factors in reinforcement learning. Advances in Neural Information Processing Systems , 32, 2019.

John Von Neumann and Oskar Morgenstern. Theory of games and economic behavior . Princeton university press, 1953.

Nir Vulkan. An economist's perspective on probability matching. Journal of economic surveys , 14(1): 101-118, 2000.

Martin L Weitzman. Gamma discounting. American Economic Review , 91(1):260-271, 2001.

John A Weymark. Measurement theory and the foundations of utilitarianism. Social Choice and Welfare , 25(2-3):527-555, 2005.

Martha White. Unifying task specification in reinforcement learning. In The Thirty-fourth International Conference on Machine Learning , 2017.

Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research , 18(136):1-46, 2017.

John I Yellott Jr. The relationship between luce's choice axiom, thurstone's theory of comparative judgment, and the double exponential distribution. Journal of Mathematical Psychology , 15(2):109-144, 1977.

Stephane Zuber. The aggregation of preferences: can we ignore the past? Theory and decision , 70(3): 367-384, 2011.

A Notation Glossary

̸

B Proofs

Theorem 4 (Impossibility) . Assume there exist distinct policies, Π , Ω , Λ , none of which is a mixture (i.e., convex combination) of the other two, and consider the aggregation of arbitrary individual preference relations {≻ i ; i ∈ I} defined on L ( P ) that individually satisfy Axioms 1-2. There does not exist aggregated preference relation ≻ Σ satisfying Axioms 1-4.

Proof. Fix s, a . Using Theorem 2, choose { r i , γ i } , r Σ , γ Σ to represent individual and aggregate preferences over L ( P ) . Define mixture policy Π β := β Π+(1 -β )Ω . W.l.o.g. assume |I| = 2 .

We use Axiom 4 (Unrestricted Domain) to set

and, using Theorem 1 to shift V 1 , V 2 , we have

̸

for some β 1 , β 2 ∈ (0 , 1) with β 1 = β 2 (again appealing to Unrestricted Domain). Intuitively, equation 7 requires individual preferences to conflict, and the β 1 = β 2 condition requires them to be non-symmetric about Λ . Equation 8 is merely a convenient choice of numerical representation.

̸

We now apply Theorem 3 followed by Theorem 2 to the expression V Σ ( s, a Π β ) -V Σ ( s, a Λ) . We again invoke Theorem 1 to shift V Σ and eliminate the constant term, so that by Theorem 3 ∃{ w i } for which,

where the second line applies Theorem 2, with rewards cancelling out.

Alternatively, applying Theorem 2 followed by Theorem 3 to the same expression yields,

Combining equations 9 and 10, we obtain:

Finally, setting β to be β 1 or β 2 in equation 11, we obtain the equalities:

The w i are non-zero (Axiom 4, Sensitivity) and the remaining values are non-zero (else β 1 = β 2 ), so we conclude that γ Σ ( s, a ) = γ 1 ( s, a ) = γ 2 ( s, a ) . But this contradicts our assumption that individual preferences {≻ i ; i ∈ I} may be chosen arbitrarily, completing the proof.

Theorem 5 (Possibility) . Consider the aggregation of arbitrary individual preference relations {≻ i ; i ∈ I} defined on L ( P ) , and consequently L ( P h ) , ∀ h , that individually satisfy Axioms 1-2. There exists aggregated preference relations {≻ h Σ } , defined on L ( P h ) , ∀ h , that satisfy Axioms 1-4.

In particular, given s, a , V Σ , { V i } , { w i ( s ) } , where (A) each V i satisfies Axioms 1-2 on L ( P ) , and V Σ satisfies Axioms 1-2 on L ( P h ) , ∀ h , and (B) V Σ ( s, a Π) = ∑ i w i ( s ) V i ( s, a Π)) , then , choosing

implies that V Σ ( sas ′ , Π) ∝ ∑ i w i ( sa ) V i ( s ′ , Π) so that the aggregated preferences {≻ sas ′ Σ } satisfy Axiom 3 on L ( P sas ′ ) . Unrolling this resultw i ( hsas ′ ) := w i ( hs ) γ i ( s, a ) -produces a set of constructive, history dependent weights { w i ( h ) } such that Axiom 3 is satisfied for all histories { h } .

Proof. We follow the proof of Theorem 4. Fix s, a . Using Theorem 2, choose { r i , γ i } , r Σ , γ Σ to represent individual and aggregate preferences over L ( P s ) and L ( P sas ′ ) . Define mixture policy Π β := β Π+(1 -β )Ω . W.l.o.g. assume |I| = 2 .

We use Axiom 4 (Unrestricted Domain) to set

and, using Theorem 1 to shift V 1 , V 2 , we have

̸

for some β 1 , β 2 ∈ (0 , 1) with β 1 = β 2 (again appealing to Unrestricted Domain).

We now apply Theorem 3 followed by Theorem 2 to the expression V Σ ( s, a Π β ) -V Σ ( s, a Λ) . We again invoke Theorem 1 to shift V Σ and eliminate the constant term, so that by Theorem 3 ∃{ w i ( s ) } for which,

where the second line applies Theorem 2, with rewards cancelling out.

It suffices to find one set of satisfactory w i ( h ) , so we can assume that, given s, a , w i ( sas ′ ) := w ( sa ) is the same for all s ′ . This will allow us to factor it out below. Then, applying Theorem 2 followed by Theorem 3 to the same expression yields,

Combining equations 16 and 17, setting β to be β 1 or β 2 in equation, and rearranging, we obtain the equalities:

from which we conclude that:

This shows the existence of weights w i ( sa ) , unique up to a constant scaling factor, for which V Σ ( T ( s, a ) , Π) ∝ ∑ i w i ( sa ) V i ( T ( s, a ) , Π) , that apply regardless of how individual preferences are chosen or aggregated at s . Unrolling the result completes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a general multi-objective sequential decision-making setting where each objective may use a different discount factor. Using an axiomatic approach, the authors prove that under some axioms (vNM axioms + dynamic consistency for the relation on each objective), an aggregated preference relation cannot simultaneously satisfy the vNM axioms, dynamic consistency, Pareto indifference, and some technical conditions. In addition, the authors discuss some ways out to this impossibility result, notably via state augmentation or relaxing dynamic consistency.\n\nSTRENGTHS:\nImpossibility result, although the result is actually not very surprising when different discount factors are allowed\n\nProposition and discussion about different solutions to this impossibility result\n\nWEAKNESSES:\nThe presentation and organization of the paper could be improved. Notably:\n\nThe results could be presented in a more accessible way to a more general audience. The authors seems to know well the related literature in decision theory and economics, which may not necessarily be the case for the NeurIPS audience. For instance, the second and third paragraphs of Section 5.1 are quite hard to follow for a non-expert.\n\nSection 4.2 should be checked. Some notations (e.g., h_{:-1}, y_i, or y) are not explained properly or are not used in a rigorous way.\n\nIt is not clear to me why Section 5 combines a presentation of other solutions to the impossibility theorem and a discussion of related work.\n\nAs far as I know, most work in multiobjective reinforcement learning applies an identical discount factor on all the objectives. Since the impossibiity theorem doesn't apply in this case, the results of this paper don't apply to most such work. Therefore, I believe researchers may be mislead by the title of this paper. I suggest the authors to use a more precise one.\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNot applicable, this is a theoretical paper", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors analyze the implications of preference aggregation within a Markov Decision Process Framework. They show that it is not possible to ensure dynamic consistency in an aggregated MDP if one also wants to be able to accommodate arbitrary preference criteria, even if the criteria are individually dynamically consistent. They further show that by relaxing the Markov condition incrementally, dynamic consistency can be recovered.\n\nSTRENGTHS:\nAddresses a fundamental representational issue in MDPs. The authors clearly have a deep understanding of the temporal consistency literature in economics and decision theory, and bring it to bear here. The technical exposition is clear, and the reasoning is sound. The example of procrastination is instructive. The construction of a patch to deal with different discount rates is perhaps the most directly useful contribution. There is also some interesting extended discussion about intemporal preferences that could be quite relevant in an AI context.\n\nWEAKNESSES:\nThe authors attempt to motivate the contribution in the context of the current \"Reward is Enough\" debate in RL. This is tantalizing, but seems to me a bit forced. Really we have a basic technical question in representation of intertemporal preferences, and the paper underlines a common lesson that aggregating across separate preferences is never as straightforward as one might think. The technical points are perhaps connected to some arguments debate participants have brought up (and they should indeed be better versed in intertemporal choice), but ultimately we face the same questions we always do about how much and what kind of state should we incorporate to keep things approximately enough Markovian.\n\nQUESTIONS:\nTechnical Comments\n\nC1. The whole concept of a discount factor in MDPs is to represent a global time preference. Why should anyone have thought it could be coherent to have different time preferences on different criteria? To the extent that discounting is really capturing dynamics in the world (e.g., consequences of work persisting), then arguably that would more properly be represented in the state space to begin with. That is, using the discount factor for this is a hack and one should not be surprised it is fragile.\n\nC2. Axiom 3 is really strong. It basically entails strong independence of criteria--what is needed to get an additive representation. Many readers will not realize that. Moreover, I suspect it is much stronger than needed to get your key impossibility result. That is, even forms with many more interaction terms will probably run into problems with temporal consistency.\n\nC3. On reflection, it seems to me that the key technical point here could be restated as saying that given Axiom 3, the *only* thing that can go wrong is mismatch of discount factors. Do you agree with that characterization?\n\nQuibbles\n\nQ1. About the procrastination story. Line 67 says that the policy must remember it had previously chosen \"play\" in order to work forever. Not true (it actually does not matter what was executed in the first step): it just needs to know it is beyond the first step.\n\nQ2. Title. \"Multi-objective agency\" is a very ill-defined term, so hard to buy the assertion it requires anything in particular. What is actually proved here is about what is required for dynamically consistent preference aggregation. \n\nQ3. Exposition had a few minor gaps, for example \\cal{P} never formally defined (clear from context, though).\n\nLIMITATIONS:\nThe paper adequately discusses assumptions and limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper examines multi-objective reinforcement learning---the setting in which multiple distinct objectives are desired, and often combined, to form a composite objective. Concretely, the paper explores the limits of aggregating different objectives by appealing to three main pools of ideas. First, to the von Neumann-Morgenstern expected utility, axioms; Second, to Pareto indifference; and Third, to dynamic consistency. The main result is an impossibility result, illustrating that objectives with different time-based objectives (that is, different discount factors), cannot be aggregated in a way that yields a Markovian reward function even if the individual objectives themselves are Markovian. The paper then explores what lies beyond this impossibility result, exploring a mechanism for expanding the state-space to collapse the non-Markovian aggregated objective down to a Markovian one. This results in a \"historical\" discount factor, a hindsight view of the discount factor.\n\nSTRENGTHS:\n**STRENGTHS**\n\nThe paper possesses many strengths:\n1) The aspirations of the work are ambitious and important. Clearly establishing when certain kinds of objectives can and cannot be captured is important.\n2) The work is rigorous, and well-connected to classical results in decision theory.\n3) The examples are clear and help to communicate the main ideas.\n4) The impossibility result on its own is interesting. Once I understood the details, it is not ultimately surprising, but I do not believe the result needs to be surprising to be useful.\n5) Historical discounting is a new and interesting idea.\n\nWEAKNESSES:\n**WEAKNESSES**\n\nAt the same time, the paper has several weaknesses:\n1) Language. First, and my biggest critique, the work commonly makes use of unusual and vague language surrounding some of the main concepts. For instance, the title, and central idea of the work---multi-objective agency---is not well-defined, and by my reading is not an appropriate choice of description for the content of the work. I would recommend moving away from \"agency\" as a term, and certainly \"multi-objective agency\", as neither are well defined in this paper. Instead, I would suggest using \"multi-critieria objectives\", or \"multi-objective RL\", as is used in prior literature. It is much more clear and precise, and more well connected with the work.\n2) Clarity, and detail of exposition. Ideas are often introduced abruptly and not explained in much detail. For instance, the axioms in section 3.1 are simply stated without any added context or explanation. While vNM is quite common, dynamic consistency is less so, and likely deserves a more thorough, careful, and simple explanation, given its central role in the work. Similarly, after some theorems are introduced, they are sometimes not discussed. Other details are often left unexplained, such as \"...none of which is a mixture of the other two\" in Theorem 4 (the main result). From a quick reading it is possible to understand this, but it would be worthwhile to spell this out carefully.\n3) Notation is sometimes defined quite precisely, but it is often overly complex or not defined. For instance, in Theorem 5, it is unclear what {$\\succ_\\Sigma^{sas'}$} is intended to mean. Conventions tend to deviate quite a lot from typical work in reinforcement learning as well (getting rid of the Q function in favor of two uses of V, using $\\Pi$ instead of $\\pi$). \n4) Unsurprising results. Lastly, I do believe most of the results are unsurprising. When discount factors vary across objectives, it is perhaps expected that their aggregation will not be representable in the same form (and that we can augment the state space to remedy this). Still, it  is useful to make these arguments carefully and rigorously.\n\nQUESTIONS:\nPrimary recommendation: My strongest recommendation is to move Section 4 to the Appendix, and to use the remaining space to provide additional clarity and exposition around key ideas around results. I believe all of the Axioms need more careful and simpler introductions and discussions (and especially Axiom 3), as well as the main results. I believe this switch will strengthen the paper considerably.\n\nMain Comments/Questions:\n- \"Our main contribution is an impossibility result from which one concludes that non-Markovian rewards are likely necessary for RL agents that pursue multiple objectives or serve multiple principals\". This seems slightly too strong, by my reading of the main result---it really only applies when the discounts across the objectives differ. Is this the role that \"likely\" is playing in this statement? If so I might suggest adjusting the language to be more precise.\n- Separately, I wonder if the paper can comment on when the discount should be associated with an _agent_ rather than an _objective_ in isolation.\n- Two pieces of related work come to mind. First, the expressivity of \"multi-dimensional\" reward was explored by Miura in 2022: On the Expressivity of Multidimensional Markov Reward at the RLDM workshop on RL as Agency. I do wonder about the connections between these two sets of results. Second, Tasse et al. propose an algebra on tasks in RL in \"A boolean task algebra for reinforcement learning\". I wonder about how this style of composition bears on the findings and perspectives from the present paper.\n\nSmall Questions:\n- \"While this allows us to design agents that implement these policies, it doesn’t quite solve the intertemporal choice problem\": what is meant by this statement? Actually designing agents that implement these policies does not seem to be in the purview of this paper. Instead, we can aggregate objectives in a way that allows us to _incentivize_ agents according to an appropriate objective, but this is different from the agent design process.\n- I do not understand Axiom 3 as stated and the explanation of the notation below is too brief to elucidate. Given the importance of the Axiom, I encourage expanding the explanation.\n\nTypos and writing suggestions:\n- As mentioned above, I think the phrase \"multi-objective agency\" is not the appropriate way to describe the main concept of the paper. I suggest replacing \"multi-objective agency\" with \"multi-criteria objectives\", or \"multiple objectives\", or some variant thereof.\n- I don't believe \"(history dependent)\" is needed in the abstract following \"non-Markovian\"\n- It looks as thought reward functions are defined as both $R$ (in the definition of the MDP) and $r$ (in, say, Theorem 2). I would encourage picking one use throughout.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-30"}