SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Silviu Pitis
University of Toronto and Vector Institute
spitis@cs.toronto.edu
As the capabilities of artificial agents improve, they are being increasingly deployed to service multiple diverse objectives and stakeholders. However, the composition of these objectives is often performed ad hoc, with no clear justification. This paper takes a normative approach to multi-objective agency: from a set of intuitively appealing axioms, it is shown that Markovian aggregation of Markovian reward functions is not possible when the time preference (discount factor) for each objective may vary. It follows that optimal multi-objective agents must admit rewards that are non-Markovian with respect to the individual objectives. To this end, a practical non-Markovian aggregation scheme is proposed, which overcomes the impossibility with only one additional parameter for each objective. This work offers new insights into sequential, multi-objective agency and intertemporal choice, and has practical implications for the design of AI systems deployed to serve multiple generations of principals with varying time preference.
The idea that we can associate human preferences with scalar utility values traces back hundreds of years and has found usage in numerous applications [9, 71, 28, 49]. One of the most recent, and perhaps most important, is the design of artificial agents. In the field of reinforcement learning (RL), this idea shows up as the reward hypothesis [74, 67, 10], which lets us define objectives in terms of a discounted sum of Markovian rewards. While foundational results from decision theory [81, 62] and inverse RL [52, 54] justify the reward hypothesis when a single objective or principal is considered, complexities arise in multi-objective scenarios [61, 77]. The literature on social choice is largely defined by impossibilities [5], and multi-objective composition in the RL and machine learning literature is typically restrictive [68, 51], applied without clear justification [29], or based on subjective evaluations of empirical efficacy [21]. Addressing these limitations is crucial for the development of artificial agents capable of effectively serving the needs of diverse stakeholders.
This paper extends previous normative work in RL by adopting an axiomatic approach to the aggregation of objectives. The approach is based on a set of intuitively appealing axioms: the von Neumann-Morgenstern (VNM) axioms, which provide a foundation for rational choice under uncertainty; Pareto indifference, which efficiently incorporates individual preferences; and dynamic consistency, which ensures time-consistent decision-making. From these axioms, an impossibility is derived, leading to the conclusion that optimal multi-objective agents with diverse time preferences must have rewards that are non-Markovian with respect to the individual objectives. To address this challenge, a practical state space expansion is proposed, which allows for the Markovian aggregation of objectives requiring only one parameter per objective. The results prompt an interesting discussion on dynamic preferences and intertemporal choice, leading to a novel "historical discounting" strategy that trades off dynamic consistency for intergenerational fairness. Finally, it is shown how both our results can be extended (albeit non-normatively) to stochastic policies.
The remainder of this paper is organized as follows: Section 2 motivates the problem by modeling human procrastination behavior as an aggregation of two objectives, work and play, and showing how a plan that appears optimal today may lead to the worst possible future outcome. Section 3 presents the axiomatic background and the key impossibility result. Section 4 presents the corresponding possibility result and a practical state expansion to implement it. Section 5 relates the results to intertemporal choice, proposes  $N$ -step commitment and historical discounting strategies for managing intergenerational tradeoffs, extends the results to stochastic policies, and discusses related topics in RL. Section 6 concludes with some final thoughts and potential future research directions.
2 Motivation: The Procrastinator's Peril
We begin with a numerical example of how the naive aggregation of otherwise rational preferences can lead to undesirable behavior. The example, which will be referred to throughout as the "Procrastinator's Peril", involves repeated procrastination, a phenomenon to which the reader might relate. An agent aggregates two competing objectives: work and play. At each time step the agent can choose to either work or play. The pleasure of play is mostly from today, and the agent doesn't value future play nearly as much as present play. On the other hand, the consequences of work are delayed, so that work tomorrow is valued approximately as much as work today.
Let us model the agent's preferences for work and play as two separate Markov Decision Processes (MDP), each with state space  $S = \emptyset$  and action space  $A = \{\mathbf{w},\mathbf{p}\}$ . In the play MDP, we have rewards  $R(\mathfrak{p}) = 0.5$ ,  $R(\mathbf{w}) = 0$  and a discount factor of  $\gamma_{\mathrm{play}} = 0.5$ . In the work MDP, we have rewards  $R(\mathfrak{p}) = 0$ ,  $R(\mathbf{w}) = 0.3$  and a discount factor of  $\gamma_{\mathrm{work}} = 0.9$ . One way to combine the preferences for work and play is to value each trajectory under both MDPs and then add up the values. Not only does this method of aggregation seem reasonable, but it is actually implied by some mild and appealing assumptions about preferences (Axioms 1 and 3 in the sequel). Using this approach, the agent assigns values to trajectories as follows:
τ1	p,p,p,p...	V(τ1) = ∑t(0.5)t·0.5	= 1.00
τ2	w,w,w,w...	V(τ2) = ∑t(0.9)t·0.3	= 3.00
τ3	p,w,w,w...	V(τ3) = 0.5 + 0.9·V(τ2)	= 3.20
τ4	p,p,w,w...	V(τ3) = 0.75 + 0.92·V(τ2)	= 3.18
We see that the agent most prefers  $\tau_{3}$ : one period (and one period only!) of procrastination is optimal. Thus, the agent procrastinates and chooses to play today, planning to work from tomorrow onward. Come tomorrow, however, the agent is faced with the same choice, and once again puts off work in favor of play. The process repeats and the agent ends up with the least preferred alternative  $\tau_{1}$ .
This plainly irrational behavior illustrates the impossibility theorem. Observe that the optimal policy  $\tau_{3}$  is non-Markovian—it must remember that the agent has previously chosen play in order to work forever. But any MDP has a stationary optimal policy [55], so it follows that we need rewards that are non-Markovian with respect to the original state-action space. Alternatively, we will see in Subsection 4.2 that we can expand the state space to make the optimal policy Markovian.
3 Impossibility of Dynamically Consistent, Pareto Indifferent Aggregation
Notation We assume familiarity with Markov Decision Processes (MDPs) [55] and reinforcement learning (RL) [74]. We denote an MDP by  $\mathcal{M} = \langle S, \mathcal{A}, T, R, \gamma \rangle$ , where  $\gamma : S \times \mathcal{A} \to \mathbb{R}^+$  is a state-action dependent discount function. This generalizes the usual "fixed"  $\gamma \in \mathbb{R}$  and covers both the episodic and continuing settings [85]. We use lowercase letters for generic instances, e.g.  $s \in S$ , and denote distributions using a tilde, e.g.  $\tilde{s}$ . In contrast to standard notation we write both state- and state-action value functions using a unified notation that emphasizes the dependence of each on the future policy: we write  $V(s, \pi)$  and  $V(s, a\pi)$  instead of  $V^{\pi}(s)$  and  $Q^{\pi}(s, a)$ . We extend  $V$  to operate on probability distributions of states,  $V(\tilde{s}, \Pi) = \mathbb{E}_{s \sim \tilde{s}} V(s, \Pi)$ , and we allow for non-stationary, history-dependent policies (denoted by uppercase  $\Pi$ ,  $\Omega$ ). With this notation, we can understand  $V$  as an expected utility function defined over prospects of the form ( $\tilde{s}$ ,  $\Pi$ ). We use the letter  $h$  to denote histories (trajectories of states and actions)—these may terminate on either a state or action, as may be inferred from the context. For convenience, we sometimes directly concatenate
histories, states, actions and/or policies (e.g.,  $hs$ ,  $sa$ ,  $s\Pi$ ,  $a\Pi$ ) to represent trajectory segments and/or the associated stochastic processes. For simplicity, we assume finite  $|\mathcal{S}|$ ,  $|\mathcal{A}|$ .
3.1 Representing rational preferences
This paper is concerned with the representation of aggregated preferences, where both the aggregation and its individual components satisfy certain axioms of rationality. We define the objects of preference to be the stochastic processes ("prospects") generated by following (potentially non-stationary and stochastic) policy  $\Pi$  from state  $s$ . Distributions or "lotteries" over these prospects may be represented by (not necessarily unique) tuples of state lottery and policy  $(\tilde{s}, \Pi) \in \mathcal{L}(\mathcal{S}) \times \Pi =: \mathcal{L}(\mathcal{P})$ . We write  $(\tilde{s}_1, \Pi) \succ (\tilde{s}_2, \Omega)$  if  $(\tilde{s}_1, \Pi)$  is strictly preferred to  $(\tilde{s}_2, \Omega)$  under preference relation  $\succ$ .
To be "rational", we require  $\succ$  to satisfy the "VNM axioms" [81], which is capture in Axiom 1:
Axiom 1 (VNM). For all  $\tilde{p},\tilde{q},\tilde{r}\in \mathcal{L}(\mathcal{P})$  we have:
Asymmetry: If  $\tilde{p} \succ \tilde{q}$ , then not  $\tilde{q} \succ \tilde{p}$ ;
Negative Transitivity: If not  $\tilde{p} \succ \tilde{q}$  and not  $\tilde{q} \succ \tilde{r}$ , not  $\tilde{p} \succ \tilde{r}$ ;
Independence: If  $\tilde{p} \succ \tilde{q}$ , then  $\alpha \tilde{p} + (1 - \alpha) \tilde{r} \succ \alpha \tilde{q} + (1 - \alpha) \tilde{r}, \forall \alpha \in (0, 1]$ ;
Continuity: If  $\tilde{p} \succ \tilde{q} \succ \tilde{r}$ , then  $\exists \alpha, \beta \in (0,1)$  such that  $\alpha \tilde{p} + (1 - \alpha) \tilde{r} \succ \tilde{q} \succ \beta \tilde{p} + (1 - \beta) \tilde{r}$ ;
where  $\alpha \tilde{p} + (1 - \alpha) \tilde{q}$  denotes the mixture lottery with  $\alpha\%$  chance of  $\tilde{p}$  and  $(1 - \alpha)\%$  chance of  $\tilde{q}$ .
Asymmetry and negative transitivity together form the basic requirements of a strict preference relation—equivalent to completeness and transitivity of the corresponding weak preference relation,  $\succeq$  (defined as  $p\succeq q\Leftrightarrow q\not\Rightarrow p$ ). Independence can be understood as an irrelevance of unrealized alternatives, or consequentialist, axiom: given that the  $\alpha \%$  branch of the mixture is realized, preference between  $\tilde{p}$  and  $\tilde{q}$  is independent of the rest of the mixture (i.e., what could have happened on the  $(1 - \alpha)\%$  branch). Finally, continuity is a natural assumption given that probabilities are continuous.
We further require  $\succ$  to be dynamically consistent [70, 40, 44]:
Axiom 2 (Dynamic consistency).  $(s, a\Pi) \succ (s, a\Omega)$  if and only if  $(T(s, a), \Pi) \succ (T(s, a), \Omega)$  where  $T(s, a)$  is the distribution over next states after taking action  $a$  in state  $s$ .
This axiom rules out the irrational behavior in the Procrastinator's Peril, by requiring today's preferences for tomorrow's actions to be the same as tomorrow's preferences. While some of these axioms (particularly independence and dynamic consistency) have been the subject of debate (see, e.g., [44]), note that the standard RL model is more restrictive than they require [54].
The axioms produce two key results that we rely on (see Kreps [39] and Pitis [54] for proofs):
Theorem 1 (Expected utility representation). The relation  $\succ$  defined on the set  $\mathcal{L}(\mathcal{P})$  satisfies Axiom 1 if and only if there exists a function  $V:\mathcal{P}\to \mathbb{R}$  such that,  $\forall \tilde{p},\tilde{q}\in \mathcal{L}(\mathcal{P})$
$$
\tilde {p} \succ \tilde {q} \iff \sum_ {z \in s u p p (\tilde {p})} \tilde {p} (z) V (z) > \sum_ {z \in s u p p (\tilde {q})} \tilde {q} (z) V (z).
$$
Another function  $V^{\dagger}$  gives this representation iff  $V^{\dagger}$  is a positive affine transformation of  $V$ .
Using Theorem 1, we extend the domain of value function  $V$  to  $\mathcal{L}(\mathcal{P})$  as  $V(\tilde{p}) = \sum_{z}\tilde{p} (z)V(z)$
Theorem 2 (Generalized Bellman representation). If  $\succ$  satisfies Axioms 1-2 and  $V$  is an expected utility representation of  $\succ$ , there exist  $R: S \times A \to \mathbb{R}$ ,  $\gamma: S \times A \to \mathbb{R}^{+}$  such that  $\forall s, a, \Pi$ ,
$$
V (s, a \Pi) = R (s, a) + \gamma (s, a) V (T (s, a), \Pi).
$$
Remark 3.1.1 Instead of preferences over stochastic processes of potential futures, one could begin with preferences over trajectories [86, 64, 10]. The author takes issue with this approach, however, as it's unclear that such preferences should satisfy Asymmetry or Independence without additional assumptions (humans often consider counterfactual outcomes when evaluating the desirability of a trajectory) [54]. By using Theorem 2 to unroll prospects, one can extend preferences over prospects to define preferences over trajectories according to their discounted reward.
Remark 3.1.2 Theorem 2, as it appeared in Pitis [54], required an additional, explicit "Irrelevance of unrealizable actions" axiom, since prospects were defined as tuples  $(\tilde{s},\Pi)$ . This property is implicit in our redefinition of prospects as stochastic processes.
Remark 3.1.3 In this line of reasoning only the preference relation  $\succ$  is primitive;  $V$  and its Bellman form  $(R,\gamma)$  are simply representations of  $\succ$  whose existence is guaranteed by the axioms. Not all numerical representations of  $\succ$  have these forms [84]. In particular, (strictly) monotonically increasing transforms preserve ordering, so that any increasing transform  $V^{\dagger}$  of a Theorem 1 representation  $V$  is itself a valid numerical representation of  $\succ$  (although lotteries will no longer be valued by the expectation over their atoms unless the transform is affine, per Theorem 1).
3.2 Representing rational aggregation
Let us now consider the aggregation of several preferences. These may be the preferences of an agent's several principals or preferences representing a single individual's competing interests. Note at the outset that it is quite natural for different objectives or principals to have differing time preference. We saw one example in the Procrastinator's Peril, but we can also consider a household robot that seeks to aggregate the preferences of Alice and her husband Bob, for whom there is no reason to assume equal time preference [26].
An intuitively appealing axiom for aggregation is Pareto indifference, which says that if each individual preference is indifferent between two alternatives, then so too is the aggregate preference.
Axiom 3 (Pareto indifference). If  $\tilde{p} \approx_{i} \tilde{q} (\forall i \in \mathcal{I})$ ,  $\tilde{p} \approx_{\Sigma} \tilde{q}$ .
Here,  $\tilde{p} \approx \tilde{q}$  means indifference (not  $\tilde{p} \succ \tilde{q}$  and not  $\tilde{q} \succ \tilde{p}$ ), so that  $\approx_{i}$  is the  $i$ th individual indifference relation,  $\mathcal{L}$  is a finite index set over the individuals, and  $\approx_{\Sigma}$  indicates the aggregate relation. There exist stronger variants of Pareto property that require monotonic aggregation (e.g., if all individuals prefer  $\tilde{p}$ , so too does the aggregate; see Axiom  $3'$  in Subsection 5.1). We opt for Pareto indifference to accommodate potentially deviant individual preferences (e.g., if all individuals are indifferent but for a sociopath, the aggregate preference may be opposite of the sociopath's).
If we require individual and aggregate preferences to satisfy Axiom 1 and, jointly, Axiom 3, we obtain a third key result due to Harsanyi [32]. (See Hammond [31] for proof).
Theorem 3 (Harsanyi's representation). Consider individual preference relations  $\{\succ_{i}; i \in \mathcal{I}\}$  and aggregated preference relation  $\succ_{\Sigma}$ , each defined on the set  $\mathcal{L}(\mathcal{P})$ , that individually satisfy Axiom 1 and jointly satisfy Axiom 3. If  $\{V_{i}; i \in \mathcal{I}\}$  and  $V_{\Sigma}$  are expected utility representations of  $\{\succ_{i}; i \in \mathcal{I}\}$  and  $\succ_{\Sigma}$ , respectively, then there exist real-valued constant  $c$  and weights  $\{w_{i}; i \in \mathcal{I}\}$  such that:
$$
V _ {\Sigma} (\tilde {p}) = c + \sum_ {i \in \mathcal {I}} w _ {i} V _ {i} (\tilde {p}).
$$
That is, the aggregate value can be expressed as a weighted sum of individual values (plus a constant).
According to Harsanyi's representation theorem, the aggregated value function is a function of the individual value functions, and nothing else. In other words, Pareto indifferent aggregation of VNM preferences that results in VNM preference is necessarily context-free—the same weights  $\{w_{i}\}$  apply regardless of state and policy.
We will also make use of two technical conditions to eliminate certain edge cases. Though sometimes left implicit, these are both common requirements for aggregation functions [5].
Axiom 4 (Technical conditions on  $\succ_{\Sigma}$ ).
Unrestricted Domain:  $\succ_{\Sigma}$  is defined for all valid individual preference sets  $\{\succ_{i}; i \in \mathcal{I}\}$ .
Sensitivity:  $\forall i\in \mathcal{I}$ , holding  $\succ_{j}, j\neq i$  constant, there exist  $\succ_{1}^{1},\succ_{i}^{2}$  resulting in different  $\succ_{\Sigma}$ .
The first condition allows us to consider conflicting objectives with different time preference. The second condition implies that the weights  $w_{i}$  in Theorem 3 are nonzero.
Remark 3.2.1 It is worth clarifying here the relation between Harsanyi's theorem and a related class of aggregation theorems, occasionally cited within the machine learning literature (e.g., [6]), based on axioms originating with Debreu [19]. One instance of this class states that any aggregate preference satisfying six reasonable axioms can be represented in the form:  $V_{\Sigma}(\tilde{p}) = \sum_{i\in \mathcal{I}}m(V_i(\tilde{p}))$ , where  $m$  is a strictly increasing monotonic function from the family  $\{x^p |0 < p\leq 1\} \cup \{\log (x)|p = 0\} \cup \{-x^p |p < 0\}$  [47]. If we (1) drop the symmetry axiom from this theorem to obtain variable weights  $w_{i}$ , and (2) add a monotonicity axiom to Harsanyi's theorem to ensure positive weights  $w_{i}$  [32], then the only difference between the theorems is the presence of monotonic function  $m$ . But
note that applying  $m$  to each  $V_{i}$  or to  $V_{\Sigma}$  individually does not change the preference ranking they represent; it does, however, decide whether  $V_{i}$  and  $V_{\Sigma}$  are VNM representations of  $\succ$ . Thus, we can understand Harsanyi's theorem as saying: if  $V_{i}, V_{\Sigma}$  are VNM representations of  $\succ$ , then  $m$  must be linear. Or conversely, if  $m$  is non-linear ( $p \neq 1$ ),  $V_{i}$  and  $V_{\Sigma}$  are not VNM representations.
Remark 3.2.2 A caveat of Harsanyi's theorem is that it implicitly assumes that all individual preferences, and the aggregate preference, use the same set of agreed upon, "objective" probabilities. This is normatively justifiable if we use the same probability distribution (e.g., that of the aggregating agent) to impose "ideal" preferences  $\succ_{i}$  on each individual, which may differ from their implicit subjective or revealed preferences [62]. As noted by Desai et al. [20], Harsanyi's theorem fails if the preferences being aggregated use subjective probabilities. Note, however, that the outcome of the "bargaining" construction in Desai et al. [20] is socially suboptimal when the aggregating agent has better information than the principals, which suggests that effort should be made to unify subjective probabilities. We leave exploration of this to future work.
3.3 Impossibility result
None of Theorems 1-3 assume all Axioms 1-4. Doing so leads to our key result, as follows.
Theorem 4 (Impossibility). Assume there exist distinct policies,  $\Pi, \Omega, \Lambda$ , none of which is a mixture (i.e., convex combination) of the other two, and consider the aggregation of arbitrary individual preference relations  $\{\succ_{i}; i \in \mathcal{I}\}$  defined on  $\mathcal{L}(\mathcal{P})$  that individually satisfy Axioms 1-2. There does not exist aggregated preference relation  $\succ_{\Sigma}$  satisfying Axioms 1-4.
Sketch of Proof. The full proof is in Appendix B. Briefly, we consider  $|\mathcal{I}| = 2$  and use Axiom 4 (Unrestricted Domain) to construct mixtures of  $\Pi$  and  $\Omega$  so that each mixture is considered indifferent to  $\Lambda$  by one of the individual preference relations. Then, by applying Theorem 2 and Theorem 3 in alternating orders to the difference between the value of the mixture policy and the value of  $\Lambda$ , and doing some algebra, we arrive at the equations
$$
w _ {1} \gamma_ {1} (s, a) = w _ {1} \gamma_ {\Sigma} (s, a) \quad \text {a n d} \quad w _ {2} \gamma_ {2} (s, a) = w _ {2} \gamma_ {\Sigma} (s, a), \tag {1}
$$
from which we conclude that  $\gamma_{\Sigma}(s,a) = \gamma_1(s,a) = \gamma_2(s,a)$ . But this contradicts our assumption that individual preferences  $\{\succ_{i}; i \in \mathcal{I}\}$  may be chosen arbitrarily, completing the proof.
Intuition of Proof. Under mild conditions, we can find two policies (a  $\Pi/\Omega$  mixture, and  $\Lambda$ ) between which individual preference  $\succ_1$  is indifferent, but individual preference  $\succ_2$  is not. Then for mixtures of this  $\Pi/\Omega$  mixture and  $\Lambda$ ,  $\succ_1$  remains indifferent, but the strength of  $\succ_2$  changes, so that  $\succ_\Sigma$  must have the same time preference as  $\succ_2$ . By an analogous argument,  $\succ_\Sigma$  must have the same time preference as  $\succ_1$ , leading to a violation of Unrestricted Domain.
The closest results from the economics literature consider consumption streams  $(S = \mathbb{R})$  [88, 15, 83]. Within reinforcement learning, equal time preference has been assumed, without justification, when merging MDPs [68, 41] and value functions for different tasks [29].
Remark 3.3.1 A consequence of Unrestricted Domain, critical to the impossibility result, is that individual preferences may have diverse time preferences (i.e., different discount functions). If discounts are equal, Markovian aggregation is possible.
Remark 3.3.2 Per Remark 3.2.2, by applying Harsanyi's theorem, we are implicitly assuming that all preferences are formed using the same "objective" probabilities over prospects; is there a notion of "objective" time preference that should be used? If so, this would resolve the impossibility (once again, requiring that we impose a notion of ideal preference on individuals that differs from their expressed preference). We leave this consideration for future work.
Remark 3.3.3 Theorem 4 applies to the standard RL setup, where  $\gamma_{1},\gamma_{2},\gamma_{\Sigma}$  are constants.
4 Escaping Impossibility with Non-Markovian Aggregation
An immediate consequence of Theorem 4 is that any scalarized approach to multi-objective RL [79] is generally insufficient to represent composed preferences. But the implications run deeper: insofar as general tasks consist of several objectives, Theorem 4 pushes Sutton's reward hypothesis to its
limits. To escape impossibility, the Procrastinator's Peril is suggestive: to be productive, repeat play should not be rewarded. And for this to happen, we must keep track of past play, which suggests that reward must be non-Markovian, even when all relevant objectives are Markovian. That is, even if we have settled on some non-exhaustive state representation that is "sufficiently" Markovian, an extra aggregation step could render it no longer sufficient.