SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Relaxing Markov Preference The way in which non-Markovian rewards (or equivalently, non-Markovian utilities) can be used to escape Theorem 4 is quite subtle. Nowhere in the proofs of Theorems 1-4 is the Markov assumption explicitly used. Nor does it obviously appear in any of the Axioms. The Markov assumption is, however, invoked in two places. First, to establish history-independent comparability between the basic objects of preference—prospects  $(s, \Pi)$ —and second, to extend that comparison set to include "prospects" of the form  $(T(s, a), \Pi)$ . To achieve initial comparability, Pitis [54] applied a "Markov preference" assumption (preferences over prospects are independent of history) together with an "original position" construction that is worth repeating here:
[I]t is admittedly difficult to express empirical preference over prospects ... an agent only ever chooses between prospects originating in the same state ... [Nevertheless,] we imagine a hypothetical state from which an agent chooses between [lotteries] of prospects, denoted by  $\mathcal{L}(\mathcal{P})$ . We might think of this choice being made from behind a "veil of ignorance" [58].
In other words, to allow for comparisons between prospect  $(s_1,\Pi)$  and  $(s_2,\Pi)$ , we presuppose some pseudo-state,  $s_0$ , and compare prospects  $(s_0s_1,\Pi)$  and  $(s_0s_2,\Pi)$ . Markov preference then lets us cut off the history, so that our preferences between  $(s_1,\Pi)$  and  $(s_2,\Pi)$  are cardinal.
The impossibility result suggests, however, that aggregate preference is not independent of history, so that construction 2 cannot be applied. Without this construction, there is no reason to require relative differences between  $V(s_{1},*)$  and  $V(s_{2},*)$  to be meaningful, or to even think about lotteries/mixtures of the two prospects (as done in Axiom 1). Letting go of this ability to compare prospects starting in different states means that Theorem 1 is applicable only to sets of prospects with matching initial states, unless we shift our definition of "prospect" to include the history; i.e., letting  $h$  represent the history, we now compare "historical prospects" with form  $(h,\Pi)$ .
Though this does not directly change the conclusion of Theorem 2,  $T(s,a)$  in  $V(T(s,a),\Pi)$  includes a piece of history,  $(s,a)$ , and can no longer be computed as  $\mathbb{E}_{s'\sim T(s,a)}V(s',\Pi)$ . Instead, since the agent is not choosing between prospects of form  $(s',\Pi)$  but rather (abusing notation) prospects of form  $(sas',\Pi)$ , the expectation should be computed as  $\mathbb{E}_{s'\sim T(s,a)}V(sas',\Pi)$ .
The inability to compare prospects starting in different states also changes the conclusion of Theorem 3, which implicitly uses such comparisons to find constant coefficients  $w_{i}$  that apply everywhere in the original  $\mathcal{L}(\mathcal{P})$ . Relaxing the application of Harsanyi's theorem to not make inter-state comparisons results in weights  $w_{i}(h)$  that are history dependent when aggregating the historical prospects.
4.1 Possibility Result
Allowing the use of history dependent coefficients in the aggregation of  $V_{i}(T(s,a),\Pi)$  resolves the impossibility. The following result shows that given some initial state dependent coefficients  $w_{i}(s)$ , we can always construct history dependent coefficients  $w_{i}(h)$  that allow for dynamically consistent aggregation satisfying all axioms. In the statement of the theorem,  $\mathcal{L}(\mathcal{P}_h)$  is used to denote the set of lotteries over prospects starting with history  $h$  of arbitrary but finite length (not to be confused with the set of lotteries over all historical prospects, of which there is only one). Note that if a preference relation satisfies Axioms 1-2 with respect to  $\mathcal{L}(\mathcal{P})$ , the natural extension to  $\mathcal{L}(\mathcal{P}_{hs})$ ,  $(hs,\Pi_1)\succ (hs,\Pi_2)\iff (s,\Pi_1)\succ (s,\Pi_2)$ , satisfies Axioms 1-2 with respect to  $\mathcal{L}(\mathcal{P}_{hs})$ . Here, we are using  $hs$  to denote a history terminating in state  $s$ .
Theorem 5 (Possibility). Consider the aggregation of arbitrary individual preference relations  $\{\succ_{i}; i \in \mathcal{I}\}$  defined on  $\mathcal{L}(\mathcal{P})$ , and consequently  $\mathcal{L}(\mathcal{P}_h)$ ,  $\forall h$ , that individually satisfy Axioms 1-2. There exists aggregated preference relations  $\{\succ_{\Sigma}^{h}\}$ , defined on  $\mathcal{L}(\mathcal{P}_h)$ ,  $\forall h$ , that satisfy Axioms 1-4.
In particular, given  $s$ ,  $a$ ,  $V_{\Sigma}$ ,  $\{V_i\}$ ,  $\{w_i(s)\}$ , where  $(A)$  each  $V_i$  satisfies Axioms 1-2 on  $\mathcal{L}(\mathcal{P})$ , and  $V_{\Sigma}$  satisfies Axioms 1-2 on  $\mathcal{L}(\mathcal{P}_h)$ ,  $\forall h$ , and  $(B)$ $V_{\Sigma}(s,a\Pi) = \sum_i w_i(s)V_i(s,a\Pi)$ , then, choosing
$$
w _ {i} \left(s a s ^ {\prime}\right) := w _ {i} (s a) := w _ {i} (s) \gamma_ {i} (s, a) \quad f o r a l l i, s ^ {\prime} \tag {3}
$$
implies that  $V_{\Sigma}(s a s', \Pi) \propto \sum_{i} w_{i}(s a) V_{i}(s', \Pi)$  so that the aggregated preferences  $\{\succ_{\Sigma}^{s a s'}\}$  satisfy Axiom 3 on  $\mathcal{L}(\mathcal{P}_{s a s'})$ . Unrolling this result— $w_{i}(h s a s') := w_{i}(h s) \gamma_{i}(s, a)$ —produces a set of constructive, history dependent weights  $\{w_{i}(h)\}$  such that Axiom 3 is satisfied for all histories  $\{h\}$ .
Sketch of Proof. The full proof is in Appendix B. Following the proof of Theorem 4, we arrive at
$$
w _ {1} (s) \gamma_ {1} (s, a) = w _ {1} (s a) \gamma_ {\Sigma} (s, a) \quad \text {a n d} \quad w _ {2} (s) \gamma_ {2} (s, a) = w _ {2} (s a) \gamma_ {\Sigma} (s, a), \tag {4}
$$
from which we conclude that:
$$
\frac {w _ {2} (s a)}{w _ {1} (s a)} = \frac {w _ {2} (s) \gamma_ {2} (s , a)}{w _ {1} (s) \gamma_ {1} (s , a)}. \tag {5}
$$
This shows the existence of weights  $w_{i}(sa)$ , unique up to a constant scaling factor, for which  $V_{\Sigma}(T(s,a),\Pi)\propto \sum_{i}w_{i}(sa)V_{i}(T(s,a),\Pi)$ , that apply regardless of how individual preferences are chosen or aggregated at  $s$ . Unrolling the result completes the proof.
From Theorem 5 we obtain a rather elegant result: rational aggregation over time discounts the aggregation weights assigned to each individual value function proportionally to its respective discount factor. In the Procrastinator's Peril, for instance, where we started with  $w_{\mathfrak{p}}(s) = w_{\mathfrak{w}}(s) = 1$ , at the initial (and only) state  $s$ , we might define  $w_{\mathfrak{p}}(sp s) = 0.5$  and  $w_{\mathfrak{w}}(sp s) = 0.9$ . With these non-Markovian aggregation weights and  $\gamma_{\Sigma}(sp) = 1$ , you can verify that (1) the irrational procrastination behavior is solved, and (2) the aggregated rewards for work and play are now non-Markovian.
Remark 4.1 (Important!) The discount  $\gamma_{\Sigma}$  is left undetermined by Theorem 5. One might determine it several ways: by appealing to construction 2 with respect to historical prospects in order to establish inter-state comparability, by setting it to be the highest individual discount (0.9 in the Procrastinator's Peril), by normalizing the aggregation to weights to sum to 1 at each step, or perhaps by another method. In any case, determining  $\gamma_{\Sigma}$  would also determine the aggregation weights, per equation 4 (and vice versa). We leave the consideration of different methods for setting  $\gamma_{\Sigma}$  and establishing inter-state comparability of  $V_{\Sigma}$  to future work. (NB: This is a normative question, which we leave unanswered. While one can make assumptions, as we will for our numerical example in Subsection 5.2, future research should be wary of accepting a solution just because it seems to work.)
4.2 A Practical State Space Expansion
The basic approach to dealing with non-Markovian rewards is to expand the state space in such a way that rewards become Markovian [27, 14, 2]. However, naively expanding  $S$  to a history of length  $H$  could have  $O((|S| + |\mathcal{A}|)^H)$  complexity. Fortunately, the weight update in equation 4 allows us to expand the state using a single parameter per objective. In particular, for history  $hsa$  and objective  $i$ , we append to the state the factors  $y_{i}(hsa) \coloneqq y_{i}(h)\gamma_{i}(s,a) / \gamma_{\Sigma}(s,a)$ , which are defined for every history, and can be accumulated online while executing a trajectory. Then, given a composition with any set of initial weights  $\{w_i\}$ , we can compute the weights of augmented state  $s_{\mathrm{aug}} = (s,y_i(hs))$  as  $w_{i}(s_{\mathrm{aug}}) = y_{i}(hs)\cdot w_{i}$ . Letting  $\mathcal{L}(\mathcal{P}^{(y)})$  be the set of prospects on the augmented state set, we get the following corollary to Theorem 5:
Corollary 1. Consider the aggregation of arbitrary individual preference relations  $\{\succ_{i}; i \in \mathcal{I}\}$  defined on  $\mathcal{L}(\mathcal{P})$ , and consequently  $\mathcal{L}(\mathcal{P}^{(y)})$ , that individually satisfy Axioms 1-2. There exists aggregated preference relation  $\{\succ_{\Sigma}\}$ , defined on  $\mathcal{L}(\mathcal{P}^{(y)})$ , that satisfies Axioms 1-4.
5.1 A Fundamental Tension in Intertemporal Choice
The state space expansion of Subsection 4.2 allows us to represent the (now Markovian) values of originally non-Markovian policies in a dynamically consistent way. While this allows us to design agents that implement these policies, it doesn't quite solve the intertemporal choice problem.
In particular, it is known that dynamic consistency (in the form of Koopmans' Stationarity [38]), together with certain mild axioms, implies a first period dictatorship: the preferences at time  $t = 1$  are decisive for all time [25] (in a sense, this is the very definition of dynamic consistency!). Generally speaking, however, preferences at time  $t \neq 1$  are not the same as preferences at time  $t = 1$  (this is
what got us into our Procrastinator's Peril to begin with!) and we would like to care about the value at all time steps, not just the first.
A typical approach is to treat each time step as a different generation (decision maker), and then consider different methods of aggregating preferences between the generations [33]. Note that (1) this aggregation assumes intergenerational comparability of utilities (see Remark 4.1), and (2) each generation is expressing their personal preferences about what happens in all generations, not just their own. Since this is a single aggregation, it will be dynamically consistent (we can consider the first period dictator as a benevolent third party who represents the aggregate preference instead of their own). A sensible approach might be to assert a stronger version of Axiom 3 that uses preference:
Axiom  $3^{\prime}$  (Strong Pareto Preference) If  $\tilde{p} \succeq_{i} \tilde{q} (\forall i \in \mathcal{I})$ , then  $\tilde{p} \succeq_{\Sigma} \tilde{q}$ ; and if, furthermore,  $\exists j \in \mathcal{I}$  such that  $\tilde{p} \succ_{j} \tilde{q}$ , then  $\tilde{p} \succ_{\Sigma} \tilde{q}$ .
Using Axiom  $3^{\prime}$  in place of Axiom 3 for purposes of Theorem 3 gives a representation that assigns strictly positive weights to each generation's utility. Given infinite periods, it follows (e.g., by the Borel-Cantelli Lemma for general measure spaces) that if utilities are bounded, some finite prefix of the infinite stream decides the future and we have a "finite prefix dictatorship", which is not much better than a first period one.
The above discussion presents a strong case against using dynamic consistency to determine a long horizon policy. Intuitively, this makes sense: preferences change over time, and our current generation should not be stuck implementing the preferences of our ancestors. One way to do this is by taking time out of the equation, and optimizing the expected individual utility of the stationary state-action distribution,  $d_{\pi}$  (cf. Sutton and Barto [74] (Section 10.4) and Naik et al. [50]):
$$
J (\pi) = \sum_ {s} d _ {\pi} (s) V _ {\pi} (s) \tag {6}
$$
Unfortunately, as should be clear from the use of a stationary policy  $\pi$ , this time-neutral approach falls short for our purposes, which suggests the use of a non-Markovian policy. While optimizing equation 6 would find the optimal stationary policy in the Procrastinator's Peril (work forever with  $V(\tau_{2}) = 3.0$ ), it seems clear that we should play at least once ( $V(\tau_{3}) = 3.2$ ) as this hurts no one and makes the current decision maker better off—i.e., simply optimizing (6) violates the Pareto principle.
This discussion exemplifies a known tension in intertemporal choice between Pareto optimality and the requirement to treat every generation equally—it is impossible to have both [16, 42]. In a similar spirit to Chichilnisky [16], who has proposed an axiomatic approach requiring both finite prefixes of generations and infinite averages of generations to have a say in the social preference, we will examine a compromise (to the author's knowledge novel) between present and future decision makers in next Subsection.
5.2 N-Step Commitment and Historical Discounting
We now consider two solutions to the intertemporal choice problem that deviate just enough from dynamic consistency to overcome finite period dictatorships, while capturing almost all value for each decision maker. In other words, they are "almost" dynamically consistent. We leverage the following observation: due to discounting, the first step decision maker cares very little about the far off future. If we play once, then work for 30 steps in the Procrastinator's Peril, we are already better off than the best stationary policy, regardless of what happens afterward.
This suggests that, rather than following the current decision maker forever, we allow them to commit to a non-Markovian policy for some  $N < \infty$  steps that brings them within some  $\epsilon$  of their optimal policy, without letting them exert complete control over the far future. To implement this, we could reset the accumulated factors  $y_{i}$  to 1 every  $N$  steps. This approach is the same as grouping every consecutive  $N$  step window into a single, dynamically consistent generation with a first period dictator.
A more fluid and arguably better approach is to discount the past when making current decisions ("historical discounting"). We can implement historical discounting with factor  $\eta \in [0,1]$  by changing the update rule for factors  $y_{i}$  to
$$
y _ {i} (h s a) := \eta \left[ y _ {i} (h) \frac {\gamma_ {i} (s , a)}{\gamma_ {\Sigma} (s , a)} \right] + (1 - \eta) w _ {i} ^ {n} (s),
$$
where  $w_{i}^{n}(s)$  denotes the initial weight for objective  $i$  at the  $n$ th generation (if preferences do not change over time,  $w_{i}^{n} = w_{i}$ ). This performs an exponential moving average of preferences over time—
high  $\eta < 1$  initially gives the first planner full control, but eventually discounts their preferences until they are negligible. This obtains a qualitatively different effect from  $N$ -step commitment, since the preferences of all past time steps have positive weight (by contrast, on the  $N$ th step,  $N$ -step commitment gives no weight to the  $N - 1$  past preferences). As a result, in the Procrastinator's Peril, any sufficiently high  $\eta$  returns policy  $\tau_{3}$ , whereas  $N$ -step commitment would play every  $N$  steps.
Historical discounting is an attractive compromise because it is both Pareto efficient, and also anonymous with respect to tail preferences—it both optimizes equation 6 in the limit, and plays on the first step. Another attractive quality is that historical discounting allows changes in preference to slowly deviate from dynamic consistency over time—the first period is not a dictator.
As a numerical example, we can consider what happens in the Procrastinator's Peril if we start to realize that we value occasional play, and preferences shift away from the play MDP to a playn MDP. The playn MDP has fixed  $\gamma_{\mathrm{playn}} = 0.9$  and non-Markovian reward  $R(\mathfrak{p}|$  no play in last N-1 steps)  $= 0.5$ ,  $R$  (anything else)  $= 0$ . We assume preferences shift linearly over the first 10 timesteps, from  $w_{\mathrm{play}} = 1$ ,  $w_{\mathrm{playn}} = 0$  at  $t = 0$  to  $w_{\mathrm{play}} = 0$ ,  $w_{\mathrm{playn}} = 1$  at  $t = 10$  (and  $w_{\mathrm{work}} = 1$  throughout). Then, the optimal trajectories  $\tau^{*}$  for different  $\eta$ , and resulting discounted reward for the first time step,  $V^{1}(\tau^{*})$ , are as follows:
η	τ*	V1(τ*)
0.00	play for 5 steps, then play every 10 steps	2.635
0.30	play for 5 steps, then play every 10 steps	2.635
0.50	play for 3 steps, then play every 10 steps	2.932
0.90	play, then work for 14 steps, then play every 10 steps	3.105
0.95	play, then work for 23 steps, then play every 10 steps	3.163
0.98	play, then work for 50 steps, then play every 10 steps	3.198
1.00	play, then work forever (same as τ3)	3.200
When  $\eta = 0$ , each time step acts independently, and since the play MDP has high weight at the start, we experience a brief period of consistent play in line with the original Procrastinator's Peril, before preferences fully shift to occasional play. With high  $\eta < 1$ , we capture almost all value for the first time step, while also eventually transitioning to the equilibrium "play every 10 steps" policy.
5.3 Extension to Boltzmann Policies
The preference relation  $\succ$  is deterministic, but the associated policy does not have to be. In many cases—partially-observed, multi-agent, or even fully-observed settings [74, 30]—stochastic policies outperform deterministic ones. And for generative sequence models such as LLMs [13], stochastic policies are inherent. We can extend our analysis—impossibility, possibility, state space expansion, and intertemporal choice rules—to such cases by adopting a stochastic choice rule.
To formalize this, we will take the stochastic choice rule as primitive, and use it to define a (deterministic) relation  $\succ$  that, for practical purposes, satisfies Axioms 1-4 [18]. We assume our choice rule satisfies Luce's Choice Axiom [43], which says that the relative rate of choosing between two alternatives in a choice set of  $n$  alternatives is constant regardless of the choice set. This can be implemented numerically with a Bradley-Terry choice model [11] by associating each alternative  $a_{i}$  with a scalar  $\Omega(a_{i})$ , so that given choice set  $\{a_{i}, a_{j}\}$ ,  $p(a_{i}) = \Omega(a_{i}) / (\Omega(a_{i}) + \Omega(a_{j}))$ . (An interesting interpretation for  $\Omega(a_{i})$  that connects to both probability matching [82, 65] and statistical mechanics [45] is as the number of "outcomes" (microstates) for which  $a_{i}$  is the best choice.)
We then simply "define" preference as  $a_i \succ a_j \iff \Omega(a_i) > \Omega(a_j)$ , and utility as  $V(a_i) \coloneqq k \log \Omega(a_i)$ , so that the policy is a softmax of the utilities. The way these utilities are used in practice (e.g., [30, 17]) respects Axioms 1-2. And summing utilities is a common approach to composition [21, 29], which is consistent with Harsanyi's representation (Theorem 3). For practical purposes then, the impossibility result applies whenever composed objectives may have different time preference.
Remark 5.3 Unlike our main results, this extension to Boltzmann policies is motivated by practical, rather than normative, considerations. Simple counterexamples to Luce's Choice Axiom exist [87] and probability matching behavior is evidently irrational in certain circumstances [65]. We note, however, that certain theoretical works tease at the existence of a normative justification for Boltzmann policies [18, 8, 73, 23]; given the practice, a clear justification would be of great value.
5.4 Related Work in RL
Task Definition Tasks are usually defined as the maximization of expected cumulative reward in an MDP [74, 55, 67]. Preference-based RL [86] avoids rewards, operating directly with preferences (but note that preference aggregation invokes Arrow's impossibility theorem [5, 48]), while other works translate preferences into rewards [17, 72, 12, 35]. This paper joins a growing list of work [54, 1, 76, 77] that challenges the implicit assumption that "MDPs are enough" in many reward learning papers, particularly those that disaggregate trajectory returns into stepwise rewards [22, 56, 59].
Discounting Time preference or discounting can be understood as part of the RL task definition. Traditionally, a constant  $\gamma$  has been used, although several works have considered other approaches, such as state-action dependent discounting [85, 66] and non-Markovian discounting [24, 63]. Several works have considered discounting as a tool for optimization [80] or regularization [36, 4, 57].
Task Compositionality Multi-objective RL [61] represents or optimizes over multiple objectives or general value functions [75], which are often aggregated with a linear scalarization function [7, 3]. Rather than scalarizing to obtain a single solution to a multi-objective problem, one can also seek out sets of solutions, such as the set of Pareto optimal policies [78] or the set of acceptable policies [46]. Several works have also considered formal task decompositions [14, 51] where simple addition of MDPs is insufficient [68]. More broadly, in machine learning, composition can be done via mixtures of experts and/or energy-based modeling [34, 21], which have also been applied to RL [29, 41]. Our results provide normative justification for linear scalarization when time preference is the same for all objectives, but call for non-Markovian adjustments when time preferences differ.
Non-Markovian Rewards The necessity of non-Markovian rewards was demonstrated in other settings by Abel et al. [1] and more recently, in a concurrent work by Skalse and Abate [69]. Though several papers explicitly consider RL with non-Markovian rewards [27, 60], this is usually motivated by task compression rather than necessity, and the majority of the RL literature restricts itself to Markovian models. Many popular exploration strategies implicitly use non-Markovian rewards [37, 53]. Our work is unique in that non-Markovian rewards arise from aggregating strictly Markovian quantities, rather non-Markovian quantities present in the task definition or algorithm.
The main contribution of this work is an impossibility result from which one concludes that non-Markovian rewards (or an equivalent state expansion) are likely necessary for agents that pursue multiple objectives or serve multiple principals. It's possible that this will be the case for any advanced agent whose actions impact multiple human stakeholders. To accurately align such agents with diverse human preferences we need to endow them with the capacity to solve problems requiring non-Markovian reward, for which this paper has proposed an efficient state space expansion that uses one new parameter per aggregated objective. While the proposed state space expansion allows multi-objective agents to have dynamically consistent preferences for future prospects, it does not, in itself, solve the intertemporal choice problem. To that end, we have proposed "historical discounting", a novel compromise between dynamic consistency and fair consideration of future generations.
Interesting avenues for future work include quantifying the inefficiency of Markovian representations, investigating normative approaches to aggregating preferences based on subjective world models (Remark 3.2.2), considering the existence of an "objective" time preference (Remark 3.3.2), improving methods for determining subjective time preference (e.g., [63]), implementing and comparing approaches to determining  $\gamma_{\Sigma}$  (Remark 4.1), investigating historical discounting in more detail (Subsection 5.2), and considering the existence of a normative justification for Boltzmann policies and their composition (Subsection 5.3).