SlowGuess/NeurIPS_2023_OCR huggingface dataset:
I thank Elliot Creager, for a fruitful discussion that prompted Subsection 5.1 and motivated me to turn this into a full paper; Duncan Bailey, who assisted with an earlier workshop version; the anonymous reviewers, who provided detailed reviews and suggestions that helped improve the final manuscript; and Jimmy Ba and the Ba group for early discussions. This work was supported by an NSERC CGS D Award and a Vector Research Grant.
A Notation Glossary
Table 1: Notation summary.
Basic notation
L(·)	the set of probability distributions over its argument
{·}	a set of its argument
M	a generalized MDP (S, A, T, T0, R, γ)
S, s	the state space S with generic state s ∈ S
A, a	the action space A with generic action a ∈ A
T	the transition function T: S × A → L(S)
R	the reward function R: S × A → R
γ	the generalized discount factor γ: S × A → R+
τ	a trajectory τ = [(s0, a0), (s1, a1), ...,], which may or may not terminate
h	a history (trajectory to up to time t)
π, ω	stationary policies π, ω: S → L(A)
Π, Ω, Λ	non-stationary policies (πt | τ0t, πt+1 | τ0t+1, ...)
Π	policy space, so π, ω, Π, Ω ∈ Π; note L(Π) = Π
P	the space P = S × Π of prospects over which preferences are expressed; note L(P) = L(S) × Π
Ph	the set of historical prospects starting with h; NB: (h2, Π)∉Ph1 for h1 ≠ h2.
P(y)	the set of prospects on augmented state space S ∪ {yi}
(s, Π) ∈ P	a prospect (stochastic process representing the controlled future)
(h, Π)	a historical prospect (state space has been expanded to include h)
hs, sa, sΠ, aΠ, ...	concatenations of subcomponents of trajectories/histories/policies; for example, the non-stationary policy sΠ := (a, Π0, Π1, ...)
V	value function V: L(S) × Π → R, which is equal to the discounted sum of future rewards: V(s, Π) = E_{s0~sΠ} \{ ∑_{t=0}^∞ [Π_{k=1}^t γ(st-1, at-1)]r(st, at) \}
supp(p)	the support of distribution p
I	index set I of individuals (assumed finite)
wi, wi(s), wi(h)	theith aggregation weight, either a constant, or a function of state or history
yi(h)	theith aggregation factor for history h
dπ	the stationary state-action distribution for policy π
Generic modifiers
·i	index i of a sequence, vector, or collection of agents
·j	the slice from i to j of a sequence or vector; e.g., τt+1t is the trajectory slice [(st, at), ..., (st+k, at+k)]
·ε	indicates an aggregated item (e.g., >ε is social/aggregated preference)
·′	indicates next timestep when time implicit (e.g., s, s′)
~	indicates a probability distribution (e.g., ~s ∈ L(S))
γh	indicates a preference relation on Ph
Operators
:= (=:)	defined as (is the definition of)
>	strict preference
<	weak preference (p ≥ q ↔ q ≠ p)
≈	indifference (p ≈ q and q ≠ p)
Proof. Fix  $s, a$ . Using Theorem 2, choose  $\{r_i, \gamma_i\}, r_\Sigma, \gamma_\Sigma$  to represent individual and aggregate preferences over  $\mathcal{L}(\mathcal{P})$ . Define mixture policy  $\Pi_{\beta} \coloneqq \beta \Pi + (1 - \beta) \Omega$ . W.l.o.g. assume  $|\mathcal{I}| = 2$ .
We use Axiom 4 (Unrestricted Domain) to set
$$
(T (s, a), \Pi) \succ_ {1} (T (s, a), \Lambda) \succ_ {1} (T (s, a), \Omega),
$$
$$
\begin{array}{l} (T (s, a), \Omega) \succ_ {2} (T (s, a), \Lambda) \succ_ {2} (T (s, a), \Pi), \end{array} \tag {7}
$$
and, using Theorem 1 to shift  $V_{1}, V_{2}$ , we have
$$
V _ {1} (T (s, a), \Lambda) = V _ {2} (T (s, a), \Lambda) = 0,
$$
$$
V _ {1} (T (s, a), \Pi) > 0 > V _ {1} (T (s, a), \Omega) \quad \text {s . t .} \quad V _ {1} (T (s, a), \Pi_ {\beta_ {1}}) = 0, \tag {8}
$$
$$
V _ {2} (T (s, a), \Omega) > 0 > V _ {2} (T (s, a), \Pi) \quad \text {s . t .} \quad V _ {2} (T (s, a), \Pi_ {\beta_ {2}}) = 0,
$$
for some  $\beta_{1},\beta_{2}\in (0,1)$  with  $\beta_{1}\neq \beta_{2}$  (again appealing to Unrestricted Domain). Intuitively, equation 7 requires individual preferences to conflict, and the  $\beta_{1}\neq \beta_{2}$  condition requires them to be non-symmetric about  $\Lambda$ . Equation 8 is merely a convenient choice of numerical representation.
We now apply Theorem 3 followed by Theorem 2 to the expression  $V_{\Sigma}(s, a\Pi_{\beta}) - V_{\Sigma}(s, a\Lambda)$ . We again invoke Theorem 1 to shift  $V_{\Sigma}$  and eliminate the constant term, so that by Theorem 3  $\exists \{w_i\}$  for which,
$$
\begin{array}{l} V _ {\Sigma} (s, a \Pi_ {\beta}) - V _ {\Sigma} (s, a \Lambda) = \sum_ {i \in \mathcal {I}} w _ {i} V _ {i} (s, a \Pi_ {\beta}) - \sum_ {i \in \mathcal {I}} w _ {i} V _ {i} (s, a \Lambda) \\ = w _ {1} \gamma_ {1} (s, a) \left[ V _ {1} (T (s, a), \Pi_ {\beta}) - V _ {1} (T (s, a), \Lambda) \right] + \tag {9} \\ w _ {2} \gamma_ {2} (s, a) [ V _ {2} (T (s, a), \Pi_ {\beta}) - V _ {2} (T (s, a), \Lambda) ] \\ = w _ {1} \gamma_ {1} (s, a) V _ {1} (T (s, a), \Pi_ {\beta}) + w _ {2} \gamma_ {2} (s, a) V _ {2} (T (s, a), \Pi_ {\beta}). \\ \end{array}
$$
where the second line applies Theorem 2, with rewards cancelling out.
Alternatively, applying Theorem 2 followed by Theorem 3 to the same expression yields,
$$
\begin{array}{l} V _ {\Sigma} (s, a \Pi_ {\beta}) - V _ {\Sigma} (s, a \Lambda) = \gamma_ {\Sigma} (s, a) V _ {\Sigma} (T (s, a), \Pi_ {\beta}) - \gamma_ {\Sigma} (s, a) V _ {\Sigma} (T (s, a), \Lambda) \\ = w _ {1} \gamma_ {\Sigma} (s, a) \left[ V _ {1} (T (s, a), \Pi_ {\beta}) - V _ {1} (T (s, a), \Lambda) \right] + w _ {2} \gamma_ {\Sigma} (s, a) \left[ V _ {2} (T (s, a), \Pi_ {\beta}) - V _ {2} (T (s, a), \Lambda) \right] \tag {10} \\ = w _ {1} \gamma_ {\Sigma} (s, a) V _ {1} (T (s, a), \Pi_ {\beta}) + w _ {2} \gamma_ {\Sigma} (s, a) V _ {2} (T (s, a), \Pi_ {\beta}) \\ \end{array}
$$
Combining equations 9 and 10, we obtain:
$$
\begin{array}{l} w _ {1} \gamma_ {1} (s, a) V _ {1} (T (s, a), \Pi_ {\beta}) + w _ {2} \gamma_ {2} (s, a) V _ {2} (T (s, a), \Pi_ {\beta}) \\ \begin{array}{l} w _ {1} \gamma_ {1} (s, a) V _ {1} (T (s, a), \Pi_ {\beta}) + w _ {2} \gamma_ {2} (s, a) V _ {2} (T (s, a), \Pi_ {\beta}) \\ = w _ {1} \gamma_ {\Sigma} (s, a) V _ {1} (T (s, a), \Pi_ {\beta}) + w _ {2} \gamma_ {\Sigma} (s, a) V _ {2} (T (s, a), \Pi_ {\beta}). \end{array} \tag {11} \\ \end{array}
$$
Finally, setting  $\beta$  to be  $\beta_{1}$  or  $\beta_{2}$  in equation 11, we obtain the equalities:
$$
w _ {1} \gamma_ {1} (s, a) V _ {1} (T (s, a), \Pi_ {\beta_ {2}}) = w _ {1} \gamma_ {\Sigma} (s, a) V _ {1} (T (s, a), \Pi_ {\beta_ {2}}),
$$
$$
w _ {2} \gamma_ {2} (s, a) V _ {2} \left(T (s, a), \Pi_ {\beta_ {1}}\right) = w _ {2} \gamma_ {\Sigma} (s, a) V _ {2} \left(T (s, a), \Pi_ {\beta_ {1}}\right). \tag {12}
$$
The  $w_{i}$  are non-zero (Axiom 4, Sensitivity) and the remaining values are non-zero (else  $\beta_{1} = \beta_{2}$ ), so we conclude that  $\gamma_{\Sigma}(s,a) = \gamma_1(s,a) = \gamma_2(s,a)$ . But this contradicts our assumption that individual preferences  $\{\succ_{i}; i \in \mathcal{I}\}$  may be chosen arbitrarily, completing the proof.
$$
w _ {i} \left(s a s ^ {\prime}\right) := w _ {i} (s a) := w _ {i} (s) \gamma_ {i} (s, a) \quad f o r a l l i, s ^ {\prime} \tag {13}
$$
Proof. We follow the proof of Theorem 4. Fix  $s, a$ . Using Theorem 2, choose  $\{r_i, \gamma_i\}, r_\Sigma, \gamma_\Sigma$  to represent individual and aggregate preferences over  $\mathcal{L}(\mathcal{P}_s)$  and  $\mathcal{L}(\mathcal{P}_{sas'})$ . Define mixture policy  $\Pi_{\beta} := \beta \Pi + (1 - \beta) \Omega$ . W.l.o.g. assume  $|\mathcal{I}| = 2$ .
$$
(T (s, a), \Omega) \succ_ {2} (T (s, a), \Lambda) \succ_ {2} (T (s, a), \Pi), \tag {14}
$$
$$
V _ {1} (T (s, a), \Pi) > 0 > V _ {1} (T (s, a), \Omega) \quad \text {s . t .} \quad V _ {1} (T (s, a), \Pi_ {\beta_ {1}}) = 0, \tag {15}
$$
for some  $\beta_{1},\beta_{2}\in (0,1)$  with  $\beta_{1}\neq \beta_{2}$  (again appealing to Unrestricted Domain).
We now apply Theorem 3 followed by Theorem 2 to the expression  $V_{\Sigma}(s, a\Pi_{\beta}) - V_{\Sigma}(s, a\Lambda)$ . We again invoke Theorem 1 to shift  $V_{\Sigma}$  and eliminate the constant term, so that by Theorem 3  $\exists \{w_i(s)\}$  for which,
$$
\begin{array}{l} V _ {\Sigma} (s, a \Pi_ {\beta}) - V _ {\Sigma} (s, a \Lambda) = \sum_ {i \in \mathcal {I}} w _ {i} (s) V _ {i} (s, a \Pi_ {\beta}) - \sum_ {i \in \mathcal {I}} w _ {i (s)} V _ {i} (s, a \Lambda) \\ = w _ {1} (s) \gamma_ {1} (s, a) \left[ V _ {1} (T (s, a), \Pi_ {\beta}) - V _ {1} (T (s, a), \Lambda) \right] + \tag {16} \\ w _ {2} (s) \gamma_ {2} (s, a) \left[ V _ {2} (T (s, a), \Pi_ {\beta}) - V _ {2} (T (s, a), \Lambda) \right] \\ = w _ {1} (s) \gamma_ {1} (s, a) V _ {1} (T (s, a), \Pi_ {\beta}) + w _ {2} (s) \gamma_ {2} (s, a) V _ {2} (T (s, a), \Pi_ {\beta}). \\ \end{array}
$$
It suffices to find one set of satisfactory  $w_{i}(h)$ , so we can assume that, given  $s, a$ ,  $w_{i}(s a^{\prime}) := w(s a)$  is the same for all  $s^{\prime}$ . This will allow us to factor it out below. Then, applying Theorem 2 followed by Theorem 3 to the same expression yields,
$$
\begin{array}{l} V _ {\Sigma} (s, a \Pi_ {\beta}) - V _ {\Sigma} (s, a \Lambda) = \gamma_ {\Sigma} (s, a) V _ {\Sigma} (T (s, a), \Pi_ {\beta}) - \gamma_ {\Sigma} (s, a) V _ {\Sigma} (T (s, a), \Lambda) \\ = w _ {1} (s a) \gamma_ {\Sigma} (s, a) \left[ V _ {1} (T (s, a), \Pi_ {\beta}) - V _ {1} (T (s, a), \Lambda) \right] + \\ w _ {2} (s a) \gamma_ {\Sigma} (s, a) \left[ V _ {2} (T (s, a), \Pi_ {\beta}) - V _ {2} (T (s, a), \Lambda) \right] \tag {17} \\ = w _ {1} (s a) \gamma_ {\Sigma} (s, a) V _ {1} (T (s, a), \Pi_ {\beta}) + w _ {2} (s a) \gamma_ {\Sigma} (s, a) V _ {2} (T (s, a), \Pi_ {\beta}) \\ \end{array}
$$
Combining equations 16 and 17, setting  $\beta$  to be  $\beta_{1}$  or  $\beta_{2}$  in equation, and rearranging, we obtain the equalities:
$$
w _ {1} (s) \gamma_ {1} (s, a) = w _ {1} (s a) \gamma_ {\Sigma} (s, a) \quad \text {a n d} \quad w _ {2} (s) \gamma_ {2} (s, a) = w _ {2} (s a) \gamma_ {\Sigma} (s, a), \tag {18}
$$
$$
\frac {w _ {2} (s a)}{w _ {1} (s a)} = \frac {w _ {2} (s) \gamma_ {2} (s , a)}{w _ {1} (s) \gamma_ {1} (s , a)} \tag {19}
$$