SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Han Shao
han@ttic.edu
Lee Cohen
leecohencs@gmail.com
Ayrim Blum
avrim@ttic.edu
Yishay Mansour
Tel Aviv University and Google Research
mansour.yishay@gmail.com
Aadirupa Saha
aadirupa@ttic.edu
Matthew R. Walter
mwalter@ttic.edu
In this work, we propose a multi-objective decision making framework that accommodates different user preferences over objectives, where preferences are learned via policy comparisons. Our model consists of a known Markov decision process with a vector-valued reward function, with each user having an unknown preference vector that expresses the relative importance of each objective. The goal is to efficiently compute a near-optimal policy for a given user. We consider two user feedback models. We first address the case where a user is provided with two policies and returns their preferred policy as feedback. We then move to a different user feedback model, where a user is instead provided with two small weighted sets of representative trajectories and selects the preferred one. In both cases, we suggest an algorithm that finds a nearly optimal policy for the user using a number of comparison queries that scales quasilinearly in the number of objectives.
Many real-world decision making problems involve optimizing over multiple objectives. For example, when designing an investment portfolio, one's investment strategy requires trading off maximizing expected gain with minimizing risk. When using Google Maps for navigation, people are concerned about various factors such as the worst and average estimated arrival time, traffic conditions, road surface conditions (e.g., whether or not it is paved), and the scenery along the way. As the advancement of technology gives rise to personalized machine learning (McAuley, 2022), in this paper, we design efficient algorithms for personalized multi-objective decision making.
While prior works have concentrated on approximating the Pareto-optimal solution set<sup>1</sup> (see Hayes et al. (2022) and Roijers et al. (2013) for surveys), we aim to find the optimal personalized policy for a user that reflects their unknown preferences over  $k$  objectives. Since the preferences are unknown, we need to elicit users' preferences by requesting feedback on selected policies. The problem of eliciting preferences has been studied in Wang et al. (2022) using a strong query model that provides stochastic feedback on the quality of a single policy. In contrast, our work focuses on a more natural and intuitive query model, comparison queries, which query the user's preference over two selected policies, e.g., 'do you prefer a policy which minimizes average estimated arrival time or policy which minimizes the number of turns?' Our goal is to find the optimal personalized policy using as few queries as possible.
<sup>1</sup>The Pareto-optimal solution set contains a optimal personalized policy for every possible user preference.
To the best of our knowledge, we are the first to provide algorithms with theoretical guarantees for specific personalized multi-objective decision-making via policy comparisons.
Similar to prior works on multi-objective decision making, we model the problem using a finite-horizon Markov decision process (MDP) with a  $k$ -dimensional reward vector, where each entry is a non-negative scalar reward representative of one of the  $k$  objectives. To account for user preferences, we assume that a user is characterized by a (hidden)  $k$ -dimensional preference vector with non-negative entries, and that the personalized reward of the user for each state-action is the inner product between this preference vector and the reward vector (for this state-action pair). We also distinguish between the  $k$ -dimensional value of a policy, which is the expected cumulative reward when selecting actions according to the policy, and the personalized value of a policy, which is the scalar expected cumulative personalized reward when selecting actions according to this policy. The MDP is known to the agent and the goal is to learn an optimal policy for the personalized reward function (henceforth, the optimal personalized policy) of a user via policy comparative feedback.
Comparative feedback. If people could clearly define their preferences over objectives (e.g., "my preference vector has 3 for the scenery objective, 2 for the traffic objective, and 1 for the road surface objective"), the problem would be easy—one would simply use the personalized reward function as a scalar reward function and solve for the corresponding policy. In particular, a similar problem with noisy feedback regarding the value of a single multi-objective policy (as mapping from states to actions) has been studied in Wang et al. (2022). As this type of fine-grained preference feedback is difficult for users to define, especially in environments where sequential decisions are made, we restrict the agent to rely solely on comparative feedback. Comparative feedback is widely used in practice. For example, ChatGPT asks users to compare two responses to improve its performance. This approach is more intuitive for users compared to asking for numerical scores of ChatGPT responses.
Indeed, as knowledge of the user's preference vector is sufficient to solve for their optimal personalized policy, the challenge is to learn a user's preference vector using a minimal number of easily interpretable queries. We therefore concentrate on comparison queries. The question is then what exactly to compare? Comparing state-action pairs might not be a good option for the aforementioned tasks—what is the meaning of two single steps in a route? Comparing single trajectories (e.g., routes in Google Maps) would not be ideal either. Consider for example two policies: one randomly generates either (personalized) GOOD or (personalized) BAD trajectories while the other consistently generates (personalized) MEDIOCRE trajectories. By solely comparing single trajectories without considering sets of trajectories, we cannot discern the user's preference regarding the two policies.
Interpretable policy representation. Since we are interested in learning preferences via policy comparison queries, we also suggest an alternative, more interpretable representation of a policy. Namely, we design an algorithm that given an explicit policy representation (as a mapping from states to distributions over actions), returns a weighted set of trajectories of size at most  $k + 1$ , such that its expected return is identical to the value of the policy. It immediately follows from our formalization that for any user, the personalized return of the weighted trajectory set and the personalized value of the policy are also identical.
In this work, we focus on answering two questions:
(1) How to find the optimal personalized policy by querying as few policy comparisons as possible? (2) How can we find a more interpretable representation of policies efficiently?
Contributions. In Section 2, we formalize the problem of eliciting user preferences and finding the optimal personalized policy via comparative feedback. As an alternative to an explicit policy representation, we propose a weighted trajectory set as a more interpretable representation. In Section 3, we provide an efficient algorithm for finding an approximate optimal personalized policy, where the policies are given by their formal representations, thus answering (1). In Section 4, we design two efficient algorithms that find the weighted trajectory set representation of a policy. Combined with the algorithm in Section 3, we have an algorithm for finding an approximate optimal personalized policy when policies are represented by weighted trajectory sets, thus answering (2).
Related Work. Multi-objective decision making has gained significant attention in recent years (see Roijers et al. (2013); Hayes et al. (2022) for surveys). Prior research has explored various approaches,
${}^{2}$  A return of a trajectory is the cumulative reward obtained in the trajectory. The expectation in the expected return of a weighted trajectory set is over the weights.
such as assuming linear preferences or Bayesian Settings, or finding an approximated Pareto frontier. However, incorporating comparison feedback (as was done for Multi-arm bandits or active learning, e.g., Bengs et al. (2021) and Kane et al. (2017)) allows us a more comprehensive and systematic approach to handling different types of user preferences and provides (nearly) optimal personalized decision-making outcomes. We refer the reader to Appendix A for additional related work.
Sequential decision model. We consider a Markov decision process (MDP) known<sup>3</sup> to the agent represented by a tuple  $\langle S, \mathcal{A}, s_0, P, R, H \rangle$ , with finite state and action sets,  $S$  and  $\mathcal{A}$ , respectively, an initial state  $s_0 \in S$ , and finite horizon  $H \in \mathbb{N}$ . For example, in the Google Maps example a state is an intersection and actions are turns. The transition function  $P: S \times \mathcal{A} \mapsto \mathrm{Simplex}^S$  maps state-action pairs into a state probability distribution. To model multiple objectives, the reward function  $R: S \times \mathcal{A} \mapsto [0,1]^k$  maps every state-action pair to a  $k$ -dimensional reward vector, where each component corresponds to one of the  $k$  objectives (e.g., road surface condition, worst and average estimated arrival time). The return of a trajectory  $\tau = (s_0, a_0, \dots, s_{H-1}, a_{H-1}, s_H)$  is given by  $\Phi(\tau) = \sum_{t=0}^{H-1} R(s_t, a_t)$ .
A policy  $\pi$  is a mapping from states to a distribution over actions. We denote the set of policies by  $\Pi$ . The value of a policy  $\pi$ , denoted by  $V^{\pi}$ , is the expected cumulative reward obtained by executing the policy  $\pi$  starting from the initial state,  $s_0$ . Put differently, the value of  $\pi$  is  $V^{\pi} = V^{\pi}(s_0) = \mathbb{E}_{S_0 = s_0}\left[\sum_{t=0}^{H-1} R(S_t, \pi(S_t))\right] \in [0, H]^k$ , where  $S_t$  is the random state at time step  $t$  when executing  $\pi$ , and the expectation is over the randomness of  $P$  and  $\pi$ . Note that  $V^{\pi} = \mathbb{E}_{\tau}[\Phi(\tau)]$ , where  $\tau = (s_0, \pi(s_0), S_1, \dots, \pi(S_{H-1}), S_H)$  is a random trajectory generated by executing  $\pi$ .
We assume the existence of a "do nothing" action  $a_0 \in \mathcal{A}$ , available only from the initial state  $s_0$ , that has zero reward for each objective  $R(s_0, a_0) = 0$  and keeps the system in the initial state, i.e.,  $P(s_0 \mid s_0, a_0) = 1$  (e.g., this action corresponds to not commuting or refusing to play in a chess game.)<sup>4</sup> We also define the (deterministic) "do nothing" policy  $\pi_0$  that always selects action  $a_0$  and has a value of  $V^{\pi_0} = 0$ . From a mathematical perspective, the assumption of "do nothing" ensures that  $\mathbf{0}$  belongs to the value vector space  $\{V^{\pi} | \pi \in \Pi\}$ , which is precisely what we need.
Since the rewards are bounded between  $[0,1]$ , we have that  $1\leq \| V^{\pi}\|_{2}\leq \sqrt{k} H$  for every policy  $\pi$ . For convenience, we denote  $C_V = \sqrt{k} H$ . We denote by  $d\leq k$  the rank of the space spanned by all the value vectors obtained by  $\Pi$ , i.e.,  $d\coloneqq \mathrm{rank}(\mathrm{span}(\{V^{\pi}|\pi \in \Pi \}))$ .
Linear preferences. To incorporate personalized preferences over objectives, we assume each user is characterized by an unknown  $k$ -dimensional preference vector  $w^{*} \in \mathbb{R}_{+}^{k}$  with a bounded norm  $1 \leq \| w^{*} \|_{2} \leq C_{w}$  for some (unknown)  $C_{w} \geq 1$ . We avoid assuming that  $C_{w} = 1$  to accommodate for general linear rewards. Note that the magnitude of  $w^{*}$  does not change the personalized optimal policy but affects the "indistinguishability" in the feedback model. By not normalizing  $\| w^{*} \|_{2} = 1$ , we allow users to have varying levels of discernment. This preference vector encodes preferences over the multiple objectives and as a result, determines the user preferences over policies.
Formally, for a user characterized by  $w^{*}$ , the personalized value of policy  $\pi$  is  $\langle w^{*}, V^{\pi} \rangle \in \mathbb{R}_{+}$ . We denote by  $\pi^{*} \coloneqq \arg \max_{\pi \in \Pi} \langle w^{*}, V^{\pi} \rangle$  and  $v^{*} \coloneqq \langle w^{*}, V^{\pi^{*}} \rangle$  the optimal personalized policy and its corresponding optimal personalized value for a user who is characterized by  $w^{*}$ . We remark that the "do nothing" policy  $\pi_0$  (that always selects action  $a_0$ ) has a value of  $V^{\pi_0} = 0$ , which implies a personalized value of  $\langle w^{*}, V^{\pi_0} \rangle = 0$  for every  $w^{*}$ . For any two policies  $\pi_1$  and  $\pi_2$ , the user characterized by  $w^{*}$  prefers  $\pi_1$  over  $\pi_2$  if  $\langle w^{*}, V^{\pi_1} - V^{\pi_2} \rangle > 0$ . Our goal is to find the optimal personalized policy for a given user using as few interactions with them as possible.
Comparative feedback. Given two policies  $\pi_1, \pi_2$ , the user returns  $\pi_1 \succ \pi_2$  whenever  $\langle w^*, V^{\pi_1} - V^{\pi_2} \rangle > \epsilon$ ; otherwise, the user returns "indistinguishable" (i.e., whenever  $|\langle w^*, V^{\pi_1} - V^{\pi_2} \rangle| \leq \epsilon$ ). Here  $\epsilon > 0$  measures the precision of the comparative feedback and
<sup>3</sup>If the MDP is unknown, one can approximate the MDP and apply the results of this work in the approximated MDP. More discussion is included in Appendix 5.
<sup>4</sup>We remark that the "do nothing" action require domain-specific knowledge, for example in the Google maps example the "do nothing" will be to stay put and in the ChatGPT example the "do nothing" is to answer nothing.
is small usually. The agent can query the user about their policy preferences using two different types of policy representations:
1. Explicit policy representation of  $\pi$ : An explicit representation of policy as mapping,  $\pi : S \to \text{Simplex}^A$ .
2. Weighted trajectory set representation of  $\pi$ : A  $\kappa$ -sized set of trajectory-weight pairs  $\{(p_i, \tau_i)\}_{i=1}^{\kappa}$  for some  $\kappa \leq k+1$  such that (i) the weights  $p_1, \ldots, p_\kappa$  are non-negative and sum to 1; (ii) every trajectory in the set is in the support<sup>5</sup> of the policy  $\pi$ ; and (iii) the expected return of these trajectories according to the weights is identical to the value of  $\pi$ , i.e.,  $V^\pi = \sum_{i=1}^{\kappa} p_i \Phi(\tau_i)$ . Such comparison could be practical for humans. E.g., in the context of Google Maps, when the goal is to get from home to the airport, taking a specific route takes 40 minutes  $90\%$  of the time, but it can take 3 hours in case of an accident (which happens w.p.  $10\%$ ) vs. taking the subway which always has a duration of 1 hour.
In both cases, the feedback is identical and depends on the hidden precision parameter  $\epsilon$ . As a result, the value of  $\epsilon$  will affect the number of queries and how close the value of the resulting personalized policy is to the optimal personalized value. Alternatively, we can let the agent decide in advance on a maximal number of queries, which will affect the optimality of the returned policy.
Technical Challenges. In Section 3, we find an approximate optimal policy by  $\mathcal{O}(\log \frac{1}{\epsilon})$  queries. To achieve this, we approach the problem in two steps. Firstly, we identify a set of linearly independent policy values, and then we estimate the preference vector  $w^{*}$  using a linear program that incorporates comparative feedback. The estimation error of  $w^{*}$  usually depends on the condition number of the linear program. Therefore, the main challenge we face is how to search for linear independent policy values that lead to a small condition number and providing a guarantee for this estimate.
In Section 4, we focus on how to design efficient algorithms to find the weighted trajectory set representation of a policy. Initially, we employ the well-known Carathéodory's theorem, which yields an inefficient algorithm with a potentially exponential running time of  $|S|^{H}$ . Our main challenge lies in developing an efficient algorithm with a running time of  $\mathcal{O}(\mathrm{poly}(H|S||\mathcal{A}|))$ . The approach based on Carathéodory's theorem treats the return of trajectories as independent  $k$ -dimensional vectors, neglecting the fact that they are all generated from the same MDP. To overcome this challenge, we leverage the inherent structure of MDPs.
3 Learning from Explicit Policies
In this section, we consider the case where the interaction with a user is based on explicit policy comparison queries. We design an algorithm that outputs a policy being nearly optimal for this user. For multiple different users, we only need to run part of the algorithm again and again. For brevity, we relegate all proofs to the appendix.
If the user's preference vector  $w^{*}$  (up to a positive scaling) is given, then one can compute the optimal policy and its personalized value efficiently, e.g., using the Finite Horizon Value Iteration algorithm. In our work,  $w^{*}$  is unknown and we interact with the user to learn  $w^{*}$  through comparative feedback. Due to the structure model that is limited to meaningful feedback only when the compared policy values differ at least  $\epsilon$ , the exact value of  $w^{*}$  cannot be recovered. We proceed by providing a high-level description of our ideas of how to estimate  $w^{*}$ .
(1) Basis policies: We find policies  $\pi_1, \ldots, \pi_d$ , and their respective values,  $V^{\pi_1}, \ldots, V^{\pi_d} \in [0, H]^k$ , such that their values are linearly independent and that together they span the entire space of value vectors. These policies will not necessarily be personalized optimal for the current user, and instead serve only as building blocks to estimate the preference vector,  $w^*$ . In Section 3.1 we describe an algorithm that finds a set of basis policies for any given MDP.
(2) Basis ratios: For the basis policies, denote by  $\alpha_{i} > 0$  the ratio between the personalized value of a benchmark policy,  $\pi_1$ , to the personalized value of  $\pi_{i + 1}$ , i.e.,
$$
\forall i \in [ d - 1 ]: \alpha_ {i} \left\langle w ^ {*}, V ^ {\pi_ {1}} \right\rangle = \left\langle w ^ {*}, V ^ {\pi_ {i + 1}} \right\rangle . \tag {1}
$$
<sup>5</sup>We require the trajectories in this set to be in the support of the policy, to avoid trajectories that do not make sense, such as trajectories that "teleport" between different unconnected states (e.g., commuting at 3 mph in Manhattan in one state and then at 40 mph in New Orleans for the subsequent state).
Recall that  $d \leq k$  is the rank of the space spanned by all the value vectors obtained by all policies.
We will estimate  $\widehat{\alpha}_i$  of  $\alpha_{i}$  for all  $i\in [d - 1]$  using comparison queries. A detailed algorithm for estimating these ratios appears in Section 3.2. For intuition, if we obtain exact ratios  $\widehat{\alpha}_i = \alpha_i$  for every  $i\in [d - 1]$ , then we can compute the vector  $\frac{w^{*}}{\|w^{*}\|_1}$  as follows. Consider the  $d - 1$  equations and  $d - 1$  variables in Eq (1). Since  $d$  is the maximum number of value vectors that are linearly independent, and  $V^{\pi_1},\ldots V^{\pi_d}$  form a basis, adding the equation  $\| w\| _1 = 1$  yields  $d$  independent equations with  $d$  variables, which allows us to solve for  $w^{*}$ . The details of computing an estimate of  $w^{*}$  are described in Section 3.3.
3.1 Finding a Basis of Policies
In order to efficiently find  $d$  policies with  $d$  linearly independent value vectors that span the space of value vectors, one might think that selecting the  $k$  policies that each optimizes one of the  $k$  objectives will suffice. However, this might fail—in Appendix N, we show an instance in which these  $k$  value vectors are linearly dependent even though there exist  $k$  policies whose values span a space of rank  $k$ .
Moreover, our goal is to find not just any basis of policies, but a basis of policies such that (1) the personalized value of the benchmark policy  $\langle w^{*}, V^{\pi_{1}} \rangle$  will be large (and hence the estimation error of ratio  $\alpha_{i}$ ,  $|\widehat{\alpha}_{i} - \alpha_{i}|$ , will be small), and (2) that the linear program generated by this basis of policies and the basis ratios will produce a good estimate of  $w^{*}$ .
Choice of  $\pi_1$ . Besides linear independence of values, another challenge is to find a basis of policies to contain a benchmark policy,  $\pi_1$  (where the index 1 is wlog) with a relatively large personalized value,  $\langle w^*, V^{\pi_1} \rangle$ , so that  $\hat{\alpha}_i$ 's error is small (e.g., in the extreme case where  $\langle w^*, V^{\pi_1} \rangle = 0$ , we will not be able to estimate  $\alpha_i$ ).
For any  $w \in \mathbb{R}^k$ , we use  $\pi^w$  denote a policy that maximizes the scalar reward  $\langle w, R \rangle$ , i.e.,
$$
\pi^ {w} = \underset {\pi \in \Pi} {\arg \max } \left\langle w, V ^ {\pi} \right\rangle , \tag {2}
$$
and by  $v^w = \langle w, V^{\pi^w} \rangle$  to denote the corresponding personalized value. Let  $\mathbf{e}_1, \ldots, \mathbf{e}_k$  denote the standard basis. To find  $\pi_1$  with large personalized value  $\langle w^*, V^{\pi_1} \rangle$ , we find policies  $\pi^{\mathbf{e}_j}$  that maximize the  $j$ -th objective for every  $j = 1, \ldots, k$  and then query the user to compare them until we find a  $\pi^{\mathbf{e}_j}$  with (approximately) a maximal personalized value among them. This policy will be our benchmark policy,  $\pi_1$ . The details are described in lines 1-6 of Algorithm 1.
Algorithm 1 Identification of Basis Policies
1: initialize  $\pi^{\mathbf{e}^*}\gets \pi^{\mathbf{e}_1}$   
2: for  $j = 2,\dots ,k$  do  
3: compare  $\pi^{\mathbf{e}^*}$  and  $\pi^{\mathbf{e}_j}$   
4: if  $\pi^{\mathbf{e}_j}\succ \pi^{\mathbf{e}^*}$  then  $\pi^{\mathbf{e}^*}\leftarrow \pi^{\mathbf{e}_j}$   
5: end for  
6:  $\pi_1\gets \pi^{\mathbf{e}^*}$  and  $u_{1}\gets \frac{V^{\pi^{\mathbf{e}^{*}}}}{\|V^{\pi^{\mathbf{e}^{*}}}\|_{2}}$   
7: for  $i = 2,\ldots ,k$  do  
8: arbitrarily pick an orthonormal basis,  $\rho_{1},\ldots ,\rho_{k + 1 - i}$  for  $\mathrm{span}(V^{\pi_1},\ldots ,V^{\pi_{i - 1}})^\perp$   
9:  $j_{\mathrm{max}}\gets$  arg max  $j\in [k + 1 - i]$  max  $(|v^{\rho_j}|,|v^{-\rho_j}|)$   
10: if max  $(|v^{\rho_{j\mathrm{max}}}|,|v^{-\rho_{j\mathrm{max}}}|) > 0$  then  
11:  $\pi_i\gets \pi^{\rho_{j\mathrm{max}}}$  if  $|v^{\rho_{j\mathrm{max}}}| > |v^{-\rho_{j\mathrm{max}}}|$ ; otherwise  $\pi_i\gets \pi^{-\rho_{j\mathrm{max}}}$ .  $u_i\gets \rho_{j\mathrm{max}}$   
12: else  
13: return  $(\pi_1,\pi_2,\ldots),(u_1,u_2,\ldots)$   
14: end if  
15: end for