Title: Eliciting User Preferences for Personalized Multi-Objective Decision Making through Comparative Feedback
Venue: NeurIPS 2023
Accepted: yes
Keywords: preference learning

algorithms

linear model

Markov decision processes

learning theory

multi-objective decision making

preference elicitation
Abstract: In this work, we propose a multi-objective decision making framework that accommodates different user preferences over objectives, where preferences are learned via policy comparisons. Our model consists of a known Markov decision process with a vector-valued reward function, with each user having an unknown preference vector that expresses the relative importance of each objective. The goal is to efficiently compute a near-optimal policy for a given user. We consider two user feedback models. We first address the case where a user is provided with two policies and returns their preferred policy as feedback. We then move to a different user feedback model, where a user is instead provided with two small weighted sets of representative trajectories and selects the preferred one. In both cases, we suggest an algorithm that finds a nearly optimal policy for the user using a number of comparison queries that scales quasilinearly in the number of objectives.

Eliciting User Preferences for Personalized Multi-Objective Decision Making through Comparative Feedback

Abstract

In this work, we propose a multi-objective decision making framework that accommodates different user preferences over objectives, where preferences are learned via policy comparisons. Our model consists of a known Markov decision process with a vector-valued reward function, with each user having an unknown preference vector that expresses the relative importance of each objective. The goal is to efficiently compute a near-optimal policy for a given user. We consider two user feedback models. We first address the case where a user is provided with two policies and returns their preferred policy as feedback. We then move to a different user feedback model, where a user is instead provided with two small weighted sets of representative trajectories and selects the preferred one. In both cases, we suggest an algorithm that finds a nearly optimal policy for the user using a number of comparison queries that scales quasilinearly in the number of objectives.

1 Introduction

Many real-world decision making problems involve optimizing over multiple objectives. For example, when designing an investment portfolio, one's investment strategy requires trading off maximizing expected gain with minimizing risk. When using Google Maps for navigation, people are concerned about various factors such as the worst and average estimated arrival time, traffic conditions, road surface conditions (e.g., whether or not it is paved), and the scenery along the way. As the advancement of technology gives rise to personalized machine learning (McAuley, 2022), in this paper, we design efficient algorithms for personalized multi-objective decision making.

While prior works have concentrated on approximating the Pareto-optimal solution set 1 (see Hayes et al. (2022) and Roijers et al. (2013) for surveys), we aim to find the optimal personalized policy for a user that reflects their unknown preferences over k objectives. Since the preferences are unknown, we need to elicit users' preferences by requesting feedback on selected policies. The problem of eliciting preferences has been studied in Wang et al. (2022) using a strong query model that provides stochastic feedback on the quality of a single policy. In contrast, our work focuses on a more natural and intuitive query model, comparison queries, which query the user's preference over two selected policies, e.g., 'do you prefer a policy which minimizes average estimated arrival time or policy which minimizes the number of turns?'. Our goal is to find the optimal personalized policy using as few queries as possible.

To the best of our knowledge, we are the first to provide algorithms with theoretical guarantees for specific personalized multi-objective decision-making via policy comparisons.

Similar to prior works on multi-objective decision making, we model the problem using a finitehorizon Markov decision process (MDP) with a k -dimensional reward vector, where each entry is a non-negative scalar reward representative of one of the k objectives. To account for user preferences, we assume that a user is characterized by a (hidden) k -dimensional preference vector with nonnegative entries, and that the personalized reward of the user for each state-action is the inner product between this preference vector and the reward vector (for this state-action pair). We also distinguish between the k -dimensional value of a policy, which is the expected cumulative reward when selecting actions according to the policy, and the personalized value of a policy, which is the scalar expected cumulative personalized reward when selecting actions according to this policy. The MDP is known to the agent and the goal is to learn an optimal policy for the personalized reward function (henceforth, the optimal personalized policy ) of a user via policy comparative feedback.

Comparative feedback. If people could clearly define their preferences over objectives (e.g., 'my preference vector has 3 for the scenery objective, 2 for the traffic objective, and 1 for the road surface objective'), the problem would be easy-one would simply use the personalized reward function as a scalar reward function and solve for the corresponding policy. In particular, a similar problem with noisy feedback regarding the value of a single multi-objective policy (as mapping from states to actions) has been studied in Wang et al. (2022). As this type of fine-grained preference feedback is difficult for users to define, especially in environments where sequential decisions are made, we restrict the agent to rely solely on comparative feedback. Comparative feedback is widely used in practice. For example, ChatGPT asks users to compare two responses to improve its performance. This approach is more intuitive for users compared to asking for numerical scores of ChatGPT responses.

Indeed, as knowledge of the user's preference vector is sufficient to solve for their optimal personalized policy, the challenge is to learn a user's preference vector using a minimal number of easily interpretable queries. We therefore concentrate on comparison queries. The question is then what exactly to compare? Comparing state-action pairs might not be a good option for the aforementioned tasks-what is the meaning of two single steps in a route? Comparing single trajectories (e.g., routes in Google Maps) would not be ideal either. Consider for example two policies: one randomly generates either (personalized) GOOD or (personalized) BAD trajectories while the other consistently generates (personalized) MEDIOCRE trajectories. By solely comparing single trajectories without considering sets of trajectories, we cannot discern the user's preference regarding the two policies.

Interpretable policy representation. Since we are interested in learning preferences via policy comparison queries, we also suggest an alternative, more interpretable representation of a policy. Namely, we design an algorithm that given an explicit policy representation (as a mapping from states to distributions over actions), returns a weighted set of trajectories of size at most k +1 , such that its expected return is identical to the value of the policy. 2 It immediately follows from our formalization that for any user, the personalized return of the weighted trajectory set and the personalized value of the policy are also identical.

In this work, we focus on answering two questions:

(1) How to find the optimal personalized policy by querying as few policy comparisons as possible? (2) How can we find a more interpretable representation of policies efficiently?

Contributions. In Section 2, we formalize the problem of eliciting user preferences and finding the optimal personalized policy via comparative feedback. As an alternative to an explicit policy representation, we propose a weighted trajectory set as a more interpretable representation. In Section 3, we provide an efficient algorithm for finding an approximate optimal personalized policy, where the policies are given by their formal representations, thus answering (1). In Section 4, we design two efficient algorithms that find the weighted trajectory set representation of a policy. Combined with the algorithm in Section 3, we have an algorithm for finding an approximate optimal personalized policy when policies are represented by weighted trajectory sets, thus answering (2).

Related Work. Multi-objective decision making has gained significant attention in recent years (see Roijers et al. (2013); Hayes et al. (2022) for surveys). Prior research has explored various approaches,

such as assuming linear preferences or Bayesian Settings, or finding an approximated Pareto frontier. However, incorporating comparison feedback (as was done for Multi-arm bandits or active learning, e.g., Bengs et al. (2021) and Kane et al. (2017)) allows us a more comprehensive and systematic approach to handling different types of user preferences and provides (nearly) optimal personalized decision-making outcomes. We refer the reader to Appendix A for additional related work.

2 Problem Setup

Sequential decision model. We consider a Markov decision process (MDP) known 3 to the agent represented by a tuple ⟨S , A , s 0 , P, R, H ⟩ , with finite state and action sets, S and A , respectively, an initial state s 0 ∈ S , and finite horizon H ∈ N . For example, in the Google Maps example a state is an intersection and actions are turns. The transition function P : S × A ↦→ Simplex S maps state-action pairs into a state probability distribution. To model multiple objectives, the reward function R : S × A ↦→ [0 , 1] k maps every state-action pair to a k -dimensional reward vector, where each component corresponds to one of the k objectives (e.g., road surface condition, worst and average estimated arrival time). The return of a trajectory τ = ( s 0 , a 0 , . . . , s H -1 , a H -1 , s H ) is given by Φ( τ ) = ∑ H -1 t =0 R ( s t , a t ) .

A policy π is a mapping from states to a distribution over actions. We denote the set of policies by Π . The value of a policy π , denoted by V π , is the expected cumulative reward obtained by executing the policy π starting from the initial state, s 0 . Put differently, the value of π is V π = V π ( s 0 ) = E S 0 = s 0 [ ∑ H -1 t =0 R ( S t , π ( S t )) ] ∈ [0 , H ] k , where S t is the random state at time step t when executing π , and the expectation is over the randomness of P and π . Note that V π = E τ [Φ( τ )] , where τ = ( s 0 , π ( s 0 ) , S 1 , . . . , π ( S H -1 ) , S H ) is a random trajectory generated by executing π .

We assume the existence of a 'do nothing' action a 0 ∈ A , available only from the initial state s 0 , that has zero reward for each objective R ( s 0 , a 0 ) = 0 and keeps the system in the initial state, i.e., P ( s 0 | s 0 , a 0 ) = 1 (e.g., this action corresponds to not commuting or refusing to play in a chess game.) 4 . We also define the (deterministic) 'do nothing' policy π 0 that always selects action a 0 and has a value of V π 0 = 0 . From a mathematical perspective, the assumption of 'do nothing' ensures that 0 belongs to the value vector space { V π | π ∈ Π } , which is precisely what we need.

√

Since the rewards are bounded between [0 , 1] , we have that 1 ≤ ∥ V π ∥ 2 ≤ kH for every policy π . For convenience, we denote C V = √ kH . We denote by d ≤ k the rank of the space spanned by all the value vectors obtained by Π , i.e., d := rank(span( { V π | π ∈ Π } )) .

Linear preferences. To incorporate personalized preferences over objectives, we assume each user is characterized by an unknown k -dimensional preference vector w ∗ ∈ R k + with a bounded norm 1 ≤ ∥ w ∗ ∥ 2 ≤ C w for some (unknown) C w ≥ 1 . We avoid assuming that C w = 1 to accommodate for general linear rewards. Note that the magnitude of w ∗ does not change the personalized optimal policy but affects the 'indistinguishability' in the feedback model. By not normalizing ∥ w ∗ ∥ 2 = 1 , we allow users to have varying levels of discernment. This preference vector encodes preferences over the multiple objectives and as a result, determines the user preferences over policies.

Formally, for a user characterized by w ∗ , the personalized value of policy π is ⟨ w ∗ , V π ⟩ ∈ R + . We denote by π ∗ := arg max π ∈ Π ⟨ w ∗ , V π ⟩ and v ∗ := 〈 w ∗ , V π ∗ 〉 the optimal personalized policy and its corresponding optimal personalized value for a user who is characterized by w ∗ . We remark that the 'do nothing' policy π 0 (that always selects action a 0 ) has a value of V π 0 = 0 , which implies a personalized value of ⟨ w ∗ , V π 0 ⟩ = 0 for every w ∗ . For any two policies π 1 and π 2 , the user characterized by w ∗ prefers π 1 over π 2 if ⟨ w ∗ , V π 1 -V π 2 ⟩ > 0 . Our goal is to find the optimal personalized policy for a given user using as few interactions with them as possible.

Comparative feedback. Given two policies π 1 , π 2 , the user returns π 1 ≻ π 2 whenever ⟨ w ∗ , V π 1 -V π 2 ⟩ > ϵ ; otherwise, the user returns 'indistinguishable' (i.e., whenever |⟨ w ∗ , V π 1 -V π 2 ⟩| ≤ ϵ ). Here ϵ > 0 measures the precision of the comparative feedback and

is small usually. The agent can query the user about their policy preferences using two different types of policy representations:

Explicit policy representation of π : An explicit representation of policy as mapping, π : S → Simplex A .

Weighted trajectory set representation of π : A κ -sized set of trajectory-weight pairs { ( p i , τ i ) } κ i =1 for some κ ≤ k + 1 such that (i) the weights p 1 , . . . , p κ are non-negative and sum to 1 ; (ii) every trajectory in the set is in the support 5 of the policy π ; and (iii) the expected return of these trajectories according to the weights is identical to the value of π , i.e., V π = ∑ κ i =1 p i Φ( τ i ) . Such comparison could be practical for humans. E.g., in the context of Google Maps, when the goal is to get from home to the airport, taking a specific route takes 40 minutes 90% of the time, but it can take 3 hours in case of an accident (which happens w.p. 10% ) vs. taking the subway which always has a duration of 1 hour.

In both cases, the feedback is identical and depends on the hidden precision parameter ϵ . As a result, the value of ϵ will affect the number of queries and how close the value of the resulting personalized policy is to the optimal personalized value. Alternatively, we can let the agent decide in advance on a maximal number of queries, which will affect the optimality of the returned policy.

Technical Challenges. In Section 3, we find an approximate optimal policy by O (log 1 ϵ ) queries. To achieve this, we approach the problem in two steps. Firstly, we identify a set of linearly independent policy values, and then we estimate the preference vector w ∗ using a linear program that incorporates comparative feedback. The estimation error of w ∗ usually depends on the condition number of the linear program. Therefore, the main challenge we face is how to search for linear independent policy values that lead to a small condition number and providing a guarantee for this estimate.

In Section 4, we focus on how to design efficient algorithms to find the weighted trajectory set representation of a policy. Initially, we employ the well-known Carathéodory's theorem, which yields an inefficient algorithm with a potentially exponential running time of | S | H . Our main challenge lies in developing an efficient algorithm with a running time of O (poly( H | S | |A| )) . The approach based on Carathéodory's theorem treats the return of trajectories as independent k -dimensional vectors, neglecting the fact that they are all generated from the same MDP. To overcome this challenge, we leverage the inherent structure of MDPs.

3 Learning from Explicit Policies

In this section, we consider the case where the interaction with a user is based on explicit policy comparison queries. We design an algorithm that outputs a policy being nearly optimal for this user. For multiple different users, we only need to run part of the algorithm again and again. For brevity, we relegate all proofs to the appendix.

If the user's preference vector w ∗ (up to a positive scaling) is given, then one can compute the optimal policy and its personalized value efficiently, e.g., using the Finite Horizon Value Iteration algorithm. In our work, w ∗ is unknown and we interact with the user to learn w ∗ through comparative feedback. Due to the structure model that is limited to meaningful feedback only when the compared policy values differ at least ϵ , the exact value of w ∗ cannot be recovered. We proceed by providing a high-level description of our ideas of how to estimate w ∗ .

(1) Basis policies : We find policies π 1 , . . . , π d , and their respective values, V π 1 , . . . , V π d ∈ [0 , H ] k , such that their values are linearly independent and that together they span the entire space of value vectors. 6 These policies will not necessarily be personalized optimal for the current user, and instead serve only as building blocks to estimate the preference vector, w ∗ . In Section 3.1 we describe an algorithm that finds a set of basis policies for any given MDP.

(2) Basis ratios : For the basis policies, denote by α i > 0 the ratio between the personalized value of a benchmark policy , π 1 , to the personalized value of π i +1 , i.e.,

We will estimate ̂ α i of α i for all i ∈ [ d -1] using comparison queries. A detailed algorithm for estimating these ratios appears in Section 3.2. For intuition, if we obtain exact ratios ̂ α i = α i for every i ∈ [ d -1] , then we can compute the vector w ∗ ∥ w ∗ ∥ 1 as follows. Consider the d -1 equations and d -1 variables in Eq (1). Since d is the maximum number of value vectors that are linearly independent, and V π 1 , . . . V π d form a basis, adding the equation ∥ w ∥ 1 = 1 yields d independent equations with d variables, which allows us to solve for w ∗ . The details of computing an estimate of w ∗ are described in Section 3.3.

3.1 Finding a Basis of Policies

In order to efficiently find d policies with d linearly independent value vectors that span the space of value vectors, one might think that selecting the k policies that each optimizes one of the k objectives will suffice. However, this might fail-in Appendix N, we show an instance in which these k value vectors are linearly dependent even though there exist k policies whose values span a space of rank k .

Moreover, our goal is to find not just any basis of policies, but a basis of policies such that (1) the personalized value of the benchmark policy ⟨ w ∗ , V π 1 ⟩ will be large (and hence the estimation error of ratio α i , | ̂ α i -α i | , will be small), and (2) that the linear program generated by this basis of policies and the basis ratios will produce a good estimate of w ∗ .

Choice of π 1 . Besides linear independence of values, another challenge is to find a basis of policies to contain a benchmark policy, π 1 (where the index 1 is wlog) with a relatively large personalized value, ⟨ w ∗ , V π 1 ⟩ , so that ̂ α i 's error is small (e.g., in the extreme case where ⟨ w ∗ , V π 1 ⟩ = 0 , we will not be able to estimate α i ).

For any w ∈ R k , we use π w denote a policy that maximizes the scalar reward ⟨ w,R ⟩ , i.e.,

and by v w = 〈 w,V π w 〉 to denote the corresponding personalized value. Let e 1 , . . . , e k denote the standard basis. To find π 1 with large personalized value ⟨ w ∗ , V π 1 ⟩ , we find policies π e j that maximize the j -th objective for every j = 1 , . . . , k and then query the user to compare them until we find a π e j with (approximately) a maximal personalized value among them. This policy will be our benchmark policy, π 1 . The details are described in lines 1-6 of Algorithm 1.

Choice of π 2 , . . . , π d . After finding π 1 , we next search the remaining d -1 polices π 2 , . . . , π d sequentially (lines 8-13 of Algorithm 1). For i = 2 , . . . , d , we find a direction u i such that (i) the vector u i is orthogonal to the space of current value vectors span( V π 1 , . . . , V π i -1 ) , and (ii) there exists a policy π i such that V π i has a significant component in the direction of u i . Condition (i) is used to guarantee that the policy π i has a value vector linearly independent of span( V π 1 , . . . , V π i -1 ) . Condition (ii) is used to cope with the error caused by inaccurate approximation of the ratios ̂ α i . Intuitively, when ∥ α i V π 1 -V π i +1 ∥ 2 ≪ ϵ , the angle between ̂ α i V π 1 -V π i +1 and α i V π 1 -V π i +1 could be very large, which results in an inaccurate estimate of w ∗ in the direction of α i V π 1 -V π i +1 . For example, if V π 1 = e 1 and V π i = e 1 + 1 w ∗ i ϵ e i for i = 2 , . . . , k , then π 1 , π i are 'indistinguishable' and the estimate ratio ̂ α i -1 can be 1 . Then the estimate of w ∗ by solving linear equations in Eq (1) is (1 , 0 , . . . , 0) , which could be far from the true w ∗ . Finding u i 's in which policy values have a large component can help with this problem.

Algorithm 1 provides a more detailed description of this procedure. Note that if there are n different users, we will run Algorithm 1 at most k times instead of n times. The reason is that Algorithm

1 only utilizes preference comparisons while searching for π 1 (lines 1-6), and not for π 2 , . . . , π k (which contributes to the k 2 factor in computational complexity). As there are at most k candidates for π 1 , namely π e 1 , . . . , π e k , we execute lines 1-6 of Algorithm 1 for n rounds and lines 8-13 for only k rounds.

3.2 Computation of Basis Ratios

As we have mentioned before, comparing basis policies alone does not allow for the exact computation of the α i ratios as comparing π 1 , π i can only reveal which is better but not how much. To this end, we will use the 'do nothing' policy to approximate every ratio α i up to some additive error | ̂ α i -α i | using binary search over the parameter ̂ α i ∈ [0 , C α ] for some C α ≥ 1 (to be determined later) and comparison queries of policy π i +1 with policy ̂ α i π 1 +(1 -̂ α i ) π 0 if ̂ α i ≤ 1 (or comparing π 1 and 1 ̂ α i π i +1 +(1 -1 ̂ α i ) π 0 instead if ̂ α i > 1 ). 7 Notice that the personalized value of ̂ α i π 1 +(1 -̂ α i ) π 0 is identical to the personalized value of π 1 multiplied by ̂ α i . We stop once ̂ α i is such that the user returns 'indistinguishable'. Once we stop, the two policies have roughly the same personalized value,

Eq (1) combined with the above inequality implies that | ̂ α i -α i | ⟨ w ∗ , V π 1 ⟩ ≤ C α ϵ . Thus, the approximation error of each ratio is bounded by | ̂ α i -α i | ≤ C α ϵ ⟨ w ∗ ,V π 1 ⟩ . To make sure the procedure will terminate, we need to set C α ≥ v ∗ ⟨ w ∗ ,V π 1 ⟩ since α i 's must lie in the interval [0 , v ∗ ⟨ w ∗ ,V π 1 ⟩ ] . Upon stopping binary search once Eq (3) holds, it takes at most O ( d log( C α ⟨ w ∗ , V π 1 ⟩ /ϵ )) comparison queries to estimate all the α i 's.

Due to the carefully picked π 1 in Algorithm 1, we can upper bound v ∗ ⟨ w ∗ ,V π 1 ⟩ by 2 k and derive an upper bound for | ̂ α i -α i | by selecting C α = 2 k .

In what follows we set C α = 2 k . The pseudo code of the above process of estimating α i 's is deferred to Algorithm 3 in Appendix C.

3.3 Preference Approximation and Personalized Policy

We move on to present an algorithm that estimates w ∗ and calculates a nearly optimal personalized policy. Given the π i 's returned by Algorithm 1 and the ̂ α i 's returned by Algorithm 3, consider a matrix ̂ A ∈ R d × k with 1 st row V π 1 ⊤ and the i th row ( ̂ α i -1 V π 1 -V π i ) ⊤ for every i = 2 , . . . , d . Let ̂ w be a solution to ̂ Ax = e 1 . Wewill show that ̂ w is a good estimate of w ′ := w ∗ ⟨ w ∗ ,V π 1 ⟩ and that π ̂ w is a nearly optimal personalized policy. In particular, when ϵ is small, we have |⟨ ̂ w,V π ⟩ - ⟨ w ′ , V π ⟩| = O ( ϵ 1 3 ) for every policy π . Putting this together, we derive the following theorem.

Theorem 1. Consider the algorithm of computing ̂ A and any solution ̂ w to ̂ Ax = e 1 and outputting the policy π ̂ w = arg max π ∈ Π ⟨ ̂ w,V π ⟩ , which is the optimal personalized policy for preference vector ̂ w . Then the output policy π ̂ w satisfying that v ∗ -〈 w ∗ , V π ̂ w 〉 ≤ O ( ( √ k +1 ) d + 14 3 ϵ 1 3 ) using O ( k log( k/ϵ )) comparison queries.

Computation Complexity We remark that Algorithm 1 solves Eq (2) for the optimal policy in scalar reward MDP at most O ( k 2 ) times. Using, e.g., Finite Horizon Value iteration to solve for the optimal policy takes O ( H |S| 2 |A| ) steps. However, while the time complexity it takes to return the optimal policy for a single user is O ( k 2 H |S| 2 |A| + k log( k ϵ )) , considering n different users rather than one results in overall time complexity of O (( k 3 + n ) H |S| 2 |A| + nk log( k ϵ )) .

Proof Technique. The standard technique typically starts by deriving an upper bound on ∥ ̂ w -w ∗ ∥ 2 and then uses this bound to upper bound sup π |⟨ ̂ w,V π ⟩ - ⟨ w ∗ , V π ⟩| as C V ∥ ̂ w -w ∗ ∥ 2 . However,

this method fails to achieve a non-vacuous bound in case there are two basis policies that are close to each other. For instance, consider the returned basis policy values: V π 1 = (1 , 0 , 0) , V π 2 = (1 , 1 , 0) , and V π 3 = (1 , 0 , η ) for some η > 0 . When η is extremely small, the estimate ̂ w becomes highly inaccurate in the direction of (0 , 0 , 1) , leading to a large ∥ ̂ w -w ∗ ∥ 2 . Even in such cases, we can still obtain a non-vacuous guarantee since the selection of V π 3 (line 11 of Algorithm 1) implies that no policy in Π exhibits a larger component in the direction of (0 , 0 , 1) than π 3 .

Proof Sketch. The analysis of Theorem 1 has two parts. First, as mentioned in Sec 3.1, when ∥ α i V π 1 -V π i +1 ∥ 2 ≪ ϵ , the error of ̂ α i +1 can lead to inaccurate estimate of w ∗ in direction α i V π 1 -V π i +1 . Thus, we consider another estimate of w ∗ based only on some π i +1 's with a relatively large ∥ α i V π 1 -V π i +1 ∥ 2 . In particular, for any δ > 0 , let d δ := min i ≥ 2:max( | v u i | , | v -u i | ) ≤ δ i -1 . That is to say, for i = 2 , . . . , d δ , the policy π i satisfies ⟨ u i , V π i ⟩ > δ and for any policy π , we have ⟨ u d δ +1 , V π ⟩ ≤ δ . Then, for any policy π and any unit vector ξ ∈ span( V π 1 , . . . , V π d δ ) ⊥ , we have ⟨ ξ, V π ⟩ ≤ √ kδ . This is because at round d δ +1 , we pick an orthonormal basis ρ 1 , . . . , ρ k -d δ of span( V π 1 , . . . , V π d δ ) ⊥ (line 8 in Algorithm 1) and pick u d δ +1 to be the one in which there exists a policy with the largest component as described in line 9. Hence, |⟨ ρ j , V π ⟩| ≤ δ for all j ∈ [ k -d δ ] . Then, we have ⟨ ξ, V π ⟩ = ∑ k -d δ j =1 ⟨ ξ, ρ j ⟩ ⟨ ρ j , V π ⟩ ≤ √ kδ by Cauchy-Schwarz inequality. Let ̂ A ( δ ) ∈ R d δ × k be the sub-matrix comprised of the first d δ rows of ̂ A . Then we consider an alternative estimate ̂ w ( δ ) = arg min x : ̂ A ( δ ) x = e 1 ∥ x ∥ 2 , the minimum norm solution of x to ̂ A ( δ ) x = e 1 . We upper bound sup π ∣ ∣ 〈 ̂ w ( δ ) , V π 〉 -⟨ w ′ , V π ⟩ ∣ ∣ in Lemma 2 and sup π ∣ ∣ ⟨ ̂ w,V π ⟩ -〈 ̂ w ( δ ) , V π 〉∣ ∣ in Lemma 3. Then we are done with the proof of Theorem 1.

Since we only remove the rows in ̂ A corresponding to u i 's in the subspace where no policy's value has a large component, ̂ w and ̂ w ( δ ) are close in terms of sup π ∣ ∣ ⟨ ̂ w,V π ⟩ -〈 ̂ w ( δ ) , V π 〉∣ ∣ .

Note that the result in Theorem 1 has a factor of k d 2 , which is exponential in d . Usually, we consider the case where k = O (1) is small and thus k d = O (1) is small. We get rid of the exponential dependence on d by applying ̂ w ( δ ) to estimate w ∗ directly, which requires us to set the value of δ beforehand. The following theorem follows directly by assigning the optimal value for δ in Lemma 2.

Notice that the algorithm in Theorem 2 needs to set the hyperparameter δ beforehand while we don't have to set any hyperparameter in Theorem 1. The improper value of δ could degrade the performance of the algorithm. But we can approximately estimate ϵ by binary searching η ∈ [0 , 1] and comparing π 1 against the scaled version of itself (1 -η ) π 1 until we find an η such that the user cannot distinguish between π 1 and (1 -η ) π 1 . Then we can obtain an estimate of ϵ and use the estimate to set the hyperparameter.

We remark that though we think of k as a small number, it is unclear whether the dependency on ϵ in Theorems 1 and 2 is optimal. The tight dependency on ϵ is left as an open problem. We briefly discuss a potential direction to improve this bound in Appendix I.

4 Learning from Weighted Trajectory Set Representation

In the last section, we represented policies using their explicit form as state-action mappings. However, such a representation could be challenging for users to interpret. For example, how safe is a car

described by a list of |S| states and actions such as 'turning left'? In this section, we design algorithms that return a more interpretable policy representation-a weighted trajectory set.

Recall the definition in Section 2, a weighted trajectory set is a small set of trajectories from the support of the policy and corresponding weights, with the property that the expected return of the trajectories in the set (according to the weights) is exactly the value of the policy (henceforth, the exact value property ). 8 As these sets preserve all the information regarding the multi-objective values of policies, they can be used as policy representations in policy comparison queries of Algorithm 3 without compromising on feedback quality. Thus, using these representations obtain the same optimality guarantees regarding the returned policy in Section 3 (but would require extra computation time to calculate the sets).

There are two key observations on which the algorithms in this section are based:

(1) Each policy π induces a distribution over trajectories. Let q π ( τ ) denote the probability that a trajectory τ is sampled when selecting actions according to π . The expected return of all trajectories under q π is identical to the value of the policy, i.e., V π = ∑ τ q π ( τ )Φ( τ ) . In particular, the value of a policy is a convex combination of the returns of the trajectories in its support. However, we avoid using this convex combination to represent a policy since the number of trajectories in the support of a policy could be exponential in the number of states and actions.

(2) The existence of a small weighted trajectory set is implied by Carathéodory's theorem. Namely, since the value of a policy is in particular a convex combination of the returns of the trajectories in its support, Carathéodory's theorem implies that there exist k +1 trajectories in the support of the policy and weights for them such that a convex combination of their returns is the value of the policy. Such a ( k +1) -sized set will be the output of our algorithms.

We can apply the idea behind Carathéodory's theorem proof to compress trajectories as follows. For any ( k +2) -sized set of k -dimensional vectors { µ 1 , . . . , µ k +2 } , for any convex combination of them µ = ∑ k +2 i =1 p i µ i , we can always find a ( k +1) -sized subset such that µ can be represented as the convex combination of the subset by solving a linear equation. Given an input of a probability distribution p over a set of k -dimensional vectors, M , we pick k +2 vectors from M , reduce at least one of them through the above procedure. We repeat this step until we are left with at most k +1 vectors. We refer to this algorithm as C4 (Compress Convex Combination using Carathéodory's theorem). The pseudocode is described in Algorithm 4, which is deferred to Appendix J due to space limit. The construction of the algorithm implies the following lemma immediately.

Lemma 4. Given a set of k -dimensional vectors M ⊂ R k and a distribution p over M , C4 ( M,p ) outputs M ′ ⊂ M with | M ′ | ≤ k +1 and a distribution q ∈ Simplex M ′ satisfying that E µ ∼ q [ µ ] = E µ ∼ p [ µ ] in time O ( | M | k 3 ) .

So now we know how to compress a set of trajectories to the desired size. The main challenge is how to do it efficiently (in time O (poly( H | S | |A| )) ). Namely, since the set of all trajectory returns from the support of the policy could be of size Ω( |S| H ) , using it as input to C4 Algorithm is inefficient. Instead, we will only use C4 as a subroutine when the number of trajectories is small.

We propose two efficient approaches for finding weighted trajectory representations. Both approaches take advantage of the property that all trajectories are generated from the same policy on the same MDP. First, we start with a small set of trajectories of length of 1 , expand them, compress them, and repeat until we get the set of trajectory lengths of H . The other is based on the construction of a layer graph where a policy corresponds to a flow in this graph and we show that finding representative trajectories is equivalent to flow decomposition.

In the next subsection, we will describe the expanding and compressing approach and defer the flow decomposition based approach to Appendix K due to space considerations. We remark that the flow decomposition approach has a running time of O ( H 2 |S| 2 + k 3 H |S| 2 ) (see appendix for

details), which underperforms the expanding and compressing approach (see Theorem 3) whenever |S| H + |S| k 3 = ω ( k 4 + k |S| ) . For presentation purposes, in the following, we only consider the deterministic policies. Our techniques can be easily extended to random policies. 9

Expanding and Compressing Approach.

The basic idea is to find k +1 trajectories of length 1 to represent V π first and then increase the length of the trajectories without increasing the number of trajectories. For policy π , let V π ( s, h ) = E S 0 = s [ ∑ h -1 t =0 R ( S t , π ( S t )) ] be the value of π with initial state S 0 = s and time horizon h . Since we study the representation for a fixed policy π in this section, we slightly abuse the notation and represent a trajectory by τ π = ( s, s 1 , . . . , s H ) . We denote the state of trajectory τ at time t as s τ t = s t . For a trajectory prefix τ = ( s, s 1 , . . . , s h ) of π with initial state s and h ≤ H subsequent states, the return of τ is Φ( τ ) = R ( s, π ( s )) + ∑ h -1 t =1 R ( s t , π ( s t )) . Let J ( τ ) be the expected return of trajectories (of length H ) with the prefix being τ , i.e.,

For any s ∈ S , let τ ◦ s denote the trajectory of appending s to τ . We can solve V π ( s, h ) for all s ∈ S , h ∈ [ H ] by dynamic programming in time O ( kH |S| 2 ) . Specifically, according to definition, we have V π ( s, 1) = R ( s, π ( s )) and

Thus, we can represent V π by

By applying C4, we can find a set of representative trajectories of length 1 , F (1) ⊂ { ( s 0 , s ) | s ∈ S} , with ∣ ∣ F (1) ∣ ∣ ≤ k +1 and weights β (1) ∈ Simplex F (1) such that

Supposing that we are given a set of trajectories F ( t ) of length t with weights β ( t ) such that V π = ∑ τ ∈ F ( t ) β ( t ) ( τ ) J ( τ ) , we can first increase the length of trajectories by 1 through Eq (4) and obtain a subset of { τ ◦ s | τ ∈ F ( t ) , s ∈ S} , in which the trajectories are of length t +1 . Specifically, we have

Then we would like to compress the above convex combination through C4 as we want to keep track of at most k +1 trajectories of length t +1 due to the computing time. More formally, let J F ( t ) := { J ( τ ◦ s ) | τ ∈ F ( t ) , s ∈ S} be the set of expected returns and p F ( t ) ,β ( t ) ∈ Simplex F ( t ) ×S with p F ( t ) ,β ( t ) ( τ ◦ s ) = β ( t ) ( τ ) P ( s | s τ t , π ( s τ t )) be the weights appearing in Eq (6). Here p F ( t ) ,β ( t ) defines a distribution over J F ( t ) with the probability of drawing J ( τ ◦ s ) being p F ( t ) ,β ( t ) ( τ ◦ s ) . Then we can apply C4 over ( J F ( t ) , p F ( t ) ,β ( t ) ) and compress the representative trajectories { τ ◦ s | τ ∈ F ( t ) , s ∈ S} . We start with trajectories of length 1 and repeat the process of expanding and compressing until we get trajectories of length H . The details are described in Algorithm 2.

Algorithm 2 Expanding and compressing trajectories

1: compute V π ( s, h ) for all s ∈ S , h ∈ [ H ] by dynamic programming according to Eq (4)

2: F (0) = { ( s 0 ) } and β (0) ( s 0 ) = 1

3: for t = 0 , . . . , H -1 do

4: J F ( t ) ← { J ( τ ◦ s ) | τ ∈ F ( t ) , s ∈ S} and p F ( t ) ,β ( t ) ( τ ◦ s ) ← β ( t ) ( τ ) P ( s | s τ t , π ( s τ t )) for τ ∈ F ( t ) , s ∈ S // expanding step

5: ( J ( t +1) , β ( t +1) ) ← C4 ( J F ( t ) , p F ( t ) ,β ( t ) ) and F ( t +1) ←{ τ | J ( τ ) ∈ J ( t +1) } // compressing step

6: end for

7: output F ( H ) and β ( H )

Theorem 3. Algorithm 2 outputs F ( H ) and β ( H ) satisfying that ∣ ∣ F ( H ) ∣ ∣ ≤ k + 1 and ∑ τ ∈ F ( H ) β ( H ) ( τ )Φ( τ ) = V π in time O ( k 4 H |S| + kH |S| 2 ) .

The proof of Theorem 3 follows immediately from the construction of the algorithm. According to Eq (5), we have V π = ∑ τ ∈ F (1) β (1) ( τ ) J ( τ ) . Then we can show that the output of Algorithm 2 is a valid weighted trajectory set by induction on the length of representative trajectories. C4 guarantees that ∣ ∣ F ( t ) ∣ ∣ ≤ k +1 for all t = 1 , . . . , H , and thus, we only keep track of at most k +1 trajectories at each step and achieve the computation guarantee in the theorem. Combined with Theorem 1, we derive the following Corollary.

Corollary 1. Running the algorithm in Theorem 1 with weighted trajectory set representation returned by Algorithm 2 gives us the same guarantee as that of Theorem 1 in time O ( k 2 H |S| 2 |A| + ( k 5 H |S| + k 2 H |S| 2 ) log( k ϵ )) .

5 Discussion

In this paper, we designed efficient algorithms for learning users' preferences over multiple objectives from comparative feedback. The efficiency is expressed in both the running time and number of queries (both polynomial in H, |S| , |A| , k and logarithmic in 1 /ϵ ). The learned preferences of a user can then be used to reduce the problem of finding a personalized optimal policy for this user to a (finite horizon) single scalar reward MDP, a problem with a known efficient solution. As we have focused on minimizing the policy comparison queries, our algorithms are based on polynomial time pre-processing calculations that save valuable comparison time for users.

The results in Section 3 are of independent interest and can be applied to a more general learning setting, where for some unknown linear parameter w ∗ , given a set of points X and access to comparison queries of any two points, the goal is to learn arg max x ∈ X ⟨ w ∗ , x ⟩ . E.g., in personalized recommendations for coffee beans in terms of the coffee profile described by the coffee suppliers (body, aroma, crema, roast level,...), while users could fail to describe their optimal coffee beans profile, adopting the methodology in Section 3 can retrieve the ideal coffee beans for a user using comparisons (where the mixing with 'do nothing' is done by diluting the coffee with water and the optimal coffee for a given profile is the one closest to it).

When moving from the explicit representation of policies as mappings from states to actions to a more natural policy representation as a weighted trajectory set, we then obtained the same optimality guarantees in terms of the number of queries. While there could be other forms of policy representations (e.g., a small subset of common states), one advantage of our weighted trajectory set representation is that it captures the essence of the policy multi-objective value in a clear manner via O ( k ) trajectories and weights. The algorithms provided in Section 4 are standalone and could also be of independent interest for explainable RL (Alharin et al., 2020). For example, to exemplify the multi-objective performance of generic robotic vacuum cleaners (this is beneficial if we only have e.g., 3 of themwe can apply the algorithms in Section 4 to generate weighted trajectory set representations and compare them directly without going through the algorithm in Section 3.).

An interesting direction for future work is to relax the assumption that the MDP is known in advance. One direct way is to first learn the model (in model-based RL), then apply our algorithms in the learned MDP. The sub-optimality of the returned policy will then depend on both the estimation error of the model and the error introduced by our algorithms (which depends on the parameters in the learned model).

References

Ailon, N. (2012). An active learning algorithm for ranking from pairwise preferences with an almost optimal query complexity. Journal of Machine Learning Research , 13(1):137-164.

Ailon, N., Karnin, Z. S., and Joachims, T. (2014). Reducing dueling bandits to cardinal bandits. In ICML , volume 32, pages 856-864.

Akrour, R., Schoenauer, M., and Sebag, M. (2012). APRIL: Active preference learning-based reinforcement learning. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 116-131.

Alharin, A., Doan, T.-N., and Sartipi, M. (2020). Reinforcement learning interpretation methods: A survey. IEEE Access , 8:171058-171077.

Balcan, M., Blum, A., and Vempala, S. S. (2015). Efficient representations for lifelong learning and autoencoding. In Proceedings of the Annual Conference on Learning Theory (COLT) , pages 191-210.

Balcan, M.-F., Vitercik, E., and White, C. (2016). Learning combinatorial functions from pairwise comparisons. In Proceedings of the Annual Conference on Learning Theory (COLT) .

Barrett, L. and Narayanan, S. (2008). Learning all optimal policies with multiple criteria. In Proceedings of the International Conference on Machine Learning (ICML) , pages 41-47.

Bengs, V., Busa-Fekete, R., El Mesaoudi-Paul, A., and Hüllermeier, E. (2021). Preference-based online learning with dueling bandits: A survey. J. Mach. Learn. Res.

Bhatia, K., Pananjady, A., Bartlett, P., Dragan, A., and Wainwright, M. J. (2020). Preference learning along multiple criteria: A game-theoretic perspective. In Advances in Neural Information Processing Systems (NeurIPS) , pages 7413-7424.

Biyik, E. and Sadigh, D. (2018). Batch active preference-based learning of reward functions. In Proceedings of the Conference on Robot Learning (CoRL) , pages 519-528.

Blum, A. and Shao, H. (2020). Online learning with primary and secondary losses. Advances in Neural Information Processing Systems , 33:20427-20436.

Chatterjee, K. (2007). Markov decision processes with multiple long-run average objectives. In Proceedings of the International Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS) , pages 473-484.

Chatterjee, K., Majumdar, R., and Henzinger, T. A. (2006). Markov decision processes with multiple objectives. In Proceedings of the Annual Symposium on Theoretical Aspects of Computer Science (STACS) , pages 325-336.

Chen, X., Ghadirzadeh, A., Björkman, M., and Jensfelt, P. (2019). Meta-learning for multi-objective reinforcement learning. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 977-983.

Cheng, W., Fürnkranz, J., Hüllermeier, E., and Park, S.-H. (2011). Preference-based policy iteration: Leveraging preference learning for reinforcement learning. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD) , pages 312-327.

Cheung, W. C. (2019). Regret minimization for reinforcement learning with vectorial feedback and complex objectives. In Advances in Neural Information Processing Systems (NeurIPS) .

Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. (2017). Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NeurIPS) .

Doumpos, M. and Zopounidis, C. (2007). Regularized estimation for preference disaggregation in multiple criteria decision making. Computational Optimization and Applications , 38(1):61-80.

Fürnkranz, J., Hüllermeier, E., Cheng, W., and Park, S.-H. (2012). Preference-based reinforcement learning: A formal framework and a policy iteration algorithm. Machine Learning , 89(1):123-156.

Hayes, C. F., Radulescu, R., Bargiacchi, E., Källström, J., Macfarlane, M., Reymond, M., Verstraeten, T., Zintgraf, L. M., Dazeley, R., Heintz, F., Howley, E., Irissappane, A. A., Mannion, P., Nowé, A., de Oliveira Ramos, G., Restelli, M., Vamplew, P., and Roijers, D. M. (2022). A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems .

Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. (2018). Reward learning from human preferences and demonstrations in Atari. In Advances in Neural Information Processing Systems (NeurIPS) .

Jain, A., Sharma, S., Joachims, T., and Saxena, A. (2015). Learning preferences for manipulation tasks from online coactive feedback. International Journal of Robotics Research , 34(10):1296-1313.

Jamieson, K. G. and Nowak, R. (2011). Active ranking using pairwise comparisons. Advances in neural information processing systems , 24.

Kane, D. M., Lovett, S., Moran, S., and Zhang, J. (2017). Active classification with comparison queries. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science (FOCS) , pages 355-366. IEEE.

Knox, W. B., Hatgis-Kessell, S., Booth, S., Niekum, S., Stone, P., and Allievi, A. (2022). Models of human preference for learning reward functions. arXiv preprint arXiv:2206.02231 .

Lee, K., Smith, L., and Abbeel, P. (2021). PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprint arXiv:2106.05091 .

Mannor, S., Perchet, V., and Stoltz, G. (2014). Approachability in unknown games: Online learning meets multi-objective optimization. In Conference on Learning Theory , pages 339-355. PMLR.

Mannor, S. and Shimkin, N. (2001). The steering approach for multi-criteria reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS) .

Mannor, S. and Shimkin, N. (2004). A geometric approach to multi-criterion reinforcement learning. Journal of Machine Learning Research , 5:325-360.

Marinescu, R., Razak, A., and Wilson, N. (2017). Multi-objective influence diagrams with possibly optimal policies. Proceedings of the AAAI Conference on Artificial Intelligence .

McAuley, J. (2022). Personalized Machine Learning . Cambridge University Press. in press.

Pacchiano, A., Saha, A., and Lee, J. (2022). Dueling RL: reinforcement learning with trajectory preferences.

Ren, W., Liu, J., and Shroff, N. B. (2018). PAC ranking from pairwise and listwise queries: Lower bounds and upper bounds. arXiv preprint arXiv:1806.02970 .

Roijers, D. M., Vamplew, P., Whiteson, S., and Dazeley, R. (2013). A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research , 48:67-113.

Rothkopf, C. A. and Dimitrakakis, C. (2011). Preference elicitation and inverse reinforcement learning. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 34-48.

Sadigh, D., Dragan, A. D., Sastry, S., and Seshia, S. A. (2017). Active preference-based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS) .

Saha, A. and Gopalan, A. (2018). Battle of bandits. In Uncertainty in Artificial Intelligence .

Saha, A. and Gopalan, A. (2019). PAC Battling Bandits in the Plackett-Luce Model. In Algorithmic Learning Theory , pages 700-737.

Saha, A., Koren, T., and Mansour, Y. (2021). Dueling convex optimization. In International Conference on Machine Learning , pages 9245-9254. PMLR.

Sui, Y., Zhuang, V., Burdick, J., and Yue, Y. (2017). Multi-dueling bandits with dependent arms. In Conference on Uncertainty in Artificial Intelligence , UAI'17.

Sui, Y., Zoghi, M., Hofmann, K., and Yue, Y. (2018). Advancements in dueling bandits. In IJCAI , pages 5502-5510.

Wang, N., Wang, H., Karimzadehgan, M., Kveton, B., and Boutilier, C. (2022). Imoˆ 3: Interactive multi-objective off-policy optimization. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22 .

Wilson, A., Fern, A., and Tadepalli, P. (2012). A Bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems (NeurIPS) .

Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. (2017a). A survey of preference-based reinforcement learning methods. J. Mach. Learn. Res.

Wirth, C., Akrour, R., Neumann, G., and Fürnkranz, J. (2017b). A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research , 18(136):1-46.

Wirth, C., Fürnkranz, J., and Neumann, G. (2016). Model-free preference-based reinforcement learning. In Proceedings of the National Conference on Artificial Intelligence (AAAI) .

Yona, G., Moran, S., Elidan, G., and Globerson, A. (2022). Active learning with label comparisons. In Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI) .

Zoghi, M., Whiteson, S., Munos, R., Rijke, M. d., et al. (2014). Relative upper confidence bound for the k -armed dueling bandit problem. In JMLR Workshop and Conference Proceedings , number 32, pages 10-18. JMLR.

A Related Work

Multi-objective sequential decision making There is a long history of work on multi-objective sequential decision making (Roijers et al., 2013), with one key focus being the realization of efficient algorithms for approximating the Pareto front (Chatterjee et al., 2006; Chatterjee, 2007; Marinescu et al., 2017). Instead of finding a possibly optimal policy, we concentrate on specific user preferences and find a policy that is optimal for that specific user. Just like the ideal car for one person could be a Chevrolet Spark (small) and for another, it is a Ford Ranger (a truck).

In the context of multi-objective RL (Hayes et al., 2022), the goal can be formulated as one of learning a policy for which the average return vector belongs to a target set (hence the term 'multi-criteria' RL), which existing work has treated as a stochastic game (Mannor and Shimkin, 2001, 2004). Other works seek to maximize (in expectation) a scalar version of the reward that may correspond to a weighted sum of the multiple objectives (Barrett and Narayanan, 2008; Chen et al., 2019) as we consider here, or a nonlinear function of the objectives (Cheung, 2019). Multi-objective online learning has also been studied, see (Mannor et al., 2014; Blum and Shao, 2020) for example.

The parameters that define this scalarization function (e.g., the relative objective weights) are often unknown and vary with the task setting or user. In this case, preference learning (Wirth et al., 2017a) is commonly used to elicit the value of these parameters. Doumpos and Zopounidis (2007) describe an approach to eliciting a user's relative weighting in the context of multi-objective decision-making. Bhatia et al. (2020) learn preferences over multiple objectives from pairwise queries using a gametheoretic approach to identify optimal randomized policies. In the context of RL involving both scalar and vector-valued objectives, user preference queries provide an alternative to learning from demonstrations, which may be difficult for people to provide (e.g., in the case of robots with high degrees-of-freedom), or explicit reward specifications (Cheng et al., 2011; Rothkopf and Dimitrakakis, 2011; Akrour et al., 2012; Wilson et al., 2012; Fürnkranz et al., 2012; Jain et al., 2015; Wirth et al., 2016; Christiano et al., 2017; Wirth et al., 2017b; Ibarz et al., 2018; Lee et al., 2021). These works typically assume that noisy human preferences over a pair of trajectories are correlated with the difference in their utilities (i.e., the reward acts as a latent term predictive of preference). Many contemporary methods estimate the latent reward by minimizing the cross-entropy loss between the reward-based predictions and the human-provided preferences (i.e., finding the reward that maximizes the likelihood of the observed preferences) (Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021; Knox et al., 2022; Pacchiano et al., 2022).

Wilson et al. (2012) describe a Bayesian approach to policy learning whereby they query a user for their preference between a pair of trajectories and use these preferences to maintain a posterior distribution over the latent policy parameters. The task of choosing the most informative queries is challenging due to the continuous space of trajectories, and is generally NP-hard (Ailon, 2012). Instead, they assume access to a distribution over trajectories that accounts for their feasibility and relevance to the target policy, and they describe two heuristic approaches to selecting trajectory queries based on this distribution. Finally, Sadigh et al. (2017) describe an approach to active preference-based learning in continuous state and action spaces. Integral to their work is the ability to synthesize dynamically feasible trajectory queries. Biyik and Sadigh (2018) extend this approach to the batch query setting.

Comparative feedback in other problems Comparative feedback has been studied in other problems in learning theory, e.g., combinatorial functions (Balcan et al., 2016). One closely related problem is active ranking/learning using pairwise comparisons (Jamieson and Nowak, 2011; Kane et al., 2017; Saha et al., 2021; Yona et al., 2022). These works usually consider a given finite sample of points. Kane et al. (2017) implies a lower bound of the number of comparisons is linear in the cardinality of the set even if the points satisfy the linear structural constraint as we assume in this work. In our work, the points are value vectors generated by running different policies under the same MDP and thus have a specific structure. Besides, we allow comparison of policies not in the policy set. Thus, we are able to obtain the query complexity sublinear in the number of policies. Another related problem using comparative/preference feedback is dueling bandits that aim to learn through pairwise feedback (Ailon et al., 2014; Zoghi et al., 2014) (see also Bengs et al. (2021); Sui et al. (2018) for surveys), or more generally any subsetwise feedback (Sui et al., 2017; Saha and Gopalan, 2018, 2019; Ren et al., 2018). However, unlike dueling bandits, we consider noiseless comparative feedback.

B Proof of Lemma 1

Proof. We first show that, according to our algorithm (lines 1-6 of Algorithm 1), the returned π 1 satisfies that

This can be proved by induction over k . In the base case of k = 2 , it's easy to see that the returned π 1 satisfies the above inequality. Suppose the above inequality holds for any k ≤ n -1 and we prove that it will also hold for k = n . After running the algorithm over j = 2 , . . . , k -1 (line 2 ), the returned policy π e ∗ satisfies that

Then there are two cases,

As π e i is the optimal personalized policy when the user's preference vector is e i , we have that

where the last inequality holds because the entries of V π e i and w ∗ are non-negative. Therefore, there exists i ∈ [ k ] such that 〈 w ∗ , V π e i 〉 ≥ 1 k 〈 w ∗ , V π ∗ 〉 = 1 k v ∗ .

Then we have

when ϵ ≤ v ∗ 2 k . By rearranging terms, we have v ∗ ⟨ w ∗ ,V π 1 ⟩ ≤ 2 k .

By setting C α = 2 k , we have | ̂ α i -α i | ⟨ w ∗ , V π 1 ⟩ ≤ C α ϵ = 2 kϵ and thus, | ̂ α i -α i | ≤ 4 k 2 ϵ v ∗ .

C Pseudo Code of Computation of the Basis Ratios

The pseudo code of searching ̂ α i 's is described in Algorithm 3.

Algorithm 3 Computation of Basis Ratios

D Proof of Theorem 1

Theorem 1. Consider the algorithm of computing ̂ A and any solution ̂ w to ̂ Ax = e 1 and outputting the policy π ̂ w = arg max π ∈ Π ⟨ ̂ w,V π ⟩ , which is the optimal personalized policy for preference vector ̂ w . Then the output policy π ̂ w satisfying that v ∗ -〈 w ∗ , V π ̂ w 〉 ≤ O ( ( √ k +1 ) d + 14 3 ϵ 1 3 ) using O ( k log( k/ϵ )) comparison queries.

Proof. Theorem 1 follows by setting ϵ α = 4 k 2 ϵ v ∗ and C α = 2 k as shown in Lemma 1 and combining the results of Lemma 2 and 3 with the triangle inequality.

Specifically, for any policy π , we have

The first inequality follows from

and applying (7) twice- once for π ∗ and once for π ̂ w . The last inquality follows by setting δ = ( C 4 V k 5 ∥ w ∗ ∥ 2 ϵ v ∗ 2 ) 1 3 .

E Proof of Lemma 2

Lemma 2. If | ̂ α i -α i | ≤ ϵ α and α i ≤ C α for all i ∈ [ d -1] , for every δ ≥ 4 C 2 3 α C V d 1 3 ϵ 1 3 α , we have ∣ ∣ 〈 ̂ w ( δ ) , V π 〉 -⟨ w ′ , V π ⟩ ∣ ∣ ≤ O ( C α C 4 V d 3 2 δ ∥ w ′ ∥ 2 2 ϵ α δ 2 + √ kδ ∥ w ′ ∥ 2 ) for all π , where w ′ = w ∗ ⟨ w ∗ ,V π 1 ⟩ .

To prove Lemma 2, we will first define a matrix, A (full) .

Given the output ( V π 1 , . . . , V π d ) of Algorithm 1, we have rank(span( { V π 1 , . . . , V π d δ } )) = d δ . Let b 1 , . . . , b d -d δ be a set of orthonormal vectors that are orthogonal to span( V π 1 , . . . , V π d δ ) and together with V π 1 , . . . , V π d δ form a basis for span( { V π | π ∈ Π } ) .

We define A (full) ∈ R d × k as the matrix of replacing the last d -d δ rows of A with b 1 , . . . , b d -d δ , i.e.,

Observation 1. We have that span( A (full) ) = span( { V π | π ∈ Π } ) and rank( A (full) ) = d .

Lemma 5. For all w ∈ R k satisfying A (full) w = e 1 , we have | w · V π -w ′ · V π | ≤ √ kδ ∥ w ′ ∥ 2 for all π .

We then show that there exists a w ∈ R k satisfying A (full) w = e 1 such that ∣ ∣ ̂ w ( δ ) · V π -w · V π ∣ ∣ is small for all π ∈ Π .

Lemma 6. If | ̂ α i -α i | ≤ ϵ α and α i ≤ C α for all i ∈ [ d -1] , for every δ ≥ 4 C 2 3 α C V d 1 3 ϵ 1 3 α there exists a w ∈ R k satisfying A (full) w = e 1 s.t. ∣ ∣ ̂ w ( δ ) · V π -w · V π ∣ ∣ ≤ O ( C α C 4 V d 3 2 δ ∥ w ′ ∥ 2 2 ϵ α δ 2 ) for all π .

We now derive Lemma 2 using the above two lemmas.

Proof of Lemma 2. Let w be defined in Lemma 6. Then for any policy π , we have

by applying Lemma 5 and 6.

F Proofs of Lemma 5 and Lemma 6

Lemma 5. For all w ∈ R k satisfying A (full) w = e 1 , we have | w · V π -w ′ · V π | ≤ √ kδ ∥ w ′ ∥ 2 for all π .

Proof of Lemma 5. Since span( A (full) ) = span( { V π | π ∈ Π } ) , for every policy π , the value vector can be represented as a linear combination of row vectors of A (full) , i.e., there exists a = ( a 1 , . . . , a d ) ∈ R d s.t.

Now, for any unit vector ξ ∈ span( b 1 , . . . , b d -d δ ) , we have ⟨ V π , ξ ⟩ ≤ √ kδ .

The reason is that at each round d δ +1 , we pick an orthonormal basis ρ 1 , . . . , ρ k -d δ of span( V π 1 , . . . , V π d δ ) ⊥ (line 8 in Algorithm 1) and pick u d δ +1 to be the one in which there exists a policy with the largest component as described in line 9. Hence, |⟨ ρ j , V π ⟩| ≤ δ for all j ∈ [ k -d δ ] .

It follows from Cauchy-Schwarz inequality that ⟨ ξ, V π ⟩ = ∑ k -d δ j =1 ⟨ ξ, ρ j ⟩ ⟨ ρ j , V π ⟩ ≤ √ kδ .

Combining with the observation that b 1 , . . . , b d -d δ are pairwise orthogonal and that each of them is orthogonal to span( V π 1 , . . . , V π d δ ) we have which implies that

Since w ′ satisfies Aw ′ = e 1 , we have

For any w ∈ R k satisfying A (full) w = e 1 , consider ˜ w = w + ∑ d -d δ i =1 ⟨ b i , w ′ ⟩ b i . Then we have A (full) ˜ w = A (full) w ′ .

Thus, applying (8) twice, we get

Hence,

where Eq (a) follows from (8), inequality (b) from Cauchy-Schwarz, and inequality (c) from applying (9).

Lemma 6. If | ̂ α i -α i | ≤ ϵ α and α i ≤ C α for all i ∈ [ d -1] , for every δ ≥ 4 C 2 3 α C V d 1 3 ϵ 1 3 α there exists a w ∈ R k satisfying A (full) w = e 1 s.t. ∣ ∣ ̂ w ( δ ) · V π -w · V π ∣ ∣ ≤ O ( C α C 4 V d 3 2 δ ∥ w ′ ∥ 2 2 ϵ α δ 2 ) for all π .

Before proving Lemma 6, we introduce some notations and a claim.

For any x, y ∈ R k , let θ ( x, y ) denotes the angle between x and y .

For any subspace U ⊂ R k , let θ ( x, U ) := min y ∈ U θ ( x, y ) .

For any two subspaces U, U ′ ⊂ R k , we define θ ( U, U ′ ) as θ ( U, U ′ ) = max x ∈ U min y ∈ U ′ θ ( x, y ) .

and where η ϵ α ,δ = 4 C α C 2 V d δ ϵ α δ 2 .

To prove the above claim, we use the following lemma by Balcan et al. (2015).

Lemma 7 (Lemma 3 of Balcan et al. (2015)) . Let U l = span( ξ 1 , . . . , ξ l ) and ̂ U l = span( ̂ ξ 1 , . . . , ̂ ξ l ) . Let ϵ acc , γ new ≥ 0 and ϵ acc ≤ γ 2 new / (10 l ) , and assume that θ ( ̂ ξ i , ̂ U i -1 ) ≥ γ new for i = 2 , . . . , l , and that θ ( ξ i , ̂ ξ i ) ≤ ϵ acc for i = 1 , . . . , l .

Then,

Proof of Claim 1. For all 2 ≤ i ≤ d δ , we have that

where Ineq (a) holds as u i is orthogonal to span( ̂ A ( δ ) 1: i -1 ) according to line 8 of Algorithm 1 and Ineq (b) holds due to ∣ ∣ ∣ ̂ A ( δ ) i · u i ∣ ∣ ∣ = | V π i · u i | ≥ δ . The last inequality holds due to ∥ ̂ α i -1 V π 1 -V π i ∥ 2 ≤ ̂ α i -1 ∥ V π 1 ∥ 2 + ∥ V i ∥ 2 ≤ ( C α +1) C V .

Similarly, we also have

For any matrix M , let M i denote the i -th row vector of M , M i : j denote the submatrix of M composed of rows i, i +1 , . . . , j , and M i : denote the submatrix composed of all rows j ≥ i .

Let span( M ) denote the span of the rows of M .

Recall that ̂ A ∈ R d × k is defined as

which is the approximation of matrix A ∈ R d × k defined by true values of α i , i.e.,

We denote by ̂ A ( δ ) = ̂ A 1: d δ , A ( δ ) = A 1: d δ ∈ R d δ × k the sub-matrices comprised of the first d δ rows of ̂ A and A respectively.

Claim 1. If | ̂ α i -α i | ≤ ϵ α and α i ≤ C α for all i ∈ [ d -1] , for every δ ≥ 4 C 2 3 α C V d 1 3 ϵ 1 3 α , we have

We continue by decomposing V π i in the direction of V π 1 and the direction perpendicular to V π 1 .

For convince, we denote v ∥ i := V π i · V π 1 ∥ V π 1 ∥ 2 , V ∥ i := v ∥ i V π 1 ∥ V π 1 ∥ 2 , V ⊥ i := V π i -V ∥ i and v ⊥ i := ∥ ∥ V ⊥ i ∥ ∥ 2 .

Then we have

If ( ̂ α i -1 V π 1 -V ∥ i ) · ( α i -1 V π 1 -V ∥ i ) ≥ 0 , i.e., ̂ α i -1 V π 1 -V ∥ i and α i -1 V π 1 -V ∥ i are in the same direction, then

where Ineq (12) follows from the fact that the derivative of arctan is at most 1 , i.e., ∂ arctan x ∂x = lim a → x arctan a -arctan x a -x = 1 1+ x 2 ≤ 1 .

Inequality (13) holds since v ⊥ i ≥ |⟨ V π i , u i ⟩| ≥ δ .

Similarly, we have

By applying Lemma 7 with ϵ acc = ϵ α C V δ , γ new = δ ( C α +1) C V , ( ξ i , ̂ ξ i ) = ( A i +1 , ̂ A i +1 ) (and ( ξ i , ̂ ξ i ) = ( ̂ A i +1 , A i +1 ) ), we have that when δ ≥ 10 1 3 ( C α +1) 2 3 C V d 1 3 δ ϵ 1 3 α ,

and

This completes the proof of Claim 1 since C α ≥ 1 .

Proof of Lemma 6. Recall that ̂ w ( δ ) = arg min ̂ A ( δ ) x = e 1 ∥ x ∥ 2 is the minimum norm solution to ̂ A ( δ ) x = e 1 .

Let λ 1 , . . . , λ d δ -1 be any orthonormal basis of span( A ( δ ) 2: ) .

We construct a vector w satisfying A (full) w = e 1 by removing ̂ w ( δ ) 's component in span( A ( δ ) 2: ) and rescaling. Formally,

It is direct to verify that A 1 · w = V π 1 · w = 1 and A i · w = 0 for i = 2 , . . . , d δ . As a result, A ( δ ) w = e 1 .

Combining with the fact that ̂ w ( δ ) has zero component in b i for all i ∈ [ d -d δ ] , we have A (full) w = e 1 . According to Claim 1, we have

Thus, there exist unit vectors ˜ λ 1 , . . . , ˜ λ d δ -1 ∈ span( ̂ A ( δ ) 2: ) such that θ ( λ i , ˜ λ i ) ≤ η ϵ α ,δ Since ̂ A ( δ ) ̂ w ( δ ) = e 1 , we have ̂ w ( δ ) · ˜ λ i = 0 for all i = 1 , . . . , d δ -1 , and therefore,

This implies that for any policy π ,

Denote by γ = V π 1 · ( ∑ d δ -1 i =1 〈 ̂ w ( δ ) , λ i 〉 λ i ) , which is no greater than C V √ d δ ∥ ∥ ̂ w ( δ ) ∥ ∥ 2 η ϵ α ,δ . We have that

where the last equality holds when C V √ d δ ∥ ∥ ̂ w ( δ ) ∥ ∥ 2 η ϵ α ,δ ≤ 1 2 .

Now we show that ∥ ∥ ̂ w ( δ ) ∥ ∥ 2 ≤ C ∥ w ′ ∥ 2 for some constant C .

Since ̂ w ( δ ) is the minimum norm solution to ̂ A ( δ ) x = e 1 , we will construct another solution to ̂ A ( δ ) x = e 1 , denoted by ̂ w 0 in a simillar manner to the construction in Eq (14), and show that ∥ ̂ w 0 ∥ 2 ≤ C ∥ w ′ ∥ .

Let ξ 1 , . . . , ξ d δ -1 be any orthonormal basis of span( ̂ A ( δ ) 2: ) .

We construct a ̂ w 0 s.t. ̂ A ( δ ) ̂ w 0 = e 1 by removing the component of w ′ in span( ̂ A ( δ ) 2: ) and rescaling. Specifically, let

Since A ( δ ) w ′ = e 1 , it directly follows that 〈 ̂ A 1 , ̂ w 0 〉 = ⟨ V π 1 , ̂ w 0 ⟩ = 1 and that 〈 ̂ A i , ̂ w 0 〉 = 0 for i = 2 , . . . , d δ , i.e., ̂ A ( δ ) ̂ w 0 = e 1 .

.

Since Claim 1 implies that θ (span( ̂ A ( δ ) 2: ) , span( A ( δ ) 2: )) ≤ η ϵ α ,δ , there exist unit vectors ˜ ξ 1 , . . . , ˜ ξ d δ -1 ∈ span( A ( δ ) 2: ) such that θ ( ξ i , ˜ ξ i ) ≤ η ϵ α ,δ .

As w ′ has zero component in span( A ( δ ) 2: ) , w ′ should have a small component in span( ̂ A ( δ ) 2: ) . In particular,

As a result, ∥ ̂ w 0 ∥ 2 ≤ 3 2 ∥ w ′ ∥ 2 when C V √ d δ ∥ w ′ ∥ 2 η ϵ α ,δ ≤ 1 3 , which is true when ϵ α is small enough.

According to Lemma 1, ϵ α ≤ 4 k 2 ϵ v ∗ , ϵ α → 0 as ϵ → 0 .

Thus, we have ∥ ∥ ̂ w ( δ ) ∥ ∥ 2 ≤ ∥ ̂ w 0 ∥ 2 ≤ 3 2 ∥ w ′ ∥ 2 and C V d δ ∥ ∥ ̂ w ( δ ) ∥ ∥ 2 η ϵ α ,δ ≤ 1 2 .

Combined with Eq (15), we get

Since C V ∥ w ′ ∥ 2 ≥ |⟨ V π 1 , w ′ ⟩| = 1 , by taking η ϵ α ,δ = 4 C α C 2 V d δ ϵ α δ 2 into the above equation, we have

which completes the proof.

G Proof of Lemma 3

Lemma 3. If | ̂ α i -α i | ≤ ϵ α and α i ≤ C α for all i ∈ [ d -1] , for every policy π and every δ ≥ 4 C 2 3 α C V d 1 3 ϵ 1 3 α , we have ∣ ∣ ̂ w · V π -̂ w ( δ ) · V π ∣ ∣ ≤ O (( √ k + 1) d -d δ C α ϵ ( δ ) ) , where ϵ ( δ ) = C α C 4 V d 3 2 δ ∥ w ′ ∥ 2 2 ϵ α δ 2 + √ kδ ∥ w ′ ∥ 2 is the upper bound in Lemma 2.

Proof. Given the output ( V π 1 , . . . , V π d ) of Algorithm 1, we have rank(span( { V π 1 , . . . , V π d δ } )) = d δ .

For i = d δ + 1 , . . . , d , let ψ i be the normalized vector of V π i 's projection into span( V π 1 , . . . , V π i -1 ) ⊥ with ∥ ψ i ∥ 2 = 1 .

Then we have that span( V π 1 , . . . , V π i -1 , ψ i ) = span( V π 1 , . . . , V π i ) and that { ψ i | i = d δ + 1 , . . . , d } are orthonormal.

For every policy π , the value vector can be represented as a linear combination of ̂ A 1 , . . . , ̂ A d δ , ψ d δ +1 , . . . , ψ d , i.e., there exists a unique a = ( a 1 , . . . , a d ) ∈ R d s.t. V π = ∑ d δ i =1 a i ̂ A i + ∑ d i = d δ +1 a i ψ i .

̸

Since ψ i is orthogonal to ψ j for all j = i and ψ i is are orthogonal to span( ̂ A 1 , . . . , ̂ A d δ ) , we have a i = ⟨ V π , ψ i ⟩ for i ≥ d δ +1 .

√

which implies that

Hence

This implies that

Since ̂ A ( δ ) ̂ w = ̂ A ( δ ) ̂ w ( δ ) = e 1 , we have term ( a ) = 0 .

We move on to bound term (c).

Note that the vectors { ψ i | i = d δ +1 , . . . , d } are orthogonal to span( V π 1 , . . . , V π d δ ) and that together with V π 1 , . . . , V π d δ they form a basis for span( { V π | π ∈ Π } ) .

Thus, we can let b i in the proof of Lemma 2 be ψ i + d δ .

In addition, all the properties of { b i | i ∈ [ d -d δ ] } also apply to { ψ i | i = d δ +1 , . . . , d } as well. Hence, similarly to Eq (9),

Consequentially, we can bound term ( c ) is by

since ∥ ∥ ̂ w ( δ ) ∥ ∥ 2 ≤ 3 2 ∥ w ′ ∥ 2 when C V √ d δ ∥ w ′ ∥ 2 η ϵ α ,δ ≤ 1 3 as discussed in the proof of Lemma 6. Now all is left is to bound term (b).

We cannot bound term (b) in the same way as that of term (c) because ∥ ̂ w ∥ 2 is not guaranteed to be bounded by ∥ w ′ ∥ 2 .

For i = d δ +1 , . . . , d , we define

For any i, j = d δ + 1 , . . . , d , ψ i is perpendicular to V π 1 , thus ∣ ∣ ∣ 〈 ψ i , ̂ A j 〉∣ ∣ ∣ = |⟨ ψ i , ̂ α j -1 V π 1 -V π j ⟩| = |⟨ ψ i , V π j ⟩| . Especially, we have

Let ̂ A ∥ i := ̂ A i -∑ d j = d δ +1 〈 ̂ A i , ψ j 〉 ψ j denote ̂ A i 's projection into span( ̂ A 1 , . . . , ̂ A d δ ) .

Since ̂ A i has zero component in direction ψ j for j > i , we have ̂ A ∥ i = ̂ A i -∑ i j = d δ +1 〈 ̂ A i , ψ j 〉 ψ j . Then, we have

where the first equation holds due to ̂ A ̂ w = e 1 .

By rearranging terms, we have

Recall that at iteration j of Algorithm 1, in line 8 we pick an orthonormal basis ρ 1 , . . . , ρ k +1 -j of span( V π 1 , . . . , V π j -1 ) ⊥ . Since ψ j is in span( V π 1 , . . . , V π j -1 ) ⊥ according to the definition of ψ j , |⟨ V π i , ψ j ⟩| is no greater then the norm of V π i 's projection into span( V π 1 , . . . , V π j -1 ) ⊥ .

Therefore, we have

where inequality (d) holds because π ρ l is the optimal personalized policy with respect to the preference vector ρ l , and Equation (e) holds due to the definition of u j (line 9 of Algorithm 1). Inequality (f) holds since ⟨ V π j , ψ j ⟩ is the norm of V π j 's projection in span( V π 1 , . . . , V π j -1 ) ⊥ and u j belongs to span( V π 1 , . . . , V π j -1 ) ⊥ .

By taking absolute value on both sides of Eq (17), we have

We can now bound ∣ ∣ ∣ ̂ w · ̂ A ∥ i ∣ ∣ ∣ as follows.

for some constant C ′ > 0 .

Eq (20) holds because ̂ A ( δ ) ̂ w = ̂ A ( δ ) ̂ w ( δ ) = e 1 and ̂ A ∥ i belongs to span( ̂ A ( δ ) ) . Eq (21) holds because ̂ w ( δ ) is the minimum norm solution to ̂ A ( δ ) x = e 1 , which implies that ̂ w ( δ ) · ψ i = 0 . The last inequality follows by applying Lemma 2.

We will bound ϵ i |⟨ ̂ w,ψ i ⟩| by induction on i = d δ +1 , . . . , d .

In the base case of i = d δ +1 ,

Then, by induction through Eq (19), we have for i = d δ +2 , . . . , d ,

Similar to the deviation of Eq (18), we pick an orthonormal basis ρ 1 , . . . , ρ k +1 -i of span( V π 1 , . . . , V π i -1 ) ⊥ at line 8 of Algorithm 1, then we have that, for any policy π ,

Then we have that term (b) is bounded by

Hence we have that for any policy π ,

H Proof of Theorem 2

Theorem 2. Consider the algorithm of computing ̂ A and any solution ̂ w ( δ ) to ̂ A ( δ ) x = e 1 for δ = k 5 3 ϵ 1 3 and outputting the policy π ̂ w ( δ ) = arg max π ∈ Π 〈 ̂ w ( δ ) , V π 〉 . Then the policy π ̂ w ( δ ) satisfies that v ∗ -〈 w ∗ , V π ̂ w ( δ ) 〉 ≤ O ( k 13 6 ϵ 1 3 ) .

Proof of Theorem 2. As shown in Lemma 1, we set C α = 2 k and have ϵ α = 4 k 2 ϵ v ∗ . We have ∥ w ′ ∥ = ∥ w ∗ ∥ ⟨ w ∗ ,V π 1 ⟩ and showed that ⟨ w ∗ , V π 1 ⟩ ≥ v ∗ 2 k in the proof of Lemma 1.

By applying Lemma 2 and setting δ = ( C 4 V k 5 ∥ w ∗ ∥ 2 ϵ v ∗ 2 ) 1 3 , we have

I Dependency on ϵ

In this section, we would like to discuss a potential way of improving the dependency on ϵ in Theorems 1 and 2.

Consider a toy example where the returned three basis policies are π 1 with V π 1 = (1 , 0 , 0) , π 2 with V π 2 = (1 , 1 , 1) and π 3 with V π 3 = (1 , η, -η ) for some η > 0 and w ∗ = (1 , w 2 , w 3 ) for some w 2 , w 3 .

The estimated ratio ̂ α 1 lies in [1 + w 2 + w 3 -ϵ, 1 + w 2 + w 3 + ϵ ] , and ̂ α 2 lies in [1 + ηw 2 -ηw 3 -ϵ, 1 + ηw 2 -ηw 3 + ϵ ] . Suppose that ̂ α 1 = 1 + w 2 + w 3 + ϵ and ̂ α 2 = 1 + ηw 2 -ηw 3 + ϵ . By solving

we can derive ̂ w 2 = w 2 + ϵ 2 (1 + 1 η ) and ̂ w 3 = w 3 + ϵ 2 (1 -1 η ) .

The quantity measuring sub-optimality we care about is sup π |⟨ ̂ w,V π ⟩ - ⟨ w ∗ , V π ⟩| , which is upper bounded by C V ∥ ̂ w -w ∗ ∥ 2 . But the ℓ 2 distance between ̂ w and w ∗ depends on the condition number of ̂ A , which is large when η is small. To obtain a non-vacuous upper bound in Section 3, we introduce another estimate ̂ w ( δ ) based on the truncated version of ̂ A and then upper bound ∥ ∥ ̂ w ( δ ) -w ∗ ∥ ∥ 2 and sup π ∣ ∣ ⟨ ̂ w,V π ⟩ -〈 ̂ w ( δ ) , V π 〉∣ ∣ separately.

However, it is unclear if sup π |⟨ ̂ w,V π ⟩ - ⟨ w ∗ , V π ⟩| depends on the condition number of ̂ A . Due to the construction of Algorithm 1, we can obtain some extra information about the set of all policy values.

First, since we find π 2 before π 3 , η must be no greater than 1 . According to the algorithm, V π 2 is the optimal policy when the preference vector is u 2 (see line 11 of Algorithm 1 for the definition of u 2 ) and V π 3 is the optimal policy when the preference vector is u 3 = (0 , 1 , -1) . Note that the angle between u 2 and V π 2 is no greater than 45 degrees according to the definition of u 2 . Then the values of all policies can only lie in the small box B = { x ∈ R 3 | ∣ ∣ u ⊤ 2 x ∣ ∣ ≤ |⟨ u 2 , V π 2 ⟩| , ∣ ∣ u ⊤ 3 x ∣ ∣ ≤ |⟨ u 3 , V π 3 ⟩|} . It is direct to check that for any x ∈ B , |⟨ ̂ w,x ⟩ - ⟨ w ∗ , x ⟩| < (1 + √ 2) ϵ . This example illustrates that even when the condition number of ̂ A is large, sup π |⟨ ̂ w,V π ⟩ - ⟨ w ∗ , V π ⟩| can be small. It is unclear if this holds in general. Applying this additional information to upper bound sup π |⟨ ̂ w,V π ⟩ - ⟨ w ∗ , V π ⟩| directly instead of through bounding C V ∥ ̂ w -w ∗ ∥ 2 is a possible way of improving the term ϵ 1 3 .

J Description of C4 and Proof of Lemma 4

Algorithm 4 C4: Compress Convex Combination using Carathéodory's theorem

1: input a set of k -dimensional vectors M ⊂ R k and a distribution p ∈ Simplex M

2: while | M | > k +1 do

3: arbitrarily pick k +2 vectors µ 1 , . . . , µ k +2 from M

4: solve for x ∈ R k +2 s.t. ∑ k +2 i =1 x i ( µ i ◦ 1) = 0 , where µ ◦ 1 denote the vector of appending 1 to µ

5: i 0 ← arg max i ∈ [ k +2] | x i | p ( µ i )

6: if x i 0 < 0 then x ←-x

7: γ ← p ( µ i 0 ) x i and ∀ i ∈ [ k +2] , p ( µ i ) ← p ( µ i ) -γx i

0

8: remove µ i with p ( µ i ) = 0 from M

9: end while

10: output M and p

Lemma 4. Given a set of k -dimensional vectors M ⊂ R k and a distribution p over M , C4 ( M,p ) outputs M ′ ⊂ M with | M ′ | ≤ k +1 and a distribution q ∈ Simplex M ′ satisfying that E µ ∼ q [ µ ] = E µ ∼ p [ µ ] in time O ( | M | k 3 ) .

Proof. The proof is similar to the proof of Carathéodory's theorem. Given the vectors µ 1 , . . . , µ k +2 picked in line 3 of Algorithm 4 and their probability masses p ( µ i ) , we solve x ∈ R k +2 s.t. ∑ k +2 i =1 x i ( µ i ◦ 1) = 0 in the algorithm.

Note that there exists a non-zero solution of x because { µ i ◦ 1 | i ∈ [ k +2] } are linearly dependent. Besides, x satisfies ∑ d +2 i =1 x i = 0 .

Therefore,

For all i , if x i < 0 , p ( µ i ) -γx i ≥ 0 as γ > 0 ; if x i > 0 , then x i p ( µ i ) ≤ x i 0 p ( µ i 0 ) = 1 γ and thus p ( µ i ) -γx i ≥ 0 .

Hence, after one iteration, the updated p is still a probability over M (i.e., p ( µ ) ≥ 0 for all µ ∈ M and ∑ µ ∈ M p ( M ) = 1 ). Besides, ∑ d +2 i =1 ( p ( µ i ) -γx i ) µ i = ∑ d +2 i =1 p ( µ i ) µ i -γ ∑ d +2 i =1 x i µ i = ∑ d +2 i =1 p ( µ i ) µ i .

Therefore, after one iteration, the expected value E µ ∼ p [ µ ] is unchanged.

When we finally output ( M ′ , q ) , we have that q is a distribution over M and that E µ ∼ q [ µ ] = E µ ∼ p [ µ ] .

Due to line 6 of the algorithm, we know that x i 0 > 0 . Hence p ( µ i 0 ) -γx i 0 = p ( µ i 0 ) -p ( µ i 0 ) x i 0 x i 0 = 0 .

We remove at least one vector µ i 0 from M and we will run for at most | M | iterations.

Finally, solving x takes O ( k 3 ) time and thus, Algorithm 4 takes O ( | M | k 3 ) time in total.

K Flow Decomposition Based Approach

We first introduce an algorithm based on the idea of flow decomposition.

For that, we construct a layer graph G = (( L (0) ∪ . . . ∪ L ( H +1) ) , E ) with H +2 pairwise disjoint layers L (0) , . . . , L ( H +1) , where every layer t ≤ H contains a set of vertices labeled by the (possibly duplicated) states reachable at the corresponding time step t , i.e., { s ∈ S| Pr( S t = s | S 0 = s 0 ) > 0 } .

Let us denote by x ( t ) s the vertex in L ( t ) labeled by state s .

Layer L ( H +1) = { x ( H +1) ∗ } contains only an artificial vertex, x ( H +1) ∗ , labeled by an artificial state ∗ .

For t = 0 , . . . , H -1 , for every x ( t ) s ∈ L ( t ) , x ( t +1) s ′ ∈ L ( t +1) , we connect x ( t ) s and x ( t +1) s ′ by an edge labeled by ( s, s ′ ) if P ( s ′ | s, π ( s )) > 0 . Every vertex x ( H ) s in layer H is connected to x ( H +1) ∗ by one edge, which is labeled by ( s, ∗ ) .

Wedenote by E ( t ) the edges between L ( t ) and L ( t +1) . Note that every trajectory τ = ( s 0 , s 1 , . . . , s H ) corresponds to a single path ( x (0) s 0 , x (1) s 1 , . . . , x ( H ) s H , x ( H +1) ∗ ) of length H +2 from x (0) s 0 to x ( H +1) ∗ .

This is a one-to-one mapping and in the following, we use path and trajectory interchangeably.

The policy corresponds to a ( x (0) s 0 , x ( H +1) ∗ ) -flow with flow value 1 in the graph G . In particular, the flow is defined as follows.

When the layer t is clear from the context, we actually refer to vertex x ( t ) s by saying vertex s .

For t = 0 , . . . , H -1 , for any edge ( s, s ′ ) ∈ E ( t ) , let f : E → R + be defined as

where q π ( τ ) is the probability of τ being sampled.

For any edge ( s, ∗ ) ∈ E ( H ) , let f ( s, ∗ ) = ∑ ( s ′ ,s ) ∈ E ( H -1) f ( s ′ , s ) . It is direct to check that the function f is a well-defined flow. We can therefore compute f by dynamic programming.

For all ( s 0 , s ) ∈ E (0) , we have f ( s 0 , s ) = P ( s | s 0 , π ( s 0 )) and for ( s, s ′ ) ∈ E ( t ) ,

Now we are ready to present our algorithm by decomposing f in Algorithm 5.

Each iteration in Algorithm 5 will zero out at least one edge and thus, the algorithm will stop within | E | rounds.

Algorithm 5 Flow decomposition based approach

1: initialize Q ←∅ .

2: calculate f ( e ) for all edge e ∈ E by dynamic programming according to Eq (23)

3: while ∃ e ∈ E s.t. f ( e ) > 0 do

4: pick a path τ = ( s 0 , s 1 , . . . , s H , ∗ ) ∈ L (0) × L (1) × . . . × L ( H +1) s.t. f ( s i , s i +1 ) > 0 ∀ i ≥ 0

5: f τ ← min e in τ f ( e )

6: Q ← Q ∪ { ( τ, f τ ) } , f ( e ) ← f ( e ) -f τ for e in τ

7: end while

8: output Q

Theorem 4. Algorithm 5 outputs Q satisfying that ∑ ( τ,f τ ) ∈ Q f τ Φ( τ ) = V π in time O ( H 2 |S| 2 ) .

The core idea of the proof is that for any edge ( s, s ′ ) ∈ E ( t ) , the flow on ( s, s ′ ) captures the probability of S t = s ∧ S t +1 = s ′ and thus, the value of the policy V π is linear in { f ( e ) | e ∈ E } .

The output Q has at most | E | number of weighted paths (trajectories). We can further compress the representation through C4, which takes O ( | Q | k 3 ) time.

Corollary 2. Executing Algorithm 5 with the output Q first and then running C4 over { (Φ( τ ) , f τ ) | ( τ, f τ ) ∈ Q } returns a ( k + 1) -sized weighted trajectory representation in time O ( H 2 |S| 2 + k 3 H |S| 2 ) .

We remark that the running time of this flow decomposition approach underperforms that of the expanding and compressing approach (see Theorem 3) whenever |S| H + |S| k 3 = ω ( k 4 + k |S| ) .

L Proof of Theorem 4

Theorem 4. Algorithm 5 outputs Q satisfying that ∑ ( τ,f τ ) ∈ Q f τ Φ( τ ) = V π in time O ( H 2 |S| 2 ) .

Proof. Correctness: The function f defined by Eq (22) is a well-defined flow since for all t = 1 , . . . , H , for all s ∈ L ( t ) , we have that

In the following, we first show that Algorithm 5 will terminate with f ( e ) = 0 for all e ∈ E .

First, after each iteration, f is still a feasible ( x (0) , x ( H +1) ) -flow feasible flow with the total flow out-of x (0) reduced by f τ . Besides, for edge e with f ( e ) > 0 at the beginning, we have f ( e ) ≥ 0 throughout the algorithm because we never reduce f ( e ) by an amount greater than f ( e ) .

Then, since f is a ( x (0) , x ( H +1) ) -flow and f ( e ) ≥ 0 for all e ∈ E , we can always find a path τ in line 4 of Algorithm 5.

Otherwise, the set of vertices reachable from x (0) through edges with positive flow does not contain x ( H +1) and the flow out of this set equals the total flow out-of x (0) . But since other vertices are not reachable, there is no flow out of this set, which is a contradiction.

In line 6, there exists at least one edge e such that f ( e ) > 0 is reduced to 0 . Hence, the algorithm will run for at most | E | iterations and terminate with f ( e ) = 0 for all e ∈ E .

Thus we have that for any ( s, s ′ ) ∈ E ( t ) , f ( s, s ′ ) = ∑ ( τ,f τ ) ∈ Q :( s τ t ,s τ t +1 )=( s,s ′ ) f τ .

Then we have

Computational complexity: Solving f takes O ( | E | ) time. The algorithm will run for O ( | E | ) iterations and each iteration takes O ( H ) time. Since | E | = O ( |S| 2 H ) , the total running time of Algorithm 5 is O ( |S| 2 H 2 ) . C4 will take O ( k 3 | E | ) time.

M Proof of Theorem 3

Theorem 3. Algorithm 2 outputs F ( H ) and β ( H ) satisfying that ∣ ∣ F ( H ) ∣ ∣ ≤ k + 1 and ∑ τ ∈ F ( H ) β ( H ) ( τ )Φ( τ ) = V π in time O ( k 4 H |S| + kH |S| 2 ) .

Proof. Correctness: C4 guarantees that ∣ ∣ F ( H ) ∣ ∣ ≤ k +1 .

We will prove ∑ τ ∈ F ( H ) β ( H ) τ Φ( τ ) = V π by induction on t = 1 , . . . , H .

Recall that for any trajectory τ of length h , J ( τ ) = Φ( τ )+ V ( s τ h , H -h ) was defined as the expected return of trajectories (of length H ) with the prefix being τ .

In addition, recall that J F ( t ) = { J ( τ ◦ s ) | τ ∈ F ( t ) , s ∈ S} and p F ( t ) ,β ( t ) was defined by letting p F ( t ) ,β ( t ) ( τ ◦ s ) = β ( t ) ( τ ) P ( s | s τ t , π ( s τ t )) .

For the base case, we have that at t = 1

Suppose that V π = ∑ τ ′ ∈ F ( t ) β ( t ) ( τ ′ ) J ( τ ′ ) holds at time t , then we prove the statement holds at time t +1 .

By induction, the statement holds at t = H by induction, i.e., V π = ∑ τ ∈ F ( H ) β ( H ) τ J ( τ ) = ∑ τ ∈ F ( H ) β ( H ) τ Φ( τ ) .

Computational complexity: Solving V π ( s, h ) for all s ∈ S , h ∈ [ H ] takes time O ( kH |S| 2 ) . In each round, we need to call C4 for ≤ ( k +1) | S | vectors, which takes O ( k 4 |S| ) time. Thus, we need O ( k 4 H |S| + kH |S| 2 ) time in total.

N Example of maximizing individual objective

Observation 2. Assume there exist k > 2 policies that together assemble k linear independent value vectors. Consider the k different policies π ∗ 1 , . . . , π ∗ k that each π ∗ i maximizes the objective i ∈ [ k ] . Then, their respective value vectors V ∗ 1 , . . . , V ∗ k are not necessarily linearly independent. Moreover, if V ∗ 1 , . . . , V ∗ k are linearly depended it does not mean that k linearly independent value vectors do not exists.

Proof. For simplicity, we show an example with a horizon of H = 1 but the results could be extended to any H ≥ 1 . We will show an example where there are 4 different value vectors, where 3 of them are obtained by the k = 3 policies that maximize the 3 objectives and have linear dependence.

Consider an MDP with a single state (also known as Multi-arm Bandit) with 4 actions with deterministic reward vectors (which are also the expected values of the 4 possible policies in this case):

Denote π a as the fixed policy that always selects action a . Clearly, policy π 1 maximizes the first objective, policy π 2 the third, and policy π 3 the second ( π 4 do not maximize any objective). However,

r (3) linearly depends on r (1) and r (2) as

In addition, r (4) is linearly independent in r (1) , r (2) : Assume not. Then, there exists β 1 , β 2 ∈ R s.t.:

Hence, the first equations imply β 2 = 1 -8 β 1 , and 4 β 1 +2 -16 β 1 = 3 , hence β 1 = -1 12 and β 2 = 5 3 . Assigning in the third equation yields -1 6 +5 = 2 which is a contradiction.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a multi-objective decision making framework that accommodates different user preferences over multiple objectives. Furthermore, the framework facilitates learning these preferences via policy comparisons. The underlying model is assumed to be a finite horizon Markov decision process, while the user preference relation is assumed to be a linear combination of objective values. The preference relation is subsequently learned using user feedback taking two forms: policy comparison, and the comparison of two small sets of representative trajectories, respectively. The paper then suggests an algorithm for finding near-optimal policies using a number of comparison queries that scales quasi-linearly with the number of objectives.\n\nSTRENGTHS:\nThe paper addresses an important problem in decision making under uncertainty with multiple objectives which has a wide range of potential applications.\n\nWEAKNESSES:\nHowever, the paper has several shortcomings. First, the presentation lacks a small running example and therefore it is quite difficult to follow the technical details of the proposed approach. Second, I was surprised by the abrupt ending of the paper (with Theorem 3) without any follow-up or discussion. It gives the impression that the submission was rushed right before the deadline which is kinda disappointing. Third, the paper lacks any empirical evidence of the proposed approach and therefore we don't have any idea whether the approach is practical or not. For example, if the MDP has a very complex state (and/or action) space, how would one best represent and present the trajectories to the user to elicit feedback. While it seems straightforward in a toy-like setting, I don't think the preference elicitation scheme based on trajectories is actually practical in a real setting.\n\nQUESTIONS:\nNo specific questions.\n\nLIMITATIONS:\nI found the paper closely related to previous work by (Marinescu et al, 2017) which also looks at multi-objective decision making in the case of influence diagram models (which are actually more general than finite-horizon MDPs) and an unknown linear preference relation between objectives. The goal however, is slightly different, in that it is concerned with finding possibly optimal policies (which are optimal under at least one instantiation of the preference vector).\n\n(Marinescu et al, 2017) Marinescu, R., Razak, A. and Wilson, N. 'Multi-Objective Influence Diagrams with Possibly Optimal Policies', Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17), San Francisco, California, USA, 4 – 9 February, pp. 3783 - 3789.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach to elicit the unknown linear weights used to aggregate multi-objective valuations in sequential decision-making problems. The authors first consider comparative queries where a user can compare policies (instead of trajectories as in previous work). They investigate then the case where a policy can be replaced and approximated by a set of weighted trajectories.\n\nSTRENGTHS:\nThe paper investigates a novel approach for preference elicitation in multi-objective problems.\n\nThe authors obtain upper-bounds on the number of queries needed to obtain a near optimal policy for a given user.\n\nThe approach scales well with the number of users.\n\nWEAKNESSES:\nI think the obtained results are mostly of theoretical interest, because the type of considered queries is not very practical, in my opinion. Directly comparing policies is naturally usually not possible for humans, which motivates the second case investigated in this paper. However, even comparing two sets of weighted trajectories would be very hard for humans. \n\nIt is not clear if the proposed approach based on geometric and algebraic results can work in presence of noisy feedback from the user, which is a more realistic setting.\n\nThe writing, while reasonably clear, seems to be a bit rushed. Notably, the paper ends abruptly, without any conclusion.\n\nQUESTIONS:\n1) In footnote 1, why do the authors mention \"nearly\"? I think under the assumptions they make, optimal policies for every possible user preferences are in the set of Pareto optimal solutions.\n\n2) Are there situations where comparing sets of weighted trajectories would be practical for humans?\n\n3) Why do you not assume that the weight vector w^* is normalized since dividing by a positive value doesn't change the preferences over policies?\n\n4) Can the proposed technique be adapted if there is noise in the user's comparisons?\n\nSome minor issues:\n- line 107: V(\\pi) -> V^\\pi\n- line 128: In line 96, policies are introduced as stochastic, which seems to contradict with the explicit policy representation.\n- line 259: vacuous -> non-vacuous?\n\nLIMITATIONS:\nNot applicable, this is a theoretical paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

comments: SUMMARY:
The paper provides a multi-objective decision-making framework to learn approximate optimal personalized policies by eliciting user preferences via comparative queries. The authors characterize theoretically two feedback models, explicit queries and weights queries sets, and derive sound algorithms with proven upper bounds on the learning complexity.

STRENGTHS:
To the best of my knowledge, the proposed algorithms and theoretical analysis are novel and interesting. Eliciting multi-objective user preferences over policies has a tremendous impact on many (real-world) settings in which decision-making systems aid humans in daily tasks. The paper is well-written. All the theoretical assumptions, results and algorithms are well-motivated and clearly described. I checked the proofs (and skimmed through the appendix), but a more rigorous analysis would require more time.

WEAKNESSES:
In my opinion, I can see two main areas of improvement (1) extending the algorithm to noisy feedback from the users and (2) providing some empirical evidence about the effectiveness of the proposed algorithms.

Lastly, the paper is dense with information, but it ends abruptly. I would have enjoyed a paragraph either summarizing the main findings or expanding the current research to consider its potential practical impact (e.g., challenges in implementation).

QUESTIONS:
- If we consider human-decision making, do you think the two proposed feedback models are realistically usable by humans or do you think they might work well with agents, for example, in cooperative AI settings?

LIMITATIONS:
The authors adequately address all the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n_**Edit:** I have read the authors' rebuttal, and they have addressed all the concerns raised in my review._\n\nThe paper considers the problem of multi-objective reinforcement learning, when there is an unknown linear preference over the objectives, and approximate preference comparisons can be performed. The goal is to find an optimal policy (according to the true preference) while minimizing the number of comparisons.\n\nThe main task is identifying the unknown preference, which the paper shows how to do using Algorithm 1, Algorithm 3, and Theorem 1. Once the preference is (approximately) identified, Theorem 1 shows that solving the RL problem for this approximate preference returns a policy that is competitive with the optimal policy for the true preference.\n\nAdditionally, the paper also considers how preference comparisons could be implemented. One option is to ask the user to compare two policies directly, which might be hard to do. Because of this, the paper considers the case when the user can compare trajectories from two policies, which might be more interpretable to the user. To this end, the paper shows how to efficiently find a small (O(k)) weighted set of trajectories such that their weighted values are equal to the value of the policy (Algorithm 2).\n\nSTRENGTHS:\n- The procedure for identification of linear preference is not limited to just RL and can be used in a wider setting, making it of general interest.\n- Algorithm 1 adapts to $d \\le k$, which could be helpful in cases when there are many redundant objectives.\n- The weighted trajectory set representation of a policy is novel, interpretable, and possibly of wider interest.\n\nWEAKNESSES:\n- Line 220: “we only need to run Algorithm 1 once for all users”. This seems to be incorrect, as Algorithm 1 uses preference comparisons which depend on the user. This means the leading factor in the bound on line 256 should change from $(k^2+n)$ to $n (k^2 + 1)$.\n- The paper assumes the existence of a “do nothing” action, which might require a modified MDP. While this appears straightforward, it seems to me it requires domain-specific knowledge in order to modify it such that the “do nothing” policy has minimal value (which is the true requirement). E.g. in the chess example, perhaps refusing to play is considered a draw, which is better than a loss. Thus, a “do nothing” action would need to somehow always lose.\n\nQUESTIONS:\n- Proof of Lemma 1: in the first inequality, shouldn't it be possible to reduce $k \\epsilon$ to $\\epsilon$? This would make Lemma 1 applicable to when $\\epsilon \\le v^\\*/(2k)$ rather than $v^*/(2k^2)$.\n- Is it possible to approximately estimate $\\epsilon$ by comparing $\\pi^1$ against a scaled version of itself (similar to the binary search in Algorithm 3)? If $\\hat{\\epsilon} \\approx \\epsilon$ can be estimated this way, can Theorem 2 be used with $\\delta = k^{5/3} \\hat{\\epsilon}^{1/3}$, avoiding the problem of having to guess the hyperparameter $\\delta$?\n- Have you considered the case when the user can compare policy value vectors? In this case, we can query with “artificial” values directly (e.g. basis vectors), requiring no interaction with the MDP at all. Perhaps this makes the problem too simple?\n- The preference identification procedure can be extended to general, non-tabular RL case. Is the same true for efficiently finding weighted trajectory sets?\n\nLIMITATIONS:\nThe paper requires that preferences are linear, which is a fairly strong assumption. However, it is a good first step.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-18"}

comments: SUMMARY:
This paper proposes a method for multi-objective RL considering two user feedback models and thereby the first algorithms with theoretical guarantees for personalized MO decision making via policy comparison.

STRENGTHS:
The paper prpoposes an interesting method for finding a personalized optimal policy for MORL problems reflecting users' unknown preferences over the objectives. It is very well written and scienitifically sound, motivates the work well. Method is presented clearly and carefully. The novelty and significance of the method are well presented.

WEAKNESSES:
The paper lacks conclusions and discusssion about limitations completely and thereby its utility is left a bit open.  The paper also emphasizes that it is important to present the obtained trajectories in an easy format to the user, but does not give any examples how this would be achieved.

QUESTIONS:
You state that the method learns a user's preference vector using minimal number of queries, and C4 is used when the number is small, but discussion of what is small in the number of trajectories or minimal for queries?

Could you give an intuitive example of how are the easily interpretable queries presented to the user.

What are the limitations of the method?

LIMITATIONS:
Not really addressed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-19