In this paper, we present a link between preference-based and multiobjective sequential decision-making. While transforming a multiobjective problem to a preference-based one is quite natural, the other direction is a bit less obvious. We present how this transformation (from preference-based to multiobjective) can be done under the classic condition that preferences over histories can be represented by additively decomposable utilities and that the decision criterion to evaluate policies in a state is based on expectation. This link yields a new source of multiobjective sequential decision-making problems (i.e., when reward values are unknown) and justifies the use of solving methods developed in one setting in the other one.
Multiobjective MDP (MOMDP) @cite_11 is an MDP @math where the reward function is redefined as @math with @math being the number of objectives. The value function @math of a policy @math is now vectorial and can be computed as the limit of the vectorial version of ) and ): @math , In MOMDP, the value function of policy @math Pareto-dominates that of another policy @math if in every state @math , @math is not smaller than @math on every objective and @math is greater than @math on at least one objective. By extension, we say that @math Pareto-dominates @math if value function @math Pareto-dominates value function @math . A value function (resp. policy) is Pareto-optimal if it is not Pareto-dominated by any other value function (resp. policy). Due to incomparability of vectorial value functions, there are generally many Pareto-optimal value functions (and therefore policies), which constitutes the main difficulty of the multiobjective setting.
@cite_11
2102660061
Decomposition-based algorithms have become increasingly popular for evolutionary multiobjective optimization. However, the effect of scalarizing methods used in these algorithms is still far from being well understood. This paper analyzes a family of frequently used scalarizing methods, the @math methods, and shows that the @math value is crucial to balance the selective pressure toward the Pareto optimal and the algorithm robustness to Pareto optimal front (PF) geometries. It demonstrates that an @math method that can maximize the search ability of a decomposition-based algorithm exists and guarantees that, given some weight, any solution along the PF can be found. Moreover, a simple yet effective method called Pareto adaptive scalarizing (PaS) approximation is proposed to approximate the optimal @math value. In order to demonstrate the effectiveness of PaS, we incorporate PaS into a state-of-the-art decomposition-based algorithm, i.e., multiobjective evolutionary algorithm based on decomposition (MOEA D), and compare the resultant MOEA D-PaS with some other MOEA D variants on a set of problems with different PF geometries and up to seven conflicting objectives. Experimental results demonstrate that the PaS is effective. This paper is devoted to fair optimization in Multiobjective Markov Decision Processes (MOMDPs). A MOMDP is an extension of the MDP model for planning under uncertainty while trying to optimize several reward functions simultaneously. This applies to multiagent problems when rewards define individual utility functions, or in multicriteria problems when rewards refer to different features. In this setting, we study the determination of policies leading to Lorenz-non-dominated tradeoffs. Lorenz dominance is a refinement of Pareto dominance that was introduced in Social Choice for the measurement of inequalities. In this paper, we introduce methods to efficiently approximate the sets of Lorenz-non-dominated solutions of infinite-horizon, discounted MOMDPs. The approximations are polynomial-sized subsets of those solutions. Consider the problem of approximating the optimal policy of a Markov decision process (MDP) by sampling state transitions. In contrast to existing reinforcement learning methods that are based on successive approximations to the nonlinear Bellman equation, we propose a Primal-Dual @math Learning method in light of the linear duality between the value and policy. The @math learning method is model-free and makes primal-dual updates to the policy and value vectors as new data are revealed. For infinite-horizon undiscounted Markov decision process with finite state space @math and finite action space @math , the @math learning method finds an @math -optimal policy using the following number of sample transitions @math where @math is an upper bound of mixing times across all policies and @math is a parameter characterizing the range of stationary distributions across policies. The @math learning method also applies to the computational problem of MDP where the transition probabilities and rewards are explicitly given as the input. In the case where each state transition can be sampled in @math time, the @math learning method gives a sublinear-time algorithm for solving the averaged-reward MDP. We consider multiobjective and parametric versions of the global minimum cut problem in undirected graphs and bounded-rank hypergraphs with multiple edge cost functions. For a fixed number of edge cost functions, we show that the total number of supported non-dominated (SND) cuts is bounded by a polynomial in the numbers of nodes and edges, i.e., is strongly polynomial. This bound also applies to the combinatorial facet complexity of the problem, i.e., the maximum number of facets (linear pieces) of the parametric curve for the parametrized (linear combination) objective, over the set of all parameter vectors such that the parametrized edge costs are nonnegative and the parametrized cut costs are positive. We sharpen this bound in the case of two objectives (the bicriteria problem), for which we also derive a strongly polynomial upper bound on the total number of non-dominated (Pareto optimal) cuts. In particular, the bicriteria global minimum cut problem in an n-node graph admits @math O(n3logn) SND cuts and @math O(n5logn) Pareto optimal cuts. These results significantly improve on earlier graph cut results by Mulmuley (SIAM J Comput 28(4):1460---1509, 1999) and Armon and Zwick (Algorithmica 46(1):15---26, 2006). They also imply that the parametric curve and all SND cuts, and, for the bicriteria problems, all Pareto optimal cuts, can be computed in strongly polynomial time when the number of objectives is fixed.