Title: Beyond Scalar Rewards: An Axiomatic Framework for Lexicographic MDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: expected utility theory

lexicographic expected utility

lexicographic MDP

axiomatic framework

reward hypothesis

reinforcement learning theory
Abstract: Recent work has formalized the reward hypothesis through the lens of expected utility theory, by interpreting reward as utility. Hausner's foundational work showed that dropping the continuity axiom leads to a generalization of expected utility theory where utilities are lexicographically ordered vectors of arbitrary dimension. In this paper, we extend this result by identifying a simple and practical condition under which preferences in a Markov Decision Process (MDP) cannot be represented by scalar rewards, necessitating a 2-dimensional reward function. We provide a full characterization of such reward functions, as well as the general d-dimensional case under a memorylessness assumption on preferences. Furthermore, we show that optimal policies in this setting retain many desirable properties of their scalar-reward counterparts, while in the Constrained MDP (CMDP) setting — another common multiobjective setting — they do not.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper takes an axiomatic approach to define Lexicographic MDPs (LMDPs) and proves key properties that LMDPs satisfy. The resulting framework enjoys desirable features such as the existence of uniformly optimal deterministic policies, something not guaranteed in CMDPs. One limitation, as noted in Section 8, is the absence of algorithms for solving the proposed LMDP framework.\n\nThough I scored 3, I'm willing to raise the score if my concerns are addressed.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe paper is generally well-written, and the theoretical results seem solid.\nWhile no algorithm is provided, I believe that this axiomatic framework lays a strong foundation for future research on LMDPs.\n\n**Weaknesses:**\n\nThe significance of the results is not well articulated.\n\nWhat specific types of problems can this framework model compared to existing ones, like CMDPs?\nIf the LMDP framework can represent the same kinds of problems as CMDPs while guaranteeing a uniformly optimal deterministic policy, that would be a nice contribution.\nHowever, I could not find any concrete examples illustrating such cases in the paper.\nAlthough Figure 1 illustrates an example of an LMDP, it is not clearly explained how this example differs from what can be modeled by existing frameworks.\n\nExplicitly identifying problems that the proposed framework can capture but existing works cannot would strengthen the paper’s contribution.\n\nQUESTIONS:\n* In line 124, what are the \"other methods\" being referred to?\n* In line 36:\n\n> Another common and practical example is when the first priority is to achieve some goal while the second priority is to minimize the time that it takes. Such objectives are also reminiscent of Asimov’s Three Laws of Robotics (Asimov, 1942), where the highest priority objective is to “not injure a human being or, through inaction, allow a human being to come to harm.”\n\nWhy is minimizing the time to achieve a goal aligned with Asimov's example? The connection is not clear to me.\n\n======\n\n* What does the $Q^\\star$ in Theorem 7 refer to? I think the $Q$ function is not defined for LMDPs. \n* Do you conjecture that the proposed LMDP framework satisfies a Bellman-type equation? The existence of a deterministic and uniformly optimal policy seems to suggest that such an equation might exist, which could be useful for constructing a dynamic programming algorithm.\n\nLIMITATIONS:\nThe limitations are well described in Section 8.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors provide an extension of Markov Decision Processes (MDPs) that supports lexicographic objectives. They take an axiomatic approach by describing criteria over preferences between histories (completeness, transitivity, independence, memorylessness, and temporal $\\gamma$-indifference) and providing a representation result in the form of a linear lexicographic utility function (theorem 3 and 4). They additionally propose a \"safety first\" axiom that leads to a 2-dimensional lexicographic utility (theorem 5) which they extend to the sequential setting (theorem 6). Finally, they prove a \"fundamental theorem of LMDPs\" (theorem 7) that shows a relationship between optimal policies and the (lexicographically) optimal value function, analogous to that of MDPs, but, notably, unlike constrained MDPs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- This is a high quality paper: the technical results are clearly formulated and appear correct to me.\n- The paper is clearly written: the authors do a good job of building up the results and connecting their framework/results to existing framework/results\n\nWeaknesses\n- Although the significance of the paper is gestured towards by comparing LDMPs with CMDPs and some comparision was done in the Appendix, I felt that the authors did not convincingly demonstrate how LMDPs can be used in place of scalar MDPs or CMDPs. Axiomatic definition of new classes of MDPs is a valuable exercise, however, there are practical reasons why the community uses MDPs as a standard model of sequential decision-making (e.g., we have lots of algorithms for solving MDPs). Unfortunately, the authors do not do very much to convince readers of the practical significance of not using LMDPs. In particular, I would have liked to see an LMDP of some reasonable difficultly actually be solved (e.g., a tabular and/or deep RL setting), either using a version of an MDP algorithm or a new algorithm. Even better would be to see how the CMDP formulation of the same problem leads to issues that LMDPs avoid. Currently, the authors only present a very simple lexicographic planning problem in Figure 1 and one in the appendix.\n-  Relatedly, the \"single unsafe utility\" formulation is interesting and seems promising, but this is a single specific case of the more general formulation that seems like a good parameterization but isn't empirically demonstrated. \n- The results presented are somewhat original. As far as I'm aware, there is no axiomatic derivation of lexicographic MDPs in the literature. However, the theoretical results seem to follow naturally from combining well-established results (e.g., expected utility, lexicographic expected utility, Bowling et al (2023)'s results on settling the reward hypothesis). In the end, the results feel like a straightforward exploration of a new way to specify sequential decision-making problems, but don't make the case for using this approach over any other in practice.\n\nQUESTIONS:\n1. Do standard MDP algorithms work with LMDPs? What changes have to be made?\n2. Have the authors tested methods for solving LMDPs? Are there any challenges not found in the standard MDP setting?\n3. Can we state more formally what classes of problems are solved by LMDPs but not CMDPs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper contributes to the theory of lexicographic Markov decision processes, where the objective is to maximize a d-dimensional vector lexicographically. The paper starts by giving a background on:\n- von-Neumann-Morgenstern (vNM) expected utility theorem: only linear utility functions satisfy the four vNM axioms\n- if we let go of the “continuity” axiom, Hausner 1954 characterization theorem shows that we’ll have a $d$-dimensional linear *lexicographic* utility function for some d\nThe RL/MDP theory has largely focused on vNM utilities, and this paper lays a more generalized theory for lexicographic MDPs.\n\nBuilding on Hausner 1954, the paper proposes a memorylessness axiom (extension of Shakerinava and Ravanbakhsh, 2022), and uses this axiom along with three of the vNM axioms to prove a characterization of linear lexicographic utility functions for MDPs or sequential decision making. Similar to the discounted reward case with the vNM utilities, the paper defines a discounted reward version and proves a characterization using another axiom it introduces.\n\nNext, as an example, the paper investigates a setting where there are unsafe outcomes with infinite cost, which breaks the continuity assumption. The paper proves that this cannot be captured by vNM utilities and requires a 2-dimensional linear lexicographic utility function. The paper extends this example to the sequential setting as well.\n\nFinally, the paper proves that for lexicographic MDPs, greedy following the adapted version of Q-value function is optimal and a stationary deterministic optimal policy exists.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper is written and presented very nicely. I appreciate the clarity. Though, none of the proofs appear in the main body. Though, the paper mentions that a reader knowledgeable of the proofs follow a structure similar to that of the scalar-reward models.\n- The axiomatic analysis and the characterizations of lexicographic utility functions for MDPs is new --- though lexicographic objectives in MDPs have been investigated before.\n- I haven’t checked the proofs, but the logical path of the theorems building on top of each other seems very reasonable and sound.\n- The example with the unsafe outcome nicely illustrates that one needs lexicographic optimization. \n\nWeaknesses:\nI don’t see any critical problems, and I appreciate how clean everything is presented in the paper --- though, I haven’t checked any of the proofs yet.\n\nI’m not currently giving a higher score mainly due to that, as I understand, the paper also fairly admits that many of the proofs and ideas are taken from prior work but nicely put together in this work for the sequential decision making setting.\nThe paper itself also mentions the algorithmic/computational challenges of solving lexicographic MDPs given the existing literature.\nIt would’ve been ideal to see some new (algorithmic/computational) insight taken away from the paper that can be used to devise new algorithms or perhaps solve or better formalize new problems. Though, I think there is value in the characterizations and the defined axioms.\n\nQUESTIONS:\nNone at the moment.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies reinforcement learning (i.e. sequential decision-making) under lexicographic preferences functions. The authors introduce the Lexicographic MDP (LMDP), and their contributions are to extend classical expected utility theory of von Neumann & Morgenstern and of Hausner to the sequential setting. Under one additional axiom that they argue is natural for the sequential setting (Memorylessness), they establish an analogue of Hausner's lexicographic expected utility theorem -- this gives rise to vector-valued utility functions taking a recursive form similar to the classical value functions. The authors then extend this to the setting of sequential decision-making with safety constraints, establishing a \"safe\" version of the lexicographic relations with an additional axiom (Safety First). It is shown that LMDPs with the Safety First axiom (and one additional technical axiom related to discount factors) have a unique globally optimal policy, similar to MDPs and in contradistinction with the Constraint MDPs.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a theoretically interesting paper on the foundational aspects of decision-making with utilities. The motivation for studying this setting is the case when different vector-value outcomes can not be compared in a scalar fashion, i.e. there exists a strict ordering between the indices of the vector-valued outcome (maximizing the first index of the outcome takes highest precedence over any other index). The main application area that I can see is for providing a different formalism for safe decision-making. Accordingly, the authors study this case in the paper, with the axiom that there exists a single \"infinitely bad\" outcome that must be avoided at all costs. \n\nThe axioms the authors introduce are natural. The writing and presentation itself is very clear in the main text. One suggestion would be to review the abstract: it is vague does not really convey or accurately discuss the technical contributions of the paper.\n\nAs for what's missing:\n- It would have been of high value to provide more discussion and perhaps some examples on how this \"safe lexicographic decision-making\" formalism differs from the Constrained MDP. Namely, is there a toy problem or perhaps a real-life example where one should use this formalism as opposed to the Constraint MDP? The authors do establish that they differ on at least one technical level (behaviours of their optimal policies), but this does not do enough to separate the two settings. \n- Beyond this, as the authors acknowledge, it would have been interested to have some algorithmic developments (how to plan/solve the value/utility functions whose structure they have spent time working out).\n- Lastly, there seems to be some missing discussion on how this work differs from related work *on a technical level*. The authors mention the works of (Shakerinava & Ravanbakhsh, 2022) and (Bowling et al., 2023) but not precisely how their results are similar or different. From a quick glance, the contents seem to be highly related. For example, Theorem 4.2 of (Shakerinava & Ravanbakhsh, 2022) studies the exact same axioms as Theorem 3 here except with the addition of the Continuity axiom, and derives an analogous conclusion. Similarly, Theorem 4.1 of (Bowling et al., 2023) is related to Theorem 4 here but without the sequential structure, and derives an analogous conclusion. This should be clearly discussed and at a glance reduces the novelty of the contributions in this paper. \n\nI am leaving a weak accept for this paper, but would be happy to revise my score after the discussion if it turns out that I have been mistaken (or if the authors revise some of the discussion, if needed).\n\nSmaller complaints:\n- Please define lexmax_a in Theorem 7.\n\nQUESTIONS:\n1. Differentiation from Prior Work: Can you provide a detailed technical comparison explaining how your contributions differ from these prior results beyond the tweaking of the axioms? What specific novel insights does your work provide that these earlier papers did not establish?\n2. Scope of Safety Applications: Can you elaborate on the types of AI safety problems where your framework would be most appropriate and when one would choose this formalism over the Constrained MDP (or vice-versa)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}