Title: Planning and Learning in Average Risk-aware MDPs
Venue: NeurIPS 2025
Accepted: yes
Keywords: average cost markov decision processes

dynamic risk measure

relative value iteration

Q-learning

reinforcement learning
Abstract: For continuing tasks, average cost Markov decision processes have well-documented value and can be solved using efficient algorithms. However, it explicitly assumes that the agent is risk-neutral. In this work, we extend risk-neutral algorithms to accommodate the more general class of dynamic risk measures. Specifically, we propose a relative value iteration (RVI) algorithm for planning and design two model-free Q-learning algorithms, namely a generic algorithm based on the multi-level Monte Carlo (MLMC) method, and an off-policy algorithm dedicated to utility-based shortfall risk measures. Both the RVI and MLMC-based Q-learning algorithms are proven to converge to optimality. Numerical experiments validate our analysis, confirm empirically the convergence of the off-policy algorithm, and demonstrate that our approach enables the identification of policies that are finely tuned to the intricate risk-awareness of the agent that they serve.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses risk-aware average MDPs. First, the authors propose a risk-aware relative value iteration algorithm (model-based) to solve average risk-aware MDPs when considering dynamic risk measures. Then, the authors introduce two model-free algorithms and theoretically analyze them. The authors empirically show the algorithms converge to the right quantities and run them in three environments considered by previous works in the field of risk-aware MDPs.\n\nSTRENGTHS AND WEAKNESSES:\nThe clarity of the paper could be improved, particularly in the introduction. There are some sentences that are a bit unclear. E.g. \"(...) the randomness of the environment is usually unknown, highlighting the importance of developing model-free learning algorithms.\" (line 37-38). However, I do not believe this should be a reason to reject the paper, but I ask the authors to review their introduction. The contributions are clearly stated. The assumptions are stated and discussed thoughout the document.\n\nThe organization of the paper could be slightly improved. In particular, I suggest moving Sec. 5.2 (which introduces a new method) to Sec. 4, and keep only the experiments in Sec. 5.\n\nThe experimental section of the paper (and respective discussion) should be further extended. First, in Tab. 1, I did not understand what is the exact difference between the \"optimal risk\" and the \"expectile risk\" column. Could the authors clarify? Maybe this should be made clearer in the text. Second, since the authors propose multiple algorithms, why don't the authors run them and show that they achieve similar performances?\n\nThe paper seems original, though I am not familiarized with the previous works cited.\n\nRegarding the quality and correctness of the paper, I cannot provide an accurate assessment of them as I am not familiarized with the previous works cited by the authors. The paper is quite technical and, here and there, I got a bit lost reading it, but I believe it was likely because I miss some background knowledge regarding some of the concepts used by the authors.\n\nQUESTIONS:\n- What does the paragraph that starts in line 48 has to do with the present work? One thing is risk-awareness (risk-aware MDPs), other thing is robustness to unknown dynamics (distributionally robust MDPs). Please clarify it as it is not clear why such cited works matter for the context of risk-awareness.\n\n- Maybe the authors could add a bit more about the differences between the dynamic risk measures considered in this work and standard risk measures applied to the random return yielded by the interaction between a given policy and the MDP. In the later setting it is known that Markovian policies do not suffice for optimality, so I was wondering if the authors could provide more details regarding the difference between the two formulations and why they decided to go with the former (i.e., why do they decided to consider dynamic risk measures).\n\n- \"Remark 2.15. Assumption 2.12 is a sufficient condition for the existence of an optimal average risk independent of the initial state and is stronger than the unichain assumption commonly used in risk neutral average MDPs.\" - How easy is it to verify whether this assumption holds in practical problems? I ask this question because even verifying whether the MDP is unichain is already a hard task.\n\n- In Tab. 1, I did not understand what is the exact difference between the \"optimal risk\" and the \"expectile risk\" column. Could the authors clarify? \n\nMinor comments/typos:\n- \"The condition we impose for convergence are weak\" -> \"conditions\"? (line 65); \"In the follows, we (...)\" (line 101).\n\nLIMITATIONS:\nThe authors discuss the limitations of their assumptions throughout the work. Some of them are standard (e.g., step size assumptions), while others seem more stringent. Unfortunately, I lack the background knowledge to assess the impact of some of the assumptions used.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the risk-aware average-cost MDPs, including the formulation of the Bellman equation, the required assumptions for the problem to be well-posed, and the formulation of the Q-function to be used for model-free reinforcement learning. Essentially, these are the theoretical foundations for RL algorithms to be used in risk-aware average-cost MDPs. \n\nThere are papers studying a special type of risk-aware measures in average-cost MDPs, and this paper generalizes the risk measures that can be used.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- I enjoy reading the paper because it is very well written. The concepts, notation, analyses, and results are clearly presented. \n- The discussion on the connection to the literature is very thorough. My understanding is that risk-aware finite-horizon or discounted MDPs have been studied; risk-neural average-cost MDPs have been studied; and this is one of the first papers that studies the general risk-aware average-cost MDPs. \n\nWeaknesses\n- I wish there are more discussions on the challenges in the formulation and analyses of risk-aware average-cost MDPs. It is understood that the average-cost formulation is harder to analyze than the discounted, due to a lack of contraction. What additional difficulties are introduced when risk-aware objectives are introduced versus the risk-neutral version? Will the challenges persist for other classes of RL algorithms such as value/policy approximation, policy gradient, etc? Such a discussion can help position this paper and its contribution.\n\nQUESTIONS:\nI have one question: in the analysis of average-cost MDPs, many of the studies use a limit argument. That is, use the discounted formulation and then take the discount factor to 1. This paper doesn't use this type of analysis. Why is that and what type of results would need such an analysis?\n\nLIMITATIONS:\n- The paper doesn't investigate other types of RL algorithms and discuss their applicability in the risk-aware average-cost setting.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of planning in average-cost MDPs with general dynamic risk measures. The authors propose three algorithms: a model-based risk-aware relative value iteration (RVI) algorithm, a model-free Q learning algorithm generalizing MLMC, and a model-free asynchronous off-policy algorithm for UBSR measures. The authors show theoretical convergence for RVI and MLMC-based, while the last algorithm is evaluated empirically on three problems: machine replacement, water reservoir management, and inventory management.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths:*\nThe paper addresses an important problem by addressing average-cost MDPs with a broad class of dynamic risk measures. The introduced algorithms are novel and of practical relevance. For two algorithms, there are rigorous formal proofs for convergence (I could not verify them in detail, though). Generally, the paper is well-written and organized.\n\n*Weaknesses:*\nConvergence of the UBSR algorithm is only shown empirically, though this seems to be the most important one. The differences in applicability between the proposed algorithms could be stated more clearly in the introduction (under which circumstances, which algorithm should one use). Assumptions 2.12 and 3.1 seem quite restricting to me. While the paper presents one model-based and two model-free algorithms, section 3 \"model-based algorithms\" introduces two algorithms, section 4 \"model-free algorithms\" just one algorithm, which is confusing to me.\nI see the main issue with this paper in the evaluation. The application to practical problems (machine replacement, water reservoir management, inventory management) is only briefly discussed, and all these systems are very small (and this limitation is not discussed). I do not understand which of the algorithms is applied for the results in Table 1 and why not all of them are applied. The results are not discussed at all, just some numbers are stated that \"confirm [their] theory, proving [their] effectiveness\". There is (almost) no qualitative discussion of the results, showing that the problem formulations and the solutions provided by the algorithms are sensible. At least, in C4 there are average costs of simulated trajectories for the inventory management problem provided, but this also lacks a detailed discussion besides stating that \"the risk-averse policy produces trajectories with lower variance\". It is also not clear there how the risk-averse policy is computed. There are no comparisons to existing risk-sensitive RL algorithms. Furthermore, the numerous abbreviations are confusing, e.g., for tasks. In many places, such as the discussion in the appendix, there would be enough space to state the full task title, which would enhance comprehensibility.\n\n*Minor Issues / Typos*\n- L. 8 \"utility-based\"\n- L. 24 \"MDPs\"\n- L. 81 \"the the\"\n- L. 101 \"in the follows\"\n- L. 147 \n- L. 248 \"a\"\n\nQUESTIONS:\n- Could you clarify under which practical conditions each algorithm is most applicable? For example, when should one prefer the MLMC Q-learning algorithm over the UBSR algorithm?\n- How limiting is Assumption 2.12 in real applications? Can it be weakened or checked easily?\n- Which algorithm was used to produce the values shown in Table 1?\n- In Table 1, what exactly is meant by \"optimal risk\" and \"expectile risk\"? Are these computed under the same policy or under different ones?\n- Is there a plan or path to proving convergence for the UBSR Q-learning algorithm?\n- Do your algorithms scale to real-world application systems?\n\nLIMITATIONS:\nSome limitations are very briefly discussed in the conclusion section, others (such as limitation to low-dimensional systems) are not.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of solving Markov Decision Processes (MDPs) under average cost criteria when using general dynamic risk measures. In the planning setting, the authors introduce a relative value iteration (RVI) algorithm that accommodates an extremely broad class of one-step risk mapping. In the learning setting, they extend both multi-level Monte Carlo (MLMC) techniques and an asynchronous simulation-based algorithm tailored to the UBSR (Upper Bound on Sample Risk) framework. The theoretical developments are supported by a set of computational experiments showing the practical behaviour of these methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- Well written paper with a comprehensive review of the literature.\n- Strong theoretical contribution with rigorous proofs.\n- Detailed experiments that support the relevance of the algorithms.\n\n**Weaknesses**:\n- The experiments only consider relatively small scale MDPs.\n- The convergence of one algorithm (UBSR Q-learning) is not addressed.\n\n**Minor remarks**:\n- On page2 (introduction), it would be beneficial if the authors explicitly mentioned that their planning contributions (Section 3) involve model-based algorithms. The current wording might lead readers to think previous work was model-based, and this paper only introduces model-free methods.\n\nQUESTIONS:\n- The results are heavily on Doeblin condition. Is there any intuition beyond this assumption?\n\n- The results are based on time-consistent, monetary risk measures assumption. What challenges arise if one tries to handle more general, possibly time-inconsistent objectives (e.g. mean–variance)?\n\n- The objective is designed with dynamic risk measures, which quite differs from the usual static risk measures. The dynamic versions are known to provide poor approximations of the static one and is generally harder to interpret [1]. Have you tried to evaluate this in the setting of average reward MDPs, in the environment you consider?\n\n- How does the convergence speed compares to risk-neutral setting?\n\n[1]  D. A. Iancu, M. Petrik, and D. Subramanian. Tight approximations of dynamic risk measures. Mathematics of Operations Research, 40(3):655–682, 2015.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}