Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tries to disprove the reward hypothesis mathematically by showing some classes of tasks cannot be expressed using any scalar Markovian reward function. They showcase a set of multi-objective reward functions, risk averse utility functions that can’t be reduced to an equivalent scalar reward function. They also introduce a new class of RL tasks, namely,  modal tasks where the reward is dependent on the transition function
strength_and_weaknesses: Strengths 
The paper does provide sound proofs that some of the multi objective and risk averse based reward functions cannot be reduced, ceteris paribus, to an equivalent scalar Markovian reward function.

Weaknesses/Questions

1. The paper states that ‘In other words, it is the hypothesis that any natural task can be expressed as a reward signal’. The definition of a natural task can vary significantly. It would also bring up the question of whether the corresponding objective functions are ‘natural’ or if they are part of ‘what we mean by goals and purposes’.

2. I think there should be clarification on why the state space has to be the same between the original and modified MDPs to showcase that the reward hypothesis holds. 

3.This also leads to the question of if there are scalar rewards conditioned on the objectives then would that be equivalent to the original MORL or Risk Sensitive RL reward functions or if there are reward function approximators that approximate such equivalent rewards then would the reward hypothesis hold. Could there even be such approximators?

4. Coming to modal reward function, there isn’t a specific example that showcases how the reward function might be dependent on the transition function.

clarity,_quality,_novelty_and_reproducibility: Clarity - The writing in general is followable and most of the proofs are straightforward.

Novelty - The attempt to disprove the reward hypothesis mathematically is novel.

Reproducability - It is reproducible

summary_of_the_review: The paper definitely provides more insight into discussing the reward hypothesis and the extent to which it can be flexible but there are other aspects that need to be clarified (mentioned in the strengths and weaknesses section) in order to firmly disprove the reward hypothesis. Even in regards to the modal tasks, additional details are needed to clarify the dependency of the reward function and the transition function. Due to these reasons, I would incline towards not accepting this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper investigates tasks in which the "reward hypothesis" is not the most appropriate framework. The authors demonstrate three types of objectives: MORL,  risk sensitive RL, and Modal objectives, that cannot mathematically be reduced to a singular MDP formulation.
strength_and_weaknesses: The mathematical demonstrations are themselves interesting and curious in their own right. The main conclusions for each of the 3 individual objective functions appear to be justified. 

However, their main purpose in this paper, as formulated by the authors, is to serve to disprove the "reward hypothesis" and this framing is the problematic part. 

The first problem with the framing of in terms of the reward hypothesis is that in Sutton & Barto itself, this statement is explicitly stated to be an informal or guiding idea rather than a solid dictum. The second problem is that the reward hypothesis is broadly about the idea that goals can be formulated as rewards, and is therefore agnostic to the methods used to achieve this control problem or even the formulation of the setting itself (whether MDP or otherwise). Therefore, mathematical proofs about the nonequivalence of MDP RL and the 3 objective functions considered cannot disprove this statement (which itself, was informal to begin with), and framing this as a disproving is flashy but doesn't necessarily contribute to the advancement of this field. 

It is not to say that the paper did not demonstrate some interesting things. Examining expressivity of singular reward functions/MDPs is a cool question, and within this more targetted focus, this paper brings some interesting new demonstrations.

All in all, the authors should either provide clarifications to rigorously justify their current framing (i.e. disproving) or, they can amend their whole manuscript angle to be closer to what the authors actually demonstrated -- i.e. that these other objective functions are nonexpressible in an (S,A, reward) MDP.  
clarity,_quality,_novelty_and_reproducibility: The mathematical statements themselves were clearly defined and very readable. They were interesting results. Regarding originality, the authors have discussed the novel case of "Modal RL" which I have not seen elsewhere, and found very interesting. The other 2 objective functions as nonexpressible in an (S,A, reward) MDP were novel/curious demonstrations as well. 

However, when the authors say in the Discussion as their main point "We argue that our results show that the reward hypothesis is false – there are tasks, which are natural to state and intuitive to understand, and which can be solved with RL methods, but which cannot be expressed using scalar Markovian reward functions", Abel et al 2021 has already investigated tasks that cannot be expressed using a single scalar reward and this (expressivity) was their main point. The authors should therefore state that their study adds to this previous demonstration. 
summary_of_the_review: 
This study can be seen as an extension of the work Abel et al 2021, by providing 3 new and interesting tasks that scalar rewards cannot express. 

With reorganization and rigorous framing, this paper can potentially be an interesting one to the community.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper operationalizes the reward hypothesis as: Any task of interest can be captured as the maximization of the expected sum of a scalar reward signal R(s, a, s’). They then show several ways in which this hypothesis fails:

1. Many natural task formulations such as “Do X, but break ties according to Y” can be formulated using multi-objective reinforcement learning. The policy ordering induced by such a formulation cannot be expressed by any reward signal R(s, a, s’).
2. It is common to be risk-sensitive in pursuit of some goal: for example, if M() captures the amount of money you have, then you often care a lot if M() becomes very low or zero, but don’t care as much if M() increases by a similar amount. In economics this is often handled by transforming M() into some new function M’, such as M’(x) = -exp(αM(x)). In RL, we might similarly want to choose a risk-averse policy that corresponds to a Q function Q’(s, a) = -exp(αQ(s, a)). However, barring uninteresting edge cases, there is no reward function R’ that induces Q’, due to the sequential nature of the task and how R’ is _summed_ to get Q’.
3. We may want task specifications that depend on counterfactuals and possibilities, e.g. “accomplish X while remaining _able to reach_ the start state”. These can be formalized as reward functions that depend on the transition function. While it is of course possible to encode this as a regular reward function for a given transition function, there is no regular reward function that works for all transition functions (again barring uninteresting edge cases).
strength_and_weaknesses: Strengths:
1. The paper adds to the literature showing that the reward hypothesis is false (as previously demonstrated by Abel et al, which received an Outstanding Paper at NeurIPS, and discussed informally in the AI alignment community, e.g. [1]). To my knowledge, while the specific failures shown in this paper were generally suspected to be issues with the reward hypothesis, this is the first time they have been shown formally. I find the risk aversion result particularly strong.

Weaknesses:
1. In its discussion of MORL, the paper primarily analyzes policy orderings, even though what we typically care about in practice is the chosen policy. When considering the chosen policy, there is a significantly stronger argument that a linear combination of (history-based) rewards captures what we want (discussed below).
2. I found the “modal reward” section discussion to be fairly unconvincing, as it is not clear why we should care about robust equivalence rather than contingent equivalence (discussed further below).

**MORL**

I think there is a fairly strong argument for single-objective learning given _trajectory-based_ rewards, based on applications of decision theory to decision-procedures over trajectories (i.e. policies). Specifically:

1. We assume that when considering each $R_i(\xi)$ individually, we would like to choose trajectories on the basis of expected value under $R_i$. When S and A are finite, and we have a finite horizon length, this can be justified by the VNM theorem, since there are only a finite number of trajectories. Of course, this may require transformations to encode e.g. risk-aversion (such as switching from amount of money to log(amount of money), to be risk-averse w.r.t.money).
2. Similarly we assume that our final decision-procedure after aggregating should also choose trajectories on the basis of expected value under some reward function (once again, this is just saying that it should be VNM-rational).
3. We make the hopefully-uncontroversial assumption that if all of the $R_i$ are indifferent between two trajectories, then so too is our final decision-procedure.
4. The Harsanyi Aggregation Theorem [2] then says that our decision-procedure can be represented as maximizing the weighted linear sum of the various $R_i$.
5. The (history-dependent) policy that maximizes the weighted linear sum of the various $R_i$ is obviously consistent with this decision-procedure, and so is a good target to learn, which can be done with single-objective RL

(I have read Appendix C and I do not think this argument falls prey to any of the mistakes you mention there. Still, I have not checked the argument carefully; there could be some problem.)

Of course, this does not save the reward hypothesis (as you define it) because it requires us to use _trajectory-based_ rewards, whereas you operationalize the reward hypothesis as using _Markovian_ rewards. Nonetheless, I think it is worth mentioning this argument in favor of single-objective RL (if only in an appendix).

What happens if you try to run your argument on this construction? The first obstacle is that I am using trajectory-based rewards while your argument assumes Markovian rewards. However, I would guess that your argument applies just as well to trajectory-based rewards, as the key linearity-based arguments would still apply.

In that case, your result would say that for specific MORL objectives, the resulting policy ordering cannot be expressed by a single trajectory-based reward. One possibility is that the MORL objectives choose policies that are not consistent with a VNM-rational decision-procedure over trajectories, violating step 2 of my argument; if this were the case I would take that as a strike against MORL objectives rather than a strike against the reward hypothesis.

However, the more likely scenario is that both claims are simultaneously true. My argument only makes claims about the final decision-procedure over trajectories, i.e. the _optimal_ policy chosen by the single trajectory-based reward. It could indeed be the case that the ordering over other, non-optimal policies is different between the MORL objective and any possible trajectory-based reward. However in this case it is unclear why I should care. Ultimately we only obtain and deploy one policy; why should I care what the ordering over other policies is?

**Modal rewards**

It is not clear to me why we should ask for robust equivalence instead of contingent equivalence. As I understand it, your argument for this claim is:

> the construction of R♢τ will invariably be laborious, and require detailed knowledge of the environment. For example, consider the task “you should always be able to return to the start state”; here, constructing R♢τ would amount to manually enumerating all the states from which the start state is reachable. This is very much against the spirit of reinforcement learning, where much of the point is that we want to be able to specify tasks which can be pursued in unknown environments. 

However, it is unclear to me how you do any better with a modal reward function: how do you write down R(s, a, s’, τ) in full generality? I’m not familiar with all of the references you cite, but in e.g. Krakovna et al., 2018; 2020a, Turner et al., 2020, and your own Appendix E, there is no R(s, a, s’, τ) that gets written down and then maximized; rather the algorithms work by automatically building the contingently-equivalent reward R(s, a, s’) by learning it from experience. Why should we not think of this as “writing down the contingently-equivalent reward, and then maximizing that”?

The one difference that I see is that the contingently-equivalent reward is built _at the same time_ that the policy is learned, but this is surely just an efficiency improvement: you could also design methods that first learned the contingently-equivalent reward, and then maximized it. Given that the reward hypothesis (as you have operationalized and analyzed it) is primarily about what is theoretically possible; an argument that depends entirely on an efficiency difference does not seem appropriate.

(You might also say that the fact that you have to learn the reward means it is very hard to provide in practice, which is a strike against the reward hypothesis, but in the introduction you emphasize that the reward hypothesis is about what can be done _in principle_ and not about what is difficult to do in practice.)

Minor issues:

Lemma 1 is false as stated: given some s’, the reward could be constant on the states reachable from s’, while being non-constant on states unreachable from s’. In the proof, the issue arises here:

> Finally, note that this means that we can construct such trajectories for any state s′, by simply composing a transition ⟨s′, a, s⟩ with each of ζ1, ζ2, ζ3.

This doesn’t work if that transition is impossible (as in the preliminaries you require trajectories to be “possible according to µ0 and τ”). (If you remove this requirement I would expect this breaks other proofs.)

[1] Ngo, Richard. “Coherent behaviour in the real world is an incoherent concept.”

[2] Harsanyi, John C. "Cardinal welfare, individualistic ethics, and interpersonal comparisons of utility." Journal of political economy 63.4 (1955): 309-321.
clarity,_quality,_novelty_and_reproducibility: Clarity: Strong; given the technical nature of the results it was fairly easy for me to follow along.

Quality: Strong; the results are non-trivial and argue against a commonly-believed hypothesis and (to my knowledge) are correct (barring minor issues that can be easily corrected).

Originality / Novelty: While the results have been discussed informally before, to my knowledge this is the first time they have been formally proved.

Reproducibility: N/A, no experiments
summary_of_the_review: This paper provides clear technical results arguing against a commonly-held hypothesis. While I do have some nitpicks, they are just that: nitpicks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper discusses three types of problems where the goal cannot be expressed as scalar values in MDP (MORL where the objective cannot be expressed as weighted sums of individual objectives, trajectory-wise risk-sensitive problems, and modal problems). They also propose potential solutions for modal problems and discussed relevant literature for solving MORL and risk-sensitive problems.
strength_and_weaknesses: Strengths:
1. The paper is very well written and easy to follow. The introduction follows a clear narrative that takes the reader through all the class of problems and considerations. The subsequent sections also follow a consistent structure. Despite being dense in theorems, it’s easy to navigate.
2. The paper offers mathematical insight into each theorem in plain language. This also highlights the significance and logical progression of different theorems in this work.
3. Beyond showing MDPs where the goal cannot be expressed as scalar values, the author also devises solutions for these problems in section 5 as well as the appendix (modal problems).


Weakness:
I believe that the paper would be stronger if the author could address the following questions. Because there is a wide range of interpretations of the reward hypothesis, I hereby divide my questions into “philosophical” and “technical” ones:

Philosophical:
1. Response by Sutton et al. The authors’ argument for refuting the reward hypothesis is based on the fact that not all “goals and purposes” can be represented by scalar reward. This reminds me of one of Rich Sutton’s responses that goals ultimately lead to some desired behavior/policies. And one can trivially design a reward function to describe such optimal policy by penalizing actions that are not in the behavior. I wonder if the authors disagree with this. This may also be a point worth clarifying in the paper.
2. It’s not that the hypothesis is False, it’s MDP’s fault. One of the main reasons that risk-averse learning and modal learning fail the reward hypothesis is because they contradict the Markov assumption. I.e. for risk-averse learning, the reward depends on the agents’ future actions and for modal learning, the reward depends on the transition function. Since the reward hypothesis, as cited, does not imply Markov conditions, I wonder if the authors should narrow the claim to “The reward hypothesis is false under the Markov assumption”?
3. Undesirable/unsatisfactory is not impossible. In modal learning, the author points out that constructing a scalar reward function for modal learning is “undesirable” as it requires an enumeration of all the states and actions and detailed knowledge of the environment. However, if I understand correctly, it is still “possible” and there exists such a scalar reward function in any specific env. Sutton et al noted this in their post saying “is it always beneficial to describe goals as rewards in practice? Probably not.” I believe this doesn’t refute the original reward hypothesis that constructing a reward function is possible. Please let me know if I didn’t understand this part correctly.

Technical:
1. Non-differentiable MORL problem. The authors mention in section 5 that MORL problems with differentiable U can be solved via existing solvers. However, if I understand correctly, MaxSat is a non-differentiable MORL problem. Do the authors have any suggestions for solving this problem?

clarity,_quality,_novelty_and_reproducibility: The paper is clear and well-written. As far as I know, it is novel in discussing MORL, risk-sensitive and modal problems in the context of the reward hypothesis.

summary_of_the_review: I’m not yet convinced by the arguments about modal problems, and I also think that risk-sensitive problems are limited by the expressivity of MDP, not necessarily because the reward hypothesis is False. I am interested in hearing the author’s response, and would either revise my review or suggest that the authors adjust the claim of the paper to be less strong than the “reward hypothesis is False”.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the ability of Markovian rewards to express tasks in some generalized RL settings, namely multi-objective RL, risk-averse RL, and the so-called modal RL. The tasks are intended as total policy ordering (multi-objective RL, modal RL) or total trajectory ordering (risk-averse RL). The paper provides examples of tasks that cannot be expressed by a Markovian reward, which they claim is a refutation of the reward hypothesis. The paper include several ancillary results, such as providing necessary and sufficient conditions for when a multi-objective RL task can be expressed through a Markovian reward, and a formulation of the modal RL problem, in which the tasks are expressed in modal terms (a notion close to counterfactuals) instead of categorical terms.
strength_and_weaknesses: *Strengths*
- Overall clarity. The paper is well-written and easy to follow;
- Originality (partial). The paper provides an original formulation of the so-called modal RL problem, which might deserve its own characterization, as well as interesting insights on multi-objective RL formulations, and when those are not harder than standard RL.

*Weaknesses*
- Novelty. The result that is presented as the major take of the work, i.e., that Markovian rewards cannot express all the possible notion of tasks, is not novel, as it was already substantiated in (Abel et al., 2021);
- Presentation. The paper includes several potentially interesting results that take a backseat with respect to the (unsurprising) refutation of the reward hypothesis;
- Motivation. The analysis includes some arbitrary choices that lack a clear motivation, such as total policy ordering as to express the notion of task.

*Main Concerns*

(C1) Presentation. I do not understand the choice of the authors of presenting this paper as a formal refutation of the reward hypothesis. Abel et al. (2021) already demonstrated that some notion of tasks cannot be expressed through a Markovian reward function. Whereas the authors are right to say that their analysis mostly rely on different arguments, the major take is neither surprising or novel.

(C2) Notion of Task. I do not think the reward hypothesis ever mentioned policy ordering in the definition of "goals and purposes". One could argue that the actual goal of the learning process is to find an optimal policy, and that the expressivity of the reward should be evaluated in terms of the optimal policies it can induce. Nonetheless, Abel et al. (2021) demonstrated that also this weaker notion of task (which they call set of admissible policies) cannot be always expressed through a Markovian reward. The refutation argument of this paper does not include this result instead.

*Other Comments*

(C3) Generalization to average reward and finite-horizon.
To my knowledge, both the results presented in this paper and (Abel et al., 2021) are limited to the discounted RL setting. Do the authors believe that similar results could be generalized to the average reward setting, i.e., infinite horizon but without discounting, and the episodic setting, in which the reward function is often non-stationary?

(C4) Sufficiency of Markovian deterministic policies.
While it is well-known that the class of deterministic Markovian policies are sufficient to optimize any Markovian reward in the discounted setting, it is not clear what policy class we should consider for the presented generalized RL settings (multi-objective, risk-averse, modal RL). Do they also admit an optimal deterministic Markovian policy?

(C5) A "universal" problem setting.
The paper shows that some notion of tasks cannot be expressed through a Markovian reward function. Do the authors believe it might exist a more general (and tractable) problem setting that include all of the presented notions of task? Perhaps the convex RL formulation (Hazan et al., 2019; Zhang et al., 2020), where the learning objective is expressed as a convex/concave function of the state-action distribution, can be an interesting candidate. Another is the vectorial reward formulation by (Cheung, 2019).

(C6) Modal RL.
I am not really familiar with the notion of modal tasks, but the modal RL seems an original and interesting formulation to me. Can the authors provide a deeper characterization of this problem formulation? What kind of tasks it can express? E.g., it clearly subsumes standard RL tasks, what about multi-objective tasks or other generalization? Are the three presented problem setting non-overlapping in terms of expressivity?

(C7) Choice of the problem settings.
The choice of the problem settings to present in against the reward hypothesis is kind of arbitrary, and it is not clear why they have been preferred over constrained RL, pure exploration (e.g., maximum state entropy), imitation learning (as divergence minimization) or any other generalization.

(C8) Related works.
I believe that some additional works could be at least mentioned in the paper, such as the convex RL formulation and vectorial reward formulation, which are interesting generalization of the Markovian reward, as well as other papers considering non-Markovian rewards (e.g., through reward machines, for which I am not an expert and I cannot provide valuable references).

Convex RL:
- Hazan et al., Provably efficient maximum entropy exploration, 2019;
- Zhang et al., Variational policy gradient method for reinforcement learning with general utilities, 2020;
- Zahavy et al., Reward is enough for convex MDPs, 2021;
- Geist et al., Concave utility reinforcement learning: The mean-field game viewpoint, 2022;
- Mutti et al., Challenging common assumptions in convex reinforcement learning, 2022.

Vectorial rewards:
- Cheung, Exploration-exploitation trade-off in reinforcement learning on online markov decision processes with global concave rewards, 2019;
- Cheung, Regret minimization for reinforcement learning with vectorial feedback and complex objectives, 2019.
clarity,_quality,_novelty_and_reproducibility: The paper reads well, and the reported statements seems to be correct from a brief inspection. However, the main take presented by this paper is not novel.
summary_of_the_review: This paper is an interesting read, and it includes several ancillary results that can have significant value. Nevertheless, I believe that this paper cannot be accepted in its current form, as the presentation is centered on a result that is not novel. I would suggest the authors to reformulate the paper putting less emphasis on the refutation of the reward hypothesis, and more on the actual original contribution. E.g., modal RL seems a very interesting idea to expand, and some interesting ideas about how to approach them are relegated to the Appendix. Also the characterization of the necessary and sufficient conditions for which multi-objective RL cannot be casted as a standard RL problem are interesting, and they could provide a valuable contribution on their own.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper is about the reward hypothesis (roughly that a reinforcement learning agent would only need one scalar signal to learn "any" task) and wants to show that this is not the case for many problems. The main portion of the paper consists of three settings (multi-objective, risk-averse and modal) and in each of these settings some formalizations and some properties are defined (and somewhat proven). The paper is purely theoretical.

strength_and_weaknesses: The main strength of the paper is the topic, which receives a lot of attention in the last few years: reward functions, what can they do, what is their expressivity, how can they be learned/engineered/constructed, model/problem classes, and how to employ them in typical RL frameworks.

However, the paper does not really deliver on its main promises stated early in the paper (and the title). I guess it is fairly well known that, in principle, many situations exist where reward functions are multi-dimensional, or dependend on time or previous states, or something else, and then it is not possible to reduce it simply to a single scalar. However, for many (all?) problems it can be done, but typically with an exponential blowup in the parameters of the RL problem. For example, in the sub-area of non-Markovian reward functions, which is highly related to the "modal setting" in Section 4, problems with highly complex, non-Markovian reward functions can systematically be "compiled" into regular (single scalar, Markovian) problems by compiling necessary sequences of states (and actions) into a new state space (which is typically exponentially larger than the original). In this case the new problem can be solved using standard methods in deep RL, without any loss in accuracy or the possibility of finding an optimal solution, at a cost of a much increased state space, and accompanying measures like need for generalization, shaping rewards, longer learning times, and so on. Similar things can be said about the other two frameworks treated in the paper.

The main weakness of the paper is that it is highly unclear what the contributions are. The paper talks about well-known frameworks, and connects it more or less to RL, but without writing about what are contributions, new insights, and connections to the literature. For RL experts, I think most of it is very intuitive to read, and even interesting in the more general sections 1 and 5, but for me it is unclear how this paper advances the state-of-the-art and understanding of the reward hypothesis (relative to the literature). Furthermore, Sections 3 and 4 are even less worked out than Section 2, and only contain some initial thoughts. The modal RL is only sketched briefly, and does not connect well to the growing literature on combinations of RL and formal verification tools (like PCTL and temporal formalisms). Section 2 should first show the state-of-the-art in MORL, and then do additional things. Overall, all the methods are not illustrated nor motivated well, and also domain descriptions (examples) nor experimental evaluations or illustrations are incorporated. The paper as a whole has interesting bits and fragments, but as a paper, in this time and sub-area, it does not do well as "a research paper" since it is too unfocused and to unclear about its achievements.

Some other things:
- Since the paper is quite unfocused, I feel the related work is too. It is unclear why exactly this set of references is chosen.
- The preliminaries are ok, and formalisations too, but some "operations research" style formalizations are chosen (G,J,..) and it is good to connect (maybe) more literally to Sutton and Barto who have a more AI-oriented flavor of notation.
- Why not illustrate Sections 3 and 4 more with AI-oriented examples, in well-known domains, or even with some experimental part? This would help a lot in motivating these formalizations. Page 9 says "we have given many examples" but I feel this is really missing.
- The modal setting should be much more connected to the current trend to combine RL with verification tools, and it should get more space (maybe a separate paper) to really explain the setting in full detail.
- Section 5 should at least discuss in more detail how typically one can still "compile" problems to single-scalar problems, with the necessary trade-offs in model size, solution times, and so on.
- Too much detail is deferred to appendices, especially on page 9. The major part of the story should be in the paper itself though, and appendices should provide some further detail. The balance is off here. 
clarity,_quality,_novelty_and_reproducibility: The paper is quite clear in a technical sense, but story-wise it is quite unfocused. The paper is fairly polished. As outlined in another part of this review, I feel the paper's novelty is quite unclear, since many things are intuitive but it does not become very clear what are the contributions. Reproducibility is not a real issue, since the formalizations are purely theoretical and no experiments are provided.
summary_of_the_review: This paper has several interesting things to say, about a topic that is currently active, but it is unclear what are the contributions, and furthermore it tries to do too many things in one paper without (almost) any illustrations, nor experiments, and overall the main portions sound not too novel at all, given the current state of the art in RL.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers agreed that the paper provides new insights on the refutation of the reward hypothesis based on three types of tasks that cannot be expressed with scalar rewards. However, the reviewers also raised several concerns and questions in their initial reviews. We thank the authors for their detailed responses and for preparing a revision to address some of these concerns. 

There was a large spread in the ratings, and the reviewers extensively discussed this paper, considering the responses and revision. Some of the reviewers raised a common concern that the positioning of the paper has changed after revision (original submission title "The Reward Hypothesis is False"; revised submission title "Three Problem Classes That Markov Rewards Cannot Express"). With this new positioning, the paper does not read coherently and is mostly a list of varied results in three domains. Based on the raised concerns and follow-up discussions, unfortunately, the final decision is a rejection. Nevertheless, this is exciting and potentially impactful work, and we encourage the authors to incorporate the reviewers' feedback when preparing a future revision of the paper.

justification_for_why_not_higher_score: The paper was discussed among all the reviewers, considering the responses and revision. The reviewers shared a common concern that the positioning of the paper has changed after revision and does not read coherently anymore. While there is still a large spread in the reviewers' ratings, the discussions are inclined toward rejection.
Decision: Reject