Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents reward modeling methods for non-markovian rewards, leveraging LSTMs to update a hidden state and encode the trajectory history. It performs experiments in custom environments with non-markovian rewards, with a baseline that assumes markovian rewards. Their various proposed methods perform better than the baseline. The theory behind the method is sound and the experiments are sound too.
strengths_and_weaknesses: Good points: I think the connection between reward modeling with trajectory-level labeling and multi-instance learning is interesting and well explained. The experiments give good (although artificial) examples of non-markovian reward decision process. The writing is good and the paper is clear, and the reviewer had a good understanding of the presented methods after reading the paper.

My main gripe with the method and theory is the use of the NMRDP (non-markovian reward decision process), which can be viewed as a specialisation of a POMDP where only the reward depend on the hidden state. This would be a useful formalism if it led to a simpler algorithm, but the described algorithm resembles exactly what you would get if you tried to solve the reward modeling problem for POMDPs using LSTMs. I realise that the experiments only contain the special case of NMRDPs, but there is a missed opportunity in presenting a more general solution here. The argument of using this formalism to model human behaviour when assigning rewards is unconvincing to me as it is not tested in the experiment.

Speaking of this, I feel the experiments are too weak and obviously make it impossible for the baseline to perform well. They do demonstrate that the presented methods can handle non-markovian rewards in these limited settings. The problem is the applicability of this setting to more complex or better-known tasks; without resorting to completely standard benchmarks where the reward is markovian, could you make a stronger case that the methods are applicable and helpful for larger-scale, real-world tasks? And more importantly, can your method actually handle the complexity of human-labels rewards as you talk about in the introduction but don’t demonstrate in your experiments? In my opinion, addressing the second question with a set of experiment with real human labels would also address the first question.

To summarize 1) I think the method would be better presented as reward-modeling for POMDPs (where lstms have been used to encode the history in many works, see Dreamer, PlaNet, reccurent DQN) at which point the solution becomes obvious. 2) The experiments, while valid, do not convince the reviewer of the wider applicability of their methods.

Small comments: 

The abstract uses a lot of specialized language RM, MIL, “provide interpretable learn hidden information” that wasn’t really clear to the reviewer until *after* reading the paper and coming back to the abstract.

I find it awkward that the reward is a function of both s_t and h_{t+1}. I realize that it leads to better performance in practice, but it shouldn’t be necessary and maybe drop it from the math.

Section 3.2, and the paper in general, completely ignores the existing body of work using LSTMs (and RNNs in general) for POMDPs, which leads to algorithms similar to yours (see for example Hausknecht, Matthew, and Peter Stone. "Deep recurrent q-learning for partially observable mdps." 2015 aaai fall symposium series. 2015.) I think at least a mention should be added; I think the idea of using LSTMs specifically to handle non-markovian reward is simply a specialisation of this existing idea of using RNNs to encode past observations in POMDPs. It's not a bad thing per se, but should be mentioned.

Line 229 mention an “interesting” case when CSC and instance-space LSTM beats the oracles; the reviewer does not think it is interesting but rather suspicious. This could be stemming from a lack of hyperparameter search or uncertainty in the results.



questions: Without resorting to completely standard benchmarks where the reward is markovian, could you make a stronger case that the methods are applicable and helpful for larger-scale, real-world tasks?

And more importantly, can your method actually handle the complexity of human-labels rewards as you talk about in the introduction but don’t demonstrate in your experiments?
limitations: It is not clear what the limitation of this work is as it has to potential to be very general; the experiments do not go far enough to find the limits of the methods. I see no obvious negative societal impact of this work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a new method to deal with non-Markovian rewards based on reformulating the problem as multiple instance learning with LSTM. The authors test a few alternative ways of doing this in 4 environments designed to demonstrate non-Markovain properties.
strengths_and_weaknesses: Originality: 

While Markovian reward modelling is quite well studied, non-Markovian rewards are investigated much less. The proposed improvements to MIL with LSTM (instance space and skip connections) are not particularly original, but they effectively address the shortcomings of the prior methods.  

Quality: 

I am not completely convinced about the baseline "Base Case: Embedding Space LSTM" model where the rewards are obtained by taking the difference in partial bag labels. Suppose the task is to grasp an object. If I am given two sequences to annotate: 1) s1=not grasped, s2=not grasped, s3=not grasped, s4=grasped, s5=grasped and 2) s1=not grasped, s2=not grasped, s3=not grasped, s4=grasped, I could assign both of them label 1. Then, using the difference between partial bags would result in the reward of s5 to be 0 which is intuitively not correct. I expect that with predictions from LSTM we may face a similar problem. If it is the case, maybe the inferior performance of this method could be due to this.
In section 3.3 the authors explain that the RL agent observes hidden states along the reward from LSTM. However, I am wondering if it is suitable to use hidden state representation of the reward model for agent training. Maybe it is better to have a recurrent agent architecture that trains its own latent representation.
For the baselines in Figure 5, I think it would be more informative to have RL with recurrent architecture + oracle rewards as the upper bound on the performance and it would avoid the situation when an agent with learnt reward outperforms the oracle.

Clarity: 

The paper is very well written and easy to follow. I found the visualizations in Figures 7 and 8 very informative. 
The paper positions its problem as originating from an unrealistic assumption on human evaluation of temporally-extended behaviour. However, I believe the non-Markovian rewards might be important in practice in some tasks that are inherently non-Markovian and the success can't be determined only by a given state, but a whole sequence must be considered. Some of such tasks are studied in the experiments, but the point that these are different types of tasks was not made clear. It could be also informative as an additional ablation to consider simple Markovian tasks (with Markovian reward), for example, like navigating to a given target to confirm that the method of this paper works in that case too.

Significance: 

My biggest concern in this paper is the complexity of the evaluation environments. While the proposed environments and tasks work sufficiently well to demonstrate the proof of concept, they are rather simplistic (grid world) and some tasks are a bit artificial. I am wondering if the proposed method would face any difficulties when dealing with a more challenging environment, for example, some control problems in simulation, or something that has image-based states. I think that in order to have a larger impact in the community, this paper should include results on more complex environments.

questions: - Could the authors clarify if the issue with the difference in partial bags as I described takes place in their experiments?
- Could the authors clarify why it is chosen for the agent to reuse latent states from reward modelling rather than to train its own LSTM?
- Could the authors elaborate why the baseline that just replaces each state with frame stacked state (several states concatenated, often used as a simplification for recurrence) would not suit this problem?
- The authors train Deep Q-Network, but as far as I understand the states are just 2-dimensional, how important is it to have deep architectures in this case?
- The authors say that the transformer architecture would be unsuitable for the temporal structure of the problem. Why would it be the case with temporally masked transformers?
- How was the dataset for RM training (from lines 210-213) split into training and test? Are the results in Table 1 on the test set?

limitations: Some limitations are discussed at the end of the paper. In the review I also pointed to/asked for clarification regarding some potential limitations. No potential societal impact is mentioned.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper defines non-Markovian reward modeling and proposes a novel LSTM-based model to capture temporal dependencies in reward modeling. It also adopts a multiple instance learning (MIL) framework to handle temporal dependencies with reward labels. It encodes hidden state information using LSTM and also produces explicit reward predictions with skip-connection to the current state and action features.
It optimizes LSTM-based models on offline trajectory datasets. Quantitative experimental results on return and reward prediction show the model's compelling reward function modeling quality. The authors also present qualitative results on the interpretability of the learned hidden embeddings.

strengths_and_weaknesses: Strengths:

The paper is well written and easy to read and understand. Figure 2 clearly demonstrates the proposed method and the baseline models. The experiments are comprehensive and the visualization result is quite inspiring. They also propose novel RL tasks that can show the effectiveness of their model. For significance, learning the non-Markovian reward function is the cornerstone of a successful real-world application using RL, especially with human interactions.

Weaknesses:

I appreciate the idea and results of the conceptual toy RL tasks experiments, however, it would be better if there were comparisons with other models in the more complex RL tasks such as Montezuma's Revenge where modeling non-Markovian rewards is important.

questions: 1. How important the proposed non-Markovian reward modeling is compared to other temporal information encoding models (such as Flare [1] or SPR [2])? 

2. How many timesteps the proposed LSTM-based model can encode?

3. What did you use for the LSTM hidden states 2D visualization? T-SNE? And do you have any other visualization results on other common RL tasks with discrete action space such as Atari games?


- [1] Reinforcement Learning with Latent Flow, Shang et. al., NeurIPS 2021
- [2] Data-Efficient Reinforcement Learning with Self-Predictive Representations, Schwarzer et. al., ICLR 2021
limitations: Yes, the authors adequately addressed the limitations and potential negative social impact of their work.
review_title: null

Meta-review: metareview: The reviewers have agreed on many points (at least after some help from the author's explanations and changes in the rebuttal): the problem formulation is interesting (in particular as it relates to evolving human preferences, but also in the practical experimental cases), the writing is clear and the technical solutions are interesting. 

While there is also a general consensus that more, larger experiments would be desirable, I note this is much more difficult to achieve in the paper's setup than most "vanilla=Markov" RL, as significant modifications are needed to any standard environment to fit this paradigm. Lunar Lander was well appreciated during the rebuttal, and I believe the paper will now have a strong impact as-is (although if the authors can find the time for another similarly sized env prior to the final version, it will be welcome). 
confidence: Less certain
Decision: Accept