Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning
Authors: Joseph Early
Tom Bewley
Christine Evers
SArvapali Ramchurn
Venue: NeurIPS 2022 Accept (2022)
Keywords: reward modelling
reinforcement learning
multiple instance learning
interpretability

Abstract: We generalise the problem of reward modelling (RM) for reinforcement learning (RL) to handle non-Markovian rewards. Existing work assumes that human evaluators observe each step in a trajectory independently when providing feedback on agent behaviour. In this work, we remove this assumption, extending RM to capture temporal dependencies in human assessment of trajectories. We show how RM can be approached as a multiple instance learning (MIL) problem, where trajectories are treated as bags with return labels, and steps within the trajectories are instances with unseen reward labels. We go on to develop new MIL models that are able to capture the time dependencies in labelled trajectories. We demonstrate on a range of RL tasks that our novel MIL models can reconstruct reward functions to a high level of accuracy, and can be used to train high-performing agent policies.

Non-Markovian Reward Modelling from Trajectory Labels via Interpretable Multiple Instance Learning
Joseph Early +18
Tom Bewley
*i8
Christine Evers
Sarvapali Ramchurn
Abstract
We generalise the problem of reward modelling (RM) for reinforcement learning (RL) to handle non-Markovian rewards. Existing work assumes that human evalua- tors observe each step in a trajectory independently when providing feedback on agent behaviour. In this work, we remove this assumption, extending RM to capture temporal dependencies in human assessment trajectories. We show how RM can be approached as a multiple instance learning (MIL) problem, where trajectories are treated as bags with return labels, and steps within the trajectories are instances with unseen reward labels. We go on to develop new MIL models that able capture the time dependencies in labelled trajectories. We demonstrate on a range of RL tasks that novel MIL models can reconstruct reward functions high level of accuracy, and can be used to train high-performing agent policies.
Introduction
There is growing consensus around the view that aligned and beneficial AI requires reframing of objectives as being contingent, uncertain, and learnable via interaction with humans [35]. In reinforcement learning (RL), this proposal has found one formalisation in reward modelling (RM): the inference of agent objectives from human preference information such as demonstrations, pairwise choices, approval labels, and corrections [29]. Prior work in RM typically assumes that human evaluates the return (quality) of a sequential trajectory of agent behaviour by summing equal and independent reward assessments of instantaneous states and actions, with the aim of RM being to reconstruct the underlying reward function. However, in reality the human's 's experience of trajectory is likely be temporally extended (e.g., via video clip [12] or real-time observation), which opens the door to dependencies between earlier events and the assessment of later ones. The independence assumption may be both psychologically unrealistic given human memory limitations [26]. and technically naive given the difficulty of building complete instantaneous state representations [25]. We thus seek to generalise RM to allow for temporal dependencies in human evaluation, by postulating hidden state information that accumulates over a trajectory. Reconstruction of the human's 's preferences now requires the modelling of hidden state dynamics alongside the reward function itself.
In tackling this generalised problem, we identify a structural isomorphism between RM (specifically from trajectory return labels) and the established field of multiple instance learning (MIL) [10]. Trajectories are recast as bags and constituent state-action pairs as instances, which collectively contribute to labels provided the bag level by interacting in potentially complex ways. This mapping inspires range of novel MIL model architectures that use long short-term memory (LSTM) modules [19] recover the hidden state dynamies, and learn instance-level reward predictions from return-labee labelled trajectories of arbitrary length. In experiments with synthetic oracle labels, we show that our MIL RM models can accurately reconstruct ground truth hidden states and reward functions for non-No Son-Markovial tasks, and can be straightforwardly integrated into RL agent training to achieve performance matching, or even exceeding, that of agents with direct access true hidden states and rewards. We then apply interpretability analysis to understand what the models have learnt.
"qqual contribution
University of Southampton, United Kingdom; (J. A. E.rlly, C. Evers, ,vers, @sotonnn......o uc.uk t University of Bristol, United Kingdom; tom. bww.eybrrstool..ooooooooooooo ac. uk Thhe Alan Turing Institute, United Kingdom
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Our contributions are as follows:
1. We generalise RM to handle mnn-Markovian rewards that depend on hidden features of the environment or the psychology of the human evaluator in addition to visible states/actions.
2. We identify structural connection between RM and MIL, creating the opportunity to transfer concepts and methods between the two fields.
3. We propose novel LSTM- MIL models for this generalised RM problem, and develop interpretability techniques for understanding and verifying the learnt reward functions.
4. We compare our proposed models to existing MIL baselines on five non-Markovian tasks, evaluating return prediction, reward prediction, robustness label noise, and interpretability.
5. We demonstrate that the hidden state and reward predictions of our MIL RM models can be used by RL agents to solve -on-Markovian tasks.
The remainder of this work is as follows. Section 2 discusses related work in RM and MIL, Section 3 gives formal problem definition and describes our MIL-inspired methodology, and Section presents our experiments and results. We discuss key findings in Section 5, and Section 6 concludes. All of our source code is available in a public repository.
2 Background and Related Work
Reward Modelling RM [29] aims to infer reward function from revealed human preference information such as demonstrations [32], pairwise choices [12], corrections [4], good/bad/neutral labels [33], or combinations thereof [24]. Most prior work assumes a human evaluates a trajectory by summing independent rewards for each state-action pair, but in practice their experience is likely to be temporally extended (e.g., via a video clip), creating the opportunity for dependencies to emerge between earlier events and the assessment of later ones. As noted by Chan et al. [11] and Bewley and Lecue [8]. such dependencies may arise from cognitive biases such as anchoring, prospect bias, and the peak-end rule [26], but they could equally reflect rational drivers of human preferences not captured by the state representation. Some efforts have been made to model temporal dependencies, such as a discrete psychological mode which evolves over consecutive queries about hypothetical trajectories [6]. or monotonic bias towards more recently-viewed timesteps due to human memory limitations [28]. Elsewhere, Shah et al. [36] use human demonstrations and binary approval labels to learn temporally extended task specifications in logical form. In comparison to these restricted examples, our work provides a more general approach to capturing temporal dependencies in RMM.
Non-Markovian Rewards In the canonical RL problem setup of a Markov decision process (MDP), rewards depend only on the most recent state-action pair. In a non- Mon-MMrkovian reward decision process (NMRDP) [2], rewards depend on the full preceding trajectory [2]. NMRDPs can be expanded into MDPs (and thus solved by RL) by augmenting the state with a hidden state that captures all reward- relevant historical information, but this is typically not known a priori. Data-driven approaches to learning NMRDP expansions [21] often make use ddomai-speccff propositions and temporal logic operators [3, 39, 41J. Outside of the RM context, recurrent architectures such LSTMs have been used in NMRDPs to reduce reliance on pre- pre-specified propositions [23]. They also have a long history of use in partially observable MDPs, where dynamics are also Man-Markovian [5, 17, 46).
Multiple Instance Learning In MIL [10], datasets are structured as collections of bags X, E X. each of which is comprised of instances {a) . } and has associated bag-level label Y; and instance-level labels {vi...., . The aim is to construct model that learns solely from bag labels; instance labels are not available during training, but may be used later to evaluate instance-level predictions. The simplest MII. approaches assume that instances are independent and that the bag is unordered, but models exist for capturing various types of instance dependencies [22, 42, 45]. LSTMs have emerged as natural architecture for modelling temporal dependencies among ordered bags, where they can be utilised to aggregate instance information into an overall bag representation. They have previously been applied to standard MIL benchmarks [44], as well as specific problems such as Chinese painting image classification [30]. As we discuss in Section 3.2, these existing models are somewhat unsuitable for use in RMM, leading to propose our own novel model architectures.
http:://ggghhuubooo//aaaaalal/// Non- Mww.aaaar-oeeeeeeeeeaaee Modaaddd

---

3 Methodology
In this section, we present the core methodology of our work. We formally define the new paradigm of non-Markovian RM (Section 3.1), before drawing on the MIL literature to propose models that can be used to solve this generalised problem (Section 3.2). We then go on discuss how can use our learnt RM models for training RL agents on non non-Markovian tasks (Section 3.3).
3.1 Formal Definition of Non-Markovian RM
Consider an agent interacting with an environment with Markovian dynamics. At discrete time I, the current environment state & E S and agent action a; E AA condition the next environment state $1+1 according to the dynamics function D 8 x A - AS. A trajectory & E E is sequence of state-action pairs, Ç = ((s0, do)...., (1T=).ar-)), and human's 's preferences about agent behaviour respect a real-valued return function G = + RR. In traditional (Markovian) RM, return is assumed to decompose into sum of independent rewards over acte-action pairs, G( Ei= R(st, az), and the aim is to reconstruct R' x R. from possibly-noisy sources of preference information. In our generalised non-Markovian model, we consider the human to observe trajectory sequentially and allow for the possibility of hidden state information that accumulates over time and parameterises It:
G(
Σ R(st, at, ha+1) where h++1 5h,, St, at), t=0
5 is a hidden state dynamics function, and he is fixed value for the initial hidden state. Reconstruction of the human's 's preferences now requires the estimation of 8' zs ō and h/ R ho alongside R' z R. We visualise the difference between Markovian and Mor-Maakovinn RM in Figure 1.
Markovian Reward Modelling
Non-Markovian Reward Modelling
Preferunce Informationn
OYS
Podf Info.
Rew. Mod.
Modelling
N2
R
Np
fy
Figure 1: In Markovian RM, the human is assumed to sum (+) over independent and equal reward assessments for the state-action pairs in a trajectory. In non-Markovian RM, per-timestep rewards additionally depend on hidden state information h that accumulates over time.
The hidden state h may be interpreted as (I) an external feature of the environment that is detectable by the human but excluded from the state, or (2) a psychological feature of the person themselves, through which their response to each new observation is influenced by what they have seen already. The latter framing is more interesting for our purposes, and connects to the psychological literature on human judgement, memory, and biases [26]. In practice, hidden state information may encode the human's 's preferences about the order which a sequence of behaviours should be performed, the effect of historic observations on their subjective mood (and in turn on their reward evaluations), or cognitive biases which corrupt the way they aggregate instantaneous rewards into trajectory- feedback. All of these complications are liable to arise in practical RM applications, but cannot be handled when the Markovian reward assumption is made. Appendix claborates on this discussion, presenting motivating use cases and limitations of non-Markovin RM.
In this work, we focus on one of the simplest and most explicit forms of preference information: direct labelling of returns G(Ex) for a dataset of N trajectories \ We aim to solve the reconstruction problem by minimising the squared error in predicted returns:
argmin R',8' .hg
G(G)) - Σ
--1
R' (81,t, az,t, Mast)) where
11,0 h/g hi 8' 6'(hlee St, a1))
Vi E (1..N).
(2)
3

---

We observe that Equation 2 perfectly matches the definition of a MIL problem. Each trajectory Ç can be considered as an ordered bag of instances ((s.,,: (i,0)...., (St,T, a4,, with unobserved instance labels R(ss,,, ai,z, 1,), an observed bag label G( = Ei=o R(si,r: di, hi.+4)), ), and temporal instance interactions via the changing hidden state f..g. This correspondence motivates us to review the space of existing MIL models (specifically those that model temporal dependencies
among instances) to provide starting point for developing our non-Markovian RM approach.
3.2
MIL RM Architectures
The MIL literature contains a variety of architectures for handling temporal instance dependencies, including graph neural networks (GNNs) [42] and transformers [37]. While effective for many problems, such architectures are an unnatural fit to non-Markovian RM as they contain no direct analogue of hidden state h' carried forward in time, instead handling dependencies via some variant of message- message-passing between instances. LSTM- based MIL architectures [30, 44] provide a more promising starting point since they explicitly represent both h' (implemented as continuous-valus vector) and its temporal dynamics function 8' (a particular arrangement of gating functions).
Starting from an existing L.TT--aase MIL. architecture, we propose two successive extensions as well as a naïve baseline that cannot handle temporal dependencies. All four architectures include a feature extractor (FE) for mapping state-action pairs into feature vectors and head network (HN) that outputs predictions. These architectures are depicted in Figure 2. Note we use the same nomenclature as [10] and [45]: embedding space approaches produce an overall bag representation that is used for prediction, while instance space approaches produce predictions for each instance in the bag and then aggregate those predictions to a final bag prediction.
Base Case: Embedding Space LSTM This architecture, proposed by Wang et al. [44], processes all instances in a bag sequentially and uses the final LSTM hidden state as a bag embedding. This is fed into the HN, which predicts the bag label g' (return in the RM context). A Although this model can account for temporal dependencies, it does not inherently produce instance predictions (rewards), which require some post hoc analysis to recover. While methods exist for computing instance importance values as a form of interpretability [13], these are not guaranteed sum to the bag label as stipulated by the reward-returt formulation. We propose new method: at time I, the predicted reward rf is calculated by feeding the LSTM hidden state at times (- 1 and into the HN to obtain two partial bag labels/returns h61 and D/: and computing the difference of the two, i.e., rz l6 K'- We define % = O. This post hoc computation is shown in purple in Figure 2.
Extension 1: Instance Space LSTM The post hoc computation of reward proposed above is rather inelegant and often yields poor predictions (see Section and Appendix D), likely because rewards are never computed or back-propagated through during learning. This leads us to propose an improved architecture, which is structurally similar but differs in how network outputs are mapped onto RM concepts. The change places reward predictions hidden state at time t, the output of the HN is taken to be the instantaneous reward r/ rather than the partial return. Rewards are computed sequentially for all timesteps in trajectory and summed to give the return prediction g'. We thereby obtain a model that both handles temporal dependencies and produces explicitly-leant reward predictions.
the back pnck-propagttion path. Given the LSTM
Concatenated Skip Connection (CSC) Instance Space LSTM In both of the preceding architectures, the LSTM hidden state h; is the sole input the HN. This requires hig to represent all reward-relevant information from both the true hidden state hn and the latest state-action pair 1, a i achieve good performance. To lighten the load on the LSTM, we further extend the Instance Space LSTM model with skip connection [18, 20] which concatenates the FE output onto the hidden state before feeding to the HN. In principle, this should allow the hidden state to solely focus on representing temporal dependencies. As well as improving RM performance compared to an equivalent model without skip connections, we find in Section 5.1 that this modification tends to yield more interpretable and disentangled hidden state representations.
Extension 2:
Markovian Baseline: Instance Space Neural Network (NN) To quantify the cost of ignoring temporal dependencies, we also run experiments with a baseline architecture that feeds only the FE output for each state-action pair into the HN. yielding fully-independent reward predictions which are summed to give the return prediction. This independent predict-and-sum architecture has precedence in both MII., where it is referred to as mi-Net by Wang et al. [45], and in RM, where it embodies the de facto standard Markovian reward assumption [12].

---

Instance Space NIN
Embedding Space LSTM
HN
HN
Jain card ousing harning)
HN
LSTM
LSTM
PR
PB
PE
02-3
FB
Instance Space LSTM
CSC Instance
Space LSTM
EDN
IIN
HN
BN
1STM
1STM
LSTM
fg
LSTM
PB
PE
02-3: 02-1
FE
Figure 2: MIL architectures used in this work. FE feature extractor; HN head network; (+) scalar summation; (-) scalar subtraction; (C") vector concatenation.
3.3 Training Agents with Non-Markovian RM Models
In this work, as in RM more widely, we are not solely interested in learning reward functions to represent human preferences, but also the downstream application of rewards to train agents' action-selection policies. After optimising our LSTM-based models on offline trajectory datasets, we deploy them at the interface between conventional prior work, where a learnt model is used to either generate reward signal for agent to maximise [12] or augment its observed state representation with hidden state information [21], our models serve dual role, providing both rewards and state augmentations, Figure 3 describes this setup in detail.
agents and their environments. Going beyond
M/
Kuvircement Dynamics
Pre-learnt 1STM MIT Model
RL Agent
d4
Figure 3: During RL agent training, our LSTM MIL models sit at the centre of the agent-environment loop by which states 8, and actions are exchanged. We focus episodic tasks, where the environment state periodically resets. The LSTM hidden state is simultancously reset to hg the start of an episode, then is iteratively updated over time t given the state-action pairs 8t, az At time t the environment state 8 is augmented with the post-update hidden state h/+1 by concatenation, and this augmented state is observed by the agent. St, a₁ and M4+1 are used to compute a reward r'4-1 following the relevant steps from Figure and the reward also sent to the agent. In the language of NMRDPs, the hidden state augmentation expands the agent's 's learning problem into an MDP by providing the additional information required to make the rewards Markovian. Note that unlike during learning of the MIL models, return predictions are never required.
4 Experiments and Results
After initially validating our models on several toy datasets (see Appendix D), we focus the bulk of our evaluation on five RL tasks. As running experiments with people is costly, we use the standard RM approach of generating synthetic preference data (here trajectory return labels) using ground truth oracle reward functions [12] (for a discussion comparing the use of oracle and human labels, see Appendix C). Unlike prior work, these oracle reward functions depend on historical information that cannot be recovered from the instantaneous environmental state, thereby emulating the disparity between the information that a human evaluator possesses while viewing trajectory sequentially, and that contained in the state alone. In this section, we introduce our RL tasks (Section 4.1), evaluate the quality of reward reconstruction (Section 4.2), investigate the use of MIL. RM models for agent training (Section 4.3), and evaluate their robustness to label noise (Section 4.4).
5

---

4.1 RL Task Descriptions
We apply our methods to five -oe-Marvviinn RL tasks, the first four of which are within a common 2D navigation environment and are specifically designed to capture different kinds of non-Markovvan structure. Each environment has two spawn zones and an episode time limit of T 100; see Figure 4. In each case, the environment state contains the I, y position of the agent only. The tasks involve moving into freasure zone, contingent on some hidden information that cannot be derived from the current.2 position, but is instead function of the full preceding trajectory. In the first two cases the hidden information varies with time only, but the other two it depends on the agent's 's past positions.
Timer For times t 50 the treasure gives reward of - 1 for each timestep that the agent spends inside it, before switching to +1 thereafter. Since time is not included in the environment state, recovering the reward function by only observing the agent's current position is impossible.
Moving The Timer task only captures a binary change, therefore we generalise it to be continuous. In this case, the treasure zone oscillates left and right at a constant speed. Again, this is not captured in the environment state, but can be recovered if the length of the preceding trajectory is known.
Key Before reaching the treasure zone, the agent must first enter a second zone to collect key; otherwise it receives 0 reward. As the key's status is not captured in the environment state, temporal dependency exists between the agent's 's past positions and the reward it obtains from the treasure.
Charger We generalise the Key task by replacing the key zone with a charging zone that builds up the amount of reward the agent will receive when it reaches the treasure. The reward now depends not only whether the agent visits zone (binary), but how long it spends there (continuous).
For the fifth and most complex task, we adapt Lunar Lander from OpenAl Gym [9], adding the condition that the lander should take off again and stably hover after 50 timesteps on the landing pad. This is analogous to the Charger task but with larger state-action space and longer episodes (T = 500). Further details on the tasks and MIL model hyperparameters given in Appendix E.
Timer
Moving
Key
Treasure:
without key +1 with key
when / S, 50 +1 when >> 50
Coostant +1 but oscillates horizentally
Key: collected when entered
Spawn zones
Charger
Lunar Lander
charge level
Spawn zone
Landing pad: when 50
Charging increments charge by 0.02 per step (max
Hover sone: when pad +1 when pad 50
when pad 50, increments pad by per step
Figure 4: Visualisations of the five non-Markovian RL tasks.
An important design decision for the L.STM-based models the size of the hidden state, as affects both performance and interpretability. For all the above tasks, we know priori that it is possible to capture the temporal dependencies in at most two dimensions, so we constrain our models use 2D) hidden states. This allows us to visualise and interpret the hidden representations in Section 5.1.
4.2 Reward Modelling Results
Below discuss the performance of the reward reconstruction for the different MIL RM models on our five RI. tasks. For each task, we generate initial trajectories to form our MII. RM datasets (see Appendix E). Results from MIL models trained on these trajectories are given in Table 1. We observe that the CSC Instance Space LSTM model is on average the best-performing model for predicting both trajectory returns and timestep rewards. While the Embedding Space LSTM model performs best at predicting return on the Key and Lunar Lander tasks (as is not constrained to the summation of reward predictions as in the other architectures), it struggles on the reward metric (due to the use of proxy post hoc method). As it is important for these models to achieve strong performance on both return and reward prediction, the Instance Space LSTM and CSC Instance Space LSTM models are better candidates than the Embedding Space LSTM. Also note that the Instance Space NN that serves as our Markovian RM baseline performs very poorly on return prediction, indicating that these tasks indeed cannot be Ieart without modelling temporal dependencies.

---

Table 1: MIL RM return (top) and reward (bottom) results, using ten repeats. The Lunar Lander
results are the average test set MSE of the top five models (with scaling; see Appendices E.4 and E.5). For the other tasks, each measurement is the test set MSE averaged over all ten repeats. The standard errors of the mean are given, and the Lunar Lander reward results are scaled by 10-5.
Model
Timer
Moving
Key
Charger
Lunar Lander
Instance Space NN
130.8 + 1.530
22.24 + 0.441 7.764 + 0.232
7.783 + 0.214
2.297 = 0.058
Embedding Space LSTM
3.151 + 0.662
13.04 + 0.899
0.360
0.055
0.689 : 0.124
0,416 = 0.048
Instance Space LSTM
7.313 + 2.627
11.13 + 1.169
0.488
0.062
0.628 + 0.126
1.223 + 0.431
CSC Instance Space LSTM
0.605 + 0.166
5.307 + 0.299
0.391 + 0.083
0.125 + 0.012
0.501 + 0.035
Instance Space NN
0.217 + 0.001
0.068 + 0.000
0.011 + 0.000
0.025 : 0.000
7.484 = 0.861
Embedding Space LSTM
101.8 + 60.35
3.033
+ 0.715
0.010 + 0.008
0,037 : 0.016
120.2 + 24.27
Instance Space LSTM
0.263 + 0.038
0.069
+ 0.005
0.002 + 0.000
0.005 0.001
9.336 - 3.116
CSC Instance Space LSTM
0.073 + 0.016
0.026
0.002
0.001 + 0.000
0.001
+ 0.000
7.365 + 1.032
4.3 RL Training Results
Following the method in Section 3.3, we then train Soft Actor-Critic [15] (Lunar Lander) and Deep Q-Network [31] (all others) RL agents to optimise the rewards learnt by the LSTM-based models. We evaluate agent performance in a post hoc manner by passing its trajectories to the relevant oracle. This evaluation provides an end-to-end measure of both reward reconstruction and policy learning, and is standard in RM [12]. We baseline against agents trained with access to: a) the oracle reward function and the oracle hidden states, and b) just the oracle reward function without hidden states (i.e, using only the environment states that are missing information). In Figure 5, observe that the CSC Instance Space LSTM model enables the best RI. agent performance, coming closest to the oracle. Interestingly, for the Timer and Lunar Lander tasks, the CSC Instance Space LSTM model actually outperforms the use of the oracle, suggesting that the learnt hidden states are easier to exploit for policy learning than the raw oracle state (we investigate what these models have learnt in Section 5.1). Note the poor performance of agents trained without hidden state information, which aligns with expectations. For further details on agent training. see Appendix F.
Timer
Moving
Key
Charger
Lunar Lander
Oncele Return 50
000
720
Note: lower
quartile for
400
400
Oracle
Oracle (without bidden state)
Embxdding Space LSTM
Instance Space LSTM
CSC Instance Space LSTM
Figure 5: RL performance for different training configurations on our five RL tasks. The results given are the medians and interquartile ranges. For the oracle results, we trained ten repeats, and for the MIL-LSTM results, we performed one RL training run for each MIL-LSTM model repeat.
For Lunar Lander, we perform a deeper analysis of RL training performance, by decomposing the oracle return curves from Figure 5 into the four reward components Rpad. Ruo contact, Rhover and Rshaping (see Appendix E. for definitions). The decomposed curves, shown in Figure 6, allow us to diagnose the origins of the performance disparity between runs using different LSTM model architectures. There is relatively little separation in performance on the shaping reward Rataping and pad contact reward Rpad (for the latter, all runs end up reliably achieving the maximum possible reward of 49, although those using Embedding Space LSTM models require significantly more training time). This suggests that all models have been able recover these components with reasonable fidelity. However, there are marked differences in performance on R Rmu_comiact and Rhover (the components relating to the second task stage of taking off and moving to the hover zone). For Roover, runs using the CSC Instance Space LSTM peak at return of around 200 from this component, while those using the other two models almost never achieve non-zero return, i.e. only the RL agents trained using the CSC Instance Space LSTM RM models reliably learn to hover. This indicates that the models have learnt very different representations of reward and hidden state dynamics, which are effective for policy learning in the case of the CSC model, and highly ineffective for the others.
7

---

Observe that runs using CSC Instance Space LSTM models outperform those with direct access the ground truth oracle on all components, and most markedly on Rnover This counterintuitive finding suggests that this model reliably learns hidden state representations that are easier for RI. agents leverage for policy learning than the ground truth ones, and potentially that certain emrors in the reward prediction may actually be beneficial for the purpose of helping agents to complete the underlying task (especially the hovering stage). In typical RL parlance: the model's reward function appears to be better shaped than the ground truth. The potential origins of this better-than-orack phenomenon are investigated in Figure A5 (Appendix G).
Rpad
Rec.contact
Rhover
Rshappng
0 Partial Retum 50
260
Episode Number
800
Oricle
Embedding Space LSTM
Iustance Space LSTM
CSC Instance Space LSTM
Figure 6: Decomposed oracle return curves for Lunar Lander.
4.4 Robustness to Mislabelling
In this work, the return labels are provided by oracles rather than real people. When using human evaluators, there is likely to be uncertainty in the labels, and it is important to evaluate model robustness against such noise [28]. We implement noise through label swapping [34]; this ensures the marginal label distribution remains the same and does not include out-of-distribution returns. In Figure 7, show how both return and reward prediction decay with noise levels increasing from O (no labels swapped) to 0.5 (half swapped). The rate, smoothness, and consistency (across three repeats) of this decay varies between tasks, with decays in return prediction generally being smoother. We observe that the CSC Instance Space LSTM model remains the strongest predictor of both return and reward in the majority of cases, indicating general robustness and providing evidence that the model should still be effective with imperfect human labels. On all metrics aside from Timer reward loss (where the mix of negative and positive rewards makes the effect of noise especially unpredictable), noise level of at least 0.3 is required for the CSC Instance Space LSTM model to perform as badly as the Instance Space NN baseline does with no noise at all.
Timer
Moving
Key
Charger
Retum MSE Les
15
100
20
0,005
0,02
0.02
Reward MSB
Noise Level 0.
0.5
0.5
0.5
Instance Space NN
Embedding Space LSTM
Instance Space LSTM
CSC Instance Space LSTM
Figure 7: Performance of MIL RM models subject to label noise. We omit the Embedding Space LSTM reward losses they are very high, and the Lunar Lander task due long training times.
5
Discussion
In this section, we seek interpret MIL. RM models, analysing the distribution of learnt hidden states (Section 5.1) as well as their temporal dynamics over the course of a trajectory (Section 5.2). Finally, in Section 5.3 discuss the limitations of this work and potential areas for future work.

---

5.1 Hidden State Analysis
The primary purpose of RM to perform accurate reward reconstruction to facilitate agent training, but there is secondary opportunity to improve understanding of human preferences through inter- pretability analysis of the learnt models. We can directly visualise the 2D LSTM hidden states of our oracle experiments, which enables qualitative comparison of the various model architectures (see Figure 8). Visualising the hidden states with respect to the temporal dependencies indicates that the CSC Instance Space LSTM model has learnt insightful hidden state representations, Breaking down the CSC Instance Space LSTM model hidden embeddings: for the Timer task, time is represented along curve, with sparser representation around t = 50 (the crossover point when the treasure becomes positíve). For the Moving task, time is similarly captured along with an additional notion of the change in treasure direction from right left. For the Key task, the binary state of no key key is separated, with additional partitioning based on & position, denoting the two different start points of the agent. In the Lunar Lander task, the model has learnt strong separation between states either side of the crossover point when the time on the pad is equal to 50, with high sparsity around the crossover point. In comparison, the Embedding and Instance Space LSTM models have not learnt as sparse a representation. We discuss the Charger task in Section 5.2.
Embedding Space LSTM
Instance Space LSTM
CSC Instance Space LSTM
Timer
50, Out is
50, Out
< 50, In
100
Timestop
Moving
Left, Out
Left, In
Right, Out
Right, In
100
Timestep
No Key, Out
Key
No Key,
Key, Out
Key. In
x Position
Charger
Charging
0
Charge Level
Lunar
Lander
On Pad Neither
300
Time on Pad
Figure 8: Learnt hidden state embeddings for MIL RM models. For each model and task, categorise the hidden state embeddings depending on the true environment state (first column for each model). In and Out environments states indicate whether the agent is in the treasure zone or not, and for the Moving task, Left and Right indicate the direction in which the goal is currently moving. We also provide labelling based temporal information (second column for each model). Furthermore, we include markers to indicate the hidden states for the centres of the agent spawn zones. In each case, we elected to use the best-performing repeat for each model assessed by the reward reconstruction (see Table 1). Note, for the Key task, as the temporal information is captured in the state categorisation (No Key vs Key), we the second column to show the relationship between the hidden embeddings and the agent's x position.
5.2 Trajectory Probing
We further interpret our models by visualising the learnt reward with respect to the environment state, and by using hand-specified probe trajectories to verify that the learnt hidden state transitions mimic the true transitions. We present the above for the CSC Instance Space LSTM model on the Charger task Figure (Appendix contains similar figures for all other tasks). The top row shows that the model has correctly learnt the relationships between position, charge, and reward (reward increases

---

in the treasure zone as charge increases). From the probes, we can see how the charge level can be recovered from the hidden states. We also note that the inflection point between under-charging and over-charging is captured, i.e., this is where the optimal charge level lies, subject to some noise based on where the agent starts the spawn zones. Furthermore, with the Challenging probe, we observe that the learnt hidden states align with the agent moving in and out of the charging zone.
Charge Level
Charge Level
Charge Level
Charge Level
Charge Level
Clarge Level
Reward
Optimal
Snder-charged (oracle return 33.6)
Over-charged (oracle return 30,00)
Challenging (oracle return 30.00)
(oracllc return
44.2)
Environment State Trajectory
99
x70
x
x21
Hidden State Trajectory
Out In Charging
Figure 9: Interpretability for the CSC Instance Space LSTM model on the Charger task. Top: the learnt relationship between agent position, charge level, and reward. Middle/bottom: Four probe trajectories demonstrating hidden state transitions as the trajectory progresses. Optimal: best possible return (charge to sufficient level then maximise time in treasure). Over-churged: continuing to charge after maximum charge of is reached. "xn" labels indicate the agent remains in position for timesteps. As in Figure 8, we use the best-performing model according to reward reconstruction.
5.3 Limitations and Future Work
Although we analyse the performance of our methods in the presence of noisy labels in Section 4.4, major area of future work is to apply our methods human labelling (for discussion of this, see Appendix C). Another area of future work involves more complex environments, for example the use of tasks with image observations, similar to the Atari environments in Open Gym [9]. Furthermore, we perform RM from either offline dataset or from only one RI. training iteration; an iterative bootstrapping approach with multiple RI. RM training iterations could lead to improved RI. results. There are also limitations with our MIL RM approach for the Lunar Lander task; see Appendix E.5 for details and suggestions for future work. More generally, we hope that our identification of the link between RM and MIL may inspire bidirectional transfer of tools and techniques.
6 Conclusion
We posed the problem of non- non-Markovian RM, which removes an unrealistic assumption about how humans evaluate temporally extended agent behaviours. After identifying an isomorphism between RM and MII,, proposed and evaluated novel MIL-inspired models that allow reconstruct non-Markovian reward functions, augment agent training, and interpret their learnt representations.
Acknowledgements
This work was funded by the AXA Research Fund, the UKRI Trustworthy Autonomous Systems Hub (EP/V00784X/1), and an EPSRC/Thales industrial CASE award. We would also like to thank the Universities of Southampton and Bristol, as well as the Alan Turing Institute, for their support. The authors acknowledge the use of the IRIDIS (Southampton) and BlueCrystal (Bristol) high- performance computing facilities and associated support services in the completion of this work.
10

---

References
[1] D. Ariely and Z. Carmon. Gestalt characteristics of experiences: The defining features of summarized events. Journal of Behavioral Decision Making, 13(2):191-011, 2000.
[2] F. Bacchus, C. Boutilier, and A. Grove. Rewarding behaviors. In Proceedings of the National Conference on Artificial Intelligence, pages 1160-1167, 1996.
[3] Bacchus, C. Boutilier, and A. Grove. Structured solution methods for non-markovian decision processes. In AAAI/IAА,, pages 112-117. Citeseer, 1997.
[4] A. Bajesy, D. P. Losey, M. K. O'Malley, and A. D. Dragan. Learning robot objectives from physical human interaction. In Conference on Robot Learning, pages 217-226. PMLR, 2017.
[5] B. Bakker. Reinforcement learning with Long Short-term Memory. Advances in Neural Information
Processing Systems, 14, 2001.
[6] C. Basu, BıyIk, He, M. Singhal, and D. Sadigh. Active learning of reward dynamics from hierarchical queries. In 2019 IEEE/RSJ International Conference on Intelligent Rohots and Systems (IROS), pages 120-127. IEEE, 2019.
[7] K. C. Berridge and J. P. O'Doherty. From experienced utility decision utility. In Neuroeco- nomics, pages 335-351 Elsevier, 2014.
[8] T. Bewley and F. Lecue. Interpretable Preference-basee Reinforcement Learning with Tree- Structured Reward Functions. In 21st International Conference on Autonomous Agents and Multiagent Systems, 2022.
[9] G. Brockman, V. Cheung, L. Pettersson, Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
[10] M.-A. Carbonneau, V. Cheplygina, E. Granger, and G. Gagnon. Multiple instance learning: A survey of problem characteristics and applications. Pattern Recognition, 77:329-353, 2018.
[11] L. Chan, A. Critch, and A. Dragan. The impacts of known and unknown demonstrator irra- tionality on reward inference. 2020.
[12] F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017.
[13] J. Early, C. Evers, and S. Ramchurn. Model Agnostic Interpretability for Multiple Instance Learning. In International Conference on Learning Representations, 2022.
[14] S. Griffith, K. Subramanian, Scholz, C. L. Isbell, and L. Thomaz. Policy shaping: Inte- grating human feedback with reinforcement learning. Advances in Neural Information Processing Systems, 26, 2013.
[15] Haarnoja, Zhou, P. Abbeel, and Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference Machine Learning, pages 1861-1870 PMLR, 2018
[16] D. Hadfield-Menell, S. Milli, Abbeel, S. J. Russell, and A. Dragan. Inverse reward design. Advances in Neural Information Processing Systems, 30, 2017.
[17] M. Hausknecht and P. Stone. Deep Recurrent Q-learning for Partially Observable MDPs. In 2015 AAAI Fall Symposium Series, 2015.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[19] Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 98): 1780, 1997.
[20] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[21] R. T. Icarte, T. Klassen, R. Valenzano, and S. Mcllraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In International Conference on Machine Learning, pages 2107-2116. PMLR, 2018
[22] M. Ilse, J. Tomezak, and M. Welling. Attention- based deep multiple instance learning. In International conference on machine learning pages 2127-2136. PMLR, 2018.

---

[23] F. Jarboui and V. Perchet. Trajectory representation learning for multi-task nmrdp planning. In
2020 25th International Conference on Puttern Recognition (ICPR), pages 6786-6793, IEEE, 2021. [24] H. J. Jeon, S. Milli, and A. Dragan. Reward Reward-rational rational (implicit) choice: A unifying formalism for reward learning. Advances in Neural Information Processing Systems, 33:4415-4422, 2020.
[25] L.. P. Kaelbling, M. Littman, and A. R. Cassandra. Planning and acting partially observable stochastic domains. Arrificial intelligence, 101(1-2):99-334 1998.
[26] D. Kahneman. Evaluation by moments: Past and future. Choices, values, and frames, pages 693-708, 2000.
[27] M. G. Kendall. Rank correlation methods. 1948.
[28] K. Lee, L. Smith, A. Dragan, and P. Abbeel. B-pref: Benchmarking preference-based rein- forcement learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and
Benchmurks Track, 2021.
[29] J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, and S. Legg. Scalable agent alignment via reward modeling: research direction. arXiv preprint arXiv: 2018.
[30] D. Li and Y. Zhang. Multi-instance learning algorithm based on lstm for chinese painting image classification. IEEE Access, 8:179336-179935
2020.
[31] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 51 518(7540):529-533, 2015.
[32] A. Y. Ng. S. J. Russell, et al. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning, volume 1, page 2. PMLR, 2000.
[33] S. Reddy, A. Dragan, S. Levine, Legg, and J. Leike. Learning human objectives by evaluating hypothetical behavior. In International Conference on Machine Learning, pages 8020-8029. PMLR, 2020.
[34] D. Rolnick, A. Veit, S. Belongie, and N. Shavit. Deep learning is robust to massive label noise. arXiv preprint arXiv: 1709 2017.
[35] S. Russell. Human Compatible: Artificial Intelligence and the Problem of Control. Penguin, 2019.
[36] A. Shah, S. Wadhwania, and J. Shah. Interactive robot training for non-markov tasks. arXiv
preprint arXiv:2003.02232 2020.
[37] Z. Shao, H. Bian, Y. Chen, Wang, Zhang, X. Ji, et Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems, 34, 2021.
[38] K. E. Stanovich. Higher-order preferences and the master rationality motive. Thinking & Reasoning, 14(1):111-12,, 2008
[39] S. Thićbaux, C. Gretton, Slaney, D. Price, and F. Kabanza. Deeosion- planning with non-markovian rewards. Journal of Artificial Intelligence Research, 25: 17-74, 2006.
[40] J. Tien, J. Z.-Y. He, Z. Erickson, A. D. Dragan, and D. Brown. A study of causal confusion in preference-basee reward learning. arXiv preprint arXiv:2204.0666 066001 2022.
[41] R. Toro Icarte, E. Waldie, T. Klassen, R. Valenzano, M. Castro, and S. Mcllraith. Learning reward machines for partially observable reinforcement learning. Advances in Neural Information Processing Systems, 32, 2019.
[42] M. Tu, J. Huang, X. He, and B. Zhou. Multiple instance learning with graph neural networks. arXiv preprint 1906.008990048 2019.
[43] H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[44] K. Wang, Oramas, and Tuytelaars. In defense of Istms for addressing multiple instance learning problems. In Proceedings of the Asian Conference on Computer Vision, 2020.
[45] X. Wang, Y. Yan, P. Tang, X. Bai, and W. Liu. Revisiting multiple instance neural networks. Pattern Recognition, 74:15-24, 2018.
[46] D. Wierstra, A. Förster, J. Peters, and J. Schmidhuber. Recurrett policy gradients. Logic Journal of the IGPL, 18(5):620-633 2010.
12


Reviews:
[{"review_id": "swivzwOOSo4", "reviewer": "Reviewer_kcLW", "summary": "This paper defines non-Markovian reward modeling and proposes a novel LSTM-based model to capture temporal dependencies in reward modeling. It also adopts a multiple instance learning (MIL) framework to handle temporal dependencies with reward labels. It encodes hidden state information using LSTM and also produces explicit reward predictions with skip-connection to the current state and action features.\nIt optimizes LSTM-based models on offline trajectory datasets. Quantitative experimental results on return and reward prediction show the model's compelling reward function modeling quality. The authors also present qualitative results on the interpretability of the learned hidden embeddings.\n", "questions": "1. How important the proposed non-Markovian reward modeling is compared to other temporal information encoding models (such as Flare [1] or SPR [2])? \n\n2. How many timesteps the proposed LSTM-based model can encode?\n\n3. What did you use for the LSTM hidden states 2D visualization? T-SNE? And do you have any other visualization results on other common RL tasks with discrete action space such as Atari games?\n\n\n- [1] Reinforcement Learning with Latent Flow, Shang et. al., NeurIPS 2021\n- [2] Data-Efficient Reinforcement Learning with Self-Predictive Representations, Schwarzer et. al., ICLR 2021", "limitations": "Yes, the authors adequately addressed the limitations and potential negative social impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe paper is well written and easy to read and understand. Figure 2 clearly demonstrates the proposed method and the baseline models. The experiments are comprehensive and the visualization result is quite inspiring. They also propose novel RL tasks that can show the effectiveness of their model. For significance, learning the non-Markovian reward function is the cornerstone of a successful real-world application using RL, especially with human interactions.\n\nWeaknesses:\n\nI appreciate the idea and results of the conceptual toy RL tasks experiments, however, it would be better if there were comparisons with other models in the more complex RL tasks such as Montezuma's Revenge where modeling non-Markovian rewards is important.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "sOYehp1fl7N", "reviewer": "Reviewer_2DcX", "summary": "This paper proposes a new method to deal with non-Markovian rewards based on reformulating the problem as multiple instance learning with LSTM. The authors test a few alternative ways of doing this in 4 environments designed to demonstrate non-Markovain properties.", "questions": "- Could the authors clarify if the issue with the difference in partial bags as I described takes place in their experiments?\n- Could the authors clarify why it is chosen for the agent to reuse latent states from reward modelling rather than to train its own LSTM?\n- Could the authors elaborate why the baseline that just replaces each state with frame stacked state (several states concatenated, often used as a simplification for recurrence) would not suit this problem?\n- The authors train Deep Q-Network, but as far as I understand the states are just 2-dimensional, how important is it to have deep architectures in this case?\n- The authors say that the transformer architecture would be unsuitable for the temporal structure of the problem. Why would it be the case with temporally masked transformers?\n- How was the dataset for RM training (from lines 210-213) split into training and test? Are the results in Table 1 on the test set?\n", "limitations": "Some limitations are discussed at the end of the paper. In the review I also pointed to/asked for clarification regarding some potential limitations. No potential societal impact is mentioned.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: \n\nWhile Markovian reward modelling is quite well studied, non-Markovian rewards are investigated much less. The proposed improvements to MIL with LSTM (instance space and skip connections) are not particularly original, but they effectively address the shortcomings of the prior methods.  \n\nQuality: \n\nI am not completely convinced about the baseline \"Base Case: Embedding Space LSTM\" model where the rewards are obtained by taking the difference in partial bag labels. Suppose the task is to grasp an object. If I am given two sequences to annotate: 1) s1=not grasped, s2=not grasped, s3=not grasped, s4=grasped, s5=grasped and 2) s1=not grasped, s2=not grasped, s3=not grasped, s4=grasped, I could assign both of them label 1. Then, using the difference between partial bags would result in the reward of s5 to be 0 which is intuitively not correct. I expect that with predictions from LSTM we may face a similar problem. If it is the case, maybe the inferior performance of this method could be due to this.\nIn section 3.3 the authors explain that the RL agent observes hidden states along the reward from LSTM. However, I am wondering if it is suitable to use hidden state representation of the reward model for agent training. Maybe it is better to have a recurrent agent architecture that trains its own latent representation.\nFor the baselines in Figure 5, I think it would be more informative to have RL with recurrent architecture + oracle rewards as the upper bound on the performance and it would avoid the situation when an agent with learnt reward outperforms the oracle.\n\nClarity: \n\nThe paper is very well written and easy to follow. I found the visualizations in Figures 7 and 8 very informative. \nThe paper positions its problem as originating from an unrealistic assumption on human evaluation of temporally-extended behaviour. However, I believe the non-Markovian rewards might be important in practice in some tasks that are inherently non-Markovian and the success can't be determined only by a given state, but a whole sequence must be considered. Some of such tasks are studied in the experiments, but the point that these are different types of tasks was not made clear. It could be also informative as an additional ablation to consider simple Markovian tasks (with Markovian reward), for example, like navigating to a given target to confirm that the method of this paper works in that case too.\n\nSignificance: \n\nMy biggest concern in this paper is the complexity of the evaluation environments. While the proposed environments and tasks work sufficiently well to demonstrate the proof of concept, they are rather simplistic (grid world) and some tasks are a bit artificial. I am wondering if the proposed method would face any difficulties when dealing with a more challenging environment, for example, some control problems in simulation, or something that has image-based states. I think that in order to have a larger impact in the community, this paper should include results on more complex environments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K3dZT8Ce91d", "reviewer": "Reviewer_2P9E", "summary": "This paper presents reward modeling methods for non-markovian rewards, leveraging LSTMs to update a hidden state and encode the trajectory history. It performs experiments in custom environments with non-markovian rewards, with a baseline that assumes markovian rewards. Their various proposed methods perform better than the baseline. The theory behind the method is sound and the experiments are sound too.", "questions": "Without resorting to completely standard benchmarks where the reward is markovian, could you make a stronger case that the methods are applicable and helpful for larger-scale, real-world tasks?\n\nAnd more importantly, can your method actually handle the complexity of human-labels rewards as you talk about in the introduction but don’t demonstrate in your experiments?", "limitations": "It is not clear what the limitation of this work is as it has to potential to be very general; the experiments do not go far enough to find the limits of the methods. I see no obvious negative societal impact of this work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Good points: I think the connection between reward modeling with trajectory-level labeling and multi-instance learning is interesting and well explained. The experiments give good (although artificial) examples of non-markovian reward decision process. The writing is good and the paper is clear, and the reviewer had a good understanding of the presented methods after reading the paper.\n\nMy main gripe with the method and theory is the use of the NMRDP (non-markovian reward decision process), which can be viewed as a specialisation of a POMDP where only the reward depend on the hidden state. This would be a useful formalism if it led to a simpler algorithm, but the described algorithm resembles exactly what you would get if you tried to solve the reward modeling problem for POMDPs using LSTMs. I realise that the experiments only contain the special case of NMRDPs, but there is a missed opportunity in presenting a more general solution here. The argument of using this formalism to model human behaviour when assigning rewards is unconvincing to me as it is not tested in the experiment.\n\nSpeaking of this, I feel the experiments are too weak and obviously make it impossible for the baseline to perform well. They do demonstrate that the presented methods can handle non-markovian rewards in these limited settings. The problem is the applicability of this setting to more complex or better-known tasks; without resorting to completely standard benchmarks where the reward is markovian, could you make a stronger case that the methods are applicable and helpful for larger-scale, real-world tasks? And more importantly, can your method actually handle the complexity of human-labels rewards as you talk about in the introduction but don’t demonstrate in your experiments? In my opinion, addressing the second question with a set of experiment with real human labels would also address the first question.\n\nTo summarize 1) I think the method would be better presented as reward-modeling for POMDPs (where lstms have been used to encode the history in many works, see Dreamer, PlaNet, reccurent DQN) at which point the solution becomes obvious. 2) The experiments, while valid, do not convince the reviewer of the wider applicability of their methods.\n\nSmall comments: \n\nThe abstract uses a lot of specialized language RM, MIL, “provide interpretable learn hidden information” that wasn’t really clear to the reviewer until *after* reading the paper and coming back to the abstract.\n\nI find it awkward that the reward is a function of both s_t and h_{t+1}. I realize that it leads to better performance in practice, but it shouldn’t be necessary and maybe drop it from the math.\n\nSection 3.2, and the paper in general, completely ignores the existing body of work using LSTMs (and RNNs in general) for POMDPs, which leads to algorithms similar to yours (see for example Hausknecht, Matthew, and Peter Stone. \"Deep recurrent q-learning for partially observable mdps.\" 2015 aaai fall symposium series. 2015.) I think at least a mention should be added; I think the idea of using LSTMs specifically to handle non-markovian reward is simply a specialisation of this existing idea of using RNNs to encode past observations in POMDPs. It's not a bad thing per se, but should be mentioned.\n\nLine 229 mention an “interesting” case when CSC and instance-space LSTM beats the oracles; the reviewer does not think it is interesting but rather suspicious. This could be stemming from a lack of hyperparameter search or uncertainty in the results.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept