Online Inverse Reinforcement Learning with Learned Observation Model
Authors: Saurabh Arora
Prashant Doshi
Bikramjit Banerjee
Venue: CoRL 2022 Poster (2022)
Keywords: Observation model
Inverse reinforcement learning
Maximum entropy

Abstract: With the motivation of extending incremental inverse reinforcement learning (I2RL) to real-world robotics applications with noisy observations as well as an unknown observation model, we introduce a new method (RIMEO) that approximates the observation model in order to best estimate the noise-free ground truth underlying the observations. It learns a maximum entropy distribution over the observation features governing the perception process, and then uses the inferred observation model to learn the reward function. Experimental evaluation is performed in two robotics tasks: (1) post-harvest vegetable sorting with a Sawyer arm based on human demonstration, and (2) breaching a perimeter patrol by two Turtlebots. Our experiments reveal that RIMEO learns a more accurate policy compared to (a) a state-of-the-art IRL method that does not directly learn an observation model, and (b) a custom baseline that learns a less sophisticated observation model. Furthermore, we show that RIMEO admits formal guarantees of monotonic convergence and a sample complexity bound.

Online Inverse Reinforcement Learning with Learned Observation Model
Saurabh Arora & Prashant Doshi
Bikramjit Banerjee
THINC Lab, Dept. of Computer Science University of Georgia, Athens, GA United States of America {sa08751, pdoshi Ouga. edu
School of CSCE
Univ. Southern Mississippi, Hattiesburg, MS United States America
Bikramjit Banerj Canerjee@usm edu
Abstract: With the motivation of extending incremental inverse reinforcement learning (12RL) real-world robotics applications with noisy observations as well as an unknown observation model, we introduce a new method (RIMEO) that approximates the observation model order to best estimate the noise-free free ground truth underlying the observations. It learns a maximum entropy distribution over the observation features governing the perception process, and then uses the inferred observation model to learn the reward function. Experimental evaluation is performed in two roboties tasks: (1) post-harvest vegetable sorting with a Sawyer arm based on human demonstration, and (2) breaching perimeter patrol by two Turtlebots. Our experiments reveal that RIMEO learns more accurate policy compared to (a) state-of-the-an IRL method that does not directly learn an observation model, and (b) a custom baseline that learns a less sophisticated observation model. Furthermore, show that RIMEO admits formal guarantees of monotonic convergence and a sample complexity bound.
Keywords: Observation model, Inverse reinforcement learning, Maximum entropy
Introduction
Inverse reinforcement learning (IRL) methods learn the specification for task that best explains the observed behavior of an expert executing that task [1, 2], in the form of reward function for Markov decision process [3]. IRL methods generally assume that the learner's perception is noise-free and that the learning occurs offline. However, real-world sensing often noisy and agents may need to learn incrementally because not all the training data is available in one batch. Motivated by the challenging goal of bringing robotic automation to post-harvest processing lines for vegetables, consider examples of noisy perception by robot learner observing human sorting onions by removing the blemished ones. To learn this complicated manipulation task from its observations, the learner needs to unambiguously understand when is onion considered blemished, and how do the various locations of the expert's 's hand help realize the task (e.g., hand hovering over table, hand near eyes of the sorter); Fig. ((a). When the expert places blemished onion in bin for bad onions, as the robot's sensing is imperfect, the blemish may not be perceived correctly by the learner's s camera. This introduces error in the perceived state because the learner incorrectly sees the human placing an unblemished onion in a destination meant for bad onions (see Fig. (bb) top). Another example of imperfect perception is shown in Fig. 1(b) bottom: although the expert is holding an onion just above the table, the learner may incorrectly perceive it to be in the inspection (or eye) region. This ambiguity occurs because these two regions are close proximity.
Shahryari and Doshi [4] approach the problem of offline IRL under noisy observation using the Robust IRL method, but they assume that the learner knows the observation (or sensor) model. This assumption may not be pragmatic because the observation model is typically a function of obscure variables including inaccuracies in sensing hardware and computer vision. Real-world robotic
6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand.

---

(b)
Figure 1: (a) (top) A human demonstrator picks an onion, inspects it closely to check if it is blemished, and places it in the bin finding it to be blemished. (bottom) The learner (Sawyer) robot uses the learned policy imitate the observed behavior. (b) (top) A blemished onion detected as being unblemished. (bottom-left) An onion held the hover region. (bottom-right) An onion held at the eye (or in inspection) region.
applications need a method that can learn the unknown observation model under noisy input and use that model to learn a reward function.
When a learner passively watches the expert perform the task, the observations arrive as a continuous stream of data over an extended period of time. However, methods such Robust IRL require all observations to be available together as a batch. Current IRI. applications need a method that can learn online from incremental noisy input without priori knowledge of the sensor model. To meet this challenge, we present a method that generalizes the batch Robust IRL to the incremental setting, and with an unknown observation model. Our method engages in robust incremental IRL with maximum entropy estimation of the observation model (labeled as RIMEO), and is an instantiation of the recent 12RL framework [5]. We formulate it as a sequence of incremental learning sessions where each session is a two-step process maximum entropy learning of the observation model of the learner, followed by inversely learning the reward function of the expert by utilizing the learned observation model.
Using noisy observations of human engaged in sorting onions, evaluate the performance of RIMEO in a Gazebo sim of the onion-sorting domain in which Sawyer processes the observations incrementally and eventually imitates the task of sorting the onions. We show that it generates significantly improved sorting performance compared to a recent end-to-end IRL baseline that accepts raw images as input. As this and other IRL. methods do not directly learn an observation model, we also evaluate the comparative advantage of RIMEO against a custom baseline which learns a less sophisticated observation model. Similar results are observed in a second task of a Turtlebot learning to breach patrol by another. Furthermore, we formally show that RIMEO's is data likelihood improves monotonically with more sessions, and admits a probabilistic bound for sample complexity.
2 Related Work
Observation noise not widely addressed by the IRL literature, which tends assume that the state-action data is accurate. However, this assumption may not hold in robotic applications where the learner using (noisy) sensors to watch the expert (a paradigm called naan-from-observation (L.fO).. Robust IRL. [4] was an early technique, which estimated the ground truth underlying the noisy observations using a known observation model. But, it did not address cases where the observation model is unknown, Bayesian model [6] was presented to enable IRL in that confound the observations of the expert may be present. The model offers way to sample the source of each observation and a hyperprior over the observation models is defined. Bayesian IRL has also recently been extended to contexts involving both occluded and noisy observations using marginal maxiimum-a-pssterii inference [7]. However, the observation or sensor model assumed to be known in order utilize this method.
when the data is only available incrementally. More recently, hierarchical
environments where elements
We also note IRL. methods that learn the stochastic transition dynamics, either by using importance sampling [8] or by decomposing each transition into its component features and learning from feature aggregates [9]. Adapting the former method for unknown observation models requires a preexisting near-perfect base observation model, which may get unrealistic. We take inspiration from

---

the latter method, where, analogously to Bogert and Doshi [9], we decompose an observation into its component observation features followed by learning the feature probabilities, and the learner generalizes from that information.
3 Background
Inverse RL aims to find reward function under which the observed behavior of the expert optimal. with the behavior modeled as incomplete Markov decision process (MDP) (S, A,T) [10, 2]. Abeel and Ng [11] modeled the reward function as linear combination of K binary features, Φk- each of which maps a state and an action to a value in {0.1). The linearized reward function is Rø(s, a) = 0f a) 2==1
Oz (s, a). where 0g are the feature weights in O. The learner's task is to find a 0 that makes the observed behavior optimal in the completed MDP (S, A, Ro, T))
Let E be the space of all possible ground truth trajectories of finite time-step length T (i.c., sequence T' (s, a) pairs) executable by the expert, and Ed be the set of trajectories observed by the learner. The cumulative value of any feature de for trajectory is given by function fs: = -> R defined as (5) = Σ (s.a),EE 7 ( a) where is the temporal index of a state-action pair in the trajectory. Ziebart et al. [12] solves IRL by finding the probability distribution over E\t that exhibits the maximum entropy and is constrained match the observed feature expectations:
max - A
- EEE
P(E) log P(E) subject to Σ
P(E) = 1, and Ez Ez(Øn] = SuVK = 1... (1) _EE=
Here, A is the space of all distributions over the finite E, and Fe [p6] & =EES P(S) fx(E). The learner empirically estimates the true feature expectation as db E ΣeER, fa(E).
3.1 Robust IRL with Known Observation Model
We define the observation model as function that gives the likelihood of a perceived state-action pair (s, a). given the demonstrated (ground truth) state-action pair (s, a)g 1. That is,
Os((s, a)., (s, a),,) A P((s, a). a))))
the RHS being the probability of the learner observing (s, a), when the expert actually demonstrated
(s, g).- Extending this observation model to trajectories, for ground truth trajectory & E. the probability of observing trajectory & Ed is
P(Eo1(g) = II
(((s,a)).(s,,))))
(3)
The batch learning method of Robust IRL [4] uses this observation model to learn the reward function under noisy observations. In the process, estimates the tth feature expectation as
Ed Σ C.EE E PIEolE::0)faAE
(4)
where (E: 0) = n (P(olE,)) P(E;: 0) and P(Eg; 0) is the probability the generation a trajectory Eg using a policy computed with O. Then the program of (1) becomes
max
P(ç., Eo) log P(E., Eo)
subject to
Σ E
P(E., = and Ez(
δe.u Vk:
E
(5)
where EzlØi] = ES.EE, Σe P(so, Eg) fx(g)). We extend this optimization problem to the case of unknown observation model and online learning in Sections and 5, respectively.
3.2 I2RL Framework
Arora et al. [5] formulate incremental IRL as a sequence of sessions with R3 the 7th session's estimate of the expert's 's reward function. A formal definition of incremental IRL follows.
Although we use state-action pair an observation, the robust-IRL method is generalizable to any form of observations [4].

---

DEFN 3.1 (SESSION). A session of 12AL, ((MDP Ed,s, (iis-1,,,1)-- N takes as input the expert's s incomplete MDP, the current (ith demonstration, Ea, the sufficient statistic summarizing previous i sessions (01:-11), and the reward function estimated previously. It yields a revised estimate of the expert's reward rewardfuncctin function, RG-
DEEN 3.2 (12RL). Incremental IRL. is a sequence of learning sessions (G(MDP/E Ed.1. 0, Ro). (2(M Ed.2, 01:1, R1); : Ss Es...,) which continue indefinitely or until stopping criterion assessing convergence met.
As measure of how well the learned reward function explains the observed data, we may use the log likelihood of the accumulated demonstration received up to the 7th session.
4 Learning the Observation Model
Our overall approach is approximate the unknown observation model using underlying feature set shared among observations, and then fit it to the noisy observations via maximum entropy optimization. Let // be this set of m observation features. Each feature, by V. E {1, m) is a predicate that is true if and only if a specific characteristic represented by that feature appears in the observed state-action pair under consideration. Let yoou be the set of indicators, whose member Wg. assumes a value based on comparing the value of a feature on ground truth pair (8, a), and on observation (s, a) v/ 1, v, )=))) where 1/j V and j E
We assume that the learner knows the feature set and that the features are independent. For each (s, a). Ed, (s, a). y E E, we approximate the observation probability (Eq. 2) to be the product of (unknown) probabilities of the features 4 according to the values taken by the corresponding indicators 2/..
Obl(s,a) (s((8)))):(s,,)))) (8,
II
P(O))
II
P(C,)
(6)
Here ((bt)) can be interpreted as the the probability of the jth feature being non-noisy, and (())) = 1- P(ey). Though the learner knows the features, cannot see the noise-free ground truth state- action pairs and those pairs not included in the demonstration. Therefore, obtaining a complete observation model requires inferring distribution over observation features for all state-action pairs, regardless of whether they are observed or not.
If deep neural network based models such YOLO [13] and SA-Net [14] are used to process raw videos of expert performing the task, we may obtain stochastic confidence scores for the state-action labels which we denote using the function ((ss,a) (Fig. 2). The likelihood of an observation (s, a), given any ground truth (s, a) can be empirically estimated as a running average of the confidence scores from observed trajectories:
Eekimum-EEntrope Observation Model
Ō((s, ()),))).))) En,eE, ElS,tes, c((ss (s.)).
EqneE, Exs.)eE, 1 (s.a). (s,a)
Utilizing the principle of maximum entropy (a form of least commitment), the learner infers a unique distribution over the observation features V which has the maximum entropy, while satisfying the constraint imposed by Eqs. 6 and that the aggregated observation feature probabilities match the corresponding empirical estimates of observation likelihoods. As the ground truth is not available to the learner, it may instead consider possible state-action pairs that could have been in the input, (s, a)2 where is the estimated ground truth from the expert, rather than considering all possible state-acto acteonction pairs. The optimization problem is:
Σ Eeeew (P(w,) log ((w;) + P(Ws) log P()))) subject to
max - As
II =1
(())
II P P(,,) ((s. 4 (s. E, and P(),) + P(5,) Viy E V.
The Lagrangian relaxation of the above program is:

---

P((;)) --
[(V,v,))=- =
P(by) + p()y) ) log P(0Ø)
+
Σ
aag
II U90=1 19-00
P())) P(
+ Σ Xo ((P(1))) P((5)) - EE
( (x, EE
(8)
where v, are vectors of multipliers 90 and Xay-
Notice that the same feature bhy could be shared by multiple, distinct (s, a)g and observation (s, a) pairs, which then activate v6g.9. Several of these active indicators for various j can be present in the LHS of the first constraint and these receive probability in aggregate from Ô. Bard [15] was the first note that maximum entropy approach is beneficial when just the aggregate prob- abilities are available. The critical points of a Lagrangian often occur at saddle points rather than local maxima or minima [16], and numerical optimization solvers may not find the sad- dle points. Therefore, we modify the Lagrangian objective to make critical points occur at local
optima, L' L'= V OC + (P()))
ac Teni
AC and use limited memory Broyden-Fletcher-
Goldfarb-Shannee (L-BFGS) [17, 18] unconstrained optimization solver to learn the maximum-entropy distribution P Equation 6 then obtains the observation model from this distribution, which paves the way to obtain P(E.lE,) using Eq. 3 making it possible to apply the previous Robust IRL technique.
5 Robust Incremental IRL
To extend this process to an online setting, the learner uses P? learned at the beginning of session to estimate the feature expectations:
0
Σ Σ () E EyEE
Eapl
Σ Σ
(9)
foEFd EyE
Let Ed,1: be the cumulative demonstration up to the 2th session, and obi be the corresponding cumulative feature expectations. The learner updates the latter from session to session as
P
4
1
Σ
Pia(EalEo:0) = Eatial Σ EyEE
Σ
Σ
(EalE fx(Ee)
Ed,itl
EoEE
EyE
Σ
pf' (Eol (ffe((())((())) 0))s(()))
(5
ا1:ا1 + Ea,i
(10)
| Earl
EyE
5.1 The RIMEO Method
Following Def. 3.1, session of RIMBO is ((MMP) Ed,i, a1:-1, 0i-1) where sufficient statistic 01:1-1 contains the summary up to the last session (i (i-1) 1), viz., cumulative size of the demonstration set set ((Ed,:;1-1))) l), the learned P* values from the optimization in Eq. 8, and the cumulative feature expec- tations (6 ). Specifically, 01:1-1 = (( & This information, together with Ed and 0011 is sufficient for Fq. 9 and thence Eq. 10. For demonstration Ed,; of the current session i, the scores <( c(sssa)) a)) for input (s, a) pairs are available from the perception pipeline. The process starts by computing Ô using the scores and Eq. 7. The resulting Ô is plugged into Eq. to get VL', using which Pr; is updated by the L-BFGS gradient ascent starting with the initial value of Pi.-1(()) The learned Pt: are then used to compute the updated observation model probabilities P((s, a) (s, a)
(Eqs. 2, 6). This extends to a corresponding update for full trajectories, PIsi(5 Eg. 0) (Eq. 3). Solving Eq. the learner uses MCMC sample ground truth trajectories from (r 0) for computing
the feature expectations Øg.. followed by computation 21: via incremental update (Fq. 10). Finally, the latter is used for the main Robust IRL optimization (Eq. 5) via L-BFGS gradient descent to learn
5

---

Perception of RGB-D streams
Robust 12RL using MaxEnt with learned øbs. model
YOLO
(s.a);
session
Max: Entropy
(5,a)z
Es5,c(5.a)
P
learn ebs. model
(s. (s.ahoaal
Eag ((ss..)),((sss
Robust-IRI
R;(s,a)
with obs. model
RL under noise
SA-Net
Figure 2: An overview RIMEO. On the left is an image stream depicting the hidden ground truth, (s, a) of the expert's bchavior. As the perception process is not perfect, the input to the learning algorithm is noisy. This input is an array possibly incorrect observations and coreesoonding (classification) scores, c((s, o).). that measure possible match with unknown (s; a).
the reward weights (0') and corresponding policy Due to space constraint, we present the detailed algorithm with links our GitHub code Appendix A included in the supplementary file which is available at http: /ttr///tinn //thtthinc..//// cs. uga. edw/files/ abdooLLL suspsp2--pppppppe .dff.
5.2 Convergence Properties of RIMEO
While some of the results in this section analogous those for I2RL [5], Lemma is the main original result this paper since it incorporates the additional uncertainty due to noisy observations and unknown observation model. Due space constraint, we present all proofs in the Appendix B. The demonstration agglikelihood for session is given by (L Q1:i-11 O²-1).
LEMMA (MONOTONICITY). The demonstration likelihood increases monotonically with each new session, (L,, 01 - - 01.
0i > 0, when Ed > Ea,a]-
Note that feature expectations estimated under noisy perception with an estimated observation model, Ta P from Eq. 10. is an approximation of the expectations when the observation model known accurately. d1: computed by sampling the hidden ground truth E, from P(S, 0²- This, in turn, an approximation of the expectations estimated under noise-free perception, 31 As in I2RL [5], we assume that the latter estimation using n1, samples obeys Hoeffding bounds, i.e.,
P( 0 S E&) 2 - 8,, where 8, 2K exp( exp(-2n,e²) Additionally for this paper, assume that observation model estimation with n. samples also obeys Hoeffding bounds as P(max 10 Ô < E.) 2 1 da/ K, where O is the true observation model (of which Ô is an estimate), and 8, 2K / (xx(=-2n,e)) We also assume that all features in V are observed in the very first session. Then the following lemma holds.
LEMMA 2 (CONSTRAINT BOUND) ). Under the assumptions stated above, with probability at least max(0, 8,): (1 7) (F=lox) 31H @brt)) cr,Vk {1, K}. where 6, 5 + 5g + 5 & = + [ c... L is the longest trajectory length, and e, 5 are as defined in Theorem I in [5].
Lemma 2 allows us to probabilistically bound the error in the log likelihood for RIMEO, leading to the main result below specifying the confidence in its convergence.
THROREM (CONFIDENCE) ). Let Er. 6, be as defined in Lemma 2, and O' be the solution of session
2Kc, with confidence at least i for RIMEO. Then (f [a,I=) LL. Ed,i, 01:i-1, 0 fr-1) S (1-7))
ma 1 -y), where fg are the true weights of the expert.
6 Experiments
Our primary domain for evaluating RIMEO is the onion sorting illustrated Fig 1. The expert aims to identify blemished onions from a collection of onions on table, and drop blemished ones in a bin while allowing others to pass. A state of the task is composed of these features: onion location and end-effector location (on table, picked up in hover region, at eye level (under inspection), near the

---

0.7
RIMEO RIFOM
0.6
KL Divergenca
Method
TP..R.N.TN
Precision Recall%
0.6
Expert
12,5
100.00, 58.33
0.4
RIMEO
8,1.
88.89, 66.67
0.3
RIFOM
8.3.9.4
72.73. 66.67
0.2
RIMEO R IIFO
AIRI
5,4.8.7
55.56, 41.67
0.1
MLIRL
7.3.9.5
70.00. 58.33
MI
10 16
Number of sassions
Number
Figure 3: (Ieft) Divergence between (())) and (()) for simulated onion sorting data, (middle) LBA of the learned policy on the human demonstration data. (right) Column label TP denotes true positive (# blemished onions in bin), FP is false positive (# good onions in bin), TN true negative (# good onions remaining on table), and FN denotes false negatives (# blemished onions remaining on tablc).
bin for blemished onion]. and quality prediction {blemished, unblemished, or unknown], resulting in 22 states. The actions involve attending to a new onion on the table, picking up, bringing the grasped onion closer to eye and inspecting it, placing it in the bin, and placing it back on the table. We utilize K 11 predicates as reward features, (s. a), listed in Appendix C. To learn the observation model, we use 8 binary predicates y as observation features," 2 also listed in Appendix C.
Baselines: We MLIRL [19] and AIRL [20] as baselines to compare with RIMEO. AIRL is state- of-the-art and incremental because the learned weights of the discriminator and generator nets are a sufficient statistic. To our knowledge, we create a custom baseline using the frequentist approach, which uses robust I2RL with frequentist estimation of the observation model (RIFOM). While RIMEO solves an optimization problem to learn P RIFOM assumes that the ground truth is the same as the observation and learns P. as the running average of normalized frequencies of observation features activated in the demonstration.
other IRI. method infers observation model. Therefore,
Setup: We use videos of a human sorting onions (Fig. I) as the training demonstration. We utilize SA-Net [14] and YOLO [13] for identifying the sequences of states in the videos. The perception pipeline considers the state with the highest prediction score c(s, a) as the current state in the input trajectory& & E This noisy trajectory and the confidence scores are input to 12RL sessions of all methods. The domain is simulated in Gazebo 7 (ROS Melodic) with the cobot Sawyer as the learner. The experiments were run desktop with Intel Xeon CPU (E5-1603, 2.8GHz) and 16GB RAM.
Performance Results: We evaluate the divergence of the learned P (1)) from a simulated dataset for which the true distribution p(t) is known. We introduce noise as P P( a())a))) o th ((s,, a) 0.3 for j = BlemishedOOnion. OnionToBin, OnionTooabie and 0 for other features. Figure 3(efft) shows the KL divergences between the distributions P(t) and P (w) with sessions i. Each data point is averaged over 50 runs, and each session has 2 trajectories of size 3 as input. The divergence for RIMEO is significantly lower than that of RIFOM for all the sessions, indicating that RIMEO Iearns more accurate P. Observe that the divergence starts stabilizing after the sixth session. As AIRL or MLIRL do not directly learn an observation model, we do not report their divergence measures.
Next, we measure the learned behavior accuracy (LBA), which is the proportion of states in which the action prescribed by the learned policy matches the actual action of the expert. Figure 3midddee) shows the average LBA for our human demonstration data sel as the number of sessions increases, for all three methods. While AIRL's LBA improves initially (until session 1 11), it flattens out to a value (maximum 0.48) that is considerably lower than the other two methods. On average, AIRI. is also slower to converge (about minutes) as compared to RIMEO (96.69 seconds). This performance is likely due to the need of neural network based adversarial techniques for more training data and that AIRL suffers when the training data (that is used by the discriminator) is noisy. Furthermore, RIFOM's LBA flattens out around 23% lower than RIMEO's. s. This is because the maximum-entrop optimization of the underlying observation features in RIMEO generalizes the learned information across (s a) pairs not present in the demonstration, but the frequentist baseline fails to generalize. RIMEO eventually reaches an LBA of 96% whereas RIFOM's 's stays around 70%. However, as RIMEO solves an additional optimization problem for learning P. it takes longer to converge (96.69 seconds, on average) compared to RIFOM's (85.99 seconds). MLIRL's LBA is similar RIFOM's,
We used Pearson product- product-moment moment correlation (0.03 with p-value 0.04, averaged over all pairs of features from demonstration) to confirm that these observation features are sufficiently independent.
7

---

RIMEO RIFOM
0.8
Divergence
0.6
0.3
LBA
0.4
0.2
RIMEO RIFOM
0.1
0.2
0.4
5
Number of sessions
Number of sessions
Figure 4: (left) Patroller (red) navigates hallways [21], and learner (green) aims to pass undetected. The colored regions (long hallway, turning points, and three small divisions in small hallways on both sides) define movement turn-around features. and G are start and goal locations for the learner. (middle) Divergence between ((t) and P()), and (right) LBA of the learned policies on Gazebo simulation data of leamer.
but its high variance and inability to improve its performance can be attributed to observation noise. As it is not incremental (it was run with accumulating data in batch mode), runtime comparison is not meaningful and hence not reported.
To evaluate sorting performance, Sawyer uses the learned policies to sort fixed number of onions. For object tracking and onion classification, we use trained YOLO model with simulated camera in Gazebo. For all techmiques, we execute the policy learned in the final session, to sort 6 moving collections of onions each (2 blemished and 2 good). The table in Fig. 3rright) shows the average precision and recall of the expert and the 4 learning methods. As the expert aims to inspect every onion whose blemish not immediately visible, which slow, it exhibits high precision but low recall. RIMEO's 's precision is closest to the expert's due to lower false positives. AIRL performs worse in both precision and recall having higher false positives and true negatives. This is consistent with its low LBA, indicating its reduced ability to successfully sort onions under noise. Among the two robust I2RL methods, as the steps after learning the observation model are the same, we attribute the better precision of RIMEO to more accurate observation model.
To establish the generality of RIMEO, we evaluate it in second domain: simplified version of the perimeter patrol domain [21]. It involves two Turtlebots (in Gazebo), one patroller continuously navigates a hallway in cyclic trajectories: the other is a learner tasked with reaching goal location without being spotted by the patroller (Fig. 4). The learner must solve its own distinct decision-makinn problem (modeled as another MDP), which is dependent on correctly predicting the patrol. The latter can be estimated from inferring the patroller's preferences given that the learner knows dynamics. The state of patroller the MDP has three dimensions (ar, y, 0), which gives the and coordinates of the cell decomposition of the corridors and hallways, and 0 the orientation the patroller. The domain has 124 states. The patroller can exccute one of four possible actions: move-forward, turn left or turn right 90 degrees, and stop. The reward function utilizes 6 binary state-action feature functions, described in Appendix D. The observation feature set I contains 4 binary predicates also described in Appendix D.
Perimeter Patrol Domain:
We introduce noise as ((v ((s, a))) ) + (s,(s, a)²) 0.3 for j = MoveForward, TurnRight, and for other features. The two figures (right) Fig. 4 show the performance of both robust I2RL methods in perimeter patrol. As in onion-sorting, KI. divergence and LBA for RIMEO is better than RIFOM, AIRL and MLIRL. These results two unrelated tasks indicate that the presented method is sufficiently general for broad applicability.
7 Conclusion
We introduced an online IRL method that accommodates noisy observations and infers the unknown observation model by maximum entropy optimization. Not only does this method fill gap in the prior work on online IRL. but also makes online learning more pragmatic, as evaluated in two robotics tasks of hast-sa onion sorting and breaching patrol.
The current set up restricts the observation features to binary functions. The assumption of feature independence and the key approximation of Eq. 6 may not be easily satisfied, and could have contributed to RIMEO's s learned observation model not being accurate (KID==0). To meet the assumption that all ve observed in the first session, it may need to contain more demonstrations.
Limitations:
8

---

Acknowledgments
We thank the anonymous reviewers for helpful comments and suggestions. This research was supported part by grant from NSF #IIS-1830421 and a Phase grant from the GA Research Alliance to PD.
References
[1] S. Arora and P. Doshi. A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress. Artificial Intelligence, 297:103500, 2021. doi.huts://doi.org// 101g:/00.001/ 103500.
[2] A. Ng and S. Russell. Algorithms for inverse reinforcement learning. In Seventeenth Interna- tional Conference on Machine Learning, pages 663-670, 2000.
[3] M. L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., New York, NY, USA, 1st edition, 1994. ISBN 0471619779.
[4] S. Shahryari and P. Doshi. Inverse Reinforcement Learning Under Noisy Observations. In Proceedings of the 16th Conference Autonomous Agents and MultiAgent Systems, AAMAS 117, Richland, SC, 2017. International Foundation for Autonomous Agents and Multiagent Systems.
[5] Arora, P. Doshi, and B. Banerjee. I2RL:
Online inverse reinforcement learning under occlusion. Autonomous Agents and Multi-Agent Systems, 35(1):4, Nov 2020. ISSN 1573-7454.
[6] K. Bogert and P. Doshi. A hicrarchical bayesian process for inverse rl in partially-comtrolled environments, page 145-153, 2022.
[7] P. S. Suresh and P. Doshi. Marginal MAP estimation for inverse RL under occlusion with observer noise. In Proceedings of the Thirty-Eighes Conference on Uncertainty Artificial Intelligence, volume 180, pages 1907-1916, 2022.
[8] A. Boularias, J. Kober, and J. Peters. Relative entropy inverse reinforcement learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS 201 Fort Lauderdale, USA, April 1-13, 201 I, pages 182-189, 2011.
[9] K. Bogert and P. Doshi. Toward Estimating Others' Transition Models Under Occlusion for Multi-robot IRL. In 24th International Joint Conference on Artificial Intelligence (ICCAI), pages 1867-1873, 2015. 2015.
[10] S. Russell. Learning agents for uncertain environments (extended abstract). In Eleventh Annual Conference on Computational Learning Theory, pages 101-103, 1998.
Twenty-first International Conference on Machine Learning (ICML), pages 1-8, 2004. [11] Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In
[12] B. Ziebart, A. Maas, J. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In 23rd National Conference on Artificial Intelligence -Vlluee 3, pages 1433-1438, 2008.
[13] G. Jocher, A. Stoken, Borovec, C. Stan, L. Changyu, A. Hogan, L. Diaconu, Marc, Doug, Ingham, Frederik, Guilhen, Hatovix, J. Poznanski, J. Fang, L. Yu, M. Wang, N. Gupta, O. Akhtar, P. Dvoracek, and Rai. ultralytics/yolos:: v3.1, Oct. 2020.
[14] N. Soans, E. Asali, Y. Hong, and P. Doshi. SA-Net: Robust State-Action Recognition for Learn- ing from Observations. In 2020 IEEE International Conference on Robotics and Automation, ICRA, pages 2153-2159, 2020.
[15] Y. Bard. Estimation state probabilities using the maximum entropy principle. IBM Journal of Research and Development, 24(5):563-569, 1980.
[16] G. R. Walsh. Methods of optimization. Wiley, London, 1975. ISBN 0471919225.

---

[17] D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical programmingg 45(1):503-528 1989.
[18] Nocedal and N. Okazaki. libLBFGS: A library Limited-memory Broyden-Fletcher- Goldfarb-Shanno (L-BFGS). htttp:://github.ccmm http.////ccoooaa//illleggg. 2013.
[19] M. C. Vroman. Maximum Likelihood Inverse Reinforcement Learning. PhD thesis, Rutgers, The State University of New lersey, 2014.
[20] J. Fu, K. Luo, and S. Levine. Learning Robust Rewards with Adverserial Inverse Reinforcement Learning. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018. URL bottps://ppenreee
A
[21] K. Bogert and P. Doshi. Multi-robot Inverse Reinforcement Learning Under Occlusion with Interactions. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems, AAMAS '14, pages 173-180, 2014.
10


Reviews:
[{"review_id": "yU18qnBqRJH", "reviewer": "Reviewer_MqYS", "summary": "This paper introduces an inverse reinforcement learning method, which learns an observation model, to deal with real world robotics applications with noisy observations. The observation model is learned using the maximum entropy principle and is then used to learn a reward function. The experiments are conducted on two tasks: (1) post-harvest onion sorting using human demonstrations; (2) a perimeter patrol task. On the presented tasks, this approach outperforms a state-of-the-art adversarial inverse reinforcement learning method (AIRL) and a custom baseline, which also uses an observation model (yet a less sophisticated one).\n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- the task of inferring a reward under noisy expert demonstration is of general interest\n\n**Weaknesses**:\n- the paper is not very well written. The paper is overly complex with a confusing set of notations.\n- the contribution is minor, only the observation model is introduced to the online inverse reinforcement learning setting. \n- the application scope of the presented algorithms is very limited --> discrete states and actions, with binary observation features.\n- the presented robotic tasks is solved on a very high-level.\n- the comparisons made to state-of-the-art are made on very specific tasks, with very specific features. It is hard to judge on the general performance of this approach.\n- no sufficient implementation details are given (e.g., in the appendix).\n- no video and no code is provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "From my perspective, the general approach of inferring a reward under noisy expert demonstration is interesting and promising. However, the paper introduced a very limited approach and benchmarked on very specific task with a small number of highly engineered features and observations. This makes it very hard to judge on the performance of the presented approach in comparison to the state-of-the-art. \nMoreover, the paper is unnecessarily complex written making it hard to follow and not straightforward to understand. The notation needs to be revised from my perspective.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "**Issues**:\n- the notation is very confusing. The combination of $\\phi$ for the binary features, $\\psi_j$ for the binary predicates and $\\psi_J^{o,g}$ is not very straight forward to understand. In addition to that, Figure 2\nconfuses even more, where $\\psi^{1,g}_j$ (where does the 1 come from?) is introduced, of which I am not sure of what it means. It would help a lot to create a more informative Figure presenting the general approach\na bit more clearly. Figure 2 is not well made in my opinion.\n- Equation (6) uses the binary predicates $\\psi_j (<s, a>)$ to define the indicators. Why is a different notation used for indicators in equation (8)s?\n- Use a consistent math font, even in the Figures.\n- It needs to be explained in greater detail, how the confidence intervals are calculated and the networks are trained.\n- Implementation details, also for AIRL, on the experiments are missing. A judgment on AIRL's computational performance without giving implementation details is not very helpful.\n- I don't think that AIRL is a good baseline for this task, due to the small state and action states. I would like to see a comparison to an approach made for discrete state and action spaces.\n\n\n**Minor Issues**:\n- please blur faces in Figures next time to ensure double blindness (even if the pictured persons are not the authors, but lab members etc.)"}}, {"review_id": "nd00Hxv5Rp6", "reviewer": "Reviewer_1CPf", "summary": "This work deals with inverse reinforcement learning (IRL), specifically in robotic manipulation tasks where the observation model is noisy.  They state this is a more realistic scenario since real-world robotic perception is noisy and models therefore must be updated incrementally.  Previous works in IRL assume that the observation model is known, and they extend this work to additionally learning an unknown observation model incrementally.\n\nThe introduced method, I2RL-MEOM, approximates the unknown observation model using a shared feature set.  They then fit this model to observations with a max-entropy optimization (Equation 8).  The learned feature expectations are then used to incrementally update the IRL objective. The authors additionally demonstrate the convergence properties of the method.\n\nThe method is compared against AIRL on a variety of robotic manipulation tasks in an onion sorting domain and a perimeter breaching task.  ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I2RL-MEOM is compared against AIRL on a variety of Sawyer robotic manipulation tasks in an onion sorting domain. Some onions are blemished and must be manipulated appropriately depending on the task (picked, sorted, etc). There are decent improvements over AIRL, but the results are not overtly impressive. The convergence analysis is also appreciated.\n\nThis method shows some decent improvements in a set of interesting and realistic Saywer sorting tasks. It also deals with a difficult and realistic problem, when the observation model is noisy in the IRL setting. There are decent improvements over AIRL, but the results are not overtly impressive.\n\nOverall, I find this paper incremental, and limited. I believe that learning the observation model incrementally is too restrictive under this task, such as its restriction to binary features. I also believe that the experimental justification is lacking.\n\nThis paper is difficult to read at times, mostly since such a large amount of notation is introduced. For example, Section 5.1 is extremely dense with rapid-fire notation heavy descriptions of each algorithm step. I feel that this entire section could be replaced by a simplified pseudo-code figure. There are many descriptions in this paper where the notation could and should be simplified. Some formula-heavy related work could be summarized better (especially Section 3.1), with the full exposition in the appendix.\n\nThe results are poorly organized, Figure 3 (Right) is missing actual columns for the TP, FP, and result value, where instead, commas are used to separate results. Figure 4 is also too small. The exhaustive list of experiments (Lines 213-224) should also be moved to the appendix and only summarized in this section.\n\nI am confused as to why the only baseline compared is AIRL. I feel that more extensive comparisons are needed against other model-based methods. It would be nice to see a comparison against Robust IRL, even with an incorrect observation model. The empirical justification for this method is, therefore, quite limited. I would also encourage the authors to compare against other SOTA Robust IRL methods such as Viano et al., 2021 (https://openreview.net/pdf?id=t8HduwpoQQv).\n\nThis method is also severely limited by the fact that features are binary functions. The authors rightfully explain this limitation in Section 7.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I still believe this work is quite restrictive, and can only deal with a subset of feature functions.  The experimental results are lacking and suffer from a lack of sufficient baseline comparisons.  The improvements over AIRL are decent, but not overtly impressive. The writing is very hard to work through and is too notation heavy.  The algorithm description therefore suffers.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "The main issue with this work is the extremely notational heavy exposition.  I believe that better summarization of the method is also needed, with only the most relevant equations concisely shown in the main text.  I also encourage the authors to examine more baselines for comparison outside of AIRL as stated in my main review."}}, {"review_id": "hieyR4K9T-kG", "reviewer": "Reviewer_Kim8", "summary": "They propose to extend incremental/online inverse reinforcement learning (I2RL) to the case with noisy observations and an unknown observation model, making it suitable to real-world robotics applications. Their method -- called Robust Incremental IRL with Maximum Entropy Estimation of Observation Model (RI2RL-MEOM) -- is formulated as a sequence of incremental learning sessions where each session is a two-step process: (1) learning a maximum entropy distribution of the observation model, and (2) using that model to learn the reward function of the expert. They present results on 2 robotics tasks: (1) onion sorting with Sawyer arm based on human demonstration and (2) breaching a perimeter patrol with 2 Turtlebots. RI2RL-MEOM learns a more accurate policy compared to (a) an IRL method that doesn't learn an observation model, and (b) a custom baseline that learns a simpler observation model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* Their approach is practical for robotic applications, where the observations are typically noisy and the observation model is unknown.\n* RI2RL-MEOM admits formal guarantees of monotonic convergence and sample complexity bound.\n* Their experiment on onion sorting uses _real_ videos of human sorting onions, which is a realistic setting.\n* Their experiment demonstrates low KL divergence for a simulated dataset of the onion sorting task, thus demonstrating the correctness of the learned model against a ground truth. (The simulated dataset is generated for the purposes of evaluating the learned observation model against a known model.)\n* Their method achieves higher learned behavior accuracy (LBA) and requires less data (i.e. sessions) compared to their baselines -- RI2RL-FQOM and AIRL.\n\nWeaknesses\n\n* They required a manually defined set of features. Although this was appropriate for the tasks considered, this might be a limitation for more complex tasks that might not have obvious features or features that are non-trivial to extract.\n* Although they did use real videos as training data for the onion sorting task, the evaluation of the learned policy was performed in simulation. However, this might be appropriate for their setting since they are testing for high-level decision making.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe this work is a good example of a learned-based method that addresses real-world constraints -- namely noisy observations and a unknown observation model -- and demonstrate results on a task that uses real-world data.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I didn't find any major issues with this paper. However, I'm not familiar with the theoretical component of this work, and I defer the judgement on this to the other reviewers.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "Nt1L9JFNLR-", "reviewer": "Reviewer_F2Vv", "summary": "This paper considers the problem of reward learning when the observation model is unknown. The authors propose a maximum entropy approach to estimate the observation model an incorporate this into an online IRL algorithm. Results on real robots show that the proposed approach outperforms AIRL and a frequentist version. Theoretical analysis shows that the proposed approach has desirable convergence properties.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- online reward learning with an unknown observation model is an important and understudied problem\n- good overview of prior work in section 3\n- nice theoretical analysis of proposed algorithm\n- real robot experiments \n\nWeaknesses\n- Seems to require fully solving the MDP as does classical MaxEnt IRL\n- Requires discrete and binary features and known reward features", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper provides a nice mix of theory, technical rigor, and empirical results on physical robots. The experimental results show the advantage off the proposed approach and the overall problem studied by the authors seems novel and important for real world robot reward learning.\n\n---\n\nThe authors have addresses my main concerns. I still think the paper should be accepted.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The algorithm abbreviation (RI2RL-MEOM) is quite long and awkward. I'd suggest something shorter and catchier :)\n\nFigure 2 has a lot of clutter with all the equations/notation in the upper right. I would recommend using words or pictures to describe as much as possible as done in the left half of the figure.\n\nHow would the authors propose that the reward features and observation features be chosen? The method seems to rely heavily on these being good representations. Could they be learned from data?\n\nDoes AIRL receive the same observation representations as the proposed approach?\n\nHow critical is the SA-network? Would the proposed approach work on raw continuous states and actions?\n\nI would be interested in seeing what is lost due to an unknown observation model. Is it possible to run prior Robust IRL in simulation with a known model and compare to see how much learning the observation model degrades performance?"}}]

Author rebuttal:

Decision: Accept (poster)