Learning Transparent Reward Models via Unsupervised Feature Selection
Authors: Daulet Baimukashev
Gokhan Alcan
Kevin Sebastian Luck
Ville Kyrki
Venue: CoRL 2024 (2024)
Keywords: Inverse reinforcement learning
Reinforcement learning
Imitation learning
Robots
Reward learning
Robot learning
TL;DR: We construct interpretable and compact reward model from observational data. This learned reward model can be directly used using standart reinforcement learning framework for solving complex tasks.

Abstract: In complex real-world tasks such as robotic manipulation and autonomous driving, collecting expert demonstrations is often more straightforward than specifying precise learning objectives and task descriptions. Learning from expert data can be achieved through behavioral cloning or by learning a reward function, i.e., inverse reinforcement learning. The latter allows for training with additional data outside the training distribution, guided by the inferred reward function. We propose a novel approach to construct compact and interpretable reward models from automatically selected state features. These inferred rewards have an explicit form and enable the learning of policies that closely match expert behavior by training standard reinforcement learning algorithms from scratch. We validate our method's performance in various robotic environments with continuous and high-dimensional state spaces.

Learning Transparent Reward Models via Unsupervised Feature Selection
Daulet Baimukasher Gokhan Alcan2, Kevin Sebastian Luck Ville Kyrki! Aalto University Tampere University VVrije Universiteit Amsterdam
Abstract: complex real-world tasks such as robotic manipulation and au- tonomous driving, collecting expert demonstrations is often more straightfor- ward than specifying precise learning objectives and task descriptions. Learn- ing from expert data can be achieved through behavioral cloning or by learn- ing reward function, i.e., inverse reinforcement learning. The latter allows for training with additional data outside the training distribution, guided by the inferred reward functioo. We propose novel approach to construct compact and transparent reward models from automatically selected state features. These inferred rewards have an explicit form and enable the learning of policies that closely match expert behavior by training standard reinforcement learning algo- rithms from scratch. We validate our method's performance in various robotic enviromments with continuous and high nigh-dimmnsional state spaces. Webpage: https://sites: wow.se..ooo.....ooo..o wwwwviww/vvccceec.e.....c....eee wwt.////eeee//raarrreeeeeeeeeeee -eeaaarret-eeedd...
Keywords: Inverse reinforcement learning, Reinforcement learning, Imitation learning, Reward learning, Robot learning
1 Introduction
Imitation learning (II.) involves learning policies from expert demonstrations. It has been success- fully applied in many real-world robotics applications such as household tasks [1, 2]. manipulation
[3, 4, 5], and autonomous driving [6. 7J. It is more straightforward for humans to demonstrate task execution than to precisely specify task descriptions and/or learning objectives [8]. More broadly, IL can be addressed by either behavioral cloning [9] or inverse reinforcement learning (IRL) [10]. The first approach learns the mapping from state to action using supervised learning but suffers from compounding errors outside the training distribution [11]. Reinforcement learning (RI.) [12] ben- efits from exploration of the environment and learns through experience. RL has been applied in robotic applications [13, 14, 15, 16, 17], but it requires well-defined reward function solve the task. Manually specifying reward function poses additional challenges for complex tasks, creat- ing need for the automatic construction of rewards from data. We propose to construct reward by automatically selecting its component features and retrieving reward using inverse reinforcement learning (IRL) setting [18, 10]. IRL. benefits from expert demonstrations while utilizing more ex- ploration.
One of the prominent approaches for learning rewards is maximum entropy inverse reinforcement learning (Max-ent IRL), which addresses the reward ambiguity problem using a probabilistic model of the behavior [19, 20, 21]. Max-ent IRL can learn an explicit reward model, but typically the features that compose the reward are specified beforehand. Both linear [22, 23, 24) and non-linear [25, 26] functions of features have been used to learn the reward. As depicted in Fig. I our method does not rely on hand-picked features as current IRI. methods, but finds relevant set of features from large candidate set. We learn transparent reward model such that the relationship between
"Corresponding Author
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Current IRL
Adversarial
Ours
Candidate
Rank
Non-linear Features
42
42
NN Features
Features
45
4
41
Reward
Reward
Reward
Full RL
1-step RL
Full
Transparent
Policy
Policy
Policy
Amendable
Figure 1: Current IRL. methods require hand-picked features to use as reward components. Adver- sarial methods learn the reward using neural networks which are less transparent and not amendable. But, the proposed method finds automatically relevant set state features and constructs transparent reward models.
state features and the resulting reward is direct, and reward function is compact and amendable if preferred.
In this work, we propose to construct a reward function from state features using unsupervised feature selection without requiring access to ground truth rewards. To achieve this, use the maximum entropy formulation for stochastic policies, in which the probability of trajectory is proportional to its cumulative reward. This approach mitigates the need for an exhaustive search for suitable fcatures and produces an explicitly represented reward model. As only state features are used for reward components, this reward formulation is also referred to as disentangled rewards and robust to changes in the environment dynamies, as presented in [27]. Our contributions are as follows:
1. We develop method for learning a compact and transparent reward function for use in reinforcement learning algorithms.
2. We demonstrate the effectiveness of our proposed method through validation on various continuous control systems with high-dimensional, continuous state spaces.
2 Related Works
2.1 Feature construction for IRL
Inverse reinforcement learning methods can retrieve rewards if their structure is specified accord- ingly. Many works have considered manually designed features [28, 23, 20, 29, 30) for reward. While human-specified rewards are effective in some scenarios, given only demonstrations, it challenging to design reward function that matches the true intent or reward of an expert. The literature on automatically learning compact reward structures is limited. Reward construction from atomic features using regression trees proposed in [31]. This method iteratively constructs the features and corresponding rewards. Another work [26] considered Gaussian processes for mod- cling nonlinear function approximation for the reward. A Bayesian approach using similar atomic features that constructs logical conjunctions features was presented [32]. Authors [33] pre-train feature vector using self-supervised loss, but uses trajectory rankings to learn the reward function. Compared to previous approaches, we not assume to have relevant atomic features or preference labels. Instead, use states only from expert trajectories and consider nonlinear basis functions of states reward components. This results in explicitly represented reward model that consists of small number of features.

---

2.2 Adversarial IRL
An alternative approach for imitation learning, effective in high-dimensional tasks, is enhanced with adversarial methods as presented [27, 34, 36]. These methods adopt the generative adversarial networks (GANs) [37] training scheme to generate trajectories similar to those of experts. The cost a learned cost reward. While adversarial methods achieve state-of-the-ar performance for wide range of applications [38, 39, 40], the retrieved reward function is neural network and not an explicitly represented reward model.
of the discriminator is usually referred
3 Problem Statement
Reinforcement learning (RL) is an approach to solve Markov Decision Processes (MDP) defined as a tuple M = (S, R, where is the set of states, A the set of actions, R the reward function, T represents transition dynamics, and y is discount factor. The solution represented by an optimal policy is defined as the one maximizing the expected discounted cumulative reward of the trajectory:
π arg max Ep(er4s (s..
r R(sr)
(1)
Here, y is a discount factor, transition probability p(s' (s, a) is fixed but unknown, and the policy is learned to optimize this objective from experience.
Now we consider MDP setting, where in addition to transition probabilities, the task reward is un- known. Our objective is to reconstruct this reward function from observational data. As proposed in [10], we represent the reward as linear combination of features which can consists of any nonlinear function of the state.
R(s) = lT (s()
where is the state, o E Rd is the feature function, mapping the state d-dimensional feature vector, and f E d is the weight vector of features. The reward construction problem then translates to finding suitable features and their corresponding weights.
Proposed Method
In this section, we present our method for learning a reward model, which consists of two aspects: finding the feature set and its weight vector. ground truth rewards are not available, we use unsupervised feature selection by generating pseudo labels from expert data. Then, we adopt the max-ent IRL framework for learning feature weights. The overview of the algorithm is provided in Appendix C.
4.1 Trajectory probability under maximum entropy
According to the maximum entropy formulation for stochastic policies [19], the probability trajectory proportional the exponential the reward of that trajectory. That is,
e ((rr))
P(T,(0)=
(3)
Z(0)
where the partition function (0) normalizes the reward function over all possible trajectories. After taking logarithm of both sides of (3), get
log P( = R(T.) log Z(0),
(4)
and noting that the partition term Z is constant:
log P(T(0) R(A)).
(5)
Due to the linearity from Eq. 2. the cumulative reward trajectory can then be defined as
R(T) 0T O(T))
(6)
3

---

where ((T) Eser ((s)). Thus, the log probability of trajectories is proportional to a linear combination of features that compose the reward.
log P(T,0)) x 8" o(T,)
(7)
Based on this proportionality. to determine whether specific feature contributes to the reward, evaluate the correlation between feature's expectation and log probability of trajectories. Since ground truth rewards are not available, use log probabilities of trajectories pseudo-labels.
4.2 Pseudo labels
We generate pseudo labels by computing log probabilitiess of trajectories empirically from training data D = (2,,Te, Tkk The probability of trajectory т {s1 82, ... Spp is given as
P(r) =
P(sm.
We simplify the computation of conditional probabilities by decomposing them into the marginal and joint probabilities of consecutive states. After applying the logarithm both sides and simpli- fying the terms, get
log P(T) log P(s))
Σ (log P(st, log P(s:))
(9)
The derivation of the Eq. 14 is included in the Appendix D.I. Using the entire training dataset, we perform multivariate kernel density estimation (KDE), to obtain marginal P(s) and the joint density of sequential states P(*,, 8141). KDE provides a non-parametric way to estimate the probability density function of the states and requires substantial less training data, unlike neural networks.
4.3 Candidate features
To perform feature selection using Eq.7, finite set of candidate state features is required, as eval- uating all possible features may be computationally infeasible. According to the moment matching method [41], the probability of the state s can be approximated by their higher moments. Using moments up to the third order, we can approximate state probability as
log P(s) Re 4 ( + 22(s) + 43(())
where 1 (s) represents the first-order moments (mean), $ $2((s) represents the second-order moments (covariance), (s) represents the third-order moments (skewness), s state vector. As proposed features evaluated on their predictive power of the log probabilities of trajectories, which in turn consist of state trajectories, we include all covariance terms in the candidate set of features (s) E (s), (s),, (3(s)). However, due to the curse of dimensionality and the finite set of demonstrations, using all features as reward components is impractical. Feature matching with IRL may fail due to noise and spurious correlations, leading to false correlations that do not exist in the original data distribution. Therefore, we present an efficient method to select only the relevant features.
4.4 Feature selection
After computing the probability and feature expectations of trajectories, we obtain the following data.
(TO)
(2(TT)
ox dx(n)
log P(T)) log P(n)
Φn(1))
(2(11)
X
y
(IN)
@2(TN)
ox
(TN)
log P(Tx)]
One way to perform feature selection is to utilize univariate feature selection method based on statistical tests. First, for each feature separately, apply F-statistics between trajectory

---

feature expectations and trajectory probabilities, i.e, between each column of X and the vec- tor Y. This gives the feature importance metric, which we can use for ranking the features. Then, the features with higher F-statistics are selected and included in the feature extractor (()). Univariate feature selection is more suitable when the number samples much smaller (K > N) than the size of the candidate set, which is often the case for high-dimensional states. Therefore, directly using linear regression for X and Y results in overfitting. Recur- sive feature elimination methods may be used here, but they are computationally demanding.
Although the univariate feature selection method does account for interactions between features, these interactions will be considered while learning the ward function through inverse reinforcement learning. Additionally, to reduce overfitting, we employ cross- validation and noise addition methods. Specifically, we add multiple non-expert trajectories by labeling them with uniformly sampled log probability values from the bottom 10th percentile of expert log probabilities. The
(a) Hopper-v4
(b) Ant-v4
time complexity of the feature selection algorithm is ((N) with respect to the size of the candidate o. This approach is efficient, as it does not involve itera- tive search of features and policy learning, which would be the case with manual feature construction. The next step to fully recover the reward function is to find the weights of each of the features, which covered the next part.
(c) Walker2d-v1
(d) HallCheetah-v4
Figure 2: Benchmark tasks used in this paper.
4.5 Reward and policy retrieval
We employ the maximum entropy IRL. [19] to learn the weights of the selected features. To maxi- mize the log probability the observed data, we formulate the optimization problem as follows:
0" = arg max
Σ
log P(r(0)
(10)
TED
To find the optimal weights 6, we take the derivative of log likelihood that is given by:
((T)
V V
Σ log TED
= Σ ((T))
E E Ro TET
ΣΣ
((s))
(11)
2(0)
TED
1=1 AE
Here, Me represents the feature expectation of the training data, is the number of trajectories from current policy. The weights of the reward can then be updated by using the gradient descent as follows:
レ 0 + V(())
(12)
where a is the learning rate. The full derivations are included in Appendix D.2. The learned reward function has an explicit form allowing us to apply any value iteration algorithm to obtain a policy. We utilize the Soft Actor-Critic (SAC)[42) to learn the expert policy by optimizing Eq.
Experiments
In this section, we consider different continuous control tasks and learn their reward functions from expert data. Then, using the learned rewards, we learn the policy and compare it against the expert performance, We will investigate the following research questions:
1. Is the proposed methodology effective in reconstructing ground truth rewards solving vari- robotic tasks?
2. Are the learned rewards transparent and amendable?

---

Linear
Kandom
Manual
Proposed
Expert
Ani
1681,871 TO00U
372.42
10M 3213523
22.00
967.46
1970,048 I 198
Hopper
#21.0-331..02
914,294 80207
2901092-16.00
Walker2d
1157,47 017,4713822 3.38./9
45.501 150.9
123. 16 1208. 76
3108 296.14
82
HallCheetthh
944 90
149,02
5514
304.02
68066.2 113.35
Table Mean cumulative rewards for policies trained using various feature sets, calculated across 10 test simulations under varying initial conditions. The last column shows the expert RI. policy with ground truth reward of the environment. The cells with bold values indicate highest scores.
5.1
Sctup
We conduct experiments on different environments with continuous state space, particularly Mu- JoCo environments such as Ant, HalfCheetah, Walker2d, and Hopper from the Gymnasium [43] benchmark (see Fig.2). The descriptions of the tasks and ground truth reward formulations provided in Appendix A.
5.2
Data collection
To collect the expert training data, train RL algorithms for the above environments, Specifi- cally, we used SAC for all the environments with continuous action space. We train RL algorithms to maximize the cumulative ground truth reward function of the environment. After reaching estab- lished benchmark results as presented in Stable Baselines3 [44] for these tasks, we refer to the RL policy as an expert policy and deploy it for data collection. For each the environments, we col- lected trajectories with random starting states, and the simulated trajectories were saved dataset D. Thus, in our IRL method, as shown in Algorithm only dataset used, and we assume that the expert policy ground truth reward function is unknown. We trained the SAC algorithm for trajec-
very limited time-40,000 simulation steps-to generate sensible but suboptimal(oon-expe tories for feature selection phase. More implementation details are provided in Appendix B.
5.3 Baselines
In this work, propose feature selection mechanism aimed at recovering the reward function that corresponds to the expert's behavior, given the dataset. Therefore, we compare our method against various baseline feature selection strategies: manual features, randomly selected features, direct use of all states as features (referred to as linear features). For manual feature construction, we evaluated several hand-craften features and selected the best ones for final experiment. Similarly, the random projection method evaluates a random subset of features from the candidate set. Given that com- paring reward functions directly across baselines does not yield a meaningful performance metric in terms of completing task, focus comparing the policies derived from these reward functions. To this end, we conduct two comparative analyses. First, execute the derived policies in multiple testing environments configured by ground truth reward functions and observe cumulative rewards. This evaluation valid because our training data comes from an expert trained with ground truth reward function as well. Second, we compare the state distributions the training data from expert and testing data from the extracted policies. For the divergence measure of multivariate distribu- tions, we exploited the 2D Wasserstein distance metric [45]. Also, we compared the performance of the method against SOTA adversarial inverse reinforcement learning methods such AIRI[[7]] and GAIL [35].
Results
The cumulative episodic rewards of poli- cies trained using different rewards using baseline methods are shown in Table The results show the average for 10 trajec- tories with three random seeds. The per-
Task
HalfChectah
Walker2d
Hopper
Ant
Advers.
609
609
609
609
Ours
12
20
Table 1: Number of reward network parameters.

---

300
Curnlaative Reward
3000
linear
manual
Wasserstein Distance
250
linear
manual
random
random
proposed
200
proposed
1000
100
50
-1000
20
Iteration Number
Iteration Number
(a) Mean cumulative rewards of learner using dif- ferent baseline methods.
(b) 2D Wasserstein distance between state distribu-
tions of expert and learner.
Figure 3: IRL training curves for the Walker2d task comparing different feature selection baselines.
formance of the proposed reward model outperforms the baselines for all of the tasks.
Next, we compare the performance of our method AIRL and GAIL algorithms from the imitation learning library [46]. Table presents the mean cumulative rewards for the tasks studied. We can that method achieves higher cumulative rewards in all domains, with significant improvement on the Walker2d, Hopper, and Ant tasks. While adversarial IRL methods achieve state-of-ther th---rrrtt (SOTA) performance, they require a large amount of training data to perform well. Table shows the number of parameters in the reward model. In adversarial IRL methods, a two-layer MLP with 32 nodes is used, while proposed method uses linear reward model with significantly less parameters. This enables to simplify learning process of the reward and therefore, proposed method performs better than adversarial methods with less data.
Task
HalfCheetah
Walker2d
Hopper
Ant
GAIL
2186 + 287
872 + 336
628 + 59
-389 + 229
AIRI.
5264 + 1856 89
1289 165
-1189 + 570
Ours
5514:204.02
3108--296.4 14
2110=197.57
2236-1987.46
Table 3: Mean cumulative rewards for different IRL algorithms
The learning curves of the IRL for Walker2d are shown for different reward models in Fig. 3a. Similarly, the Wasserstein distance metric [45] between expert and learner state distributions during training is shown in Fig. 3b. To compute the divergence metric, we use 40 trajectories of expert and learner. The divergence met- ric between expert and learner policies clearly demon- strates that for the proposed reward, not only does the policy achieve high cumulative rewards, but also bet- ter matches the expert state distribution compared other baseline methods. Next, Fig. 4a shows the correlation plot between ground truth rewards and recovered rewards for 150 trajectories with random initial conditions. The Pearson correlation score is reported in the figure for the HalfCheetah environment. We observe strong correlation between recovered and ground truth rewards, with Pearson correlation coefficient of 0.78. This finding is noteworthy, as no ground truth reward sig- nal was provided during reward learning; only expert demonstrations, specifically state trajectories, were used to infer rewards. This suggests that the obtained rewards not only result behaviors similar those of the expert but also achieve this by constructing reward signal that closely re- sembles the ground truth reward signal. In addition, Fig. 4b shows the full RL training loop with the recovered reward, suggesting the compatibility of the recovered rewards with the RL framework, meaning that RL can be trained from scratch using this reward function.
Expert
Fast
Slow
U2 0.12 0.1 0.17 0.18 0.09 0.08
Table 4: Velocity of the HalfCheetah along the x-axis changes accordingly af- ter modifying the weights of the reward component

---

6000
leamer
Recovered Rewards
Cumulative Reward
5000
expert
96
4000
3000
2000
92
1000
Correlation Line (r=0.78g
300
310
330
340
350
0.0
0.8
Ground Truth Rewards
Timestep
1e6
(a) Ground truth and recovered cumulative episodic rewards for trajectories in the expert data
(b) RL learning curve with ground truth reward (ex- pert) and recovered reward functions (Ieamer).
Figure 4: Comparison of training with ground truth and recovered reward functions for HalfCheetah.
Next, examine the characteristics of the rewards obtained. Let's consider the inferred reward model for HalfCheetah:
(1.59 0.03sp
0.57s11 0.06$16 + 0.21s 0.21s0$11)ss
(0.3181 0.3882 0.838982)88
1.24 1.03s 0.57818g
The transparency of the reward can be seen from the reward representation which consists of mono- mials. This enables the analysis of which features of the expert contributes to reward and interaction between state features. We observe that the reward terms focus on state 88, which represents the ve- locity along the æ-axis, directly relating to the agent's ' original reward. Additionally, there a high negative gain for 8g, angular velocity along the y-axis, which is sensible since the HalfChee- tah should primarily move along the z-axis. Negative gains for the angular velocity of the rotors, specifically 82 and $16, also evident, aligning with the control penalty in the ground truth reward.
One important benefit of this compact and explicit representation of the reward is the possibility of fine-tuning the reward function to generate different behaviors. The Table below shows the mean and standard deviation the HalfCheetah velocity along the x-axis after multiplying coefficient of one of the terms, $s, in the reward function by and 0.5, respectively. As result of amending reward components, faster and slower motion was achieved, which be expected as state s8 accounts for the velocity along the x-coordinate of the front tip.
One limitation of this paper is its applicability to other high-dimensional inputs, such as images. We anticipate that pre-learning features, possibly through the use of autoencoders, and combining with proposed method could be viable approach to overcoming this limitation, and we plan to explore this direction in future work. Another limitation is our experimental design. We have conducted experiments with real robots and human demonstrations. However, validated method using complex robotic environments. In future work, plan to apply the method to real-world tasks and leamn rewards from demonstrations.
7 Conclusion
This paper presented method for constructing a transparent reward model with an explicitly rep- resented form using non-linear state features. Reward components were selected using an unsuper- vised learning method from observational data by incorporating the maximum entropy formulation for the behavior. As shown above, selecting the features that allow predicting trajectory probabilities results in higher match with ground truth rewards and enables the retrieval of policies more similar to those of expert. Learning rewards from data alleviates the challenge of manual reward specifi- cation for complex tasks and allows for training with exploration outside of the training data using the reinforcement learning framework. We have demonstrated that across different environments. recovered rewards outperform other baseline methods.

---

Acknowledgments
This work was supported by the Academy of Finland under grant 347199, The authors would like acknowledge the computational resources provided by CSC IT Center for Science and Aalto Science-IT project.
References
[1] Kim, N. Cauli, P. Vicente, B. Damas, A. Bernardino, J. Santos-Victor, and F. Cavallo. Clean- ing tasks knowledge transfer between heterogeneous robots: deep learning approach. Journal of Intelligent & Robotic Systems, 98: 191-205, 2020.
[2] D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali, et al. Deep imitation learning of sequential fab- ric smoothing from an algorithmic supervisor. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9651-9658. IEEE, 2020.
[3] Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, Goldberg, and P. Abbeel. Deep imita- tion learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE international conference robotics and automation (ICRA), pages 5628-5635. IEEE, 2018.
[4] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint Mixx:v::700 10087, 2017.
[5] D. Jain, A. Li, Singhal, A. Rajeswaran, V. Kumar, and E. Todorov. Learning deep visuo- motor policies for dexterous hand manipulation. In 2019 international conference roboties and automation (ICRA), pages 1636-6643. IEEE, 2019.
[6] Y. Pan, C.A.. Cheng, K. Saigol, K. Lee, X. Yan, E. Theodorou, and B. Boots. Agile au- tonomous driving using end-to-end deep imitation learning. arXiv preprint 17XX:1709.001744. 2017.
[7] Hawke, R. Shen, C. Gurau, S. Sharma, D. Reda, N. Nikolov, P. Mazur, S. Micklethwaite, N. Griffiths, A. Shah, et al. Urban driving with conditional imitation learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 251-257. IEEE, 2020.
[8] T. Ni, H. Sikchi, Y. Wang, T. Gupta, L. Lee, and B. Eysenbach. f-irl: Inverse reinforce- ment learning via state marginal matching. Conference on Robot Learning, pages 529-551. PMLR, 2021.
[9] M. Bain and C. Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pages 103-129, 1995.
[10] A. Y. Ng. S. Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, page 2, 2000.
[11] Xu, Z. Li. and Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems, 33:15737-15749 2020.
[12] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[13] F. Stulp, J. Buchli, E. Theodorou, and S. Schaal. Reinforcement learning of full-body hu- manoid motor skills. 2010 10th IEEE-RAS International Conference on Humanoid Robots, pages 405-410. IEEE, 2010.
[14] P. Kormushev, S. Calinon, and D. G. Caldwell. Reinforcement learning in robotics: Applica- tions and real-world challenges. Robotics, 12)):1221443 2013.

---

[15] A. E. Sallab, M. Abdou, E. Perot, and S. Yogamani. Deep reinforcement learning framework for autonomous driving. Electronic Imaging, 2017(19)70-77, 2017.
[16] S. Gu, E. Holly, I. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipu- lation with asynchronous policy updates. In 20/7 IEEE International Conference on Robotics and Automation (ICRA), pages 2389-3396 IEEE, 2017.
[17] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning, In arXiv preprint 2015.
[18] Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Twenty- First International Conference on Machine Learning ICMI. '04, page 1, Banff, Alberta, Canada, 2004. ACM Press. 11111001144/001530.00111.53
[19] B. Ziebart, A. L. Maas, A. Bagnell, A. K. Dey, et Maximum entropy inverse reinforce- ment learning, In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
[20] Z. Wu, L. Sun, W. Zhan, C. Yang, and M. Tomizuka. Efficient sampling-based maximum entropy inverse reinforcement learning with application to autonomous driving. IEEE Robotics and Automation Letters, 5(4):5355-5362 2020.
[21] Y. Zhou, R. Fu, and C. Wang. Learning the car-following behavior of drivers using maximum entropy deep inverse reinforcement Iearning. Journal of advanced transportation, 2020: 1-13, 2020.
[22] N. Aghasadeghi and T. Bretl. Maximum entropy inverse reinforcement learning in continuous state spaces with path integrals. In 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1561-1566, 2011. doi: 1009/ROS.
2200000S221111 2012200019990 1.609469.
[23] M. Kuderer, S. Gulati, and W. Burgard. Learning driving styles for autonomous vehicles from demonstration. In 2015 IEEE international conference on robotics and automation (ICRA), pages 2641-2646. IEEE, 2015.
[24] G. Dexter, K. Bello, and J. Honorio. Inverse reinforcement learning in continuous state space with formal guarantees. Advances in Neural Information Processing Systems, 34:6972-6982, 2021.
[25] M. Wulfmeier, P. Ondruska, and I. Posner. Deep inverse reinforcement learning. CoRR, abs/1507.04888, 2015.
[26] S. Levine, Z. Popovic, and V. Koltun. Nonlinear inverse reinforcement learning with gaussian processes. Advances in neural information processing systems, 24, 2011.
[27] C. Finn, Christiano, Abbeel, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. w:Xiv:1710.11260 2016. URL https://arxii: org/ abs/ 1710.11248.
[28] C. Xia and A. El Kamel. Neural inverse reinforcement learning in autonomous navigation. Roboties and Autonomous Systems, 84:1-14, 2016.
[29] S. J. Lee and Z. Popović. Learning behavior styles with inverse reinforcement learning. ACM transactions on graphics (TOG), 29, 2010.
[30] H. Kretzschmar, M. Spies, C. Sprunk, and W. Burgard. Socially compliant mobile robot navi- gation via inverse reinforcement learning. The International Journal of Robotics Research, 35 (11):1289-130) 2016.
[31] S. Levine, Z. Popovic, and V. Koltun. Feature construction for inverse reinforcement learning. Advances in neural information processing systems, 23, 2010.
10

---

[32] J. Choi and K.-E. Kim. Bayesian nonparametrie feature construction for inverse reinforcement learning. In Tienty-Third International Joint Conference on Artificial Intelligence. Citeseer, 2013.
[33] D. Brown, R. Coleman, R. Srinivasan, and S. Niekum. Safe imitation learning via fast bayesian reward inference from preferences. International Conference on Machine Learning, pages 1165-1177. PMLR, 2020.
[34] C. Finn, Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. International conference on machine learning, pages 49-58. PMLR, 2016.
[35] Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[36] Fu, K. Luo, and S. Levine. Learning robust rewards with adversarial inverse reinforcement learning. arXiv preprini arXiv:/710.1 1248, 2017.
[37] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep con-
volutional generative adversarial networks. arXiv preprini 151 2015. URL https://arste hwwp:/..rr/aass/ab/eee http//b///550 htt.b///511.0000 06434.
International Conference Machine Learning, pages 7194-7201. PMLR, 2019. [38] L. Yu, J. Song, and S. Ermon. Multi-agent adversarial inverse reinforcement learning. In
[39] A. H. Qureshi, B. Boots, and M. C. Yip. Adversarial imitation via variational inverse rein- forcement learning, arXiv preprint (xXiv:88090064000 2018.
[40] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta. Robust adversarial reinforcement learning.
In International Conference Machine Learning, pages 2817-2826. PMLR, 2017.
[41] K. Pearson. Contributions the mathematical theory of evolution. Philosophical Transactions of the Royal Society of London A, 185:71-110, 1894. 101...00000//tt.189....00
[42] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint
2018.
[43] M. Towers, J. K. Terry, A. Kwiatkowski, J. U. Balis, G. d. Cola, T. Deleu, M. GoulÂfo, A. Kallinteris, A. KG, M. Krimmel, R. Perez-Vicente A. PierrÃO, S. Schulhoff, J. J. Tai, A. T. J. Shen, and O. G. Younis. Gymnasium, Mar. 2023. URL https://zenod:: //zg///./oooooo/////// record/812707 8127025.
[44] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 (268):1-8, 2021. URL http://jmlr. wwtg//ppers///222201120 html.
[45] C. Villani. Optimal Transport: Old New, volume 338 of Grundlehren der mathematischen Wissenschaften. Springer, Berlin, Heidelberg, 2008. ISBN 978-3-540--1150-9, doi:10.1007/ 978-3-540-710500
[46] A. Gleave, M. Taufeeque, J. Rocamonde, Jenner, S. H. Wang, S. Toyer, M. Ernestus, N. Belrose, S. Emmons, and S. Russell. imitation: Clean imitation learning implementations. wtwp///1.........
arXiv:2211.1199 199722 [cs.LG], 2022. URL http::t////// //axivv.
[47] T. Akiba, S. Sano, T. Yanase, Т. Ohta, and M. Koyama. Optuna: A next-generation hyper- parameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, 2019.
[48] D. Kingma. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980
2014.

---

Appendix
A Details of Experiments
Below are the short descriptions of the benchmark tasks and the corresponding ground truth reward functions:
1. Hopper-v4 The task of this one-legged robot is to move forward by applying torques to three hinges. The true reward is calculated using distance moved forward and high torque values are penalised.
2. Ant-v4 The task of this robot consisting of three links is to move forward by applying torques to rotors. The true reward is calculated using distance moved forward and high torque values are penalised.
3. Walker2d-v4 The task of this robot is move forward by applying torques to its six hinges. The true reward is calculated using distance moved forward and high torque values are penalised.
4. Half-Cheetahh-4 The task of this two-dimensional robot consisting of body parts is to move forward by applying torques to joints, hinges, and feet. The true reward calculated using distance moved forward and high torque values penalised.
Tasks
dim(S)
dim (A)
Hopper
Walker
17
HalfCheetah
17
Ant
27
Table 5: State-action dimensionality of the tasks
An additional challenge in reward learning is that the features composing the ground-truth rewards are not directly available. For example, the direct components of the ground-truth reward, partic- ularly the x-coordinate and torque values, are hidden from the states and, consequently, from the reward model. Instead, we infer the reward from the available indirect features like velocity or joint angles.
B Implementation details
The source code was implemented Python 3.8, and the source code will publicly available upon acceptance. We utilized the Stable-baselines3 library [44] for the training RL algorithms. As policy network, we used multi-layer perceptron (MLP) with two hidden layers. Furthermore, conducted thorough hyperparameter optimization for both the RL and inverse reinforcement learning (IRI.) parameters. A table with the hyperparameters used during the training is included in the supplementary materials. The total number of training iterations for IRL is 50, and the training dataset consists of 150 trajectories. To facilitate the training, we parallelized data collection and environment simulation using multi processpocessing techniques. During the feature selection process, we normalize each state feature have mean of zero and standard deviation of one. We apply the same normalization for reward computation. However, for policy learning, use the raw state inputs. We employed the Optuna[47] [47] library with grid search to tune hyperparameters, and Adam optimizer[44 [48] was used to learn reward model.
12

---

C
Algorithm
Algorithm 1: Inverse Reinforcement Learning with Unsupervised Feature Selection
input :
.M'r, expert data D, simulator E, iterations M, learning rate a, number of trajs N output: Reward r, policy æ
< -eneraae candidate feature set
Generate pseudo labels Y
3 Compute feature expectations X
H < rank_ featureea X, ,, Y
8 <(-) < select_topk H, k)
compute-features He < compute features (D, ()
7 Initialize 0 ~ -
for
M do
r construct reward ())
10
Configure simulator with new
11
- learn learn_policy( E,r)
12
G f collectt.ollouts(r. E, N)
13
No compute features G)
14
(()) :
loss gradient
15
0 - 0 +VV((0);
// update A
16
end
D Proofs
D.1
Computation of trajectory probability
The probability of trajectory T = {$1: $2 ... sn is given as
P(r) = P(s))))(sss)))(sssssssssp ... ((smlsn-1).
(13)
After applying the logarithm to both sides and simplifying the terms, we get
log P(r) = log P(si)
(log P(st, 8141) - log P(s:))
(14)
Here, conditional state probability P(se+1|se) can be written as P(s = P(s),
Then:
P(T) P(s1) 82) P(s2, s3). P(si) P(s2) P(sn-1)
P(sn-1,88)
(15)
After applying the logarithm both sides, we get:
P()1,82)
P(82,83)
P(8n-1,8n)
log P(r) log P(s1) + log
+ log
+ log
(16)
P(si)
P(s2)
P(8n-1)
8(81,
log P(si) +
log
(17)
P(s:)
log P(s1) +
(log P(s 8141)
log P(s:))
(18)
(19)
D.2 Learning of reward weights
0" arg max
log P(T
(20)
9
13

---

To find the optimal weights 8, we take the derivative of log- ligeliieldood that is given by:
e0 ((T))
VL(0) = V Σ
log
(21)
Z(0)
TED
V
Σ (0T ((T) log 2(0))
(22)
TED
Σ ((T) - V log Z(0)
(23)
TED
Σ ((T)
Σ p( ((T')e 0'('))
(24)
7(0)
TED
N'ET
Σ ((T) Σ p(T(0)((T)
(25)
TED
TET
= He
((s))
(26)
BIET
E Hyperparameters
HalfCheetah
Walker
Hopper
Ant
Number of expert trajs
150
150
100
150
Number iterations
50
50
Learning rate
0.05
0.03
0.05
0.03
Learning decay
0.985
0.99
0.99
0.985
RL timestep
le6
le6
le6
le6
Discount factor
0.99
0.99
0.99
0.99
Number of parallel envs
4
4
4
Batch size
1024
256
512
256
Number of nodes (MLP)
256x2
256x2
256x2
256x2
Table 6: Training hyperparameters.
14


Reviews:
[{"review_id": "WWWVcn4uBC", "reviewer": "Reviewer_psPt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "For learning a reward from human demonstrations, the authors propose a new method to select important state features. Based on max entropy formulation, the authors use trajectory density as labels and find best features to model the trajectory density.", "summary_of_recommendation": "The paper explores an interesting direction, but some details are not clear. Additionally, the experimental results are somewhat limited. Therefore, I recommend a 'weak reject.'", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "**Random seeds**\n\n- The authors mentioned, \"The results show the average for 10 trajectories with random seeds.\" How many random seeds were used?\n\n**Writing**\n\n- line 112: change \"states ss\" to \"state $s$\"\n\n- Eq3,4,5,7: change $P(\\tau_i)|\\theta)$ to $P(\\tau_i|\\theta)$\n\n- Algorithm 1, line6: change $(D,\\phi$ to $(D,\\phi)$\n\n- Eq 1: $t$ notation is missed inside of expectation.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hkjI5KZcgb", "reviewer": "Reviewer_rzLp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper seeks to learn interpretable reward functions from demonstrations", "summary_of_recommendation": "The paper studies an important area and has some promising results but there are insufficient details to fully evaluate and the claim of interpretability requires more validation.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "See questions above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XQlW4NFXJN", "reviewer": "Reviewer_RAMo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a feature selection method for IRL which extracts state features and values highly correlated with the expert behavior through an explicit feature extraction and density estimation step.", "summary_of_recommendation": "Not ready due to some substantial writing issues, a lack of new baselines, and insufficient experiments.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "See above questions in weaknesses.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Summary**

The paper introduces a method for learning transparent reward functions from human demonstrations by focusing on selecting important state features. Using a maximum entropy framework, the approach involves estimating trajectory density and identifying the features that best model this density. This feature selection method for IRL aims to extract state features that are highly correlated with expert behavior, enhancing the transparency of the learned reward functions.

**Summary of Reviews**

*Strengths*:
- Reviewers found the idea of automatically selecting state features for reward models compelling, novel, and principled.
- The experiments were well-designed to demonstrate the benefits of the proposed approach.

*Weaknesses*:
- Lack of clarity. All reviewers criticized the paper for its lack of clarity, missing crucial details necessary for understanding the method, the baselines, and experimental details.
- Missing technical and experimental details. All reviewers had lingering questions about the method and experiments. The authors should resolve any remaining confusions in the rebuttal.
- Insufficient experiments. Some reviewers complained about the narrow evaluation on just the Hopper, Ant, Walker and HalfCheetah environments. Others asked for more baseline comparisons to AIRL and GCL. The lack of real-world robotics experiments is concerning.
- Missing discussion of limitations.

**Outcome and Post-rebuttal**

Two reviewers substantially improved their scores and indicated their concerns being mostly if not fully addressed. I thank the authors for such a substantial and compelling rebuttal effort! Reviewers found the manuscript significantly improved, and appreciated the reframing as transparency and not interpretability. I recommend Accept for poster and I suggest the authors follow the remaining reviewer suggestions to improve their manuscript for the camera ready. In particular: more details for reproducibility, more details about the handcrafted features, a comparison with FIRL, and a code release.
Decision: Accept