Stylized Offline Reinforcement Learning: Extracting Diverse High-Quality Behaviors from Heterogeneous Datasets
Authors: Yihuan Mao
Chengjie Wu
Xi Chen
Hao Hu
Ji Jiang
Tianze Zhou
Tangjie Lv
Changjie Fan
Zhipeng Hu
Yi Wu
Yujing Hu
Chongjie Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement Learning
Diversity in RL
Offline RL
Primary area: reinforcement learning

Abstract: Previous literature on policy diversity in reinforcement learning (RL) either focuses on the online setting or ignores the policy performance. In contrast, offline RL, which aims to learn high-quality policies from batched data, has yet to fully leverage the intrinsic diversity of the offline dataset. Addressing this dichotomy and aiming to balance quality and diversity poses a significant challenge to extant methodologies. This paper introduces a novel approach, termed Stylized Offline RL (SORL), which is designed to extract high-performing, stylistically diverse policies from a dataset characterized by distinct behavioral patterns. Drawing inspiration from the venerable Expectation-Maximization (EM) algorithm, SORL innovatively alternates between policy learning and trajectory clustering, a mechanism that promotes policy diversification. To further augment policy performance, we introduce advantage-weighted style learning into the SORL framework. Experimental evaluations across multiple environments demonstrate the significant superiority of SORL over previous methods in extracting high-quality policies with diverse behaviors. A case in point is that SORL successfully learns strong policies with markedly distinct playing patterns from a real-world human dataset of a popular basketball video game "Dunk City Dynasty."

Published as a conference paper at ICLR 2024
STYLIZED OFFLINE REINFORCEMENT LEARNING: EXTRACTING DIVERSE HIGH-QUALITY BEHAVIORS FROM HETEROGENEOUS DATASETS
Yihuan Mao', Chengjie Wu', Xi Chen', Hao Hu², Jiang", Tianze Zhou', Tangjie Lv², Changjie Fan², Zhipeng Hu², Yi Wu', Yujing Hu²*, Chongjie Zhang Institute for Interdisciplinary Information Sciences, Tsinghua University, China
Fuux AI Lab, NetEase, China, 3 "Washintoon University in St. Louis (maoyh20, wucj19 hwwh2) www.ss....sss...eeeooeee.. edd. (pechenxi, jxwuyi t sitiigguaa edu. cn, chongjie@wust (jiangji, wwwotiiaaze hzl hzlvtangjie, fanchangjie zphu, huyujingJ@corp. netease. com
ABSTRACT
Previous literature on policy diversity in reinforcement learning (RL) either fo- the online setting ignores the policy performance. In contrast, offline RL, which aims to learn high-quality policies from batched data, has yet to fully leverage the intrinsic diversity of the offline dataset. Addressing this dichotomy and aiming to balance quality and diversity poses a significant challenge to extant methodologies. This paper introduces a novel approach, termed Stylized Offline RL (SORL), which is designed to extract high- nigh-performing, stylistically diverse policies from dataset characterized by distinct behavioral patterns. Drawing inspiration from the venerable Expectation-Maxiiizaaie Maximization (EM) algorithm, SORL innovatively alternates between policy learning and trajectory clustering, a mecha- nism that promotes policy diversification. To further augment policy performance, we introduce advantage- advantage-weighter weighted style learning into the SORL framework. Ex- perimental evaluations across multiple environments demonstrate the significant superiority of SORL over previous methods in extracting high-quality policies with diverse behaviors. A case in point is that SORL successfully learns strong policies with markedly distinet playing patterns from a real-world human dataset of popular basketball video game "Dunk City Dynasty."
INTRODUCTION
Learning to accomplish task with diverse behaviors, also known as quality-diversity optimization, is an emerging area within stochastic optimization research (Pugh et al.) 2016) Cully & Demiris 2018 Mouret & Clune| 2015). aims to generate a diverse set of solutions that maximize a given objective function and especially valuable in applications involving human interactions, such as games (Shen et al., 202T) and autonomous driving (Araujo et al. 2023). For instance, in online games, deploying Al bots with varied motion styles enrich the gaming environment and en- hance player engagement. Similarly, in autonomous driving, offering multiple solutions can cater to with different preferences. Additionally, in opponent modeling (Yu al., (022), high-quality diverse opponents that resemble real opponents can significantly improve the performance of the learned policy. However, a significant limitation is that most studies in this domain heavily rely on extensive online interactions (Nilsson & Cully 2021 Pierrot et al., 2022), leading to high costs and imposing constraints on systems with restricted online access. Recent advances in offline RL present promising direction, allowing for policy learning from pre-collected datasets without fur- ther interactions (Fujimoto et al. (2019) Kumar et al., 2020: Kostrikov et al.) 202I). However, they often prioritize policy quality, sidelining the inherent diversity within the dataset. In this paper, we bridge this gap by targeting both diversity and high-quality policy learning from offline datasets.
"Equal advising.
Code at https::///// / www..tt..c....e..e..c...o. wtt.c///w.weee.....ooom

---

Published as a conference paper at ICLR 2024
The central challenge lies in how optimize the performance of the policies while ensuring that their behaviors are as distinguishable as possible. Balancing this in an offline setting becomes prob- lematic due to the inherent characteristics of offline datasets. Typically, datasets capturing diverse behaviors are heterogeneous, gathered from multiple sources, leading to an action distribution with multiple modes and inconsistent data quality (Chen al.) [2022] Li et al. 2017). Studies focus on learning diverse behaviors that can capture the multi-modality of the action distribution. However, they often employ diversity objective that is task-agnostic (Eysenbach et al., 2019 Masood & Doshi- Velez, 2019). In offline settings, this task-agnostic objective can lead to policies that perform exceedingly poorly due to inconsistent data quality. Conversely, current offline RL methods priori- tize task performance maximization. To mitigate overestimation issues, conservative constraint is often imposed on the policy, ensuring that distribution aligns closely with the dataset (Fujimoto) et al. 2019, Kumar et al.] [020]] Kostrikov et al. (02T). Amplifying this conservativeness might re- tain some multi modalit of the dataset. However, it does not offer control over how distinguishable the policy should be during training.
To address this challenge, we introduce Stylized Offline Reinforcement Learning (SORL), a two- step framework designed to derive diverse and high-quality policies from heterogeneous offline dataset. In the first step, we perform a style clustering. Drawing inspiration from the venera- ble pnpetttioon-Maxiizzziioo (EM) algorithm, dataset into clusters where each represents a distinct and dominant motion style. In the second step, we employ Eivantage-weighiee style learning, an offline RL method with a novel objective considering both performance and diversity. Here we train a set of policies to maximize the task performance, with each policy specifically constrained the action distribution of a particular style identified in the first step. Unlike other offiin RL methods that constrain the policy to the entire dataset without differentiating between types of motions, our approach effectively extracts stylis- tically diverse behavior in the dataset that can be characterized by distinct behavioral patterns. In contrast to diverse RL methods, we achieve high-performing policies that are in-distribution with respect to the dataset, yet less influenced by the low-quality samples.
classify the trajectories from the heterogeneous
We evaluate SORL across various environments and offline datasets. These include a didactic game with hand-crafted dataset, set of Atari games using open-sourced human data, and the popular basketball video game "Dunk City Dynasty" with data recorded from online players. We compare SORL against the offline versions of two baseline methods that focus on learning diverse behav- ior from multi-modal datasets. The experimental results demonstrate the significant superiority of SORL over the baseline methods in achieving policies with higher performance while maintaining distinguishable behavior patterns. In summary, the contributions of this paper are as follows:
1. We introduce SORL., a novel framework that addresses the limitations of both diverse RI. and offline RL methods by incorporating both quality and diversity into the optimization objective.
2. We provide comprehensive evaluations of SORL across diverse environments using datasets recorded from humans, showcasing its capability to extract high-performing. stylistically diverse policies from heterogeneous offline datasets.
2 RELATED WORK
Offline RL and Imitation Learning Offline reinforcement learning (RL) leverages a fixed offline dataset to learn policy that achieves high performance in online evaluation. Previous work has focused on policy conservativeness to prevent over-estimation and mitigate out- didtof--ditribttion ac- tions during evaluation (Fujimoto et al., 2019 Wu al. 2019b Peng al. [019 Kostrikov al.) 2021 Kumar et al., 2020). Imitation Learning involves learning the behavior policy from the dataset, instead of using the reinforcement learning. Some studies in Imitation Learning also consider the potential mudti-modality of the dataset (Li et al. 2017 Kuefler & Kochenderfer 2017] Wang et al.) 2017 Igl et al., 2023 Shafiullah et al.] 2022, Wu et al. 2019a). Offline skill discovery shares sim- ilarities with learning diverse policies, but its goal is to improve performance in downstream tasks through the acquisition of skills. These approaches employ similar methods to model the latent variable and capture different skills (Laskin et al. 2022 Villecroze al. 2022).
Diversity in RL Diversity plays a crucial role in Reinforcement Learning algorithms. In some works, diversity is supposed to enhance the overall quality by means of encouraging exploration
2

---

Published as a conference paper at ICLR 2024
(Hong et al.
2018) better opponent modeling (Fu et al. 2022). Besides studies diverse policies,
skill discovery also shares the same requirement for diverse skills (Eysenbach et al. 2019 Campos
2020: Sharma et al.) 2020; Achiam et al., 2018). Other works consider optimizing both diversity and high quality in the optimization (Masood & Doshi- 2019, Zhang et al. 2019b] 7hoou al.] 2022). It is important to mention that studies diversity in the online environment
et al.
promote exploration or skill discovery. However, in this paper, our focus is on diversity in the offline setting, which beneficial for opponent modeling and various applications such as game AI and autonomous driving.
Quality-Divers Optimization Some research in the field of evolutionary algorithms also fo- cuses on discovering diverse policies. They formulate the problem the Quality-Divvessiy Opti- mization problem (Pugh et al. (206)) Cuully & (emiris][0188) where "quality" refers to the policy's s performance, and "diversity" emerges from the evolutionary iterations. An algorithm called MAP- Elites has been developed to generate diverse and high-quality policies (Cully) 2015 Mouret & Clune| 2015). Subsequent studies in this area aim improve the efficiency of the evolutionary al- gorithm by combining it with policy gradient methods (Pierrot et al. [2022 Nilsson & Cully 2021
Pierrot et al. 2022 or evolution strategies (Colas et al. Wang et al. 2022)
PRELIMINARY
In this paper, we consider Markov Decision Process (MDP) defined by the tuple (S, P,r,, Po. 7), where S is the state space. A is the action space, and P S x A x S - R is the transition function. The reward function is denoted by : S x A + R. We use Pa : + R denote the initial state distribution, and y to denote the discount factor. In standard reinforcement learning (RL), learning agent optimizes its polity π S A + to maximize the expected cumulative discounted return J(x) = E, E r'r(Se: at), where su ~ Po. Gg ~ r(ag)s)). and 8t-1 ~ P($ 1 (st, at). The value function V (sz) corresponds to the expected return of policy
state St, and the action value function Q" (st, ax) refers to the expected return obtained by playing action dr at state 8, and then following π. The advantage function A (st, ax) is defined A (st, ax) Q" (sz, a,) (5),,(s)>*() (st)).. We use dxe (s) = Ei p'p(st = s/T) to denote the unnormalized discounted state distribution induced by policy
In offline RL, an agent learns from a pre-collected dataset D consisting of multiple trajectories with- out online interaction with the environment. This paper further assumes that the dataset D contains behaviors of heterogeneous policies ( (3(2),...,,((k)))
The assumption is not restrictive for many real-world scenarios. For instance, in online gaming and autonomous driving, the dataset usually contains behaviors of number of different agents and humans, each of which may possess distinct behavioral patterns. The objective of our approach, termed Stylized Offline RL (SORL), is to learn set of high-quality and diverse policies ( (1), x ,r,,),..,,77 (m))) from the most representative behaviors exhibited by the dataset D.
4 METHOD
In order to leverage heterogeneous datasets and strike balance between policy diversity and per- formance, we propose Stylized Offline RL. (SORL), a novel two-step framework consisting of (1) EM-based style clustering and (2) weiihte style learning. In style clustering, for each cluster that represents distinct and dominant behavioral style, SORL assigns different weights to trajectories in the dataset. A trajectory's weight reflects its posterior probability of belonging to that style. Subsequently, SORL incorporates stylized advantage weighted regression to leam di- verse and high-quality policies by constraining each policy be conservative with respect to the corresponding weighted set of data. The SORI. algorithm is illustrated in Algorithm
4.1 EM-BASED STYLE CLUSTERING
The latent variable model is a natural way to disentangle a dataset generated by diverse policies Li et al. 2017; Wang al. 2017 Laskin et al., 2022). In our problem, each trajectory in the dataset is generated by an unknown latent policy. Therefore, drawing inspiration from the expectation- maximization (EM) algorithm, SORL alternates between trajectory clustering and policy learning to
3

---

Published as a conference paper at ICLR 2024
extract the most representative diverse behaviors from the heterogeneous dataset. SORL learns set of m most representative policies (1(1), (2(2)
(mm)
by maximizing the log-likelihood:
LL(( = Σ log Σ p(r|2= t)p(u(0)
(1)
JED
where 7 denotes a trajectory in the dataset, 2 is the latent variable indicating which policy 7 belongs to, p(7|2 = i) is the probability of r sampled under policy μ()), and p(u(() is the prior distribution. The policies serve as the generative model for generating the dataset. The main challenge is to identify latent z. The EM-based algorithm iteratively updates the policies and the estimation f for the posteriors p(=[r). Upon its convergence, the estimated posterior distribution p(z[T) offers clustering of trajectories in the dataset, and the policy p(() imitates the behavior in cluster i.
4.1.1 E-STEP
In the E-step, SORL calculates the estimated posterior distribution of the latent variable p(=|T) with respect to the current policies {u10). Formally, p(= i|-) p(u()) (s.a)e== (a)ss, In this paper, assume uniform prior. Empirically, find that the long trajectory horizon leads to numerical instability when multiplying probabilities. Hence, we employ an alternative approach to estimate the latent variable. Since all steps in the trajectory share the same latent variable, the posterior distribution p(z/T) is estimated by averaging across all the samples in a trajectory.
1
p(= i|
Σ (a)
(2)
Z
(s,a)Er
where is a normalizing factor. SORL uses equation to calculate posteriors for all trajectories of the dataset in the E-step.
4.1.2 M-STEP
According to Jensen's inequality (Durrett 2019).
p(= = i/7) log p(T|2 1)p(2(0)
( ΣΣ
(3)
p(= = i1(7)
TED
In the M-step, with the posteriors frozen, the policies {2(0) yu == updated to maximize the right side of inequality B, which is lower bound of the log likelihood equation]] [ In order to maximize the objective EneD Σ [p( a/T) p(T]( i)), SORL utilizes weighted imitation learning to
minimize the following loss:
Loss((u
Σ TED t=1
p(z= ilT)
Σ (a.0)
log (a
(4)
D
The style clustering algorithm shares the same convergence result with the original EM algorithm, and it is guaranteed to converge to a saddle point (Ng et al. 2012).
4.2 ADVANTAGE-WEIGHHED STYLE LEARNING
In the second step. SORL further improves the policies' performances with stylized advantage- weighted regression. Formally, SORL learns a set of policies (m(11, (2)) ...., x(m)) through the following constrained optimization problem:
Vi [m], T( arg max ((x(1)
/ (alssd 1, Vs.
(5)
8.1. Egdd ia (e) D ( (T (+1s) (1 (1s)) Se,
The policies learn to maximize cumulative return to avoid degenerate behavior, while still remaining diverse through constraining the KL divergence between م() and x(0) to be small. Maximizing the cumulative return (x)()) is equivalent to maximizing the expected improvement ()((())
(()()) (p) (i 1.. 1...m)) m). According to previous works, the expected improvement can be
expressed in terms of advantage: (Schulman et al. 2015)
(T((()) (0)
Eggd (1) E (a)s) (A (s)
4

---

Published as a conference paper at ICLR 2024
Equation 6 poses challenges in optimization due to the unknown discounted state distribution d To address this, a common approach is to substitute dyco with duco to provide a good estimate of ( ()) (Schulman et al. 2015 Peng et al., 2019h. It has been proved that the error can be bounded by the KL. divergence between x and yu (Schulman et al. 2015), which has already been constrained in our objective Therefore, we optimize the following objective
Vi E [m], (1)) arg max Band Les (e) (e) (- Au (s,a)
s.L. Eand aa (e) DKL (T (1b)l)))))))) se,
T (als)da 1, As.
In weighte style learning, we approximate A (s,a) by A" a), where represents the policy distribution of the entire dataset. This approximation is made because A" a) often has higher quality than An Subsequently, we calculate the Lagrangian of the optimization problem:
Bgnd (o (aa (s, X(E - (x (-1s)ll) (11))) + S. [ - 7
( (als)da)
(8)
Note that the policy x (6) of each style can be optimized independently. Taking derivative with respect (alss,, we can obtain the closed-form solution (11) ( (als)exp(3 (s, Finally, project (1)-- Ty paramterized by 0 by minimizing the KL divergence between them. It can
(i)
(t)
be proved that the parameterized policy of i-th style x can be learned by minimizing the following loss. Please refer Appendix Dfoor detailed proofs.
Loss( (
(0)) -E---EEp E,-DĀ(z i|TT
Σ (s.0)Er
log T (als) exp (A
(9)
Compared with the previous offline RL algorithm AWR Peng al. 2019) that learns single policy, SORL learns a set of diverse policies by assigning different weights to trajectories in the dataset. SORL leverages the estimated posterior distribution calculated step one to force different policies to focus on different clusters of trajectories that possess distinct behavior patterns.
The two-step SORI. algorithm illustrated in Algorithm SORL. leverages the EM- EM-based style clustering to extract distinct styles from the dataset. Furthermore, it takes into account both diversity and quality improvements to perform advantage-weighte learning. As result, the SORL algorithm extracts from the heterogeneous dataset collection of diverse policies with high quality.
EXPERIMENTS
In this experimental section, aim to address the following questions: (1) Can SORL derive high- performing policies that exhibit diverse behaviors from an offline heterogeneous dataset originating from various sources? (2) How does SORL compare to prior methods that focus learning diverse behaviors from an offline heterogeneous dataset? (3) Is SORL suitable for complex, real-world tasks, especially those involving large-scale datasets collected from human users?
Our experiments encompass a range of tasks and offline datasets. For each experiment, we train a of policies and evaluate them based on three criteria: quality, diversity, and consistency. These criteria respectively reflect the performance, behavioral diversity, and whether the learned diversity inherent in the offline dataset, addressing the question (1). To answer question (2), compare SORL against two prior methods: Off-RLPMM (Wu et al., 2019a) and InfoGAIL eLi al. 2017). This comparison spans three sets of experiments, each with increasing levels of difficulty.
The first experiment involves a grid grid-shooting game where the dataset is manually collected from two distinct winning strategies. This experiment allows for detailed analysis of the learned policies as we have full control over the game environment and access to the ground truth styles in the dataset. The second experiment features a set of Atari games. The datasets for these games are recorded from human players in controlled, semi-frame-by-fiie the quality of the dataset is relatively high. This experiment assesses the performance of the methods when learning from heterogeneous datasets with minimal interference from inconsistent data quality. The final experiment centers on the popular basketball video game "Dunk City Dynasty" (FuxiRL) 2023). Here, datasets recorded directly from online players, leading to greater diversity and
manner (Zhang et al. 2019a). As a result,
5

---

Published as a conference paper at ICLR 2024
Algorithm Stylized Offline RL (SORL)
1: Input: offline heterogeneous dataset D, number of policies m
2: Output: learned diverse policies mg)) = =(())
3: Initialize policies x²0) (m)) and p1 (m))
4: EM-based style clustering
5: while not converged do
6: # E-step
7: Calculate the posterior probability of styles p(=17) with Equation
8: # M-step
9: Calculate the loss function Loss( ()))(()))))))))))))))))))))))( defined in Equation 4
10: Update {u(0} by taking one step of gradient descent with respect to Loss (onss((ui)) (fu(0))
11: end while
12: # Advantage- -renhtge-weighhed style learning
13: Calculate the posterior probability of styles p(2(T) with Equation 2
14: while not converged do
15: Calculate the loss function Loss((=))) defined in Equation
16: Update { (2)) by taking one step of gradient descent with respect to Loss ( ()))
end while
considerable variations data quality. This experiment evaluates the effectiveness the methods in handling complex, real-world tasks using datasets collected from diverse group of humans, addressing the question (3).
In the following, introduce the baseline methods and the evaluation criteria, and then present the results of the three experiments.
5.1 BASELINES
OF-RLPMM: RLPMM (Wu et al. 2019a) adopts a similar EM framework, but it is not directly applicable in our offline setting because it requires additional online interaction train classifier for labeling all the trajectories the M-step. We implement an offline version of RLPMM (Off- RLPMM) that adopts similar M-step SORL, except that Off-RLPMM partitions the dataset into disjoint subsets instead of using SORL's weighting scheme.
InfoGAIL: InfoGAIL. Lii et al. 2017) studies imitation learning from multimodal datasets. Differ- ent from the EM stage of SORL. InfoGAIL infers the latent style of trajectories by maximizing the mutual information between the latent codes and trajectories.
In both baselines, we employ Behavior Cloning (BC) as the imitation learning algorithm. Addi- tionally, in contrast to the two-stage algorithm SORL, both baselines do not have a second policy improvement stage. Detailed explanations are provided in Appendix
5.2 EVALUATION CRITERIA
We evaluate the learned policy with the following three criteria.
Quality:
The quality of policy is assessed by the episode return of trajectories during evaluation. reflects the performance the policy in completing tasks.
Diversity is determined by the entropy of styles identified from the learned policies. With learned clustering p. each trajectory is classified into a style with the highest probability. Diversity then calculated as the entropy of the distribution of styles over the whole dataset. In conjunction with the later-introduced consistency metric, higher entropy suggests that policies exhibit diverse, distinguishable behaviors.
Diversity:
Ppotalarity = 1) = [D]
Σ I(arg max{p(2 = j(r)} = TED
j
(10)
Diversity == Entropy (popptrrip

---

Published as a conference paper at ICLR 2024
Consistency: The consistency measures the likelihood that the learned policy generates actions corresponding to the specifie cluster it associated with within the dataset. Evaluating consistency is crucial to ensure that the behaviors have learned are inherent in the heterogeneous datasets, rather than being unknown out-of-distribution actions, which could also artificially inflate diversity scores.
5.3 DIDACTIC EXAMPLE: GRID SHOOTTING
In the grid shooting game, a player navigates in 9x9 grid world, controlling a character that engages in combat against Al opponent. Both the player and the Al have the ability to move in four directions within the grid, shoot at each other, and gather stars that randomly appear in one of the grid cells. Rewards and termination conditions distributed based the subsequent events:
Shooting: If the player and the Al opponent are positioned in the same row or column, they have the opportunity to shoot at each other. A successful shot ends the game, with the shooter being declared the winner and receiving reward of 10.
Star Collection: A star is randomly placed in grid cell, and its location is known to both the player and the opponent. Whichever party reaches the location the star earns a reward of 1, After the star is collected, it reappears randomly in another grid cell. If no successful shooting occurs, the game terminates at the 100th step, and the participant with the higher accumulated rewards is announced as the winner.
Methods
Learned policies
Reward (shoot)
Reward (star)
Winning rate
policy 1
2.6 + 0.1
2.2 1.3
51.0 = 0.1%
SORL
policy 2
1.1 + 0.4
6.3 + 0.2
59.3 + 0.0%
policy I
2.7 0.1
0.0
26.5 = 0.0%
OIf-RLPMM
policy 2
0.0
8.6
54.4 + 0.0%
policy I
0.1
4.7
52.0 5.7%
InfoGAIL
policy 2
1.5 0.3
3.5
58.7 7.4%
Table 1: The reward distribution and the winning rate of the policies learned from the heteroge- neous dataset.
(a) A sample trajectory of style that shoots the en- emy.
(b) A sample trajectory of style 2 that gets the stars.
Figure 1: Visualization of policies learned by SORL, capturing the main playing styles in the dataset.
In the Grid Shooting environment, there are two distinct winning strategies or playing styles. The first strategy emphasizes shooting the enemy, while the second focuses on collecting stars and avoid- ing direct confrontations with the enemy, as depicted in Figure [ We constructed our dataset by combining trajectories from both playing styles. Using this dataset, we trained two policies with SORL and compared them with two baseline methods. The rewards earned for each event and the final winning rates are detailed in T
From the table, it is evident that SORL successfully captures the core difference between the two playing styles. This is reflected the distinct reward distributions for shooting and star collecting across the two learned policies. Furthermore, the winning rates of the policies promising, un- derscoring their robust performance. In contrast, while Off- RLMMM also captures the two playing styles, the winning rates of its policies are lower than those of SORL. InfoGAlL. the other hand,

---

Published as a conference paper at ICLR 2024
yields competitive winning rate but fails differentiate between the styles of the policies. Figure shows example trajectories for the two learned policies of SORL. More detailed quantitative results including the quality, diversity and consistency is presented in Appendix A.2
5.4
ATARI GAMES
In this experiment, we focus on six Atari games, including SpaceInvaders, MsPacman, Montezu- maRevenge, Enduro. Riverraid, and Frostbite. For each game, We train three policies using the dataset provided by Atari Head (Zhag et al. 2019a), a large-scale dataset of human players. The dataset recorded in controllable cemi-frame-by-- by-framee manner, ensuring high data quality. The experimental results, which include metrics quality, diversity, and consistency, are presented in A visual representation of these results, using a radar plot, available in Figure
Table 2
qualit
quality
quality
501L OHFLLPNN whoot
5OTL OPNUNN
0.2
0,4
diversity
ob
con prssstency
diversit
5e consistency
diversit.
corsistency
Inacelnvaders
(b) MsPacman
(c) MontezumaRevenge
quality
quality
quality
0.6 comsistency
0.5
0.5 comsistency
0.5 commistency
(d) Enduro
(c) Riverraid
(f) Frosthite
Figure 2: The radar plot of quality, diversity, and consistency in the Atari environment.
Games
Methods
Quality
Diversity
Consistency
SORL
387.1 33.3
0.96 0.11
94.7 0.1%
SpaceInvaders
Off-RLPMM
412.5 16.3
0.22 + 0.15
89.8 + 0.2%
InfoGAIL.
353.2+
0.0
92.3 = 0.7%
SORL
6
0.91 + 0.09
94.5 = 0.2%
MsPacman
Off-RLPMM
543.4
0.88 0.09
90.0 0.1%
InfoGAIL
558.0 137.8
0.00 0.00
94.2 2.1
SORL
306.7 19.1
1.05
95.1 0.1%
MontezumaRevenge
Off-RLPMM
216.7 95.3
0.86
94.8 + 0.1
InfoGAII.
2
0.00 + 0.00
92.3 1.1
SORL
371.1
0.84
96.2 0.1%
Enduro
Off-RLPMM
317.2
0.00 -00000 0.00
89.1 19.1=00 0.6%
InfoGAIL
351.3+31..
0.00 + 0.00
84.4 = 0.8%
SORL
1.00 0.04
0.2%% 95.1 111100 0.2%
Riverraid
Off-RLPMM
+
0.39 + 0.29
89.7 + 3.9%
InfoGAII.
1882.6 149.8
0.00 0.00
89.2 1.5%
SORL
2056.0 + 391.9
0.97 0.04
93.5 = 0.4%
Frostbite
Off-RLPMM
2183.2 + 502.4
0.60 + 0.07
91.6 = 0.0%
InfoGAIL
2584.7 206.5
0.00 0.00
91.8 = 0.3%
Table 2: The experiment results of quality, diversity and consistency in the Atari environment.
From T we see results consistent with those in the grid shooting game. SORL outperforms the baseline methods, attaining the highest scores in both diversity and consistency for all games, and
8

---

Published as a conference paper at ICLR 2024
secures the top quality score in four out of the six games. While Off Off-RLPMM does exhibit some diversity, its policy quality is weaker. Conversely, InfoGAIL achieves competitive quality scores but struggles to learn diverse policies. Visualizations of the stylized policies are in Appendix H
5.5 VIDEO GAME APPLICATION
Dunk City Dynasty" (FuxiRL p023 is an online mobile game where players control a character to play in 3v3 basketball match. The game presents formidable challenge to existing algorithms due to its high-dimensional state and action spaces. For this experiment, we collect a dataset of over 100,000 steps, directly from online players. Compared to the other two experiments, the dataset used here has notable variations in both behavioral styles and data quality. Same as Atari games, we train three policies with SORL and compare them with baseline methods. The experimental results on quality, diversity, and consistency are presented in Table The results highlight that SORL con- sistently outperforms in all three evaluation metries, underscoring its robustness and adaptability in handling complex, real- world tasks, especially when working with large- scale datasets from diverse human players.
Methods SORL
Quality
Diversity
Consistency
3.4
1.05 + 0.0
40.7 + 4.6%
Off-RLPMM InfoGAIL
1.7
38.6 5.4% 0.51 0.18
5.0
0.73 + 0.22
39.8 + 3.3%
Table 3: The experiment results of quality, diversity and consistency in the "Dunk City Dynasty" environment.
(a) Screenshot
(b) Policy
(c) Policy 2
(d) Policy
Figure 3: A scene in the game, and the shooting areas favored by each policy. The grayscale repre- sents the distribution of shooting action locations, with darker shades indicating higher probabilities and lighter shades signifying lower probabilities.
We observed distinct shooting range preferences among the three policies. For example, policy tends to favor short-range shots, while policy is inclined towards long-range shots. The preferred shooting areas for each policy visualized in Figure 3 Additional screenshots showcasing typical behaviors these policies can be found in Figure 5 in the Appendix B To better illustrate the
behaviors exhibited by the learned policies, have included gameplay videos in the supplementary materials..
CONCLUSION
In this paper, explored the extraction of diverse and high-quality behaviors from offline heteroge- neous datasets. These datasets, sourced from multiple origins, inherently possess a multimodal data distribution with inconsistent data quality. Such setting poses significant challenges to existing methods, which can be affected by low-quality data or lack the control to differentiate the behav- ioral while learning. To address these challenges, we introduced the Stylized Offline Reinforce- ment Learning (SORL) framework. SORL employs an EM-based style clustering combined with advantage- traantage-wiighted policy learning. This design not only optimizes the performance of the poli- cies but also preserves the inherent behavioral diversity found in heterogeneous datasets. Through extensive experiments, we compared SORL with two prior methods across various tasks and hetero- geneous offline datasets. Our results underscore SORL's superior capability in extracting behaviors that are both high-quality and diverse. Our future work aims to offer a more flexible formulation of policy behavior, potentially allowing shared behaviors between policies, making it more relevant for real-world applications. Furthermore, we plan to integrate adaptive task learning using the diverse policies derived from SORL, enabling dynamic switching in different task scenarios.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
The source code for the SORL algorithm and the baselines used in this study are included in the supplementary materials. The proof sketch can be found in Section 4.2, and comprehensive proof is provided in the Appendix [4.2] For the Grid Shooting environment, the data processing steps involve using the state and action directly from the environment output. As for the Atari Head dataset, the processing steps follow the methodology outlined in the original package.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms, 2018.
Hugo Araujo, Mohammad Reza Mousavi, and Mahsa Varshosaz. Testing, validation, and veri- fication of robotic and autonomous systems: A systematic review. ACM Trans. Sofhw. Eng.
Methodol., 32(2), mar 2023. ISSN 1049-331X. doi: 10.1145/3542945. URL https:
//doi. org/ 10. 11 44/53542945
Victor Campos, Alexander Trott, Caiming Xiong, Richard Socher, Xavier Giro-I-Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Confer- ence Machine Learning, volume 119 of Proceedings of Machine Learning Reseurch, pp-
1317-1327. PMLR, 13-18 Jul 2020. URL https: https:://proceeding..oo mir..mm.r.... press/v119// stg.add/d....m.r.pppsssss///.. campos20a. html
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. Lapo: Latent-variable advantage-weighe policy optimiza- tion for offline reinforcement learning. Advances in Neural Information Processing Systems, 35: 36902-36913, 2022.
Cédric Colas, Vashisht Madhavan, Joost Huizinga, and Jeff Clune. Scaling map-elites to deep neuroevolution. In Proceedings of the 2020 Genetic and Evolutionary Computation Confer- ence, GECCO '20. pp. 67-75, New York, NY, USA, 2020. Association for Computing Machin- cry. ISBN 1281450371285 doi: 10.1145/3377930 3390227.. URL httpt:tt///do org/Ic. 1145/3377930 .390211
Antoine Cully and Yiannis Demiris. Quality and diversity optimization: A unifying modular framework. IEEE Transactions on Evolutionary Computation, 22(2):245-259, 2018. doi:
10.1109/EEC22217277777.1
Clune J. Tarapore et al Cully, A. Robots that can adapt like animals. Nature, 521:503-507, 2015. doi: https://doi.org/ 1t.g//1.1100 w030//0ttrree/444c
Rick Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In International Conference on Learning Representa- tions, 2019. URL L /tttt::t/// //tpprevee/eeeeeeiieeee
Haobo Fu, Ye Tian, Hongxiang Yu, Weiming Liu, Shuang Wu, Jiechao Xiong, Ying When, Kai Li, Junliang Xing, Qiang Fu, and Wei Yang. Greedy when sure and conservative when uncertain about the opponents. In International Conference on Machine Learning. PMLR, 2022.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration, 2019.
FuxiRI.. Dunk city dynasty: Multi-agent basketball environment. htttps://iithub.. com/ 2023. 2023.
Fuxi R
tydynasty St./Dunkkiityyyaatty/
Zhang-War Wei Hong, Tzu- Yun Shann, Shih-Yang
Yang Su, Yi-Hsiang Chang, and Chun-Yi Lee. Diversity-
driven exploration strategy for deep reinforcement learning, 2018. URL https://
10

---

Published as a conference paper at ICLR 2024
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Maximilian Igl. Pumit Shah, Paul Mougin, Sirish Srinivasan, Tarun Gupta, Brandyn White, Kyriacos Shiarlis, and Shimon Whiteson. Latent hierarchical imitation learning for stochastic environ- ments, 2023. URLhttps. http:sss///ppeeeveeeee net formmmormmmm//mm wwte:iid=dswwwwmmmm ZP DOH
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning, 2021.
Alex Kuefler and Mykel J. Kochenderfer. Bur-iin demonstrations for mudti-modal imitation learn- ing. CoRR, abs/17 2017.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning, 2020.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel.
Cic: Contrastive intrinsic control for unsupervised skill discovery, 2022.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Proceedings of the 31st International Conference on Neural Information Pro- cessing Systems, NIPS' 17, pp. 3815-3825, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 97815108800 1786569669
Yicheng Luo, Zhengyao Jiang, Samuel Cohen, Edward Grefenstette, and Marc Peter Deisenroth. Optimal transport for offline imitation learning, 2023.
Muhammad Masood and Finale Deshi-Velez. Diversity Diversity-inducing -inuucing policy gradient: Using maximum mean discrepancy to find set of diverse policies. In Proceedings of the Twenty-Fighth Interna-
tional Joint Conference on Artificial Intelligence, IJCAI-19, pp. 5923-5929, International Joint
Conferences on Artificial Intelligence Organization, 7 2019. doi: 10.24966/rjeai221198811 URL https: org.orggg/oo 10. 24 24963/ /////////aa....////// /tt.:/.a............... 2019/821
Jean-Baptiste Mouret and Jeff Clune. Illuminating search spaces by mapping clites. CoRR, abs 04909
abs/1504.00409, 2015. URL http::ttpp/.....oooo// http://wwwwv..gaa////abbbaaa/a/e
Gerhard Neumann and Jan Peters. Fitted q-iteration by advantage weighted regression. In Proceed- ings of the 21st International Conference on Neural Information Processing Systems, NIPS'08, pp. 1177-1184 1184, Red Hook, NY, USA, 2008. Curran Associates Inc. ISBN 2781605609492.
Shu Kay Ng, Thriyambakam Krishnan, and Geoffrey McLachlan. The Algorithm, pp. 139-172 Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3-642- 21551-3.
doi: 10.1007798833-62221155-333
URL
978 3- 642-664222222 21551-336
Olle Nilsson and Antoine Cully. Policy gradient assisted map-elites. In Proceedings the Genetic and Evolutionary Computation Conference, GECCO '21, pp. 866-875, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781 4781453333599 doi: 10.1145/3449639. 3459304. URL
100. 1ee.11114444933 3459304
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019.
Thomas Pierrot, Valentin Macé, Félix Chalumeau, Arthur Flajolet, Geoffrey Cideron, Karim Beguir, Diversity policy gradient for sample
Antoine Cully, Olivier Sigaud, and Nicolas efficient -uality-diversity optimization, 2022.
Justin K. Pugh, Lisa B. Soros, and Kenneth O. Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers Robotics AI, 3:40, 2016. URL https::/api. semant cesiaaantsccclarr.... wwwpolla..rrrrooooooooooooo :D::://ooru:u::11110
John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. Trust region policy optimization. ICML'15, pp. 1889-1897.. Jor.R 2015.
11

---

Published as a conference paper at ICLR 2024
Nur Muhammad Mahi Shafiullah, Zichen Jeff Cui, Ariuntuya Altanzaya, and Lerrel Pinto. Behavior transformers: Cloning & modes with one stone, 2022.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
URL httpttts:///peereee/aaaaae net wttp:e..eee//rurrrieeeem
forum? Barpummid-HJgglii Hd-
Ruimin Shen, Yan Zheng. Jianye Hao, Zhaopeng Meng. Yingfeng Chen, Changjie Fan, and Yang Liu. Generating behavior-diverse game ais with evolutionary multi-objective deep reinforcement learning. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelli- gence, IJCAI'20, 2021. ISBN 97809992411655
Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Ne- unert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Confer- ence on Learning Representations, 2020. URL https:: ptttps://operrevieee net www.:/eet/orume... forum?id'
Valentin Villecroze, Harr Braviner, Panteha Naderian, Chris Maddison, and Gabriel Loaiza- Ganem. Bayesian nonparametries for offine skill discovery. In International Conference Machine Learning, pp. 22284-22299) PMIR, 2022.
Qing Wang. Jiechao Xiong. Lei Han, peng sun, Han Liu, and Tong Zhang. Ex- ponentially weighted imitation learning for batched historical data. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: http::::s/ppooeeednggeee www.:ddinnnn....i.s..o.m fil A.le/4aec
files/paper/2018/ F33. 7e 41 100e-Paper..pff
Yutong Wang, Ke Xue, and Chao Qian. Evolutionary diversity optimization with clustering- based selection for reinforcement learning. In International Conference on Learning Representations, 2022. /ttpstts:///ppeeeeeeee http://wwww...e..r.a.aaeeee////.
Ziyu Wang, Josh Merel, Scott Reed, Greg Wayne, Nando de Freitas, and Nicolas Heess. Robust imitation of diverse behaviors. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 17, pp. 5326 5335,, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Weichang Wu, Junchi Yan, Xiaokang Yang. and Hongyuan Zha. Reinforcement learning with policy mixture model for temporal point processes clustering. ArXiv, abs/1905. 12345, 2019a. URÍ.
https: htaps://ap.... wwwanteatiiio icchoic org/CorpusID:: 6816 :et.ppuusI:::111888888811
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. 119.../1111111111111111111111111
CoRR, abs/1911.11361. 2019b. URL
XiaoPeng Yu, Jiechuan Jiang, Wanpeng Zhang, Haobin Jiang, and Zongqing Lu. Model-based opponent modeling. In S. Koyejo, S. Mohamed, A. Agarwal, D. Bel- grave, K. Cho, and A. Oh (eds.). Advances in Neural Information Processing Systems, volume 35, pp. 28208-28221. Curran Associates, Inc., 2022. URL
b528 b5285599e221aaddd88223222-----00
Ruohan Zhang, Calen Walshe, Zhuode Liu, Lin Guan, Karl S. Muller, Jake A. Whritner, Luxin Zhang, Mary M. Hayhoe, and Dana H. Ballard. Atari-head: Atari human eye-tracking and demon- stration dataset, 2019a.
Yunbo Zhang, Wenhao Yu, and Greg Turk. Learning novel policies for tasks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 7483-7492. PMLR, 09-15 Jun 2019b.
mmr. /tt.://d.nn.....pp.ss/////////// http://99//aaaggg..g html
Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward-swing wwirt--withhhng policy optimization. In International Conference on Learning Representations, 2022. URL httpttttts:///pot
net. wtw.tt/.oorrmmmieecccctt..
12

---

Published as a conference paper at ICLR 2024
Appendix
A
GRID SHOOTING
A.1 ENVIRONMENT DETAILS
The state space dimension is 56. The action space is discrete action space with 9 actions. The opponent is an agent with fixed strategy that move randomly, and shoot with probability once the shooting action cools down.
A.2 QUANTITATIVE EXPERIMENT RESULTS OF THE GRID SHOOTING ENVIRONMENT
episode return
SORL IIODAL
0.1
0.2
Quality 55.1 + 4.6%
Diversity
Consistency
diversity
0.4
accuracy
0.5
SORL Off-RLPMM InfoGAIL
0.60 + 0.07
86.4 + 0.4% 90.5 + 0.0%
40.4 0.7%
0.68 + 0.01
55.3 + 2.1%
0.00 + 0.00
84.0 + 0.3%
Figure 4: The radar plot of qual- ity, diversity and consistency in the Table The experiment results of quality, diversity and Grid Shooting environment.
consistency in the Grid Shooting environment.
B "DUNK CITY DYNASTY"
B.1 ENVIRONMENT DETAILS
The state space dimension is 468, including global state, allies' states and enemies' states. The action space is a discrete space with 52 actions. The opponent in evaluation is an agent with moderate strength, that is learned by vanilla Behavioral Cloning for similar training steps (100,000 steps).
B.2 SCREENSHOTS IN "DUNK CITY DYNASTY"
Table presents pressenss three metrics in the experiment, illustrating SORL's ability learn diverse policies while achieving satisfactory performance. Figure Shows screenshots from the videos showcasing each style's self-playing behavior. Supplementary materials include videos that provide additional visual demonstrations, For instance, style demonstrates preference for shooting in the short range, while style 3 favors long-range shots. The videos reveal that the proportion of goals scored inside the restricted area is 83.3%, 66.7%, and 33.3% for styles 1, 2 and 3, respectively. Furthermore, the proportion of two-point shots is 91.6%, 88.9% and 66.7%. Figure depicts the probabilities of shooting in different regions. In Figure 6, (a) and (c) showcase typical shots of Style and 3. Panel (a) demonstrates shots within the restricted zone, while panel (c) displays long- range shots. Style 2 is characterized ball possession and running across the arena. The videos indicate that Style has the fewest number of passes (41, 29, 39, respectively). and the lowest proportion of goals scored directly after pass (27.7% 27,7%, 22. 22.2%, 44.4%).
B.3 THE PLOT OF SHOOTING POSITIONS
We plot the shooting positions Figure 0 based on the gameplay data. The shooting positions in the plot corresponds the distribution in Figure
13

---

Published as a conference paper at ICLR 2024
(a) Style prefers short-range shots.
0
(b) Style 2 prefers ball handling run.
(c) Style 3 prefers long-range shots.
Figure 5: The screenshots of styles.
(a) Policy
(b) Policy
(c) Policy
Figure 6: The Plot of Shooting Positions.
C ABLATION STUDY
In this section, we performed ablation experiments on the advantage-weighted policy learning. Based on the algorithm description provided in Section the SORL algorithm can be divided into two parts. Therefore, when the advantage-weit weighted policy learning is excluded, SORL reduces to the EM-based style clustering. Table 5 presents the results, indicating that the consistency and diversity metrics show similar performance, while the quality metric improves as a result of the weighte style learning.
14

---

Published as a conference paper at ICLR 2024
Quality
Diversity
Consistency
SORL
387.1 + 33.3
0.96 + 0.11
94.7 = 0.1%
SpaceInvaders
SORL w/o advantage
349.9 50.2
0.90 + 0.14
93.9 0.2%
SORL
622.2 + 65.3
0.91 + 0.09
94.5 0.2%
MsPacman
SORL w/o advantage
590.0 +: 40.0
1.02 + 0.05
93.9 + 0.2%
SORI.
306.7 19.1
1.05 0.03
95.1 0.1%
MontezumaRevenge
SORL w/o advantage
211.1 + 39.7
1.01 + 0.05
94.7 + 0.1%
SORL
371.1
0.84 + 0.25
96.2 = 0.1%
Enduro
SORL w/o advantage
9.7
0.91 0.21
95.8 0.1%
SORL
1931.2 432.
1.00 = 0.04
95.1=1 0.2%
Riverraid
SORL w/o advantage
1842.4 282.2
0.95 0.00
94.1
SORL
2056.0 = 391.9
0.97 0.04
Frostbite
SORL w/o advantage
1787.8 + 250.6
0.90
0.14
SORL
5.3 = 3.4
1.05 0.0
40.7 4.6%
DunaCityDynasty
SORL w/o advantage
0.7 - 0.9
0.92 0.11
39.8 1
Table The ablation study of SORL, comparing the SORL algorithm without advantage-weighted
style learning.
D PROOF DETAILS
We provide detailed proof of solving the constrained optimization problem in Section 4.2 The original problem is as follows.
Vi [m]. (6) arg max Egnd .e9 Baxr (-10) (s,a)
8.1. Egwd (1)()))))))))))))) DKL ( (0) (11))]1 (11)) Se,
(11)
(1) (als)da Vs.
In -yvanttge-wiightee style learning, we approximate A (s, a) by (" a). where represents the policy distribution of the entire dataset. This approximation is made because A",s, a) often has higher quality than A Subsequently, we calculate the Lagrangian of the optimization problem:
L( A, a) =Ea-d .n (5) [
A(E (x (11s)la
(12)
+ / as 10 (als)da)
Differentiating on (1),
aL
d (x00,))44((ssa)) Alog (a(ss) Alog / (als) A] as
(13)
(als)
We
to 0. and can get the closed-form solution (a(ss) 2 ( (als) exp( & A (s, a)). where the normalization term ( (s) exp [6 + 1).
set
97.
Finally, we project - (1)= T paramterized by 0 by minimizing the KL divergence between them. D(1) denotes the trajectories corresponding style in the dataset, which unknown. Hence,
15

---

Published as a conference paper at ICLR 2024
rewrite the expression by incorporating the probability of each trajectory being the i-th style. arg min B (D ( (-1s)[) (8) (-1))
arg min E D()) (T (als) (als) T (als) log m8 ( (a|s)))
arg min D
(ax)) (als))
(a|s))
arg min DD
(s) (a|s) exp( a)) log m (als))
(14)
arg min E DWD Dín E BaD log T (als) (8) exp( 1
π (*
arg min E log m (a(s) ((0(s)
exp( (8,
arg min E,. i|-) log (als) (s)
To
exp( π (s)
Similar to AWR Peng et al. 2019) and other prior work (Neumann & Peters
2008 Siegel et al.
2020 Wang et al. 2018) 8). we neglect the per-state normalizing constant Z (s). The policy update
can be expressed as follows:
arg min -E,~Dp(z = i|7) log = (als) exp( A"
(15)
The original problem II has solution because it satisfies the Linear Indepence Constraint Qualifica- tion (LICQ)..
E EXPERIMENT DETAILS
Network structure We construct the network based the default network of the relative task according to the codebase we use. The network of Grid Shooting and Dunk City Dynasty a 3 3-layer MLP. and the network of Atari environments has three convolution layers and two linear layers.
Policy
Policy
Policy m
B E RR rx
Brxk
Shared Weights
W pdik
Ag RAXY
Ag Rdxr
Am Rddr
Figure 7: The usage of LoRA in the network structure of multiple policies.
Besides. in order to balanced learning among all the styles, share the main network and use a LoRA module to discriminate different styles. LoRA (Hu et al. (20TT) is a widely used network structure, that substitutes the original matrix of the linear layer by matrix W and low-
16

---

Published as a conference paper at ICLR 2024
rank multiplication of two other matrices A, and B, In our setting shown in Figure 7 the matrix W is shared among all styles of policies, while A,, B, varies. The input dimension and output dimension & depends on the linear layer, and the rank is set to 9 in this work.
Number of styles In Grid Shooting, we use 2 styles because we want to recover the two real styles of behaviors in the dataset. In other environments, we learn styles from the dataset.
Hyperparameters and other details In Grid Shooting, size 20000 and the number of epochs 10. In Atari environments, we use the batch size of 32 x 5 where we sample 32 (s, a) pairs for 5 times, with the number of epochs 30. In Dunk City Dynasty, we use the batch size of around 300 and 100, 000 steps for the quality, diversity, and consistency metrics. We use 900, 000 steps for generating the playing videos of different styles.
the batch size 128, offline dataset
We re-implement the InfoGAIL with continuous latent variables in all the environments. The 3 styles are extracted by applying (1, O, 0), (0, 1, 0), (0, 0, 1) the latent variable to the policy model and get the 3 different policies.
The Grid Shooting and Atari results in the tables show the mean and standard deviation among random seeds.
F ADDITIONAL METRICS EVALUATING DIVERSITY
In the main text, we use the diversity metric of popularity, which is dataset-related metric that evaluates diversity. However, the definition of diversity, especially in the offline setting, is not unique. We provide additional diversity metries in this part. The definitions of metries are as follows.
1. Skill metric measures the dissimilarity of the skill set. The skill set is defined as the vector of rewards obtained by each skill, c.g., skill (hshoots star in the grid shoot- ing environment. And the dissimilarity between two skill sets skill,, skilly is defined as the value of their cross product daku((skill skill2) = ||skill] x skillz]]. The dis- similarity metric is obtained by first sampling trajectories with different policies, and calculating the average dissimilarity between pair-wise skill sets, i.e., Diversilyssat = E Ekktil,, dkk.kkkkkkklal(fff (dsuu(skill, skill,) where skill,, skill, are sampled from different styles.
2. OT (optimal transport) metric is based on the optimal transport distance (also known as Wasserstein distance). The similar idea used in recent work imitation learn- ing (Luo et al.) (023). When calculating the trajectory-level OT distance, align steps with neighbor states together and sum up to the overall distance, i.c., dor(r' ()' arg min ue M Etn Er |ls! - s² lle,e where M = {o T ,1l = $1, = 1).. And the diversity metric is calculated by averaging all trajectory pairs T", TJ from different policies Diversilyor E,,, do (7', -²).
We employ a normalization technique to enhance the interpretability of the OT metric. The normalization is achieved by applying the formula parmalize(Diversityy Diversityor-a where the OT metric on trajectories sampled from the same policy.
3. Discrimination metric represents how different policies can be discriminated by a neural network. After collecting trajectories of different policies, we first train a neural network to predict the policy index, and the discrimination metric is just the evaluation accuracy.
Popularty metric
Skill metric
OT metric
Discrimination metric
SORL SORL w/o advantage
0.60
15.7
0.022
0.75
0.61
21.2
0.108
0.97
Off-RLPMM InfoGAIL
0.68
23.2
0.048
0.99
0.00
2.0
0.000
0.69
Table 6: The diversity under differen metrics.
Based on the results of the additional diversity metrics presented in Table 6 we can conclude that the SORL algorithm is capable of obtaining diverse policies.
17

---

Published as a conference paper at ICLR 2024
G COMPARISON WITH OFFLINE RL BASELINES
SORL (ours) 387.1 + 33.3
CQL
AWR
IQL 361.5 86.9 505.0
SpaceInvaders
136.5 + 27.5
339.3 57.1
MsPacman
622.2 65.3
MoneezumaRevenge
306.7 19.1
113.4 196.3
32.1
Enduro
371.1 +: 2.9
0.0 + 0.1
345.7 40.2
290.1 + 44.2
Riverraid
1931.2 + 432.1
1127.7 160.6
1710.3 + 270.0
1909.3 332.1
Frostbite
2056.0 + 391.9
76.3 + 16.7
1528.0 + 329.4
2184.7 = 337.5
Table 7: The comparision with offline RL baselines.
present the results of standard offline RL methods in Table which do not focus on diverse policy learning.
H ATARI ENVIRONMENT VISUALIZATION
H.1 SPACEINVADERS
0000
0000
R
R
R
R
Bunker
Shooter
Figure 8: The game interface SpaceInvaders.
In SpaceInvaders Figure g the shooter can move horizontally, and there are three stationary bunkers positioned above the shooter. The shooter can shoot from bottom to top, in order to destroy the aliens above to get scores. During playing, there are random attacks from top bottom that can destroy the shooter. However, staying under the bunker also be destroyed by continuously attacking it. In addition, the aliens slowly swing left and right together all the time. In the game, players moves horizontally shoot from a correct position with an alien above, and avoid being attacked.
prevent those attacks. The bunker
Leftmost Bunker
Bunker Bunker 2
Bunker 2 Bunker 3
Bunker 3 Rightmost
Style
23%
19%
36%
22%
Style
10%
57%
15%
18%
Style
32%
38%
26%
4%
Table 8: In SpaceInvaders, the probability of the shooter in different regions.
18

---

Published as a conference paper at ICLR 2024
(a) Style
(b) Style
(c) Style 3
Figure 9: In Spacelnvaders, the probability distribution of the shooter's position.
To evaluate the diversity of learned policies, we calculate the frequency of the shooter's position over 30,000+ steps for each style. Table and Figureedddmmnnnnntn @demonstrates that different styles of policies tend to prefer different positions.
To better understand the style learned clusters, we analyze the game mechanics which partially explains the preferences of styles. In Spacelnvaders, shortcut of getting high scores is to staying close to the left of the Bunker (and also staying close to the right of the Bunker 2) to destroy all aliens in the leftmost column (and the rightmost column), because there fewer attacks from top to bottom and close to the bunker, where destroying the aliens in the leftmost/rightmostt column provides large amount scores. As result, style and style prefers staying in the leftmost, and style and style 2 prefers staying the leftmost.
H.2 MSPACMAN
In MsPacman as shown in Figure I0] the map and starting position are fixed. To evaluate the diversity of learned policies, we calculate the frequency of the beginning trajectory until the first corner for each style, based on 30 trajectories per style. There are a total of six possible corners. Table displays the different preferred corners for each style.
Figure The game interface MsPacman.
Besides, we analyze the visualizations of the dataset trajectories. Atari datasets in our experiment are collected by three players, which exhibit some diversity in terms of policy variations. As shown the following table I0 in the game MsPacman, different human players (named J. K and R) have varying preferences for the first visited corners. Additionally, even within a single player's gameplay, there can be variations in visitation patterns.
19

---

52.2% Published as a conference paper at ICLR 2024
Which corner
3rd left
2nd left
1st left
1st right
2nd right
3rd right
Style 0%
33%
67%
0%
0%
0%
Style 20%
7%
7%
3%
27% 37%
Style 3
0%
97%
3%
0%
0%
0%
Table 9: In MsPacman, the probability of the pacman choosing different routes in the beginning.
Which corner
3rd left
2nd left
1st left
1st right
2nd right
3rd right
Player J (2 trajs)
0%
0%
50%
50%
0%
0%
Player K trajs)
0%
0%
50%
50%
0%
0%
Player R (16 trajs)
6%
44%
6%
19%
19%
6%
Table 10: the dataset MsPacman, the probability of the pacman choosing different routes the beginning.
I SENSITIVITY ANALYSIS ON THE NUMBER OF POLICIES
Sensitivity analysis We have performed experiments to evaluate the diversity, quality, and consis- tency of SORL under different values of m, where m is the number of policies. The performance of SORL in Table [ remains similar across different values, indicating that the algorithm is highly sensitive to the hyperparameter m.
m = 2
m7 = 3
m7 = 4
m =
Diversity
0.61
0.85
0.65 88.7%
0.69
Quality
55.1%
50.3%
52.5%
Consistency 86.4%
88.5%
88.7%
Table 11: The diversity, quality and consistency under different values of m.
The reward distribution of learned policies The reward distribution of the learned policies is presented in Table When taking larger m value, the learned policies still differs lot from each other. Some of them focus on shooting enemies, while others prefer collecting stars.
(m 3)
Policy
Policy 2
Policy 3
Reward (shoot)
0.0
3.3
2.7
Reward (star)
8.2
0.4
3.8
Winning rate
59%
36%
56%
(m
Policy
Policy
Policy 3
Policy
Reward (shoot)
1.3
2.2
0.2
3.1
Reward (star)
6.0
3.2
7.6
0.5
Winning
57%
57%
56%
40%
(m 5)
Policy 1
Policy 2
Policy 3
Policy 4
Policy 5
Reward (shoot)
1.1
2.6
2.9
0.4
1.5
Reward (star)
8.0
0.5
0.5
8.6
6.2
Winning rate
50%
38%
37%
65%
61%
Table 12: The reward distribution when m =3,4,5
J DISCUSSION ON POSTERIOR APPROXIMATION
In the main text, we approximate the true posterior $(2 il]) x S(a.jEr (als) with p(z
i/7) (10 because the consecutive multiplication leads to numerical instability.
We plot the distribution of p(2 = 1/7) for all trajectories the dataset in Figure 11 Figure Tla uses consecutive multiplication, while Figure 11b is the average of all steps. The results show that using
20

---

Published as a conference paper at ICLR 2024
consecutive multiplication causes trend of polarization in clustering, which causes unexpected numerical instability. The experiment was conducted in the grid shooting environment with the number of clusters set to = 2.
230-
433
332
200
5500 (h) The distribution of the approximated p(= 1/7), which is calculated by averag- ing the probability at each step.
5400
0.423
3:400
0.433
0323
5000
0375
0.00
(a) The distribution of the true posterior
p( = 1/7), which is calculated by consecu- tive multiplication of the probability each
step.
Figure 11: The histogram of the distribution of the posterior.
21


Reviews:
[{"review_id": "XqlyjYvfXr", "reviewer": "Reviewer_pT1t", "summary": "The paper addresses the problem of learning diverse policies based on datasets of trajectories collected by humans. This is particularly relevant in the context of video gaming, where the goal is to develop bots that are not only proficient but also exhibit varied behavioral patterns based on human player data. The authors introduce a purely offline solution that eliminates the need for environmental interaction. This approach is underpinned by a dual-step method. Initially, a clustering technique, leveraging the EM algorithm, assigns trajectories to different clusters by learning  a style-sensitive policy. Subsequently, to foster policies that are both effective and stylistically aligned, Advantage Weighted Regression (AWR) is employed in conjunction with a style-regularization component based on the style-sebsitive policies. The effectiveness of this method is demonstrated through a series of tests conducted in a simplistic environment, a handful of Atari games, and a commercial video game, all of which confirm the algorithm's capability to generate diverse and competent policies.", "questions": "(see previous section)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and will be of the interest of a large audience. The model is quite simple (clustering then offline learning) and easy to apply to different use-cases, it can be a good baseline for many future works.  More importantly, as far as I know, this paper is the first one to propose a set of experiments on a real video game and a large dataset of collected traces which is certainly where this paper has the most value and the dataset and  environment will be release (can you confirm I am right on that point ?)", "weaknesses": "The paper presents a compelling methodology, yet it notably omits a benchmark against \"robust imitation of diverse behaviors,\" which is a reference work within this domain. Although primarily an online training paper, like infoGAIL, its principles could potentially be adapted for offline training, serving as a relevant comparison.\n\nThere appears to be an implied relationship between what the authors denote as 'style' and the rewards associated with a particular trajectory. Commonly, one might categorize trajectories by skill level, segregating expert from intermediate or novice plays. However, in such a scenario, the operation of the Advantage Weighted Regression (AWR) on these distinct clusters is not thoroughly explained. The connection between the 'style' of play and the 'reward' outcome merits a deeper examination.\n\nThe simplicity of the clustering model raises concerns regarding its ability to discern more nuanced styles, such as specific repetitive actions (e.g., \"jump twice\"). A more critical discussion on the model's capacity to identify and differentiate between complex styles would enhance the paper.The algorithm seems limited in capturing policies that would need memory to characteriwe their styles.\n\nRegarding the implementation of AWR, it seems to be applied to each cluster individually. This approach suggests that in a situation where ten clusters are identified, only one-tenth of the training trajectories are utilized during the AWR phase for each cluster. This potentially limits the method's scalability when dealing with numerous styles, possibly making it impractical for extensive style differentiation.\n\nLastly, the paper could explore the potential of employing more advanced offline reinforcement learning algorithms in the second step of the methodology. Such a discussion could provide insights into improving the efficiency and effectiveness of the learning process in diversifying policies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n0d2ViUODr", "reviewer": "Reviewer_Xp9f", "summary": "This paper presents a new approach, Stylized Offline RL (SORL), which seeks to derive high-quality, stylistically diverse policies from offline datasets with distinct behavioral patterns. While most reinforcement learning (RL) methodologies prioritize either online interactions or policy performance, SORL combines the Expectation-Maximization (EM) algorithm with trajectory clustering and advantage-weighted style learning to promote policy diversification and performance enhancement. Through experiments, SORL has been shown to outperform previous methods in generating high-quality and diverse policies, with a notable application being in the basketball video game \"Dunk City Dynasty\". The effectiveness of SORL is evaluated in various settings, including a basketball video game. Compared to other methods, SORL consistently yields better-performing policies that also maintain distinct behavior patterns. The paper's contributions include:\n* The introduction of SORL, a framework that combines quality and diversity into the optimization objective, addressing limitations in both diverse RL and offline RL methods.\n* Extensive evaluations showing SORL's ability to generate high-quality, stylistically diverse policies from diverse offline datasets, including human-recorded data.", "questions": "1.\tThe metric for evaluating diversity seems to rely on the learned clustering p. I wonder why don’t evaluate the diversity of the learned policy? Otherwise, this metric cannot be used for algorithms that don’t learn the clustering of datasets. In addition, the goal of clustering the dataset is to learn diverse policies for online evaluation, so the diversity of the policy is what we really care about.\n2.\tCould the authors provide some visualization or example of the mean of the clusters in Atari games? It is not intuitive how the diverse behavior looks like in those games. Similarly, in the Dunk City Dynasty game, the visualization in Figure 3 is too simple. Could the authors plot the shooting positions of each policy? Also, besides the shooting position, are there other differences between these policies?\n3.\tCould the authors provide more details about the setting of the Dunk City Dynasty experiments? For example, the action space, and model structure. Appendix C only describes the structure of the first two experiments. \n4.\tWhat does it mean by “we share the main network and use a lora module to discriminate different styles.” In Appendix C?\n5.\tResults in Table 3 show that SORL has a very high variance (5.3 ± 3.4) in terms of quality, which only slightly outperforms InfoGAIL (5.0 ± 0.8). Does this mean pursuing high-quality sacrifices for the performance of the policy? Then, what do we gain from the high diversity?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe structure of the paper is clear and easy to understand. \n2.\tThe idea of using the EM framework to do trajectory clustering and policy optimization is interesting. \n3.\tUsing offline RL to solve real-world tasks using a human-generated dataset shows the scalability of the proposed method.", "weaknesses": "1.\tNo standard Offline RL baseline compared. As quality and diversity are both metrics for evaluation, it would be good to compare the performance with other standard offline RL methods, e.g., CQL, TD3+BC, AWR. \n2.\tThe motivation for increasing the diversity of policy is not clear. In the related work section, the authors only discuss the importance of diversity in online RL settings, for example, encouraging exploration, better opponent modeling, and skill discovery. However, in the offline RL setting, there is no exploration problem or skill discovery since the dataset is fixed. In addition, in the preliminary section, the authors aim to “learn a set of high-quality and diverse policies” without any explanation of the advantage of learning a set of diverse policies over a single policy with diverse behaviors (e.g., using multi-modal distribution as policy distribution).\n3.\tMany details are missing in the experiment of the “Dunk City Dynasty”. The code does not include this experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bM5aNPW6X8", "reviewer": "Reviewer_HX1W", "summary": "This paper discusses the problem of extracting diverse as well as high-quality policies from multi-modal datasets via offline reinforcement learning. The core of the proposed method lies in clustering trajectories within the dataset. Behavior policyies to induce such clusters are learned, which are later used for constraining policy learning to ensure that the offline RL policies are high-performing as well as aligning with the diverse multi-modal dataset. Extensive experiments are conducted, and results seem positive. But I still have some concerns for this paper, please refer to the weaknesses.", "questions": "Please refer to the weaknesses. If the authors can address my concerns, I'm happy to increase my score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is straightforward and easy to comprehend.\n2. Extensive experiments are conducted. Resutls on all three benchmarks show the proposed method SORL achieves balance between performance and diversity of the learned policies.\n3. Procedure of SORL is clearly described.", "weaknesses": "1. Transformation from the true posterior to Eq. 2 needs more explanation. The current context is too weak. And I assume the basic assumption for this transformation is that all behavior policies $\\mu_{1,..,m}$ are diverse enough, because the authors use transtion-wise action probability to replace the trajectory probability. This makes sense if behavior policies are diverse enough that they take different actions for each step. But if the policies only slightly differ from each other, the consecutive multiplication of all steps within the trajectory will make their trajectory distributions very different from each other (while the action distribution is not much different). As a result, Eq. 2 provides very inaccurate estimation of the posterior.\n2. The proposed SORL needs to know the number $m$ of policy primitives constituting the dataset in order to learn the diverse policies. But it is hard to know this prior under many ciucumstances. I think there should be a study about how sensitive SORL is  to this hyperparameter.\n3. How diverse are the induced policies? The case studies are great but there should be a quantitive study. My further question on this is, if we set $m$ larger than the actual number of policy primitives $\\mu_{1,..,m}$, what will the resulted policies be like?\n4. Some typos, e.g. line 10 in Algorithem 1: $\\mu^{i}$ instead of $\\mu^{1}$\n5. As the author claims the induced policies are high-performing, the baselines should include some strong offline RL methods for comparison. This will also show SORL's advantage in policy diversity compared to them. The current baselines are too few.\n6. How do the authors collect online user data? Where is the user agreement to collect this data? This should appear at least in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7FGEtuhj3m", "reviewer": "Reviewer_BZGK", "summary": "This paper introduces an EM-inspired algorithm called Stylized Offline RL (SORL) to extract diverse strategies from heterogeneous offline RL datasets. Based on the learned behavior policies, the paper then applies an advantage-weighted style learning algorithm to improve their performance further. The authors demonstrated their algorithm's effectiveness with experiments on six Atari games and one online mobile basketball game, where SORL outperforms other baselines regarding quality, diversity, and consistency.", "questions": "1. §5.3 states that the character has to combat against an AI opponent. What is the strategy of the opponent? Is SORL robust to the changes in the opponent's strategy?\n\n2. All experiments were conducted on environments with discrete action spaces. How does SORL perform in continuous action environments?\n\n3. How was the diversity metric measured for InfoGAIL? To my knowledge, InfoGAIL does not explicitly split the policy into multiple clusters but instead learns a multi-modal policy.\n\n4. In Appendix C, the paper states that\n\n   > Besides, in order to ensure balanced learning among all the styles, we share the main network and use a lora module to discriminate different styles.\n\n   The explanation on the \"lora\" module seems missing. Also, I recommend moving this part to the main paper.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Extracting diverse behaviors from an offline RL dataset is an interesting problem. SORL efficiently solves the problem following an EM-based approach. The proposed evaluation criteria, considering quality, diversity, and consistency, provide a nice guideline for other researchers to follow. The algorithm also performs better than other existing baselines in multiple offline RL datasets, including the \"Dunk City Diversity\" dataset, which contains extremely diverse behaviors. Finally, the paper is overall well-written and easy to understand.", "weaknesses": "1. There is no theoretical ground for naively replacing $A^{\\mu^{(i)}}$ with $A^\\mu$ without importance sampling corrections. At least an empirical ablation study should be provided if it is difficult to devise a theoretical justification.\n\n2. It isn't easy to understand the proof presented in Appendix B.\n\n    (1) $\\pi^{(i)}$ needs to satisfy the constraint $\\int \\pi^{(i)}(a\\mid s)\\,da=1$ for all $s$. The optimal solution might not be a critical point.\n\n    (2) $A^\\mu(s, a)-\\lambda \\mu^{(i)}(a\\mid s)+\\lambda \\pi^{(i)}(a\\mid s)+\\lambda=0$ does not imply $\\pi^{(i)*}(a\\mid s)\\propto \\mu^{(i)}(a\\mid s)\\exp(\\frac{1}{\\lambda}A^\\mu(s, a))$.\n\n    (3) The normalization constant for $\\pi^{(i)*}$ is ignored in (14).\n\n3. The diversity metric proposed by the authors does not consider how different the styles are. For example, consider the case where $\\pi^{(i)}(a=k\\mid s)=\\frac{1}{K}+\\epsilon_k(s)$ where $K$ is the number of possible actions and $\\epsilon_k(s)$ is a small number chosen arbitrarily. Then $\\hat{p}(z=j\\mid traj)$ would be determined by the values of $\\epsilon_k(s)$, so $p_{popularity}$ would be close to a uniform distribution, which is the distribution that maximizes the entropy. However, the learned styles are far from being diverse.\n\n### Minor comments:\n\n1. How about using $\\tau$ instead of $traj$? I think this notation is widely accepted.\n\n2. The $-$ sign seems missing in (9).\n\n3. §5.1 Off-RLMPP → Off-RLPMM (appears twice on the fourth line)\n\n4. Appendix B: Unmatched parentheses $($ in (13) and on the first line of p.15\n\n5. Appendix B: $exp$ → $\\exp$ on the second line of p.15 and in the second and third equation of (14)\n\n6. Appendix B: In (14), $t$ is the index of summation, but it does not appear in the summand.\n\n7. Appendix B: Second $=$ → $\\approx$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)