Ess-InfoGAIL: Semi-supervised Imitation Learning from Imbalanced Demonstrations
Authors: Huiqiao Fu
Kaiqiang Tang
Yuanyang Lu
Yiming Qi
Guizhou Deng
Flood Sung
Chunlin Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Generative adversarial imitation learning
semi-supervised learning
multi-modal behaviors
imbalanced data
TL;DR: We enhance InfoGAIL by introducing a semi-supervised architecture that specifically tackles the challenge of learning multi-modal behaviors within imbalanced demonstrations.

Abstract: Imitation learning aims to reproduce expert behaviors without relying on an explicit reward signal. However, real-world demonstrations often present challenges, such as multi-modal, data imbalance, and expensive labeling processes. In this work, we propose a novel semi-supervised imitation learning architecture that learns disentangled behavior representations from imbalanced demonstrations using limited labeled data. Specifically, our method consists of three key components. First, we adapt the concept of semi-supervised generative adversarial networks to the imitation learning context. Second, we employ a learnable latent distribution to align the generated and expert data distributions. Finally, we utilize a regularized information maximization approach in conjunction with an approximate label prior to further improve the semi-supervised learning performance. Experimental results demonstrate the efficiency of our method in learning multi-modal behaviors from imbalanced demonstrations compared to baseline methods.

Ess-InfoGAIL: Semi-supervised Imitation Learning from Imbalanced Demonstrations
Huiqiao Fu', Kaigiang Tang', Yuanyang Lu', Yiming Qi', Guizhou Deng',
Flood Sung", Chunlin Chen
Nanjing University, China, Moonshot AI, China [hqfu, kqtang, yylu, ymqi) @ @mmail.jj..edu.co (guizhoudeng, floodsung) | @gmail.com, cichen@nju.ede @jj..ed..cn
Abstract
Imitation learning aims to reproduce expert behaviors without relying on an explicit reward signal. However, real-world demonstrations often present challenges, such as multi mouti-modal, data imbalance, and expensive labeling processes. In this work, propose novel semi-supervisct imitation learning architecture that learns dis- entangled behavior representations from imbalanced demonstrations using limited labeled data. Specifically, method consists of three key components. First, we adapt the concept of semi-supervised generative adversarial networks to the imitation learning context. Second, employ learnable latent distribution align the generated and expert data distributions. Finally, we utilize a regularized information maximization approach in conjunction with an approximate label prior to further improve the semi-supervised learning performance. Experimental results demonstrate the efficiency of our method in learning multi-modal behaviors from imbalanced demonstrations compared to baseline methods.
Introduction
The recent success of Reinforcement Learning (RL) has shown great potential in solving complex decision-making tasks 1 2 3) A key prerequisite for RL. to design a reward function, which can be challenging in many real-world applications 5] By contrast, Imitation Learning (II.) methods, such as Behavior Cloning (BC) [6], Inverse Reinforcement Learning (IRL. 7 and Generative Adversarial Imitation Learning (GAIL) [8], can reproduce expert behaviors without relying an explicit reward signal. Among them, GAIL performs more effective learning process by learning a
policy directly from demonstrations, and has been widely applied in various fields 9 TO [2].
Real-world demonstrations are typically coupled by multi-modee behaviors. pretable and meaningful behavior representations, InfoGAIL [13] utilizes the concept of maximizing mutual information between discrete continuous latent variables to generate corresponding samples in an unsupervised manner. However, unsupervised learning of desired decodable factors from unstructured data can be challenging, particularly in the case of behavior imitation, where commonly used methods such as contrastive learning often underperform. Recent researches have uncovered that unsupervised learning of disentangled representations is inherently difficult without the presence of inductive biases on both the models and data [14). Additionally, real-world demonstrations often consist of imbalanced behavior modes. For example, walking constitutes the majority of human behavior, whereas jumping only represents small fraction. It is difficult for InfoGAIL to learn disentangled behavior representations from imbalanced demonstrations, as shown Fig. [ One intuitive solution is to introduce supervised learning with fully labeled data [15]. However, manually segmenting and labeling from the raw demonstrations can be a costly endeavor.
disentangle inter-
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

8
8
(a) Expert
(b) GAIL
(c) InfoGAIL
(d) Ess-InfoGAIL
Figure 1: 2D trajectory environment. An agent can move freely within plane, seeking to mimic expert trajectories, each represented by a different color to indicate a distinct mode of expert behavior. (a) The imbalanced expert demonstrations, where the red and green trajectories are predominant, while the blue and purple trajectories only represent a small portion. (b) Trajectories generated by GAIL. (c) Trajectories generated by InfoGAIL. (d) Trajectories generated by the proposed Ess- InfoGAIL. Our method is able to learn high quality disentangled behavior modes from the imbalanced expert demonstrations.
To tackle these challenges, we introduce novel cuni--uperviied imitation learning architecture that can acquire disentangled behavior representations from imbalanced demonstrations, guided by only limited amount of labeled data (as little as 0.5% to maximum of 2% of the dataset). Our model, Elastic semi-supervised InfoGAIL (Ess-InfoGAIL), makes three key improvements to the related InfoGAIL. First, we draw inspiration from InfoGNN [16] and focus on learning meaningful and controllable behavior representations by maximizing two mutual information terms: i) The mutual information between a latent skill variable, drawn from a categorical distribution, and the labeled expert state-action transitions, ii) and the mutual information between latent shifting variable, drawn from a continuous uniform distribution, and the state-action transitions produced by a policy. The former term enforces that the latent skill variable corresponds to label categories in semi- supervise manner, while the latter allows for style shifting within a given skill, Second, we tackle the issue of imbalanced data and align the distribution of generated transitions with that of expert demonstrations by utilizing a latent skill variable drawn from a differentiable Gumbel-Softmax distribution [17], instead of using a latent skill variable drawn from fixed categorical distribution. Third, to make use of the intrinsic information in the unlabeled imbalanced data and improve the efficiency of the remi-supervised learning, we leverage the Regularized Information Maximization (RIM) [18] with label prior that approximates the learned latent skill distribution.
In the experiment, first validated method in simple 2D trajectory environment to obtain more intuitive visualization, as shown in Fig. []] and subsequently tested our method in 4 challenging MuJoCo environments (Reacher, Pusher, Walker-2D, and Humanoid). The experimental results illustrate the efficiency of our method disentangling multi-mode behaviors from imbalanced demonstrations compared baseline methods. To the best of our knowledge, tackle the problem of learning disentangled behavior representations from imbalanced demonstrations, with the aid of only a limited amount of labeled data. The code is available at htttss://gith:: com/ com/ t//AA//NAsOssss InfoGAIL In particular, the contributions of this work are threefold:
work is the first to
1. We extend the InfoGAIL with supervuper imitation learning architecture, resulting in Ess- InfoGAIL,, which enables the learning of desired disentangled behavior representations.
2. We utilize a learnable latent distribution to align the generated and expert distributions in the imbalanced data setting, and leverage RIM with an approximate label prior improve the semi-supervised learning.
3. We demonstrate the efficiency of our method in 4 challenging MuJoCo environments, showcasing that even in scenarios with highly imbalanced demonstrations, the policy can still reproduce the desired behavior modes.
2 Related Work
Generative adversarial imitation learning GAIL, adapts techniques developed for Generative Adversarial Networks (GANs) [1), and has demonstrated promising results in challenging imitation

---

learning tasks, such as robot control [10], imitation from raw visual data 200, ageet gaming [ text generation (T], and driver behavior imitation [22]. Subsequent research has proposed various improvements to GAIL. 23 24 resulting in more efficient and reliable imitation results. However, discovering rich disentangled factors from an unstructured dataset using GAIL. can be challenging due to the absence of inductive biases on the data or model. One promising approach to alleviate this issue is to introduce a latent variable model and use the technique of maximizing the mutual information [25] between the latent variable and the data, which can enable effective disentanglement of behaviors.
Learning multi-modal behaviors Tasks can often be accomplished in multiple ways, and the expert may exhibit different modes of behavior [26 27 Therefore, it is crucial to extend GAIL to learn and replicate these multi-modal behaviors. One intuitive approach is to train separate auxiliary classifier allowing label- discriminators for each labeled mode [28). Or can [15]. These supervised approaches have conditional imitation learning about multiple intentions
demonstrated their effectiveness and reliability in various applications, from controlling wheeled robots (28] to task allocation in taxi companies [29] However, they are constrained by their reliance on labeled data. To overcome this limitation, recent research has focused on unsupervised methods that typically leverage mutual information between latent variables and the generated data (30 12] For example. InfoGAIL maximizes this mutual information to learn from visual demonstrations 13]. while Burn-InfoGAIL maximizes it from the perspective of Bayesian inference to draw modal variables from burn-in demonstrations BT. However, unsupervised methods often struggle to distinguish latent factors without considering semantic information or task context [32], and proved unreliable in theory [14. Therefore, we propose novel semi-superyie approach that leverages the advantages of both supervised and unsupervised methods to learn semantically interpretable behavior representations with a small portion of labeled data.
Learning from imbalanced data Real-world data often exhibits imbalanced distributions, which can impede learning by causing the model to become biased towards the dominant category and potentially overlook the minority category. In supervised learning setting, the problem of imbalanced data can be mitigated through various techniques such as data re-sampling and category re-weighting These methods are useful when the category distributions are known beforehand. Several methods have been proposed for the unsupervised setting to deal with the imbalanced data with unknown category distributions [35] 36 37). A recent approach called Elastic-InfoGAN (38) has been proposed to address the issue of distribution alignment in GANs by utilizing a learnable latent distribution to handle imbalanced data. Unlike these methods, our work focuses on multi-modal semi-supers supervised imitation learning from raw imbalanced data without relying on prior knowledge of category distributions.
3 34
3 Approach
In this part, we first provide brief background on RL, GAIL, and InfoGAIL, then provide a detailed description of how to extend InfoGAIL to a semi-supervised architecture and address the disentangled behavior learning using both the imbalanced unlabeled expert demonstrations and limited labeled expert demonstrations.
3.1 Background
Reinforcement learning We consider a standard Markov Decision Process (MDP) setting ① rep- resented by tuple AS R, 7). where S is the state space, is the action space, P (St+ [Sr, ax) is the transition probability of state S at time step / + given state S, and action a, at time step R(s, a) is the reward function, and 7 E (0, 1) is the discount factor. In RL process, an agent interacts with the environment and learns a policy π (a|s) to maximize the expected discounted return Ep(ri=)
ETD R'R a.) where p(T(T) = ((5)) IT p (s |s,, a.) π (Ta (s:) represents the likelihood of trajectory T = (so, ag, ro, Si..... 8T ar PT- S7 under æ. The reward func- tion serves as an interface through which users can specify the task that an agent needs to perform. However, creating effective reward functions that elicit the desired behaviors from an agent can often
be a laborious and challenging process.
3

---

GAIL and InfoGAIL As one of the apprenticeship learning method, GAIL adapts techniques developed for GANs to directly learn a stylized policy through expert demonstrations without the need for handcrafted rewards. The formal GAIL. objective is denoted as
min max VGAIL D) = Edd flg ((s, a)] (d [log(1 ((s, a))] (T))
(I)
D
where D is discriminator which tries to distinguish state-action pairs (s, a) drawn from the gener- ated state- satte-action distribution d= (s. a). induced by т, and the state-action distribution of the expert demonstration d (s, a). (T) is the causal entropy of the policy The GAIL objective is optimized alternately by taking gradient step to increase Equation with respect to D and an RL step to decrease Equation 1 with respect
The Lensen-Shannon divergence between dl (s, a) and d a)
eventually minimized [39].
GAIL may encounter difficulties in learning multi-modal behaviors from unstructured demonstrations the absence of meaningful representations. To overcome this limitation, InfoGAIL introduces an additional regularization term to maximize the mutual information [(c; a) between the latent variable e and state-action transitions (s, a), and the objective is then approximated by a variational lower bound [25]:
(T, T Q) = dr [log Q(cls, a)] + ((c) < / (c)
where Q(cls, a) is approximation of the true posterior P(cls, a). The lower bound is tight if Q = P. This regularization allows InfoGAlL. infer the interpretable latent structure in unsupervised manner, and the final objective becomes:
min max VIntoGAn. (,, = VGAA. D) XIL((T
(3)
x D
where is a weighting coefficient.
3.2 Semi-supervised Imitation Learning from Imbalanced Demonstrations
Datasct
D
p(c)
Env.
a
Doody
D
Q
c ((c)
Q
Figure 2: Schematic overview of the Ess- Ess-InfoGAIL network architecture. Here, represents latent skill variable sampled from the differentiable Gumbel-Softmax distribution, e denotes latent shifting variable sampled from continuous uniform distribution, and y signifies the category label. Doody serves as shared backbone network for extracting features from state-action pairs. Parameters 0, Φ. and vi correspond the policy ma, the discriminator D&, and the encoders Q Q respectively.
As InfoGAIL is learned in an unsupervised manner, obtaining the desired disentangled behavior representations can prove to be challenging. The difficulty is compounded when the expert demon- strations are imbalanced, as shown in Fig. To address this, we propose Elastic semi-supervised InfoGAlL. (Ess-InfoGAIIL with three improvements to InfoGAlL:: i) A semi-supervised learning architecture, ii) learnable latent skill distribution and iii) RIM with an approximate label prior. The network architecture of Ess- Ess-InfoGAIL. is shown in Fig. 2
Semi-supervised InfoGAIL utilize both labeled and unlabeled data, we decompose the latent variable into a semi-supervised part e and an unsupervised part inspired by ss-InfoGAN [16]. Unlike ss-InfoGAN, here we focus on imitation learning under sequential decision-making tasks. Additionally,
latent skill variable which is sampled from categorical distribution.

---

allowing for encoding the same information as the label y. and use e as a latent shifting variable which is sampled from continuous uniform distribution, allowing for style shifting within given skill.
Then seek maximize two mutual information terms s(; a) and T(c; a), where the first term can be approximated using Equation 2 with a posterior approximation (e(s,,)) a), and the second term can be approximated using the following variational lower bounds:
Lis(O)) = all)) Dog Q (v/s, a)] + (v) S I(y: s, a).
(4)
(5) where, Equation is a supervised term that uses the state-action distribution of the labeled expert demonstration d == (s, a, y) to predict y, and Equation 5 an unsupervised term that utilizes the acte-actionn distribution produced by a policy me to learn the inherent semantic meaning of y. The technique of lower bounding mutual information is known as variational information maximization With Q Q Qu obtain the semi-supervised regularization term:
Lis (T, 2%) = Epte).d (s.ale) (Log 2 (c|s, a)] + H(c) - ((c a),
25
(T Q+)) Lis (Q.) + Lis(To (T),
(6)
where, is is optimized with respect to Q and Lis is optimized with respect to Te. Specifically, Lis trained as a classification process with the cross-entropy function, and L in Equation 2 is trained as regression process with the mean absolute error.
Learnable latent skill distribution The majority of GAN and GAIL algorithms assume that the latent variable is drawn from uniform distribution with fixed parameters. However, in the presence of imbalanced data, this assumption may lead to suboptimal optimization results. To align the state-action distribution produced by a policy Ty with the state-action distribution of the imbalanced expert demonstration, we utilize latent skill variable c drawn from a differentiable distribution p(c) (17), similar to
[8). The Gumbel- -Sfftexx distribution is continuous distribution over the simplex that can approximate samples from a categorical distribution
in differentiable way, and this trick has been widely used in many areas 44 41
Let P2, ... PK be the class probabilities of a categorical distribution, the sampling of a K- dimensional latent skill variable can be done
exp (log (p.) + /T)
C, = Σドー exp (log (p,) + +;) /T)
for i = I,..., K,
where 91 'KK are i.idd samples drawn from Gumbel(O, 1). The coefficient T is a temperature parameter that determines the degree of smoothing applied to the probability distribution. A smaller value of r leads to a distribution that is closer to the one-hot encoding, while larger value of results in a distribution that closer to the categorical distribution.
Since the objective of GAIL is to minimize the dL (s, a), we can eventually estimate the label distribution of the unlabeled expert demonstrations by passing the gradient back to the latent skill distribution p(c) using Gumbel-Softmax reparameteriza- tion trick.
divergence between d' (s, a) and
RIM with an approximate label prior Discriminative clustering techniques can automatically identify the boundaries or distinctions between categories in unlabeled data [42]. To leverage the intrinsic information in the unlabeled imbalanced demonstrations and enhance the efficiency of the semi-supervised learning process in Equation 6
we employ Regularized Information Maximization
(RIM) [8] with a label prior that approximates the learned latent skill distribution:
LRIM(Q) = - Σ H (Q2 - DKL. (Q9())lp(c)) - B(V),
(8)
where the first term the cluster assumption 43
corresponding to the idea that datapoints should be classified with large margin. The second term is performed to avoid degenerate solutions by minimizing the KL divergence between the empirical label distribution (0) ao Qc a.) and the learned latent skill distribution p(c), while the last term is a parameter regularization (e.g.. L2 regularization) performed to avoid complex solutions.
5

---

3.3 Overall Algorithm
With the above three improvements, the objective function of Iss-InfoGAIL eventually becomes:
min maaxVEs-InfGGlll Do, Ve P(c))
No. Qy p(c) D
(9)
VimooGAL(TT Do p(c)) >2 (To, Q p(c)) AIL ((c)))
where A2 and A3 are two weighting coefficients. 0, vh are parameters for the policy no. the discriminator Dz. and the encoders Q and Q respectively. We optimize Da. Q% and Q with the stochastic gradient method. To is optimized using Proximal Policy Optimization (PPO) 44 with advantages computed using GAE(A) [45, and the value function is updated using (A) I. The latent skill variable concatenated with the state, serves as input to the policy, where is sampled from the learnable latent skill distribution p(c) which updated along with the policy. The schematic overview of the Ess-InfoGAIL network architecture is shown in Fig. 2 and the pseudo code is shown in Algorithm[] [ the experiments, we show that the proposed Ins-Inf can learn disentangled behavior representations efficiently from imbalanced demonstrations using limited labeled data.
Algorithm 1: Ess-InfoGAIL
Input: Unlabeled and labeled demonstrations ds. a), dEL (s, a, y), initial parameters of the policy, value function, discriminator and encoders Oo, Bo, do, vo, initial latent skill distribution po(c), weighting coefficients d1, 22, A3
Output: Learned policy TT-
1 for i
2, do Sample batch of latent variables ~ p(e),
3
Interact with the environment using Te, and c, and obtain d' (s,
Sample batch of state-action pairs 5 ~ d=, b ~ d JEL ~ dEL Update de 0i+1 by ascending with gradients according to VGALL Update di; to 1h441 by descending with gradients according to Ves-InfoGAA. Update O,, P:(c) P (c) by taking policy step using the PPO update rule
4
6
according to VEss-IifoGAIL
Update the value function using (A)
end
Experiments
In this section, we validate our method by conducting experiments in variety of environments, in- cluding a simple 2D trajectory environment, as well as 4 challenging MuJoCo environments (Reacher, Pusher, Walker-2D, and Humanoid). First, perform the quantitative analysis demonstrate the superiority of our method in discovering disentangled behavior representations from imbalanced demonstrations with limited labeled data, compared to baseline methods. Second, analyze the amount of labeled data required for the model to effectively encode the semantic meaning of the labels. Finally, we analyze the effect of varying the degree of data imbalance and the number of behavior modes on the imitation of multi-modal behaviors.
Task setup (1) 2D trajectory environment (Fig. is plane where the agent can move freely a constant velocity by selecting its direction p, at discrete time t. The agent's observations at time t consist of positions from 4 t. The expert demonstrations consist of 4 distinct modes, each produced by stochastic expert policy that generates circular trajectory. The setting is similar to the experiment in [ (2) Reacher environment (Fig. Left) consists a 2-DoF robot arm. The goal is to move the robot's 's end effector close to a target position. The original environment is modified by setting evenly distributed targets on circle with a radius of 1.5, and the target position removed from the observation space. The initial angles of the two joints are randomly sampled between [-n, π| and |- //2 2, π 1/2], respectively. (3) Pusher environment (Fig. Right) consists a 4-DoF robot arm. The goal is to move a target cylinder to goal position using the robot's end effector. Similar to Reacher, we use evenly distributed goal positions and remove the goal position from the observation
space. (4) Walker-2D environment (Fig. 4 Left) is 6-DoF bipedal robot consisting of two legs

---

7
Figure 3: Left: Reacher with 6 targets: random initial state, reaching one target, reaching another target. Right Pusher with 6 goal positions: random initial state, reaching one goal position, reaching another goal position.
Figure 4: Left: Walker-2D with 3 goals: running forwards, running backwards, balancing. Right: Humanoid with 3 goals: running forwards, running backwards, balancing.
and feet attached to a common base. The goal to make coordinate both sets of legs and feet to move the robot in the right direction, including rumning forward, running backward and balancing. (5) Humanoid environment (Fig. Right) is a high-dimensional robot with 17 degrees of freedom. The goal is similar to the Walker- 2D, including running forward, running backward and balancing.
Multi-modal demonstrations preparation In the experiments, we first pre-train expert policies, each corresponding K different goals (or K behavior modes). Subsequently, policies to sample sets of expert demonstrations. From each set of expert demonstrations, we extract small portion and label them with the one-hot behavioral categories, while the remaining expert demonstrations are randomly sampled and mixed to create imbalanced unlabeled expert data. Moreover, in the real-world scenario, one can directly use the raw motion capture data (e.g., motion capture data of an animal over a day) without the need to train additional expert policies.
these K expert
Baselines and evaluation metries We design various baselines to demonstrate the efficiency of our method in learning multi-modal behaviors from imbalanced demonstrations. (1) GAII:: The original GAIL. (2) InfoGAIL: The original InfoGAll. with fixed uniform categorical distribution. (3) ACGAIL: Reproduction of ACGAIL [15] using an auxiliary classifier with full labeled data. (4) Gaaltie-I Modification of Elastic-InfoGAN [38] imitation tasks with a learnable latent distribution and contrastive loss. (5) Ess-InfoGAIL GS: Ess-InfoGAIL without using the Gumbel-Softmax technique. (6) Ess-InfoGAIL RIM: InfoGAnG without using the RIM technique. (7) Ess-InfoGAIL: Our final method.
To evaluate the behavior disentanglement quality of the above methods, we use pre-trained behavior classifier and employ two commonly used metrics in clustering algorithms similar to [38]: (a)
Average Entropy (ENT) evaluates two properties: whether state-action pairs generated for given latent variable belong to the same ground-truth class and ii) whether each ground-truth class associated with single unique latent variable. (b) Normalized Mutual Information (NMT) quantifies the correlation between two clusterings, with value ranging from 0 to 1. A higher NMI value indicates a stronger correlation between the clusterings.
Additionally, we use the normalized average task reward to assess the quality of learned policies. Note that, we modify the original MuJoCo environment and compute the task reward differently for each behavior mode. The final task reward, derived by averaging across all modes, serves exclusively as an evaluation metric and does not affect the policy training process.
4.1 Comparison with Baseline Methods
We quantitatively analyze Ess-InfoGAlLLs behavior disentanglement performance, as presented Table I complex tasks with numerous behavior modes, both GAIL and InfoGAIL often encounter mode collapse, where their policies manifest behavior corresponding to only subset of the behavior modes due to lack of prior behavior representation knowledge. For instance, in the Reacher environment, the robot arm reaches only few target positions. Leveraging limited labeling
7

---

Table 1: Behavior disentanglement quality measured by NMI (1) and ENT (L). Where Ess- InfoGAIL AS removes in Ass-InfoAALL For each experiment, we use 10 different random seeds and collect 50 episodes for each seed. The results demonstrate the efficiency of Ess-InfoGAIL in learning multi-modal behaviors compared to baseline methods. (See supplementary for error bars.)
in Ess-InfoGAIL, and Ess-InfoGAIL ARMM RIM removes RIM
2D trajectory
Reacher
Pusher
Walker-2D
Humanoid
Metrics
NMI
ENT
NMI
HNT
NMI
ENT
NMI
ENT
NMI
ENT
GAIL
8
0.392
0.529
0.153
1.055
0.376
0.723
0.439
0.406
0.388
0.597
InfoGAIL
13
0.742
0.371
0.301
1.113
0.604
0,551
0.657
0.284
0,550
0.487
ACGAIL 15
0.783
0.324
0.537
0.781
0.754
0.409
0.658
0.340
0.544
0.478
Elastic-InfoGAlL.
0.773
0.330
0.311
1.101
0,650
0.537
0.615
0.351
0.503
0.498
Eas-InfoCAALL.
0.892
0.157
0.607
0.661
0.857
0.194
0.740
0.262
0.638
0.360
Ess Iss-nnoGAIILRI
0.893
0.159
0.575
0.725
0.875
0.188
0.715
0.274
0.642
0.347
Ess-InfoGAIL (Ours)
0.910
0.131
0.662
0.587
0.906
0.144
0.755
0.237
0.696
0.206
guidance, our approach, Ess- InfoGAL, improves the average NMI by 0.436 and 0.215 and decreases the average ENT by 0.401 and 0.300 compared to GAIL and InfoGAIL, respectively. across all environments. Notably, Ess-InfoGAll. attains superior behavior disentanglement quality with fewer labels than ACGAIL, which utilizes full labeled data. This success might stem from imbalanced data hindering ACGAIL's auxiliary classifier learning, while our method balances the small number of labels by extracting equal-duration data for each behavior mode.
Furthermore, we adapt Elastic-InfoGAN to the imitation learning framework using Humbel--offmaal and contrastive learning loss, augmenting negative samples with Gaussian noise. However, contrastive learning's 's efficacy is limited visual-free imitation tasks, and indirectly addresses behavior disen- tanglement, yielding uncertain performance. In contrast, Ess- InsonnGGALL directly tackles mode collapse and achieves semantically interpretable behavior disentang glemmnt via limited labels. exhibits an average NMI improvement of 0.215 and an average ENT decrease of 0.302 over across all environments. We perform ablation experiments, confirming the enhancements brought by Gumbel-Softmax and RIM techniques under the semi-supervised framework and imbalanced data. Ess-InfoGAIL demonstrates an average NMI increase of 0.039 and 0.046 and an average ENT reduction of 0.066 and 0.078 compared to Ess-InfoGAIL GS and Ess-InfoGAIL RIM, respectively.
To assess the quality of learned policies, we analyze the normalized average task rewards within the Reacher environment, depicted in Fig. 5 demonstrating the superiority of our method over all baseline methods. Results for different environments can be found in the supplementary material. Due to mode collapse and data imbalance, some methods (e.g., GAIL and InfoGAIL) may end up learning only one or two behavior modes, leading to lower average task rewards compared to a random policy.
4.2 Amount of Labeled Data Requirements
In real-world scenarios, obtaining labeled behavior data (e.g., from motion capture or raw video) is difficult due to the interleaving of various behavior modes. Thus, minimizing labeling costs is
1.0
Nornalioed Average Task Reward
0.9
0.8
0.8
8.6
GAIL
Ascinay
InfoCIATL
0.7
ACGAIL
0.1%
0.4
-.m.
Elastic InfoGAIL Ess-infoAAALGS
0.6
0.5%
1%
02
Es--hnloAATURIM
0.
10%
Fa--IiiGAAlI (Oun)
100%
0.3
0.5
1.0
2.0
0.4
0
Soaps
106
Steps
Figure 5: Normalized average task reward of each
Figure 6: Classification accuracy using dif- ferent amount of labeled data.
method during training (only for evaluation).
8

---

crucial. We assess the prediction accuracy of encoders trained with limited labeled data across varying proportions within all unlabeled data. Results are shown in Fig. (using Reacher environment as an example). Proportions of labeled data are set at 0.1 %, 0.5%, 1,, 10%, and 100%. For Reacher, 0.1% data equates to 2 episodes (100 time steps) per mode. The encoder demonstrates nearly 92% prediction accuracy with just 0.5% 5% labeled data. Walker-2D and Humanoid tasks, with higher state dimensions, require more labeled data for effective behavior mode separation. Default label ratios, not specified, are: 2D-Trajectory: I 1, Reacher: 0.5%, Pusher: 1%, Walker-2D: 1%, Humanoid: 2%. Guided by limited labeled data, the agent adeptly disentangles specific styles from labeled data and more varied behavioral styles from unlabeled data.
Table 2: Degree of data imbalance.
Table 3: Learning more behavior modes.
InfoGAIL
Ess-InfoGAIL
InfoGAIL
Ess-InfoGAlL
Metrics
NMI
ENT
NMI
ENT
Metrics
NMI
ENT
NMI
ENT
20
0.328
1.035
0.704
0.510
2 0.329
1.031
0.711
0.504
40
0.317
1.050
0.693
0.521
4
0.319
1.073
0.703
0.522
60
0.313
1.061
0.687
0.583
6
0.301
1.113
0.662
0,587
80
0.306
1.073
0.681
0.561
8
0.308
L.087
0.651
0.619
100
0.301
1.113
0.662
0.587
10
0.303
1.076
0.617
0.657
200
0.291
1.121
0.625
0.607
12
0.298
1.115
0.613
0.653
4.3
Degree of Data Imbalance
The degree of data imbalance directly affects the learning quality of the policy, discriminator, and encoder. We set 6 levels of data imbalance in the Reacher environment, ranging from 20 to 200. For example, a data imbalance level of 20, the majority class has at most 20 times the number of samples as the minority class. The proportions of labeled data is set to 0.5%. At the maximum data imbalance level of 200, the minority class has only about 2 episodes of trajectories for each mode. The experimental results are shown in Table 2 It can be observed that as the degree of data imbalance increases, the NMI gradually decreases while the ENT gradually increases. However, even when the degree of the data imbalance is 200, the NMI of Ess-InfoGAll. only decreases by 0.079 compared to the case of a data imbalance of 20, and the performance is still better than InfoGAll. with data imbalance of 20. Our proposed method demonstrates the ability achieve effective disentanglement of behaviors, even in the presence of highly imbalanced data distributions.
4.4 Learning More Behavior Modes
The disentanglement task becomes increasingly challenging with increase in the number of behavior modes. We set up 6 levels of behavior modes ranging from 2 to 12 in the Reacher environment, where the agent needs to learn disentangled behavior representations from these unstructured behavior data. We maintain a fixed degree of data imbalance at 100 and the proportions of labeled data is set to 0.5%. The experimental results are presented in Table 3 The results demonstrate that even with 12 behavior modes, our method consistently achieves higher NMI values and lower ENT values, compared InfoGAlI. with only behavior modes. This affirms the controllability and scalability of our method in learning disentangled behavior representations while preserving category semantic information.
5 Conclusion
In this work, we proposed a novel semi-supervised imitation learning method, Ess-InfoGAIL, which utilizes a learnable latent distribution and an improved RIM to enable the agent to learn better disentangled multi-modal behaviors from unstructured expert demonstrations. We qualitatively and quantitatively validated the efficiency of our method, compared with other baseline methods. We also demonstrated that our method only requires a very small amount of labeled data to achieve good results, and further verified its controllability and scalability in scenarios with higher imbalance degrees and more behavior modes, which are crucial for multi-modal imitation tasks in the real world. Although there are some limitations, such as the need to provide small amount of labeled data for each category and preset the number of modal categories, we believe that our work lays the foundation for imitation learning in large-scale unstructured demonstrations.

---

Acknowledgments
This work was supported part by the National Natural Science Foundation of China (No. 62073160).
References
[1] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[2] Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62)seabk2822, 2022.
[3] Huiqiao Fu, Kaiqiang Tang, Peng Li, Wenqi Zhang, Xinpeng Wang, Guizhou Deng, Tao Wang, and Chunlin Chen. Deep reinforcement learning for multi-contact motion planning hexapod robots. In International Joint Conferences on Artificial Intelligence, pages 2381-2388, 2021.
[4] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transfor- mations: Theory and application to reward shaping. In International Conference on Machine Learning, volume 3, pages 278-287. 1999.
[5] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence, page 3207-3214, 2018.
[6] Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.
[7] Brian D) Ziebart, Andrew I. Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. AAAI Conference on Artificial Intelligence, volume 8, pages 1433-1453 1438, 2008.
[8] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, volume 29, pages 4565-4573, 2016.
[9] Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adver- sarial imitation learning. In Advances in Neural Information Processing Systems, volume 31, pages 7461-7472, 2018.
[10] Xue Bin Peng. Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (TOG), 40(4):1-20 2021.
[11] Alejandro Escontrela, Xue Bin Peng, Wenhao Yu, Tingnan Zhang, Atil Iscen, Ken Goldberg, and Pieter Abbeel. Adversarial motion priors make good substitutes for complex reward functions. In IEEE/RSJ International Conference Intelligent Robots and Systems, pages 25-32, 2022.
[12] Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 41(4):1-17, 2022.
[13] Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, volume 30, pages 3812-3822, 2017.
[14] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Ractsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pages 4114 4124, 2019.
[15] Jiahao Lin and Zongzhang Zhang. Acgail: Imitation learning about multiple intentions with auxiliary classifier gans. 15th Pacific Rim International Conference on Artificial Intelligence, pages 321-334, 2018.
10

---

[16] Adrian SpurT, Emre Aksan, and Otmar Hilliges. Guiding infogan with semi-supervision. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 119-134, 2017.
[17] Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, pages 1-13, 2017.
[18] Andreas Krause, Pietro Perona, and RyanG. Gomes. Discriminative clustering by regularized information maximization. In Advances in Neural Information Processing Systems, pages 775-783, 2010.
[19] Ian Goodfellow, Jean Pouget- Abagee, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 1):1391144, 2020.
[20] Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observa- tion. arXiv preprint arXiv: /00///877..6.8. 2018.
[21] Qingyang Wu, Lei Li, and Zhou Yu. Textgail: Generative adversarial imitation learning for text generation. In AAA/ Conference on Artificial Intelligence, volume 35, pages 14067-14075, 2021.
[22] Raunak Bhattacharyya, Blake Wulfe, Derek Phillips, Alex Kuefler, Jeremy Morton, Ransalu Senanayake, and Mykel Kochenderfer. Modeling human driving behavior through genera- tive adversarial imitation learning. IEEE Transactions on Intelligent Transportation Systems, 24:2874 24:2874-2887, 2023.
[23] Xin Zhang, Yanhua Li, Ziming Zhang, and Zhi-Li Zhang. f-gail: Learning f-divergence for generative adversarial imitation learning. Advances Neural Information Processing Systems, volume 33, pages 12805-12815, 2020.
[24] Yoshihisa Tsurumine and Takamitsu Matsubara. Goal-aware generative adversarial imitation learning from imperfect demonstration for robotic cloth manipulation. Robotics and Autonomous Systems, 158: 104264, 2022.
[25] David Barber and Felix Agakov. The im algorithm: variational approach to information maximization. In Advances Neural Information Processing Systems, volume 16, pages 201-208, 2004.
[26] Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph J Lim. Multi- modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neurul Information Processing Systems, pages 1235-1245, 2017.
[27] Xinpeng Wang, Huiqiao Fu, Guizhou Deng, Canghai Liu, Kaiqiang Tang, and Chunlin Chen. Hierarchical free gait motion planning for hexapod robots using deep reinforcement learning. IEEE Transactions on Industrial Informatics, 19(1 1090011-11111 2023.
[28] Eric Vollenweider, Marko Bjelonic, Victor Klemm, Nikita Rudin, Joonho Lee, and Mareo Hutter. Advanced skills through multiple adversarial motion priors in reinforcement learning. IEEE International Conference on Robotics and Automation, pages 5120-5126, 2023.
[29] Xin Zhang, Yanhua Li, Xun Zhou, and Jun Luo. Unveiling taxi drivers' strategies via cgail: Conditional generative adversarial imitation learning. IEEE International Conference on Data Mining, pages 1480-1485, 2019.
[30] Chenhao Li, Sebastian Blaes, Pavel Kolev, Marin Vlastelica, Jonas Frey, and Georg Martius. Versatile skill control via self-supervised adversarial imitation of unlabeled mixed motions. In IEFE International Conference on Robotics and Automation, pages 2944-2950, 2023.
[31] Alex Kuefler and Mykel J Kochenderfer. Burn-in demonstrations for multi-modal imitation learning. International Conference on Autonomous Agents and Multiagent Systems, pages 1071-1078, 2018
11

---

[32] Cong Fei, Bin Wang, Yuzheng Zhuang, Zongzhang Zhang, Jianye Hao, Hongbo Zhang, Xuewu Ji, and Wulong Liu. Triple-gail: multi-modal imitation learning framework with generative adversarial nets. In International Joint Conference Artificial Intelligence, pages 2929-2935 2020.
[33] Guo Haixiang. Li Yijing, Jennifer Shang, Gu Mingyun, Huang Yuanyue, and Gong Bing. Learning from class inualanbbled data: Review of methods and applications. Expert systems with applications, 73:220-239, 2017.
[34] Xinting Hu, Yi Jiang, Kaihua Tang, Jingyuan Chen, Chunyan Miao, and Hanwang Zhang. Learning segment the tail. In IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 14045-14054, 2020.
[35] Aaron Reite, Scott Kangas, Zackery Steck, Steven Goley, Jonathan Von Stroh, and Steven Forsyth. Unsupervised feature learning in remote sensing. In Applications of Machine Learning. volume 11 1113, pages 132-144, 2019.
[36] Mengye Ren, Tyler R Scott, Michael L luzzolino, Michael C Mozer, and Richard Zemel. Online unsupervised learning of visual representations and categories. arXiv preprint drXiv:2109.05675, 2021.
[37] Tanmoy Dam, Sreenatha G Anavatti, and Hussein A Abbass. Improving clustergan ing ceff-augmmeeedifoormatioo maximization of disentangling latentspaces. arXiv preprint arXiv:2107.12706, 2021.
[38] Utkarsh Ojha, Krishna Kumar Singh. Cho-Jui Hsich, and Yong Jac Lee. Elastic-infogan: Unsupervised disentangled representation learning in dass-imbalanced data. In Advances in Neural Information Processing Systems, volume 33, pages 18063-18075, 2020.
[39] Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srini- vasa. Imitation learning as f-divergence minimization. In Algorithmic Foundations of Robotics XIV, pages 313-329, 2021.
[40] Matt J Kusner and José Miguel Hernández-Lobato. Gans for sequences discrete elements with the gumbel-softmax distribution. arXiv preprint 1rXiv:661.040051 2016.
[41] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprini 18X2:::8//.0555 2018.
[42] Attr Ghosal, Arunima Nandy, Amit Kumar Das, Saptarsi Goswami, and Mrityunjoy Panday. short review on different clustering techniques and their applications. In Emerging Technology Modelling and Graphics, pages 69 69-83, 2020.
[43] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems, volume 17, pages 529-536, 2005.
[44] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 7077 170.006377 2017.
/707
[45] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint drXiv:1506.02438,200 2015.
12


Reviews:
[{"review_id": "3ivMsQ4OD2", "reviewer": "Reviewer_iRnn", "summary": "The paper introduces a semi-supervised imitation learning architecture that addresses challenges associated with real-world demonstrations, such as multimodality, data imbalance, and expensive labeling processes. The proposed method utilizes three key components: adapting semi-supervised generative adversarial networks to the imitation learning context, employing a learnable prior to align generated and expert data distributions, and utilizing a regularized information maximization approach along with the learned prior to enhancing semi-supervised learning performance. Experimental results highlight the effectiveness of the proposed method in learning multimodal behaviors from imbalanced demonstrations, outperforming baseline methods.", "questions": "* Please provide more details about how to collect imbalanced multi-modal demonstrations used for training.\n* I am also confused about the evaluation metrics. It seems that the authors only measure the distance between agent demonstrations and multi-modal expert demonstrations. However, the average reward, which is a basic evaluation metric should also be provided for comparison.", "limitations": "The authors have briefly discussed the limitations of the paper in the last section.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The idea of this paper is well-motivated and the investigated problem is practical.\n* Empirical results show great improvement over baseline and other compared methods. The visualized results are good to demonstrate the efficiency", "weaknesses": "* The novel seems to be limited. This paper seems to be an extension of ss-InfoGAN [1] in the imitation learning fields. And the authors do not fully discuss the difference between ss-InfoGAN and the proposed method.\n* The quality of the paper can be improved. There seem to be a lot of types in the paper. For example, the missing reference of the figure in line 208, 212, 216. Moreover, I also can not find the definition of $D_body$ in Figure 2, which makes it hard to understand the framework of the whole algorithm.\n\n[1] Spurr, Adrian, Emre Aksan, and Otmar Hilliges. \"Guiding infogan with semi-supervision.\" Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Cham: Springer International Publishing, 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer iRnn\n\n> Q1. The novel seems to be limited. More discussions with ss-InfoGAN needs to be added.\n\nA1. Indeed, we draw inspiration from ss-InfoGAN and extended it to the imitation learning framework, addressing key issues that still persist in the field of imitation learning, such as multi-modal behavior, data imbalance, and expensive labeling processes. The main differences from ss-InfoGAN are as follows: 1. We focus on imitation learning under sequential decision-making tasks, and 2. We introduce a learnable latent skill distribution and an improved RIM approach to tackle critical challenges commonly present in imitation tasks based on a substantial amount of imbalanced raw data. Relevant discussions have been added to section 3.2. Please refer to Common Response A1 for more details.\n\n> Q2. Some minor errors need to be corrected. The definition of $D_\\text{body}$ in Fig. 2 is not clear.\n\nA2. (1) Thank you for your helpful comments. We have rectified these minor errors.\n\n(2) $D_{\\text{body}}$ serves as a shared backbone network for the discriminator $D_{\\phi}$ , and the encoders $Q_{\\psi}^{\\epsilon}$ and $Q^{\\mathbf{c}}_{\\psi}$ , responsible for extracting features from state-action pairs. We have added explanations in Fig. 2 and included the architecture diagram of the backbone network in the Appendix.\n\n> Q3. How to collect imbalanced multi-modal demonstrations used for training?\n\nA3. To collect imbalanced multi-modal data, we first train an expert policy for each mode. Subsequently, based on these expert policies, the agent interact with the environment to collect expert trajectories. Finally, we extract data from these trajectories in different proportions to create imbalanced multi-modal data. Please refer to Common Response A2 for more details.\n\n> Q4. The average reward should also be provided for comparison.\n\nA4. Thank you for your valuable suggestion. We have supplemented the average reward in both the Experiments section of the main paper and the Appendix. All the results demonstrate the advantages of the proposed method. Please refer to Common Response A3 for more details."}, {"review_id": "pWXMJVziMI", "reviewer": "Reviewer_fzCM", "summary": "# Problem Statement\n\nThe paper addresses the problem of imitation learning in the context of real-world demonstrations that often present challenges such as multimodality, data imbalance, and expensive labeling processes.\n\n# Main Contributions\nThe authors propose a novel semi-supervised imitation learning architecture, called Elastic semi-supervised InfoGAIL (Ess-InfoGAIL), that learns disentangled behavior representations from imbalanced demonstrations using limited labeled data. The method adapts the concept of semi-supervised generative adversarial networks to the imitation learning context, employs a learnable prior to align the generated and expert data distributions, and utilizes a regularized information maximization approach to further improve the semi-supervised learning performance. The authors demonstrate the efficiency of their method in challenging MuJoCo environments, showing that even in scenarios with highly imbalanced demonstrations, the policy can still reproduce the desired behavior modes.\n\n# Methodology\nThe paper introduces a novel approach called Elastic semi-supervised InfoGAIL (Ess-InfoGAIL) for semi-supervised imitation learning from imbalanced demonstrations. The approach is based on three key improvements to InfoGAIL:\n\n1. **Semi-supervised InfoGAIL**: The authors decompose the latent variable into a semi-supervised part and an unsupervised part. The semi-supervised part, denoted as '$c$', is a latent skill variable sampled from a categorical distribution, encoding the same information as the label '$y$'. The unsupervised part, denoted as '$ϵ$', is a latent shifting variable sampled from a continuous uniform distribution, allowing for style shifting within a given skill. The authors seek to maximize two mutual information terms $I(ϵ|s, a)$ and $I(c|s, a)$.\n\n2. **Learnable Latent Skill Variable**: To align the state-action transitions produced by a policy $π_θ$ with the imbalanced expert demonstrations, the authors utilize a differentiable skill variable drawn from a Gumbel-Softmax distribution.\n\n3. **Regularized Information Maximization (RIM) with an approximate label prior**: The authors leverage RIM with a label prior that approximates the learned latent skill distribution to make use of the intrinsic information in the unlabeled imbalanced data and improve the efficiency of semi-supervised learning.\n\nThe authors validate their method in a simple 2D trajectory environment for visualization and test it in four challenging MuJoCo environments.\n\n# Experiments\nThe paper conducted experiments in a 2D trajectory environment and four MuJoCo environments (Reacher, Pusher, Hopper, and Humanoid) to validate the proposed method's ability to discover disentangled behavior representations from imbalanced demonstrations with limited labeled data. The experiments also analyzed the amount of labeled data required for the model to effectively encode the semantic meaning of the labels and the effect of varying the degree of data imbalance on the imitation of multimodal behaviors. The results demonstrated the efficiency of the proposed method, Ess-InfoGAIL, in learning multimodal behaviors compared to baseline methods such as GAIL, InfoGAIL, ACGAIL, and Elastic-InfoGAIL. The paper also conducted ablation experiments to verify the performance improvement of Gumbel Softmax and RIM techniques under the semi-supervised framework and imbalanced data.", "questions": "- In equation (9) and algorithm 1, which part corresponds to the learnable latent skill variable? What are the learnable parameters and what is their update in each iteration?\n- What is $D_\\text{body}$ in Figure 2?\n- What is the label ratio in the data imbalance experiments described in section 4.3?\n- How are the expert demonstrations generated?", "limitations": "The authors note that the limitations include the requirement of a small amount of labeled data for each category and the preset number of modal categories.\n\nIn addition, the limitation of low difficulty of tasks and the limited evaluation metrics are mentioned in the \"Weakness\" section.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "# Originality and significance\n\nThe work innovatively introduces semi-supervised learning to online adversarial imitation learning of multimodal behaviors by extending InfoGAIL with mechanisms proposed in Ess-InfoGAN. The imitation learning from demonstrations that contain distributional multimodality is a topic of great interest, and the approach shows promising performance.\n\n# Quality\n\nThe algorithm derivation and experiments are solid.\n\n# Clarity\n\nOverall the article is clear, although some details are not provided.", "weaknesses": "- Minor typo: The minimization should have $Q$ as a optimization variable in addition to $\\pi$.\n\n- The tasks in all experiments are not difficult due to the clear separation of the modes. As a reference, [1] shows that randomized search of the latent codes without encoder or posterior estimator can discern more subtly separated behavior modes in MuJoCo environments similar to those tested in this article (e.g. walker going forward and backward with various speeds).\n\n- The evaluation metrics are limited. Only entropy and mutual information are reported, while for the environments and tasks appear in the article, there are commonly-acknowledged reward or score that can be used to quantify the quality of learned policy. Without those metrics reported, the quality of the policies is unclear to the readers.\n\n- The implementation codes are not provided.\n\n[1]: Vahabpour, A., Wang, T., Lu, Q., Pooladzandi, O., & Roychowdhury, V. (2022). Diverse Imitation Learning via Self-Organizing Generative Models. arXiv preprint arXiv:2205.03484.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer fzCM\n\n> Q1. Minor typo: The minimization should have value function as a optimization variable in addition to $\\pi$.\n\nA1. Thank you for your valuable suggestion. We have incorporated the optimization of the value function into Section 3.3 of the main paper.\n\n> Q2. The tasks in all experiments are not difficult due to the clear separation of the modes. Taking [1] as a reference.\n\nA2. (1) Learning distinguishable behavior modes from multi-modal data remains challenging, especially when the data size of each behavior mode is imbalanced.\n\n(2) The provided reference [1] is an intriguing work that utilizes a generator model without an encoder for behavior cloning, capable of distinguishing and imitating different behavior modes. However, fully unsupervised searching for latent variables has been proven to be difficult, as indicated in [2], which is evident from the exhaustive search for latent variables in the procedure of [1]. This is also the motivation behind our introduction of semi-supervised learning. In addition, our method can efficiently learn both discrete behavior modes and continuous behavior styles (e.g., walking speed) without the need for exhaustive search. Please refer to Appendix C for a simple visualized example. It is promising to combine and complement our work with [1]. Unfortunately, the code for [1] is not available yet.\n\n[1] Vahabpour A, Wang T, Lu Q, et al. Diverse Imitation Learning via Self-Organizing Generative Models[J]. arXiv preprint arXiv:2205.03484, 2022.\n\n[2] Locatello F, Bauer S, Lucic M, et al. Challenging common assumptions in the unsupervised learning of disentangled representations[C]//international conference on machine learning. PMLR, 2019: 4114-4124.\n\n> Q3. The evaluation metrics are limited. The average reward should also be provided.\n\nA3. Thank you for your valuable suggestion. We have supplemented the average reward in both the Experiments section of the main paper and the Appendix. All the results demonstrate the advantages of the proposed method. Please refer to Common Response A3 for more details.\n\n> Q4. The implementation codes are not provided.\n\nA4. To enhance the readability of the code, we have organized it and uploaded it to an anonymous GitHub repository. In accordance with the rebuttal policy, which prohibits the inclusion of links to external pages in posted content, we have provided an anonymized link to the Area Chair (AC) in a separate comment. The code will be released to the public promptly after the paper's publication.\n\n> Q5. In equation (9) and algorithm 1, which part corresponds to the learnable latent skill distribution? What are the learnable parameters and what is their update in each iteration?\n\nA5. (1) The latent skill variable $\\mathbf{c}$, concatenated with the state, serves as an input to the policy, and the learnable latent skill distribution $p(\\mathbf{c})$ is updated along with the policy. Specifically, the objective function of $p(\\mathbf{c})$ aligns with the policy, which aims at minimizing the first two parts of $V_{Ess-InfoGAIL}$ in Equation 9.\n\nThe revised objective function of Ess-InfoGAIL in Equation 9 is defined as follows:\n\n$\\min_{\\pi, Q^{\\epsilon}, Q^{\\mathbf{c}}, p(\\mathbf{c})} \\max_{D}(V_{InfoGAIL} - \\lambda_{2}L_{IS} - \\lambda_{3}L_{RIM})$\n\nAnd the update of $p(\\mathbf{c})$ is as follows:\n\n$p_{i+1}(\\mathbf{c})=\\arg\\min_{p(\\mathbf{c})}(V_{InfoGAIL} - \\lambda_{2}L_{IS})$\n\nWe have revised Algorithm 1 and the description of Equation 9 to enhance readers' understanding.\n\n(2) In our model, the learnable parameters include $\\theta, \\beta, \\phi, \\psi, p(\\mathbf{c})$, corresponding to the parameters of the policy, value function, discriminator, encoders and the latent skill distribution. Their update processes in each iteration are detailed in the revised Algorithm 1.\n\n> Q6. What is $D_\\text{body}$ in Fig. 2?\n\nA6. $D_{\\text{body}}$ serves as a shared backbone network for the discriminator $D_{\\phi}$ , and the encoders $Q_{\\psi}^{\\epsilon}$ and $Q^{\\mathbf{c}}_{\\psi}$ , responsible for extracting features from state-action pairs. We have added explanations in Fig. 2 and included the architecture diagram of the backbone network in the Appendix.\n\n> Q7. What is the label ratio in the data imbalance experiments described in section 4.3?\n\nA7. In the data imbalance experiments of Section 4.3, the label ratio is set to 0.5%, where 0.1\\% of the data corresponds to 2 episodes (100 time steps) of each behavior mode in the Reacher environment. We have included relevant descriptions in Section 4.3. Specifically, if not otherwise specified, the default label ratios are as follows: 2D-Trajectory: 1%, Reacher: 0.5%, Pusher: 1%, Walker: 1%, Humanoid: 2%. The corresponding descriptions have also been added to Section 4.2.\n\n> Q8. How are the expert demonstrations generated?\n\nA8. To collect imbalanced multi-modal demonstrations, we first train an expert policy for each mode. Subsequently, based on these expert policies, the agent interact with the environment to collect expert trajectories. Finally, we extract data from these trajectories in different proportions to create imbalanced multi-modal demonstrations. Please refer to Common Response A2 for more details."}, {"review_id": "exnNJ8HihD", "reviewer": "Reviewer_18Ku", "summary": "This paper focuses on semisupervised imitation learning with imbalanced data. Mainly, the approach extends InfoGAIL with a semisupervised learning architecture, inspired by ss-InfoGAN, where the latent variable is decomposed into a semisupervised part and an unsupervised part. The semisupervised part is defined as a learnable latent skill variable to deal with the imbalance in the data. Regularises Information Maximisation with a label prior is further used to improve the robustness against different models of behaviour and limited data. The proposed method is evaluated on a toy example as well as high-dimensional control tasks in Mujoco, like humanoid.\n\nPOST-REBUTTAL COMMENTS\nThe authors have addressed my comments satisfactorily. They presented further experimental results. Therefore I have increased my score from \"borderline reject\" to \"borderline accept\".", "questions": "- What is the amount of data used in the experiments? Only the percentages of labelled data have been provided. However, still 0.1% can be too much, if the number of total samples is large, like 1M. Explain clearly how the experiments were designed. Also present the cumulative rewards to demonstrate the effective of the proposed method.", "limitations": "The authors briefly discussed the limitations. Potential negative impact does not apply.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-written paper with clear motivation and objectives. The authors have intention to share the code. \n- Experiments are well designed to showcase the contribution. For example, the method is evaluated with different levels of imbalance in the data (Table 2) and with different numbers of behaviour modalities (Table 3).", "weaknesses": "- Some of the arguments in the paper are not clear to me. Looking at figure 1, do we want the agent to imitate the expert's different behaviour styles? or do we want agent to do the task? Why is learning different modes of behaviour important in this context? In addition, how is this achieved in the Mujoco environments? For example, how are different expert behaviours generated in the reacher environment? Why is classification relevant in this context?  \n- Experimental results are limited. Why are the results presented in terms of NMI and ENT only? How do we know the task performance? Expected cumulative reward should be presented or a video supplementary is needed to show that it is actually working. [56] is an image representation learning method, it doesn't make sense to use the same metrics in this paper where the goal is imitation learning. In addition, experiments regarding degree of data imbalance and learning more modalities are shown only in the reacher environment. We don't know whether the findings can be generalised to different environments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer 18Ku\n\n> Q1. Confused about Fig. 1. Do we want the agent to imitate the expert's different behaviour styles? Why is learning different modes of behaviour important in this context? How is this achieved in the Mujoco environments? Why is classification relevant in this context?\n\nA1. (1) Fig. 1 illustrates a simple 2D-Trajectory scenario, facilitating the visual comparison of algorithm performance. In the environment, an agent tries to mimic 4 expert trajectories on a plane, each represented by a different color to indicate distinct modes (styles) of expert behavior. Fig. 1 (a) shows the expert trajectories, where the red and green trajectories are predominant, while the blue and purple trajectories only represent a small portion. Our Ess-InfoGAIL algorithm can successfully imitate distinguishable behavior styles, while GAIL and InfoGAIL can only imitate the more abundant red and green expert trajectories, and they cannot differentiate between trajectory styles. The corresponding descriptions have been added to Fig. 1.\n\n(2) In the MuJoCo environment, taking Reacher as an example, the expert behavior styles can be the robot arm reaching different target points. In all experiments, we first pre-train K expert policies. Then the agents interact with the environment based on the trained policies to sample K expert trajectories with different styles as training data. In addition, to imitate real world motion capture data, the final expert data used for training is obtained by randomly sampling from the K expert trajectories. Please refer to Common Response A2 for more details.\n\n(3) Our method can learn disentangled multi-modal behaviors from raw imbalanced expert data, utilizing only limited labeled data to guide the classification. This enables the policy network to focus more on learning representations related to behavior categories. Therefore, semi-supervised behavior style classification plays a crucial role in our approach. Please refer to Common Response A1 for more details.\n\n> Q2. Experimental results are limited. Expected cumulative reward should be presented. NMI and ENT metrics are not necessary. Experiments regarding degree of data imbalance and learning more modalities are shown only in the Reacher environment.\n\nA2. (1) Thank you for your valuable suggestion. We have supplemented the average reward in both the Experiments section of the main paper and the Appendix. All the results demonstrate the advantages of the proposed method. Please refer to Common Response A3 for more details. Furthermore, as this work focuses on semi-supervised imitation learning, a majority of the training data are unlabeled. In this context, NMI and ENT remain crucial metrics to assess whether the agent has learned distinct behavior modes. Therefore, we retain them in the main paper.\n\n(2) Reacher serves as a relatively convenient environment for evaluating the impact of data imbalance and the number of behavior modes. However, in other MuJoCo environments like Humanoid, generating expert policies with diverse behaviors (e.g., walking, jumping, crouching, etc.) through manual reward engineering is challenging (since we need to pre-train the expert policy). Nevertheless, such behavior data can be easily obtained using motion capture technology in the real world. In our future work, we will consider directly utilizing motion capture data to learn richer human behavior modes, as demonstrated in reference [1].\n\n[1] Peng X B, Guo Y, Halper L, et al. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters[J]. ACM Transactions On Graphics (TOG), 2022, 41(4): 1-17.\n\n> Q3. Explain clearly how the experiments were designed in Amount of Labeled Data section. Also present the cumulative rewards to demonstrate the effective of the proposed method.\n\nA3. (1) Thank you for your valuable comment. If not specified otherwise, the default label ratio for the environments is as follows: 2D-Trajectory: 1%, Reacher: 0.5%, Pusher: 1%, Walker: 1%, Humanoid: 2%. For instance, in the Reacher environment, 0.1% of labeled data corresponds to 2 episodes (100 time steps) of each behavior mode. The labeled data used for training is very limited, covering only a few episodes in each environment of each behavior mode. More comprehensive details have been included in Section 4.2.\n\n(2) Thank you for your valuable suggestion. We have supplemented the average reward in both the Experiments section of the main paper and the Appendix. All the results demonstrate the advantages of the proposed method. Please refer to Common Response A3 for more details.\n\nWe hope that the above response has addressed your concerns."}, {"review_id": "4VSMbULKgm", "reviewer": "Reviewer_dvkr", "summary": "The paper presents a semi-supervised approach to InfoGAIL for learning from demonstrations that enables disentangled learned behaviors. The proposed approach outperforms all considered baselines in MuJoCo simulation in imbalanced settings, and is robust to very little labeled data as well as varying levels of imbalance in the dataset.", "questions": "See above weaknesses section.", "limitations": "The authors mentioned some limitations for the proposed method, although this section could use more depth.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The theme of this paper in handling imbalanced demonstration data is timely and interesting. \n* The paper is well written.\n* The experiments diligently consider the potential relevant settings showcasing the capabilities of Ess-InfoGAIL in terms of handling imbalanced data, and a small fraction of labeled datapoints.", "weaknesses": "* There are several points of confusion for me in terms of the setting considered. From Fig. 1, it is unclear what about the expert trajectories indicates imbalance. I might have missed this, but what are the considered labels in the experiments? How many label classes are there?\n* The code is not available as of yet.\n* Table 1 is missing some measure of statistical significance.\n* In line 263, two numbers are listed twice. Why is the improvement over both ablation variants the same for both cases?\n* In Fig. 5, it seems that 0.5% labeled data achieves good performance (above 90%) on both Reacher and Pusher tasks? This observation does not seem to agree with the discussion. Could both Fig. 5(a) and 5(b) have the same y-scale to make it easier to compare?\n\n* The following is a non-exhaustive list of errors/typos and stylistic improvements I found:\n1. Line 45: 'correspond' -> 'corresponds'.\n2. Lines 79-93: missing space between words and citations. In general, citations should use 'abbrv' option in the bibliography.\n3. Hanging section heading (4.1).\n4. The references should be proofread (e.g., IEEE should not be listed twice in a citation, appropriate words (e.g., Bayesian) should be capitalized).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer dvkr\n\n> Q1. From Fig. 1, it is unclear what about the expert trajectories indicates imbalance. What are the considered labels in the experiments? How many label classes are there?\n\nFig. 1 illustrates a simple 2D-Trajectory scenario, facilitating the visual comparison of algorithm performance. In the environment, an agent tries to mimic 4 expert trajectories on a plane, each represented by a different color to indicate distinct modes (styles) of expert behavior (a small portion of the trajectories are labeled with one-hot behavioral categories).\n\nFig. 1 (a) shows the expert trajectories, where the red and green trajectories are predominant, while the blue and purple trajectories only represent a small portion. Our Ess-InfoGAIL algorithm can successfully imitate distinguishable behavior styles, while GAIL and InfoGAIL can only imitate the more abundant red and green expert trajectories, and they cannot differentiate between trajectory styles. The corresponding descriptions have been added to Fig. 1. For more detailed information on expert demonstrations, please refer to Common Response A2.\n\n> Q2. The code is not available as of yet.\n\nA2. To enhance the readability of the code, we have organized it and uploaded it to an anonymous GitHub repository. In accordance with the rebuttal policy, which prohibits the inclusion of links to external pages in posted content, we have provided an anonymized link to the Area Chair (AC) in a separate comment. The code will be released to the public promptly after the paper's publication.\n\n> Q3. Table 1 is missing some measure of statistical significance.\n\nA3. Thank you for your valuable comment. Tables 1, 2, and 3, along with error bars (standard deviation) of the data, have been incorporated into the Appendix.\n\n> Q4. In line 263, two numbers are listed twice. Why is the improvement over both ablation variants the same for both cases?\n\nA4. Thank you for your valuable feedback. This is mainly due to the precision setting of the data. In the revised version, we have uniformly kept three decimal places throughout the manuscript to avoid unnecessary confusion.\n\n> Q5. Fig. 5 does not seem to agree with the discussion. Could both Fig. 5(a) and 5(b) have the same y-scale to make it easier to compare?\n\nA5. Thank you for your valuable advice. We have adjusted Fig. 5, and revised the discussion in Section 4.2 to eliminate any ambiguity.\n\n> Q6. A non-exhaustive list of errors/typos and stylistic improvements are provided.\n\nA6. Thank you very much for providing your valuable suggestions. We have carefully corrected the minor errors in the paper and incorporated the suggested changes."}]

Author rebuttal:
# Common Response

We are thankful to the reviewers for their valuable feedback. We first address the comments that are common to multiple reviewers and then response to the reviewers individually.

> Q1. The significance of this work.

A1. Our method draws inspiration from semi-supervised GANs and extends it to the multi-modal imitation task under imbalanced data, which plays a crucial role in enabling agents to effectively imitate real-world demonstrations. By utilizing the proposed Ess-InfoGAIL, high-quality multi-modal behaviors can be learned from a large amount of **raw expert demonstrations** without the need for:

* Labeling behavior modes (e.g., human walking, running, jumping, etc.) for each data point.
* Extracting independent segments of behavior modes.
* Establishing a balanced distribution of data among behavior categories.

All we require is a very limited amount of labeled data for learning guidance. These advantages are currently lacking in the majority of existing imitation learning algorithms.

> Q2. How to collect multi-modal demonstrations?

A2. In our experiments, we first pre-train K expert policies, each corresponding to K different goals (or K behavior modes). Subsequently, we use these K expert policies to sample K sets of expert demonstrations. From each set of expert demonstrations, we extract a small portion and label them with the one-hot behavioral categories, while the remaining expert demonstrations are randomly sampled and mixed to create imbalanced unlabeled expert data.

Moreover, in the real-world scenario, one can directly use the raw motion capture data (e.g., motion capture data of an animal over a day) without the need to train additional expert policies. The collection of multi-modal demonstrations has been clarified and added in Section 4 of the main paper.

> Q3. The average reward needs to be provided.

A3. We sincerely appreciate the valuable suggestions from the reviewers. We have incorporated relevant discussions of the average reward in section 4.1 of the main paper. The average reward of Reacher environment during the training process is added in Fig. 5. Due to space constraints, the average reward of other environments have been included in the Appendix. Furthermore, the normalized average reward quantification table, encompassing the outcomes of all algorithms, has been included in the Appendix, presented below. The results are also included in an additional single page PDF file. All the results demonstrate the advantages of the proposed method.

|  | 2D trajectory | Reacher | Pusher | Walker-2D | Humanoid |
|---|---|---|---|---|---|
| GAIL | 0.071$\pm$ 0.019 | 0.189$\pm$ 0.087 | 0.338$\pm$ 0.029 | 0.459$\pm$ 0.014 | 0.508$\pm$ 0.033 |
| InfoGAIL | 0.155$\pm$ 0.028 | 0.223$\pm$ 0.054 | 0.431$\pm$ 0.043 | 0.552$\pm$ 0.059 | 0.526$\pm$ 0.127 |
| ACGAIL | 0.540$\pm$ 0.061 | 0.616$\pm$ 0.131 | 0.790$\pm$ 0.042 | 0.703$\pm$ 0.040 | 0.614$\pm$ 0.040 |
| Elastic-InfoGAIL  | 0.189$\pm$ 0.059 | 0.261$\pm$ 0.079 | 0.483$\pm$ 0.042 | 0.616$\pm$ 0.039 | 0.544$\pm$ 0.111 |
| Ess-InfoGAIL$\backslash$GS  | 0.845$\pm$ 0.023 | 0.868$\pm$ 0.050 | 0.867$\pm$ 0.040 | 0.812$\pm$ 0.050 | 0.817$\pm$ 0.017 |
| Ess-InfoGAIL$\backslash$RIM  | 0.882$\pm$ 0.036 | 0.826$\pm$ 0.035 | 0.772$\pm$ 0.061 | 0.770$\pm$ 0.019 | 0.751$\pm$ 0.065 |
| Ess-InfoGAIL (Ours)  | **0.956$\pm$ 0.040** | **0.933$\pm$ 0.033** | **0.967$\pm$ 0.022** | **0.911$\pm$ 0.042** | **0.920$\pm$ 0.036** |

It is important to note that we make modifications to the original MuJoCo environment, and the computation of the task reward varies with different target behavior modes. We take the average value across all behavior modes as the final average task reward, and this task reward is only used as an evaluation metric and does not participate in the policy training process. Due to the issue of mode collapse and data imbalance, some methods (e.g., GAIL and InfoGAIL) may end up learning only one or two behavior modes, resulting in average task rewards across all modes that could be lower than those achieved by using a random policy.

> Q4. The implementation codes are not available.

A4. To enhance the readability of the code, we have organized it and uploaded it to an anonymous GitHub repository. However, in accordance with the rebuttal policy, which prohibits the inclusion of links to external pages in posted content, we have provided an anonymized link to the Area Chair (AC) in a separate comment. The code will be released to the public promptly after the paper's publication.

> Q5. Minor errors.

A5. We have corrected minor errors in the paper and have incorporated the suggested changes.

Please let us know if there are any remaining questions!

Meta-review: This submission proposes an approach for semi-supervised imitation learning, as an extension to a popular work InfoGAIL. There were differing opinions among reviewers regarding the submission, especially in the initial version. On the positive side, all reviewers agreed that the idea of the submission is well-motivated and also the submission showed significant improvement over baseline methods. On the other hand, there were a few criticisms on the limited novelty of the method and the fact that the tasks in all experiments are not too difficult. The initial submission also lacked some key performance metrics (e.g. total reward-based comparison). However, the authors addressed most of the concerns fairly (including pointing out that the key performance metrics were already in the supp. material). Thus, all reviewers changed their opinions to be positive, and the AC also concurs with the overall feedback. Therefore, we think this submission is worth sharing with others in the community.
Decision: Accept (poster)