Title: Ess-InfoGAIL: Semi-supervised Imitation Learning from Imbalanced Demonstrations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Generative adversarial imitation learning

semi-supervised learning

multi-modal behaviors

imbalanced data
Abstract: Imitation learning aims to reproduce expert behaviors without relying on an explicit reward signal. However, real-world demonstrations often present challenges, such as multi-modal, data imbalance, and expensive labeling processes. In this work, we propose a novel semi-supervised imitation learning architecture that learns disentangled behavior representations from imbalanced demonstrations using limited labeled data. Specifically, our method consists of three key components. First, we adapt the concept of semi-supervised generative adversarial networks to the imitation learning context. Second, we employ a learnable latent distribution to align the generated and expert data distributions. Finally, we utilize a regularized information maximization approach in conjunction with an approximate label prior to further improve the semi-supervised learning performance. Experimental results demonstrate the efficiency of our method in learning multi-modal behaviors from imbalanced demonstrations compared to baseline methods.

Ess-InfoGAIL: Semi-supervised Imitation Learning from Imbalanced Demonstrations

Abstract

Imitation learning aims to reproduce expert behaviors without relying on an explicit reward signal. However, real-world demonstrations often present challenges, such as multi-modal, data imbalance, and expensive labeling processes. In this work, we propose a novel semi-supervised imitation learning architecture that learns disentangled behavior representations from imbalanced demonstrations using limited labeled data. Specifically, our method consists of three key components. First, we adapt the concept of semi-supervised generative adversarial networks to the imitation learning context. Second, we employ a learnable latent distribution to align the generated and expert data distributions. Finally, we utilize a regularized information maximization approach in conjunction with an approximate label prior to further improve the semi-supervised learning performance. Experimental results demonstrate the efficiency of our method in learning multi-modal behaviors from imbalanced demonstrations compared to baseline methods.

1 Introduction

The recent success of Reinforcement Learning (RL) has shown great potential in solving complex decision-making tasks [1, 2, 3]. A key prerequisite for RL is to design a reward function, which can be challenging in many real-world applications [4, 5]. By contrast, Imitation Learning (IL) methods, such as Behavior Cloning (BC) [6], Inverse Reinforcement Learning (IRL) [7] and Generative Adversarial Imitation Learning (GAIL) [8], can reproduce expert behaviors without relying on an explicit reward signal. Among them, GAIL performs a more effective learning process by learning a policy directly from demonstrations, and has been widely applied in various fields [9, 10, 11, 12].

Real-world demonstrations are typically coupled by multi-modal behaviors. To disentangle interpretable and meaningful behavior representations, InfoGAIL [13] utilizes the concept of maximizing mutual information between discrete or continuous latent variables to generate corresponding samples in an unsupervised manner. However, unsupervised learning of desired decodable factors from unstructured data can be challenging, particularly in the case of behavior imitation, where commonly used methods such as contrastive learning often underperform. Recent researches have uncovered that unsupervised learning of disentangled representations is inherently difficult without the presence of inductive biases on both the models and data [14]. Additionally, real-world demonstrations often consist of imbalanced behavior modes. For example, walking constitutes the majority of human behavior, whereas jumping only represents a small fraction. It is difficult for InfoGAIL to learn disentangled behavior representations from imbalanced demonstrations, as shown in Fig. 1. One intuitive solution is to introduce supervised learning with fully labeled data [15]. However, manually segmenting and labeling from the raw demonstrations can be a costly endeavor.

(a) Expert

(c) InfoGAIL

(d) Ess-InfoGAIL

O

0

To tackle these challenges, we introduce a novel semi-supervised imitation learning architecture that can acquire disentangled behavior representations from imbalanced demonstrations, guided by only a limited amount of labeled data (as little as 0.5% to a maximum of 2% of the dataset). Our model, Elastic semi-supervised InfoGAIL (Ess-InfoGAIL), makes three key improvements to the related InfoGAIL. First, we draw inspiration from ss-InfoGAN [16] and focus on learning meaningful and controllable behavior representations by maximizing two mutual information terms: i) The mutual information between a latent skill variable, drawn from a categorical distribution, and the labeled expert state-action transitions, ii) and the mutual information between a latent shifting variable, drawn from a continuous uniform distribution, and the state-action transitions produced by a policy. The former term enforces that the latent skill variable corresponds to label categories in a semi-supervised manner, while the latter allows for style shifting within a given skill. Second, we tackle the issue of imbalanced data and align the distribution of generated transitions with that of expert demonstrations by utilizing a latent skill variable drawn from a differentiable Gumbel-Softmax distribution [17], instead of using a latent skill variable drawn from a fixed categorical distribution. Third, to make use of the intrinsic information in the unlabeled imbalanced data and improve the efficiency of the semi-supervised learning, we leverage the Regularized Information Maximization (RIM) [18] with a label prior that approximates the learned latent skill distribution.

In the experiment, we first validated our method in a simple 2D trajectory environment to obtain a more intuitive visualization, as shown in Fig. 1, and subsequently tested our method in 4 challenging MuJoCo environments (Reacher, Pusher, Walker-2D, and Humanoid). The experimental results illustrate the efficiency of our method in disentangling multi-modal behaviors from imbalanced demonstrations compared to baseline methods. To the best of our knowledge, our work is the first to tackle the problem of learning disentangled behavior representations from imbalanced demonstrations, with the aid of only a limited amount of labeled data. The code is available at https://github. com/tRNAoO/Ess-InfoGAIL . In particular, the contributions of this work are threefold:

We extend the InfoGAIL with a semi-supervised imitation learning architecture, resulting in Ess-InfoGAIL, which enables the learning of desired disentangled behavior representations.

We utilize a learnable latent distribution to align the generated and expert distributions in the imbalanced data setting, and leverage RIM with an approximate label prior to improve the semi-supervised learning.

We demonstrate the efficiency of our method in 4 challenging MuJoCo environments, showcasing that even in scenarios with highly imbalanced demonstrations, the policy can still reproduce the desired behavior modes.

2 Related Work

Generative adversarial imitation learning GAIL adapts techniques developed for Generative Adversarial Networks (GANs) [19], and has demonstrated promising results in challenging imitation

learning tasks, such as robot control [10], imitation from raw visual data [20], multi-agent gaming [9], text generation [21], and driver behavior imitation [22]. Subsequent research has proposed various improvements to GAIL [23, 24], resulting in more efficient and reliable imitation results. However, discovering rich disentangled factors from an unstructured dataset using GAIL can be challenging due to the absence of inductive biases on the data or model. One promising approach to alleviate this issue is to introduce a latent variable model and use the technique of maximizing the mutual information [25] between the latent variable and the data, which can enable effective disentanglement of behaviors.

Learning multi-modal behaviors Tasks can often be accomplished in multiple ways, and the expert may exhibit different modes of behavior [26, 27]. Therefore, it is crucial to extend GAIL to learn and replicate these multi-modal behaviors. One intuitive approach is to train separate discriminators for each labeled mode [28]. Or one can use an auxiliary classifier allowing labelconditional imitation learning about multiple intentions [15]. These supervised approaches have demonstrated their effectiveness and reliability in various applications, from controlling wheeled robots [28] to task allocation in taxi companies [29]. However, they are constrained by their reliance on labeled data. To overcome this limitation, recent research has focused on unsupervised methods that typically leverage mutual information between latent variables and the generated data [30, 12]. For example, InfoGAIL maximizes this mutual information to learn from visual demonstrations [13], while Burn-InfoGAIL maximizes it from the perspective of Bayesian inference to draw modal variables from burn-in demonstrations [31]. However, unsupervised methods often struggle to distinguish latent factors without considering semantic information or task context [32], and proved unreliable in theory [14]. Therefore, we propose a novel semi-supervised approach that leverages the advantages of both supervised and unsupervised methods to learn semantically interpretable behavior representations with a small portion of labeled data.

Learning from imbalanced data Real-world data often exhibits imbalanced distributions, which can impede learning by causing the model to become biased towards the dominant category and potentially overlook the minority category. In a supervised learning setting, the problem of imbalanced data can be mitigated through various techniques such as data re-sampling and category re-weighting [33, 34]. These methods are useful when the category distributions are known beforehand. Several methods have been proposed for the unsupervised setting to deal with the imbalanced data with unknown category distributions [35, 36, 37]. A recent approach called Elastic-InfoGAN [38] has been proposed to address the issue of distribution alignment in GANs by utilizing a learnable latent distribution to handle imbalanced data. Unlike these methods, our work focuses on multi-modal semi-supervised imitation learning from raw imbalanced data without relying on prior knowledge of category distributions.

3 Approach

In this part, we first provide a brief background on RL, GAIL, and InfoGAIL, then provide a detailed description of how to extend InfoGAIL to a semi-supervised architecture and address the disentangled behavior learning using both the imbalanced unlabeled expert demonstrations and limited labeled expert demonstrations.

3.1 Background

Reinforcement learning We consider a standard Markov Decision Process (MDP) setting [1] represented by a tuple ⟨S , A , P , R , γ ⟩ , where S is the state space, A is the action space, P ( s t +1 | s t , a t ) is the transition probability of state s t +1 at time step t +1 given state s t and action a t at time step t , R ( s , a ) is the reward function, and γ ∈ (0 , 1) is the discount factor. In an RL process, an agent interacts with the environment and learns a policy π ( a | s ) to maximize the expected discounted return E p ( τ | π ) [ ∑ T -1 t =0 γ t R ( s t , a t ) ] , where p ( τ | π ) = p ( s 0 ) ∏ T -1 t =0 p ( s t +1 | s t , a t ) π ( a t | s t ) represents the likelihood of a trajectory τ = { s 0 , a 0 , r 0 , s 1 , . . . , s T -1 , a T -1 , r T -1 , s T } under π . The reward function serves as an interface through which users can specify the task that an agent needs to perform. However, creating effective reward functions that elicit the desired behaviors from an agent can often be a laborious and challenging process.

GAIL and InfoGAIL As one of the apprenticeship learning method, GAIL adapts techniques developed for GANs to directly learn a stylized policy through expert demonstrations without the need for handcrafted rewards. The formal GAIL objective is denoted as

where D is a discriminator which tries to distinguish state-action pairs ( s , a ) drawn from the generated state-action distribution d π ( s , a ) , induced by π , and the state-action distribution of the expert demonstration d E ( s , a ) . H ( π ) is the causal entropy of the policy π . The GAIL objective is optimized alternately by taking a gradient step to increase Equation 1 with respect to D and an RL step to decrease Equation 1 with respect to π . The Jensen-Shannon divergence between d π ( s , a ) and d E ( s , a ) is eventually minimized [39].

GAIL may encounter difficulties in learning multi-modal behaviors from unstructured demonstrations in the absence of meaningful representations. To overcome this limitation, InfoGAIL introduces an additional regularization term to maximize the mutual information I ( c ; s , a ) between the latent variable c and state-action transitions ( s , a ) , and the objective is then approximated by a variational lower bound [25]:

where Q ( c | s , a ) is an approximation of the true posterior P ( c | s , a ) . The lower bound is tight if Q = P . This regularization allows InfoGAIL infer the interpretable latent structure in an unsupervised manner, and the final objective becomes:

where λ 1 is a weighting coefficient.

3.2 Semi-supervised Imitation Learning from Imbalanced Demonstrations

Env.

Dataset

As InfoGAIL is learned in an unsupervised manner, obtaining the desired disentangled behavior representations can prove to be challenging. The difficulty is compounded when the expert demonstrations are imbalanced, as shown in Fig. 1. To address this, we propose Elastic semi-supervised InfoGAIL (Ess-InfoGAIL) with three improvements to InfoGAIL: i) A semi-supervised learning architecture, ii) a learnable latent skill distribution and iii) RIM with an approximate label prior. The network architecture of Ess-InfoGAIL is shown in Fig. 2.

Semi-supervised InfoGAIL To utilize both labeled and unlabeled data, we decompose the latent variable into a semi-supervised part c and an unsupervised part ϵ , inspired by ss-InfoGAN [16]. Unlike ss-InfoGAN, here we focus on imitation learning under sequential decision-making tasks. Additionally, we use c as a latent skill variable which is sampled from a categorical distribution,

allowing for encoding the same information as the label y , and use ϵ as a latent shifting variable which is sampled from a continuous uniform distribution, allowing for style shifting within a given skill.

Then we seek to maximize two mutual information terms I ( ϵ ; s , a ) and I ( c ; s , a ) , where the first term can be approximated using Equation 2 with a posterior approximation Q ϵ ψ ( ϵ | s , a ) , and the second term can be approximated using the following variational lower bounds:

where, Equation 4 is a supervised term that uses the state-action distribution of the labeled expert demonstration d EL ( s , a , y ) to predict y , and Equation 5 is an unsupervised term that utilizes the state-action distribution produced by a policy π θ to learn the inherent semantic meaning of y . The technique of lower bounding mutual information is known as variational information maximization [25]. With Q 1 ψ = Q 2 ψ = Q c ψ we obtain the semi-supervised regularization term:

where, L 1 IS is optimized with respect to Q c ψ , and L 2 IS is optimized with respect to π θ . Specifically, L IS is trained as a classification process with the cross-entropy function, and L I in Equation 2 is trained as a regression process with the mean absolute error.

Learnable latent skill distribution The majority of GAN and GAIL algorithms assume that the latent variable is drawn from a uniform distribution with fixed parameters. However, in the presence of imbalanced data, this assumption may lead to suboptimal optimization results. To align the state-action distribution produced by a policy π θ with the state-action distribution of the imbalanced expert demonstration, we utilize a latent skill variable c drawn from a differentiable Gumbel-Softmax distribution p ( c ) [17], similar to Elastic-InfoGAN [38]. The Gumbel-Softmax distribution is a continuous distribution over the simplex that can approximate samples from a categorical distribution in a differentiable way, and this trick has been widely used in many areas [40, 41].

Let p 1 , p 2 , · · · , p K be the class probabilities of a categorical distribution, the sampling of a Kdimensional latent skill variable c can be done as

where g 1 · · · g K are i.i.d samples drawn from Gumbel(0, 1). The coefficient τ is a temperature parameter that determines the degree of smoothing applied to the probability distribution. A smaller value of τ leads to a distribution that is closer to the one-hot encoding, while a larger value of τ results in a distribution that is closer to the categorical distribution.

Since the objective of GAIL is to minimize the Jensen-Shannon divergence between d π ( s , a ) and d E ( s , a ) , we can eventually estimate the label distribution of the unlabeled expert demonstrations by passing the gradient back to the latent skill distribution p ( c ) using Gumbel-Softmax reparameterization trick.

RIM with an approximate label prior Discriminative clustering techniques can automatically identify the boundaries or distinctions between categories in unlabeled data [42]. To leverage the intrinsic information in the unlabeled imbalanced demonstrations and enhance the efficiency of the semi-supervised learning process in Equation 6, we employ Regularized Information Maximization (RIM) [18] with a label prior that approximates the learned latent skill distribution:

where the first term is the cluster assumption [43], corresponding to the idea that datapoints should be classified with large margin. The second term is performed to avoid degenerate solutions by minimizing the KL divergence between the empirical label distribution Q c ψ (ˆ y ) ≈ 1 N ∑ i Q c ψ (ˆ y | s i , a i ) and the learned latent skill distribution p ( c ) , while the last term is a parameter regularization (e.g., L2 regularization) performed to avoid complex solutions.

3.3 Overall Algorithm

With the above three improvements, the objective function of Ess-InfoGAIL eventually becomes:

where λ 2 and λ 3 are two weighting coefficients. θ, ϕ, ψ are parameters for the policy π θ , the discriminator D ϕ , and the encoders Q ϵ ψ and Q c ψ , respectively. We optimize D ϕ , Q ϵ ψ and Q c ψ with the stochastic gradient method. π θ is optimized using Proximal Policy Optimization (PPO) [44], with advantages computed using GAE ( λ ) [45], and the value function is updated using TD ( λ ) [1]. The latent skill variable c , concatenated with the state, serves as an input to the policy, where c is sampled from the learnable latent skill distribution p ( c ) which is updated along with the policy. The schematic overview of the Ess-InfoGAIL network architecture is shown in Fig. 2, and the pseudo code is shown in Algorithm 1. In the experiments, we show that the proposed Ess-InfoGAIL can learn disentangled behavior representations efficiently from imbalanced demonstrations using limited labeled data.

Algorithm 1: Ess-InfoGAIL

Input: Unlabeled and labeled demonstrations d E ( s , a ) , d EL ( s , a , y ) , initial parameters of the policy, value function, discriminator and encoders θ 0 , β 0 , ϕ 0 , ψ 0 , initial latent skill distribution p 0 ( c ) , weighting coefficients λ 1 , λ 2 , λ 3

Output: Learned policy π θ ∗

for

1

i

= 0

,

1

,

2

,

· · ·

do

2 Sample a batch of latent variables ϵ ∼ p ( ϵ ) , c ∼ p i ( c )

3

Interact with the environment using

π

θ

i

,

ϵ

and

c

, and obtain

d

(

s

,

a

)

4 Sample a batch of state-action pairs b π ∼ d π , b E ∼ d E , b EL ∼ d EL

5 Update ϕ i to ϕ i +1 by ascending with gradients according to V GAIL

6 Update ψ i to ψ i +1 by descending with gradients according to V Ess-InfoGAIL

7 Update θ i , p i ( c ) to θ i +1 , p i +1 ( c ) by taking a policy step using the PPO update rule according to V Ess-InfoGAIL

8 Update the value function using TD ( λ )

9

end

4 Experiments

In this section, we validate our method by conducting experiments in a variety of environments, including a simple 2D trajectory environment, as well as 4 challenging MuJoCo environments (Reacher, Pusher, Walker-2D, and Humanoid). First, we perform the quantitative analysis to demonstrate the superiority of our method in discovering disentangled behavior representations from imbalanced demonstrations with limited labeled data, as compared to baseline methods. Second, we analyze the amount of labeled data required for the model to effectively encode the semantic meaning of the labels. Finally, we analyze the effect of varying the degree of data imbalance and the number of behavior modes on the imitation of multi-modal behaviors.

Task setup (1) 2D trajectory environment (Fig. 1) is a plane where the agent can move freely at a constant velocity by selecting its direction p t at discrete time t. The agent's observations at time t consist of positions from t -4 to t . The expert demonstrations consist of 4 distinct modes, each produced by a stochastic expert policy that generates a circular trajectory. The setting is similar to the experiment in [13]. (2) Reacher environment (Fig. 3 Left ) consists a 2-DoF robot arm. The goal is to move the robot's end effector close to a target position. The original environment is modified by setting 6 evenly distributed targets on a circle with a radius of 1.5, and the target position is removed from the observation space. The initial angles of the two joints are randomly sampled between [ -π, π ] and [ -π/ 2 , π/ 2] , respectively. (3) Pusher environment (Fig. 3 Right ) consists a 4-DoF robot arm. The goal is to move a target cylinder to a goal position using the robot's end effector. Similar to Reacher, we use 6 evenly distributed goal positions and remove the goal position from the observation space. (4) Walker-2D environment (Fig. 4 Left ) is a 6-DoF bipedal robot consisting of two legs

π

<

L

7

..:

：：：

：：：

火

and feet attached to a common base. The goal is to make coordinate both sets of legs and feet to move the robot in the right direction, including running forward, running backward and balancing. (5) Humanoid environment (Fig. 4 Right ) is a high-dimensional robot with 17 degrees of freedom. The goal is similar to the Walker-2D, including running forward, running backward and balancing.

Multi-modal demonstrations preparation In the experiments, we first pre-train K expert policies, each corresponding to K different goals (or K behavior modes). Subsequently, we use these K expert policies to sample K sets of expert demonstrations. From each set of expert demonstrations, we extract a small portion and label them with the one-hot behavioral categories, while the remaining expert demonstrations are randomly sampled and mixed to create imbalanced unlabeled expert data. Moreover, in the real-world scenario, one can directly use the raw motion capture data (e.g., motion capture data of an animal over a day) without the need to train additional expert policies.

Baselines and evaluation metrics We design various baselines to demonstrate the efficiency of our method in learning multi-modal behaviors from imbalanced demonstrations. (1) GAIL : The original GAIL. (2) InfoGAIL : The original InfoGAIL with a fixed uniform categorical distribution. (3) ACGAIL : Reproduction of ACGAIL [15] using an auxiliary classifier with full labeled data. (4) Elastic-InfoGAIL : Modification of Elastic-InfoGAN [38] in imitation tasks with a learnable latent distribution and a contrastive loss. (5) Ess-InfoGAIL \ GS : Ess-InfoGAIL without using the Gumbel-Softmax technique. (6) Ess-InfoGAIL \ RIM : Ess-InfoGAIL without using the RIM technique. (7) Ess-InfoGAIL : Our final method.

To evaluate the behavior disentanglement quality of the above methods, we use a pre-trained behavior classifier and employ two commonly used metrics in clustering algorithms similar to [38]: (a) Average Entropy (ENT) evaluates two properties: i) whether the state-action pairs generated for a given latent variable belong to the same ground-truth class and ii) whether each ground-truth class is associated with a single unique latent variable. (b) Normalized Mutual Information (NMI) quantifies the correlation between two clusterings, with a value ranging from 0 to 1. A higher NMI value indicates a stronger correlation between the clusterings.

Additionally, we use the normalized average task reward to assess the quality of learned policies. Note that, we modify the original MuJoCo environment and compute the task reward differently for each behavior mode. The final task reward, derived by averaging across all modes, serves exclusively as an evaluation metric and does not affect the policy training process.

4.1 Comparison with Baseline Methods

We quantitatively analyze Ess-InfoGAIL's behavior disentanglement performance, as presented in Table 1. In complex tasks with numerous behavior modes, both GAIL and InfoGAIL often encounter mode collapse, where their policies manifest behavior corresponding to only a subset of the behavior modes due to a lack of prior behavior representation knowledge. For instance, in the Reacher environment, the robot arm reaches only a few target positions. Leveraging limited labeling

guidance, our approach, Ess-InfoGAIL, improves the average NMI by 0.436 and 0.215 and decreases the average ENT by 0.401 and 0.300 compared to GAIL and InfoGAIL, respectively, across all environments. Notably, Ess-InfoGAIL attains superior behavior disentanglement quality with fewer labels than ACGAIL, which utilizes full labeled data. This success might stem from imbalanced data hindering ACGAIL's auxiliary classifier learning, while our method balances the small number of labels by extracting equal-duration data for each behavior mode.

Furthermore, we adapt Elastic-InfoGAN to the imitation learning framework using Gumbel-Softmax and contrastive learning loss, augmenting negative samples with Gaussian noise. However, contrastive learning's efficacy is limited in visual-free imitation tasks, and it indirectly addresses behavior disentanglement, yielding uncertain performance. In contrast, Ess-InfoGAIL directly tackles mode collapse and achieves semantically interpretable behavior disentanglement via limited labels. It exhibits an average NMI improvement of 0.215 and an average ENT decrease of 0.302 over Elastic-InfoGAIL across all environments. We perform ablation experiments, confirming the enhancements brought by Gumbel-Softmax and RIM techniques under the semi-supervised framework and imbalanced data. Ess-InfoGAIL demonstrates an average NMI increase of 0.039 and 0.046 and an average ENT reduction of 0.066 and 0.078 compared to Ess-InfoGAIL \ GS and Ess-InfoGAIL \ RIM, respectively.

To assess the quality of learned policies, we analyze the normalized average task rewards within the Reacher environment, depicted in Fig. 5, demonstrating the superiority of our method over all baseline methods. Results for different environments can be found in the supplementary material. Due to mode collapse and data imbalance, some methods (e.g., GAIL and InfoGAIL) may end up learning only one or two behavior modes, leading to lower average task rewards compared to a random policy.

4.2 Amount of Labeled Data Requirements

In real-world scenarios, obtaining labeled behavior data (e.g., from motion capture or raw video) is difficult due to the interleaving of various behavior modes. Thus, minimizing labeling costs is

1.0

dAverage Task Reward

0.8

0.6

InfoGAIL

ACGAIL

Normalized

0.4

Elastic-InfoGAIL

Ess-InfoGAIL\GS

0.2

Ess-InfoGAIL\RIM

Ess-InfoGAIL (Ours)

0.0

0.5

1.0

1.5

2.0

Steps

1e6

1.0

0.9

0.8

0.7

0.1%

0.6

0.5%

1%

0.5

10%

100%

0.4

0

2

4

D

8

Steps

1e5

crucial. We assess the prediction accuracy of encoders trained with limited labeled data across varying proportions within all unlabeled data. Results are shown in Fig. 6 (using Reacher environment as an example). Proportions of labeled data are set at 0.1%, 0.5%, 1%, 10%, and 100%. For Reacher, 0.1% data equates to 2 episodes (100 time steps) per mode. The encoder demonstrates nearly 92% prediction accuracy with just 0.5% labeled data. Walker-2D and Humanoid tasks, with higher state dimensions, require more labeled data for effective behavior mode separation. Default label ratios, if not specified, are: 2D-Trajectory: 1%, Reacher: 0.5%, Pusher: 1%, Walker-2D: 1%, Humanoid: 2%. Guided by limited labeled data, the agent adeptly disentangles specific styles from labeled data and more varied behavioral styles from unlabeled data.

4.3 Degree of Data Imbalance

The degree of data imbalance directly affects the learning quality of the policy, discriminator, and encoder. We set up 6 levels of data imbalance in the Reacher environment, ranging from 20 to 200. For example, at a data imbalance level of 20, the majority class has at most 20 times the number of samples as the minority class. The proportions of labeled data is set to 0.5%. At the maximum data imbalance level of 200, the minority class has only about 2 episodes of trajectories for each mode. The experimental results are shown in Table 2. It can be observed that as the degree of data imbalance increases, the NMI gradually decreases while the ENT gradually increases. However, even when the degree of the data imbalance is 200, the NMI of Ess-InfoGAIL only decreases by 0.079 compared to the case of a data imbalance of 20, and the performance is still better than InfoGAIL with a data imbalance of 20. Our proposed method demonstrates the ability to achieve effective disentanglement of behaviors, even in the presence of highly imbalanced data distributions.

4.4 Learning More Behavior Modes

The disentanglement task becomes increasingly challenging with an increase in the number of behavior modes. We set up 6 levels of behavior modes ranging from 2 to 12 in the Reacher environment, where the agent needs to learn disentangled behavior representations from these unstructured behavior data. We maintain a fixed degree of data imbalance at 100 and the proportions of labeled data is set to 0.5%. The experimental results are presented in Table 3. The results demonstrate that even with 12 behavior modes, our method consistently achieves higher NMI values and lower ENT values, compared to InfoGAIL with only 2 behavior modes. This affirms the controllability and scalability of our method in learning disentangled behavior representations while preserving category semantic information.

5 Conclusion

In this work, we proposed a novel semi-supervised imitation learning method, Ess-InfoGAIL, which utilizes a learnable latent distribution and an improved RIM to enable the agent to learn better disentangled multi-modal behaviors from unstructured expert demonstrations. We qualitatively and quantitatively validated the efficiency of our method, compared with other baseline methods. We also demonstrated that our method only requires a very small amount of labeled data to achieve good results, and further verified its controllability and scalability in scenarios with higher imbalance degrees and more behavior modes, which are crucial for multi-modal imitation tasks in the real world. Although there are some limitations, such as the need to provide a small amount of labeled data for each category and preset the number of modal categories, we believe that our work lays the foundation for imitation learning in large-scale unstructured demonstrations.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (No. 62073160).

References

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics , 7(62):eabk2822, 2022.

Huiqiao Fu, Kaiqiang Tang, Peng Li, Wenqi Zhang, Xinpeng Wang, Guizhou Deng, Tao Wang, and Chunlin Chen. Deep reinforcement learning for multi-contact motion planning of hexapod robots. In International Joint Conferences on Artificial Intelligence , pages 2381-2388, 2021.

Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning , volume 3, pages 278-287, 1999.

Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI Conference on Artificial Intelligence , page 3207-3214, 2018.

Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation , 3(1):88-97, 1991.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence , volume 8, pages 1433-1438, 2008.

Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems , volume 29, pages 4565-4573, 2016.

Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. In Advances in Neural Information Processing Systems , volume 31, pages 7461-7472, 2018.

Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (TOG) , 40(4):1-20, 2021.

Alejandro Escontrela, Xue Bin Peng, Wenhao Yu, Tingnan Zhang, Atil Iscen, Ken Goldberg, and Pieter Abbeel. Adversarial motion priors make good substitutes for complex reward functions. In IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 25-32, 2022.

Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG) , 41(4):1-17, 2022.

Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems , volume 30, pages 3812-3822, 2017.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning , pages 4114-4124, 2019.

Jiahao Lin and Zongzhang Zhang. Acgail: Imitation learning about multiple intentions with auxiliary classifier gans. In 15th Pacific Rim International Conference on Artificial Intelligence , pages 321-334, 2018.

Adrian Spurr, Emre Aksan, and Otmar Hilliges. Guiding infogan with semi-supervision. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 119-134, 2017.

Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations , pages 1-13, 2017.

Andreas Krause, Pietro Perona, and RyanG. Gomes. Discriminative clustering by regularized information maximization. In Advances in Neural Information Processing Systems , pages 775-783, 2010.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. arXiv preprint arXiv:1807.06158 , 2018.

Qingyang Wu, Lei Li, and Zhou Yu. Textgail: Generative adversarial imitation learning for text generation. In AAAI Conference on Artificial Intelligence , volume 35, pages 14067-14075, 2021.

Raunak Bhattacharyya, Blake Wulfe, Derek J Phillips, Alex Kuefler, Jeremy Morton, Ransalu Senanayake, and Mykel J Kochenderfer. Modeling human driving behavior through generative adversarial imitation learning. IEEE Transactions on Intelligent Transportation Systems , 24:2874-2887, 2023.

Xin Zhang, Yanhua Li, Ziming Zhang, and Zhi-Li Zhang. f-gail: Learning f-divergence for generative adversarial imitation learning. In Advances in Neural Information Processing Systems , volume 33, pages 12805-12815, 2020.

Yoshihisa Tsurumine and Takamitsu Matsubara. Goal-aware generative adversarial imitation learning from imperfect demonstration for robotic cloth manipulation. Robotics and Autonomous Systems , 158:104264, 2022.

David Barber and Felix Agakov. The im algorithm: a variational approach to information maximization. In Advances in Neural Information Processing Systems , volume 16, pages 201-208, 2004.

Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph J Lim. Multimodal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neural Information Processing Systems , pages 1235-1245, 2017.

Xinpeng Wang, Huiqiao Fu, Guizhou Deng, Canghai Liu, Kaiqiang Tang, and Chunlin Chen. Hierarchical free gait motion planning for hexapod robots using deep reinforcement learning. IEEE Transactions on Industrial Informatics , 19(11):10901-10912, 2023.

Eric Vollenweider, Marko Bjelonic, Victor Klemm, Nikita Rudin, Joonho Lee, and Marco Hutter. Advanced skills through multiple adversarial motion priors in reinforcement learning. In IEEE International Conference on Robotics and Automation , pages 5120-5126, 2023.

Xin Zhang, Yanhua Li, Xun Zhou, and Jun Luo. Unveiling taxi drivers' strategies via cgail: Conditional generative adversarial imitation learning. In IEEE International Conference on Data Mining , pages 1480-1485, 2019.

Chenhao Li, Sebastian Blaes, Pavel Kolev, Marin Vlastelica, Jonas Frey, and Georg Martius. Versatile skill control via self-supervised adversarial imitation of unlabeled mixed motions. In IEEE International Conference on Robotics and Automation , pages 2944-2950, 2023.

Alex Kuefler and Mykel J Kochenderfer. Burn-in demonstrations for multi-modal imitation learning. In International Conference on Autonomous Agents and Multiagent Systems , pages 1071-1078, 2018.

Cong Fei, Bin Wang, Yuzheng Zhuang, Zongzhang Zhang, Jianye Hao, Hongbo Zhang, Xuewu Ji, and Wulong Liu. Triple-gail: a multi-modal imitation learning framework with generative adversarial nets. In International Joint Conference on Artificial Intelligence , pages 2929-2935, 2020.

Guo Haixiang, Li Yijing, Jennifer Shang, Gu Mingyun, Huang Yuanyue, and Gong Bing. Learning from class-imbalanced data: Review of methods and applications. Expert systems with applications , 73:220-239, 2017.

Xinting Hu, Yi Jiang, Kaihua Tang, Jingyuan Chen, Chunyan Miao, and Hanwang Zhang. Learning to segment the tail. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14045-14054, 2020.

Aaron Reite, Scott Kangas, Zackery Steck, Steven Goley, Jonathan Von Stroh, and Steven Forsyth. Unsupervised feature learning in remote sensing. In Applications of Machine Learning , volume 11139, pages 132-144, 2019.

Mengye Ren, Tyler R Scott, Michael L Iuzzolino, Michael C Mozer, and Richard Zemel. Online unsupervised learning of visual representations and categories. arXiv preprint arXiv:2109.05675 , 2021.

Tanmoy Dam, Sreenatha G Anavatti, and Hussein A Abbass. Improving clustergan using self-augmentedinformation maximization of disentangling latentspaces. arXiv preprint arXiv:2107.12706 , 2021.

Utkarsh Ojha, Krishna Kumar Singh, Cho-Jui Hsieh, and Yong Jae Lee. Elastic-infogan: Unsupervised disentangled representation learning in class-imbalanced data. In Advances in Neural Information Processing Systems , volume 33, pages 18063-18075, 2020.

Liyiming Ke, Sanjiban Choudhury, Matt Barnes, Wen Sun, Gilwoo Lee, and Siddhartha Srinivasa. Imitation learning as f-divergence minimization. In Algorithmic Foundations of Robotics XIV , pages 313-329, 2021.

Matt J Kusner and José Miguel Hernández-Lobato. Gans for sequences of discrete elements with the gumbel-softmax distribution. arXiv preprint arXiv:1611.04051 , 2016.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Attri Ghosal, Arunima Nandy, Amit Kumar Das, Saptarsi Goswami, and Mrityunjoy Panday. A short review on different clustering techniques and their applications. In Emerging Technology in Modelling and Graphics , pages 69-83, 2020.

Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems , volume 17, pages 529-536, 2005.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438 , 2015.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a semi-supervised approach to InfoGAIL for learning from demonstrations that enables disentangled learned behaviors. The proposed approach outperforms all considered baselines in MuJoCo simulation in imbalanced settings, and is robust to very little labeled data as well as varying levels of imbalance in the dataset.\n\nSTRENGTHS:\n* The theme of this paper in handling imbalanced demonstration data is timely and interesting. \n* The paper is well written.\n* The experiments diligently consider the potential relevant settings showcasing the capabilities of Ess-InfoGAIL in terms of handling imbalanced data, and a small fraction of labeled datapoints.\n\nWEAKNESSES:\n* There are several points of confusion for me in terms of the setting considered. From Fig. 1, it is unclear what about the expert trajectories indicates imbalance. I might have missed this, but what are the considered labels in the experiments? How many label classes are there?\n* The code is not available as of yet.\n* Table 1 is missing some measure of statistical significance.\n* In line 263, two numbers are listed twice. Why is the improvement over both ablation variants the same for both cases?\n* In Fig. 5, it seems that 0.5% labeled data achieves good performance (above 90%) on both Reacher and Pusher tasks? This observation does not seem to agree with the discussion. Could both Fig. 5(a) and 5(b) have the same y-scale to make it easier to compare?\n\n* The following is a non-exhaustive list of errors/typos and stylistic improvements I found:\n1. Line 45: 'correspond' -> 'corresponds'.\n2. Lines 79-93: missing space between words and citations. In general, citations should use 'abbrv' option in the bibliography.\n3. Hanging section heading (4.1).\n4. The references should be proofread (e.g., IEEE should not be listed twice in a citation, appropriate words (e.g., Bayesian) should be capitalized).\n\nQUESTIONS:\nSee above weaknesses section.\n\nLIMITATIONS:\nThe authors mentioned some limitations for the proposed method, although this section could use more depth.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on semisupervised imitation learning with imbalanced data. Mainly, the approach extends InfoGAIL with a semisupervised learning architecture, inspired by ss-InfoGAN, where the latent variable is decomposed into a semisupervised part and an unsupervised part. The semisupervised part is defined as a learnable latent skill variable to deal with the imbalance in the data. Regularises Information Maximisation with a label prior is further used to improve the robustness against different models of behaviour and limited data. The proposed method is evaluated on a toy example as well as high-dimensional control tasks in Mujoco, like humanoid.\n\nPOST-REBUTTAL COMMENTS\nThe authors have addressed my comments satisfactorily. They presented further experimental results. Therefore I have increased my score from \"borderline reject\" to \"borderline accept\".\n\nSTRENGTHS:\n- Well-written paper with clear motivation and objectives. The authors have intention to share the code. \n- Experiments are well designed to showcase the contribution. For example, the method is evaluated with different levels of imbalance in the data (Table 2) and with different numbers of behaviour modalities (Table 3).\n\nWEAKNESSES:\n- Some of the arguments in the paper are not clear to me. Looking at figure 1, do we want the agent to imitate the expert's different behaviour styles? or do we want agent to do the task? Why is learning different modes of behaviour important in this context? In addition, how is this achieved in the Mujoco environments? For example, how are different expert behaviours generated in the reacher environment? Why is classification relevant in this context?  \n- Experimental results are limited. Why are the results presented in terms of NMI and ENT only? How do we know the task performance? Expected cumulative reward should be presented or a video supplementary is needed to show that it is actually working. [56] is an image representation learning method, it doesn't make sense to use the same metrics in this paper where the goal is imitation learning. In addition, experiments regarding degree of data imbalance and learning more modalities are shown only in the reacher environment. We don't know whether the findings can be generalised to different environments.\n\nQUESTIONS:\n- What is the amount of data used in the experiments? Only the percentages of labelled data have been provided. However, still 0.1% can be too much, if the number of total samples is large, like 1M. Explain clearly how the experiments were designed. Also present the cumulative rewards to demonstrate the effective of the proposed method.\n\nLIMITATIONS:\nThe authors briefly discussed the limitations. Potential negative impact does not apply.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n# Problem Statement\n\nThe paper addresses the problem of imitation learning in the context of real-world demonstrations that often present challenges such as multimodality, data imbalance, and expensive labeling processes.\n\n# Main Contributions\nThe authors propose a novel semi-supervised imitation learning architecture, called Elastic semi-supervised InfoGAIL (Ess-InfoGAIL), that learns disentangled behavior representations from imbalanced demonstrations using limited labeled data. The method adapts the concept of semi-supervised generative adversarial networks to the imitation learning context, employs a learnable prior to align the generated and expert data distributions, and utilizes a regularized information maximization approach to further improve the semi-supervised learning performance. The authors demonstrate the efficiency of their method in challenging MuJoCo environments, showing that even in scenarios with highly imbalanced demonstrations, the policy can still reproduce the desired behavior modes.\n\n# Methodology\nThe paper introduces a novel approach called Elastic semi-supervised InfoGAIL (Ess-InfoGAIL) for semi-supervised imitation learning from imbalanced demonstrations. The approach is based on three key improvements to InfoGAIL:\n\n1. **Semi-supervised InfoGAIL**: The authors decompose the latent variable into a semi-supervised part and an unsupervised part. The semi-supervised part, denoted as '$c$', is a latent skill variable sampled from a categorical distribution, encoding the same information as the label '$y$'. The unsupervised part, denoted as '$ϵ$', is a latent shifting variable sampled from a continuous uniform distribution, allowing for style shifting within a given skill. The authors seek to maximize two mutual information terms $I(ϵ|s, a)$ and $I(c|s, a)$.\n\n2. **Learnable Latent Skill Variable**: To align the state-action transitions produced by a policy $π_θ$ with the imbalanced expert demonstrations, the authors utilize a differentiable skill variable drawn from a Gumbel-Softmax distribution.\n\n3. **Regularized Information Maximization (RIM) with an approximate label prior**: The authors leverage RIM with a label prior that approximates the learned latent skill distribution to make use of the intrinsic information in the unlabeled imbalanced data and improve the efficiency of semi-supervised learning.\n\nThe authors validate their method in a simple 2D trajectory environment for visualization and test it in four challenging MuJoCo environments.\n\n# Experiments\nThe paper conducted experiments in a 2D trajectory environment and four MuJoCo environments (Reacher, Pusher, Hopper, and Humanoid) to validate the proposed method's ability to discover disentangled behavior representations from imbalanced demonstrations with limited labeled data. The experiments also analyzed the amount of labeled data required for the model to effectively encode the semantic meaning of the labels and the effect of varying the degree of data imbalance on the imitation of multimodal behaviors. The results demonstrated the efficiency of the proposed method, Ess-InfoGAIL, in learning multimodal behaviors compared to baseline methods such as GAIL, InfoGAIL, ACGAIL, and Elastic-InfoGAIL. The paper also conducted ablation experiments to verify the performance improvement of Gumbel Softmax and RIM techniques under the semi-supervised framework and imbalanced data.\n\nSTRENGTHS:\n# Originality and significance\n\nThe work innovatively introduces semi-supervised learning to online adversarial imitation learning of multimodal behaviors by extending InfoGAIL with mechanisms proposed in Ess-InfoGAN. The imitation learning from demonstrations that contain distributional multimodality is a topic of great interest, and the approach shows promising performance.\n\n# Quality\n\nThe algorithm derivation and experiments are solid.\n\n# Clarity\n\nOverall the article is clear, although some details are not provided.\n\nWEAKNESSES:\n- Minor typo: The minimization should have $Q$ as a optimization variable in addition to $\\pi$.\n\n- The tasks in all experiments are not difficult due to the clear separation of the modes. As a reference, [1] shows that randomized search of the latent codes without encoder or posterior estimator can discern more subtly separated behavior modes in MuJoCo environments similar to those tested in this article (e.g. walker going forward and backward with various speeds).\n\n- The evaluation metrics are limited. Only entropy and mutual information are reported, while for the environments and tasks appear in the article, there are commonly-acknowledged reward or score that can be used to quantify the quality of learned policy. Without those metrics reported, the quality of the policies is unclear to the readers.\n\n- The implementation codes are not provided.\n\n[1]: Vahabpour, A., Wang, T., Lu, Q., Pooladzandi, O., & Roychowdhury, V. (2022). Diverse Imitation Learning via Self-Organizing Generative Models. arXiv preprint arXiv:2205.03484.\n\nQUESTIONS:\n- In equation (9) and algorithm 1, which part corresponds to the learnable latent skill variable? What are the learnable parameters and what is their update in each iteration?\n- What is $D_\\text{body}$ in Figure 2?\n- What is the label ratio in the data imbalance experiments described in section 4.3?\n- How are the expert demonstrations generated?\n\nLIMITATIONS:\nThe authors note that the limitations include the requirement of a small amount of labeled data for each category and the preset number of modal categories.\n\nIn addition, the limitation of low difficulty of tasks and the limited evaluation metrics are mentioned in the \"Weakness\" section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a semi-supervised imitation learning architecture that addresses challenges associated with real-world demonstrations, such as multimodality, data imbalance, and expensive labeling processes. The proposed method utilizes three key components: adapting semi-supervised generative adversarial networks to the imitation learning context, employing a learnable prior to align generated and expert data distributions, and utilizing a regularized information maximization approach along with the learned prior to enhancing semi-supervised learning performance. Experimental results highlight the effectiveness of the proposed method in learning multimodal behaviors from imbalanced demonstrations, outperforming baseline methods.\n\nSTRENGTHS:\n* The idea of this paper is well-motivated and the investigated problem is practical.\n* Empirical results show great improvement over baseline and other compared methods. The visualized results are good to demonstrate the efficiency\n\nWEAKNESSES:\n* The novel seems to be limited. This paper seems to be an extension of ss-InfoGAN [1] in the imitation learning fields. And the authors do not fully discuss the difference between ss-InfoGAN and the proposed method.\n* The quality of the paper can be improved. There seem to be a lot of types in the paper. For example, the missing reference of the figure in line 208, 212, 216. Moreover, I also can not find the definition of $D_body$ in Figure 2, which makes it hard to understand the framework of the whole algorithm.\n\n[1] Spurr, Adrian, Emre Aksan, and Otmar Hilliges. \"Guiding infogan with semi-supervision.\" Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Cham: Springer International Publishing, 2017.\n\nQUESTIONS:\n* Please provide more details about how to collect imbalanced multi-modal demonstrations used for training.\n* I am also confused about the evaluation metrics. It seems that the authors only measure the distance between agent demonstrations and multi-modal expert demonstrations. However, the average reward, which is a basic evaluation metric should also be provided for comparison.\n\nLIMITATIONS:\nThe authors have briefly discussed the limitations of the paper in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}