ILPO-MP: Mode Priors Prevent Mode Collapse when Imitating Latent Policies from Observations
Authors: Oliver Struckmeier
Ville Kyrki
Venue: Accepted by TMLR ()

Abstract: Imitation learning from observations (IfO) constrains the classic imitation learning setting to cases where expert observations are easy to obtain, but no expert actions are available. Most existing IfO methods require access to task-specific cost functions or many interactions with the target environment. Learning a forward dynamics model in combination with a latent policy has been shown to solve these issues. However, the limited supervision in the IfO scenario can lead to mode collapse when learning the generative forward dynamics model and the corresponding latent policy. In this paper, we analyze the mode collapse problem in this setting and show that it is caused by a combination of deterministic expert data and bad initialization of the models. Under the assumption of piecewise continuous system dynamics, we propose ILPO-MP, a method to prevent the mode collapse using clustering of expert transitions to impose a mode prior on the generative model and the latent policy. We show that ILPO-MP prevents mode collapse and improves performance in a variety of environments.

Published in Transactions on Machine Learning Research (09/2023)
ILPO-MP: Mode Priors Prevent Mode Collapse when Imi- tating Latent Policies from Observations
Oliver Struckmeier Intelligent Robotics Group Aolto University, Espoo, Finland
oliver. Nibenn.ttuuckmmierB@oottot
Ville Kyrki
allle.grrki@oaltto
Intelligent Robotics Group Aolto University, Espoo, Finland
Reviewed on OpenReview: openrevveee....w.....mm.m net/ net/forum? id= f3JLnnZsAm
Abstract
Imitation learning from observations (IfO) constrains the classic imitation learning setting to cases where expert observations are easy to obtain, but no expert actions are available. Most existing TfO methods require access task-specific cost functions or many interactions with the target enviromment. Learning forward dynamics model in combination with latent policy has been shown to solve these issues. However, the limited supervision in the IFO scenario can lead to mode collapse when learning the generative forward dynamics model and the corresponding latent policy. In this paper, we analyze the mode collapse problem in this setting and show that it is caused by a combination of deterministic expert data and bad initialization of the models. Under the assumption piecewise continuous system dynamics, we propose IL.PO-MP, method to prevent the mode collapse using clustering of expert transitions to impose mode prior on the generative model and the latent policy. We show that ITPO--MP prevents mode collapse and improves performance in variety of environments.
1 Introduction
Imitation learning (IL) as a paradigm to imitate the policy of another agent or expert has been introducod in the 90s (Michie et al. 1990 Pomerleau 1991 Bain & Sammuul 1995). The recent advances in machine learning and the adoption of deep learning have enabled researchers process high-dimensional data and model more complex policies. As result, deep learning based IL methods have been applied in various
domains, such controlling autonomous vehicles (Pomerleau 1991 Codevilla et al. 2018 Bojarski et al. 2016 Pan et al.] 2017 Giusti et al. 2015 Abbeel & Ng. 2004), robotic manipulation (Fang et al. 2019 Finn et a.. 2016 Radosavovic et al. 2021) and learning to play video games and complete tasks in simulated environments (Ho Ermon 2016 Aytar et al.) 2018). However, most state-of-the-art IL methods require large amount of expert state-action data or rely on reinforcement. learning methods which often not possible in real-world applications.
Imitation learning from observations only (IfO) tries to circumvent the need for expert actions. To this end, Edwards et al. (2019) proposed ILPO, semi-supervised model-based IfO method in which latent policy is Icarned concurrently with a forward dynamics model conditioned on latent actions and expert state observations. Using the latent. policy, a data-efficient mapping from latent. actions to real action space can learned from few environment interactions without the need for task-specific knowledge or expert actions. However, the lack of supervision through expert actions can make difficult. recover from mode collapse, state in which the latent dynamics model collapses and expresses the observed dynamics with only one latent action. Since forward dynamics and latent policy are learned together, useful latent. policy

---

Published in Transactions on Machine Learning Research (09/2023)
be obtained in this collapsed state. This problem makes the method sensitive to the initialization of the model parameters and becomes especially an issue when expert demonstrations have been collected using a deterministic expert policy.
This work, analyzes the previously unaddressed mode collapse problem in model-based IfO and discusses possible solutions. Furthermore, we propose ILPO-MP, an unsupervised clustering approach to pre-train the latent policy and dynamics model and alleviate mode collapse during latent policy learning. We evaluate the performance of the learned latent policies and show that pre-training can prevent mode collapse and improve the performance of the baseline ILPO approach.
2
Related Work
2.1 Imitation Learning
Imitation Learning (IL) describes methods in which an imitation policy is learned to mimic the behavior of an expert or a target agent. IL methods usually assume that expert demonstrations consist of state and action pairs. Behavioral cloning (BC) (Michie al. 1990 Pomerleau 1991 Bain & Sammut 1995; Bagnell et al.) 2006 Ross al.) 2011 Daftry al., 2016), directly clones the expert policy from the expert state-action pairs. Drawbacks of BC are mainly related to covariate shift. Feedback loops and accumulating uncertainly in the imitated behavior may lead the imilator to unknown states in which the learned behavior
might fail (Ross & Bagnell 2010 Spencer et al. 2021
The second approach to IL based on inverse reinforcement learning (IRL), where expert demonstrations are used to infer the expert's reward function and then use reinforcement. learning methods and access to the environment learn an imitation policy. While those methods are considered more robust to covariate shift, most of the IRI-based methods require an extensive amount of environment interactions (Ng et al.
2000 Abbeel & Ng. 2004 Russell 1998
Finn et al. 2016 Ho & Ermon 2016).
Another approach to the imitation learning problem is aligning state and action distributions via some simi- larity metric such as lemporal similarity of state action pairs (Schroecker & Isbell 2017) or the Wasserstein distance (Dadashi et al.] 2020 Fickinger et al.] 2021). These methods, however make assumptions that source and target state-action distributions are alignable via their respective similarity metrics.
2.2 Imitation Learning From Observations Only
Imitation Learning from Observations Only (IfO) constrains the IL setting further by not relying on expert action information (Torabi et al., 2019) and is considcred strictly harder than IL with expert actions Kidambi et al. (021). The motivation is that precise recording of expert actions is costly or even impossible in many weal-world environments. However, often there is abundant video material or other expert state recordings to learn from.
In their review, Torabi et al. 2019 categorize IfO methods into model-free and model-based approaches. Model-free approaches usually address the lack of expert actions using adversarial methods or reward engi- neering as substitute supervision signals. Adversarial methods (Torabi et al. 2019 Nair et al. 2017 Ho & Ermon 2016 Schroecker & Isbell 2017) usually require large amounts of interactions with the environment, which is not always feasible or safe. Reward engineering can address this, but requires either human knowl- odge of the task or other data-driven methods to extract supervision signals from the expert data (Sermanct et al. 2018 Liu et al. 20118 Gupta et. al., 2017). A mixture of both approaches was presented by Aylar et al. 2018 where representations learned from videos were used to generate reward trajectories to guide the exploration of a standard Reinforcement Learning agent trained in the target environment.
Unlike model-free methods, model-based IIO methods can be used to learn efficient control policies allowing more flexible planning. Behavioral Cloning from Observation (BCO) (Torabi et al.) 2018)) learns an inverse dynamics model P(a)s{_1:ss) Sc) to infer actions from transitions while interacting with the environment and then uses the actions to learn the imitation policy. In a very similar fashion Radosavovic et al. (2021) learn inverse dynamics model and policy to support. the training of Reinforcement Learning agent with
2

---

Published in Transactions on Machine Learning Research (09/2023)
state-only expert data. While both methods show promising results, they require many interactions with the target environment to learn a dynamics model. Furthermore, (Sun et al. (2019) argue that there is no guarantee that obtaining an inverse dynamics model is always possible. Consider that the inverse dynamics model can be rewritten as
P(a ((4,8)41881) P(s 11, Sc ( (a(ss) P(s
(1)
P(s
Thus,
8( x P(s++ la (alss)
(2)
shows that the inverse model is ill-defined without considering the corresponding policy, in the case of stochastic dynamics. To learn a probabilistic inverse model for particular expert, the data to learn the model needs to be gathered from the same expert. Yang et al. (2019) term a similar issue inverse dynamics disagreement and discuss it by comparing LfO and learning from demonstration (LFD) which assumes access to expert actions. Minimising this disagreement difficult because it requires knowledge of euvironment dynamics and expert actions. However, an upper bound can be derived and minimized. The authors compare their method, inverse dynamics disagreement minimization, model-free LfO methods and show significant improvement. The theoretical and experimental results suggest that inverse dynamics models in LfO are impacted negalively by the reliance the expert policy.
To circumvent the need for the true expert policy, Edwards et al. (019) have proposed ILPO using forward dynamics model, to the best of our knowledge the first and only method taking this approach. A latent policy Tw (2(8;) is learned parameterise the models. Later the latent policy is mapped to the real action space with few environment interactions. Those two steps are referred as (1) latent policy learning and (2) action remapping. The dependency on the expert policy is approximated by
parallel with latent forward dynamics model Go Ge(se+1l81,22 where w and 0
π (aass)) ~ ((alz)m=((lss)
(3)
and the expert forward dynamics model by
(*(s++lsstaa) ~ Go(se+1 (x)s)s))))
(4)
Note that ILPO assumes a discrete action space and therefore discrete latent action space. In summary, the reliance on the expert policy is split into an intermediate learning problem in which low dimensional latent action space Z is learned in an unsupervised way from offline expert state observations.
In practice, the forward dynamics model Go is trained by predicting |Z| possible state transitions A₂ S411 - &t and picking the best prediction. The minimization loss
Cmin = min | - G)(s
(5)
assures gradient updates are performed the instance of G which leads to the best prediction to learn to separate the different modes/ modes/latent latent actions. The latent policy is trained by using the forward model and minimizing the loss
Lexp 11..
Σ Goossss
(6)
training the latent policy to predict probabilities that explain the observed transitions.
Ideally Z should be low dimensional such that the behavior of the expert can be sufficiently expressed with the available latent actions while the mapping to real actions P(a|z) is as easy as possible to learn. Edwards et al. (2019) have investigated by hyperparameter search that a good initial guess for the number latent actions is the true nunber actions that the expert takes.
3 Mode collapse when imitating latent policies
ILPO promises high data efficiency compared model-free and even many model-based LfO methods with respect to the number environment interactions. However, can not always guarantee that good latent
3

---

Published in Transactions on Machine Learning Research (09/2023)
policy can be learned. A core component of ILPO, the forward dynamics model, which is generative model, is susceptible to mode collapse, which has not been previously discussed.
3.1 Mode Collapse in Generative Models
Mode collapse, usually discussed in the context of Generative Adversarial Neural Networks (GANs), describes a failure scenario in which a multimodal generative model collapses to one mode, and the generator network generates data with low varioty (Salimans et al.) 2016; Che et al. 2016). Invostigations of the mode collapse problem have shown that it related to catastrophic lorgetting and the optimization process in GANs, which prevents the generator from breaking out of the model collapse (Che et al. 2016 Thanh-TTnn & Tran 2020 A symptom of mode collapse is that the generator fails to generate diverse data, which is well separated in observation space from the training data. As consequence this also prevents the discriminator from learning meaningful features.
Solutions to mode collapse in GANs include clustering the data based on knowledge about the classes in the training dataset (mode priors), mode regularization, minibatch discrimination (Che et al., 2016), continual learning or using optimizers with momentum to propagate knowledge during training to prevent catastrophic forgetting (Thanh-Tung & Tran 2020
3.2 Observing mode collapse
In ILPO, mode collapse occurs when the learned forward dynamics model can express the expert observations with few latent actions or, worst case, collapses to one mode. A unimodal latent action cannot be remapped to the real action space successfully. A deterministic expert policy a =s:)) is especially susceptible to mode collapse. In this case, the approximation of a deterministic expert policy can be simplified to
a = x (st) ~ (z))s) = (s)(=(ss)
(7)
which is independent of z. Furthermore, the forward dynamics can be simplified
G(st+ 1s,, 2) G(84+1,8
G(2,8e)
(8)
G())G((441,8) G( G(z)
which does not depend on the latent action. As result, the minimization loss term in Eq. collapses the mode for which by means of random initialization Lmin is smallest. We discuss the elfects of random initialisation in 6.4
Mode collapse in ILPO can be obscrved by monitoring the diversity of predicted latent actions by the latent policy The during latent policy learning. For example, let. us look the MountainCar Gym environment [Brockman al. 2016). In Fig. [a we can see that the average task performance of ILPO with deterministic expert demonstrations shown in blue is equal to random policy. The task not be learned. Let us now demonstrate how to identify mode collapse in the latent policy learning step of ILPO. Fig. Ib shows the number of unique latent actions in a batch predicted by the latent policy at the beginning of the latent policy learning process. With deterministic expert demonstrations, the number of unique actions drops to 1 early in the training and can not recover. The forward dynamics model collapses to one mode and predicts only one latent action, which is insufficient. to learn a meaningful mapping to the real action space.
Next, we demonstrate which properties of the expert demonstrations facilitate mode collapse in the latent policy learning step. Fig. 2a shows the 2D deterministic expert demonstrations in the MountainCar experi- ment colored by the ground truth expert actions. The state-action space exhibits distinct clusters of expert actions. Within each cluster, transitions are caused by the same expert action, and even the boundaries are clearly defined. As a result, there is very little variation in the action selection at any point in the state space. We highlight a few trajectories of the plotted expert data in the figure. These trajectories are deterministic paths through the statc-action space, highlighting that the expert is never uncertain about its action selection. As a result, the dynamics in the expert data can be explained by a model that is not conditioned on actions, as shown in Eq. g
4

---

Published in Transactions on Machine Learning Research (09/2023)
MountainCar
Number of Unique Latent Actions
Dae. Dopp
ILOO, Da Ear. Da. Exp. IPO. bet Bur. Noby.
Repert
ELPO, Neay Exp.
Average Episodie Reward
Pretnin, Nikk Rep.
Rander
Uniqque 2
Remapping Steps
Latent Police Learning Progress
%
(a) MountainCar task performance when per- forming ILPO from deterministic and noisy ex- pert demonstrations.
Diversity of latent actions during the first 17% of the latent policy learning.
Figure 1: A demonstration of mode collapse in the MountainCar experiment and the resulting imitation performance.
4 Addressing Mode Collapse
the following, we will discuss how mode collapse imitation learning has been addressed so far and propose new method for pre-training latent policies prevent mode collapse.
4.1 Noisy Expert Demonstrations in ILPO
MountainCar Deterministic State- State-Action Space
MountainCar Naiay Action Space
0.00
0/22
Vardey
-0.04
-0.2
Expest Action
Eapel Aciau
Expes Actia
-0.6
04
0.33
Poattion
Poetsen
(a) Deterministic expert.
(b) Noisy expert.
Figure 2: Visualisation of the expert states, actions and transitions in the MountainCar experiment.
While mode collapse has not been discussed by Edwards el al. (2019), the authors brielly note that ILPO performs better when the expert actions are noisy and that this is usually the case when learning from humans. Fig. shows a stato-action space with noisy expert actions. Compared Fig. [a such noisy expert demonstrations display locally diverse export actions. The performance and latent action diversity with noisy expert actions is shown orange in Figures Ta and 1b the task can be learned. This, limits

---

Published in Transactions on Machine Learning Research (09/2023)
the range of applications since the noise must be part of the expert's behavior while the demonstrations are recorded. It is impossible to introduce noise to deterministic expert states after the data collection since the states are a direct result of the action selection of the expert. One major issue is that tasks like MountainCar roquire accurate control to solve the task, which is not always possible when the expert is uncertain in its action selection. In this example, the agent must build enough momentum over an extended period of time which is significantly more difficult and often impossible when random actions counteract the momentum. In Fig. Ia the blue and orange dashed lines show the expert performance of the deterministic expert and the same expert with 20%chance of taking random actions while recording the MountainCar experd"] The expert performance with this amount of action noise is close to random performance (-200 for a random policy, 186 with noisy actions compared to -105 with deterministic actions). Finally, we want to discuss the assumption in Edwards et al. (2019), that humans provide noisy demonstrations, which we think does not generally hold. Previous work has shown that humans provide noisy demonstrations in robot control tasks such as teaching robot manipulation tasks Mandlekar et al. (2021). However, demonstrations in this setting are usually continuous and concerned with low level control which is different from ILPO Edwards et al. (2019). In ILPO, noise in the action selection means that at each decision there is probability to sample discrete action from a uniform distribution. However, in many problems, humans select discrete actions based on the expected utility of an action following the Boltzmann distribution (see softmax action selection). For example, when uncertain, humans will usually choose the best perceived option, given the uncertain situation, not select a random action. Furthermore, humans do not select a random action when they are quite certain about the action selection. More formally, in a state 8t, the probability of taking action -Quam)
is proportional to the value of the action and current state: P(als) x e A x The temperature parameter À describes the uncertainty in the action selection resulting from external factors such as the skill level of the demonstrator. The Q-function usually implies that action selection becomes more uncertain at decision boundaries, especially for novices. The noise exhibited by a human expert focuses around these decision boundarics where uncertainty is high and therefore not affect the mode collapse problem. For example, let look at. Figure in the manuscripl. In stochastic environment, the decision boundaries are less clear compared to Figure 2a but the action selection is still mostly uniform across the state space, leading to mode collapse. Figure 2b shows that ILPO assumes uniform noise across the state-action space, which is a different type of noise in action selection.
In conclusion, noisy expert demonstrations alleviate the mode collapse problem but impose constraints on the expert data collection procedure. Furthermore, collecting noisy expert data using the procedure in Edwards et al.] (2019) can significantly limit the capability of the expert and is not always representative of how humans make decisions.
Algorithm 1 Pre-training
Require:
Expert state-only data De, latent policy parameters w, latent dynamics parameters 0
1: T + Gettrransitionss D*)
2: zl +
CisbommrraiieCChsteringe
3: w + E2E Pre-train Tw (D*, x))
0 +- E2F Pre-train Go Co(D",v )
LLOODD,W,,0
4.2 ILPO-MP: Pre-training ILPO with a Mode Prior
To circumvent the need for noisy expert actions in ILPO, we propose ILPO-MP, a pre-training method to prime the latent policy and latent dynamics model with a mode prior such that bad initializations that lead to mode collapse are less likely. We chose mode priors over other methods of preventing mode collapse discussed in 3.1 because of their simplicity and effectiveness. Mode priors allow to incorporate prior knowledge about the underlying modes of a expert data distribution, enabling the generative dynamics model to be guided towards producing a diverse set of latent actions.
This is how Edwards al. 2019) introduced noise to the expert demonstrations
6

---

Published in Transactions on Machine Learning Research (09/2023)
But how obtain prior from observations alone? The underlying dynamics of many systems, from simple physical ones to robots, can be described using differential equations like 8 f(s a), often subject to some constraint c(s) I O. In many robotics and RL scenarios, it is assumed that f and c are continuous functions, meaning that small variations s and a cause small variations the outputs š. Piccewise continuity and e relaxes this assumption to accommodate abrupt changes like, for example, contact forces robotics. We leverage this assumption of piecewise continuous dynamics demonstrations into modes where behavior remains relatively consistent. In practice, the raw expert state demonstrations usually do not permit meaningful separation of modes. Instead, base our mode prior on the observation that the same actions in the expert data often lead to similar transitions in state-space.
cluster the expert
Table 1: Analysis of stale-action and transitions-actio similarity using the Procrustes Disparity between state ttannsttannnnon and action distance matrices. Using 5000 states from the expert data.
Experiment
Procrustes state-action disparity
Procrustes transition-action disparity
Acrobat
0.99
0.94
Cacipole
0.93
MountainCla
0.66
0.61
LonseLander
0.90
0.42
Prnn (vontor)
0.90
0.64
We can quantify this principle by measuring the correlation between states and true expert actions as well as transitions and true expert actions. the assumption of piecewise continuity holds, we expect to see high correlation between transitions and actions. We obtain three pairwise distance matrices: one for expert states, one for the transitions, and one for the ground truth actions. An example of those matrices from left to right is shown in fig. State and transition distance matrices are computed using the 2-norm. The action-distance matrix represents the equality between elements (actions) i and and is 1 if the action is equal to the action at j, and 0 otherwise. We then compute the Procrustes disparity Gower (1975)
Datsts
Divmasiiin
Daniin
175
130
126
120
100
75
75
70
25
25
25
0
20
50
73
100
125
1.0
175
25
50
75
1000
125
150
375
25
50
100
1.5
150
175
Figure 3: Visualisation of the pairwise distance matrices in the LunarLander expert data for the first 200 samples. From left to right: (left) distance matrix of states, (middle) distance matrix of transitions and (right) distance matrix of the expert actions. Qualitative analysis indicates correlation between transitions and actions.
Krzanowski (2000) for state-action and transition-action distance matrices
The results are shown in table D We can see that generally, the transition-action matrices have significantly lower Procrustes disparities, indicating higher similarity. The similarity assumption holds in all environments except the Acrobot environment, which we investigate in more detail in 6.2
PProcrustes analysis aligus two sets of data points by minimising differences in translation, rotation, and scaling, by min- imising the Procrustes disparity or distance which usually sum of the squares of the pointwise differences between two datasets.

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm shows how the pre-training step in ILPO-MP precedes the baseline ILPO training. We use agglomerative clustering on the expert state transitions to obtain latent action labels z! which serve a pseudo-labels for each transition in the expert dataset The number of clusters is thereby the number of latent actions for the baseline ILPO method. Fig. shows an example from the Pong environment and
Acrobot Deterministic State Action Space Aczobot Deterministic Transition Action Space
Transitions
Chustered
Uhster
0.0
Fesl
Exgert Action
Chater 0
0.4
Repert Action?
0.0
0.1
Expart Action
Cliste 1
0.-
02
Chutee
0.0
-02
42
0.2
-0.4
-0.-
-04
Espert Action
Expert Action
-0.6
Exper: Action
Poction
Figure 4: From left to right: (1) visualisation of transitions state-space as lines connecting states, colored by expert action. We observe no patterns in the expert data. (2) Transitions in transition space (A, 1-++1-81 colored by expert action. The higher dimensional transition data have been reduced to 2 dimensions using PCA. We can see that there are clusters where similar transitions have similar expert actions. (3) Clustering result with 6 latent actions. The clustering gives a good initialisation for the real expert actions.
highlights how in transition space clustering succeeds. The state-only expert data and the resulting labels are then used to pre-train the latent policy Tw and the forward dynamics model Go end-to-end:
Cpretruin || Go 2 I12 + ||z n (zls:))1
ILPO-MP uses the cluster labels z! as targets to train tu and as prior to replace the minimum in Eq. selecting through which mode of Ga to backpropagate.
After performing the pre-training, the latent policy learning and remapping is performed like in the baseline ILPO mothod with the pro-training network weights w and 0.
4.3 Guided ILPO-MP
We observed that the latent policy learning can end up in a collapsed state despite the pre-training when the latent policy learning is running for many epochs. This can be observed with the tools introduced in 3.2. During large number latent policy learning epochs, the number of different latent actions can collapse. Longer training means more optimisation steps that modify the model parameters of Go and Tu and as a result more chances to end up in bad local minimum in which the latent policy collapsed. This is an issue when more training epochs are required to learn more complex feature extractors. To this end, we propose variant of ILPO-MP to guide the latent policy learning using the prior obtained from clustering.
The proposed guided ILPO-MP approach extends the ILPO latent policy learning loss terms Lman (Eq. and Leep (Eq. 6 by the pre-training loss term presented in Eq.
5
Lguidance (e) = -1 ) + Lexp + Emin
C
(10)
where N is the total number of epochs and e is the current epoch. In practice, guided ILPO-MP trains the pre-training objective in parallel with the latent policy to prevent later collapse with the strongest regularisation applying at the beginning of the training to prevent mode collapse early in the training.
3 "Agglomerative clustering a hierarchical clustering method Johnson 1967) and merges individual data points or clusters iteratively based on measure of similarity, forming a hierarchical (dendrogram).
8

---

Published in Transactions on Machine Learning Research (09/2023)
5
Experiments
In this section provide more evidence for our claims that mode collapse is indeed a problem for ILPO and show how ILPO-MP prevents it. In 4 out of 6 environments, we notice that ILPO is unreliable and worst case, fails entirely. Implementation details are discussed in the Appendix in A.1
5.1 Pre-training recover mode collapse
First, let briefly discuss the CartPole environment. ILPO is able to fully solve the task every single time with both deterministic and noisy expert demonstrations. No mode collapse occurs in this environment. ILPO-MP is able to reproduce the same behavior.
Recall Fig. 1 which we used this environment as an example of mode collapse in the Mountain Car environment. Comparing ILPO (blue) and ILPO-MP (green) we can see that pre-training helped with the mode collapse where ILPO failed learn. Although ILPO-MP can not achieve expert performance, we perform better than baseline ILPO-MP with noisy expert demonstrations (red) performs best and vastly outperforms the expert it is trying to learn from (dashed orange). We hypothesize that pre-training helped ILPOO to separate the modes while ILPO alleviates the effects of the noise in the expert data. Priming the latent dynamics model with the assumption of piccewise contimuuu dynamics might have made the latent policy learning more robust to occasional deviations caused by random expert actions.
with noisy expert demonstrations (orange). Interestingly find that
LunarLander
Pong
Pong
BINA biler
Eqpat
Avernige Expocolie Beward
Average Exisooile Bewaard
Average Epicodi Rewsard
Pattak, Ust Ees.
ELFO Stt rpp
Expett
Cakkent cqp
BIP Dot Pop
Papert
Bemppcing ing Steps
Remapping Sleps
Remapping Stepe
(a) LunarLander task performance.
(b) Pong task performance.
(c) Pong from pixels task perfor- mance.
Figure 5: Task performance
The results in the LunarLander environment presented in Fig. 5a are clear. Baseline ILPO (blue) does not outperform random policy. Noisy expert data slightly improves task performance, but qualitative analysis does not indicate any behavior resembling the expert. ILPO-MP almost. achieves the performance of the noisy expert when learning from deterministic expert actions.
The last vector-based environment is our custom built two player Pong environment, shown in Fig. 5b Baseline IT.PO (blue) performs poorly with deterministic expert demonstrations. Qualitative analysis of the imitated behavior shows that the agent moves randomly and sometimes gets a lucky hit and scores, resulting in performance close to a random policy. ILPO with noisy expert demonstrations (orange) achieves its respective expert performance. ILPO-MP from deterministic and noisy expert achieved better and com- parable performance. In this experiment, we also found that using the guidance mode proposed in 4.3 yields further improvements. IPLO-MP with deterministic demonstrations is another significant improvement over baseline ILPO and ILPO-MP. Furthermore, ILPO-MP with noisy cemonstrations achieves the best lask performance and equals the deterministic expert.
The Pong from pixels experiment based on the same expert dataset as the vector pong experiment, but observations in pixel space. The results shown in Fig. 5c Although one would expect similar

---

Published in Transactions on Machine Learning Research (09/2023)
results to the vector Pong experiment, we observed that mode collapse still occurs and that ILPO with noisy expert demonstrations (orange) performs much worse. ILPP-MMP performs on par with baseline ILPOO. The pre-training is likely 'forgotten' during the longer latent policy learning step process when compared to the simple vector-based environments. We found that instead of pro-training, the guidance approach yiclds better results. With our proposed guidance method, we can achieve great results in both cases, deterministic (green) and noisy (red) expert demonstrations.
In conchusion, our proposed pre-training approach leads to improved task performance and works in envi- ronments where ILPO previously failed. Furthermore, we show that learning latent policies is applicable to adversarial environments, meaning environments in which the imitator has to play against another agent that is adapting to its behavior.
6 Discussion
6.1 Complex datasets
If the expert observations are generated by an underlying physical system with generative factors n like for example in vision-based tasks, the dynamics in observation space are likely not continuous. However, if the underlying physical system adheres to the piccewisc contimuity assumption, one learn a mapping g()), which maps pixel observations into a space that approximates the true underlying dynamics i = f(g(s) a). Extending ILPO-MP to more complex environments is a task of learning good representations that capture the underlying system dynamics. Recent works such as Zhang et al. (2020) and Lu et al. (022) have shown that invariant representations can capture task-relevant features. Other works extract meaningful representations from sequential data that capture the sparse nature of transitions in video data Klindt et al. (2020) and take into account temporal coherence in sequential data to better model latent dynamics Struckmeier et al. (2023). We illustrate this in the Pong environment, where ILPO-MP works from both, vector and pixel states (same underlying system dynamics) but with a more expressive CNN- based architecture for the pixel environment. In the future we aim to explore these represcntation learning methods to exlend ILPO-MP to learning in more complex pixel based environments.
6.2 Limitations
ILPO can solve the Acrobot environment with case demonstrated by (Edwards et al. 2019), shown in blue in Fig, 6al Our proposed method, ILPO-MP (green), achieves significantly lower lask performance, which is surprising given the good performance of ILPO and the simplicity of the task. We find that the Acrobot expert exploits an implementation detail of the physics in the Acrobot environment that violates the assumption we make on the expert demonstrations discussed in 4.2 The expert has learned to spin the outer joint extremely fast so that during one step of the environment physics, multiple rotations can occur. Stiff systems, modeled by ordinary differential equations, are prone to this issue. Such systems occur when the desired solution changes slowly while nearby solutions undergo rapid variations. This effect is closely tied to the assumption of piecewise continuity we make in our pre-training method. Piecewise continuity assumees overall system continuity but acknowledges the possibility of rapid changes or discontimuities at specific points or intervals. Stiff systems often violate this assumption due to significant variations in the rates of change among variables or components.
In the case of the Acrobot environment, the states consisting of the sine and cosine of the joint angles can not express multiple rotations and discontinuities are introduced into the expert data. Fig. 6b shows the states and next states colored in red and gray and their transition colored by the ground truth action. Lines representing transitions cross the circular state space. There is no discernible continuous trajectory like in the MountainCar data (see Fig. 2a The correlation metric for the Acrobot environment in Table further confirms this.
While the issue of modeling physics in the Acrobot environment breaks the assumption we make for our pre-training approach, we argue that this does not necessarily reflect limitation of our proposed method in general. The Acrobot environment is simplified simulation that focuses specific dynamics and constraints
10

---

Published in Transactions on Machine Learning Research (09/2023)
Acrobot
Actohot Deserministic Sate-Action Space
Average Episodic Reward
Expert
ILPO,
-12
Reprrtt
ILPO. Natey Fap
Repert
Prsteain, Det. Exg.
-0.6
-0.4
-02
0.2
Remapping Steps
Acrobot task performance.
(b) Acrobot deterministic expert data (2000 sam- ples).
Figure 6: Acrobot results
Correlation Anolyts
Comrlotion Andlyss
conytheas corellie Claa L
(a) Cartpole.
(b) Mountaincar.
Cormelation
Curolotiun
Trocart
Dorta Char
Seatio of 2lutten
Mardet of Clance
(c) LunarLander.
(d) Pong.
Figure 7: Correlation between the latent action labels obtained from clustering. In the LunarLander exper- iment the data was from the stochastic environment with deterministic expert. In all other environments the environment was deterministic.
and we can see that baseline ILPO already works well in such environments despite the discontimuities. More complex systems or real-world applications often introduce additional damping or stabilization mechanisms that prevent the occurrence of largo-scale dynamic responses. Furthermore, the tools presented in our work can be used to analyze expert data and evaluate the applicability of our pre-training based approach beforehand.
11

---

Published in Transactions on Machine Learning Research (09/2023)
6.3 Selecting the Number of Latent Actions
A critical parameter in ILPO and for our pre-training extension is the number of clusters/lates actions. Edwards et al. 2019)) have investigated the effect of the number of latent actions by repeating the experiments and plotting performance against the number of used latent actions. The results show that not all numbers of latent actions lead lo good results, and the number of real actions available is used as an initial guess. In the IfO setting, we assume that we have no access to the expert's action data Therefore we can not use this information when choosing the number of latent actions. Instead, we propose a method to identify the number of latent actions/chtt /clusters based on how well a clustering given the number latent actions conforms to the assumption we make on the transition-action similarity.
In Fig. we plot the Procrustes disparity like in table for range from 2 to 20 latent mode According to the proposed metric, the optimal number of latent actions/clusters maximizes the similarity (minimizes Procrustes disparity) between transitions and clusters shown in dark red. Those values are shown by a vertical dashed red line in the figures. The quality of this measure can be verified by comparing it to the similarity between cluster labels and ground truth expert actions shown in purple. We can also see that the similarity between expert states and clusters (in light blue) does not work well as an indicator in 3 out of 4 cases.
Furthermore, this method of visualizing the quality of the obtained clusters can be used to predict if a given expert observation-only dataset can be clustered successfully. If the Procrustes disparity between transitions and cluster labels is close to one for any number of clusters, we can assume that clustoring was not successful and can not yield a good prior for the pre-training. This holds for example for the Acrobot environment.
MountainCar
MountainCar
MountainCar
Averagee Exjoods Roward
Averoeo Eykcodx Roward
Average Epiancii Resond
Remapping Steps
Remapping Staps
Remopping Strps
(a) ILPO Deterministic Expert.
(b) ILPO Noisy Expert.
(c) ILPO-MP Deterministic Expert.
Figure 8: Task performance in the MountainCar experiment, plotted for each seed.
6.4 Effects of initialisation
As mentioned earlier, baseline ILPO susceptible to the random initialization of the neural network param- eters. The initialization can have a negative impact on the mode separation using the minimum operation in Luin Eq. 5 and consoquently lead collapse one latent action.
Vs E S arg min | - Go(s,
(11)
One batch S of data points is passed through the neural network, and Go z) is computed for all possible latent actions x. Tf, however, all samples in S minimize the loss for the same latent action z*, we only
We omit the Acrohot environment in this analysis as our clustering approach fails in this case, see 6.2
12

---

Published in Transactions on Machine Learning Research (09/2023)
backpropagate through mode, which is then also much more likely to minimize the loss for the next batch. This effect can accumulate and make mode separation impossible.
As a result, we observe a large spread in task performance for different random initializations. In Fig, we demonstrate this effect in the MountainCar experiment. In Fig. Sa we can see how ILPO with deterministic demonstrations manages only for one seed by chance achieve a small reward. With noisy expert demonstrations (Fig. Sb), ILPO manages solve the task a few times, but most of the time fails learn the task. With pre-training (ILPO-MP) (Fig. 8ch these catastrophic failures prevented.
6.5 Stochastic Environments
Lastly, we will briefly discuss the issue of stochasticity in the environment instead of the ex- pert demonstrations. We found that repeating the experiments in stochastic versions of the envi- ronments will not have much effect on the mode collapse problem. Stochastic environments add noise and make the states more diverse but do not introduce variance in the local action seloc- tion that enables mode separation. Fig. tion with deterministic expert policy in a stochastic version of the environment. Compared to the baseline in Fig. [a there is more noise around the origin, but the original problem persists. The performance and latent action diversity of the deterministic policy in the stochastic version of the MountainCar environment exhibits the same col- lapsed state.
shows the MountainCar expert state-action distribu-
MountainCar Noisy State-Action Space
7 Conclusion and Further Work
In this work, we investigate the mode collapse prob- lem when imitating latent policies from observations only. We show that mode collapse caused by combination of bad initialization and unfavorable properties of the expert demonstratinns. To address both issues, we proposed ILPO-MP, a clustering- based method to pre-train latent policy and la- tent dynamics model. We show that with this pre- training step, we can learn latent policies in envi- ronments where was previously not possible and improve imitation learning performance. Further- more, propose method to automatically decide important hyperparameter of latent policy learn- ing methods, the number of latent actions.
0.00
-0.01
Actian
Expet
0.02
Expect Action
0.4
Figure 9: Visualisation of the expert states, actions and transitions in the stochastic MountainCar experi- ment
In the future, we see potential in further exploring design choices in both latent policy learning and action remapping. In the action remapping step, the exploration mechanism a key component when obtaining high-quality data from the environments and greatly contributes to the data-efficiency of the method. In the latent policy learning step, more advanced representation learning methods leveraging the temporal smoothness of the expert data using sequences instead of single states as input could improve the quality of the learned latent policy and dynamics model.
References
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. Proceedings of the twenty-fire international conference on Machine learning, pp. 1, 2004.
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando De Freitas. Playing hard exploration games by watching youtube. Advances in neural information processing systems, 31, 2018.
13

---

Published in Transactions on Machine Learning Research (09/2023)
J Bagnell, Joel Chestnutt, David Bradley, and Nathan Ratliff. Boosting structured prediction for imitation learning. Advances in Newral Information. Processing Systems, 19, 2006.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. (03-129, 1995.
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, et al. End to end learning for sclf-driving cars. Xive preprint or dixi:16644003111 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Tong Che, Yanran Li, Athul Paul Jacob, Yoshua Bengio, and Wenjic Li. Mode regularized generative adversarial networks. arXiv preprint 1rXis:1612.0211 02136, 2016.
Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driv- ing via conditional imitation learning. In 2018 IEEE international conference robolics and aulomation (ICRA), pp. 4693 4700. IEEE, 2018.
Robert Dadashi, Léonard Hussenot, Matthieu Geist, and Olivier Pietquin. Primal wasserstein imitation learning. ar preprint arXiv:2006.00 0et.200060,678,, 2020.
Shreyansh Daftry, J Andrew Bagnell, and Martial Hebert. Learning transferable policies for monocular reactive mav control. In International Symposium on Experimental Robotics, pp. 11. Springer, 2016.
Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating latent policies from observation. In International conference on machine learning, pp. 1755 1763. PMLR, 2019.
Bin Fang, Shidong Jia, Di Guo, Muhua Xu, Shuhuan Wen, and Fuchun Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications, 3(4):362-369, 2019.
Arnaud Fickinger, Samuel Cohen, Stuart Russell, and Brandon Amos. Cross-domain imitation lcarning via
optimal transport. arXiv preprint arXiv:2110. 0.06684, 2021.
Chelsca Finn, Sergey Levine, and Picter Abboel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on mochine learning, pp. 49 49-58. PMLR, 2016.
Alessandro Giusti, Jérôme Guzzi, Dan C Ciresan, Fang-Lin Hc, Juan Rodríguez, Flavio Foutana, Matthias Faessler, Christian Forster, Jürgen Schmidhuber, Gianni Caro, et al. A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Letters, (22):661-667, 2015.
John C Gower. Goneralized procrustes analysis. Psychometrika, 40:33-50 51, 1975.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces to transfer skills with reinforcement learning. arXiv preprint arXiv: 1XX3.1103.000 1111170....0 02949, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information
processing systems, 29, 2016.
Stephen Johuson. Hierarchical clustering schemes. Psychometrika, 32(3):241 254, 1967.
Rahul Kidambi, Jonathan Chang, and Wen Sun. Mobile: Model-based imitation lcarning from observation alone. Advances in Newral Information Processing Systems, 34:28598 28611, 2021.
David Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding, arxiv preprint arXic:2007.10930 2020.
Wojtek J Krzanowski. Principles of multivariate analysis. Oxford University Press, 2000.
14

---

Published in Transactions on Machine Learning Research (09/2023)
YuXuan Liu, Abhishek Gupta, Picter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 TEEE International Conference on Robotics and Automation (ICRA), pp. 1118 1125. LEEE, 2018.
Chaochao Lu, José Miguel Hernández-Lobato, and Bernhard Schölkopf. Invariant causal representation learn- ing for generalization in imitation and reinforcement learning. In ICLR2022 Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Martin. What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXi::2188.022 032998, 2021.
Donald Michie, Michael Bain, and J Hayes-Miches, Cognitive models from subcognitive skills. IEE control engineering series, 44:71-99, 1990.
Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining self-supervisod learning and imitation for vision-basod rope manipulation. In 2017 IEEE international conference on robolics and aulomation (ICRA), pp. 2146 2153. IEEE, 2017.
Andrew Y Ng, Stuart Russell, ct al. Algorithms for inverse reinforcement learning. In Ioml, volume 1, pp-
2, 2000.
Yunpeng Pan, Ching-An Cheng, Kamil Saigol, Keuntack Lee, Xinyan Yan, Evangelos Theodorou, and Byron Boots. Agile autonomous driving using end-to-end deep imitation learning. arXiv preprinl 1rXiv:1709.000 2017.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88 97, 1991.
Tijaa Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. State-only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp.
IEEE, 2021.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. btable-baselines:: Reliable reinforcement learning implemennaations. Journal of Machine Learning Re- search, 22(268):1-8 2021. URL http://jmlr ortp/paperrs//22222-11344 htmll
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the thir- teenth international conference on artificial intelligence and statistics, pp. 661 668. JMLR Workshop and Conference Proceedings, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured pre- diction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 227-635. JMLR Workshop and Conference Proceedings, 2011.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory, pp. 101-103, 1998.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Yannick Schroecker and Charles L Isbell. State aware imitation learning. Advances in Neural Information Processing Systems, 30, 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. arXio preprint arXiv: 2017.
Pierre Sermanct, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sorgey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference robotics and audomation (ICRA), pp. 1134-1141. TEFE, 2018.
15

---

Published in Transactions on Machine Learning Research (09/2023)
Jonathan Spencer, Sanjiban Choudhury, Arun Venkatraman, Brian Ziebart, and J Andrew Bagnell. Feedback in imitation learning: The three regimes of covariate shift. arXiv preprint urXin:210..0280 2021.
Oliver Struckmeier, Kshitij Tiwari, and Ville Kyrki. Autoencoding slow representations for semi-supervised data-efficient regression. Machine Learning, pp- 1-19, 2023.
Wen Sun, Anirudh Vemula, Byron Boots, and Drew Bagnell. Provably efficient. imitation learning from observation alone. In International conference on machine learning, pp. 6036 6045, PMLR, 2019.
Hoang Thanh-Tung and Truyen Tran. Catastrophic forgetting and mode collapse in gans. In 2020 interna- tional joint conference on neural networks (ijenn), pp. 10. IEEE, 2020.
Faraz Torabi, Garrett Warnell, and Peter Stone. arXiv:1805.0100 2018.
Behavioral cloning from observation. ar preprint
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observation. arXiv preprinl ar Xiv: 1905. 13566, 2019.
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhon Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. Aduances in neural information processing systems, 32, 2019.
Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant repre- sentations for reinforcement learning without reconstruction. arXiv preprint arxi2:0006.. 10742, 2020.
16

---

Published in Transactions on Machine Learning Research (09/2023)
A Appendix
A.1 Experiment Details
Our implementation is based on the ILPO implementation by Edwards et al. (2019 and ported from Ten- sorFlow to Py PyToc.. We aim to publish our code on GitHub after publication.
We used the same neural network architectures and same hyperparameters for latent policy learning and remapping as EEwwrds et al. (2019). The only change being that we use 50 instead of 100 steps for the latent policy learning. For the pre-training we use 20 epochs for the Acrobot, Cartpole, MountainCar and LunarLander experiment and 30 for the Pong and Pixel Pong experiment.
We used 12 seeds for the latent policy learning and recorded 25 remapping experiments for each latent policy. Every task performance plot therefore comprised of 300 experiment runs per method. We made sure to maintain the same initialisation of network weights (Xavier uniform for weights and zeros for biases) since we found that using different initialisation, such as the standard PyTorch weight initialisation leads to differences in performance when compared to the TensorFlow implementation. This further strengthens our finding that ILPO is very sensitive to model initialisation.
The Pong environment is a custom environment we developed conforming to the OpenAl Gym API. The enviromment is available on github The LunarLander environment in its baseline version is stochastic and the other environments are deterministic. For the experiments, we made stochastic versions of each environment, most notably the MountainCar environment which we discuss in this paper. The stochastic MountainCar environment applies uniform noise to the force applied to the cart.
The number of latent actions used for ILPO was selected based on the number of available actions in the enviromment. For ILPO-MP determined the number of latent actions described in 6.3 Table 2 shows the number of latent actions used for ILPO-MP.
Table 2: Number of latent actions used the experiments for ILPO-MP.
Environment
Expert
Latent actions
Acrobot
Det. Expert
19
Cartpole
Det. Expert
2
MountainCar
Det. Expert
5
MountainCar
Noisy Expert
3
Lander
Det. Expert
12
Pong (vector)
Det Expert
6
Pong (vector)
Noisy Expert
8
Pong (pixel)
Det Expert
6
Pong (pixel)
Noisy Expert
8
The experts we used have been trainod using the PPO implementation (Schulman al.] 2017] from stable baselines3 (Raliin al., 2021). Deterministic expert data. has been recorded by taking the mode of the PPO action distribution and noisy data by uniformly sampling from the action space 20% of the time. Like Edwards et al. (2019) we recorded 50000 samples consisting of state and next state from each expert.
17


Reviews:
[{"review_id": "THVG71lOCN", "reviewer": "Reviewer_bKym", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n\n+The paper studies a relevant and significant questions of mode collapse in imitation learning. Although much of the text is about a specific method, ILPO, but I believe it could shed some light of other approaches as well, as action ambiguity has been plagued ILfO, even for IRL-based method like [1], which has also flagged similar issues.\n\n+Detailed analysis of the model collapse problem in the context of ILfO and ILPO, with both empirical evidence and theoretical analysis, motivint the solution pretty well.\n\n+The proposed method, ILPO-MP, which uses data cluster labels to anchor the latent policy, is technically sound and has demonstrated its effectiveness on adressing the mode collapse issue. The method is backed with comprehensive results, and the authors are very transparent on many of their boundaries and limitations (with related results as well). I appreciate it.\n\n+The authors suggest a method to select the optimal number of latent actions/clusters, which is a useful addition to the learning process.\n\n[Weaknesses]\n\n-Although it's ok to focus on a certain approach, how the authors shape the presentation of this paper could potentially impair its impact, as it does not seem clear how can this issue of mode collapse and their solution be elevated to other imitation learning from observation schemes, or imitation learning in general.\n\n-Ovearll, the scope of the experiment section could be a bit limited, as most results are from MountainCar, CartPole, Pong, etc, which are relatively simple for ILfO. This could raise questions on the generalization and real-world performances of the proposed method. The authors are suggested to include more challenging domains, ex. MuJoCo tasks, more Atari games, etc. \n\n-While I agree that a TMLR submission could simply focus on addressing the issue of a certain algorithm, but it would help the readers understand the big picture of the field and the position of the method here if some existing ILfO solutions are compared. Also, since ILPO is posed as a latent variable model learning problem, some other approaches on addressing mode collapse, ex. those from the generative learning community, could be compared here as well.\n\nMinor:\n\n-[1] raises similar issue mentioned in this manuscipt, i.e. ill-posed inverse dynamics and indeed is a concurrent work to Sun et al., 2019. The authors should consider citing this paper.\n\n[1] https://arxiv.org/abs/1910.04417", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript focues on understanding and addressing the model collapse issue of a imitation learning from observation algorithm called ILPO. The authors start off with some theoretical analysis on how mode collapse could happen in case of a deterministic expert and bad model initialization, and empirically demonstrate how can the issue be mitigated with the help of a noisy expert. Further, a method called ILPO-MP is proposed to alleviate mode collapse without the need for noisy expert, which could be unrealistic in some tasks. The main idea is to use the cluster labels of the observations (transitions) to anchor the latent policy as a mean of pretraining. Moreover, the authors introduces guided ILPO-MP, which tag the pretraining loss as an auxllilary task during ILPO when the training episodes are long, ex. vision-based learning. There are comprehensive experimental results on several environments to justify the the effectiveness of the proposed method, especially when in absense of noisy demonstrations. Some additional analysis on domains that ILPO-MP does not work (Acrobot), selection of hyperparameters, initialization, environment stochasticity are also provided.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See [Weaknesses]", "broader_impact_concerns": "See [Weaknesses]", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tzC5tMr2Ro", "reviewer": "Reviewer_BpQB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n+ The paper is very well written and easy to follow.\n\n+ The proposed approach to solving the mode collapse issue when doing imitation learning from observations only is well motivated via a detailed analysis on toy examples, and is also simple to implement.\n\n**Weaknesses:**\n\n- My main concern is how the assumption that \"*the same actions in the expert data often lead to similar transitions*\" is practical in real world-like applications. Although Table 1 shows the assumption probably holds true in simple RL environments, it is still doubtful if it can be also satisfied in more challenging RL environments with high-dimensional and complicated sensory input. Furthermore, how valid is the assumption especially when the irrelevant backgrounds change in the environments (e.g., section 6.1 of [https://arxiv.org/pdf/2006.10742.pdf](https://arxiv.org/pdf/2006.10742.pdf))?\n\n- At the current stage, the main experiments are only limited to three simple RL environments: CartPole, LunarLander, and Pong. I suggest the authors conduct more experiments and analysis on more complex environments to further demonstrate their assumption and the proposed approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a simple approach to alleviating the mode collapse issues when imitating latent policies from observational data only, not relying on expert actions. The main idea is, by clustering expert transitions, to impose a mode prior on the generative model and the latent policy, so that it is less likely to initialise the model leading to model collapse. Experiments on CartPole, LunarLander, and Pong demonstrates the validity of the proposed method.  ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the **Weaknesses** above.", "broader_impact_concerns": "The authors do not describe the ethical implications of this work in the paper. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "iFcQFafu3a", "reviewer": "Reviewer_QVpZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper tackles an important problem\n- Interesting analysis on the effect of dataset for learning latent policies\n- Improvement over ILPO on considered tasks\n\n**Weaknesses**\n- Paper is not formal and not self-contained as several notations are not formally defined and the paper assumes the knowledge of various techniques without providing the background. This makes it very difficult to fully understand the paper \n- Assumption that the clean expert demonstrations is more prevalent is questionable.\n- Assumption on the piecewise continuous dynamics is not clearly defined so it's not sure what it exactly means, and this makes it difficult to expect how this proposed method can scale to more complex setups.\n\n**Summary**\nThe paper tackles an interesting problem and proposes a method based on interesting analysis and it has a substantial potential to be a useful method for this domain. But current writing of the draft makes it fully understand what's going on and what does several components exactly mean. And it's missing empirical support on the important assumptions made in the paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This paper is concerned with tackling the problem of imitation learning from observations where the expert actions are not available from the dataset. This is an important problem because it would enable us to use a vast amount of action-free videos, like YouTube videos, for behavior learning. \n- The main contribution of this paper is to improve the previous method for this setup, i.e., Imitating Latent Policies from Observation (ILPO), by (a) identifying the problem of ILPO where latent policies collapse and thus fail to learn useful latents from expert demonstrations and (b) addressing the problem by imposing a mode prior based on clustering of the expert demonstrations.\n- This paper provides interesting analysis on how the mode prior can happen per the characteristics of the datasets\n- This paper provides experimental results that show the proposed method can improve the performance of ILPO when the datasets are very clean, e.g., collected with deterministic expert policies.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\nImprovement in writing is crucial for recommending the acceptance of this paper. Currently it's not formal and writing makes it difficult to fully understand the paper.\n- It's not clear what $\\Delta_{t}$ is from the paper. I had to have a look at the original ILPO paper for more formal definition on the notations. From my understanding, G is predicting the difference $\\Delta_{t}$ between $s_{t}$ and $s_{t+1}$  instead of directly predicting $s_{t+1}$. Then equation (6) does not make sense, because G that predicts the difference is trained to regress the full state $s_{t+1}$. The paper should add a mention that the notation is a bit misused for clarity like as in original ILPO paper, or should try to make the notation and equation be more formal.\n- Understanding this sentence seems very important for the paper but it's really difficult to understand: `Instead, we base our mode prior on the observation that the same actions in the expert data often lead to similar transitions`. What does 'transitions' exactly mean here, and what does the 'same actions in the expert data often lead to similar transitions' mean? How can same $a_{t}$ lead to the similar transition $(s_{t}, a_{t}, s_{t+1})$? After reading all the pages, it seems to me that it meants `same actions often lead to similar state differences`, but it's not clear. \n- Above point really makes it difficult to understand the exact meaning and interpretation of the analysis in section 4.2. It's not clear what does 'state' and 'transition' mean here, and what does the distance exactly mean. Adding more explanation on how to interpret the figures in Fig 3 and Fig 4 could be helpful for making the readers understand exactly what's going on.\n- Moreover, what does 'Procrustes disparities' mean here? Please provide the intuitive meaning of the distance and also provide the reference, without assuming such knowledge from the readers.\n- Explaining the reason behind `We observed that the latent policy learning can end up in a collapsed state despite the pre-training when the latent policy learning is running for many epochs.` should be more helpful for understanding the intuition behind introducing Guided ILPO-MP\n- For self-contained writing, please add the reference to agglomerative clustering. And providing more clearer definition and examples for $z^{l}$ should be helpful.\n\nThe paper also needs a more support on the assumption on the datasets.\n- The paper claims that, unlike the prior work that claimed humans usually collect noisy data, humans are more likely to collect the very clean dataset without noises. But it seems much more natural to assume that humans are likely to collect the noisy data -- for instance, see RoboMimic [Mandlekar et al., 2021] dataset where average humans usually collect very noisy demonstrations and also proficient humans often fail to collect perfectly clean data. The paper can much be more strengthened by providing some experimental evidences to support this claim.\n- The paper assumes that dataset follows piecewise continuous dynamics. But first of all, it's not clear what does this exactly mean, so it would be nice to be more formal on this front. Moreover, it's not clear how this would hold for more complex datasets, for instance, locomotion tasks with quadrupeds or humanoids. And this assumption could severly limit the applicability of the proposed method in pixel-based environments except simple environments like Pong where the visual observation is very clean. The paper can much be more strengthened by providing additional experimental results on complex environments. For instance, the paper could provide experimental results on DMC with bang-bang control [Seyde et al., 2021] to see that the method can work on more complex locomotion tasks and this should be feasible because the action space is discrete with bang bang controller. And also the paper could consider conducting additional experiments on CoinRun as in the original ILPO paper to show that the proposed method can work on more visually complex environments.\n\n[Mandlekar et al., 2021] Mandlekar, Ajay, et al. \"What matters in learning from offline human demonstrations for robot manipulation.\" arXiv preprint arXiv:2108.03298 (2021).\n[Seyde et al., 2021] Seyde, Tim, et al. \"Is bang-bang control all you need? solving continuous control with bernoulli policies.\" Advances in Neural Information Processing Systems 34 (2021): 27209-27221.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers acknowledge the importance of the problem studied in this paper and the value of the analysis performed by the authors and of the solution they designed to address the mode collapse problem. Nevertheless, the reviewers expressed concerns regarding the clarity of some aspects of the paper, including its scope as well as some more technical notation, and regarding the experimental evaluation, limited to relatively simple RL environments. The authors addressed these concerns in their responses, convincing two reviewers to recommend acceptance (Leaning Accept). The third reviewer failed to provide their final recommendation. The reviewers' main reason to remain on the fence is the experimental evaluation.

The AE believes that the authors have convincingly justified their current choice of evaluation environments, and that there is sufficient value in this work for it to be accepted. As the authors have already updated their manuscript based on their responses to the reviewers, the paper can be accepted as is.
Decision: Accept (poster)