Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes the application of the f-VIM framework (Ke et. al., 2019) to the problem of imitation learning from observations (no expert actions). The authors first identify a potential source of numerical instability in the application of f-VIM to imitation learning – the rewards for the policy-gradient RL are given by a combination of a convex conjugate and an activation function. To alleviate this, f-VIM is reparameterized by curating the activation using conjugate inverse (Equation 8), yielding a potentially more stable reward for deep-RL. 

I have the following concerns about the paper:

1.	Lack of novelty – Although I appreciate the reparameterization applied to f-VIM to make it potentially more stable for imitation learning in large state- and action-spaces, I don’t think that by itself meets the bar for ICLR. Algorithm 1 is basically the GAILFO algorithm (Torabi et al. 2018) written in the f-Vim framework, with the proposed reparameterization. The discriminator regularization (Section 4.4) has been used before.

2.	Experiments – Figure 2 shows the improvement with TV when using the reparameterization, and the authors mention in text about the difficulty with KL and reverse-KL. What about the JS divergence (GAIL)? Does reparameterization help or affect that?

3.	In Figure 3, is GAIL from the original paper, or does it use the sigmoid rewards? Figure 3 does not offer any evidence that the proposed methods in the paper lead to algorithms that should be preferred over the current state-of-the-art in imitation learning with divergence minimization such GAIL and WAIL.

Minor comment:
In Table 1: GAN is not a divergence. Please use Jensen-Shannon, with the corresponding tweaks to the columns. 
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Summary: The submission performs empirical analysis on f-VIM (Ke, 2019), a method for imitation learning by f-divergence minimization. The paper especially focues on a state-only formulation akin to GAILfO (Torabi et al., 2018b). The main contributions are:
1) The paper identifies numerical proplems with the output activations of f-VIM and suggest a scheme to choose them such that the resulting rewards are bounded.
2) A regularizer that was proposed by Mescheder et al. (2018) for GANs is tested in the adversarial imitation learning setting.
3) In order to handle state-only demonstrations, the technique of GAILfO is applied to f-VIM (then denoted f-VIMO) which inputs state-nextStates instead of state-actions to the discriminator.

Contribution / Significance:
I think that the contributions of the paper are rather marginal. I do think that the choice of output activation may have large impact on the performance and it seems that the activation suggested by Ke et al. (2019) are somewhat arbitrary. However, the activations proposed in the current submission are also seem somewhat arbitrary and are not accompanied by any theoretical analysis. 
2) and 3) are marginal combinations of existing work that are only insufficiently evaluated and do not seem particular effective.
Hence, I think that the current submission is of rather limited interest.

Soundness:
The "reparametrization" of f-VIM is motivated based on exploding policy gradients when using unbounded reward functions, especially when minimizing the (R)KL. 
I am not convinced by this motivation, given that GAIL and AIRL (which approximatly minimizes the RKL) use unbounded reward functions and do not seem to suffer from such problems.

Evaluation:
The effect of the "reparametrization" is only evaluated for total variation. The regularization loss is only evaluated with a single fixed coefficient of 10 on all experiment. I think that a sweep over the coefficient would be mandatory, especially given that current experiments do not show a clear benefit of the regularization loss (the regularized version performs worse on roughly half of the experiments). 
When learning from observations only, the submission only evaluates the proposed combination of f-VIM and GAILfO. However, it seems like it would be perfectly possible to handle state-only observations by simply making the discriminator independent of the actions, i.e. using D(s,a) = D(s). Such technique matches the marginal distributions over states and is commonly applied to GAIL, e.g. by Peng et al. [1].
It is not clear whether the reported problems of learning from observations only is really a general problem of the learning setting (as claimed in the submission) or a problem of the proposed method.

Clarity:
The paper is well written and easy to follow. Using different linestyles to distinguish the learning with regularization versus without regularization would help a lot.

Decision:
Due to the marginal contribution and the insufficient evaluation I have to recommend rejection.


Question:
I am maily interested in the authors response to my critique, especially regarding
- the choice not to compare with state-only f-VIM, and
- the motivation of the proposed output activations.


[1] Peng, Xue Bin, et al. "Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow." arXiv preprint arXiv:1810.00821 (2018). 
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: * Summary:
The paper proposes an IL method based on the f-divergence. Specifically, the paper extends f-VIM (Ke et al., 2019), which uses the f-divergence for IL, by using a sigmoid function for discriminator output’s activation function. This choice of activation function yields an alternative objective function, where the reward function for an RL agent does not directly depend on the convex conjugate of the function f; the paper claims that this independency improves stability of policy learning. This proposed method is named f-VIM-sigmoid. The paper extends f-VIM-sigmoid to the setting of IL with observation and proposes f-VIMO-sigmoid. Experiments on Mujoco locomotion tasks show that f-VIM-sigmoid and f-VIMO-sigmoid perform better than existing methods.

* Rating:
The paper proposes a simple but interesting approach to improve stability of adversarial IL. However, the paper has issues regarding baseline methods, motivation, supports of the claim, and experiments (see below). These issues should be addressed. At the present, I vote for rejection.

* Major comments:
- Discussion and comparing against a simple baseline method based on swapping distributions: 
To make the reward function be independent of the convex conjugate f*, it is possible to simply swapping the distributions P and Q in the definition of the f-divergence. More specifically, instead of minimizing D_f(P||Q), we can minimize D_f(Q||P), where P is a data distribution and Q is a generator. In this case, pi* and pi_theta in Eq. (7) swap, and the RL agent minimizes the cost function g_f(V_w(s,a)). This cost function does not directly depend on f*, similarly to the reward function r(V_w(s,a)) in Eq. (8). This swapping is simpler and more flexible than re-parameterizing, while achieving the same goal as f-VIM-sigmoid. This swapping method should be discussed and compared against the proposed methods.

- Need stronger baseline methods for ILfO: 
The paper should evaluate f-VIMO-sigmoid against stronger baselines, e.g., forward adversarial IL (Sun et al., 2019) which outperforms GAIL-based methods in the ILfO setting. 

[1] Wen Sun, Anirudh Vemula, Byron Boots, and J Andrew Bagnell. Provably efficient imitation learning from observation alone. ICML, 2019.

- Using the f-divergence for ILfO is not well motivated: 
The paper does not provide good motivations for using f-divergence in ILfO. This makes the paper quite difficult to follow, since there is no connection between f-divergence and ILfO.

- The experiments focus on evaluating existing methods rather than the proposed methods: 
Specifically, the proposed methods are evaluated with only one choice of the divergence (TV) in Figure 2. Meanwhile, most of Section 6 and results (Figure 3 and 4, and additional results in the appendix) focus on evaluating the existing methods (f-VIM and f-VIMO) with different choices of divergence. 

- The experiments in Figure 2 do not support the claim regarding stability: 
The paper claims to improve stability of IL by using the proposed re-parameterization. However, the experimental results do not support this claim, and the questions asked in Section 5 are not related to this claimed. Instead, it seems that re-parameterization helps avoiding local optima (possibly due to a biased reward function, see below), while stability is improved by regularizing the discriminator. I could not see how the re-parameterization improves the policy stability as claimed. 

- The experiments in Figure 2 seem unfair, since TV-VIM-sigmoid incorporates priors about survival bonuses: 
Specifically, TV-VIM-sigmoid uses sigmoid which yields strictly positive rewards, while TV-VIM uses tanh which yields positive and negative rewards. As discussed by Kostrikov et al., 2019, using strictly positive rewards incorporate strong priors about the survival bonuses, which exist in the locomotion task used in the experiments. Therefore, TV-VIM-sigmoid uses strong priors while TV-VIM does not. In order to make the comparison fairer, I suggest the authors to evaluate TV-VIM with sigmoid reward output, or include environments that do not have survival bonuses.

[2] Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. ICLR, 2019

* Minor comments: 
- The abstract is long and could be shorten. 
- Figures are too small and difficult to see, especially the legends.
- Table 1 should describe the form of f in addition to its conjugate.
- The title of the Algorithm 1 should be f-VIMO-sigmoid instead of f-VIMO. 

** Update after response.
I read the response. I thank the authors for clarifying the claims as well as the new experiments with the swap formulation. However, improving clarity of the claims is considered a major revision. I still keep the vote of rejection. 

Regarding reward bias. As the authors acknowledge, the improvement achieved by using reparameterization+sigmoid can be explained by two equally-plausible reasons: 1) reparameterization+sigmoid improves stability (as claimed) and 2) sigmoid gives biased rewards. The issue here is that we do not know which is the actual reason, given the current experiments in the paper. As I commented, evaluating TV-VIM with sigmoid but without reparameterization will help address this issue.

review_title: Official Blind Review #2

Meta-review: The submission performs empirical analysis on f-VIM (Ke, 2019), a method for imitation learning by f-divergence minimization. The paper especially focues on a state-only formulation akin to GAILfO (Torabi et al., 2018b). The main contributions are:
1) The paper identifies numerical proplems with the output activations of f-VIM and suggest a scheme to choose them such that the resulting rewards are bounded.
2) A regularizer that was proposed by Mescheder et al. (2018) for GANs is tested in the adversarial imitation learning setting.
3) In order to handle state-only demonstrations, the technique of GAILfO is applied to f-VIM (then denoted f-VIMO) which inputs state-nextStates instead of state-actions to the discriminator.

The reviewers found the submitted paper hard to follow, which suggests a revision might make more apparent the author's contributions in later submissions of this work. 
Decision: Reject