Venue: ICLR 2022 Track Blog

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: # Section "Rethink ValueDice in the Offline Setting"


The conclusions from Theorem 1 are based on (1) using a deterministic expert policy and (2) using a dataset of state-action tuples. However, in ValueDICE/GAIL/DAC the expert policy is (1) non-deterministic and (2) the dataset is of state-action-next-state tuples from a combination of expert and non-experts (one of the critical novelties of these AIL methods is to utilize this extra transition info). Therefore, it would be great for the authors to either adjust or clarify the limitations of Theorem 1, as it does not apply directly to the case considered in the ValueDICE paper. In fact, the blog post’s Rajaraman citation mentions an improvement in the minimax lower-bound when an imitation learning algorithm has knowledge of the MDP’s transition dynamics.

It would also be great for the blogpost to more concretely clarify to the reader the ramifications of a “minimax” bound. Namely, despite the existence of worst-case synthetic scenarios where BC and ValueDICE are equivalent, there may be many other practical instances where ValueDICE significantly outperforms BC (similar to the tabular example in Fig 1 of the ValueDICE paper).

# Section "Connection Between ValueDice and BC"

This section had some great insights, showing that ValueDICE can have a solution similar to BC under certain conditions. However, the conclusion/summary of this section could be clarified in a few ways. Namely, in contrast to BC, one of the key novelties of ValueDICE is that it can incorporate additional state-action transitions, and these can give an advantage over BC in the tabular case (see again Fig 1 in the ValueDICE paper). Moreover, the ideas from ValueDICE can be extended to offline reinforcement learning (AlgaeDICE, CQL) unlike BC.

# Section "Overfitting of ValueDice and BC"

It would be nice for the blogpost to note that the original ValueDICE paper aimed for a fair comparison between ValueDICE and BC by maintaining the same implementation details such as architecture and regularizations for both algorithms (e.g., the BC implementation in ValueDICE uses the same orthogonal regularization as ValueDICE:
https://github.com/google-research/google-research/blob/master/value_dice/twin_sac.py#L323). 

Conversely, the BC policy parameterization and regularization proposed by the blogpost (or other implementation changes) might improve the performance of ValueDICE in the blogpost’s Fig 2. Of course, extensive architecture and hyperparameter searches are time consuming, but it would still be good for the blogpost to highlight these limitations of their conclusions. 

# Section "Rethink ValueDice in the Online Setting"

"More surprisingly, ValueDice successfully performs policy optimization without the useful technique target network. This contradicts common sense that optimization of Bellman backup objectives with function approximation can easily diverge (see the counter-example in [Baird, 1995];"

This statement seems incorrect at first glance: Baird, 1995 demonstrates that the residual algorithm, which does not use a target network, converges.

# Footnote 2: “Our implementation of BC is different from the one in [Kostrikov et al., 2020]. Unlike the implementation in [Kostrikov et al., 2020], our BC policy does not learn the covariance. Our implementation is more nature and is widely used in the previous literature [Ho and Ermon, 2016, Kostrikov et al., 2019].”

It is important for the blogpost to emphasize that parameterization of the BC baseline in [Ho and Ermon, 2016, Kostrikov et al., 2019] is inherited from PPO since the algorithm achieved state-of-the-art performance in online RL. However, the parameterization in [Kostrikov et al., 2020] is taken from SAC, which significantly improves over PPO.
review_title: Great post but requires some tuning

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: **Is there a significant added value in comparison to the cited papers?**
yes, authors provide a set of experiments comparing ValueDice to regularized behavioral cloning in offline setting while giving useful insights on the relation between both of them.

**Is this added value supported by accurate, convincing, and clear arguments?*i
The authors showed the relation between behavioral cloning and ValueDice based on a set of experiments and a theoretical analysis. 
They showed that behavioral cloning objective is one of global optimum for ValueDice. Then, they created a set of experiments comparing ValueDice to behavioral cloning showing that both are closely related. Overall, the blog post show an interesting analysis to ValueDice in both offline and online setting while showing the limitations of existing benchmarks.
review_title: Review

Decision: Accept