Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

The authors propose a method to make exploration in really sparse reward tasks more efficient. They propose a method called Workflow Guided Exploration (WGE) which is learnt from demonstrations but is environment agnostic. Episodes are generated by first turning demonstrations to a workflow lattice. This lattice encodes actions which are in some sense similar to those in the demonstration. By rolling out episodes which are randomly sampled from this set of similar actions for each encountered state, it is claimed that other methods like Behavor Cloning + RL (BC-then-RL) can be outperformed in terms of number of sample complexity since high reward episodes can be sampled with much higher probability.

A novel NN architecture (DOMNet) is also presented which can embed structured documents like HTML webpages.

Comments:

- The paper is well-written and relevant literature is cited and discussed.
- My main concern is that while imitation learning and inverse reinforcement learning are mentioned and discussed in related work section as classes of algorithms for incorporating prior information there is no baseline experiment using either of these methods. Note that the work of Ross and Bagnell, 2010, 2011 (cited in the paper) establish theoretically that Behavior Cloning does not work in such situations due to the non-iid data generation process in such sequential decision-making settings (the mistakes grow quadratically in the length of the horizon). Their proposed algorithm DAgger fixes this (the mistakes by the policy are linear in the horizon length) by using an iterative procedure where the learnt policy from the previous iteration is executed and expert demonstrations on the visited states are recorded, the new data thus generated is added to the previous data and a new policy retrained. Dagger and related methods like Aggrevate provide sample-efficient ways of exploring the environment near where the initial demonstrations were given. WGE is aiming to do the same: explore near demonstration states.
- The problem with putting in the replay buffer only episodes which yield high reward is that extrapolation will inevitably lead the learnt policy towards parts of the state space where there is actually low reward but since no support is present the policy makes such mistakes. 
- Therefore would be good to have Dagger or a similar imitation learning algorithm be used as a baseline in the experiments.
- Similar concerns with IRL methods not being used as baselines.

Update: Review score updated after discussion with authors below. 

review_title: WGE seems cool but why not have an IL or IRL baseline?

final_score: rating: 6: Marginally above acceptance threshold
review_content: SUMMARY

The paper deals with the problem of training RL algorithms from demonstration and applying them to various web interfaces such as booking flights.  Specifically, it is applied to the Mini world of Bids benchmark (http://alpha.openai.com/miniwob/).

The difference from existing work is that rather than training an agent to directly mimic the demonstrations, it uses demonstrations to constrain exploration. By pruning away bad exploration directions. 

The idea is to  build a lattice of workflows from demonstration and randomly sample sequence of actions from this lattice that satisfy the current goal.    Use the sequences of actions to sample trajectories and use the trajectories to learn the RL policy.



COMMENTS


In effect, the workflow sequences provide more generalization than simply mimicking, but It not obvious, why they don’t run into overfitting problems.  However experimentally the paper performs better than the previous approach.

There is a big literature on learning from demonstrations that the authors could compare with, or explain why their work is different.  

In addition, they make general comparison to RL literature such as hierarchy rather than more concrete comparisons with the problem at hand (learning from demonstrations.)

What does DOM stand for?  The paper is not self-contained.  For example, what does DOM stand for?  


In the results of table 1 and Figure 3.  Why more steps mean success?

In equation 4 there seems to exist an environment model.  Why do we need to use this whole approach in the paper then?  Couldn’t  we just do policy iteration?

review_title: The paper has good empirical results, but algorithmic novelty is small.  It could benefit from more concrete comparisons in the literaure

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a new exploration policy for Reinforcement Learning for agents on the web called "Workflow Guided Exploration". Workflows are defined through a DSL unique to the domain.

The paper is clear, very well written, and well-motivated. Exploration is still a challenging problem for RL. The workflows remind me of options though in this paper they appear to be hand-crafted. In that sense, I wonder if this has been done before in another domain. The results suggest that WGE sometimes helps but not consistently. While the experiments show that DOMNET improves over Shi et al, that could be explained as not having to train on raw pixels or not enough episodes.
review_title: Interesting, but hard to know the novelty

Meta-review: 
PROS:
1. well-written and clear
2. added extra comparison to dagger which shows success
3. SOTA results on open ai benchmark problem and comparison to relevant related work (Shi 2017)
4. practical applications
5. created new dataset to test harder aspects of the problem

CONS:
1. the algorithmic novelty is somewhat limited
2. some indication of scalability to real-world tasks is provided but it is limited
Decision: Accept (Poster)