Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper presents a technique for adversarial generation of environments for the interesting problem of web navigation, and provides an environment that enables learning to design complex websites out of a set of compositional primitives. Then, it also proposes a method to adversarially generate a curriculum of increasingly complicated websites, and uses it to train agents which can navigate more challenging, high-dimensional websites.

Strengths:
1. An interesting novel problem domain; which is going to be very useful in a number of human computer interaction applications.
2. The web navigation environment is interesting - it will hopefully spur more research for this problem
3. The training of agents with an autoregressive adversary policy is interesting.

Weaknesses:
1. The discussion on related work on this application seems sparse; hence, for me, it was hard to judge the novelty of this work.
2. More discussion of the environment - some examples, what makes it hard, or easy would help the reader understand the key challenges.
3. More discussion on past work in interactive learning with autoregressive adversarial policies would be helpful. It will help the reader understand why this is a different interactive task and what makes it more interesting or challenging.
4. The experimental section is too sparse. Some more ablation studies on different parts of the model - e.g. budget enforcing on the adversary would be helpful.
5. The b-paired and flexible b-paired agents seem to be very similar to each other - especially for some problems. Some more analysis of this would be useful.

review_title: Interesting application, seemingly solid work

final_score: rating: 5: Marginally below acceptance threshold
review_content: The main research contribution of this paper is the Flexible-PAIRED, and bPAIRED algorithms for generating Web Sites that can be then "used" by automatic agents. The modifications to the original PAIRED algorithm is quite simple but proven to be effective in the paper.

Honestly, I believe the paper has some merit but in the current form is quite difficult to be understood.

First and foremost, it is really not clear what are the motivations behind the research work. I understand that you want to learn a policy for filling in a web form but, honestly, I am not sure I got that. The examples you make are not super clear, you should give an idea, for instance, of what's the complexity of the tasks in the examples. For instance, booking a flight to LAX for Friday involves: i) inputting LAX in an entry box, clicking search button, select flights according to user preferences (how are these taken into account, BTW?) and then pay using a credit card. Now, all of these passages are quite difficult and complex and I am not sure the technique presented in the paper actually can do that. From the paper it is not clear what are the tasks considered and how they are generated.

The second question I have regards the relevance of the results. While the difference between PAIRED and Flexible-PAIRED and Flexible-bPAIRED are large, it is not clear that a success rate of 80% can be considered high enough for real systems.

One thing that should be improved is readability, in general. For instance, the description of the PAIRED algorithm is quite difficult to follow. I understand that this is not an algorithm that is developed by the authors but the description should be done in a correct way that will make the reader understand what are the shortcomings and why you had to propose Flexible-PAIRED and bPAIRED.

A relatively minor point that needs explanation regards the impact of the simplification of the generation process described in Section 4. It is not clear if the limitations of the primitives used also limit greatly the diversity of the generated pages making the task simpler.

It is true, though, that results are much better than the baseline and for this reason I believe the paper has some merit.

Minor issues:
1. The first sentence in the Introduction is difficult to understand and does not set correctly the goals of the paper. What do you learn to generate? DOMs or Tasks + DOMs?
2. The regret will be highest for easy --> the regret will be higher
3. "assume links between pages are implicitly defined by events attached to certain elements." --> This sentence is not clear
review_title: Generating Environments through Flexible-(b)PAIRED Algorithms

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper studies the problem of learning to autonomously navigate the web, such as filling out web forms. It proposes a curriculum learning method that uses Adversarial Environment Generation (AEG) to build a curriculum of challenging web navigation tasks. It is based on the idea of creating training environments for RL agents. The paper is well written and easily accessible. The problem in this work is an interesting application of RL. The idea is not very new and extends from previous works using AEG.

It is difficult to compare the difference between the paper and PAIRED (Dennis et al., 2020). Because PAIRED (Dennis et al., 2020) is not public yet. However, as described in the related work, it seems like that the paper just applies AEG methods on complex, high-dimensional environments with some extension. The novelty of the work is limited. 

In addition, the GAN part is also similar to Goal GAN (Florensa et al., 2017). The difference is how to label an environment (or a goal in Goal GAN). It is better to have more details about the adversary architecture. For example, how to label environments and train the adversary architecture.

It is unclear why the Adversarial Environment Generation (AEG) can provide a curriculum of learning different tasks. It is worth providing more experimental results about the progress of curriculum learning. 

Overall, the main weaknesses are the novelty of the idea compared to previous works and how curriculum learning works for navigating the web tasks.
review_title: The problem is interesting. The paper extends previous works and lacks details and results about their designed curriculum.

final_score: rating: 7: Good paper, accept
review_content: This paper improves upon existing approaches for learning to fill forms on the web automatically. The main idea is to train an adversary to generate a curriculum of environments to train an agent to learn to fill forms on the web. Training such an adversary can be challenging since the adversary may prove to be too strong for the main agent to learn anything from. Thus, the paper proposes few techniques to control or shape this adversary such that the main agent is able to learn quickly as compared to similar existing approach. 

The paper is well-written and clear for most parts. I feel the improvements the paper suggests are insightful and significant. The empirical analysis is also quite satisfactory and clearly shows the superiority of the proposed approach over existing baselines.

I dont have any major criticism for the paper as I quite like it. Here are a few minor points that I think the paper can improve on:
- The paper can benefit from a more general formulation of the task it solves. Currently, the paper is focused on web navigation tasks. This is not a weakness per se, but an abstract and general formulation of the task and solution can enhance the paper.
- On similar lines, the paper should consider toning down the claims a litte. Web navigation is quite a complex task that may use more than one aspect (if I may say so) of human intelligence. The paper presents an approach for matching provided information to the correct box in a web-form. This is quite simple as compared to web navigation. 
- The paper can benefit from a few figures explaining the proposed system and experimental setup.

review_title: Good paper

Meta-review: This paper considers the problem of agents learning to autonomously navigate the web, specifically by focusing on filling out forms. The focus is on using adversarial environment generation to form a curriculum of training tasks.
Thank you for the revisions to the manuscript, which have particularly improved readability.
The presented problem is really interesting and seems an important real-world problem for RL.
Despite this, as the paper stands the results are not completely convincing. It seems like there is also scope to rigourously analyse the proposed method on other, better known domains to better quantify its limitations.
Decision: Reject