Venue: ICLR 2018 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper introduces House3D, a virtual 3D environment consisting of in-door scenes with a diverse set of scene types, layouts and objects. This was originally adapted from the SUNCG dataset, enhanced with the addition of a physics model and an API for interacting with the environment.  They then focus on a single high-level instruction following task where an agent is randomly assigned at a location in the house and is asked to navigate to a destination described by a high-level concept (“kitchen”) without colliding with objects.  They propose two models with gated-attention architecture for solving this task, a gated-CNN and a gated-LSTM. Whilst the novelty of the two models is questionable (they are adaptations of existing models to the task),  they are a useful addition to enable a benchmark on the task.  The paper in general is well written, and the environment will be a useful addition to the community.

General Comments
- In the related work section the first part talks about several existing environments.  Whilst the table is useful, for the “large-scale” and “fast-speed”columns, it would be better if there were some numbers attached - e.g.  are these orders of magnitude differences?  Are these amenable to Bayesian optimisation?
- I  didn’t  see  any  mention  of  a  pre-specified  validation  set  or  pre-defined cross-validation sets.  This would surely be essential for hyperparameter tuning
- For the discrete action space state what the 12 actions are.
- The  reward  function  should  be  described  in  more  detail  (can  be  in  appendix).  How is the shortest distance calculated?  As a general comment it seems that this is a very strong (and unrealistic) reward signal, particularly for generalisation.
- There are a number of hyperparameters (αDDPG, αA3C,  entropy bonus terms, learning rates etc).  Some discussion of how these were chosen and the sensitivity to these parameters were helpful
- Figures 3 and 4 are hard to compare, as they are separated by a page, and the y-axes are not shared.
- The additional 3D scenes datasets mentioned by Ankur Handa should be cited.

Typographical Issues
- Page 1:  intelligence→intelligent 
- Page  4:  On  average,  there  is→On  average,  there  are;  we  write→we wrote
- Page 7:  softmax-gumbel trick→softmax-Gumbel trick; gumbel-softmax→Gumbel-softmax
- References.  The references should have capitalisation where appropriate.For example,  openai→OpenAI, gumbel→Gumbel,  malmo→Malmo
review_title: The paper in general is well written, and the environment will be a useful addition to the community

final_score: rating: 4: Ok but not good enough - rejection
review_content: Paper Summary: The paper proposes a simulator for the SUNCG dataset to perform rendering and collision detection. The paper also extends A3C and DDPG (reinforcement learning methods) by augmenting them with gated attention. These methods are applied for the task of navigation.

Paper Strengths:
- It is interesting that the paper shows generalization to unseen scenes unlike many other navigation methods.
- The renderer/simulator for SUNCG is useful.

Paper Weaknesses:
The paper has the following issues: (1) It oversells the task/framework. The proposed task/framework is not different from what others have done. (2) There is not much novelty in the paper. The SUNCG dataset already exists. Adding a renderer to that is not a big deal. There is not much novelty in the method either. The paper proposes to use gated attention, which is not novel and it does not help much according to Figures 3b and 4b. (3) Other frameworks have more functionalities than the proposed framework. For example, other frameworks have physics or object interaction while this framework is only useful for navigation. (4) The paper keeps mentioning "Instructions". This implies that the method/framework handles natural language, while this is not the case. This is over-selling as well.

Questions and comments:

- Statements like "On the contrary, we focus on building a flexible platform that intersects with multiple research directions in an efficient manner allowing users to customize the rules and level of complexity to their needs." are just over-selling. This environment is not very different from existing platforms.

- What is referred to as “physics” is basically collision detection. It is again over-selling the environment. Other environments model real physics.

- It is not clear what customizable mean in Table 1. I searched through the paper, but did not find any definition for that. All of the mentioned frameworks are customizable.

- "we compute the approximate shortest distance from the target room to each location in the house" --> This assumption is somewhat unrealistic since agents in the real world do not have access to such information.

- Instruction is an overloaded word for "go to a RoomType". The paper tries to present the tasks/framework as general tasks/framework while they are not.

- In GATED-LSTM, h_t is a function of I. Why is I concatenated again with h_t?

- Success rate is not enough for evaluation. The number of steps should be reported as well.

- The paper should include citations to SceneNet and SceneNet RGBD.


review_title: Not much novelty; overselling the framework and the task

final_score: rating: 5: Marginally below acceptance threshold
review_content: Building rich 3D environments where to run simulations is a very interesting area of research. 

Strengths:
1.	The authors propose a virtual environment of indoor scenes having a much larger scale compared to similar interactive environments and access to multiple visual modalities. They also show how the number of available scenes greatly impacts generalization in navigation based tasks. 
2.	The authors provide a thorough analysis on the contribution of different feature types (Mask, Depth, RGB) towards the success rate of the goal task. The improvements and generalization brought by the segmentation and depth masks give interesting insights towards building new navigation paradigms for real-world robotics. 

Weaknesses:
1.	The authors claim that the proposed environment allows for multiple applications and interactions, however from the description in section 3, the capacities of the simulator beyond navigation are unclear.
The dataset proposed, Home3D, adds a number of functionalities over the SUNCG dataset. The SUNCG dataset provides a large number of 3D scanned houses.  The most important contributions with respect to SUNCG are:
- An efficient renderer: an important aspect.
- Introducing physics: this is very interesting, unfortunately the contribution here is very small. Although I am sure the authors are planing to move beyond the current state of their implementation, the only physical constraint currently implemented is an occupancy rule and collision detection. This is not technically challenging. 
Therefore, the added novelty with respect to SUNCG is very limited.
2.	The paper presents the proposed task as navigation from high level task description, but given that the instructions are fixed for a given target, there are only 5 possible instructions which are encoded as one-hot vectors. Given this setting, it is unclear the need for a gated attention mechanism. While this limited setting allows for a clear generalization analysis, it would have been good to study a setting with more complex instructions, allowing to evaluate instructions not seen during training.
3.	While the authors make a good point showing generalization towards unseen scenes, it would have been good to also show generalization towards real scenarios, demonstrating the realistic nature of House3D and the advantages of using non-RGB features.
4.	It would have been good to report an analysis on the number of steps performed by the agent before reaching its goal on the success cases. It seems to me that the continuous policy would be justified in this setting. 
Comments
-	It is unclear to me how the reward shaping addition helps generalize to unseen houses at test time, as suggested by the authors.
-	I miss a reference to (https://arxiv.org/pdf/1609.05143.pdf) beyond the AI-THOR environment, given that they also approach target driven navigation using an actor-critic approach.


The paper proposes a new realistic indoor virtual environment, having a much larger number of scenes than similar environments. From the experiments shown, it seems that the scale increase, together with the availability of features such as Segmentation and Depth improve generalization in navigation tasks, which makes it a promising framework for future work on this direction. However, the task proposed seems too simple considering the power of this environment, and the models used to solve the task don’t seem to bring  relevant novelties from previous approaches. (https://arxiv.org/pdf/1706.07230.pdf)


review_title: The authors propose a virtual environment built from SUNCG with interactive indoor scenes where agents can perform a variety of human-like tasks. They later focus on the task of navigation, where agents are asked to navigate to a destination without colliding with objects given a high level natural language instruction.

Meta-review: The authors present an environment for semantic navigation that is based on an existing dataset, SUNCG. Datasets/environments are important for deep RL research, and the contribution of this paper is welcome. However, this paper does not offer enough novelty in terms of approach/method and its claims are somewhat misleading, so it would probably be a better fit to publish it at a workshop. 
Decision: Invite to Workshop Track