Venue: ICLR 2017 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper proposed to use unsupervised learning to learn features in a reinforcement learning setting. It is unclear what "unsupervised" means here since the "causality prior" uses reward signals for training. This is reinforcement learning, not unsupervised learning.

The experiments are also very premature. The task is as simple as moving the head of the robot left or right. There is also no comparison to baselines.

In conclusions section, the authors claim the proposed method can be used for transfer learning without experiments to backup the claim.

Overall this paper is confusing and premature.
review_title: null

final_score: rating: 3: Clear rejection
review_content: The paper proposes to use the representation learning approach of [Jonschkowski & Brock, 2015] with a deep network as function approximator. The general task and approach are interesting, but contribution of this work is limited, and experimental evaluation is absolutely unsatisfactory, so the paper cannot be accepted for publications. 

The approach is tested on a simple synthetic task with very small training and test sets and very little variation in the data. The authors admitted themselves that the results are preliminary. The proposed method is not compared with existing approaches or simple hand-crafted baselines. It is impossible to judge if the proposed method is useful and/or performs well compared to existing approaches. This makes the paper unfit for publication. 

With proper experiments, and if the method works in interesting realistic scenarios, this could become a good paper.

review_title: Review

final_score: rating: null
review_content: Some questions:
- As discussed in the paper, the experiments are preliminary. Do you have plans to add more complex, higher-dimensional tasks?
- How does this approach compare to other state representation learning approaches (e.g. [1], [2], [3], [4])?
- Can this approach be used with continuous action spaces?
- What is the correlation if you simply use PCA on the images?

[1] Lange, Sascha, et al. "Autonomous reinforcement learning on raw visual input data in a real world application." IJCNN, 2012.
[2] Watter, Manuel, et al. "Embed to control: A locally linear latent dynamics model for control from raw images." NIPS, 2015.
[3] Finn, Chelsea, et al. "Deep Spatial Autoencoders for Visuomotor Learning." ICRA, 2016.
[4] van Hoof, Herke, et al. "Stable reinforcement learning with autoencoders for tactile and visual data." IROS, 2016.

review_title: Pre-Review Questions

final_score: rating: 3: Clear rejection
review_content: This paper implements the method of Jonschkowski & Brock to learn a low-dimensional state representation represented as the last layer of a neural network. The experiments apply the method for learning a one-dimensional state representation of a simulated robot’s head position from synthetic images.

Learning state representations is an active and useful area of research for learning representations in interactive domains such as robotics. However, there seems to be no novelty in the method, over Jonschkowki & Brock. The primary contribution is the experimental evaluation performed on one task, where the paper evaluates the correlation between the learned state representation and the ideal state representation for the task (which is the robot’s head position).

As acknowledged by the authors, the experiments are very preliminary, only showing one simple task with a one-dimensional learned representation and a two-dimensional discrete action space. To make the experiments compelling, there need to be comparisons to prior methods such as Lange et al. ’12, Watter et al. NIPS ’15, and Finn et al. ICRA ’16 which also learn state representations from raw images. PCA on the images would also be a useful comparison, especially for simple tasks. Without these comparisons, it is impossible to evaluate the effectiveness of the method.

Lastly, as mentioned in the pre-review questions, the related work should include a discussion of other state representation learning methods such as Watter et al. NIPS ’15, Finn et al. ICRA ’16, and van Hoof et al. IROS ’16.

In summary, this paper lacks novelty and significance, as the paper implements an existing method and demonstrates results on only one simple task. Without comparisons, the results are impossible to interpret. More challenging tasks and experimental comparisons would significantly improve the paper. Additionally, this paper does not introduce any novel contributions to state representation learning for solving challenges in this domain. One pro is that the paper is generally written clearly.
review_title: Review

final_score: rating: null
review_content: Hello,

A few questions:

1. Why do you use so few sequences for training and only one (!) for validation? Why do you not simulate more data? Are you sure the training and the validation sets are sufficiently different?

2. You argue deep network is better because it can learn invariance to illumination and noise. This is known very well, and moreover, could probably be achieved with some simple hand-designed features. Have you tried introducing more variation in the task and the visual input to better make use of deep network's capabilities?

3. Have you compared to any baselines? For example, E2C (Watter et al., 2015).

4. Have you tried other tasks?

Thanks.
review_title: pre-review questions

Meta-review: The authors apply an already-published method for state representation learning in a very simple experimental scenario. They give no additional contribution or comparison, nor do they offer any empirical or analytical study.
Decision: Reject