Title: Unsupervised Deep Learning of State Representation Using Robotic Priors 
Authors: Timothee LESORT
David FILLIAT
Venue: ICLR 2017
Keywords: Deep learning
Computer vision
Unsupervised Learning

Abstract: Our understanding of the world depends highly on how we represent it.  Using background knowledge about its complex underlying physical rules, our brain can produce intuitive and simplified representations which it can easily use to solve problems. The approach of this paper aims to reproduce this simplification process using a neural network to produce a simple low dimensional state representation of the world from images acquired by a robot. As proposed in Jonschkowski & Brock (2015), we train the neural network in an unsupervised way, using the "a priori" knowledge we have about the world as loss functions called "robotic priors" that we implemented through a siamese network. This approach has been used to learn a one dimension representation of a Baxter head position from raw images. The experiment resulted in a 97,7% correlation between the learned representation and the ground truth, and show that relevant visual features form the environment are learned.

TL;DR: This paper introduces a method for training a deep neural network to learn a  representation of a robot's environment state using a priori knowledge.

Review discussion:
Hello,
A few questions:
1. Why do you use so few sequences for training and only one (!) for validation? Why do you not simulate more data? Are you sure the training and the validation sets are sufficiently different?
2. You argue deep network is better because it can learn invariance to illumination and noise. This is known very well, and moreover, could probably be achieved with some simple hand-designed features. Have you tried introducing more variation in the task and the visual input to better make use of deep network's capabilities?
3. Have you compared to any baselines?
For example, E2C (Watter et al., 2015).
4. Have you tried other tasks?
Thanks.
Dear reviewer,
thank you for your questions, here are the answers:
1. Why do you use so few sequences for training and only one (!) for validation? Why do you not simulate more data? Are you sure the training and the validation sets are sufficiently different?
This is quite preliminary work, and we did not had time to generate more data before people working on this had to leave.
However, we are quite confident that the validation set is different enough from the training set because the arms of the robot are in a position never seen during training. We expect the trained neural network to generalize to new configurations of the environment which means in this case new arms position. We will update our results with cross-validation results.
Beside these limitations, our longer term goal is to learn with reasonably limited data so as to be able to work with demonstration made by humans on a real robot, thus working with a few tenth of training data is a reasonable size given these constraints. The data augmentation we use, aims to bring good generalization without more data.
2. You argue deep network is better because it can learn invariance to illumination and noise. This is known very well, and moreover, could probably be achieved with some simple hand-designed features. Have you tried introducing more variation in the task and the visual input to better make use of deep network's capabilities?
You are right that hand-designed features could be used to solve this, however, while obviously not new, it remain interesting for us to learn this from data.
Furthermore one of the side goals of the paper is to show that a deep neural network can be trained using the robotics priors introduce in (Jonschkowski & Brock, 2015). The visual perturbations introduced where chosen in order to simulate real world perturbations.
That is why some techniques of data augmentation where not adapted, like non-linear deformations of images.
Furthermore, the perturbation must not change the head angle estimation possibility.
For example, cropping the images can bring uncertainty because two different images can become identical after cropping.
Therefore we did not introduced more variations in the image, but even in the current setting, the network has basically to learn to ignore the robot arm in arbitrary configuration so as to be not perturbed by its position for the head direction estimation, which is not completely trivial (even if once again could be possible with other techniques). The image variations are currently limited to the random arms position but we are planning to work on real image with dynamic background in the next steps.
Regarding tasks variations, we tried a push-button task (see 4. below).
3. Have you compared to any baselines?
For example, E2C (Watter et al., 2015).
No, we did not had time to do that, but the reference you mention is indeed very relevant. See the answers to the other reviewers for some element of comparison.
4. Have you tried other tasks?
Yes, we tried a button-pushing task with the goal of learning a three dimensional state space corresponding to the robot arm end-effector position, but as stated in the paper, we did not achieve success on this task yet. This is probably in part because the evaluation procedure by comparison with the ground truth is not relevant, so we are planning to integrate reinforcement learning to have a coherent evaluation system.
Thanks
Some questions:
- As discussed in the paper, the experiments are preliminary. Do you have plans to add more complex, higher-dimensional tasks?
- How does this approach compare to other state representation learning approaches (e.g. [1], [2], [3], [4])?
- Can this approach be used with continuous action spaces?
- What is the correlation if you simply use PCA on the images?
[1] Lange, Sascha, et al. "Autonomous reinforcement learning on raw visual input data in a real world application." IJCNN, 2012.
[2] Watter, Manuel, et al. "Embed to control: A locally linear latent dynamics model for control from raw images." NIPS, 2015.
[3] Finn, Chelsea, et al. "Deep Spatial Autoencoders for Visuomotor Learning." ICRA, 2016.
[4] van Hoof, Herke, et al. "Stable reinforcement learning with autoencoders for tactile and visual data." IROS, 2016.
Dear Reviewer,
thanks for your questions, here are the answers :
- As discussed in the paper, the experiments are preliminary. Do you have plans to add more complex, higher-dimensional tasks?
Yes, in the short term, we plan to learn a 3-dimensional state space representation corresponding to the robot arm end-effector position with a button-pushing task, but as stated in the paper, we did not achieve success on this task yet. On the longer term, we plan to extend to tasks where the state are relative, for example object grasping where the state should encode distance form effector to a randomly located object.
- How does this approach compare to other state representation learning approaches (e.g. [1], [2], [3], [4])?
[1] Lange, Sascha, et al. "Autonomous reinforcement learning on raw visual input data in a real world application." IJCNN, 2012.
[2] Watter, Manuel, et al. "Embed to control: A locally linear latent dynamics model for control from raw images." NIPS, 2015.
[3] Finn, Chelsea, et al. "Deep Spatial Autoencoders for Visuomotor Learning." ICRA, 2016.
[4] van Hoof, Herke, et al. "Stable reinforcement learning with autoencoders for tactile and visual data." IROS, 2016.
Those papers are relevant to our approach and we will update our related work.
We have the same overall objective as [1],[2],[3] and [4]: Learning a low dimension state representation in an unsupervised way which could be used for RL.
A first difference is that, in all these approaches, training is done with auto-encoders (AE, DAE, VAE, ...) to learn a representation able to reconstruct the input images when we learn a representation which only respect the priors we impose. We assume that, depending on the environment and tasks, some information learned for reconstructing the images may be not relevant to the task.
For example in our head direction task estimation with random arm positions, reconstructing an image would require to encode arm position, which is quite complex and not directly necessary for the head direction estimation.
Another distinctive point is the use of the dynamics or of the action performed by the robot as constraints for learning a relevant state representation. [1] does not use this information, and [3] use it in an indirect way by imposing the states to be positions in the images (additionally filtered by a Kalman filter). We use this informations in our priors, which like is shown in [2] and [4] is important for learning a relevant representation.
Additionally, the causality prior (eq (5)) of our approach require the use of the reward for state learning (which is not the case in [1-4].
While this is more dependent on the particular task that images and actions, this is a good way to bias representation toward relevant states (as shown in (Jonschkowski & Brock, 2015)) that could be advantageous in some tasks.
We did not had time to experiment these approaches to compare results, it will be done in future work with integration with reinforcement learning.
- Can this approach be used with continuous action spaces?
We think the approached can be extended to continuous actions by modifying the priors (3) (4) and (5) to have a loss function that takes action differences into account instead of finding states with identical actions.
For example, (2) would become (|Delta_s1|-|Delta_s2|)^2.exp(-|a1-a2|^2) .
However, this would raise the problem of sampling enough similar actions so that the priors could be applied on enough pairs or quadruples of images, which would be problematic with actions of higher dimensions.
- What is the correlation if you simply use PCA on the images?
We did not achieve this experiment, but in the case we study, we assume that the PCA would not find a relevant representation in just one dimension from the image given that an important part of the image variations is due to the arm movement. The goal of the approach is to find a better comprehension of the data than a PCA could achieve.
However, it is a baseline we will consider in future work.
Thanks
This paper proposed to use unsupervised learning to learn features in a reinforcement learning setting. It is unclear what "unsupervised" means here since the "causality prior" uses reward signals for training. This is reinforcement learning, not unsupervised learning.
The experiments are also very premature. The task is as simple as moving the head of the robot left or right. There is also no comparison to baselines.
In conclusions section, the authors claim the proposed method can be used for transfer learning without experiments to backup the claim.
Overall this paper is confusing and premature.
Dear Reviewer,
As written in the paper we use rewards information to train our neural network.
But we don't aim to maximize those rewards during training and that why it is not reinforcement learning. We call our training unsupervised because we don't give any label and we only impose some priors to the representation in order to learn it.
The paper proposes to use the representation learning approach of [Jonschkowski & Brock, 2015] with a deep network as function approximator. The general task and approach are interesting, but contribution of this work is limited, and experimental evaluation is absolutely unsatisfactory, so the paper cannot be accepted for publications.
The approach is tested on a simple synthetic task with very small training and test sets and very little variation in the data. The authors admitted themselves that the results are preliminary. The proposed method is not compared with existing approaches or simple hand-crafted baselines. It is impossible to judge if the proposed method is useful and/or performs well compared to existing approaches. This makes the paper unfit for publication.
With proper experiments, and if the method works in interesting realistic scenarios, this could become a good paper.
This paper implements the method of Jonschkowski & Brock to learn a low-dimensional state representation represented as the last layer of a neural network. The experiments apply the method for learning a one-dimensional state representation of a simulated robot’s head position from synthetic images.
Learning state representations is an active and useful area of research for learning representations in interactive domains such as robotics.
However, there seems to be no novelty in the method, over Jonschkowki & Brock. The primary contribution is the experimental evaluation performed on one task, where the paper evaluates the correlation between the learned state representation and the ideal state representation for the task (which is the robot’s head position).
As acknowledged by the authors, the experiments are very preliminary, only showing one simple task with a one-dimensional learned representation and a two-dimensional discrete action space. To make the experiments compelling, there need to be comparisons to prior methods such as Lange et al. ’12, Watter et al. NIPS ’15, and Finn et al. ICRA ’16 which also learn state representations from raw images. PCA on the images would also be a useful comparison, especially for simple tasks. Without these comparisons, it is impossible to evaluate the effectiveness of the method.
Lastly, as mentioned in the pre-review questions, the related work should include a discussion of other state representation learning methods such as Watter et al. NIPS ’15, Finn et al. ICRA ’16, and van Hoof et al. IROS ’16.
In summary, this paper lacks novelty and significance, as the paper implements an existing method and demonstrates results on only one simple task. Without comparisons, the results are impossible to interpret. More challenging tasks and experimental comparisons would significantly improve the paper.
Additionally, this paper does not introduce any novel contributions to state representation learning for solving challenges in this domain. One pro is that the paper is generally written clearly.
Dear reviewers,
we agreed that comparisons with other works need to be done and that other tasks have to be tested to improve evaluation and validation.
It will be done in future experiments.

Meta-review:
role: PC
data: comment: The authors apply an already-published method for state representation learning in a very simple experimental scenario. They give no additional contribution or comparison, nor do they offer any empirical or analytical study.

Decision: Reject