Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper introduces a representation learning step in the Intrinsically Motivated Exploration Process (IMGEP)  framework.

Though this work is far from my expertise fields I find it quite easy to read and a good introduction to IMGEP.
Nevertheless I have some major concerns that prevent me from giving an acceptance decision.

1) The method uses mechanisms than can project back and forth a signal  to the "outcome" space. Nevertheless only the encoder/projection part seems to be used in the algorithm presented p6. For example the encoder part of an AE/VAE is used as a preprocesing stage of the phenomenon dynamic D. It should be obviously noticed that the decoder part could also be used for helping the inverse model I but apparently that is not the case in the proposed method.

2) The representation stage R seems to be learned at the beginning of the algorithm and then fixed. When using DNN as R (when using AE/VAE) why don't you propagate a gradient through R when optimizing D and I ? In this way, learning R at the beginning is only an old good pre-training of DNN with AE.

3) Eventually, Why not directly considering R as lower layers of D and using up to date techniques to train it ? (drop-out, weight clipping, batch normalization ...).
Why not using architecture adapted to images such as CNN ?


review_title: Review of Unsupervised Learning of Goal Spaces for Intrinsically Motivated Exploration

final_score: rating: 6: Marginally above acceptance threshold
review_content: [Edit: After revisions, the authors have made a good-faith effort to improve the clarity and presentation of their paper: figures have been revised, key descriptions have been added, and (perhaps most critically) a couple of small sections outlining the contributions and significance of this work have been written. In light of these changes, I've updated my score.]

Summary:

The authors aim to overcome one of the central limitations of intrinsically motivated goal exploration algorithms by learning a representation without relying on a "designer" to manually specify the space of possible goals. This work is significant as it would allow one to learn a policy in complex environments even in the absence of a such a designer or even a clear notion of what would constitute a "good" distribution of goal states.

However, even after multiple reads, much of the remainder of the paper remains unclear. Many important details, including the metrics by which the authors evaluate performance of their work, can only be found in the appendix; this makes the paper very difficult to follow.

There are too many metrics and too few conclusions for this paper. The authors introduce a handful of metrics for evaluating the performance of their approach; I am unfamiliar with a couple of these metrics and there is not much exposition justifying their significance and inclusion in the paper. Furthermore, there are myriad plots showing the performance of the different algorithms, but very little explanation of the importance of the results. For instance, in the middle of page 9, it is noted that some of the techniques "yield almost as low performance as" the randomized baseline, yet no attempt is made to explain why this might be the case or what implications it has for the authors' approach. This problem pervades the paper: many metrics are introduced for how we might want to evaluate these techniques, yet there is no provided reason to prefer one over another (or even why we might want to prefer them over the classical techniques).

Other comments:
- There remain open questions about the quality of the MSE numbers; there are a number of instances in which the authors cite that the "Meta-Policy MSE is not a simple to interpret" (The remainder of this sentence is incomplete in the paper), yet little is done to further justify why it was used here, or why many of the deep representation techniques do not perform very well.
- The authors do not list how many observations they are given before the deep representations are learned. Why is this? Additionally, is it possible that not enough data was provided?
- The authors assert that 10 dimensions was chosen arbitrarily for the size of the latent space, but this seems like a hugely important choice of parameter. What would happen if a dimension of 2 were chosen? Would the performance of the deep representation models improve? Would their performance rival that of RGE-FI?
- The authors should motivate the algorithm on page 6 in words before simply inserting it into the body of the text. It would improve the clarity of the paper.
- The authors need to be clearer about their notation in a number of places. For instance, they use \gamma to represent the distribution of goals, yet it does not appear on page 7, in the experimental setup.
- It is never explicitly mentioned exactly how the deep representation learning methods will be used. It is pretty clear to those who are familiar with the techniques that the latent space is what will be used, but a few equations would be instructive (and would make the paper more self-contained).

In short, the paper has some interesting ideas, yet lacks a clear takeaway message. Instead, it contains a large number of metrics and computes them for a host of different possible variations of the proposed techniques, and does not include significant explanation for the results. Even given my lack of expertise in this subject, the paper has some clear flaws that need addressing.

Pros:
- A clear, well-written abstract and introduction
- While I am not experienced enough in the field to really comment on the originality, it does seem that the approach the authors have taken is original, and applies deep learning techniques to avoid having to custom-design a "feature space" for their particular family of problems.

Cons:
- The figure captions are all very "matter-of-fact" and, while they explain what each figure shows, provide no explanation of the results. The figure captions should be as self-contained as possible (I should be able to understand the figures and the implications of the results from the captions alone).
- There is not much significance in the current form of the paper, owing to the lack of clear message. While the overarching problem is potentially interesting, the authors seem to make very little effort to draw conclusions from their results. I.e. it is difficult for me to easily visualize all of the "moving parts" of this work: a figure showing the relationship bet
- Too many individual ideas are presented in the paper, hurting clarity. As a result, the paper feels scattered. The authors do not have a clear message that neatly ties the results together.
review_title: Some interesting ideas, yet no clear message

final_score: rating: 7: Good paper, accept
review_content: The paper investigates different representation learning methods to create a latent space for intrinsic goal generation in guided exploration algorithms.  The research is in principle very important and interesting.

The introduction discusses a great deal about intrinsic motivations and about goal generating algorithms. This is really great, just that the paper only focuses on a very small aspect of learning a state representation in an agent that has no intrinsic motivation other than trying to achieve random goals.
I think the paper (not only the Intro) could be a bit condensed to more concentrate on the actual contribution. 

The contribution is that the quality of the representation and the sampling of goals is important for the exploration performance and that classical methods like ISOMap are better than Autoencoder-type methods. 

Also, it is written in the Conclusions (and in other places): "[..] we propose a new intrinsically Motivated goal exploration strategy....". This is not really true.  There is nothing new with the intrinsically motivated selection of goals here, just that they are in another space. Also, there is no intrinsic motivation. I also think the title is misleading.

The paper is in principle interesting. However, I doubt that the experimental evaluations are substantial enough for profound conclusion. 

Several points of critic: 
- the input space was very simple in all experiments, not suitable for distinguishing between the algorithms, for instance, ISOMap typically suffers from noise and higher dimensional manifolds, etc.
- only the ball/arrow was in the input image, not the robotic arm. I understand this because in phase 1 the robot would not move, but this connects to the next point:
- The representation learning is only a preprocessing step requiring a magic first phase.
    -> Representation is not updated during exploration
- The performance of any algorithm (except FI) in the Arm-Arrow task is really bad but without comment. 
- I am skeptical about the VAE  and RFVAE results. The difference between Gaussian sampling and the KDE is a bit alarming, as the KL in the VAE training is supposed to match the p(z) with N(0,1). Given the power of the encoder/decoder it should be possible to properly represent the simple embedded 2D/3D manifold and not just a very small part of it as suggested by Fig 10. 
I have a hard time believing these results. I urge you to check for any potential errors made. If there are not mistakes then this is indeed alarming.

Questions:
- Is it true that the robot always starts from same initial condition?! Context=Emptyset. 
- For ISOMap etc, you also used a 10dim embedding?

Suggestion:
- The main problem seems to be that some algorithms are not representing the whole input space.
- an additional measure that quantifies the difference between true input distribution and reproduced input distribution could tier the algorithms apart and would measure more what seems to be relevant here.  One could for instance measure the KL-divergence between the true input and the sampled (reconstructed) input (using samples and KDE or the like). 
- This could be evaluated on many different inputs (also those with a bit more complicated structure) without actually performing the goal finding.
- BTW: I think Fig 10 is rather illustrative and should be somehow in the main part of the paper
 
On the positive side, the paper provides lots of details in the Appendix.
Also, it uses many different Representation Learning algorithms and uses measures from manifold learning to access their quality.

In the related literature, in particular concerning the intrinsic motivation, I think the following papers are relevant:
J. Schmidhuber, PowerPlay: training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Front. Psychol., 2013.

and

G. Martius, R. Der, and N. Ay. Information driven self-organization of complex robotic behaviors. PLoS ONE, 8(5):e63400, 2013.


Typos and small details:
p3 par2: for PCA you cited Bishop. Not critical, but either cite one the original papers or maybe remove the cite altogether
p4 par-2: has multiple interests...: interests -> purposes?
p4 par-1: Outcome Space to the agent is is ...
Sec 2.2 par1: are rapidly mentioned... -> briefly
Sec 2.3 ...Outcome Space O, we can rewrite the architecture as:
  and then comes the algorithm. This is a bit weird
Sec 3: par1: experimental campaign -> experiments?
p7: Context Space: the object was reset to a random position or always to the same position?
Footnote 14: superior to -> larger than
p8 par2: Exploration Ratio Ratio_expl... probably also want to add (ER) as it is later used
Sec 4: slightly underneath -> slightly below
p9 par1: unfinished sentence: It is worth noting that the....
one sentence later: RP architecture? RPE?
Fig 3: the error of the methods (except FI) are really bad. An MSE of 1 means hardly any performance!
p11 par2: for e.g. with the SAGG..... grammar?

Plots in general: use bigger font sizes.


review_title: Interesting, but not substantial enough -> updated now good enough

Meta-review: This paper aims to improve on the intrinsically motivated goal exploration framework by additionally incorporating representation learning for the space of goals. The paper is well motivated and follows a significant direction of research, as agreed by all reviewers. In particular, it provides a means for learning in complex environments, where manually designed goal spaces would not be available in practice. There had been significant concerns over the presentation of the paper, but the authors put great effort in improving the manuscript according to the reviewers’ suggestions, raising the average rating by 2 points after the rebuttal. 
Decision: Accept (Poster)