Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper starts by pointing out the need for methods that perform both state and temporal representation learning for RL and which allow gaining insight into what is being learned (perhaps in order to allow a human operator to intervene if necessary). This is a very important goal from a practical point of view, and it is great to see research in this direction. For this reason, I would like to encourage the authors to pursue this further. However, I am not at all convinced that the current incarnation of this work is the right answer. Part of the issues are more related to presentation, part may require rethinking.
In order to get the "interpretability", the authors opt for some fairly specific ways of performing abstraction. For example, their skills always start In a single skill initiation state, and likewise end in one state. This seems unnecessarily restrictive, and it is not clear why this restriction is needed (other than convenience). Similarly, clustering is the basis for forming the higher level states, and there is a specific kind of clustering used here. Again, it is not clear why this has to be done via clustering as opposed to other methods. Ensuring temporal coherence in the particular form employed also seems restrictive. There is a reference to supplementary material where some of these choices are explained, but I could not find this in the posted version of the paper. The authors should either explain clearly why these specific choices are necessary, or (even better) try to think if they can be relaxed while still keeping interpretability. 
From a presentation point of view, the paper would benefit from formal definitions of AMDP and SAMDP, as well as from formal descriptions of the algorithms employed in constructing these representations (eg Bellman equations for the models, and update rules for the algorithms learning them). While intuitions are given, the math is not precisely stated. The overhead of constructing an SAMDP (computation time and space) should be clarified as well.
The experiments are well carried out and it is nice to have both gridworld experiments, where visualization are easy to perform and understand, as well as Atari games (gridworld still have their place despite what other reviewers might say). The results are positive, but because the proposed approach has many moving parts which rely on specific choices, significance and general ease of use are unclear at this point. Perhaps having the complete supplementary would have helped in this respect.
Small comment: The two lines after Eq 2 contain typos in the notation and a wrong sign in the equation.
review_title: Great goal, but the PAPER NEEDS MAJOR WORK

final_score: rating: 4: Ok but not good enough - rejection
review_content: The framework of semi-Markov decision processes (SDMPs) has been long used to model skill learning and temporal abstraction in reinforcement learning. This paper proposes a variant of such a model called a semi-aggregated MDP model. 

The formalism of SAMDP is not defined clearly enough to merit serious attention. The approach is quasi heuristic and explained through examples rather than clear definition. The work also lacks sufficient theoretical rigor. 

Simple experiments are proposed using 2D grid worlds to demonstrate skills. Grid worlds have served their purpose long enough in reinforcement learning, and it is time to retire them. More realistic domains are now routinely used and should be used in this paper as well. 


review_title: Variant of an SMDP model for skill learning 

final_score: rating: null
review_content: Can you please revise the paper to clarify the purpose of SAMDP? In the description of the method, it appears that SAMDP is an learning architecture for training. Reading the results, it appears to have been used post training for analysis.

Can this method be applicable to continuous states and actions? 
review_title: Clarification

final_score: rating: null
review_content: Your references are in completely the wrong format! Please pay attention to the formatting of references, otherwise reading your paper is well nigh impossible! If you cite a paper in line in the text, such as Peng and Williams (1993), you should formulate your sentence as "Peng and Williams (1993) focused on simpler problems". Or you can write a citation in the passive mode as "Some researchers focused on simpler problems, e.g. (Peng and Williams, 1993). Please do not use the current format, as you are in violation of basic rules in scientific papers. 

Second, your literature review of work on skill learning in MDPs is woefully adequate. and does not even begin to touch on the bulk of the important work. There is a large thread of work on the use of "bottleneck" states and regions (e.g, Simsek, Hengst, et al.), which is not cited. There is work by Johannsen and Barto (JMLR) on VISA using causal analysis of action models to infer options. There is much work on spectral methods to infer option hierarchies as well. Finally, there is a large and growing body of work on hierarchical methods in deep RL (e.g, Durugkar et al., "Deep RL with Macro Actions", Arxiv, 2016; Kulkarni et al., Integrating intrinsic motivation and deep RL, Arxiv 2016). A search on Arxiv will retrieve many other related papers. 

Please avoid excessive use of acronyms, and in particular, don't use acronyms like AMDP, before they are defined in the paper. 

review_title: Incorrect format for references and inadequate literature survey 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper presents a method for visualization and analysis of policies from observed trajectories that the policies produce. The method infers higher level skills and clusters states. The result is a simplified, discrete higher-order state and action transition matrix. This model can be used for analysis, modeling, and interpretation. To construct semi-aggregated MDP the authors propose combining ideas for creating semi-MPDs and agregrated-MDPs. The method consists of choosing features, state clustering, skill inference, reward and skill length inference, and model selection. The method was demonstrated on a small grid-world problem, and DQN-trained agent for playing Atari games.

The authors correctly identify that tools and means for interpretibility of RL methods are important for analysis, and deployment of such methods for real-world applications. This is particularly true in robotics and high-consequence systems.

The end-result of the presented method is a high-level transition matrix. There is a big body of literature looking into hierarchical RL methods where lower level skills are combined with higher level policies. The presented method has the similar result, but the advantage of the presented method is that it comes up with a structure and analyzes already trained agent, which is very interesting. The paper would benefit from emphasizing this difference, and contrasting with the broader body of literature.

To build the model, the authors propose combining the ideas from two existing ideas, semi-MPDs and agregrated-MDPs with using modified k-means for state clustering. It appears that the novelty of the presented method is limited. The paper would have been stronger if the authors explicitly stated the contributions over combining existing methods, and better highlighted the practical utility of the method. 

The evaluation section would be made stronger with more analytical results and precise evaluation, showing full strength of the method.

The paper is difficult to read. To improve readability:
- The Semi-Aggregated MDP section should include more precise description of the methods. The narrative that builds intuition is welcome. In addition to the existing narrative, algorithms and formulas where applicable should be included as well. 
- The paper should be self-contained. For example, more background on Occams Razor principle should be included.
- Reduce the number of acronyms, in particular similarly sounding acronyms. Define acronyms before using. 
- Be more clear on the contributions, contrast with relevant literature, and the specific benefits of the presented method.
- Fix typos, formatting mistakes etc., as they can be distracting for reading. 

The approach of reverse engineering the hierarchy, and learning high-level transition matrix is very interesting and promising. Perhaps the method can be used to outperform single network approach by using the model as an input to more specialized hierarchical trainers and learn complex behaviors more optimally then possible with one large network approach. Unfortunately, the paper falls short in the novelty, precision, and clarity.
review_title: Review of Spatio-Temporal Abstractions .. Paper

final_score: rating: null
review_content: No questions
review_title: No questions

Meta-review: The paper combines semi-Markov models with state aggregation models to define a Semi-Aggregated Markov Decision Process (SAMDP). As per the reviews, the need for spatio-temporal representations is recognized as an important problem. Where the work currently falls short is the loose description of framework, which should be formalized for clarity and analysis, and in terms of its contributions and benefits with respect to the broader literature on hierarchical RL methods.
 The anonymous reviewers (and the additional public review/comment) provide a number of clear directions to pursue for the next iteration of this work; indeed, there is strong general enthusiasm for the greater goals (as shared by the relevant body of work in the research community), but the work is seen as requiring further clarity in formalization and presentation.
Decision: Reject