Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes Graph Schemas, a particular type of action-conditioned HMM with deterministic emissions. The uncertainty comes from the transition matrix, as well as observation "clones", i.e. multiple nodes that output the same observation. These schemas can then be transferred by keeping T fixed while changing E, enabling fast learning and planning in different simulated environments.
strength_and_weaknesses: - Strengths
    - I think the paper is clear both in its motivation, what it sets to achieve, and the future remaining directions.
    - Within simulated discrete environments, I believe the types of environments was pretty diverse.
- Weaknesses
    - The overall setting feels quite contrived, the discreteness of the states and actions, the exact clones, etc, are motivated from a cognitive perspective, but the realism of those assumptions isn't well-motivated.
    - Deriving from the previous point, the experiments were quite contrived as well, no realistic experiments were performed, which doesn't allow me to see a way of bringing these ideas to eventual useful systems.
    - The model isn't explained that much and the intellectual novelty compared to standard HMM literature is not explained.
clarity,_quality,_novelty_and_reproducibility: - Quality and novelty
    - The technical contribution of this paper feels, to me, an easy-to-compute subcase of HMMs. In this regard, I didn't find it too novel. If the subcase was very well-justified and how this subcase allows us to be much better/faster than generic HMMs, then I could buy it, but unfortunately I wasn't convinced.
    - The experimental contribution felt a bit weak for two reasons: A) the paper sets to check all the proposed method can do, but little of the different things it cannot do. B) All the domains feel very toy in multiple dimensions (little variability, contrived creation, discrete).
- Clarity
    - Low-level writing was very clear. High-level, I felt too little detail was spent on the proposed model and too much detail on the experiment. The experimental section feels very slow and simple concepts are over-explained. In contrast, I felt I could've received more insights into how the proposed model worked.
    - Small: I would avoid citing the same work in consecutive sentences, it's a bit distracting. This happens a few times in the introduction.
    - In the MNIST experiment: were the domains different digits (i.e. 50k, 80k) or 1 possible image per digit (i.e. 10 different images)?
summary_of_the_review: The paper reads well and is well-motivated from a CogSci perspective. However, the realism of its final assumptions is not supported, neither by writing or formulation, nor by experiments. To improve the writing I would devote more time on the method section and less on the experiment section. For an ICLR-level bar, I believe there needs to be some non-toy experiment or a very strong motivation. Furthermore, the differences with the HMM literature need to be discussed in much more detail to justify the novelty of this approach.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a method which employs an instantiation of Clone-structured cognitive graphs to learn structured representations of environments, which can be used downstream for transfer, inference and planning. The performance of the proposed approach is evaluated using grid-world style navigation tasks in primarily 2D and in one 3D environment.
strength_and_weaknesses: Strengths:

The paper is trying to address a very important research problem. The high-level idea of using graphs as schemas is a very cool concept and I like the overall direction a lot.

Improvements:

1. The work introduces CSCGs as graph schemas for general tasks but only tests it on task with primarily navigation objectives. If the scope of the work is to use graph schemas for only navigation tasks, then this needs to be explicitly stated early in the paper.

2. The related work is very much lacking. References are completely lacking for many related works on using graph-structured schemas to represent long-duration temporal tasks, for much more complex tasks in the real-world in various (supervised, unsupervised and RL) settings. The paper also cites LLMs in first paragraph of section 2 and I'm unsure how they are relevant here. Examples of related works:
A. Multi-modal Cooking Workflow Construction for Food Recipes (https://arxiv.org/abs/2008.09151)
B. Predicting the Structure of Cooking Recipes (https://aclanthology.org/D15-1090.pdf)
C. Procedure Planning in Instructional Videos (https://arxiv.org/abs/1907.01172)
D. Procedure Planning in Instructional Videos via Contextual Modeling and Model-based Policy Learning (https://arxiv.org/abs/2110.01770)
E. proScript: Partially Ordered Scripts Generation via Pre-trained Language Models (https://arxiv.org/abs/2104.08251)
F. Neural Task Graphs: Generalizing to Unseen Tasks from a Single Video Demonstration (https://arxiv.org/abs/1807.03480)

3. The CSCG structure described in section 3 (preliminaries) seems hard to scale. It is basically describing a POMDP setup with nodes as states and edges with transitions corresponding to taking actions. But in most practical problems, we have infinite states so this graph is going to be intractable to maintain, except in small grid worlds.

3. Section 3.1 mentions matrices, so are the state space and observation state considered countably finite? Are the #nodes in graph G known a priori? What about the number of clones per observation?

4. Section 3.1: Do we know the emission model E a priori? If not, how can we keep it fixed throughout learning? Seems like a restrictive and unrealistic assumption to me for most applications!

6. The paper only presents experiments on navigation-style environments in 2D/3D grid-world settings. This implies countably finite states and observations and only 3-4 actions. Hence, the CSCGs are small and tractable. In the case of the 3D T-maze experiment, it is unclear if the vector quantizer used for the 3D T-maze experiments would generalize to more complex 3D environments with complex objects in it.

7. I might have missed the first usage of EPN, but Section 4.1 suddenly brings it up with no prior citation or description. Please clarify if I missed its description in the paper.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, easy to understand, with limited novelty in terms of the high-level idea but a novel implementation of a graph schema (using CSCGs) and appears to be reproducible.
summary_of_the_review: Currently the paper has a ton of scope for improvement (see my comments above) and I cannot recommend acceptance. Hence, my overall score is going to be a weak reject.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes and analyzes a computational model for the learning and transfer of schemas as a mean for abstraction. A computational model that decouples learning a schema from observation/emission is suggested, and its performance across a range of empirical simulations is assessed.
strength_and_weaknesses: The paper clearly illustrates the strenghts and the limitations of the model on a number of experimental settings designed to highlight different features of the model. I found the presentation of the model in Section 3.1 quite synthetic, and I would have appreciated a more detailed definition of it; also, in Equation (1) I am puzzled by the first \sum whose range is not well specified and by P(z_1). 
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, especially in the experimental part where different experiments are clearly introduced and justified. More details about the model and/or an open implementation would improve quality, clarity and reproducibility. 
summary_of_the_review: The paper provides a good empirical analysis of a computational model using graph schema for abstraction and learning. More discussion about the model and the sharing of the experimental data would enrich this contribution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies ‘clone structured cognitive graphs’ (CSCG), a particular type of action-conditioned Hidden Markov Model (HMM) with specific assumptions on the emission function (deterministic emissions and partial observability, where multiple hidden states can generate the same observation). They investigate the ability of this model to transfer between tasks, primarily when the task topology remains the same but the emission functions change and need to be relearned. Experiments on a range of tasks show that the CSCG model is indeed capable of fast adaptation in transfer experiments. 
strength_and_weaknesses: Strength:
* Fast adaptation and transfer in navigation tasks is a relevant and important research topic. 
* The paper has extensive experiments. 
* The method shows quick adaptation in new settings. 

Weaknesses: 
* The approach is essentially based on a Hidden Markov Model, where the emission function gets relearned in a new task. This is useful and works well, but Hidden Markov Models and all their variants have of course been extensively studied in previous decades. A major challenge for HMMs is the inability to scale to higher-dimensional problems, and the authors do not really address these concerns (see discussion in conclusion of review).  
* It is unclear to me how much prior information these methods need. For example, how is the number of z variables chosen per task? I suppose it usually matches the true number of nodes in the source task, or the number of clusters in the 3D task? This seems to be a strong form of prior knowledge, i.e., the designer already knowns how many nodes to specify.  
* After the number of z variables has been specified, they can all possibly be connected in the T matrix right? In other words, the learned topologies that you visualize after learning eliminate all elements of T matrix that get estimated to zero? Or do you also put certain restrictions in the T matrix before training? 
* Your methods use hardcoded exploration strategies. You indicate exploration is not part of the current method, which is fine, but it will influence performance. It is unclear to me what exploration strategies the baseline (EPN) uses, and therefore hard to compare performance. In MPG (Fig 2) for example, to what extend is the performance of EPN influenced by a suboptimal exploration strategy? I think these are things you should address. 
* In general, you have a very extensive experiments section, which is good, but your methodology section is rather short.

* In StreetLearn (Fig 2), it seems you use pixel based observation in the figure, but I do not read anything about it in the results? Do you use an emission function back to pixel level? I guess you would then need the clustering approach of your last experiment again? Otherwise the task looks much more complicated in the figure than it really is. 

clarity,_quality,_novelty_and_reproducibility: Clarity: Well written, although a bit brief on methodology section. 
Quality: Mostly clear. 
Novelty: Uses an extensively studied approach (HMMs where we re-estimate part of the model after transfer). 
Reproducibility: Should be reproducible. 
summary_of_the_review: This paper studies a relevant topic (fast adaptation in navigation tasks), has a lot of experiments, and shows good performance. However, there is one big issue, as mentioned above, which is the use of Hidden Markov Models. Previous literature has extensively shown how HMMs can be specified, estimated, and successfully used in smaller tasks. The authors indeed confirm these ideas, by showing that HMM structure can be efficiently reused between tasks, for example transferring the latent transition structure while re-estimating the emission function. 

However, the main issue that HMMs ran into was their inability to scale to more complex, high-dimensional tasks. The authors only touch upon this topic in their final experiment, where they use clustering on pixel observations to make the HMM still applicable in a basic T-Maze. I do not say we need a neural network in every ICLR paper, but I do think this is an issue the paper should address much more. Would it be possible to use the HMM architecture with a learned embedding function? Can you also jointly optimize high-capacity embedding function and latent dynamics, and still transfer one of them afterwards? There is much more work in this direction, see for example [1,2,3,4], and EPN itself of course. I think you need to address these issues, because when it purely comes to the properties of the HMM approach, then I do not think the current paper presents enough novelty. 

[1] Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A disentangled recognition and nonlinear dynamics model for unsupervised learning. In Advances in Neural Information Processing Systems, pages 3604–3613, 2017.
[2] Matthew Johnson, David K Duvenaud, Alex Wiltschko, Ryan P Adams, and Sandeep R Datta. Composing graphical models with neural networks for structured representations and fast inference. In Advances in neural information processing systems, pages 2946–2954, 2016.
[3] Rahul G Krishnan, Uri Shalit, and David Sontag. Structured inference networks for nonlinear state space models. In AAAI, pages 2101–2109, 2017.
[4] Rangapuram, Syama Sundar, et al. "Deep state space models for time series forecasting." Advances in neural information processing systems 31 (2018).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes graph schemas which are a simplified version of an HMM. Here, the authors learn the transition matrix and keep it fixed for several tasks while the emmision models are adapted for each task. The model has been shown to exhibit fast learning and planning in different simulated environments.

The paper is well written and looks at a diverse set of environments and applications. However, the setting and the experiments feel quite contrived for some reviewers resulting in a poor evaluation. Reviewers were also concerned about realism of the used assumptions as well as too simple experiments that did not go beyond the level of toy tasks. Also, the method section in the paper needs to be extended. The authors provided an extensive response. Unfortunately, none of the reviewers engaged in the discussion leaving the scores unchanged. While the authors address many of these concerns adequately, I think it remains to be shown that the method scales also to higher-D cases. As no reviewer was very enthusiastic of the paper, I recommend rejection. 
justification_for_why_not_higher_score: The authors need to show that the algorithms also scales to higher dimensionality.
Decision: Reject