Title: Skill-Driven Neurosymbolic State Abstractions
Venue: NeurIPS 2025
Accepted: yes
Keywords: state abstraction

reinforcement learning
Abstract: We consider how to construct state abstractions compatible with a given set of abstract actions, to obtain a well-formed abstract Markov decision process (MDP). We show that the Bellman equation suggests that abstract states should represent distributions over states in the ground MDP; we characterize the conditions under which the resulting process is Markov and approximately model-preserving, derive algorithms for constructing and planning with the abstract MDP, and apply them to a visual maze task. We generalize these results to the factored actions case, characterizing the conditions that result in factored abstract states and apply the resulting algorithm to Montezuma's Revenge. These results provide a powerful and principled framework for constructing neurosymbolic abstract Markov decision processes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper attempts to formalize the connection between state abstraction and action abstraction, using probability as a language. More specifically, it discusses the conditions under which, given temporal abstraction (i.e., options), one can find a good state abstraction that aligns with the temporal abstraction.\n\nSTRENGTHS AND WEAKNESSES:\nBridging the option framework to state abstraction is an important goal that has not yet been fully achieved. The approach in this paper is original, and the use of probabilistic state representation is very interesting. It makes a significant and original contribution, and I enjoyed reading it.\n\nMost of my concerns are related to clarity. I’ve outlined them in the Questions section below.\n\nQUESTIONS:\nMajor Concerns:\n- The title, introduction, and abstract do not clearly communicate that by “skill-driven,” the authors assume that options (and their associated initiation sets) are given rather than learned. This is a crucial assumption and limitation. It would be helpful to state this more explicitly in the abstract and introduction, so readers can better understand the setting. Similarly, while the current title uses appealing terminology, it does not set the correct expectations for the paper.\n- Lines 20–21: I disagree with the claim that current HRL does not address state abstraction. For instance, Leslie Kaelbling’s work on task and motion planning includes a hierarchical structure that clearly resembles state abstraction. Similarly, Peter Dayan’s feudal reinforcement learning explicitly tackles state abstraction. Interestingly, in state-abstraction-oriented work, temporal abstraction is often implicitly assumed through the use of macro actions. However, the reverse is not generally true—i.e., options do not necessarily address state abstraction. It’s valuable that this paper explores ways to bridge that gap.\n- It would be enjoyable to see a deeper discussion of the relationship among state abstraction, temporal abstraction, and action abstraction, as the connection is both subtle and important.\n- Line 129: I find point 3 confusing. It seems possible to slightly augment the MDP definition to resolve the issue: assume all options are executable everywhere, but selecting an option outside its initiation set returns a huge cost of -100 and consumes no time. Under this formulation, the learned optimal policy should remain the same. Therefore, I do not fully understand why point 3 must hold in order for the process to be Markovian.\n- Definition 1: The functions p and I depend on the policy in the previous state. This policy dependency could introduce complications, but it appears under-discussed in the paper. Does this dependency play an essential role?\n- Figure 4: I’m confused as to why DQN with hindsight experience replay fails to improve. Its performance seems unchanged from 1K to 10K steps. Why?\n\nMinor Concerns:\n- It’s a bit confusing that Equation 1 uses “≈” while Equations 2 and 3 use “=.”\n- In Section 3.1, it took me some time to realize that S is being treated as a distribution, which is a shift from how it is handled in Section 2. Although this is mentioned later in the text, it may be helpful to introduce this point earlier—perhaps around line 108—so that readers are better prepared for the formulation that follows.\n- Definition 1: I initially interpreted p as a scalar, likely due to the presentation in Figure 2b. But just to confirm—it’s a density, correct?\n- Figure 3b: Why do some of the distributions in the middle appear multimodal?\n\nLIMITATIONS:\nYes, but in the appendix. Also, I think some of the theoretical downside (e.g., the connection between state abstraction, temporal abstraction, and action abstraction) is not discussed in detail.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an approach for learning neurosymbolic abstract Markov decision processes, develops the mathematical foundations and theory of the approach, applies it to both simple and more complex RL problems in synthetic environments, and shows the advantages over vanilla (purely neural) deep reinforcement learning, as in DQN.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper is well motivated and clear, but I am not an expert in RL and so I did not even attempt to follow all the technical details.  It appears technically strong, and a solid implementation of a compelling concept.   The results are reasonable. \n\nWeaknesses:  I am not an expert in the paper's area and so I can't really judge the novelty or level of contribution.  But I'm not sure how it fundamentally advances on earlier work such as https://arxiv.org/abs/2002.05518.   To my naive eye, both the present paper and that earlier work learn a distribution-based representation for abstract states.  The present paper adds an extension to consider learning abstract representations defined by options, but I can't tell how much of a fundamental advance that is. \n\nThe paper is strong on theory, but the applications are both quite simple.  The double digit MNIST example is helpful pedagogically but rather contrived; what problems in the real world have such a simple factorization?  The Montezuma's revenge example is more representative of a real world long-horizon planning, but it is still quite simple and the proposed system doesn't solve the whole game from scratch; it starts from a problematic expert policy, and improves it, and it considers only the first room of a long game. The visual maze example is nice, but is still simplified from the real problems of navigation, in ways that play to the algorithm's strength and make it unclear how success there will generalize. \n\nI would also like to understand better how this approach relates to and in practice improves upon other more symbolic ways of learning abstract state and action representations for planning, such as the learned bilevel planners of Silver et al. (AAAI 2022).  That work is briefly mentioned and dismissed here as facing scaling challenges, but it seems that both that work and the present RL-based framework face scaling challenges albeit possibly different ones. \n\nI have to admit that as a non-RL person, I share some of the biases that many people have: RL is nice in theory but just is not a viable approach to building real agents that work in the real world, with reasonable resources.  This paper is elegant but doesn't change my view on this fundamental question, and in some sense only hardens it: we have all this beautiful math, but what does it really buy us?  I recognize this objection may feel unfair to the authors because it targets the whole field of RL, but given the elegance of the work, I feel the authors might be well positioned to answer it.\n\nQUESTIONS:\nAny further information the authors could give that addresses the weaknesses above would be appreciated!\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The authors propose an approach to construct state abstractions which are then combined with action-abstraction (options) framework in a hierarchical reinforcement learning (HRL) algorithm. The authors provide some empirical evidence of the effectiveness of their approach.

STRENGTHS AND WEAKNESSES:
Strengths:
* A novel approach for state abstraction which is combined with action abstraction approach in a theoretically justified way
* Some experimental evidence demonstrating the efficiency of the proposed approach

Weaknesses:
* Regarding experimental efficiency, it is not fully clear whether this approach actually learns well from scratch on Atari Montezuma Revenge, the authors did not provide any learning curves and comparisons with DQNs. I think generally making options-based agents learning efficiently from scratch is a challenging task.
* At the moment, state abstraction seems to be using k-means to come up with these, which is quite similar to [1]. I think this hurts the novelty of the proposed approach.

Minor suggestions:
* I find the notation \bar{s}' quite confusing, in the text between eq.1 and eq.2. From what I read, `\bar{s}'_i` is a probability of s' under mixture component i. Maybe you can change  `\bar{s}'_i` to actually reflect that it is a probability?

References:
[1] Reinforcement Learning with Soft State Aggregation, Satinder P. Singh, Tommi Jaakkola, Michael I. Jordan, NeurIPS 1994

QUESTIONS:
What it would take to scale this method up to learning from scratch on Atari?

LIMITATIONS:
The method does not seem to be scalable yet.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an approach for building a fully abstract MDP from a ground MDP while previous works either focus on action abstraction or on state abstraction only. To cope with this difficult task the authors propose to focus on a strategy that constructs a state abstraction from an action abstraction, for which a number of approaches have been proposed. They generalize their results to the case of factored actions and factored abstract states. Building such MDPs may be viewed as a step towards powerful reasoning machines.\nThe work is validated on a number of tasks that seem to be built for this evaluation so that comparative results have likely been gained with state of the art strategies run (for instance DQN) on the same tasks. Experimental results show significant improvement of the proposed method over the baselines it is compared to.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper deals with a difficult task and does a great job at arguing the choices the authors made to achieve their work and at explaining the method in its details. The paper is very well written and structured. Still some parts remain unclear to me but it is likely because I am not an MDP expert. As far as I understand the peer focuses on the case where one gets a ground MDP and aim at constructing an abstract MDP. Is there any method to directly cope with the construction of an abstract MDP? \n\nThe experimental section does show clear superiority of the method over baseline on a few tasks on evaluation measure but it does not compare to alternate methods for building an abstract MDP. Shouldn’t the experimental section compare the proposed method with an alternative method (DQN is not such an alternative)?\n\nPart of the contributions are actually described in the Appendix. This is the case for instance of Algorithm 1 which nevertheless looks as a main component of the method. Moreover a number of elements in the method look ad-hoc and need to be chosen by hand by the designer. \nFor instance the use of Kmeans (section 3.4) and the use of an auxiliary loss in section 4.2. How far is this true, how far one needs to instantiate the method to a particular case to get a relevant result?\n\nIn appendix C.1 when you detail the use of a classifier two-sample test, is the k-means what you call the classifier? Couldn’t you use a true classifier, i mean a model learned to discriminate?\n\nI am not sure to understand what is given as input to the method in the examples of section 3.4. As far as i understand from the inputs of the algorithm in Appendix (Algorithme 1), all transitions s_i, I_i, \\bar{a}_i, r_i etc are given, meaning the ground MDP?\n\nIn Appendix A.3 are all options defined by hand ? \n\nCan you define what “approximately model-preserving” mean? It is mentioned in the paper (for instance page 1 in the introduction).\n\nQUESTIONS:\nAs far as I understand, the peer focuses on the case where one gets a ground MDP and aims at constructing an abstract MDP. Is there any method to directly cope with the construction of an abstract MDP? \n\nShouldn’t the experimental section compare the proposed method with an alternative method (DQN is not such an alternative)?\n\nA number of elements in the method look ad-hoc. How far one needs to instantiate the method to a particular case to get a relevant result?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-15"}