Title: Meta-learning how to Share Credit among Macro-Actions
Venue: NeurIPS 2025
Accepted: yes
Keywords: deep reinforcement learning

macro-actions

exploration
Abstract: One proposed mechanism to improve exploration in reinforcement learning is the use of macro-actions, a form of temporal abstractions over actions.
Paradoxically though, in many scenarios the naive addition of macro-actions does not lead to better exploration, but rather the opposite. 
In this work, we argue that the difficulty stems from the trade-offs between reducing the average number of decisions per episode versus increasing the size of the action space. 
Namely, one typically treats each potential macro-action as independent and atomic, hence strictly increasing the search space and making typical exploration strategies inefficient. 
To address this problem we propose a novel regularization term that exploits the relationship between actions and macro-actions to improve the credit assignment mechanism reducing the effective dimension of the action space and therefore improving exploration. The term relies on a similarity matrix that is meta-learned jointly with learning the desired policy.
We empirically validate our strategy looking at macro-actions in Atari games, and the StreetFighter II environment. Our results show significant improvements over the Rainbow-DQN baseline in all environments. Additionally, we show that the macro-action similarity is transferable to other environments with similar dynamics.
We believe this work is a small but important step towards understanding how the similarity-imposed geometry on the action space can be exploited to improve credit assignment and exploration, therefore making learning more efficient.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an algorithm to find similarities across actions such that actions with similar outcomes should have similar Q values. This reduces the “effective” action space since the typical assumption is that actions are independent of each other. It includes experiments on various Atari games and the proposed algorithm shows strong performance compared to baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed algorithm seems simple to implement and is intuitive, without introducing many hyperparameters.\n- The core intuition to exploit structure between similar actions in terms of similarity of outcomes aligns well intuitively.\n- The empirical results are on many Atari games, giving credibility to their algorithm.\n\nWeakness:\n- The comparison to normal Rainbow and naive Rainbow with macro actions make sense, but it seems strange there are no other baselines. In particular, I think the action representation learning literature is particularly relevant. I would encourage the authors to see the following works since they seem relevant (finding actions that are similar in terms of outcomes) [2, 3]. I think these works seem to get at similar questions of exploiting structure in the actions to learn similar representations.\n- The paper should discuss the above action representation learning framework and also work in the options literature [1, 4]. There seem to be strong connections to this literature, and it needs to be discussed.\n\n[1] Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Sutton et al. 1998.\n\n[2] Learning Action Representations for Reinforcement Learning. Chandak et al. 2019.\n\n[3] Learning Pseudometric-based Action Representations for Offline Reinforcement Learning. Gu et al. 2023. \n\n[4] Temporal Abstraction in Reinforcement Learning with the Successor Representation. Machado et al. 2023.\n\nQUESTIONS:\nThere might be a misunderstanding on my end which needs clarification. The paper first makes it seem like the action space is being augmented, but after looking at the method and psuedcode, it does not appear to be the case. Instead it appears that the method is simply finding structure among existing actions in the existing native space (or an augmented action space with artificially added actions). This is an important clarification since the paper makes it seem like the action space is being augmented, which would be similar to the options framework [1], which attempts to add options to the original action space.\n\n[1] Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Sutton et al. 1998.\n\nLIMITATIONS:\nYes, I appreciated the clarity in acknowledging the limitations of their method", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

comments: SUMMARY:
The paper proposes the Macro-Action Similarity Penalty (MASP). The idea is to use the similarity between macro actions to distribute the credit received by a chosen macro-action to all other macro-actions. The problem of using macro actions in Reinforcement Learning it that it can increase dramatically the state space and lead to much worse learning and exploration. So the appropriate use of macro actions has to be dealt cautiously. The principle here is to consider the macro actions jointly. The experiments are done with Atari games and Street Fighter II. The proposed MASP gives better results than Rainbow-DQN and Rainbow-DQN + Macro Actions.

STRENGTHS AND WEAKNESSES:
The paper has good experimental results and a novel idea.

The approach enable faster convergence and increased cumulative reward.

However it might not always be the case in other domains that macro actions are so beneficial and they require a design overhead. The authors wrote a limitation section which is good. It would improve the paper to expose the properties of the problem that make their approach beneficial. 

Strengths

- A novel way to distribute the credits to similar macro actions
- Good results on Atari and Street Fighter II

Weaknesses

- Problems with large action spaces

QUESTIONS:
- The Atari games you test have a small action space. What are the Atari games that pose problem to MASP?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper states that although access to a good set of macro-actions should, in principle, help improve exploration and credit assignment in RL, it leads to worse performance in practice. This is due to the wider range of choices available to the agent at each step, and treating macro-actions as entirely independent of one another.\n\nTo address this, the authors introduce a new regularization term, the Macro-Action Similarity Penalty (MASP), which enables more effective utilization of macro-actions. MASP encourages similar action values (Q) for similar macro-actions (or actions), where the notion of similarity is learned from meta-gradients of the RL objective.\n\nExperiments in Atari and StreetFighter environments show that MASP improves performance in comparison to two baselines – Rainbow DQN that only uses primitive actions and Rainbow DQN that uses macro-actions in the ‘standard’ way (without MASP).\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n**S1.** The paper identifies and addresses a new and interesting problem in using macro-actions for RL. \n\n**S2.** The paper is well-written and easy to follow.\n\n**S3.** The authors are quite measured in their claims and openly discuss the limitations of their proposed approach.\n\n**S4.** The experiments demonstrate that MASP enhances RL with macro-actions. However, the exact reasons for the improvement should be analysed further (see weaknesses).\n\n## Weaknesses\n\n\n\n**W1.** While the use of meta-gradients to learn the $\\Sigma$ (similarity) matrix was quite interesting, it is unclear whether this ties back to the semantic notion of similarity between actions/macro-actions. Since the motivation behind the ideas presented was that macro-actions share a lot of structure, it would be important to verify if this is indeed the case. It seems unclear whether this is the precise reason the idea works. \n\n**W2.** Based on the above point, additional experiments could be useful for a better understanding of the method.  Do the authors have results for lower numbers of macro-actions, like 4, 8, and 16? I would also be very curious to see results with no macro-actions (k=0), as the MASP regularizer can be applied to primitive actions.\n\n**W3.** The paper lacks empirical comparisons and discussions related to options-based exploration, a well-established line of research (e.g., [1]). Do the same problems identified in this paper also apply in that context? Similarly, it would be interesting to know how the presented approach compares to a simple option/macro-action type baseline, such as ez-greedy [2].\n\nOn a minor note, connections to work in temporal credit assignment could further improve the paper. Prior work discusses related ideas, albeit without an explicit emphasis on macro-actions. E.g., Hindsight credit assignment credits all actions that could have led to a future consequence [3, 4]. Chunking or compression can also be applied to abstract away subsequences of state-actions and directly connect to a future result that was reachable through many possible sequences of actions [5]. \n\nI am open to increasing the score if the weaknesses and questions are adequately addressed and clarified.\n\n### References\n\n[1] Klissarov, Martin, and Marlos C. Machado. \"Deep Laplacian-based Options for Temporally-Extended Exploration.\" ICML 2023\n\n[2] Dabney, Will, Georg Ostrovski, and André Barreto. \"Temporally-extended {\\epsilon}-greedy exploration.\" ICLR 2021\n\n[3] Harutyunyan, Anna, et al. \"Hindsight credit assignment.\" NeurIPS 2019\n\n[4] Meulemans, Alexander, et al. \"Would i have gotten that reward? long-term credit assignment by counterfactual contribution analysis.\" NeurIPS 2023\n\n[5] Ramesh, Aditya A., et al. \"Sequence compression speeds up credit assignment in reinforcement learning.\" ICML 2024.\n\nQUESTIONS:\nIncluded in “Strengths and Weaknesses”\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Macro-Action Similarity Penalty (MASP), a regularization term that treats primitive actions and macro-actions as lying in a shared similarity space and encourages the Q-values of similar actions to stay close; this reduces the effective action‐space dimension, improving credit assignment and exploration efficiency. MASP meta-learns the similarity matrix jointly with the policy via meta-gradients, so the inductive bias adapts online instead of relying on handcrafted clusters. Built on a Rainbow-DQN, the method makes large gains on Atari and Gym-Retro StreetFighter II, remains robust as the number of macro-actions grows or when macros are noisy, and the learned similarity matrix transfers well across related games. These results show that exploiting action-space geometry can make macro-action augmentation practical without expensive hierarchical architectures.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n- The paper clearly raises the challenges of simply applying macro-actions, noting that adding macro-actions actually enlarges the exploration space. To address this issue, the paper introduces the Macro-Action Similarity Penalty (MASP), which reduces the exploration action space by penalizing the similarity of macro-actions, with clear motivation and a simple formalization using an ℓ₂ penalty term.\n\n- The similarity matrix $\\Sigma$ is optimized jointly with the policy via a meta-gradient procedure, allowing the inductive bias to evolve with the agent and environment dynamics rather than being fixed a-priori .\n\n- Integrating MASP into a Rainbow-DQN backbone yields significant improvements across a broad range of Atari games and complex Street Fighter II combos, while keeping the underlying network and training hyperparameters unchanged.\n\n- MASP maintains high performance as the macro-action set grows from 64 to 1024 actions (Table 2) and when up to 75 % of macros are replaced by random sequences (Table 4), where the naïve macro-action baseline collapses .\n                                                                                                                                                           \n\nWeaknesses:\n- The similarity matrix $\\Sigma$ grows quadratically with the size of the (primitive + macro) action set and must be projected at every update, so training time and memory rise steeply as the action space becomes very large or continuous.\n\n- MASP’s gains presume a reasonably “good’’ pool of macro-actions. If the extracted sequences are irrelevant or mis-aligned with the task, performance can deteriorate; automatic discovery of high-quality macros remains an unsolved prerequisite.\n\n- Although $\\Sigma$ sometimes transfers across games with similar dynamics, its interpretability is unclear and performance drops markedly when the source and target environments differ substantially.\n\nQUESTIONS:\n- Do you have any intuition about why transfer learning with $\\Sigma$ works well even though $\\Sigma$ is not a variable dependent on any states or history?\n\n- Since $\\Sigma$ is not dependent on any states or the history of macro-actions, even with the same source and target environments, if the order of the macro-actions is changed in the target environment, performance may degrade. Could you provide an experiment in which the order of macro-actions is modified?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}