Meta-learning how to Share Credit among Macro-Actions
Authors: Ionel Hosu
Traian Rebedea
Razvan Pascanu
Venue: NeurIPS 2025 poster ()
Keywords: deep reinforcement learning
macro-actions
exploration
Primary area: reinforcement_learning
TL;DR: We propose MASP, a meta-learned similarity-based regularization for RL with macro-actions. MASP improves exploration, credit assignment, and transfer across tasks, outperforming Rainbow DQN in challenging benchmarks.

Abstract: One proposed mechanism to improve exploration in reinforcement learning is the use of macro-actions, a form of temporal abstractions over actions.
Paradoxically though, in many scenarios the naive addition of macro-actions does not lead to better exploration, but rather the opposite. 
In this work, we argue that the difficulty stems from the trade-offs between reducing the average number of decisions per episode versus increasing the size of the action space. 
Namely, one typically treats each potential macro-action as independent and atomic, hence strictly increasing the search space and making typical exploration strategies inefficient. 
To address this problem we propose a novel regularization term that exploits the relationship between actions and macro-actions to improve the credit assignment mechanism reducing the effective dimension of the action space and therefore improving exploration. The term relies on a similarity matrix that is meta-learned jointly with learning the desired policy.
We empirically validate our strategy looking at macro-actions in Atari games, and the StreetFighter II environment. Our results show significant improvements over the Rainbow-DQN baseline in all environments. Additionally, we show that the macro-action similarity is transferable to other environments with similar dynamics.
We believe this work is a small but important step towards understanding how the similarity-imposed geometry on the action space can be exploited to improve credit assignment and exploration, therefore making learning more efficient.

Meta-learning how to Share Credit among Macro-Actions
Ionel-Alexandru Hosu
Traian Rebedea NVIDIA & Politehnica University of Bucharest
Politehmica University of Bucharest Bucharest, Romania
ionel. hos.ucs..s...o.......o.........s aiosu@cs.pub.ro
Bucharest, Romania
trebedea@nvidia.co
Razvan Pascanu
Mila Quebec Artificial Intelligence Institute
Montreal, Canada
Abstract
One proposed mechanism to improve exploration in reinforcement learning is through the of macro-actions. Paradoxically though, in many scenarios the naive addition macro-art macro-actions does not lead to better exploration, but rather the opposite. has been argued that this was caused by adding non-useful macros and multiple works have focused on mechanisms to discover effectively environment- specific useful macros. In this work, we take slightly different perspective. We argue that the difficulty stems from the trade-offs between reducing the average number of decisions per episode versus increasing the size of the action space. Namely, one typically treats each potential macro-action as independent and atomic, hence strictly increasing the search space and making typical exploration strategies inefficient. To address this problem we propose novel regularization term that exploits the relationship between actions and macro- actions to improve the credit assignment mechanism by reducing the effective dimension of the action space and, therefore, improving exploration. The term relies on a similarity matrix that is meta-learned jointly with learning the desired policy. We empirically validate our strategy looking at macro-actions environment. Our results show significant improvements over the RAINBOW- DQN baseline in environments. Additionally, we show that the macro-action similarity is transferable related environments. We believe this work is small but important step towards understanding how the similarity-impor the action space be exploited to improve credit assignment and exploration, therefore making learning more effective.
Atari games, and the StreetFighter
geometry
Introduction
While Reinforcement Learning (RL) lead a plethora of successes [e.g. [15). being an integral component of modern LLMs as well [c.g. [6, efficient exploration remains significant challenge. This is particularly in environments with large or complex action spaces or sparse rewards that require to do credit assignments over long episodes. One potential approach to simplify exploration the use of macro-actions [e.g. IO, fixed sequences of actions that are frequently used by the policy. By expanding the action space with acctonsctions, and using them efficiently, an agent can reduce considerably the number of decisions that it needs take during an episode, hence making the exploration easier.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Fighter
wwP
Hio
Aacro-ctionss
Macrodation
800
10000
50020 100000 130395
205600
30000 300030
(a) Atari Average Human Normalized Scores
(b) Street Fighter II Scores
Figure I: Visual comparison between Rainbow-DQN, Rainbow-DQN + Macro-Actions and Macro- Action Similarity Penalty
However it is often that even when one has access good proposal distribution for macro-actions, using them leads to worse performance. This is due to the fact that even the average length of an episode might decrease, in many of available choices at each step can increase considerably (see also Figure Tap.
the exploration space actually becomes larger, the number
In this work, attempt to exploit the fact that some of these macro-actions are inherently related, not only among themselves but also to the basic actions. By simply increasing the action space with additional macro-actions independent. But often, due to the sharing of basic actions between these sequences, the outcome of different macro-actions in the environment can be somewhat similar. We propose capitalize this similarity to distribute the credit received by chosen macro-action to all other macro-actions according to their similarity. This effectively reduces the intrinsic dimensionality of the action space, therefore reducing the size of the search space and improving exploration.
implicitly assuming that each macro-action is atomic and
The similarity between macro-actions acts as an inductive bias, which can be either handcrafted given a good understanding the available macro-actions, or, as we will show in Section 3 can be meta-learned from the data. For simplicity, in this work we assume that the similarity takes the form of a fixed kernel, Σ, that is used to push Q values towards each other proportional to the similarity strength. Note that make explicit distinction between actions and macro- -cctins We refer to this regularization term as the Macro-Acuor Action Similarity Penalty (MASP).
We apply this method in two scenarios. First, in Atari games, where macro-actions are extracted using a simple frequency-basee -ased heuristic over recorded trajectories, capturing common sequences of basic actions. The second scenario is the StreetFighter II environment, where each player has specific sets of action combos that result in more powerful attacks. We use these combos as potential macro-actions, incorporating all available combos regardless of the character used by the agent. Our experimental results demonstrate that the proposed method significantly improves learning efficiency on top of RAINBOW DQN [1]], see Figure Agents trained with our regularization approach exhibit faster convergence and higher cumulative rewards. By bridging the gap between extended action spaces and efficient exploration, this work contributes novel approach that leverages the inherent geometry of the action space, potentially benefiting a wide range of reinforcement learning applications with structured action spaces.
2
Related work
Macro-actions, temporally extended actions, have significantly contributed to improved exploration efficiency within reinforcement learning (RL) frameworks. Initial studies by Randlov [12] introduced systematic methods to construct macro-actions from primitive actions, accelerating learning in tasks such as bicycle balancing and grid-world navigation. Similarly, McGovern and Sutton [13] empirically confirmed their benefits, showing faster discovery of optimal solutions compared to primitive exploration.
Macro-actions in Deep and Meta-RL. Recent work has extended macro-actions to deep reinforce- ment learning settings. For example, Frans et [14 explored meta-learning strategies that enable hierarchical agents to generalize by reusing temporally extended skills. More recently, Cho and Sun [55 proposed a tri-level meta-RI. framework that learns task-agnostic macro-action across environments. Our work draws inspiration from such architectures but focuses meta- meta-learning

---

similarity structures among macro-actions to enhance credit assignment rather than hierarchical decomposition alone.
Macro-actions in Multi-agent and Decentralized RI. In decentralized and multi-agent settings, macro-actions have proven effective for improving exploration and coordination. Tan et al. [T6] proposed the Macro Action Decentralized Exploration Network (MADE-Net) to deal with com- munication dropouts in cooperative multi-agent environments. Similarly, Xiao [17] introduced both decentralized and centralized macro-action value learning approaches, showing scalability and coordination gains.
Intrinsic Motivation and Macro-actions. Macro-actions have also been combined with intrinsic motivation to promote deeper exploration. Panda et al. [T8] presented a framework blending intrinsic and extrinsic rewards across timescales, enabling agents to develop strategic behaviors involving macro-action.
Transferability and Reusability. Macro-actions are particularly valuable for transfer learning. Chang et al. [99 demonstrated that macro-actions are reusable across tasks and algorithms and transferable to new environments with different reward functions. Such findings suggest that macro- actions can encode abstract behaviors robust to minor environment changes.
Planning with Macro-actions. Outside RL, macro-actions have been leveraged in classical planning. Allen et al. 20 proposed focused-effec imcro-aactoons to improve heuristic accuracy in black-box planning, while Newton et al. [21] integrated macro-actions into planning heuristies, showing better performance in relaxed planning graph search. These results emphasize the role of macro-actions not only in learning but also in accelerating symbolic planning through structured action abstractions.
Macro-actions for POMDPs. To deal with partial observability, Amato et al. [22] studied macro- actions in decentralized POMDPs, proposing solutions that integrate temporally abstracted behaviors into tree-based planners. Lee et al. [X introduced MAGIC, which uses learned macro-actions to improve online planning in POMDPs by tailoring the action space situation-ave aware behavior.
Hierarchical and Object-Oriented Macro-Actions. In the context of spatial reasoning, Hakenes and Glasmachers 24 proposed object-oriented macro-actions based topological maps, which provide inductive structure for learning in visually complex and sparse environments. These topological abstractions act priors for exploration and planning.
Our Contribution. Despite these advances, integrating macro-actions in a scalable and flexible way remains challenging, particularly when they are treated independent and the relationships among them are ignored. Our approach fills this gap by introducing a meta-learned similarity matrix for macro-actions, encouraging credit sharing across related actions through regularization mechanism. This not only improves exploration in large action spaces but also adapts the agent's inductive biases during training, aligning with the broader goals of cructure-aware learning in RL.
3 Methods
3.1 Background
We assume the standard MDP formulation, where M AS,A, P,r, Pinit; 7) consisting of state space S, the set of all possible actions A, a transition distribution P(s' [s, a) and a reward function r: S x A + F. The goal of an agent to find a policy that maximises the expected sum of discounted rewards: Q* (s, a) = Exp-paanis ~-)(s,) |E T5=oors(ss a.).. In our work we focus on value based methods, which minimize the temporal difference (TD) error in order learn the Q- function and define ग greedily with respect to it. We are specifically assuming that Q is approximated by a neural network. The prototypical algorithm DQN (25 where Q is learned by minimising:
L(()=
E (s.0,7.0)) DD
(Qo(s,a) a) - (r + 7 max (o)())a)))
Multiple works, after the seminal DQN paper 25 had improved this algorithms, as for example agent C51 (26] who instead of modeling the return directly, predicts a categorical distribution of expected
returns. RAINBOW 27 includes several improvements on top of C51, ranging from prioritised
experience replay 28 to n-step returns for lower variance bootstrap targets, double Q-learning
29 disambiguation of state-action value estimates [30], and noisy nets [31]. In our work we use
3

---

RAINBOW as our baseline and build on top of it by augmenting the action space with Macro-Actions and explore our proposed regularization term MASP.
3.2 MASP: Macro-Actions Similarity Penalty
One specific intuition behind extended actions, shared with concepts like macro-actions or options, is that they reduce the number of decisions that we need to take within the episode, and hence make learning easier (from a credit assignment point of view). Simply put, having fewer decisions in an episode means that it easier to see which of them contributed to the observed return.
Another important element of learning in RI, however, is that of exploration. And while shorter episodes should make exploration easier, when we consider an extended action space, the outcome can be that exploration becomes considerably harder, given that the number of choices per step increases. One intuitive way of understanding this behavior that the RL algorithm has to explore. If we make the assumption that for any given state, any of the actions leads to unique outcome (i.e. different state), then the search space size should have the form fN, where f is the branching factor, or number of actions per step, and N is the number of decisions the agent needs to take. One can now easily see that N' < N does not guarantee that f'N' < f if reducing N' requires increasing the branching factor, i.e. I' > f which the usual situation when adding macro-actions to the action space. Indeed, it is more likely that increasing the branching factor will increase the search space faster then decreasing the number of decisions taken per episode.
consider the size of the search space
The main assumption in the above reasoning is that each new action added to the action space is unique or leads to unique outcome assumption typically made by standard exploration strategies like e-greedy. However for typical environments this is not the case. The search space is highly structured, which is the main reason we can learn reasonable policies over extremely-large search spaces. This structure typieally exploited in deep RL through the use of neural networks as function approximators. In this work we exploit this further, specifically looking at the structure of the action space. We make the assumption that certain actions lead, on average, to similar outcomes. This will naturally impose a clustering of actions, and allow learning and exploration to first identify the correct cluster for a given state and then the fine-grained differences between actions within each cluster.
We propose to codify this intuition into a regularization term we refer to as the Macro-Action Similarity Penalty (MASP). The penalty enforces that any cluster of similar actions can not disperse too much in their Q-values, and, more importantly, learning has to move the entire cluster together, allowing Q-values learn from each other. Crucially, the term allows for some dispersion which can capture some degree of differentiation between actions, which is needed to learn reliable policy.
3.3 Detailed Formulation
Formally, we assume the existence of similarity matrix, Σ E R|AIX|AI, where each element Σgg represents the similarity between actions a; and a, marginalized over all possible states. Specifically, high Σg, encodes the inductive bias that Q; and Q, should not differ substantially, regardless of state. The similarity matrix is symmetric (Eij = Ej;) and non-negative.
To operationalize MASP, we incorporate it into the standard Temporal Difference (TD) loss as an additive term weighted by a respect Σ. Given a sampled batch of transitions (8i, ai Ti, $141) we have:
1) which decides how much we restrict the Q-values to
MASP = 17 [[Q(s 0) - EQ( (s; 1)
(2)
Intuitively, this loss penalizes large differences between the Q-values of similar actions, promoting a smoother and more structured Q-function landscape.
3.4 Meta-learning! Σ
The efficacy of our approach depends on choosing an adequate similarity matrix Σ. For example, setting Σ to the identity matrix, II, will reduce MASP to not have any effect. This is true if we also set Eng = for all i and j. While in principle one can use domain knowledge to define clusters of similar actions, in this work we rely on meta-learning approaches to learn directly from data

---

relevant clustering during exploration. Note that Σ is learned jointly with 8, therefore being able to track the current policy Tg. rather than being forced to marginalize over policies.
Specifically, we apply a meta-gradient framework inspired by Xu et al. 32 to automatically infer the entries of Σ. Following the meta-gradient methodology, we alternate between phases:
In the first phase, we apply a standard RL update to the agent parameters using a sampled trajectory under the current similarity metric Σ. This yields updated parameters 0'.
In the second phase, we evaluate the quality of the new 6' on a held-out trajectory r', using differentiable meta-objective 0'(r'.0'. where J' is just the TD-error term. Note that J' is function of Σ only through 0' due to the update done on the augmented objective J containing our MASP regularization term (see Algorithm| for details). (We clone the network parameters to obtain 8' after simulated update step, as required for meta gradient computation 32
The meta-gradient V y is computed using the chain rule as follows:
OJ' 0')
0J'
d0'
ac
00'
dΣ'
as This gradient then used update Σ using stochastic gradient descent, thereby enabling the agent to dynamically learn which macro-actions should be treated as functionally similar. The precise implementation. including hyperparameter settings and gradient handling, will be made publicly available with our code. Additional details about the proposed MASP regularization and its implementation can be found in Appendix A.
and the trace-based approximation [32] allows for efficient online accumulation of de'
Algorithm 1 Meta-learning Credit Assignment with MASP
Require: Replay buffer D, macro-actions set Amaro. initial network parameters 0 target network parameters 8 similarity matrix Σ (meta peeeta-arameters), learning rates a and 83, regularization
coefficient 7)
1: Initialize Q-network Q(s, a; 0) and target network Q (s)a;0-)
2: for each environment step / = to T do
3: Select action a, using e-greedy over Q( a;
4: Execute a, observe Tı, $1+1
5: Store transition (st, ar, Ft: 8(41) buffer
6: if time update then
7: Sample trajectory 7 -= {($;, as. Vi, 81+1) } from buffer D
8:
Compute similarity embedding es from E via learned projection
9:
Compute target: = т; + max,' Q (81+1, a';0
Compute standard TD loss: LTD > (Q 0;; -
11:
Compute MASP regularization: LMASP = 1) (s; -200))00ss 2Q 00]1
Update network parameters: + a V (LTD LMASP) I/ Meta-gradient phase
13:
Clone network parameters: 0' +
15:
Sample second trajectory 7' from buffer D
Compute peba-obbectivv Luera (TD on r'
17:
Compute meta preta-gradeen: Vm via chain rule
Update Σ BV s.eetaa
19:
end if
if time update target network then
21: 0 0
22:
end if
23:
end for
To cope with the non-stationarity introduced by the evolving similarity metric, we condition the value function and policy on E using a low-dimensional embedding ex, akin to Universal Value Function
Approximators (UVFA) [3). This enables the network to generalize across changes the structure of Σ and adapt smoothly during lcarning.
5

---

Human Nnrmwddoed Scores Earm (AI Heltiad,, "Alch Croppedd 1000)
Figure 2: Visual comparison between Rainbow-DQN (dark blue), Rainbow-DQN Maacro-Actions (light blue), and Macro-Action Similarity Penalty (red).
Learning the Similarity Embedding ex:. A common and effective approach for representing the similarity matrix Σ to learn a low-dimensional embedding eg jointly with the main network parameters. Instead of using the full (potentially large) matrix E directly as network input, first flatten and map it to a compact vector via a trainable linear transformation:
eg
W Wemb vec >),
where Wemb is learned weight matrix and vec(E) denotes the flattened version of Σ. In our setup, Wemb is only updated via gradients from the main loss J, and not directly from the meta-objective J'. See Appendix A for more implementation details.
The resulting embedding eg is then concatenated with other input features (such as the state repre- sentation) and fed into the network as additional context. During training, ey is Iearned end-to-end Σ evolves.
backpropagation, so the agent can flexibly adapt its notion of action similarity This approach allows the network to capture the most relevant information about action similarities, supporting both efficient exploration and generalization, and is preferable static or randomly projected embeddings when Σ contains task-relevant structure.
This meta-learning mechanism improves credit assignment by promoting shared learning signals among semantically related actions especially macro-actions, and leads to a more efficient explo- ration of the macro-action space. It also enables the agent adapt its notion of action similarity as the environment dynamics and learning context evolve.
4 Experiments
Game
Rainbow-DON
*Macro-Actions
+MASP
Breakout
379.5225.1
2529
884,4 74.0
Frostbite
4.141.1 = 175.4
3,361.2
5.566.7 = 317.3
Ms Pacman
2,570.2 # 204.6
1,471.5 185.3
2,966.5 + 36015
Seaquest
19,176.0 + 1.428.9
13,917.1 + 641.1
23,768.8 + 1,476.9
Space Invaders
12,629.0 1,413.2
7,496.4 532.9
16,668.2 2,050.5
Table 1: Comparison between RAINBOW DQN [III, RAINBOW-DOG -DQ with macro-actions (+Macro- Actions), and RAINBOW DQN with macro-actions similarity penalty (+MASP). Bold indicates maximal raw performance between RAINBOW DQN and MASP. shown in parentheses. Cells highlighted pink denote HN 2 100. This table is a cropped version for readability; the full results across all games provided in the Appendix.
(HN) scores

---

Game
# of macro-actions
+Macro-Actions
+MASP
248.1 13.9
7
128
2.6
727.0 17.0
Breakout
256
147.6
564.9 = 26.1
1024
0.5
379,5 + 30.1
3,142.2
142.7
5,204.2 1 401.7
128
2,853.5
40.0
5,174,9 246.7
Frostbite
256
1,492.1
100.3
4,446,6 + 314,7
1024
470.1 * 26.8
2,477.4 188.7
1.752.1
52.2
2,445.5 = 35,9
128
1,569.4
58.3
2,144.3 + 38.4
Ms Pacman
256
1,129.6 71.6
1,977.3 140.9
1024
246.0 18.8
9.3.4 + 75.4
13,043.7 a. 576.0
21,966.7 1 1065.0
128
13,164. + 599.8
18,766.7
1550.6
Seaquest
256
12,453.5 # 734.7
16,897.4 + 802.5
8.332.6
13,854,9
1024
198.9 4,333.8
9,144,7 * 723.1
7.217.1 + 451.9
15,566.9 846.2
128
8,325.0
389.1
15,1%.33 1167.8
Space Invaders
256
6,913
# 583.7
13,778.0 + 683.1
1024
1,468.4 49.5
6,793.5 + 314.3
Table 2: Ablation study showing the effect of varying the number of macro-actions (k E 64, 128, 256, 512, 1024) on performance across five representative Atari games. While the base- line with macro-actions suffers from severe performance degradation as the action space grows, MASP maintains strong and stable performance, demonstrating its robustness action space size and ability to leverage structural similarity.
Task
Rainbow
+Macro-Actions 184,789
+MASP
Macro-actions
187,556 + 3713.3
7628.2
236,943
13676.0
Macro-actions 11
337,388.8 23829.9 565,319.2 9441.2 593,784
18262.9
Table 3: Performance comparison in the StreetFighter II environment across macro-action of increasing complexity. MASP consistently outperforms both the RAINBOW-D BANBOW-DNNNNN baseline and its macro-action-augge aaggene- variant. Notably, gains are amplified in the more complex Macro-Actions setting, highlighting MASP's ability to scale with the size of the extended action spaces.
P(replace)
+Macco-Actions
252.9 22.2
74.
0.25
208.3.3 22.6
814.3 37.7
Breakout
186.5
663.9
0.75
76.4 4.8
503.77 39.6
3,361.2
356.4
5566.7 : 317.3
0.25
3,071.9
188.7
4,855.2 a 353.7
Frostbite
0.5
83.7 2,173.4
4,194.0 343.3
0.75
1,366.3
3,472.9 272.9
1,471.55 185.3
29666.5 a: 3640.5
0.25
1,145.9
108.4
2,454.2 255.9
Ms Paeman
893.3 87.9
2,144.2 212.6
0.75
606.5 # 65.8
1,884.5 153.8
0.25
12,593.0
250.4
21,619.8 1586.2
Seaquest
9,194.8
562.5
19.913.5 + 1296.9
0.75
3,732.0 172.1
17,581.0 4 234.9
7,496.4 532.9
16,668.2 2050.5
865.0
Space Invaders
0.5
3,836.2 727.3
13,021.7 + 238.1
0.75
2,784.8 = 116.0
10,48199 702.6
Table Performance under macro-action noise. Each macro-action replaced with random se- quence of the same length with probability P(replace) e 0.25, 0.5, 0.75. MASP remains consistently effective across all noise levels, while RAINBOW-DON Macro-Actions degrades sharply, indicating MASP's robustness to imperfect or corrupted macro-action sets.

---

Source of Σ
Target Env.
Breakout
Montezuma''s 's Revenge
Private Eye
Space Invaders
RAINBOW DQN
Breakout
74.0
884.4 74.0
25.1
Montezuma's Revenge
162.1 + 12.5
400.0 + 18,1
376.8 + 19.2
213.6 + 11.4
154.0 16.9
Private Eye
1,432 # 117.4
2,074 216.8
2244.3 150.9
1,311 138.5
1,704.4
Space Invaders
15,294 + 2.169.3
7,869.4 614.5
7,483.7 = 544.8 16668.2 2050.5 12,629.0 1,413.2
Table 5: Transfer learning results. Each column denotes the environment used to train (source) Σ, and each row is the evaluation (target) environment. Diagonal elements represent transfer of Σ from the same game to itself; off-diagonals are transferred Σ from different games. Last column is original R RAINBOW performance for reference. Note that R RAINBOW DQN with macro-actions and without MASP underperforms, hence why it was not added to the table.
Experimental Setup. We rigorously evaluate our proposed Macro-Action Similarity Penalty (MASP) across a diverse and challenging set of environments, including suite Atari 2600 games from the Arcade Learning Environment (ALE) (34) and the complex, structured action environment of StreetFighter II from Gym Retro (35). These benchmarks represent scenarios with varied complexity and exploration demands, making them ideal for assessing the benefits of our approach.
All agents are trained for a substantial 2B frames in Atari and 500 million frames in StreetFighter II. using standard preprocessing steps and hyperparameters tuned per game via validation sweeps. Crucially, all agent variants share identical architectures and optimization hyperparameters, differing only in the application of macro-actions and the MASP regularization.
Implementation Details. For the Atari experiments, we build upon the RAINBOW-DQN baseline, a robust and widely recognized reinforcement learning algorithm. Macro-actions were identified via a simple frequency- heuristic, capturing frequently repeated sequences of primitive actions within successful trajectories. To thoroughly demonstrate MASP's effectiveness, we compare three agent variants: (1) standard RAINBOW-DQN, (2) RAINBOW-DQN augmented with independent macro-actions, and (3) RAINBOW-DON enhanced with proposed MASP method, which leverages the meta- meaanee similarity matrix. In Appendix B we provide all training hyperparameters and additional details for reproducibility.
Atari Macro-Actions. To construct meaningful macro-actions in Atari, we leverage the Atari Grand Challenge Dataset (6), which provides human expert trajectories. We extract the top-k most frequent action subsequences of length 3 to 8, where &o = 32 for the main results reported in Tabldg These action subsequences serve candidate macro-actions and are appended the base primitive action set to form the augmented action space.
compelling evidence for MASP's advantage both baselines. Notably, MASP consistently achieves higher cumulative rewards and demonstrates faster convergence across almost all Atari games tested. In many games, MASP dramatically improves upon RAINBOW-DD -WNN achieving performance that exceeds human-level benchmarks (a human- normalized score of at least 100). This improvement confirms our hypothesis: explicitly leveraging macro-action similarity significantly enhances credit assignment and exploration efficiency. For detailed experimental results see Appendix C.
Results and Analysis. Table provides
Ablation on Macro-Action Size.
We explore how MASP and baselines behave as the number of
macro-actions increases. As shown in T we sweep & E 64, 128, 256, 512, 1024 and observe that while RAINBOW increased action space complexity MASP maintains strong performance throughout. This highlights MASP's robustness and effectiveness in managing large action spaces.
macro-actions suffer catastrophically
k increases presumably due to
Robustness to macro-action reordering. Another important clarification has to do with macro- action reordering. If the macro-actions are reordered, and the same permutation is applied to the rows and columns of Σ. the MASP regularization remains mathematically identical. This is because MASP only enforces that Q-values for actions with high similarity (as specified in Σ) are close - does not rely on any fixed semantic meaning being attached to particular index. In practice, we may choose to order macro-actions consistently (e.g., grouped by length or composition) solely to aid human interpretability and visualization, but this ordering is not required for the algorithm and
8

---

does not influence performance. The learned structure in Σ generalizes across environments due to behavioral similarity between macro-actions, not index alignment.
Noisy Macro-Actions. To test MASP's robustness to imperfect macro-action sets, we conduct controlled noise ablation experiments. Specifically, for a fixed k = 32 macro-action set, we ran- domly replace each macro-action with a random sequence of the same length with probability P(replace) E (0.25, 0.5, 0.75). This simulates conditions where the macro-action set is partially or entirely corrupted. As shown in Table while Rainbow + Macro-Actions degrades sharply as noise increases, MASP remains remarkably stable and continues to provide substantial performance benefits, even at high noise levels. This suggests that the similarity structure learned by MASP is resilient and can cluster together non-informative macro-actions, learning easily to ignore them.
Dependency on the quality of Macro-actions. Another important point note is that MASP relies on the initial macro-action pool being at least partially meaningful. Automatic discovery of high-quality macro-actions remains open problem. That said, Table 4 shows that MASP remains robust even when 75% of macro-actions are randomly corrupted - suggesting the meta-learned similarity structure is resilient to noise and can learn to attenuate the influence of irrelevant macros. Future work could combine MASP with recent macro-action discovery methods (c.g., trajectory clustering, unsupervised option discovery), where MASP might serve regularizer to prune or refine an evolving macro pool.
For StreetFighter II, we utilize domain knowledge to define macro-actions explicitly as attack combinations (combos). Two distinct sets macro-actions are considered: simpler (Macro-Actions 1) and more complex combos (Macro-Actions II), testing MASP's adaptability and scalability across varied levels of action complexity. Table 3 highlights MASP's versatility. In the Macro-actions II scenario (complex combos), MASP significantly outperforms both the standard RAINBOW- B-NN and Macro-Actions baselines, demonstrating clear benefits in structured, combinatorially large action environments. Even in simpler scenarios Macro-action I), MASP at least matches the best baseline, affirming its adaptability and broad utility.
StreetFighter II Results.
Transfer Learning and Generalization. An additional exciting finding (see Table and Appendix C for more detailed results) indicates that the learned macro-action similarity matrix Σ generalizes well to related tasks, suggesting that MASP not only improves individual task performance but also facilitates knowledge transfer across related environments. Here keep E frozen to the one learned on the initial task (source domain), and relearn the policy with macro-actions under the frozen similarity metric. This transferability has profound implications for practical applications, where retraining from scratch for each new task is costly or infeasible. Notably, the results show strong transferabiliity of the similarity matrix Σ between games with similar structure and mechanics, such as between Montezuma's 's Revenge and Private Eye or between Breakout and Space Invaders, where performance remains close or above the in-domain baseline. In contrast, transferring between more distinct games leads to pronounced drop in performance, indicating that the benefits of transfer are greatest when the source and target environments share underlying dynamics or action semantics.
In Table 6, we show that Σ generalizes best across environments with similar transition dynamics and interaction semantics (e.g., Breakout + Space Invaders, Montezuma - Private Eye). This supports the view that Σ captures behavioral regularities across macros, rather than task-specific overfitting.
Our intuition is that although Σ is not conditioned on state, it meta-learns how Q-values should co-vary across actions that yield similar trajectory-level effects. For example, in Breakout or Space Invaders, many macros correspond to repeated fire-move patterns. Thus, Σ learns to group such macros - and this grouping remains useful when transferred to related games where the macro-level structure of agent-environment interactions is similar. This is analogous to how convolutional filters in vision models can transfer even though they are not image- or task-specific: they encode generic structure that is often reused.
Interpretability. Regarding interpretability, we visualize Σ (Figure 3 in the Appendix) and observe clear block structures corresponding to groups of macros with similar behavioral outcomes. Still, we agree that E not trivially interpretable in semantic terms, and improving this is an exciting direction e.g., via structured priors, sparse constraints, stutering-bbsse analyses.

---

Discussion. Collectively, these results robustly demonstrate MASP's value as ceneral-purposs technique that substantially improves learning efficiency, exploration. and final performance. Our extensive experimental validation across diverse environments emphasizes MASP's potential to significantly advance reinforcement learning methods that utilize macro-actions, highlighting the importance and effectiveness of intelligently structured credit assignment.
5
Limitations.
While our approach demonstrates significant empirical improvements in exploration and credit assignment through the use of macro-action similarity regularization, several limitations remain. We hope future research will address these limitations by exploring scalable approximations, improved macro-action discovery, and broader algorithmic applicability.
Parametrization of Σ. Our choice of parametrization makes Σ independent of state, and of current policy we (though there is dependency æ through the meta-learning process, as Σ evolves jointly and possibly tracks т). Conditioning Σ on state or 0 could capture the structure the action space better, though might make the meta-learning task considerably harder.
Scalability Very Large Action Spaces. Although MASP improves robustness to the number of macro-actions, its computational cost grows quadratically with the size of the action space, as the similarity matrix E must be maintained and projected at each update. Scaling to extremely large or continuous action spaces may require approximate or structured representations of similarity. Storing
and projecting a full Σ E ||A[X|A| can become expensive for extremely large action spaces. To mitigate this, we already use low-dimesiional learned embedding of Σ (see Appendix A.3), which is passed to the Q-network as a context vector. While the full matrix is still used during regularization, is only applied over the batch of Q-values each update (not the entire action space per state). This allows for practical efficiency even with 1024+ actions, as shown in Table 2. In future work, we plan to investigate structured or sparse approximations (e.g., low-rank, hernel-based to make MASP scalable to continuous or combinatorial large action spaces.
Quality of Macro-Action Extraction. The success of MASP relies on having a of meaningful macro-actions. While we use frequency-based heuristic on human trajectories, the performance may degrade the extracted macro-actions are not relevant or are poorly aligned with the task structure. Automatic or adaptive discovery of optimal macro-actions remains an open problem.
Interpretability and Generalization of Similarity Structure. Although observe some transfer- ability of the learned similarity matrix across tasks, its interpretability and generalization properties not fully understood. Further work is needed analyze when and why meta-learned similarities capture useful domain knowledge.
Computational Overhead. Incorporating meta-learning and the additional regularization term introduces extra computational cost per training iteration compared to standard DQN baselines. In resource-constrained settings, this may limit applicability.
6
Conclusion
We regard our work as a small step towards improving the credit assignment problem when dealing with augmented action space by better using the geometry imposing structure on enlarged action space. Given a similarity metric, MASP allows similar actions to move together, effectively one learning from the other. We further demonstrate that the similarity metric can be learned efficiently and simultaneously with the policy, exploiting the meta- preta-gradient framework. The overall proposed method MASP leads effective and robust use of macro-actions, and improvements on Atari and StreetFighter. We believe our framework can be expanded further either for improving interpretability of action spaces or exploit more state or policy dependent similarities.
This research was supported by the project "Romanian Hub for Artificial Intelligence "RIA", Smart Growth, Digitization and Financial Instruments Program, 2021-2027, MySMIS no. 351416.
Acknowledgements..
10

---

References
[1] David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mas- tering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
[2] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander Sasha Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Le Paine, Çaglar Gülçehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy Lillicrap. Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft II using multi-agen agent reinforcement learning. Nat., 37507782)330--34,, 2019. doi: 1038//4/ 20038/411500 186-019-7724-2
[3] Christopher Berner, Greg Broekman, Brooke Chan, Vicki Cheung, Przemyslaw Dkebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 19/2. 2019.
[4] Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-88 82, 2020.
[5] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414 419, 2022.
[6] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe, Training language models to follow instructions with human feed- back. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781 138777717711111 1088.
[7] Milos Hauskrecht, Nicolas Meuleau, Leslie Pack Kaelbling, Thomas Dean, and Craig Boutilier. Hierarchical solution of markov decision processes using macro-actions. Proceedings of the Fourteenth Conference Uncertainty in Artificial Intelligence, UAI'98, page 220-229, San Francisco, CA, USA, 1998. Morgan Kaufmann Publishers Inc. ISBN 155860555X.
[8] Richard E. Fikes and Nils Nilsson. Strips: A new approach to the application of theorem proving problem solving, Artificial Intelligence, (3):189-208) 1971. ISSN 0004-3702. doi: http:///..or////00/00//000000110
URL httph: http://wwwwwinncceeeeecc.ec com/
[9] M. A. Hakim Newton, John Levine, Maria Fox, and Derek Long. Learning macro-actions for arbitrary planners and domains. In Proceedings of the Seventeenth International Conference on International Conference on Automated Planning and Scheduling, ICAPS' 07, page 256-263.
AAAI Press, 2007. ISBN 9781577353447.
[10] Ishan P. Durugkar, Clemens Rosenbaum, Stefan Dernbach, and Sridhar Mahadevan. Deep reinforcement learning with macro-actions, 2016. URL https://arxiv.orrs /t/////rxxivv.or//b////6666
04615
[11] Matteo Hessel, Joseph Modayil. Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improve- ments in deep reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), 2018
[12] Jette Randlov. Learning macro-actions in reinforcement learning. Advances in Neural Informa- tion Processing Systems, 11, 1998.
11

---

[13] Amy McGovern and Richard S Sutton. Macro-actions in reinforcement learning: An empirical analysis. 1998.
[14] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference on Learning Representations (ICLR), 2018.
[15] Minjae Cho and Chuangchuang Sun. Hierarchical meia-reinforrement learning via automated macro-action discovery. arXiv preprint arXiv:2412. 1930, 2024.
[16] Aaron Hao Tan, Federico Pizarro Bejarano, Yuhan Zhu, Richard Ren, and Goldie Nejat. Deep reinforcement learning for decentralized multi-robot exploration with macro actions. IEEE Robotics and Automation Letters, 8(1):272-279, 2022.
[17] Yuchen Xiao. Macro-Action- -caed Multi-Agent/Rooo Deep Reinforcement Learning under Partial Observability. PhD thesis, Northeastern University, 2022.
[18] Sourav Panda, Aviral Srivastava, and Jonathan Dodge. Unlocking new strategies: Intrinsic exploration for evolving macro and micro actions, In intrinsically-Motiivtee and Open-Ended Learning @orkshop@ NewrIPS2024, 2024.
[19] Keng- Yu Chang, Hung-Y Yi Lee, and Chun-Yi Yi Chen. Macro actions: Learning to act efficiently through reusable temporally extended actions. arXiv preprint arXiv:1908.01478, 2019.
[20] Cameron Allen, Michael Katz, Tim Klinger, George Konidaris. Matthew Riemer, and Gerald Tesauro. Efficient black-box planning using macro-actions with focused effects. arXiv preprint arXiv:2004. 13242, 2020.
[21] Muhammad Abdul Hakim Newton, John Levine, Maria Fox, and Derek Long. Learning macro-actions for arbitrary planners and domains. In ICAPS, volume 2007, pages 256-263, 2007.
[22] Christopher Amato, George Konidaris, Leslie P Kaelbling, and Jonathan How. Modeling and planning with macro-actions in decentralized pomdps. Journal of Artificial Intelligence Research, 64:817-859, 2019.
[23] Yiyuan Lee, Panpan Cai, and David Hsu. Magic: Learning macro-actions for online pomdp planning, arXiv preprint arXiv:201 2020.
[24] Simon Hakenes and Tobias Glasmachers. Deep reinforcement learning based navigation with macro actions and topological maps. arXiv preprint urXiv:2504. 18300, 2025.
[25] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529533,, 2015.
[26] Mare Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforce- ment learning, 2017.
[27] Matteo Hessel, Joseph Modayil, H. V. Hasselt, T. Schaul, Georg Ostrovski, W. Dabney, Dan Hor- gan, B. Piot, Mohammad Gheshlaghi Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
[28] T. Schaul, John Quan, Ioannis Antonoglou, and D. Silver. Prioritized experience replay. CoRR, 1l.59221 2016.
[29] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[30] Ziyu Wang. Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003. PMLR, 2016.
[31] Meire Fortunato, Mohammad Gheshlaghi Azar, B. Piot, Jacob Menick, Ian Osband, A. Graves, Vlad Mnih, Rémi Munos, Demis Hassabis, O. Pietquin, Charles Blundell, and S. Legg. Noisy networks for exploration. ArXiv, abs/1706.10295, 2018.
12

---

[32] Zhongwen Xu, Hado P van Hasselt, and David Silver. Meta-gres gradient reinforcement learning. Advances neural information processing systems, 31, 2018.
[33] Tom Schaul, Dan Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International Conference Machine Learning, pages 1312-1320. PMLR, 2015.
[34] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253- 2013.
[35] Brockman. Openai gym. arXiv preprint arXiv:1606.01540 2016.
[36] Vitaly Kurin, Sebastian Nowozin, Katja Hofmann, Lucas Beyer, and Bastian Leibe. The atari grand challenge dataset. arXiv preprint arXiv: 10998, 2017.
[37] Maxime Chevalier-Boisver Lucas Willems, and Suman Pal. Gym-minigrid: Minimalistic gridworld environment for openai gym. https: wtt...h.....oo...eeeeeeeeeeeeeee cymm-inigrrd
2018.
13

---

A MASP Regularization and Implementation Details
A.1 A.1 Detailed Formulation of MASP
The Macro-Action Similarity Penalty (MASP) is a regularization term designed to enforce smoothness across the Q-values of similar actions (both primitive and macro-actions) in the augmented action space. To achieve this, we define similarity matrix Σ E represents the degree of similarity between actions a, and aj. In our implementation, E is constrained to be symmetric and non-negative, but do not require it to be positive-definit.
where each entry Eig 2 0
Given a batch of transitions {(st at rs 8141))==1- MASP adds the following regularization the TD loss:
MASP = 7) Σ
(0(s))+0)) - (s, ,0)1
where (;) E TR|A| is the vector of Q-values for all actions at state 8,, and oj is a tunable regularization coefficient.
Intuition: This penalty encourages the Q-values of similar actions (according to Σ) to close, effectively sharing credit among macro-actions that are functionally related. Unlike options or hierarchical RL, our penalty "soft" aallowigg some dispersion for exploration and differentiation.
A.2 A.2 Meta-learning Procedure for
The similarity matrix E is meta-learned jointly with the agent's main network parameters 0 via meta-gradients. This is achieved as follows:
1. Inner Update (Agent Step):
Sample trajectory from the replay buffer.
Perform a standard TD update with the MASP penalty, updating 0' using Σ fixed.
2. Outer Update (Meta Step):
Sample a new trajectory T'.
Evaluate the performance of the updated O' using meta-objective (the standard TD loss).
Compute the meta- eeta-gradieet of this meta-objective w.r.t. Σ (backpropagating through the inner update step).
Update Σ with a separate learning rate
This approach is similar to the meta-gradient method introduced by Xu et al. 32 In practice, we use automatic differentiation and checkpointing to efficiently compute de'
Algorithmic Details:
See Algorithm in the main text for the pseudocode.
Σ is parameterized as free matrix with symmetry enforced by averaging with its transpose after each update.
prevent divergence, we clip the entries Σ to the range [o, 1] after each update.
A.3 A.3 Practical Implementation Details
Embedding E: For large action spaces, representing Σ explicitly can be expensive. We instead flatten and use a learned projection:
eg Wemb vec(E)
where W.mb amb is trainable matrix, and ex: a low-dimensional embedding vector. This is concatenated with the state embedding and fed into the Q-network.
14

---

Additional Training Details:
Both E and Wemb updated via backpropagation from the main loss.
The meta-objective for Σ. updates does not include gradients through W Wmmb-
To prevent Σ from degenerating to the identity or to a rank-one matrix, we add small entropy penalty to row-normalized version during meta-learning.
Computational Cost:
The MASP regularization is vectorized using batched matrix multiplications.
The additional overhead is roughly 20% over standard DQN (wallclock time), dominated by meta-gradient computation and matrix operations.
B
Experimental Details and Hyperparameters
We summarize Atari preprocessing settings in Table 6 and the main algorithm hyperparameters in
Table
Hyperparameter
Value
Max frames per episode
108,000 (84, 84)
Observation down-sampling
Num. action repeats
4
Num. stacked frames
Terminal state loss of life true
Random noops range
30
Sticky actions
true
Frames max pooled
3 and 4
Grayscaled/RGB
Grayscaled
Action set
Full
Table 6: Atari pre-processing hyperparameters.
Reproducibility and Code Availability
To facilitate reproducibiliiyy all code used the experiments and MASP implementation available thub. http:///lssubmssssssssmseaeaeaee cewproo-rredttttmmeeeeee
at: https://gi thub.
B.1 Compute Resources and Training Time
Our experiments were conducted on a compute cluster equipped with 192GB RAM and a combination of NVIDIA GPUs: 1x RTX 5090, 2x RTX 4090, and 4x RTX 3090 cards. All Atari experiments typically required approximately one week to complete per full experimental run (including all seeds, ablations, and sweeps). Training for StreetFighter environments generally completed in approximately two days.
C
Additional Experimental Results
C.1
Atari Results
Full Atari performance for all methods and human-normalizz scores are reported in Table g
C.2 Streetfighter II Experimental Details
Streetfighter II experiments were conducted with domain-specific preprocessing and macro-action settings, including a reduced action set and macro-actions corresponding to common combos. All environment and training hyperparameters are detailed in Table
15

---

Hyperparameter
Value
Optimizer
Adam
Learning rate
6.25 10-5
Batch size
32
Discount factor 7
0.99
Replay buffer size
1 x 106
Target network update period
10,000 steps
Gradient clipping
10
Exploration (imitial final)
1.0 0.01
Exploration decay schedule
IM steps
Multi-step returns (n)
Noisy nets
true
Distributional atoms
51
Distributional min/max values Dueling network
-10 10
true
Prioritized replay a
0.5
Prioritized replay /
0.4 -> 1.0
Macro-action size (k)
32 (see ablations)
Macro-action length
3-8
MASP penalty weight 77
0.1, 0.3, 0.5, 0.7, 1 (swept)
Meta-learning rate 3
0,001, 0.005, 0.01 (swept)
Σ embedding dim (eg)
32
Table 7: Main hyperparameters used for Rainbow-DQN, macro-action augmentation, and MASP
regularization in all experiments.
A visualization of the learned similarity matrix E for macro-actions in Street Fighter II can be found Figure 3 this appendix, which illustrates the clustering related macro-actions discovered by the agent during training.
C.3 MiniGrid Experiments
Experimental Setup. We also tested MASP on MiniGrid [37] environments designed to require planning and structured exploration. We selected tasks such DoorKey, FourRooms, LockedRoom, and ObstructedMarz--ull!
Implementation Details. Macro-actions consist of fixed sequences derived from typical interaction patterns (e.g., aewwardforward--ume We compare RAINBOW DQN, RAINBOW with macro-actions, and RAINBOW with MASP. Each agent is trained for 500,000 steps. Hyperparameter tuning for was done per environment.
Results and Analysis. As shown in Table 10 MASP improves success rates over both baselines across all tasks. In simpler settings, the improvements are significant, and in more complex environments like ObstructeddMazz--ulll MASP is critical to achieving high success.
C.4 Transferability
Another key motivation for the Macro-Action Similarity Penalty (MASP) framework to improve transferability across tasks and environments. In principle, MASP regularization encourages the agent to learn more robust and generalizable representations by smoothing the Q-values among similar macro-actions, potentially facilitating adaptation to tasks where action semanties overlap.
Transfer Protocol: our experiments, we evaluated transferability by taking agents trained with MASP on subset of tasks and fine-tuning them (with or without additional MASP updates) on related environments. In practice, the learned similarity matrix Σ and macro-action set can be reused or adapted for downstream tasks, reducing the need for retraining from scratch.
16

---

+MASP
Game
Rainbow-DQN
+Macro-Actions
Score
HN Score
Alien
6,022.9:7188
3,714.8 = 246.0
7614.4 388.5
(121.4)
Amidar
202.8 + 23.4
185.8 8.0
272.5 # 7.0
(48.7)
Assault
14,491.7 759.0 759.0
11,368.2 1174.3
15665.7 1682.2
(540.4)
Asterix
280, # 23760.7
225,501.
+ 25686.6
356834.2 + 22386,9
(2102.2)
Asteroids
2 191.5
2,093.2 = 148.5
3544.9 + 362.3
(52.3)
Atlantis
814,684.0 42022.1
766,181.5
52739.2
933826.3 = 48727.3
(6721.3)
Bank Heist
826.0 + 60.3
708.5 20.8
1159,4 + 70.6
(219.2)
Battle Zone
52,040.0 4502.2
30,290.9
2185.3
67537.1 5791.1
(398.2)
Beam Rider
21,768.5 1356.5
16,221.3 1208.4
29374.2 3309.8
(232.5)
Berzerk
1,793.4 96.1
1,026.2 + 36.7
2635.8 122.0
(180.3)
Bowling
39.4 + 4.2
35.1 + 2.6
66.6 # 6.0
(34,2)
Boxing
54.9 + 2.7
50.6
100.0 10.3
(1000.0)
Breakout
379.5 25.1
252.9
884.4 74.0
(1011.2)
Centipede
7,160.9 + 211.7
4.48
7489.6 445.0
(72.4)
Chopper Command
10,916.0 # 1034.1
7.464.7
11592.2 1 876,8
(73.3)
Crazy Climber
143,962.0 + 13254.7
80,474.4
158672.8 + 7774,1
(207.5)
Defender
47,671.3 4264.2
30,844.1
58679.4 7205.4
(346.3)
Demon Attack
109,670.7 2672.4
72,716.0
117663.3 + 11120.4
(1014.3)
Double Dunk
-0.6 0.0
0.1
-0.2 = 0.0
(53.7)
Enduro
2,061,1 83.3
1,324.7 148.7
2266.6 + 58.3
(67.6)
Fishing Derby
22.6 2.8
22.6 0.6
36.9.6 1.0
(75.3)
Freeway
29.1 + 2.7
25.2
30.3 # 3.7
(101.0)
Frostbite
4,141.1 175.4
5566.7 317.3
(65.5)
Gopher
72,595.7 7, 1 6706.9
45,801.3
5387.9 5387.9
78992.5 +: 3703.2
(803.5)
Gravitar
567.5 20.5
367.6 42.66
645.5 # 31.8
(60.5)
Hero
50,496.8 3850.7
1228.2 42,269.0
62730.1
3469.7
(261.0)
Ice Hockey
0.1
0.0
-0.1 0.0
(53.6)
Kangaroo
10,8410 1247.0
8,271.9 1008.4
11225.7 1304.9
(132.6)
Krull
6,715.5 823.5
4,597.9 # 342.5
8251.1 1 858.9
(65.1)
Kung Fu Master
28,999.8 2080.4
18,165.9 = 1865.3
36837.2 + 1430.2
(151.2)
Montezuma's Revenge
154.0 16.9
5.3
400.0 18.1
(3.5)
Ms Pacman
2,570.2 = 204.6
1,471,5 185.3
2966.5 360.5
(67.6)
Name This Game
11,686.5 315.2
7,252.3 + 180.3
12745.8 + 685.4
(131.5)
Phoenix
103,061.6 + 10294.9
89,925.5 + 10494.5
116444.7 5868.9
(231.2)
Pitfall
-37.6 + 2.0
-49.8 4.8
-12.8 1.1
(3.3)
Pong
19.0 + 1.4
19.0
19.0 # 0.3
(104.2)
Private Eye
1,704.4 41.7
2244.3 # 150.9
(13.2)
Q Bert
18,397.6 637.1
12,276.2
854.1
22774.8
1373.3
(417.0)
Road Runner
54,261.0 + 2330.3
45,952.7 + 1687.5
62633.7 3796,5
(220.4)
Robotank
6.4
6.3
64.5 + 7.6
(75.1)
Seaquest
19,176.0 1428.9
13,917.1
23768.8 1476.9
(240.0)
Skiing
-11,685.8 787.3
2.6.22
-10114.6 575.5
(42.6)
Solaris
2,860.7 # 153.5
1.764.4+217..
4488.9 # 267.5
(75.3)
Space Invaders
12,629.0 1413.2
1.99..45322
16668.2 + 2050.5
(197.6)
Star Gunner
123,853.0 4413.1
12667.8 112,043.0
169778.8 + 11049.3
(533.5)
Surround
0.3
6.3
8.72 = 0.72
(69.8)
Tennis
0.2
0.4
12.6 t 0.6
(60.6)
Time Pilot
11,190.5 410.0
8,3314 931.3 931.3
15583.3 # 707.4
(209.7)
Tutankham
126.9 + 5.0
70.5 7.9
179.6 14.9
(48.2)
Venture
4.4
2..0
133.7 # 8.4
(20.5)
Video Pinball
506,817.2 + 16888.8
290,039.4
11557.8
577339.3
20750.9
(327.4)
Wizard of Wor
14,631.5 948.6
10,125.1 891.3
18866.5 + 1771.3
(259.5)
Yarr's Revenge
73,007.9 5494.8
55,584.7 + 3232.2
103544.9 + 3201.0
(134,8)
Zaxxon
19,658.0 2229.6
17,553.0 1147.8
26566.7 2208.2
(103.3)
Table 8: Comparison between RAINBOW DQN [II]. RAINBOW-DON with macro-actions (+Macro-
Actions), and RAINBOW DQN with aaccons similarity penalty (+MASP). Bold indicates maximal raw performance between RAINBOW DQN and MASP. Human-normalized (HN) scores are shown in parentheses. Cells highlighted in pink denote HN > 100.

---

Hyperparameter
Value
Observation shape
(128, 128, 3) 2
Frame skip Action repeat
Num. stacked frames
Reward clipping
[-1, 1]
Opponent
Random
AI Level 3
Max episode steps
18,000
Terminal on round loss
true
No-op start range
0-10
Sticky actions
false
Action
Reduced (15 discrete moves)
Combo macro-actions
true
Macro-action set size (k)
24
Macro-action length
2-6
MASP penalty weight 1)
0.3, 0.5 (swept)
Meta Meta-learning rate 8
0,001, 0.003 (swept)
Σ embedding dim (eg)
16
Optimizer
Adam
Learning rate
x 10
Batch size
32
Replay buffer size
x 10
Discount factor
0.99
Target network update period
5,000 steps
Exploration (initial/final)
1.0 / 0.05
Exploration decay schedule
200k steps
Dueling network Distributional RI.
true
true
Table 9: Hyperparameters for Streetfighter II experiments. Settings reflect domain-specific differences. including observation size, combo macro-actions, and action set.
Task
Rainbow DQN
Macro-Actions
MASP
Door Key
0.83
0.88
Four Rooms
0.69
0.87
Locked Room
0.58
0.77
0.97
Obstructed Maze
0.57
0.84
Table 10: Comparison between the success rate of RAINBOW DQN, RAINBOW DQN with macro-
actions and MASP for MiniGrid environments.
Observations: We found that MASP-trained agents generally adapted more quickly and achieved higher initial performance on transfer tasks compared to standard Rainbow DQN baselines. This sug- gests that MASP helps encode transferable structure in the Q-function and macro-action embeddings.
Limitations: The degree transfer benefit depends on the similarity between source and target task action spaces. Large discrepancies may require re- re-learning adaptation of Σ.
18

---

Hyperparameter
Value
Max frames per episode
2,000
Num. action repeats
Num. stacked frames
Terminal state loss of life
N/A
Random noops range
Sticky actions
false
Replay buffer size
x 10*
Batch size
64
Learning rate
1 x 10
Discount factor
0.99
Target network update period
1,000 steps
Exploration (initial final) Exploration decay schedule
0.2 0.01
50k steps
Multi-step returns (n)
Noisy nets
false false
Dueling network
Macro-action set size (k)
Macro-action length
2-4
MASP penalty weight 17
0.05, 0.1, 0.3 (swept)
Meta-lcarning rate /
0.001
Σ embedding dim (ex)
Table 11: MiniGrid-sjeeific hyperparameters. Only hyperparameters that differ from Atari are shown.
Figure 3: Sample E matrix from Street Fighter II experiments, illustrating the learned similarities between different catcroacctions. Distinct clusters with higher values indicate groups of macro-actions that are functionally related or often co-activated. contrast, the regions of the matrix with the lowest values and lacking visible structure correspond to primitive actions, which are entirely independent and dissimilar to each other.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
19

---

Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-tex. system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[NA]
Justification:
No theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
20

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The appendix contains all hyperparameters used, and we will release code that can be used to reproduce our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- spen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will release the code and the environments use (Atari and Streetfighter) are already available for research.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
21

---

Please see the NeurIPS code and data submission guidelines (https://nips.co)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/nips.cc// cttpppscccpplbcccuuuuceeee guides/ httppp/pw...cuuuiioooooooeoeaaaa
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Due to space limits, all of this details are in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
22

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide most of these details in the appendix due page limit.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttpp///berrro...ccp/poll////la
Answer: [Yes]
Justification: The work is mostly theoretical, and answers how RL can learn more effi- ciently with macro-actions. As the method applies to any RL system in can have indirect implications by making the systems more powerful. But it does not have any direct etichal implication in the form it in the paper, and the model sit had been applied to have no ethical implications (small scale and applied to games).
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: As the work does not have any direct ethical implications, such discussion does not seem necessary. The paper is theoretical and just improve exploration RL in general. While cutting edge RL system have societal impact, this work has not been run that scale or those kind of systems, but rather on toy RL setups like Atari.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Not necessary as the paper does not have any direct ethical implcations.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We only use environemtns that have been opensourced for research and are widely used in the community (Atari and StreetFighter)
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:Wi We use opensourced widely used RL environments by previous works.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human data used.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human data used.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
25

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLM have been used to develop the proposed method.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
26


Reviews:
[{"review_id": "3rwoOY1zfx", "reviewer": "Reviewer_tHfT", "summary": "The paper proposes Macro-Action Similarity Penalty (MASP), a regularization term that treats primitive actions and macro-actions as lying in a shared similarity space and encourages the Q-values of similar actions to stay close; this reduces the effective action‐space dimension, improving credit assignment and exploration efficiency. MASP meta-learns the similarity matrix jointly with the policy via meta-gradients, so the inductive bias adapts online instead of relying on handcrafted clusters. Built on a Rainbow-DQN, the method makes large gains on Atari and Gym-Retro StreetFighter II, remains robust as the number of macro-actions grows or when macros are noisy, and the learned similarity matrix transfers well across related games. These results show that exploiting action-space geometry can make macro-action augmentation practical without expensive hierarchical architectures.", "questions": "- Do you have any intuition about why transfer learning with $\\Sigma$ works well even though $\\Sigma$ is not a variable dependent on any states or history?\n\n- Since $\\Sigma$ is not dependent on any states or the history of macro-actions, even with the same source and target environments, if the order of the macro-actions is changed in the target environment, performance may degrade. Could you provide an experiment in which the order of macro-actions is modified?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strenghts:\n- The paper clearly raises the challenges of simply applying macro-actions, noting that adding macro-actions actually enlarges the exploration space. To address this issue, the paper introduces the Macro-Action Similarity Penalty (MASP), which reduces the exploration action space by penalizing the similarity of macro-actions, with clear motivation and a simple formalization using an ℓ₂ penalty term.\n\n- The similarity matrix $\\Sigma$ is optimized jointly with the policy via a meta-gradient procedure, allowing the inductive bias to evolve with the agent and environment dynamics rather than being fixed a-priori .\n\n- Integrating MASP into a Rainbow-DQN backbone yields significant improvements across a broad range of Atari games and complex Street Fighter II combos, while keeping the underlying network and training hyperparameters unchanged.\n\n- MASP maintains high performance as the macro-action set grows from 64 to 1024 actions (Table 2) and when up to 75 % of macros are replaced by random sequences (Table 4), where the naïve macro-action baseline collapses .\n                                                                                                                                                           \n\nWeaknesses:\n- The similarity matrix $\\Sigma$ grows quadratically with the size of the (primitive + macro) action set and must be projected at every update, so training time and memory rise steeply as the action space becomes very large or continuous.\n\n- MASP’s gains presume a reasonably “good’’ pool of macro-actions. If the extracted sequences are irrelevant or mis-aligned with the task, performance can deteriorate; automatic discovery of high-quality macros remains an unsolved prerequisite.\n\n- Although $\\Sigma$ sometimes transfers across games with similar dynamics, its interpretability is unclear and performance drops markedly when the source and target environments differ substantially.", "was_revised": true, "final_justification": "The authors addressed the reviewer's concerns about scalability and transfer sensitivity regarding the similarity matrix $\\Sigma$ . Thus, I maintain my original score: borderline accept (4). (The reason I did not give a score of 5 is that the performance of MASP does not significantly outperform the basic RAINBOW. Therefore, I still have some concerns about the effectiveness of macro actions.)", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback, and are pleased that you found the motivation, simplicity, and empirical performance of MASP compelling. We address your concerns and questions below.\n\n**(1) Cost of $\\Sigma$: Scalability and memory in large action spaces**\n\nWe agree that storing and projecting a full $\\Sigma \\in \\mathbb{R}^{|\\mathcal{A}| \\times |\\mathcal{A}|}$ can become expensive for extremely large action spaces. To mitigate this, we already use a low-dimensional learned embedding of $\\Sigma$ (see Appendix A.3), which is passed to the Q-network as a context vector. While the full matrix is still used during regularization, it is only applied over the batch of Q-values at each update (not the entire action space per state). This allows for practical efficiency even with 1024+ actions, as shown in Table 2. In future work, we plan to investigate structured or sparse approximations (e.g., low-rank, kernel-based) to make MASP scalable to continuous or combinatorial large action spaces.\n\n**(2) Dependency on the quality of macro-actions**\n\nIndeed, MASP does rely on the initial macro-action pool being at least partially meaningful. We share your view that automatic discovery of high-quality macro-actions remains an open problem. That said, Table 4 shows that MASP remains robust even when 75\\% of macro-actions are randomly corrupted — suggesting the meta-learned similarity structure is resilient to noise and can learn to attenuate the influence of irrelevant macros. Future work could combine MASP with recent macro-action discovery methods (e.g., trajectory clustering, unsupervised option discovery), where MASP might serve as a regularizer to prune or refine an evolving macro pool.\n\n**(3) Limited interpretability and transfer sensitivity of $\\Sigma$**\n\nWe appreciate this concern. In Table 6, we show that $\\Sigma$ generalizes best across environments with similar transition dynamics and interaction semantics (e.g., Breakout → Space Invaders, Montezuma → Private Eye). This supports the view that $\\Sigma$ captures behavioral regularities across macros, rather than task-specific overfitting.\n\nRegarding interpretability, we visualize $\\Sigma$ (Figure 3 in the Appendix) and observe clear block structures corresponding to groups of macros with similar behavioral outcomes. Still, we agree that $\\Sigma$ is not trivially interpretable in semantic terms, and improving this is an exciting direction — e.g., via structured priors, sparse constraints, or clustering-based analyses.\n\n\n**(4) Response to Q1: Why does $\\Sigma$ transfer across environments even though it is not state- or history-dependent?**\n\nOur intuition is that although $\\Sigma$ is not conditioned on state, it meta-learns how Q-values should co-vary across actions that yield similar \\textit{trajectory-level effects}. For example, in Breakout or Space Invaders, many macros correspond to repeated fire-move patterns. Thus, $\\Sigma$ learns to group such macros — and this grouping remains useful when transferred to related games where the macro-level structure of agent-environment interactions is similar.\n\nThis is analogous to how convolutional filters in vision models can transfer even though they are not image- or task-specific: they encode generic structure that is often reused.\n\n**(5) Response to Q2: Performance sensitivity to macro-action ordering**\n\nThank you for raising this point — we’d like to clarify that the ordering of macro-actions does not affect MASP’s behavior or transfer performance.\n\nIf the macro-actions are reordered, and the same permutation is applied to the rows and columns of $\\Sigma$, the MASP regularization remains mathematically identical. This is because MASP only enforces that Q-values for actions with high similarity (as specified in $\\Sigma$) are close — it does not rely on any fixed semantic meaning being attached to a particular index.\n\nIn practice, we may choose to order macro-actions consistently (e.g., grouped by length or composition) solely to aid human interpretability and visualization, but this ordering is not required for the algorithm and does not influence performance. The learned structure in $\\Sigma$ generalizes across environments due to behavioral similarity between macro-actions, not index alignment.\n\nApologies for the confusion caused by our presentation. We’ll revise the paper to make this clearer.\n\n**Summary**\n\nWe appreciate your thoughtful assessment of both the method’s strengths and limitations. We hope our clarifications address your concerns and provide additional insight into the robustness, transferability, and future potential of MASP. Please don’t hesitate to let us know if additional experiments or clarifications would be helpful — we would be happy to include them in a revision."}, {"review_id": "noO5hJN1ps", "reviewer": "Reviewer_997v", "summary": "The paper states that although access to a good set of macro-actions should, in principle, help improve exploration and credit assignment in RL, it leads to worse performance in practice. This is due to the wider range of choices available to the agent at each step, and treating macro-actions as entirely independent of one another.\n\nTo address this, the authors introduce a new regularization term, the Macro-Action Similarity Penalty (MASP), which enables more effective utilization of macro-actions. MASP encourages similar action values (Q) for similar macro-actions (or actions), where the notion of similarity is learned from meta-gradients of the RL objective.\n\nExperiments in Atari and StreetFighter environments show that MASP improves performance in comparison to two baselines – Rainbow DQN that only uses primitive actions and Rainbow DQN that uses macro-actions in the ‘standard’ way (without MASP).", "questions": "Included in “Strengths and Weaknesses”", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n**S1.** The paper identifies and addresses a new and interesting problem in using macro-actions for RL. \n\n**S2.** The paper is well-written and easy to follow.\n\n**S3.** The authors are quite measured in their claims and openly discuss the limitations of their proposed approach.\n\n**S4.** The experiments demonstrate that MASP enhances RL with macro-actions. However, the exact reasons for the improvement should be analysed further (see weaknesses).\n\n## Weaknesses\n\n\n\n**W1.** While the use of meta-gradients to learn the $\\Sigma$ (similarity) matrix was quite interesting, it is unclear whether this ties back to the semantic notion of similarity between actions/macro-actions. Since the motivation behind the ideas presented was that macro-actions share a lot of structure, it would be important to verify if this is indeed the case. It seems unclear whether this is the precise reason the idea works. \n\n**W2.** Based on the above point, additional experiments could be useful for a better understanding of the method.  Do the authors have results for lower numbers of macro-actions, like 4, 8, and 16? I would also be very curious to see results with no macro-actions (k=0), as the MASP regularizer can be applied to primitive actions.\n\n**W3.** The paper lacks empirical comparisons and discussions related to options-based exploration, a well-established line of research (e.g., [1]). Do the same problems identified in this paper also apply in that context? Similarly, it would be interesting to know how the presented approach compares to a simple option/macro-action type baseline, such as ez-greedy [2].\n\nOn a minor note, connections to work in temporal credit assignment could further improve the paper. Prior work discusses related ideas, albeit without an explicit emphasis on macro-actions. E.g., Hindsight credit assignment credits all actions that could have led to a future consequence [3, 4]. Chunking or compression can also be applied to abstract away subsequences of state-actions and directly connect to a future result that was reachable through many possible sequences of actions [5]. \n\nI am open to increasing the score if the weaknesses and questions are adequately addressed and clarified.\n\n### References\n\n[1] Klissarov, Martin, and Marlos C. Machado. \"Deep Laplacian-based Options for Temporally-Extended Exploration.\" ICML 2023\n\n[2] Dabney, Will, Georg Ostrovski, and André Barreto. \"Temporally-extended {\\epsilon}-greedy exploration.\" ICLR 2021\n\n[3] Harutyunyan, Anna, et al. \"Hindsight credit assignment.\" NeurIPS 2019\n\n[4] Meulemans, Alexander, et al. \"Would i have gotten that reward? long-term credit assignment by counterfactual contribution analysis.\" NeurIPS 2023\n\n[5] Ramesh, Aditya A., et al. \"Sequence compression speeds up credit assignment in reinforcement learning.\" ICML 2024.", "was_revised": true, "final_justification": "Response to rebuttal: I appreciate the time and effort spent in conducting the additional experiments. It's definitely positive to see the trend of improvement along the lower values of $k$ (and subsequent increase). The response addresses most of my concerns. I have updated my score.", "rebuttal": "We thank the reviewer for their clear and thoughtful feedback. We are glad that you found the paper well-written, measured in its claims, and that you see value in the identified problem and empirical contributions. Below we address each of your comments in turn.\n\n**(W1) Semantic alignment of the learned similarity matrix $\\Sigma$**\n\nThis is an excellent question. While MASP encourages shared learning signals among Q-values, it is true that the meta-learned $\\Sigma$ is not explicitly grounded in human-defined semantic similarity. That said, we do observe that $\\Sigma$ clusters together macro-actions that share overlapping primitives or lead to similar environmental transitions (see Appendix C, Figure 3). These clusters remain stable across seeds, suggesting that $\\Sigma$ does encode structured similarity.\n\nTo further clarify this, we will add a quantitative analysis: we compute pairwise edit distances between macro-actions and correlate these with the learned similarities in $\\Sigma$. Preliminary results show a statistically significant inverse correlation (Spearman’s $\\rho$ = -0.41), indicating that more similar macros (in terms of overlapping primitives) tend to receive higher similarity scores in $\\Sigma$.\n\n**(W2) Results with fewer or no macro-actions**\n\nWe appreciate this suggestion and agree it would help isolate the contribution of MASP. We have now run additional experiments in Atari (Breakout and Frostbite) with $k \\in {0, 4, 8, 16}$. Results show:\n\n* At $k=0$, MASP applied to primitive actions alone yields modest gains (+5–10\\%) over Rainbow-DQN. This suggests MASP can still be beneficial in structuring credit assignment even without macros.\n\n* At $k=4,8$, MASP gives strong improvements (+15–40\\%), while the naive macro-action baseline is sometimes unstable or worse than Rainbow.\n\n* Gains saturate or taper off beyond \\$k=32\\$–64, as shown in Table 2.\n\nWe will include these new results and plots in the appendix to provide a clearer picture of MASP’s benefits across macro-action set sizes, including in the extreme case of $k=0$.\n\n**(W3) Comparison to options-based exploration and ez-greedy**\n\nThank you for this important point. Our focus was on the effect of MASP in fixed macro-action settings, but we agree that connecting to options and comparing with other temporally-extended methods is valuable.\n\n* Regarding the applicability of our observations to options: yes, the same issue of increased branching factor exists when using a large pool of options, especially if they are treated as independent. MASP’s core idea — that value estimates for related extended actions should be regularized — can directly extend to options, and we see this as a promising direction for future work.\n* We have now implemented a simplified version of ez-greedy [2] and run it on two Atari games (Breakout and Space Invaders). MASP consistently outperforms ez-greedy (e.g., +20–30% on Breakout, +12–15% on Space Invaders), while using fewer assumptions and no decaying schedule. That said, we emphasize that these results should be taken with a grain of salt: due to the lack of open-source code and our limited rebuttal window, we reimplemented the method to the best of our understanding but could not perform extensive tuning. We will include these preliminary results in the updated appendix to offer a starting point for further comparison.\n* We will also add brief discussions of [1] and [3–5] in the related work section, highlighting the connection between MASP and credit propagation via structural biases.\n\n**Minor Point: Temporal Credit Assignment**\n\nWe thank you for these pointers. While MASP does not explicitly aim to solve the long-horizon credit assignment problem, we agree that it touches on related ideas. In particular, hindsight credit assignment [3] and counterfactual contributions [4] share the spirit of redistributing credit across related causes. MASP operates in a similar way but within action space rather than time. We will note this connection in the discussion section.\n\n**Summary**\n\nWe appreciate the reviewer’s useful suggestions and also openness to revising their score and hope the additional experiments and clarifications help demonstrate the robustness and generality of MASP. We agree that further understanding and benchmarking — particularly against option-learning baselines — will enrich the paper, and we are actively extending our work in this direction."}, {"review_id": "bvxXuZBYxl", "reviewer": "Reviewer_jyLn", "summary": "The paper proposes the Macro-Action Similarity Penalty (MASP). The idea is to use the similarity between macro actions to distribute the credit received by a chosen macro-action to all other macro-actions. The problem of using macro actions in Reinforcement Learning it that it can increase dramatically the state space and lead to much worse learning and exploration. So the appropriate use of macro actions has to be dealt cautiously. The principle here is to consider the macro actions jointly. The experiments are done with Atari games and Street Fighter II. The proposed MASP gives better results than Rainbow-DQN and Rainbow-DQN + Macro Actions.", "questions": "- The Atari games you test have a small action space. What are the Atari games that pose problem to MASP?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The paper has good experimental results and a novel idea.\n\nThe approach enable faster convergence and increased cumulative reward.\n\nHowever it might not always be the case in other domains that macro actions are so beneficial and they require a design overhead. The authors wrote a limitation section which is good. It would improve the paper to expose the properties of the problem that make their approach beneficial. \n\nStrengths\n\n- A novel way to distribute the credits to similar macro actions\n- Good results on Atari and Street Fighter II\n\nWeaknesses\n\n- Problems with large action spaces", "was_revised": true, "final_justification": "The rebuttal provided a better analysis of when the proposed algorithm works.", "rebuttal": "We thank the reviewer for their thoughtful feedback and for recognizing the novelty of our approach and the strength of our results on Atari and Street Fighter II. Below we clarify the properties that make MASP effective, and we address your question about where MASP might struggle.\n\n**(1) When does MASP help? What properties make it effective?**\n\nThe reviewer is absolutely right that macro-actions are not always helpful by default — they often introduce a much larger branching factor, which can hurt exploration. MASP is specifically designed to counteract that by learning to “tie together” similar actions and reduce the effective dimensionality of the expanded action space.\n\nFrom our experiments and ablations, MASP tends to work well in environments with the following properties:\n\n* High macro-action overlap: When many macro-actions share primitives or result in similar transitions, MASP clusters their Q-values and helps propagate credit.\n\n* Combinatorial macro-structure: MASP is especially useful when macro-actions are constructed from frequent patterns (e.g., Atari combos) or exhibit compositional regularity (e.g., fighting game moves).\n\n* Sparse reward tasks: Since MASP shares learning signals across related actions, it improves sample efficiency in sparse settings (e.g., Montezuma’s Revenge, Street Fighter II).\n\n* Large action spaces with redundancy: When the number of macro-actions is high (up to 1024 in Table 2), MASP still maintains good performance, likely due to its ability to filter and organize this space.\n\nWe will make this clearer in the paper by adding a short subsection in the discussion or limitations that articulates these properties.\n\n**(2) When does MASP perform worse?**\n\nTo your question: most Atari games start with small action spaces (typically 18 primitive actions), but our macro-augmented agents operate on much larger spaces (primitive + up to 1024 macro-actions). We stress that the difficulty here comes from the size and redundancy of the macro-action pool — not just the primitive action space.\n\nThat said, MASP can perform suboptimally when:\n\n* The macro-actions are unstructured or highly noisy (although Table 4 shows MASP is robust to up to 75\\% corruption).\n* The underlying task lacks meaningful macro-structure (e.g., environments where repeated action sequences are not helpful or where fine-grained control is critical at every timestep).\n* The size of $\\Sigma$ becomes too large to efficiently store or regularize, especially in continuous or highly combinatorial domains — which we flag in the limitations.\n\nWe will update the paper to make these boundaries more explicit, and note specific games where gains were minimal (e.g., Pong or Bowling).\n\n**(3) Clarification on action space size in Atari**\n\nWhile the primitive action set in Atari is small, our experiments used macro-augmented action spaces with up to 1024 macro-actions appended — turning the effective action space into a much harder setting. MASP was introduced precisely to handle this explosion in action choices. Our ablations (Table 2) show that naive use of macro-actions without MASP collapses performance as the number of macro-actions increases.\n\n**Summary**\n\nWe appreciate your question and agree that MASP's benefits depend on the structure and redundancy in the action space. We've added additional clarifications in our updated manuscript to explain both the strengths and limitations of MASP more concretely, and hope this provides the insight you were looking for."}, {"review_id": "n2SsbwO40X", "reviewer": "Reviewer_sFMe", "summary": "The paper proposes an algorithm to find similarities across actions such that actions with similar outcomes should have similar Q values. This reduces the “effective” action space since the typical assumption is that actions are independent of each other. It includes experiments on various Atari games and the proposed algorithm shows strong performance compared to baselines.", "questions": "There might be a misunderstanding on my end which needs clarification. The paper first makes it seem like the action space is being augmented, but after looking at the method and psuedcode, it does not appear to be the case. Instead it appears that the method is simply finding structure among existing actions in the existing native space (or an augmented action space with artificially added actions). This is an important clarification since the paper makes it seem like the action space is being augmented, which would be similar to the options framework [1], which attempts to add options to the original action space.\n\n[1] Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Sutton et al. 1998.", "limitations": "Yes, I appreciated the clarity in acknowledging the limitations of their method", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The proposed algorithm seems simple to implement and is intuitive, without introducing many hyperparameters.\n- The core intuition to exploit structure between similar actions in terms of similarity of outcomes aligns well intuitively.\n- The empirical results are on many Atari games, giving credibility to their algorithm.\n\nWeakness:\n- The comparison to normal Rainbow and naive Rainbow with macro actions make sense, but it seems strange there are no other baselines. In particular, I think the action representation learning literature is particularly relevant. I would encourage the authors to see the following works since they seem relevant (finding actions that are similar in terms of outcomes) [2, 3]. I think these works seem to get at similar questions of exploiting structure in the actions to learn similar representations.\n- The paper should discuss the above action representation learning framework and also work in the options literature [1, 4]. There seem to be strong connections to this literature, and it needs to be discussed.\n\n[1] Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Sutton et al. 1998.\n\n[2] Learning Action Representations for Reinforcement Learning. Chandak et al. 2019.\n\n[3] Learning Pseudometric-based Action Representations for Offline Reinforcement Learning. Gu et al. 2023. \n\n[4] Temporal Abstraction in Reinforcement Learning with the Successor Representation. Machado et al. 2023.", "was_revised": true, "final_justification": "The authors have adequately addressed my concerns.", "rebuttal": "We thank the reviewer for the thoughtful review and helpful pointers to related literature. We are pleased that you found our method intuitive and effective in experiments. Below we clarify a key misunderstanding and respond to your concerns.\n\n**(1) Clarification: Yes, the action space is explicitly augmented**\n\nThank you for flagging this — we’d like to clarify that the action space is indeed augmented. In all our experiments, we append a set of macro-actions (fixed sequences of primitive actions, typically 3–8 steps long) to the native action space. For example, with $k = 256$ macros and 18 primitive actions, the total action space has $274$ choices per decision point. MASP then regularizes the Q-values of this augmented space, which includes both primitive and macro-actions.\n\nWe have added a clearer description of this in the Methods section and highlighted it again in Section 3.2. You are absolutely right that this makes our setting analogous to the options framework, and we now make that connection explicit.\n\n**(2) Relation to options and temporal abstraction frameworks [1, 4]**\n\nOur setup is structurally similar to options-based approaches in that we augment the action space with temporally extended actions (i.e., macro-actions), but MASP is agnostic to their initiation or termination conditions. Unlike option-critic or SMDP-based frameworks, MASP does not assume or learn hierarchical policies. Instead, it addresses a different but complementary problem: how to improve credit assignment and exploration in a large, structured action space once it has been extended.\n\nWe will now include a short subsection explicitly relating our setup to [1, 4] and clarify that MASP can be viewed as a regularization layer on top of any temporal abstraction method that increases the action space — including learned options.\n\n**(3) Relation to action representation learning [2, 3]**\n\nWe agree these works are relevant and thank you for pointing them out.\n\n* ([2, 3]) focus on learning latent representations of primitive actions or policies to improve generalization or offline learning. MASP, in contrast, operates directly in the Q-value space by imposing a soft similarity constraint between actions, without explicitly learning action embeddings.\n* MASP could benefit from or extend [2, 3] by applying its regularization in the learned latent space. We see this as a promising direction and will note it in the discussion section.    \n\nWe will cite and discuss [2, 3] in our revised Related Work section and position MASP as a complementary technique that can apply regardless of the action parameterization.\n\n**(4) On novelty and contribution}**\n\nWhile MASP builds on existing ideas in temporal abstraction and structure-aware RL, we believe its novelty lies in:\n\n* Proposing a meta-learned similarity-based regularizer that scales to hundreds of actions (macro or primitive),\n* Showing that it enables robust credit assignment even with large, noisy, or redundant macro pools (Tables 2 and 4),\n* Demonstrating strong empirical gains without hierarchical policies or policy distillation.\n\nWe hope this clarifies how MASP offers a new perspective on action-space regularization that complements — rather than replaces — option discovery or action embedding.\n\n**Summary**\n\nWe thank you again for your constructive feedback and the helpful literature suggestions. We will revise the paper to clarify the action space setup, discuss connections to action representation learning and options, and position MASP more clearly as a regularization framework for structured action spaces. We hope this addresses your concerns and demonstrates that the work makes a distinct and meaningful contribution to the field."}]

Author rebuttal:

Meta-review: The paper proposes a method to create macro actions in reinforcement learning problem without the disadvantages.  The key is to add a special regularization that improves the credit assignment mechanism.  Extensive experiment on the Atari benchmark show that the new method (called MASP) can improve over the Rainbow-DQN baseline.  One might discuss how large the improvement over the baseline is, but the method is original and well executed so worthy of acceptance (as suggested by all reviewers).
Decision: Accept (poster)