Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization
Authors: Jian-Ting Guo
Yu-Cheng Chen
Ping-Chun Hsieh
Kuo-Hao Ho
Po-Wei Huang
Ti-Rong Wu
I-Chen Wu
Venue: NeurIPS 2025 poster ()
Keywords: Trajectory Optimization
Human-likeness
Reinforcement Learning
VQVAE
Primary area: reinforcement_learning

Abstract: Human-like agents have long been one of the goals in pursuing artificial intelligence.
Although reinforcement learning (RL) has achieved superhuman performance in many domains, relatively little attention has been focused on designing human-like RL agents.
As a result, many reward-driven RL agents often exhibit unnatural behaviors compared to humans, raising concerns for both interpretability and trustworthiness.
To achieve human-like behavior in RL, this paper first formulates human-likeness as trajectory optimization, where the objective is to find an action sequence that closely aligns with human behavior while also maximizing rewards, and adapts the classic receding-horizon control to human-like learning as a tractable and efficient implementation.
To achieve this, we introduce Macro Action Quantization (MAQ), a human-like RL framework that distills human demonstrations into macro actions via Vector-Quantized VAE.
Experiments on D4RL Adroit benchmarks show that MAQ significantly improves human-likeness, increasing trajectory similarity scores, and achieving the highest human-likeness rankings among all RL agents in the human evaluation study.
Our results also demonstrate that MAQ can be easily integrated into various off-the-shelf RL algorithms, opening a promising direction for learning human-like RL agents. 
Our code is available at https://rlg.iis.sinica.edu.tw/papers/MAQ.

Learning Human-Like RL Agents through Trajectory Optimization with Action Quantization
Jian-Ting Guo Yu-Cheng Chen' Ping-Chun Hsieh 'f, Kuo-Hao Ho! Po-Wei Huang', Ti-Rong Wu²t, I-Chen Wu
Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan "Institute of Information Science, Academia Sinica, Taiwan 3 *Research Center for Information Technology Innovation, Academia Sinica, Taiwan
tgjt.co cs13, fecheng.csl.. pinghsieh, lukewayne httpaaamneee..co5oenneeoee. cs05))nycu.eeeeeeeeooooooo edu. a311551048. cs 1@nycu. edu.tw, tirongwuuiis www.ii..siiiisiiieeeeee edu. tw, icwuêcs nyc.. edu.
Abstract
Human-like agents have long been one of the goals in pursuing artificial intelligence. Although reinforcement learning (RL. has achieved superhuman performance many domains, relatively little attention has been focused on designing human-like RI. agents. As result, many reward-driven RI. agents often exhibit unnatural behaviors compared humans, raising concerns for both interpretability and trustworthiness. To achieve human-like behavior in RL, this paper first formu- lates human- action sequence that closely aligns with human behavior while also maximizing rewards, and adapts the classic receding- horizon control to human-like learning as a tractable and efficient implementation. To achieve this, we introduce Macro Action Quantization (MAQ), human- like RL framework that distills human demonstra- tions into macro actions via benchmarks show that MAQ significantly improves human-likenese, likeness, increasing trajectory similarity scores, and achieving the highest among all RL agents in the human evaluation study. Our results also demonstrate that MAQ can be easily integrated into various off-the-shelf RL algorithms, open- ing promising direction for learning human-like RL agents. Our code available at https://rlg..in .is.lggii..... www.il/g.issssiiia..oi sinica.edu.tee www.../wwwwpappre/M.......
trajectory optimization, where the objective to find an
VAE. Experiments on D4RL Adroit
rankings
1 Introduction
Designing human-like agents has been an important goal on the path toward artificial intelligence. One of the most well-known benchmarks is the Turing Test, which evaluates whether agent perform intelligent behavior that is indistinguishable from human. To pass the Turing Test, agents designed not only to achieve the goal of the given task but also behave in human- like manner.
While the pursuit of human-like agents has been investigated in natural language processing (NLP), such as in large language models (LLMs) that aim to generate responses aligned with human preferences [1. it has not yet been widely explored in the field deep reinforcement learning (DRL). In recent years, DRL has made significant progress in various domains, such as gaming [2 3 4 or robotics control 6) However, researchers have noticed RL agents often exhibit unnatural behaviors,
"These authors contributed equally. CCorrepoondig authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

making they easily distinguishable from those of human players, e.g., smoothness 7 8 9 10]. navigation issues [T 13), and (unnatural) shaking and spinning actions [14. Although RL agents can accomplish these tasks, they fail to achieve the goal of designing human-like intelligence. This disparity raises an important and underexplored challenge human-like RI. research: How to design agents that not only succeed in tasks but also behave like humans?
To address this challenge, this paper proposes formulate as a trajectory optimization problem, where the objective is to produce trajectories that closely align with human behavior. Following the principle of receding-horizee control, we optimize over short action segments rather than entire trajectories to capture human behavior more effectively. To achieve this, we introduce Macro Action Quantization (MAQ), a human- -ikennsss aware RL framework for developing competitive agents with human-like behavior. MAQ first distills human behavior into macro actions sequences of actions from offline human demonstrations using a conditional Vector-Quantizae Variational Autoencoder (VQVAE) [5]. The VQVAE generates a discrete set of macro actions, stored in a learned codebook, where each entry represents possible human behavior segments for a given state. By leveraging these macro actions, MAQ transforms the action space from low-level primitive actions to high-level, human-like macro actions, effectively constraining the agent to operate within human-like behavioral space.
in reinforcement learning
We evaluate MAQ on one of the standard RL benchmarks. the Adroit tasks in D4RL [6], which has readily available human demonstrations. We apply MAQ to three RL algorithms, including IQL, SAC, and RLPD. Experimental results show that MAQ not only achieves higher success rates task completion but also substantially improves human-likeness, using Dynamic Time Warping (DTW) and Wasserstein distance metries, MAQ significantly increases the similarity scores across all tasks. In addition, we conduct human evaluation study similar to the Turing Test- in which participants are asked to distinguish between human demonstrations and agent behaviors. The results show that while participants can easily identify traditional RL agents as non-human, they struggle to distinguish MAQ agents from humans. In conclusion, our findings demonstrate that MAQ effectively captures human-like behavior, offering a promising direction for future research in human-like like RL studies.
When measuring trajectory similarity
2 Background
2.1 Human-like Reinforcement Learning
While most RL research focuses on designing reward-driven RL agents, there are few works exploring human-like RL in two directions. The first involves adopting behavioral constraints to enforce human-likeness. For example, Fujii et al. [17 proposed to penalize actions identified as non-human, while Ho et al. [14) introduced the adaptive behavior cost to discourage non-human-like behavior, such as spinning and shaking. Although effective, these methods rely on pre-defined behavior constraints or alle-based penalties, requiring substantial effort for handcrafted design. An alternative approach to avoid the handcrafted design of behavior costs is to directly learn from offtine human datasets. Notably, human demonstrations are available and widely used for RL training in various domains, such as Atari games [18], self-driving car [19] and robot arm manipulation [16]. These pre-collected demonstrations can be used to capture the characteristics of human behavior through imitation learning 200, such as behavior cloning or inverse RL [2T]. However, while imitation learning improves human likeness, its performance is often limited by the quality of human demonstrations and fails to compete with the non-human-liie RL benchmarks. In summary, how to effectively train human-like RI. agents remains open and underexplored challenge.
2.2 Semi-Markov Decision Process and Macro Action
Semi-Markov Decision Process (SMDP) extends MDP by incorporating macro actions, which are sequences of consecutive actions executed over multiple timesteps. Given a state 8t, macro MDP, SMDP is denoted by (S,M, R.P. A.P..) 7), where M is the set of macro actions and R is the cumulative reward
action with a length I, is defined as m, = (at, ac+1, (++ Compared
obtained over the macro actions: R(s,, m²) = > x ax) 22 Similarly, the objective
to learn a policy that selects maero actions to maximize the cumulative rewards: J(xo) = (e
(2 (2207 R(s,, m.)], where m, e M, and represents the timestep for selecting macro
2

---

actions. SMDP simplifies complex tasks by enabling agents to focus on long-term planning, as macro actions reduce decision-makinn frequency while capturing sequential dependencies.
2.3 VQVAE
Vector quantized variational autoencoder (VQVAE) [1]] an extension of the variational autoencoder (VAE) [23] is generative model that incorporates vector quantization to learn discrete latent embeddings. VQVAE consists of an encoder, a codebook of discrete latent embeddings, and decoder. The encoder transforms input data into a latent vector, which is then quantized by replacing with the nearest vectors from the codebook. The decoder aims to reconstruct the same input data using this discrete latent code. By utiliizigg discrete latent representations, VQVAE allows learning more structured representations.
Recent studies have explored the application of VQVAE in reinforcement learning. For example, Ozair et (24 and Antonoglou et al. [25] propose using conditional- conditional-VQVAE 26 as a state transition model in stochastic environments, using the codebook's ' latent vectors to represent possible chance events. Given state and a latent embedding (chance event) from the codebook, the decoder aims to generate the corresponding next states. In addition, Luo et al. [27] proposes utilizing conditional- VQVAE to discretize continuous action spaces in robotic control tasks. Specifically, each latent embedding in the codebook represents a possible action for given state. The decoder reconstructs the action from the corresponding latent embedding. This approach transforms continuous action spaces into discrete action spaces, improving learning efficiency in robotic control tasks.
3 Human-Like Trajectory Optimization
Trajectory optimization.
The canonical trajectory optimization problem is formulated based on an
MDP AS R. 7. T. Po) with continuous state space S and action space .4, transition function P: x A
A reward function R : S x A - R. discount factor E [0. 1), episode length T, and initial state distribution Po- Let = (80: 0o, , -T 1, (T--1, 8T) denote a trajectory generated under an action sequence 00:T-1 (ao: 01 .1) Our goal is to find action sequence that can maximize the total discounted return.
T-1
T = arg max
-o.r- -
Σ
R(Rs, az) ~ Po; ao:r
1=
where the expectation is taken over the randomness of the initial state, reward realizations, and state transitions.
Human-like receding-horizon control (HRC). As exact full-horizon optimization in equation ( can be intractable even for moderately large T,, we resort to the technique of receding- -orizon control, which is generic control method that involves repeatedly solving a optimization problem on short moving time horizon to choose action sequences. Specifically, the decision step f, receding-horizon control would find an action sequence length H (usually much smaller than T) by solving
H+H-1 -1
Σ
7 Fi**R(si ai) 815 a1s 11
arg max
Notably, under seceding-horizon control, such short-term replanning done every k steps (with 1 k < H), i.e., the first j actions in 02.4+H-1 --1 (i are executed and the environment returns Sttj and equation 2 is re-optimized. This receding- horizon loop reproduces the closed-loop robustness observed in (28 while bounding per-step computation.
To enforce human-lliknnss, we introduce the concept of human-like sequence constraint as follows: Let D ( be a set of semaan-generatd action sequences and define human manifold H = (r T D}. Accordingly, human- can be enforced by constraining the search space of action sequences to H. Hence, human-likk mreding-horizon control can be formally described as finding an action sequence as
+ Σ 7 RR ai)
0 -1 = arg max
BULHH H- IEH
Throughout this paper, we use A(X) denote set probability distributions over a set X.

---

Under HRC, executing a larger prefix j of the H-step action sequence generates longer, uninterrupted human-style motion, whereas j maximizes reactivity.
quantization for efficient HRC. Implementing equation 3 presents two ob- stacles: (1) Searching for all [H] demonstration segments at every step is prohibitively slow. (2) Although the dataset records each visited state, the starting state of a stored segment still rarely matches the current state St exactly, so naively re-using segments requires expensive alignment or local optimization. To alleviate these issues, we propose to perform segment-level planning: HRC first proposes complete H-step sequence (at, 0t+1..... a(+H-1), executes all H actions, observes the resulting state St-H- and then replans a new H H-step action sequence from that state. This commit-and and-replan loop amortizes the optimization overhead over fixed-length segments while fully re- re-using each trajectory rollout.
Action-segment
Moreover, searching for this segment in continuous action space is still computationally very costly, so we propose to adopt the macro action quantization strategy. Inspired by (27), we replace each segment of continuous actions with a compact sequence of discrete codebook indices, effectively reducing the search over the full space A to a lookup on a finite codebook. Since all demonstrations can be rolled out and scored offline, the HRC controller needs only a constant-time table lookup runtime, slashing planning latency while still retaining the multi-step look-ahead and human-style fidelity provided by H H-step segments.
4 Macro Action Quantization
To concretize the trajectory optimization introduced in Section 3 we propose a Huuman-likeness aware framework called Macro Action Quantization (MAQ). MAQ addresses the challenges of enforeing human-likeness in receding- horizon control by learning a quantized set of action segments macro actions from offline human demonstrations. These macro actions serve as H-step segments in the trajectory optimization process. The primary goal of MAQ is to achieve high performance while preserving the key characteristics of human-like behavior. We present the details of the MAQ framework as follows.
codebook
Encodee
Decoder
Macro Action Quantisation
ER
$
(L.....
Decoder
Figure 1: The overall training process of MAQ. (1) Human behavior distillation: Conditional-
VQVAE is trained on the state (s;) and the macro action (m, = (az, 4441,... H-+H to learn discrete codebook; the macro actions (m²) are extracted from human demonstrations via a sliding window over action trajectories. (2) Reinforcement learning with MAQ: An online policy (#a) acts in the learned discrete code space by selecting codebook indices, which the VQVAE decodes into macro action executed in the environment.
Human behavior distillation. To capture the characteristics of human behavior, specifically the action sequence patterns, we propose training Conditional-VQVaa to distill macro actions from human demonstrations, as illustrated on the left in Figure I Specifically, given state 8; and macro action my, where m, (az, at.tl ..., H -1) represents sequence of H consecutive actions executed by human at st, the encoder generates latent vector e. This vector is then compared with the codebook to identify the nearest latent vector eg. Finally, e along with the state 8; is passed the decoder to produce a reconstructed macro action my, which aims to replicate the original macro action m, During training, the loss function is updated as follows:
L Im - m + [|>g[e] - ex + Bl|e sg[ex]1
where sg denotes the stop-gradient operation and /3 is the coefficient of commitment loss. MAQ not only captures the human behavior of specific state and macro action pair (st, m)) but also

---

generalizes the macro actions across similar states, with the codebook serving as a collection of human-like behaviors mapped different states.
MAQ significantly reduces the macro action space. Since macro actions consist of sequences of actions, the number of possible macro actions grows exponentially with the sequence length. For example given a sequence length I and assuming each state has n potential primitive actions, there are n possible macro actions. Fortunately, as the goal of MAQ is to capture human-like behavior, vast number of these macro actions can be disregarded in practice since they do not appear in human demonstration. Therefore, we limit the codebook size to K, focusing only on macro actions that align with human behavior, effectively reducing the action space and distilling the key macro actions.
Reinforcement learning with MAQ. MAQ can be scamlessly integrated into any reinforcement learning algorithm. Conceptually, by quantizing macro actions, the action space is transformed from primitive actions (MDP) macro actions (SMDP), while all other aspects of the RL training remain unchanged. Since these macro actions are distilled from human demonstration, RL agents aim to achieve high performance while operating within the constraints of human-likk like behaviors.
Specifically, we train a policy Ty to select the index & corresponding to the codebook entry. The policy To takes given state 8 as input and produces distribution with K policy logits, each corresponding to one macro action embedding in the codebook. At each state 8t. To samples an index according to the policy distribution. Suppose the index k is selected, the corresponding latent vector ex is then retrieved from the codebook. With this latent vector and the state st, the decoder reconstructs a macro action me to interact with the environment, as illustrated on the right in Figure
!
To optimize Te. we follow equation to find macro action that maximizes the expected reward
m arg max m. EH E R(s, me) sti m
(5)
where m, = 0t=+H-1 is the macro action and m)) = H+ 7 RR=-R((s a.) is the cumu- lative reward over the segment. Through this process, MAQ allows agents to plan and act with human-like behavior while optimizing for long-term return.
5
Experiments
This section empirically evaluates the proposed MAQ framework. Our experiments are designed to answer the two main questions: Q1) Can MAQ accomplish tasks while optimizing trajectories that align with human demonstrations, and Q2) Can MAQ exhibit human-like behavior and convince human evaluators into believing it is human in a Turing Test evaluation?
5.1 Experiment Setup
Our experiments are conducted on four Adroit tasks from D4RL [6), including Door (opening a door), Hammer (hammering a nail). Pen (twirling a pen), and Relocate (lifting and moving a ball), We first train three off-the-sher RL algorithms, including IQL 29. SAC 30) and the state-of-the-ar offline algorithm RLPD (6. Next, we apply MAQ to each of these algorithms, resulting in three human-like RL agents- MAQ+IQL, MAQ+SAC, and MAQ+RLPD to evaluate their generality across different RL algorithms. When incorporating MAQ, each RL algorithm learns policy over discrete codebook, replacing the original primitive action space with macro actions distilled from human demonstrations. This flexibility makes MAQ easy to apply to wide range of RI algorithms.
Note that to support the discrete action space introduced by MAQ, we adopt Discrete SAC (DSAC) 31 when applying it to SAC. In addition, we train Behavioral Cloning (BC) 32 agent bascline for evaluating human-likeness.
Each task includes 25 human demonstration trajectories. which are split into training and testing datasets with 9:1 ratio. Agents that require human demonstrations, including BC, IQL, and all MAQ-based agents, are trained on the same training datasets. All agents are evaluated based on trajectory similarity on the testing dataset. To ensure fair comparison, all agents (including baseline and MAQ-based agents) are trained for total of 106 steps. For IQL and MAQ+IQL, which include both offiine and online training stages, we train 10 steps to each stage. For MAQ-based agents, we first train VQVAE on human demonstrations to obtain the macro action codebook. The VQVAE is
5

---

trained with 3 = 0.25 in Eq. 4. hidden size of 256, a batch size of 32, and 100 training episodes. The same trained VQVAE is shared across all MAQ-based agents to ensure consistency. Other detailed hyperparameters are provided in Appendix
5.2
Trajectory Similarity Evaluation
5.2.1 Evaluation Metrics
To evaluate how closely agent behaviors align with human demonstrations, we use two trajectory similarity metrics: Dynamic Time Warping (DTW) and the Wasserstein Distance (WD). DTW measures the alignment between trajectories, while WD captures distributional similarity. Both metrics are computed between agent and human trajectories from the same testing dataset.
Dynamic time warping. DTW is a technique used to measure the similarity between two se- quences, even if they differ in length timing. It finds the optimal alignment between two sequences by stretching or compressing segments to minimize the overall distance. Generally, lower distance indicates higher similarity. Specifically, let T {- 7 , = and T = [Tí², T2 ..... =A] denote the sets of trajectories played by the human and the agent, consist- ing of n and trajectories, respectively. The DTW distance between T and T is calculated by DTW = + Σ Σj= (T"( AT)).. Moreover, we compare two types of DTW distances: state distance DTW, and action distance DTW,, The DTW, and DTW, are calculated by applying DTW to the state and action sequences in DTW(7 AA), respectively. A lower DTW, indicates that the agent frequently encounters states in similar order to those in human demonstrations, while lower DTW, suggests that the agent performs actions in order similar human demonstrations.
Wasserstein distance. Compared to DTW, WD evaluates distributional similarity with the Wasser- stein distance between the agent and human demonstrations: W (Pagent: Phumann) The calculation is performed on the same normalized feature space used for DTW and uses the POT library's 's emd2 solver compute the Wasserstein distance. Similar DTW, we evaluate Wasserstein variants: state-based Wasserstein WD, and action- Wasserstein WD,, The WD, and WD, are calculated by applying W (Pagent; Phuman) to the state and action.
5.2.2 Performance of MAQ-based Agents
Table summarizes results for the four Adroit tasks, reporting state and action-based Dynamic Time Warping (DTW,, DTW,) and Wasserstein distance (WD., WD.) scores together with task success rates. For better comparability across tasks, we normalize the similarity scores for each task using 1 agent human
random score-human score where the agent score is the similarity between the agent and human trajectories, the human score the similarity between human trajectories, and the random score is the similarity between a random agent and human trajectories. After transformation, higher values indicate more human like behavior.
Table shows that incorporating MAQ significantly increases trajectory similarity across all tasks, according to both DTW and WD scores. For example, in the Door task, DTW, improves from 0.43 with IQL to 0.84 with MAQ+IQL, from -0.39 with SAC to 0.8 with MAQ+SAC, and from -0.06 with RLPD to 0.76 with MAQ+RLPD. Similarly, in the Hammer task, WD,, improves from -0.19 with SAC to 0.78 with MAQ+SAC, and from 0.2 with RIPDD to 0.85 with MAQ+RLPP. Most importantly, these improvements trajectory similarity are achieved without significantly sacrificing task success rate. In addition, although the BC agent learns directly from human demonstrations and achieves moderate similarity scores, performs the worst in terms of task success rate among all agents. This is because the BC agent learns to imitate state- acteonction pairs from the demonstrations, without optimizing for successful outcomes. Overall, the largest improvement observed between SAC and MAQ+SAC, where the average DTW, increases from -0.49 to 0.56. This is because SAC learns solely from environment interaction without using any human demonstrations, unlike IQL and RLPD. As result, its trajectories deviate more significantly from those of humans. Our findings also suggest that MAQ has significant potential when applied to more complex problems, such as tasks requiring intricate scenarios with numerous macro actions (c.g., continuous action spaces).

---

Table 1: The results for Adroit benchmark from D4RL. The baseline of each algorithm incorproate with MAQ generally improves all of the original algorithm similarity score in each control task.
Tasks
||
BC
IQI.
MAQ+IQI
SAC
MAQ+SAC
RL.P
MAQ-RI,R
DTW,(1)
0.18 = 0.09
0.13 + 0.06
0.84 =: 0.06
-0.39 4
0.80 0.80=00 0.08
-0.05
0.01
0.76 1: 0.04
DTW.
0,2
0.61
0.01
0.04
0.28
0.05
Door
(1,
0.32 = 0.05
0.18 4 0.05
0.75 2: 0.05
-0.28
0.02
0.71 0.08
-0.14 0.04
0.71 1: 0.03
WD.(")
0.03
0.02
0.03
Success(13
0.02 = 0.01
0.16 + 0.06
0.93 2: 0.04
0.43
1 0.23
0.56 0.50
0.96 1: 0,07
0.93 1: 0.05
(Tw.(t)
0016 0.07
-0.14
0.17
-1.10
0.35
-0.03 0.14
0.68 0.17
DTW,(")
0.47 = 0.02
0.45 + 0.20
0.92 0.06
-0.33
4 0.11
0.91 = 0.10
0.37 4: 0.08
0.94 1: 0.07
Hammer
WD,(t)
0.11
0.12
0.03
-0.44
0.09
0.12
-0.03
0.76 0.04
WD.(")
0,30 a: 0,03
0.30 +: 0.10
0.84 +: 0.02
-0.19 1 0.04
0.78 0,11
0.20 0.03
0.85 1: 0.03
Success
0.00
0.01
0.37
DTW.(1)
0.53 = 0.13
0.34 + 0.09
0.55 + 0.17
0.06 + 0.20
0.58 0.17
0.48 0.24
0.54 + 0.18
DTW..
0.58 0.05
0.51 9.05
0.58 0,09
-0.34 0.16
0.11
0.40 0.17
0.59 + 0.13
Pen
(D,
0.59 = 0.12
0.54 + 0.11
0.59 0.13
0.29 + 0.10
0.61 0.12
0.49 0.08
0.59 + 0.12
WD..(")
0.65 0.14
0.63
0.66 0,15
0.22 + 0.15
0.14
0.122
0.66 0.14
(1ccess(1)
0.40 = 0.03
0.40 + 0.08
0.42 0.07
0.32 + 0.09
0.41
0.01
0162 0.09
0.42 + 0.05
(fW,(f)
0.09 = 0.14
0.20 : 0.20
0.52 = 0.06
-0.55 + 0.20
0.25
0.19
0.03 0.13
0.27 1 0.14
(T),(")
0.47 + 0.15
0.51 + 0:11
0.82 0.01
-0.10 + 0.16
0.66 0.09
0.32 0.13
0.69 + 0.10
Relocate
(t,
0.27 = 0.15
0.36 0.06
0.47 = 0.07
-0.22 1 0.06
0.40 0.07
0.02 0.07
0.38 1: 0.09
(D,(")
0.45 + 0.12
0.50 + 0.04
0.65 0
-0.05
+ 0.03
0.61
0.03
0.20 +200+100 0.03
0.55 + 0.08
Success(t)
0.01 = 0.02
0.00 0.00
0.20 :: 0.10
0.00 1 0,00
0.14
0.07
0.14 1 0.03
0.17 1: 0.10
DTW,(1)
0.16 = 0.29
0.21 4: 0.25
0.63 1: 0.14
-0.49 1: 0.48
0.56
0.23
0.10 1: 0.25
0.56 1: 0.21
DTW,(")
0.49 0.07
0.52 0.06
0.82 0.17
-0.26
+ 0.11
0.76 0.17
0.34 : 0.05
0.78 :: 0.17
Average
(D)
0.32 = 0.20
0.38 1 0.18
0.64
2: 0.14
-0.17
1: 0.32
0.59 0.14
0.08 0.28
0.61 1: 0.17
(D.(")
0,45 0.15
0.48 : 0.14
0.74 0,10
-0.01
1 0.18
0.71
0.08
0.24 0.15
0,70 1: 0.13
Success(1)
0.11 2 0.19
0.14 4 0.19
0.39 0.40
0.19
1: 0.22
0.28 0.25
0.68 0.40
0.52 1: 0.32
5.2.3 Human-Like Trajectory Optimization with Different Macro Action Lengths
We examine how the macro action length H in the MAQ codebook affects the human-likeness similarity score. We evaluate the MAQ+RLPD across different sequence lengths (H to 9), using the same five metrics illustrated in Table [ averaged over the four Adroit tasks. Figure [|shows the results, where each cell is color-coded with darker shades indicating higher values. The color scale is normalized independently for each metric. The results show that H = 9 yields the highest similarity scores as well as the best task success rate. Notably, WD, remains relatively stable across all values of H. likely due to the nature of the Adroit action space, which consists solely of Shadow Hand joint positions. This suggests that even shorter sequences can appear human-like in action space without being effective at completing the task. As H increases, both trajectory similarity and success rate improve, indicating that longer macro actions not only enhance human-likeness similarity score but also contribute to more effective task execution, highlighting the benefits of temporally extended planning in human-like trajectory optimization. We have also provided additional analysis of MAQ in Appendix B
5.3 Human Evaluation Study
While the trajectory similarity scores demonstrate that MAQ-based agents align more closely with human behavior compared to other methods, we also conduct a human evaluation study with 19 human evaluators to verify whether their behavior is recognized as human-like by human evaluators. Specifically, each evaluator study is asked to complete two sets of questions, including Turing Test and a linannlikeeees ranking test. Detailed settings for human evaluation stduy are provided in The following subsections provide the results on each evaluation test.
Appendix [
5.3.1 Turing Test
In the Turing Test, we conduct several two-alternative forced-choice (2AFC) questions, where evaluators are shown two videos one from human demonstrations and the other from one of the seven trained agents listed Table and asked to choose which one was performed by humans. To ensure fairness, the evaluation consists of seven questions for each Adroit task, each corresponding to a different agent. Namely, each agent appears exactly once, and the order of appearance is randomized for each evaluator.
7

---

9
0.96:00211
0.78-00117
0.61-0.17
0.70
0.52:0.32
0.51:::24
0.77=0.11
0.69
0:42:0.32
0.54 19
0.77-0.18
0:58
0.69-00.13
0.47.00330
Sequence Lengts
0.57:0020
0.80::0:15
0.71:0012
0:40
5
0. 02:2:0 22
0.76 20
0:57=0015
0.69:0.13
0:37:0.38 38
0.72:0:19
0.65:0013
0.46:00.8 25
0.76:0017
0:52:0017
0.68
0.38:0.37
0.45-0222
0.73-0.15
0.53-0.14
0.67-0.10
0.30+0.26
0.46-0.17
0.73-0.17
0. 0.54-0.11
0.70-0.13
0.25+0.25
DTW,
DTW,
WD,
WD,
Success
Matrics
Figure 2: Trajectory similarity scores and success rates with different lengths in MAQ+RLPD.
BC
MAQ- (Can))
SAC
MAQ (AAC(CNn
MAQQ PrgDDw))
Win
Dioor Harnerr
Pei
Telocaaa
Ave
Environment
Figure 3: Turing Test. Error bars represent 96% confidence intervals.
Figure shows the results of the Turing Test. Bars indicate the win rate of each agent, i.e., the percentage of questions in which evaluators were fooled by the agent and believed its behavior was more human-like than the human demonstration. The results are consistent with the findings Subsection 5.2, where MAQ-based agents achieve higher win rates compared to RL algorithms without MAQ. This further corroborates that optimizing trajectories with longer macro action lengths strongly correlated with
Overall, when comparing the average win rate across four tasks, the win rate evaluated by the evaluators is: MAQ+RLPD (39%) > MAQ+SAC (34%) > MAQ+IQL (32%) > BC (24%) = RLPD (24%) > SAC (19%) > IQL (13%). Notably, MAQ+RLPD improves win rate of 1.5% higher than non-MAQ agents, demonstrating that our method significantly enhances the perceived human-likeness of agent behavior.
The figure also shows some interesting results. The Pen task exhibits the highest win rate for all agents, indicating that evaluators have the greatest difficulty distinguishing agent behavior from human demonstrations in this setting. This suggests that pen manipulating is relatively easy for agents to master, and that the visual differences between human and agent behavior are subtle, making deception more likely. In addition, the SAC agent shows nearly 0% win rates in the Door task. Despite achieving a 43% success rate, its behavior remains noticeably different from human demonstrations. This discrepancy clearly demonstrates that reward-driver RL agents not necessarily human-like
5.3.2 Human- LikennL Ranking Test
The questions in the human-likenes ranking test are similar to those in the Turing Test, but with a key difference, as described below. Since the goal is to compare human-likeness among agents, each question presents evaluators with two videos; however, both videos may be from trained agents.

---

Evaluators are asked to choose the video that appears more human-like, For fairesss, each agent pair appears only once per evaluator, and not all possible agent pairs are shown for evaluators to reduce fatigue. Each evaluator is asked to complete approximately 17 questions during this test. After completing the test, each evaluator is shown two videos one selected from the most human-like and the other from the least human-like agents or humans based on their responses and is invited to leave comments explaining their choices.
Agent A Win Rate
0.77:0.08
0.22+0.07
2:36:0.09 03
0.83 08
0.70=0 09
0.77 08
0.62=0.00
0.74:0.06
MAQ-RUFO(00m)
0.53-0.13
0.8 10
35440015
0.35:0.07
2.
0.72+0.10
0.38:00.0
0.71=0.20
Agent A
SAC(Oug)
0 76-6-0144
0.81-0115
3.5140117
0. 0:94-0111
0.61-0 16
0.47
0:30-0000
0.0--:21
SAC
0:11:0:10
6.10:0.15
0.0520.07
0.16:0.11
0.22:0:12
0.05=0.07
0.17:0.08
0.14:0.77
MAQQKOLOO KQLOun)
0.57:0016
6:60:00144 14
0.06:00.01 07
0.40:00117
0.67:0.11 18
0.36
0. 0.32::0.00
0.6:0025
0 0.50--017
0,2040 14
0:81:-0:15
0. 0:19:0::5
0.08 17
0.12:0030
0. 0.18::0:07 07
0:33-2 20
0.50-00
0:89:0:10
a1
RC
WAD-SAD
Agent B
Figure 4: Human-likenese ranking test.
Figure 4 summarizes the results across the four tasks. The heatmap shows the win rate, where each cell indicates how often evaluators judged agents on the y-axis to behave more human-like than agents on the x-axis when comparing their videos directly. Overall, the results are consistent with the Turning Test, where the overall ranking among all agents and humans as follows: Human (74%) > MAQ+RLPD (71%) > MAQ+SAC (61%) MAQ+IQL (61%) > BC (43%) > RLPD (38%) IQL (38%) > SAC (14%). Interestingly, the ranking order remains the same as in the Turing Test, except that the positions of IQL and SAC are reversed. This indicates that human evaluators are consistent in their judgments and can distinguish between human behavior and that of RL agents. Moreover, MAQ+RLPD achieves 71% win rate across all agent pairs, achieving performance comparable to the human demonstration's 's 74% win rate. This suggests that MAQ+RLPD can convincingly fool human evaluators into believing its behavior human-like.
5.3.3 Human Feedback on Behavior Analysis
We further investigate the feedback provided by human evaluators. Figure shows video clips in Door. The MAQ+RLPD agent demonstrates precise control by firmly holding, rotating the door handle, and pulling to open the door, closely mimicking human behavior. In contrast, while the RLPD agent achieves high success rate this task, it employs an unnatural method to open the door using its backhand to press the handle while simultaneousi, pulling the door without properly holding the handle. Although this approach successfully opens the door, the behavior appears less human-like. Therefore, several evaluators judged the MAQ+RLPD agent to be more human-like. As one evaluator noted, "Because shows how a human casually opens door the grip on the handle is stable, without forcefully grabbing it all the way". Conversely, many evaluators believe RLPD agent does not exhibit human-likk like behavior, remarking, "Instead of grabbing, it is glitching its hand through the door handle" and "Its wrist performed a weird gesture and opened the door rudely."
6 Discussions
In this paper, we discuss critical and yet underexplored challenge in reinforcement learning for designing human-like agents. We first formulate suman-likness as a trajectory optimization problem, and realize this formulation by proposing MAQ. human-likenee aware framework that distills human behaviors into macro actions. Our experiments show that MAQ not only completes tasks but also aligns with human behaviors by improving the trajectory similarity scores across four Adroit control tasks, and achieves the highest human-likeness rankings in human evaluation. These results demonstrate that MAQ agents are both effective and convincingly human-like.

---

(a) MAQ+RLPD
(b) RLPD
Figure 5: Agent behavior in Door.
Although MAQ exhibits more human-like behavior, one limitation is its reliance on human demonstra- tions distill macro actions. The quality of these demonstrations can affect the effectiveness of MAQ. However, since our primary goal is to pursue necessary and appropriate within the scope of this work. Moreover, our experiments demonstrate that MAQ can be easily integrated into various RL algorithms. We also show that MAQ has the potential to generalize to more complex domains, such as real-time strategy games, where agents often act at unnaturally high frequencies to maximize performance. Rather than relying manually designed constraints (e.g., delaying action execution [3],, MAQ naturally regulates decision frequency through learned human-like macro actions, allowing agents to behave more realistically. In conclusion, MAQ offers promising direction for learning human-like RL agents.
obtaining human demonstrations is both
Broader Impact
This work investigates human-like reinforcement learning by introducing a numan-likeness aware framework, called MAQ, that allows Al agents to behave in a more natural and human-like like manner. Human-like behavior can offer practical benefits in real-world applications, particularly in scenarios involving human-robot collaboration. For example, a robot that assists with human-like movements can improve safety and trust, reducing the risk of accidents. On the other hand, human-like agents may be misused deceptive or manipulative ways, such as cheating summary, the proposed method and human-likeness metrics provide standardized approach for evaluating and quantifying human-like behavior, benefiting the broader RL community.
competitive games. In
Acknowledgement
This research is partially supported by the National Science and Technology Council (NSTC) of the Republic of China (Taiwan) under Grant Numbers 1 -00----E00--00--M-- 113-2634-F-A49000 and 1 114-262--R-A49-002 We also thank the National
114-2221-FA -2-21-----49005, Center for High-performance Computing (NCHC) for providing computational and storage resources. The authors would also like to thank the anonymous reviewers for their valuable comments and the anonymous participants for the Human Evaluation Study.
References
[1] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, December 2022.
[2] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy
10

---

Lillicrap, and David Silver. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature, 588(7839):604-600 609, December 2020.
[3] Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom Paine, Caglar Gulcehre, Ziyu Wang. Tobias Pfaff, Yuhuai Wu, Roman Ring. Dani Yogatama. Dario Wünsch, Katrina McKin- ney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782)3350-354 November 2019.
[4] OpenAI, Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with Large Scale Deep Reinforcement Learning, December 2019.
[5] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pages 1-7, April 2025.
[6] Philip J. Ball. Laura Smith, Ily Kostrikov, and Sergey Levine. Efficient Online Reinforcement Learning with Offline Data. In Proceedings of the 40th International Conference on Machine Learning, pages 1577-1594. PMLR, July 2023.
[7] Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko. Regularizing Ac- tion Policies for Smooth Control with Reinforcement Learning. 2021 IEEE International Conference Robolics and Automation (ICRA), pages 1810-1816. IEEE Press, May 2021.
[8] I Lee, Hoang- Hoang-Giang Cao, Cong- Tinh Dao, Yu-Cheng Chen, and I-Chen Wu. Gradient-basse Regularization for Action Smoothness in Robotic Control with Reinforcement Learning. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 603-610, October 2024.
[9] Qianli Shen, Yan Li, Haoming Jiang, Zhaoran Wang, and Tuo Zhao. Deep Reinforcement Learning with Robust and Smooth Policy. In Proceedings of the 37th International Conference on Machine Learning, pages 8707-8718. PMLR, November 2020.
[10] William Koch. Flight Controller Synthesis Via Deep Reinforcement Learning, September 2019.
[11] Stephanie Milani, Arthur Juliani, Ida Momennejad, Raluca Georgescu, Jaroslaw Rzepecki, Alison Shaw, Gavin Costello, Fei Fang, Sam Devlin, and Katja Hofmann. Navigates Like Me: Understanding How People Evaluate Human-Like Al in Video Games. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, pages 1-18. Association for Computing Machinery, April 2023.
[12] Sam Devlin, Raluca Georgescu, Ida Momennejad, Jaroslaw Rzepecki, Evelyn Zuniga, Gavin Costello, Guy Leroy, Ali Shaw, and Katja Hofmann. Navigation Turing Test (NTT): Learning to Evaluate Human-Like Navigation. In Proceedings of the 38th International Conference on Machine Learning, pages 2654-2653. PMLR, July 2021.
[13] Evelyn Zuniga, Stephanie Milani, Guy Leroy, Jaroslaw Rzepecki, Raluca Georgeseu, Ida Momennejad, Dave Bignell, Mingfei Sun, Alison Shaw, Gavin Costello, Mikhail Jacob. Sam Devlin, and Katja Hofmann. How Humans Perceive Human-like Behavior in Video Game Navigation. In Extended Abstracts the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA 22, pages 1-11. Association for Computing Machinery, April 2022.
[14] Kuo-Hao Ho, Ping-Chun Hsieh, Chiu-Chou Lin, You-Ren Lou, Feng-Jian Wang, and I.-Chen Wu. Towards Human-Like RL: Taming Nontraturaeii Behavior in Deep RL. via Adaptive Behavioral Costs in 3D Games. In Proceedings of the 1 15th Asian Conference on Machine Learning, pages 33-453. PMLR, February 2024.

---

[15] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural Discrete Representation Learning. Advancees in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[16]
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning, February 2021.
[17] Nobuto Fujii, Yuichi Sato, Hironori Wakama, Koji Kazai, and Haruhiro Katayose. Evaluat- ing Human-like Behaviors Video-Game Agents Autonomously Acquired with Biological Constraints. In Proceedings of the 10th International Conference on Advances in Computer Entertainment Volume 8253, ACE 2013, pages 61-76. Springer-Vetie November 2013.
[18] Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Amold, John Agapiou, Joel Leibo, and Audrunas Gruslys. Deep Q-learning From Demonstrations, Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), April 2018.
[19] Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao, and Karol Zieba. End to End Learning for Self-Driving Cars, April 2016.
[20] Maryam Zare, Parham M. Kebria, Abbas Khosravi, and Saeid Nahavandi. A Survey of Imitation Learning: Algorithms, Recent Developments, and Challenges. IEEE Transactions on Cybernetics, 54(12):7173-7)
December 2024.
[21] Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
[22] Ishan P. Durugkar, Clemens Rosenbaum, Stefan Dembach, and Sridhar Mahadevan. Deep Reinforcement Learning With Macro-Actions, June 2016.
[23] Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2022. URL https
//arxiv.org setrziv.org/abs/13111 wawp../11...600 6114
[24] Sherjil Ozair, Yazhe Li, Ali Razavi, Ioannis Antonoglou, Aaron Van Den Oord, and Oriol Vinyals. Vector Quantized Models for Planning. In Proceedings of the 38th International Conference on Machine Learning, pages 8302-8313, PMLR, July 2021.
[25] Ioannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K. Hubert, and David Silver. Planning in Stochastic Environments with a Learned Model. In International Conference on Learning Representations, October 2021.
[26] Andros Tjandra, Berrak Sisman, Mingyang Zhang, Sakriani Sakti, Haizhou Li, and Satoshi Nakamura. VQVAE Unsupervised Unit Discovery and Multi-Scale Code2Spec Inverter for Zerospeech Challenge 2019. Proc. Interspeech 2019, pages 1118-1122 2019.
[27] Jianlan Luo, Perry Dong, Jeffrey Wu, Aviral Kumar, Xinyang Geng, and Sergey Levine. Action- Quantized Offline Reinforcement Learning for Robotic Skill Learning. In Proceedings of The 7th Conference on Robot Learning, pages 1348-1361. PMLR, December 2023.
[28] Nicklas A. Hansen, Hao Su, and Xiaolong Wang. Temporal Difference Learning for Model Predictive Control. In Proceedings of the 39th International Conference on Machine Learning, pages 8387-8406. PMLR, June 2022.
[29] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit O-Learning In International Conference on Learning Representations, October 2021.
[30] Tuomas Haamoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft Actor-Critie: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. In Proceedings of the 35th International Conference Machine Learning, pages 1861-1870. PMLR, July 2018. 88.
[31] Haibin Zhou, Tong Wei, Zichuan Lin, junyou li, Junliang Xing, Yuanchun Shi, Li Shen, Chao Yu, and Deheng Ye. Revisiting Discrete Soft Actor-Critic Critic, November 2024.
12

---

[32] Christopher G. Atkeson and Stefan Schaal. Robot Learning From Demonstration. In Proceedings of the Fourteenth International Conference on Machine Learning. ICML '97, pages 12-20. Morgan Kaufmann Publishers Inc., July 1997.
[33] Antonin Rafiin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann.
Reliable Reinforcement Learning Implementations. Journal of Machine Learning Research, 22(268):1-8, 2021.
[34] gwthomas. Iql-pytorch. attps://gitee http://wwhhu..coomthhooaeeee hamm/ggthoom//LLL-PYPOooo 2021.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contributions are summarized in both the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
We describe our assumption in Section 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all the information for reproducing the results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification: We have provided all necessary information for reproducing our main results. Once the paper accepted, we will release all codes and trained models.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips. //nips.cc/ppl wwwppbblccgguiicuceccccccoeooopp /eegguddsse/Ceebbbiiiinn htt.SSbbbiiiiooooiiiii.oooi for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental details are discussed in both Section 5 Jnd Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide the error bars for all tables and figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The experiment details are provided in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed broader impacts in the Discussion.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't 't release models or datasets.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, the dataset in this paper is properly credited and mentioned.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't release models datasets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We have provided detailed instructions for all participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper has no such issues.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: LLM is used only for editing to polishing the writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neur// /ttppp/p...///oooeeeere////////0 for what should or should not be described.
20

---

A Experimental settings
Experimental Setup We conducted our experiments on machine equipped with E5-2678 CPU and four NVIDIA GeForce GTX 1080 Ti GPUs. Training was performed on various games, each requiring a different number of GPU hours depending on the specific task. Our method begins by training a VQVAE with varying sequence lengths H and codebook sizes K. For each game, we used three different random seeds: I, 10, and 100. On average, VQVAE training required less than 5 minutes. Each MAQ-based method is built upon a pre-trained VOVAE. All MAQ variants share the same VQVAE model when trained with the same random seed. We consistently selected the model from the final training iteration as the VQVAE used in our online reinforcement learning (RI.) experiments. For online RL training, we evaluated the best performing checkpoint using two similarity metrics: Dynamic Time Warping (DTW) and Wasserstein distance. Among all variants, MAQ+IQL was the most computationally intensive, requiring GPU hours per training run, In contrast, MAQ+SAC and MAQ+RLPD required only GPU hour each. Regarding the vanilla RL, IQL, and RLPD, the training required approximately GPU hours per run, and SAC training required 1 GPU hour per run. This difference stems from the experimental setup: MAQ+IQL was trained in a single-environment setting, while MAQ+SAC and MAQ+RLPD were trained using 8 parallel environments. As result, the training time for MAQ+IQL was approximately 8 times longer, and for MAQ+SAC and MAQ+RLPD, updates were performed 8 times per iteration.
Hyperparameters All hyperparameters used in the online RL methods are detailed in Table and those for Conditional VOVAE are listed in Table For SAC, we used the default hyperparameters from Stable-Baseling repository released on Github 34 which most closely follows the original paper. For RLPD (), we
33 For IQL 29),, we adopted the PyTorch implementation by gwthomas's
used the official implementation released on GitHub.
In our MAQ-based methods, we made the following modifications. In MAQ+IQL, we built upon the IQL codebase and integrated the VQVAE decoder after the policy, enabling the policy to make decisions conditioned on the codebook size K. In MAQ+SAC, we extended the implementation of Discrete SAC [1 by incorporating the VQVAE decoder. For MAQ+RLPD we built on MAQ+SAC and additionally implemented the symmetric sampling scheme from RLPD, using a symmetric ratio of 0.5 (equivalent to the offline ratio shown in Table
Dataset Segmentation For data partitioning, we used the human dataset provided by D4RL. Each task consists of 25 successful human demonstrations, which we split into training and testing sets in a 9:1 ratio, resulting in 22 trajectories for training and for testing. We used the testing dataset to evaluate similarity between agents. The normalization procedure mentioned in the paper involves generating a random agent and using its performance on the testing dataset to normalize the similarity scores. Table presents all the detailed results, including the raw data, training data, and testing data.
Training Curves for All Environments Figure 6 presents the training curves of the experiments described in Subsection 5.2 The results are based on the normalized rewards across different environments. For Offline-to-nnie Online RL (O2ORL) methods such as IQL and MAQ+IQL, training curves are plotted in environment steps and the x-axis is scaled by 0.5, so that 2M steps are compressed to the same span as 1M steps in other settings. Shaded regions indicate the standard deviation across three different random seeds.
Environment Reward and Success Rate In D4RL, the success rate indicates whether the agent
completes the task within the episode limit (e.g., opening the Door within 200 steps). In contrast, the normalized reward is heuristic score defined by the environment that reflects how close the agent is to completing the task. For example, in the Door task, the reward increases as the door approaches a fully open state, but the episode is marked as successful only when the door is completely opened.
As shown in Figure 6 and summarized in Table 4 RLPD and MAQ+RLPD achieve similar success rates. However, their normalized rewards differ substantially: RLPD tends to maximize reward by opening the door quickly, whereas MAQ+RLPD opens the door more slowly and naturally while maintaining the same success rate.
Computational Cost Analysis of MAQ Since there are 25 trajectories per task, the VQVAE can be trained in a very short time. Specifically, training VQVAE takes only 3 to 4 minutes, whereas RL
21

---

training requires several hours. While MAQ-based RL introduces some additional computational overhead, 3 a comparison of inference time for each RL algorithm, with and without MAQ.
It is also worth noting that MAQ offer better inference efficiency, it executes entire maero action (e.g., 9 consecutive actions in the paper) with single forward pass, whereas vanilla RL must do forward pass at every timestep. In this case, MAQ-based RL can even achieve lower overall inference time than vanilla RL, especially when longer action sequences are used.
Table 2: Hyperparameters for training Adroit tasks.
Parameter
BC
IQL.
MAQ+IQL
SAC
MAQ+SAC
RLPD
MAQ+RLPD
Batch size
256
256
128
256
Leurning rute
3e-4
3e-4
3e-4
3e-4
3e-4
Actor learning rate
3e-4
3e-4
Critic learning
3o-4
le-3
Temperature learnnng rate
3e-4
3e.4
Optimizer
Adam
Adum IM
Adam
Adam
Adam
Adam
Adam
Offline training steps
IM
1M
Online truining steps
IM
1M
IM
1M
1M
1M
Discount factor 7
0,99
0.99
Warm-up steps
le2
8c3
le4
8c3
Update epoch
Value coeff
Entropy coeff
auto
auto
auto
auto
Offline ratio
0.5
Temperature alpha
0.2
1.0
Replay buffer size
2M
2M
IM
IM
1M
1M
Target network update rate
0.005
0.005
0.005
0.005
0.005
0.005
Advantage coeff A
Asymmetric loss cooff
0.7
Inverse temperature 8
3.0
Std of Gaussian exploration noise
0.03
0.03
Range clip noise
0.5
MAQ Codebook size K
16
16
MAQ Macro action length 1/
9
Table 3: Hyperparameters of training MAQ's Conditional-VQVAl
Parameter
Adroit
Latent size
256
Learning rate
3e-4
Codebook size K
[8, 16, 32]
Batch size
32
Commitment loss coeff ,3
0.25
Macro length H Optimizer
[I ... 9]
Adam
Table 4: Detailed numerical results corresponding to the main table, including raw scores, training and testing data, and normalization baselines used for similarity evaluation.
MAQHSAC
ILFO
MAQHILLE
Danant
Randen
N600 190544
136844 & 3ATAS
277.M 6.10
129 SAB 1 21.00
374 13.m
121 No15
375
41.00
300 83 12.33
1066 36972
V0. 30
5007 0453
2.09 JAT
3446
4.31
3381 6.07
5006 0115
3134 6.6
1364
0110
2.77 1345
5720 0.0
100 41. 91
237 251077
541 02 73-64
ATLLNG 111.6
115 1130
12 010
10 991
110 10 21
1001
0110
0.90
01-00
1100- 6.3
114 0714
75179
38-45
COLST9 30.00
34.0
GALN6 1..00
6.50
6550
6715
6335
130
1500
FALTAS
30
Ex17
138
8.80
7581 6.0
0115
2015 0019
E300 24090
630 699
030
610 686
0175 69
1000-00E
1000 130
0/80 006
22

---

Table Comparison of computational costs with and without MAQ
Vanilla RI (ms)
MAQ (ms)
Ratio (MAQ/Vanilla RL)
SAC/MAQ+SAC IQL/MAQ+IQL
1.35
1.12
RLPDIMAQ=RLPD
0.28
0.81
2.93
Door
(b) Hammer
(c) Pen
(d) Relocate
Figure Training curves for Adroit tasks.
23

---

B Further Analysis
In this section, we analyze MAQ from several aspects: (i) the effect of codebook size in Conditional- VQVAE (Appendix [ (ii) robustness to suboptimal demonstrations, including randomly shuffled and sorted datasets (Appendix B (iii) the impact of extending the environment horizon on success rates in the Hammer task (Appendix B.3
B.1 Impact of Codebook Size on Similarity Score in MAQ
previously observed that longer sequence lengths in VQVAE tend to lead to higher similarity scores when compared against human demonstrations. In this subsection, further investigate whether varying the codebook size K exhibits similar trend. We detail the models used in this experiment and analyze how different codebook sizes influence the similarity scores across MAQ variants, including MAQ+IQL, MAQ+SAC, and MAQ+RLPD. It is important to note that the similarity score is metric we define to approximate the behavioral similarity between agents and human demonstrations. While it provides a quantitative means comparison, it does not capture the full semantics or intent of human-like behavior. As also mentioned in Section [52.3] in D4RL, control tasks, the action space only includes the positions of the Shadow Hand, while the state space contains additional information about the target object. Accordingly, we evaluate how different codebook sizes affect the similarity score for different MAQ agents. Due to the computational constraints, we limited our experiments to VQVAEs trained with codebook sizes of 8, 16, and 32.
As discussed in Section 5.2.3
As shown in Figures action length increases, the similarity score also improves, regardless of the codebook size. Moreover, the similarity scores remain within narrow variance range, indicating stable performance across different configurations.
and we observe consistent trend across all MAQ variants: as the macro
0.
6.6066177
0.226.17
042
0.0000:0
3.00a1166
02708.11
101:011
1000
96:0
075
6:5680:3
36
6.6266117
5.7 16
02440.15
038
0.7881117
5
0:5466:35
30
6.116..4
3.77
0.6k8.16
3.776
0:5T20:00
56.0
0.5220.22
5.7903 16
0.888.11
$066.36
02366.20
7
(a) DTW,
(b) DTW,
0:5200:12
34
0:128.11
1.74:2.00
0.4e5.11
3.7263.07
6.6266.55
3.7
67208:1:
97550
3.755511
6:
68846.11
6.701.06
a72-6.11
373-116
03
11.50 25
0.558.14
1/223110
8.265.11
18961116
6:0000:55
3.
62208.12
3
WD,
(d) WD.
Figure 7: Similarity heatmaps of MAQ+IQL under different codebook sizes, measured by DTW and WD on state and action.
24

---

5:752117
3:720034
6:3886:22
$6.00
07556.18
312
0.5086:1
3.75
ATE
Sig
0:4246:44
4:4000 23
6.3
3.71.1 16
366
0.3000.33
2.002:11
3,70a3
063.6.12
3.734.M
6. 6.8066.1
3.370022
6:3646.15
3.7003 15
0.8866114
(a) DTW,
(b) DTW
2.332.066
055.0.08
34520.20
6555
5.1b1111
2.922.34
25220.23
cass
2.6722000
0.288111
36723.06
012
153:2.07
35329034
37
3.1.b.10 12
64740:33
1/7
0.
38723
344
2.
6412022
13660:5
5.0011 11
0ste5.14
0.09
2
E4020.32
34529.2
<CA
5.70-3
0.58-0.11
5.00AM
022
26
20
34
(c) WD,
(d) WD.
Figure 8: Similarity heatmaps MAQ+SAC under different codebook sizes, measured by DTW and WD on state and action sequences.
21
0.3525.25
1.7ts1111
0.7888117
3896313
0.558::2
24
0.226.20
07768:13
0:5220:25 25
57500
0.5226.00
3.773323
04fe5.11
3.782017
6.54.0:3
20
000
1140
0.405:.00
99:00
0.4768.22
0.Mos.17
8.723320
1444
(a) DTW,
(b) DTW.
6:6366:16
6
026o0.11
3.73-
0:3880:3
83650 29
5.703311
03328.14
2.110.00
163:300
0.59.0:17
90.00
6.568834
3.70a2
022a8.12
0:308:25
85720 25
03580.15
2.00 14
03908.11
3.663.06
12.00
6.226.166
0.0460.13
03328:15
33350 24
03888.14
3.60:3112
05725.12
aze
WD,
(d) WD
Figure 9: Similarity heatmaps of MAQ+RLPD under different codebook sizes, measured by DTW and WD on state and action.
25

---

In contrast, Figure highhighht how the effect of varying codebook sizes interacts differently with each underlying RL algorithm. For MAQ+IQL, which benefits from offline pretraining, success rates remain relatively consistent across different codebook sizes. MAQ+SAC, on the other hand, struggles under sparse reward conditions and fails to achieve success rate above 30% regardless of codebook size or macro action length. Interestingly, MAQ+RLPP, which incorporates symmetric sampling, achieves significantly higher success rate of up to 52%, demonstrating its advantage in such challenging environments.
0404
02511
(a) MAQ+IQL
(b) MAQ+SAC
(c) MAQ+RLPD
Figure 10: Similarity heatmaps of MAQ agents under different codebook sizes, measured by success rates.
B.2 Impact of Using Suboptimal Demonstration
In this subsection, we evaluate MAQ under different demonstration qualities, including suboptimal ones. We conduct an ablation study using subsets of trajectories with the lowest task rewards as well as random subsets. The results are shown in T
In this experiment, using the Pen task as an example, the success rate remains stable across demon- strations of varying quality. Moreover, MAQ trained small subsets of demonstrations (e.g., 25% of the dataset under random sampling or the bottom 25% in terms of trajectory- reward) can still achieve higher similarity scores (DTW and WD values in the table are normalized; lower is better). Even when using different random subsets, which may include higher-return trajectories, MAQ maintains competitive success rate.
26

---

Table 6: Ablation of MAQ trained on suboptimal demonstrations. Similarity scores remain stable across demonstration ratios in random and lowest, and MAQ+RLPD achieves the best success rate and similarity.
75%
75%
50%
50%
25% random
Tasks
MAQ+RLPD
random
lowest
random
lowest
lowest
(4),(4)
302.19
261.29
239.77
277.42
298.20
350.23
414.27
DTW,
275.20
257.20
242.36
263.66
294.37
335,20
408 408.44
Door
(D,(L)
4.59
4.38
4.70
4.42
4.77
5.19
5.85
WD&(U)
3.73
3.37
3.85
3.27
3.77
4,10
4.49
S1ccess(1)
0.93
0.98
0.79
0.85
0.55
0.48
0.03
(L),(L)
578.82
593.07
736.09
605.79
643.57
623.78
706.92
(U)
528.52
519.13
641.82
515.70
550.62
545.67
519.61
Hammer
(D.(L)
5.20
5.37
5.54
5.15
5.26
5,48
6.34
(D.(1)
3.32
3.37
3.53
3.21
3,40
3.40
3.58
Success(1)
0.56
0.33
0.31
0.30
0.28
0.23
0.51
(T)
740.45
679.96
590.02
669.14
649.56
686.26
686.74
(4)
664.75
629.19
560.92
624.25
607.43
654.41
652.16
(D.(L)
8.55
8.02
9.70
7.79
8.47
8.24
8.61
(J,
6.05
5.95
8,42
5,53
6,01
5.85
6.24
Success(1)
0.42
0.33
0.32
0.33
0.23
0.23
0.20
(L)
564.73
582.06
596.79
641.00
625.72
652.59
637.79
(J).())
691.59
739.38
741.85
857.08
772.72
862.77
853.56
Relocate
WD.(L)
8.05
7.84
8.08
8.34
8.83
8.87
8.57
(D)(1)
6.13
6.11
5.89
6.54
6,47
6.60
6.32
Success(t)
0.17
0.04
0.02
0.00
0.01
B.3 Impact of Different Environment Horizon
In Appendix A we discussed the distinction between environment reward and success rate, well training curves. In the Hammer task,
as the score gap between RLPD and MAQ+RLPD success rates of RLPD and MAQ+RLPD still differ markedly. We conducted deeper analysis of this task. We found that in the 25 human demonstration trajectories, humans take an average of 451.4 steps to successfully complete the task. However, in the D4RL environment, the agent restricted to a maximum of 200 steps per trajectory, after which the episode terminated and directly considered failure.
Since MAQ learns from human demonstrations, tends to favor smooth but slow movements, while non-human-like RL agents tend to complete the task more quickly but less naturally. To validate this hypothesis, we increased the maximum episode length from 200 450 steps. As shown in Table 7 MAQ+RLPD's success rate improves significantly from 0.56 to 0.72 while maintaining human- likenes scores.
Table 7: Ablation of MAQ+RLPD on Hammer under different training horizons h.
Tasks
RLPD 200)
MAQ+RLPD 200)
MAQ+RLPD 450)
DTW,(U)
845.88
578.82
648.32
DTW.(())
1178.74
528.52
637.25
Hammer
WD.
9.57
5.20
6.17
WD.(())
7.25
3.32
4.15
Success(t)
0.56
0.72
27

---

C Details of Human-likeness Survey
C.1 Survey Setup and Evaluation Protocol
This subsection describes the interface shown to the evaluators. Each questionnaire contains four "tasks," where each task is split into two stages, and each stage consists of series of two-alternative forced-choice (2AFC) trials.
Stage 1: (luman-Detectiin Phase (Figure Ila). The assessors are told that least one of the two clips was produced by a human and must identify which one. They also mark their confidence on a five-point scale, but our main analysis considers only the binary choice and ignores the confidence scores.
Stage 2: Ranking Phase (Figure l1b). Here, we test whether the behavior generated by MAQ- based agents appears more human-like like than that of baseline agents or even the human demonstrations. Model clips (optionally mixed with human reference clips) are presented in 2AFC pairs that are scheduled with a shuffled single-elimination and round-robin mini-tourrament. Up to eight clips yield no more than 17 head-to-head comparisons; each win earns one point. Clips are ranked by wins,
win-rate W; == appearances, with mean reported confidence used only to break ties, producing an ordering from the least to the most human-like one.
Post-ranking feedback. After the Ranking Phase, each evaluator is shown the clips judged the most and the least human-like and may optionally explain those judgments in free text (Figure Icc.. We analyze these comments in Section C.2
C.2 Additional Results of Human Feedback on Behavior Analysis
In this subsection, we discuss the additional behavioral results and the qualitative feedback received for each game. Figure 12] displays the distribution of the Top-1 and Top-2 rankings obtained in the Ranking Phase. Across all games, human demonstrations achieve the highest Top-1 I probability, confirming that evaluators can reliably identify genuine human behavior. Among the learned agents, MAQ+IQL attains the second-highes Top-1 I rate, followed by MAQ+RLPD, showing that our methods frequently convince evaluators that their behavior is indeed human-like.
In the Hammer task, five evaluators placed the human demonstration at Rank The next most human-like agents were the MAQ Srtants-MAQ+RLPP and MAQ+IQL, each with four first-place votes, followed by MAQ+SAC with one first-place vote.
What evaluators liked about MAQ policies. Comments converge three strengths: deliberate grip preparation, a realistic multi-strike rhythm, and a smooth tolloh-troough.
"Is able to hammer the nail."
"First takes hold of the hammer; because you need to aim, the first hit is lighter and the second harder."
"Humans can aim accurately when hammering a nail and probably won't II drive the nail completely in at once."
"It performs smoothly and hits the nail several times."
"There is a back-and-fortte hammer-swinging motion."
"The feeling of driving it in on the last strike is very human-like."
Why baseline agents were judged less human-like. remarks:
Non-MAQ policies drew noticeably harsher
" *Can't 't even lift the hammer."
"It threw the hammer."
"It fails the task."
"It cannot even lift the hammer."
"Hits too far from the nail and releases the hammer in unnatural manner."
28

---

MAQ does more than replicating the gross kinematics of hammering: several evaluators remarked that the policy "re-aims and strikes again" after an initial impact, behavior they naturally expect from humans. Figure 13 contrasts MAQ+RLPD with the plain RLPD agent. MAQ+RI RPD delivers series of well-timed blows that gradually seat the nail, whereas RLPD drives the nail in a single, overly forceful hit. Although MAQ+RLPD does not chase the maximum task score, it acts purposefully the way a person would, delivering several well-timed blows, while vanilla RLPD, trained only to maximize reward, drives the nail in a single, overly forceful hit that evaluators consistently judged "unlike a human."
In the Pen task, six evaluators placed MAQ+IQL at Rank I more than any other policy, while vanilla RLPD received five first-place votes and the human demonstration received only two.
What evaluators liked about MAQ policies. Written feedback centers on natural, well- coordinated grip and fine adjustments that keep the pen aligned with the target:
"It using all the fingers." "It feels like all fingers are used."
"Humans would likely adjust the pen to be as close as possible to the target model."
"The middle and ring fingers move with the rest of the hand, which aligns with ergonomic principles."
"Holding the pen like this is very steady and human-like."
"The finger angles are not too strange and the task is completed quickly."
Why baseline agents were judged less human-like. Agents trained without MAQ conditioning were criticized for awkward finger placement and lack of re-aiming:
"It's s unnatural for a human to leave one finger open when trying to grasp."
feel like fingers are out of control."
"Atter holding the pen, wouldn't deliberately stick out single finger."
"It does not adjust the pen."
"He basically shows no intention of aiming or aligning."
"People don't hold a pen vertically."
"The angles of the fingers appear twisted."
These remarks echo the ranking statistics: MAQ conditioning encourages full-hand coordination and incremental aligmment, which are features that evaluators intuitively associate with human pen manipulation, whereas reward-only policies often adopt grasp patterss that look distinetly non-human.
In the Relocate task, six evaluators placed the human demonstration at Rank 1, but MAQ+IQL was close second with five first-place votes, followed by MAQ+RLPD (three) and MAQ+SAC (one).
What evaluators liked about MAQ policies. Positive remarks emphasize three recurring traits: direct, purposeful approach to the ball, continuous motion once the object is secured, and smooth point-to-poin transfer.
"It moves towards the ball."
"After the ball is grasped, the motion continues without any pause."
"Humans should smoothly move an object from the source to the destination point the hand is gently and smoothly picking, moving, and putting."
"At least it puts the ball near the target position."
"The hand shows slight grasping motions, and the movement trajectory feels quite natural." "
"People normally go straight to grab the ball."
Why baseline agents were judged less human-like. pattern, the evaluators reacted strongly:
When an agent broke the
29

---

"I moves away from the ball."
"The hand stopped moving before it actually grasped the ball."
"It looks like it can't It' even pick up the ball."
"It does not move the object all."
"The arm and finger movements are both very chaotic." " "When grab the ball, would align my palm to the ball, not my wrist."
Taken together, these comments show that MAQ conditioning steers agents toward the straight- line reach, uninterrupted grasp-and-plase sequence that evaluators intuitively associate with human relocation behavior, whereas reward-only policies often hesitate, wander, or execute awkward wrist- first contacts that look distinctly non-human.
Summary of MAQ Advantages. Across all appendix tasks, including Hammer, Pen, and Relocate, the MAQ-coonitionee agents consistently receive the highest human-likeness rankings among the learned policies. Evaluators repeatedly highlight three qualities that MAQ alone imparts:
Preparatory alignment. MAQ policies re-aim, re-grip, or re-strike ways that mirror how humans make minor adjustments before the decisive action.
2. Appropriate force modulation. MAQ agents hammer with several well-timed taps, guide the pen with gentle finger coordination, and transport the ball with a continuous grasp-and- place siannenceeeehavoor evaluators intuitively regard as natural.
These observations reinforce our central claim: MAQ does not simply optimize task scores; it systematically shapes behavior toward the timing, grip strategy, and motion fluidity that humans recognize as their own.
30

---

D4RL Adroit door
DARL Adtlot -do0
Purman Celecion Pheae
(a) Human Detection Phasc.
(b) Ranking Phase.
D4RL Adroit door
Ranking
Finll Ranking
derre Merieys
Leest HurrenBie
why (sptiona)
why
(eprional)
You lave glotad ganer the qued sndiue. Bcrsnncs
Sani tu dili
Inbck
(c) Post-ranking feedback.
Figure 11: Evaluation protocol.
31

---

3:00
MMQ
Figure 12: Probability distribution of agents ranked top 1 and top Ranking Phase.
(a) MAQ+RLPD
(b) RLPD
Figure 13: Agent behavior Hammer.
32


Reviews:
[{"review_id": "Oqs3Oz99rj", "reviewer": "Reviewer_HZYt", "summary": "- The paper explores the question of how to design agents that accomplish tasks like humans do. \n- The paper tries to generate more human-like and relatively smooth trajectories while avoiding traits like shakiness, and other unnatural behaviors. \n- The authors introduce MAQ (Macro-Action Quantization) framework where human action sequences are distilled into macro-actions (sequence of actions) using a conditional VQ-VAE with an aim to match the generated action sequences with human-like demonstration along with maximizing the required rewards. \n  - MAQ (state-conditioned VQ-VAE) transforms that primitive low-level action space into a high-level behavior-based macro actions space. \n  - Enforces the policy to select these macro-actions that can be decoded back to more human line action sequences (trajectories).\n  - The VQ-VAE codebook, on a high-level, represents a collection of human-like meta-actions or behavior or skills that can be combined together to form a trajectory. It becomes a reduced macro-action-space as compared to the original action space.\n- The RL agent tries to optimize (in a semi-MDP setting) over short action sequences to capture detailed human behavior.\n- The trajectories within an horizon (of sequence len = H) are executed based on receding-horizon control and are replanned after k (<= H) steps.\n- The paper uses Dynamic Time Warping and Wasserstein distance as trajectory similarity metrics to calculate human-likeness in the generated trajectories. \n- Proposes an interesting work to extend research on designing human-like agents/trajectories.", "questions": "- How are we making sure that we have proper coverage of the state-action pair for human demonstrations for different experiments to have effective/generalized quantization?\n- Any particular way/frequency of updating the codebook (clusters) or having more robust codebooks that don't change drastically with the new information.\n- Line 123: Are there any ablations for j to have reactivity vs human-likeness metrics comparison?\n- Equation 4: What is this loss term $||m - \\tilde{m}||^2$ ? Is this the L2 norm over the action sequences or log-likelihood?\n\n\n**Suggestion:**\nIt seems that this framework can also be leveraged for hierarchical RL with the high-level (low frequency) behaviour planner and a low-level (high-frequency planner).", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n- The paper does a good job of having an exhaustive analysis of the experiments based on the mathematical and human-evaluation metrics.\n- The authors very well presented answers to the questions:\n  - Can MAQ accomplish tasks while aligning with human demonstrations?\n  - Can and how likely MAQ generated trajectories fool human evaluators?\n- Presents different evaluation methods to showcase the results. \n  - Turing test with human participants to distinguish between the generated (using different RL baselines) and human trajectories and had the following results:\n   - MAQ+RLPD (39%) > MAQ+SAC (34%) > 288 MAQ+IQL (32%) > BC (24%) = RLPD (24%) > SAC (19%) > IQL (13%).\n  - For the human-likeness ranking test: MAQ+RLPD achieved a 71% win rate, very close to the human demonstrations which had a 74% win rate. \n  (*References for these baselines in the paper)\n\n- The authors claimed the MAQ seems to have more potential to solve tasks requiring intricate scenarios with numerous macro actions. \n- The results demonstrated that MAQ based agents can effectively complete the tasks and are human-like as well. \n\n\n**Weaknesses/Limitations**:\n- The proposed framework would require expert human or optimal demonstration data in order to encode near-human-like behaviors. \n- The paper assumes that the human-demo are optimal trajectories and in that case, it might miss the intent of the task if they are near-suboptimal. \n- The paper does not talk about any dynamic environments where the frequency of executing actions might play an important role in having more reactive policies and yet human-like. \n\nMultiple aspects of this work can be derived for other tasks related to human-like task completion or human-like trajectory classification, or distilling/enforcement of particular conditional behaviors.", "was_revised": true, "final_justification": "Addressed Concerns:\n- The authors have addressed the concern related to dynamic environments.\n- They conducted experiments to test the proper coverage of the state-action pairs.\n- They conducted experiments/ablations for j to have reactivity vs human-likeness metrics comparison.\n- Compared the results with other IL algorithms like IQL.\n- The authors successfully clarified that the MAQ framework does not require optimal or expert demonstrations.\n- The use of codebook reinitialization was explained as a method to ensure the robustness and effective usage of the codebook.\n\nIt seems that the quality of the paper can be/would be improved after the rebuttal discussions and experiments are included in the paper. I would like to go my initial score of 4 considering the limitations pointed out in the overall rebuttal.", "rebuttal": "We thank the reviewer for constructive feedback and thoughtful reviews. We address each concern and question as follows.\n\n> W1: The proposed framework would require expert human or optimal demonstration data in order to encode near-human-like behaviors\n\nWe would like to clarify that our proposed MAQ framework does not require expert or optimal human demonstrations. Instead, MAQ is designed to distill human-like behaviors from any human demonstration and integrate them with RL algorithms to improve task performance.\n\nFor example, in D4RL [1][2], two types of datasets are provided: (a) a human demonstration dataset (25 trajectories per task) collected from humans, and (b) an expert demonstration dataset (5000 trajectories per task) generated by a fine-tuned expert RL policy. As shown in the table below, the rewards of the human dataset are substantially lower than those of the expert dataset (except Pen), indicating that the human demonstrations are far from optimal.\n\n|Task|Human dataset reward|Expert dataset reward|\n|-|-|-|\n|Door|784.9|2899.8|\n|Hammer|3049.7|12275.5|\n|Pen|6290.2|3283.5|\n|Relocate|3643.2|4293.5|\n\nIn our experiments, MAQ is trained solely on the human dataset, without using any expert dataset. Although these trajectories are not optimal, our results show that MAQ is still able to achieve both human-likeness and high success rates. This shows that MAQ can effectively capture human behavior even from suboptimal demonstrations.\n\n[1] Rajeswaran, Aravind, et al. \"Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.\" arXiv preprint arXiv:1709.10087 (2017).\n\n[2]   Fu, Justin, et al. \"D4rl: Datasets for deep data-driven reinforcement learning.\" arXiv preprint arXiv:2004.07219 (2020).\n\n> W2: The paper assumes that the human-demo are optimal trajectories and in that case, it might miss the intent of the task if they are near-suboptimal.\n\nFollowing our previous response, MAQ does not require optimal trajectories. However, we agree with the reviewer that training on suboptimal demonstrations may affect task performance. We have also noted this in the Discussion section: \"The quality of these demonstrations can affect the effectiveness of MAQ.\"\n\nTo investigate the impact of demonstration quality, we conduct an ablation study using different subsets of trajectories with the lowest task rewards. Specifically, we split the training/testing dataset based on the reward of each trajectory and evaluate three scenarios:\n* 75% low-reward: train on 75% lowest-reward trajectories and test on the remaining 25% highest-reward ones\n* 50% low-reward: train on 50% lowest-reward trajectories and test on the remaining 50%\n* 25% low-reward: train on 25% lowest-reward trajectories and test on the remaining 75%\n\nSince reducing the size of the training dataset will affect the performance, we also include an experiment using the same data proportions but with trajectories selected randomly. The results are shown below.\n\n||||75%|75%|50%|50%|25%|25%|\n|-|-|-|-|-|-|-|-|-|\n|||MAQ+RLPD|random|lowest|random|lowest|random|lowest|\n|Door|DTW$_s$ (↓)|302.19|261.29|239.77|277.42|298.26|350.23|414.27|\n||DTW$_a$ (↓)|275.20|257.20|242.36|263.66|294.37|335.20|408.44|\n||WD$_s$ (↓)|4.59|4.38|4.70|4.42|4.77|5.19|5.85|\n||WD$_a$ (↓)|3.73|3.37|3.85|3.27|3.77|4.10|4.49|\n||Success|0.93|0.98|0.79|0.85|0.55|0.48|0.03|\n||\n|Pen|DTW$_s$ (↓)|740.45|679.96|590.02|669.14|649.56|686.26|686.74|\n||DTW$_a$ (↓)|664.75|629.19|560.92|624.25|607.43|654.41|652.16|\n||WD$_s$ (↓)|8.55|8.02|9.70|7.79|8.47|8.24|8.61|\n||WD$_a$ (↓)|6.051|5.95|8.42|5.53|6.01|5.85|6.24|\n||Success|0.417|0.33|0.32|0.33|0.23|0.23|0.20|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\nAs expected, training with fewer demonstrations reduces performance in both low-reward and random settings. Moreover, training on low-reward trajectories consistently leads to lower success rates compared to random. This demonstrates that the quality of demonstrations affects task performance. However, DTW and WD scores remain consistent across all scenarios, even when training on the lowest-reward trajectories. This suggests that MAQ can still effectively capture human behavior from suboptimal trajectories. We would be happy to include this additional ablation study in the revision if the reviewer finds it satisfactory.\n\n> W3: The paper does not talk about any dynamic environments where the frequency of executing actions might play an important role in having more reactive policies and yet human-like.\n\nTo evaluate using MAQ in dynamic environments, we conduct an additional experiment on Enduro, a dynamic Atari environment with rapidly changing scenes, frequent decision-making, and the need for real-time reactions to avoid collisions.\n\nWe train both PPO and MAQ+PPO on Enduro. For MAQ, the macro-action sequence length is set to 9. To assess human-likeness in the discrete action space, we adopt the method proposed in [3], which calculates the Wasserstein distance between action distributions, where xN represents the aggregated action histogram over N steps. The results below show that MAQ+PPO outperforms PPO in both scores and human-likeness, demonstrating that MAQ can effectively operate in dynamic environments while preserving human-like behavior.\n\n||PPO|MAQ+PPO|\n|-|-|-|\n|WD$_a$ x1 (↓)|0.86|**0.45**|\n|WD$_a$ x8 (↓)|2.49|**1.55**|\n|WD$_a$ x16 (↓)|3.55|**2.59**|\n|Score|25.66|**425.33**|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\n[3] Pearce, Tim, et al. \"Imitating human behaviour with diffusion models.\" arXiv preprint arXiv:2301.10677 (2023).\n\n> Q1: How are we making sure that we have proper coverage of the state-action pair for human demonstrations for different experiments to have effective/generalized quantization?\n\nTo investigate the coverage of state-action pairs from human demonstrations, we analyze whether actions from testing trajectories appear within the VQVAE codebook. Specifically, for each state-macro action pair $(s, m)$ from the testing trajectories, we input the state $s$ into the trained VQVAE decoder to reconstruct all macro actions corresponding to the codebook entries. Then, we check whether the macro action $m$ appears among these decoded outputs.\n\nSince D4RL uses continuous actions, we calculate the L2 distance between ground-truth macro action $m$ and each decoded macro action $\\tilde{m}$. If the distance to any codebook entry is below a predefined threshold $t$, we consider $m$ to be covered by the codebook. The table below shows the evaluation results. For comparison, we also applied the same analysis to expert trajectories, which do not exhibit non-human-like behavior. The results demonstrate that our VQVAE trained on human demonstrations achieves high coverage on human trajectories but very low coverage on expert trajectories.\n\n|Threshold ($t$)|Human|Expert|\n|-|-|-|\n|0.1|98.77|52.33|\n|0.05|95.34|0.13|\n|0.04|93.84|0.008|\n|0.03|90.70|0|\n|0.02|82.22|0|\n|0.01|29.19|0|\n\n> Q2: Any particular way/frequency of updating the codebook (clusters) or having more robust codebooks that don't change drastically with the new information.\n\nWe adopt a commonly used technique in VQVAE: codebook reinitialization [4], which helps maintain effective codebook usage and avoids codebook collapse. Specifically, if certain codebook entries remain unused for a period, they are reinitialized with the most frequently used embeddings. This strategy ensures that the codebook remains robust.\n\n[4] Williams, Will, et al. \"Hierarchical quantized autoencoders.\" Advances in Neural Information Processing Systems 33 (2020): 4524-4535.\n\n> Q3: Line 123: Are there any ablations for j to have reactivity vs human-likeness metrics comparison?\n\nWe conduct an experiment in the Pen environment using MAQ+RLPD with different values of $j$, including 1, 3, 5, and 7. The table below shows that both human-likeness and performance remain similar across different $j$. This is likely because $H$ (action sequence length) is set to 9 during MAQ training, so even when $j=1$, the agent is still able to execute human-like behavior.\n\nWe choose $j=H$ in the paper because longer sequences tend to generate smoother, uninterrupted human-style motion. In addition, this setting reduces the replanning frequency, which improves training efficiency.\n\n|j=9 (paper setting)|j=7|j=5|j=3|j=1|\n|-|-|-|-|-|\n|DTW$_s$ (↓)|740.45|722.54|730.76|738.15|730.31|730.31|\n|DTW$_a$ (↓)|664.75|651.99|664.43|666.73|673.95|673.95|\n|WD$_s$ (↓)|8.55|8.49|8.54|8.52|8.53|8.53|\n|WD$_a$ (↓)|6.05|6.00|6.05|6.03|6.09|6.09|\n|Success|0.42|0.37|0.44|0.41|0.42|0.42|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\n> Q4: Equation 4: What is this loss term ||m-~m||^2? Is this the L2 norm over the action sequences or log-likelihood?\n\nWe use the L2 norm over the action sequences."}, {"review_id": "IkYWXhdVlk", "reviewer": "Reviewer_rcnz", "summary": "This paper proposes learning human-like RL agents to be explicable and facilitate human-AI interaction. The key idea is to generate behavior via connecting human-like behavior segments learned from human demonstrations.These segments are then viewed as options in RL. Generalization to different situations is achieved by the application of vector quantized variational autoencoder (VQVAE). The system is evaluated in four Adroit tasks (including opening door, hammering a nail, twirling a pen, moving a ball), where the generated behaviors are compared with human demonstrations using dynamic time warping and Wasserstein distances, success rate, as well as via human studies. Results show that the generated behaviors achieve higher similarity scores and success rates, while being less distinguishable from human demonstrations.", "questions": "1. Any reason for why prior works that leverage human demonstrations and can be used to generate \"human-like\" behaviors are not considered or compared, such as GAIL (see below)? Note that directly comparing with imitation learning that does not have the ability to adapt/generalize (such as BC) is not exactly fair (\"This is because the BC agent learns to imitate state-action pairs from the demonstrations, without optimizing for successful outcomes.\"). \n\nHo, Jonathan, and Stefano Ermon. \"Generative adversarial imitation learning.\" Advances in neural information processing systems 29 (2016).\n\n2. Three different RL methods are chosen as the baselines to evaluate the generalizability of the proposed work. Why is this efficient use of space given that the learned macro actions are essentially treated as actions (assuming my understanding is correct) so it should generalize to whatever standard RL method? After all, we are not really interested in comparing these RL methods. \n\n3. In general, there is significant variance in the performance across different tasks. What do you think the problem is and how you plan to address it?\n\n4. In 5.3.2, it is claimed that \"Moreover, MAQ+RLPD achieves a 71% win rate, nearly indistinguishable from the human demonstrations, which have a 74% win rate. This suggests that MAQ+RLPD can convincingly fool human evaluators into believing its behavior is human-like.\" It this true? Even though MAQ+RLPD has an overall win rate of 71%, Figure 4 still shows that human demonstration dominates it, no?\n\n5. How do you set the reward for the macro actions? I did not find information about that in the paper.", "limitations": "Some of the limitations are discussed by the authors while others remain unaddressed. Please refer to my comments above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well motivated and clearly written. \n2. The behavior generation pipeline seems to be effective in creating more human-like behaviors than the selected baselines. \n\nWeaknesses:\n\n1. The idea of using macro actions is restrictive for RL agents in contrast to prior work that may be leveraged to achieve similar results. Furthermore, the human-likeliness relies on the assumption of compositionality of macro actions, which may or may not hold in general.  \n2. Learning VQVAE requires expert demonstrations that are carefully mapped to critical subtasks to be generalizable. \n3. Even though the proposed approach seems to have improved human likeliness with respect to the selected baselines, the result is highly domain dependent, revealing a critical gap to be filled.", "was_revised": true, "final_justification": "Although I still maintain concerns regarding the fixed length macro actions and the use of RL to chain these seemingly \"arbitrary\" macro actions, the authors have provided reasonable evidence that demonstrated its effectiveness at generating human-like behaviors. To some extent, I was quite surprised at the results given the simplicity of the approach but kudos to the authors. Hence, I am adjusting my rating to \"weak accept\". \n\nOn the other hand, in addition to incorporating the relevant discussion during the author-reviewer discussion phase into the final version, I would encourage the authors to more carefully discuss the limitations of their approach, such as fixed length macro actions and potentially the difficulty with out of distribution scenarios. I would also encourage the authors to discuss the limitations of the evaluation. I feel that the performance improvement against the baselines could as well be explained away by other factors, such as whether or not the task objective is optimized (BC) or the computational advantage of using macro actions with RL (IQ-learn), not necessarily the proposed human-like behavior generation mechanism. Along this line, how would you expect MAQ to compare against RL with options where the options are manually constructed or learned? I would be curious to see the results.", "rebuttal": "We thank the reviewer for constructive feedback and thoughtful reviews. We address each concern and question as follows.\n\n> W1: The idea of using macro actions is restrictive for RL agents in contrast to prior work that may be leveraged to achieve similar results. Furthermore, the human-likeliness relies on the assumption of compositionality of macro actions, which may or may not hold in general.\n\nWe chose to use macro actions based on the assumption that sequences of actions better reflect human behavior. While prior work has attempted to produce human-like behavior using hand-crafted cost functions or reward shaping at the primitive action level, such approaches often require task-specific domain knowledge, limiting their generality. In contrast, our method relies only on human demonstrations and avoids manual task-specific design, making it more applicable across domains.\n\nTo address concerns regarding potential restrictions in executing macro actions, we also conducted an ablation study in which agents still learn macro actions, but at test time only execute the first few steps of each macro action to allow more reactive behavior. Specifically, with a macro-action sequence length of 9, we tested executing only the first $j \\in${$1,3,5,7$} actions. The table below shows that both human-likeness and performance remain similar across different $j$. Notably, even when $j=1$, the agent remains human-like while increasing reactivity.\n\n|j=9 (paper setting)|j=7|j=5|j=3|j=1|\n|-|-|-|-|-|\n|DTW$_s$ (↓)|740.45|722.54|730.76|738.15|730.31|730.31|\n|DTW$_a$ (↓)|664.75|651.99|664.43|666.73|673.95|673.95|\n|WD$_s$ (↓)|8.55|8.49|8.54|8.52|8.53|8.53|\n|WD$_a$ (↓)|6.05|6.00|6.05|6.03|6.09|6.09|\n|Success|0.42|0.37|0.44|0.41|0.42|0.42|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\nRegarding the compositionality assumption, we agree that this is an assumption. However, our empirical results suggest that the learned macro actions are reasonably composable in practice. We also agree that a study of macro-action compositionality would be a valuable direction for future work.\n\n> W2: VQVAE requires expert demonstrations …\n\nWe would like to clarify that our proposed MAQ framework does not require expert or optimal human demonstrations. Instead, MAQ is designed to distill human-like behaviors from any human demonstration and integrate them with RL algorithms to improve task performance.\n\nFor example, in D4RL [1][2], two types of datasets are provided: (a) a human demonstration dataset (25 trajectories per task) collected from humans, and (b) an expert demonstration dataset (5000 trajectories per task) generated by a fine-tuned expert RL policy. As shown in the table below, the rewards of the human dataset are substantially lower than those of the expert dataset, indicating that the human demonstrations are far from optimal.\n\n|Task|Human dataset reward|Expert dataset reward|\n|-|-|-|\n|Door|784.9|2899.8|\n|Hammer|3049.7|12275.5|\n|Pen|6290.2|3283.5|\n|Relocate|3643.2|4293.5|\n\nIn our experiments, MAQ is trained solely on the human dataset, without using any expert dataset. Although these trajectories are not optimal, our results show that MAQ is still able to achieve both human-likeness and high success rates. This shows that MAQ can effectively capture human behavior even from suboptimal demonstrations.\n\n[1] Rajeswaran, Aravind, et al. \"Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.\" arXiv preprint arXiv:1709.10087 (2017).\n\n[2] Fu, Justin, et al. \"D4rl: Datasets for deep data-driven reinforcement learning.\" arXiv preprint arXiv:2004.07219 (2020).\n\n> W3: … the result is highly domain dependent …\n\nWe agree with the reviewer that MAQ requires access to human demonstrations in the target domain, as noted in our Discussion section: \"*Although MAQ exhibits more human-like behavior, one limitation is its reliance on human demonstrations to distill macro actions. The quality of these demonstrations can affect the effectiveness of MAQ. However, since our primary goal is to pursue human-likeness, obtaining human demonstrations is both necessary and appropriate within the scope of this work.*\"\n\nHowever, we would like to emphasize that, compared to prior human-like RL approaches that rely on domain knowledge to manually design human-likeness cost functions, our method only requires collecting demonstrations. This greatly reduces the effort and cost of adapting to new domains, making MAQ more practical for real-world applications. While extending MAQ to cross-domain generalization remains an interesting direction for future work, we believe our method already represents a significant step in human-like RL by offering a general, data-driven solution that avoids handcrafted, domain-specific constraints.\n\n> Q1: Any reason for why prior works that leverage human demonstrations and can be used to generate \"human-like\" behaviors are not considered or compared, such as GAIL (see below)? \n\nWe include BC as a baseline to represent a simple and widely used method. As the reviewer noted, BC captures human-likeness to some degree but does not optimize with task success rate. We did not include GAIL for several reasons. First, GAIL relies on high-quality expert demonstrations, while our focus is on imitating human behavior, which can be suboptimal but diverse. Second, GAIL is known to require extensive training time to train in high-dimensional continuous control tasks. For example, training GAIL on the 6-dimensional Walker task takes ~25M steps. In contrast, our method learns macro actions for the 24-dimensional Pen task in only 1M steps. We also attempted to train GAIL for 1M steps on the four D4RL tasks, but GAIL failed to produce any successful trajectories, possibly requiring more training and computational resources.\n\nHowever, we agree that comparing other imitation learning methods would strengthen our results. To this end, we included IQ-Learn [1], one of the strongest IL algorithms. Since IQ-Learn was not originally tested on D4RL tasks, we applied it (using the hyperparameters recommended in the original paper) to all four tasks. The results below show that IQ-Learn performs poorly in both human-likeness and task success, while MAQ+IQ-Learn improves both performance. We believe this additional experiment offers a more convincing comparison and also demonstrates that MAQ is complementary to IL algorithms. We can include this experiment if the reviewer finds it satisfactory.\n\n[1] Garg, Divyansh, et al. \"Iq-learn: Inverse soft-q learning for imitation.\" Advances in Neural Information\n\n|||IQ-Learn|MAQ+IQ-Learn|\n|-|-|-|-|\n|Door|DTW$_s$ (↓)|633.88|358.11|\n||DTW$_a$ (↓)|796.35|345.33|\n||WD$_s$ (↓)|9.10|5.11|\n||WD$_a$ (↓)|8.18|4.04|\n||Success|0.00|0.25|\n||\n|Hammer|DTW$_s$ (↓)|844.00|627.05|\n||DTW$_a$ (↓)|1261.88|607.84|\n||WD$_s$ (↓)|9.58|6.25|\n||WD$_a$ (↓)|7.41|4.18|\n||Success|0.00|0.04|\n||\n|Pen|DTW$_s$ (↓)|1025.75|778.53|\n||DTW$_a$ (↓)|768.69|684.22|\n||WD$_s$ (↓)|10.32|8.86|\n||WD$_a$ (↓)|7.11|6.38|\n||Success|0.09|0.20|\n||\n|relocate|DTW$_s$ (↓)|641.08|534.19|\n||DTW$_a$ (↓)|1261.29|591.09|\n||WD$_s$ (↓)|10.20|7.72|\n||WD$_a$ (↓)|8.77|5.62|\n||Success|0.00|0.02|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\n> Q2: Three different RL methods are chosen … Why is this efficient use of space given that the learned macro actions are essentially treated as actions (assuming my understanding is correct) so it should generalize to whatever standard RL method?\n\nWe fully agree with the reviewer's point! As noted, MAQ is fundamentally a framework that transforms actions into learned macro actions, making it to be easily integrated into various RL algorithms. However, we were concerned that reviewers/readers might expect empirical evidence of this generality. Therefore, we included experiments with three representative RL algorithms, IQL, SAC, and RLPD, to demonstrate that MAQ consistently improves human-likeness while maintaining high task success rates across different RL algorithms. Note that our intention is not to compare these RL algorithms themselves, but to show that each benefits from integrating MAQ, thereby highlighting its flexibility and generality. We welcome any suggestions for improving our experimental design.\n\n> Q3: … significant variance in the performance across different tasks.\n\nThe performance variance is primarily due to inherent differences in task difficulty. For example, the D4RL paper [2] (Table 2) reports that among the four Adroit tasks, Pen and Door are relatively easier, while Relocate (requiring precise grasping, lifting, and moving of an object) is the most challenging. This result also aligns with our experimental results. In summary, the performance variance is not specific to MAQ, but is based on differences in task difficulty.\n\n> Q4: Even though MAQ+RLPD has an overall win rate of 71%, Figure 4 still shows that human demonstration dominates it, no?\n\nIt is true that when directly comparing human demonstrations against MAQ+RLPD in the ranking test, human demonstrations have a 62% win rate. However, our original claim is based on the **overall win rate across all agent pairs**. For example, human demonstrations only achieve an 87% win rate against SAC, whereas MAQ+RLPD achieves a 95% win rate against SAC. This suggests that human evaluators found MAQ+RLPD to be more human-like than human demonstrations when both are compared to SAC. Therefore, our intention is to highlight MAQ+RLPD's overall performance in fooling human evaluators across all comparisons, rather than to claim it outperforms human demonstrations directly. We thank the reviewer for the feedback and will revise the phrasing in the revision to clarify this more precisely.\n\n> Q5: How do you set the reward for the macro actions?\n\nThe reward of macro action is computed as the sum of each step's reward over the entire action sequence. We will include this information in the revision."}, {"review_id": "wgYKdVgMQP", "reviewer": "Reviewer_HJQk", "summary": "This paper formulates the challenge of producing human-like behavior in reinforcement learning (RL) as a trajectory optimization problem. To address this, the authors introduce Macro Action Quantization (MAQ), a framework that distills human demonstrations into macro-actions using a Conditional Vector Quantized Variational Autoencoder (VQVAE). The VQVAE is trained to encode sub-trajectories of length H into a discrete codebook of size K. A policy is then learned over this discrete action space, selecting macro-actions via codebook indices to maximize environment rewards. Each selected index is decoded into a sequence of primitive actions conditioned on the current state.\n\nEmpirical evaluation on the D4RL Adroit benchmark shows that integrating MAQ with standard RL algorithms (IQL, SAC, RLPD) leads to a significant improvement in human-likeness, as measured by Dynamic Time Warping and Wasserstein distance, without substantial loss in task performance. Additionally, a human evaluation study is conducted using a Turing Test style setup and a pairwise ranking test, demonstrating that MAQ-trained agents are consistently judged to behave more like humans compared to baseline agents.", "questions": "1. Statistical Validity of Human Evaluation: How many human evaluators participated in the Turing Test and ranking tests? Please include this number and report significance tests to support the reliability of these results.\n2. RLPD Degradation: In Table 1 and Appendix Figure 6, RLPD seems to suffer significantly when combined with MAQ. Can the authors clarify this discrepancy between reported success rate and observed reward curves? Can you provide more details on how the success rate is computed vs reward curve normalized score\n3. Demonstration Quality: Were the human demonstrations used in the evaluation (e.g., Turing Test) all successful executions? Were agent demonstrations sampled from successful rollouts? This information is crucial for fair comparison.\n4. Motivation and Impact: The paper briefly mentions the importance of human-likeness but does not provide a compelling argument or citations. Could the authors expand on the value and implications of designing human-like agents, including societal risks or limitations?", "limitations": "The paper does not adequately discuss the limitations or potential societal impacts of building human-like agents. The Discussion section does not meaningfully reflect on possible negative consequences or ethical concerns (e.g., manipulation, human trust, or deceptive agents). I recommend the authors include a Broader Impacts section that honestly addresses these issues and situates their work in a wider context.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper invests in analyzing the human-likeness of policies trained with RL, which, although tangentially implicit when using human data for training, is a relatively novel evaluation of the actual behavioral outcome. A Turing Test-style human evaluation provides additional validation of perceived human-likeness.\n- The proposed MAQ framework integrates well with existing RL algorithms and empirically improves human-likeness metrics across tasks.\n\n**Weaknesses:**\n\n- Figures are poorly described, particularly Figure 1, which lacks subfigure explanations and adequate detail.\n- If right, The evaluation uses only ~2.5 human demonstration trajectories per task, raising concerns about robustness of human likenliness mesures\n- There doesn’t seem to be a clear observation  that MAQ  degrades task performance for the highest scoring RL algorithm, RLPD. Difference is even more obious when looking at the appendix reward curves, Figure 6.\n- The motivation for human-like behavior is weakly argued and under-cited, beyond bringing in discussion the Turing Test.\n- The number of human evaluators is not reported, and no statistical significance analysis is provided for the Turing Test or ranking results.\n- The paper lacks a Broader Impact or Societal Impact discussion, especially relevant when aiming to mimic human behavior.\n- It remains unclear whether demonstrations used in the human-likeness evaluation are successful ones. This could unfairly bias comparisons.", "was_revised": true, "final_justification": "The rebuttal effectively clarified several concerns, e.g., RLPD degradation, evaluation details, and demonstration quality. However, motivation for human-likeness, broader impact discussion, and figure clarity remain for future edits. The method is technically solid but limited in significance, so I keep my overall rating, although quality has gone up.", "rebuttal": "We thank the reviewer for constructive feedback and thoughtful reviews. We address each concern and question as follows.\n\n> W1: Figures are poorly described, particularly Figure 1, which lacks subfigure explanations and adequate detail.\n\nThank you for the suggestion. We will revise the caption to include clearer subfigure explanations and more detailed descriptions in the revision.\n\n> W2: If right, The evaluation uses only ~2.5 human demonstration trajectories per task, raising concerns about robustness of human likenliness mesures\n\nYes, due to the limited number of human demonstration trajectories (25 per task), we adopted a 9:1 train/test to maximize the training data available for learning human behavior. As a result, the test data contains only 3 trajectories. To mitigate concerns about robustness, each algorithm in the paper is trained three times using three different randomly selected train/test splits (while maintaining the 9:1 ratio). This reduces the chance of overfitting to any particular test data and helps ensure the evaluation results are reliable and robust.\n\nIn addition, to further assess robustness, we conduct an ablation study using different amounts of training data in Door and Pen environments: 90% (as in the paper), 75%, 50%, and 25%. The results below show that while the success rate decreases as the amount of training data is reduced (as expected), the trajectory similarity scores remain consistent. This demonstrates the robustness of our method.\n\n|||90% (22:3)|75% (18:7)|50% (12:13)|25% (6:19)|\n|-|-|-|-|-|-|\n|Door|DTW$_s$ (↓)|302.19|261.29|277.42|350.23|\n||DTW$_a$ (↓)|275.20|257.20|263.66|335.20|\n||WD$_s$ (↓)|4.59|4.38|4.42|5.19|\n||WD$_a$ (↓)|3.73|3.37|3.27|4.10|\n||Success|0.93|0.98|0.85|0.48|\n|||||||\n|Pen|DTW$_s$ (↓)|740.45|679.96|669.14|686.26|\n||DTW$_a$ (↓)|664.75|629.19|624.25|654.41|\n||WD$_s$ (↓)|8.55|8.02|7.79|8.24|\n||WD$_a$ (↓)|6.05|5.95|5.53|5.85|\n||Success|0.42|0.33|0.33|0.23|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\n> W3: There doesn’t seem to be a clear observation that MAQ degrades task performance for the highest scoring RL algorithm, RLPD. Difference is even more obious when looking at the appendix reward curves, Figure 6.\n> Q2: RLPD Degradation: In Table 1 and Appendix Figure 6, RLPD seems to suffer significantly when combined with MAQ. Can the authors clarify this discrepancy between reported success rate and observed reward curves? Can you provide more details on how the success rate is computed vs reward curve normalized score\n\nWe were also curious about why MAQ+RLPD performed worse than RLPD, especially in the Hammer environment, where the success rate drops from 1.00 to 0.56. After submitting the paper, we conducted a deeper analysis of this task. We find that in the 25 human demonstration trajectories, humans take an average of 451.4 steps to successfully complete the task. However, in the D4RL environment, the agent is restricted to a maximum of 200 steps per trajectory, after which the episode is automatically considered a failure.\n\nSince MAQ learns from human demonstrations, it tends to favor smooth but slow movements, while non-human-like RL agents tend to complete the task more quickly but less naturally. To validate this hypothesis, we increased the maximum episode length from 200 to 450 steps. As shown in the table below, MAQ+RLPD's success rate improves significantly from 0.56 to 0.72 while maintaining human-likeness scores. We will include this analysis with an explanation in the revision.\n\n||RLPD|MAQ+RLPD (200)|MAQ+RLPD (450)|\n|-|-|-|-|\n|DTW$_s$ (↓)|845.88|578.82|648.32|\n|DTW$_a$ (↓)|1178.74|528.52|637.25|\n|WD$_s$ (↓)|9.57|5.20|6.17|\n|WD$_a$ (↓)|7.25|3.32|4.15|\n|Success|1.00|0.56|0.72|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\nRegarding the success rate and normalized reward, in D4RL, the success rate indicates whether the agent successfully completes the task within the episode limit (e.g., 200 steps). In contrast, the normalized reward is a heuristic score defined by the D4RL environment to reflect how close the agent gets to completing the task. For example, in the Door task, the reward increases as the door gets closer to being fully opened. However, only when the door is completely opened, the task marked as successful. Overall, tasks like those in D4RL are primarily evaluated based on whether the agent actually succeeds since the rewards are often used to guide learning, but do not directly indicate task success. This is why we report and compare success rates in the main experiment.\n\n> W4: The motivation for human-like behavior is weakly argued and under-cited, beyond bringing in discussion the Turing Test.\n\nWe understand the reviewer's concern. Indeed, the topic of human-like behavior in RL has been relatively underexplored, as most of the prior work has focused primarily on improving task performance rather than human-like behavior. This is why we start with the concept of the Turing Test to emphasize the importance of developing agents that are not only effective but also human-like. We will revise the paper to strengthen the motivation for human-like RL and include a Broader Impact section (see the response to W6). If the reviewer has any recommended citations or related works, we would be grateful for the suggestions.\n\n> W5: The number of human evaluators is not reported, and no statistical significance analysis is provided for the Turing Test or ranking results.\n> Q1: Statistical Validity of Human Evaluation: How many human evaluators participated in the Turing Test and ranking tests? Please include this number and report significance tests to support the reliability of these results.\n\nThank you for pointing this out. In our human study, a total of 19 human evaluators participated in the Turing Test and ranking test. We will clarify this in the revision.\n\nIn addition, we follow the similar method used in [1] reporting 96% confidence intervals for both the Turing Test and the ranking test. Thank you again for raising this point.\n\n[1] Tirinzoni, Andrea, et al. \"Zero-shot whole-body humanoid control via behavioral foundation models.\" arXiv preprint arXiv:2504.11054 (2025).\n\n> W6: The paper lacks a Broader Impact or Societal Impact discussion, especially relevant when aiming to mimic human behavior.\n> Q4: Motivation and Impact: The paper briefly mentions the importance of human-likeness but does not provide a compelling argument or citations. Could the authors expand on the value and implications of designing human-like agents, including societal risks or limitations?\n> The paper does not adequately discuss the limitations or potential societal impacts of building human-like agents. The Discussion section does not meaningfully reflect on possible negative consequences or ethical concerns (e.g., manipulation, human trust, or deceptive agents). I recommend the authors include a Broader Impacts section that honestly addresses these issues and situates their work in a wider context.\n\nThank you for the suggestion. We will include a Broader Impact section in the revision to address both the potential benefits and risks of human-like agents. The following are several key points we plan to cover. On the positive side, human-like behavior can offer practical benefits in real-world applications, particularly in scenarios involving human-robot collaboration. For example, a robot that assists with human-like movements can improve safety and trust, reducing the risk of accidents. On the other hand, human-like agents may be misused in deceptive or manipulative ways, such as cheating in competitive games. We will include these aspects in the revision. If reviewers feel there are additional aspects worth addressing, we would be happy to include them.\n\n> W7: It remains unclear whether demonstrations used in the human-likeness evaluation are successful ones. This could unfairly bias comparisons.\n> Q3: Demonstration Quality: Were the human demonstrations used in the evaluation (e.g., Turing Test) all successful executions? Were agent demonstrations sampled from successful rollouts? This information is crucial for fair comparison.\n\nYes, all 25 human demonstrations used in our experiments are successful trajectories, as provided in [2]. We will clarify this in the revision. Thank you for pointing it out.\n\n[2] Rajeswaran, Aravind, et al. \"Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.\" arXiv preprint arXiv:1709.10087 (2017)."}, {"review_id": "8PG4OZoIpx", "reviewer": "Reviewer_gwPt", "summary": "The paper addresses the challenge of creating RL agents that not only excel at tasks but also behave in a manner that resembles human actions. \nTo tackle this, the paper formulates human-likeness as a trajectory optimization with human constraints within the RHC framework.\nBased on the formulation, the paper proposes a method (MAQ) to distill human demonstrations into discrete macro actions using VQVAE and training the RL agent in this macro action space, biasing it toward more human-like behavior.\nEmpirical evaluation of the MAQ-enhanced agents show improved alignment with human demonstration trajectories (both in metrics and human evaluations) without sacrificing task performance.", "questions": "- Human evaluation settings: The exact number of evaluators for the human study and the agreement among evaluators are not provided to ensure statistical robustness.\n\n- Additional computational costs: Training the VQVAE component can lead to more computational costs and training time. A cost comparison between the MAQ-based RL training and vanilla RL training should be conducted and discussed.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- Important problem: Addressing the gap between high-performing RL agents and human-like behavior is important, especially for applications where trust, interpretability, and natural behavior are essential (e.g., human–robot interaction and social agents).\n\n- Human evaluation included: The experimental evaluations provide both quatitative (trajectory similarity metrics) and qualitive results (Turing Test and human-likeness ranking) to demonstrate the efficiency of incorporating MAQ (distilled from human demonstrations) into RL training to improve human likeness.\n\nWeaknesses:\n\n- Limited task/domain evaluation: The experiments are confined to a limited benchmark set (Adroit tasks). While this can be a strong testbed, the generality beyond this, especially those tasks or domains heavily demanding for human-like behaviors (e.g., navigation, games, role-playing), is not empirically validated.\n\n- Negative impact on task success (RLPD): The results for MAQ+RLPD suggest MAQ can sometimes negatively impact task performance, especially when the underlying RL agent is highly performant. This needs a more nuanced discussion or a proposed mitigation.\n\n- Insufficient ablation studies: Lack experimental analysis on the (least) amount of human demonstrations required for MAQ-based RL agents, and the the codebook size K in VQVAE.\n\n- Lack of strong IL baselines: Comparison only to BC and not to state-of-the-art imitation learning or inverse RL baselines.\n\n- Limited novelty:  While the particular integration with RHC and the specific quantization method represent an incremental but meaningful improvement, the combination of IL and macro-action frameworks is not entirely new. So the novelty is somewhat restricted by the reliance on existing architectures and frameworks (e.g., VQVAE and RHC), even though the way they are combined is original.", "was_revised": true, "final_justification": "The authors conduct detailed responses, including additional benchmark and baselines, explanation on Negative impact on task success, ablation studies, etc. If these responses can be integrated into the revised paper, the quality will be improved a lot.", "rebuttal": "We thank the reviewer for constructive feedback and thoughtful reviews. We address each concern and question as follows.\n\n> W1: Limited task/domain evaluation: The experiments are confined to a limited benchmark set (Adroit tasks). While this can be a strong testbed, the generality beyond this, especially those tasks or domains heavily demanding for human-like behaviors (e.g., navigation, games, role-playing), is not empirically validated.\n\nYes, as the reviewer noted, we chose Adroit because it is a strong testbed, which requires continuous control and fine-grained skills. We believe it is well-suited for evaluating human-like manipulation behavior.\n\nHowever, we also agree with the reviewer that demonstrating the generality of MAQ is important. Therefore, we conduct additional experiments in two Atari environments:\n* Venture: a navigation and role-playing style task in which player movements are clearly visible\n* Enduro: a third-person racing game with a highly dynamic environment\n\nNext, we train both PPO and MAQ+PPO on these games. For MAQ, the macro-action sequence length is set to 9. To assess human-likeness in the discrete action space, we adopt the method proposed in [1], which calculates the Wasserstein distance between action distributions, where xN represents the aggregated action histogram over N steps. The results below show that MAQ+PPO outperforms PPO in both scores and human-likeness, demonstrating that MAQ can generalize beyond Adroit tasks. We would be happy to include these additional experiments in the revision if the reviewer finds them satisfactory.\n\n|||PPO|MAQ+PPO|\n|-|-|-|-|\n|Enduro|WD_a x1|0.86|**0.45**|\n||WD_a x8|2.49|**1.55**|\n||WD_a x16|3.55|**2.59**|\n||Score|25.66|**425.33**|\n|||||\n|Venture|WD_a x1|1.19|**0.23**|\n||WD_a x8|3.26|**0.91**|\n||WD_a x16|1.67|**1.56**|\n||Score|66.66|**436.66**|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\n[1] Pearce, Tim, et al. \"Imitating human behaviour with diffusion models.\" arXiv preprint arXiv:2301.10677 (2023).\n\n> W2: Negative impact on task success (RLPD): The results for MAQ+RLPD suggest MAQ can sometimes negatively impact task performance, especially when the underlying RL agent is highly performant. This needs a more nuanced discussion or a proposed mitigation.\n\nThis is a great question! We were also curious about why MAQ+RLPD performed worse than RLPD, especially in the Hammer environment, where the success rate drops from 1.00 to 0.56. After submitting the paper, we conducted a deeper analysis of this task. We find that in the 25 human demonstration trajectories, humans take an average of 451.4 steps to successfully complete the task. However, in the D4RL environment, the agent is restricted to a maximum of 200 steps per trajectory, after which the episode is automatically considered a failure.\n\nSince MAQ learns from human demonstrations, it tends to favor smooth but slow movements, while non-human-like RL agents tend to complete the task more quickly but less naturally. To validate this hypothesis, we increased the maximum episode length from 200 to 450 steps. As shown in the table below, MAQ+RLPD's success rate improves significantly from 0.56 to 0.72 while maintaining human-likeness scores. We will include this analysis with an explanation in the revision. Thank you for highlighting this!\n\n||RLPD|MAQ+RLPD (200)|MAQ+RLPD (450)|\n|-|-|-|-|\n|DTW$_s$ (↓)|845.88|578.82|648.32|\n|DTW$_a$ (↓)|1178.74|528.52|637.25|\n|WD$_s$ (↓)|9.57|5.20|6.17|\n|WD$_a$ (↓)|7.25|3.32|4.15|\n|Success|1.00|0.56|0.72|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\n> W3: Insufficient ablation studies: Lack experimental analysis on the (least) amount of human demonstrations required for MAQ-based RL agents, and the the codebook size K in VQVAE.\n\nRegarding the amount of human demonstrations, we conduct an ablation study using different amounts of training data in Door and Pen environments: 90% (as in the paper), 75%, 50%, and 25%. The results below show that while the success rate decreases as the amount of training data is reduced (as expected), the trajectory similarity scores remain consistent. This demonstrates that MAQ can still capture human-like behavior even with as few as 6 trajectories.\n\n|||90% (22:3)|75% (18:7)|50% (12:13)|25% (6:19)|\n|-|-|-|-|-|-|\n|Door|DTW$_s$ (↓)|302.19|261.29|277.42|350.23|\n||DTW$_a$ (↓)|275.20|257.20|263.66|335.20|\n||WD$_s$ (↓)|4.59|4.38|4.42|5.19|\n||WD$_a$ (↓)|3.73|3.37|3.27|4.10|\n||Success|0.93|0.98|0.85|0.48|\n|||||||\n|Pen|DTW$_s$ (↓)|740.45|679.96|669.14|686.26|\n||DTW$_a$ (↓)|664.75|629.19|624.25|654.41|\n||WD$_s$ (↓)|8.55|8.02|7.79|8.24|\n||WD$_a$ (↓)|6.05|5.95|5.53|5.85|\n||Success|0.42|0.33|0.33|0.23|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\nRegarding the codebook size K, please refer to our Appendix. C: \"Impact of Codebook Size on Similarity Score in MAQ\", which provides a detailed analysis of how different values of $K$ affect performance. \n\n> W4: Lack of strong IL baselines: Comparison only to BC and not to state-of-the-art imitation learning or inverse RL baselines.\n\nWe include BC as a baseline to represent a simple and widely used method. As the reviewer noted, BC captures human-likeness to some degree but does not optimize with task success rate. We did not include GAIL for several reasons. First, GAIL relies on high-quality expert demonstrations, while our focus is on imitating human behavior, which can be suboptimal but diverse. Second, GAIL is known to require extensive training time to train in high-dimensional continuous control tasks. For example, training GAIL on the 6-dimensional Walker task takes ~25M steps. In contrast, our method learns macro actions for the 24-dimensional Pen task in only 1M steps. We also attempted to train GAIL for 1M steps on the four D4RL tasks, but GAIL failed to produce any successful trajectories, possibly requiring more training and computational resources.\n\nHowever, we agree that comparing other imitation learning methods would strengthen our results. To this end, we included IQ-Learn [1], one of the strongest IL algorithms. Since IQ-Learn was not originally tested on D4RL tasks, we applied it (using the hyperparameters recommended in the original paper) to all four tasks. The results below show that IQ-Learn performs poorly in both human-likeness and task success, while MAQ+IQ-Learn improves both performance. We believe this additional experiment offers a more convincing comparison and also demonstrates that MAQ is complementary to IL algorithms. We can include this experiment if the reviewer finds it satisfactory.\n\n[1] Garg, Divyansh, et al. \"Iq-learn: Inverse soft-q learning for imitation.\" Advances in Neural Information\n\n|||IQ-Learn|MAQ+IQ-Learn|\n|-|-|-|-|\n|Door|DTW$_s$ (↓)|633.88|358.11|\n||DTW$_a$ (↓)|796.35|345.33|\n||WD$_s$ (↓)|9.10|5.11|\n||WD$_a$ (↓)|8.18|4.04|\n||Success|0.00|0.25|\n||\n|Hammer|DTW$_s$ (↓)|844.00|627.05|\n||DTW$_a$ (↓)|1261.88|607.84|\n||WD$_s$ (↓)|9.58|6.25|\n||WD$_a$ (↓)|7.41|4.18|\n||Success|0.00|0.04|\n||\n|Pen|DTW$_s$ (↓)|1025.75|778.53|\n||DTW$_a$ (↓)|768.69|684.22|\n||WD$_s$ (↓)|10.32|8.86|\n||WD$_a$ (↓)|7.11|6.38|\n||Success|0.09|0.20|\n||\n|relocate|DTW$_s$ (↓)|641.08|534.19|\n||DTW$_a$ (↓)|1261.29|591.09|\n||WD$_s$ (↓)|10.20|7.72|\n||WD$_a$ (↓)|8.77|5.62|\n||Success|0.00|0.02|\n\n\\* Note: DTW and WD are not normalized; lower is better.\n\n> W5: Limited novelty ...\n\nOur contribution lies in how we integrate these existing components to address the underexplored challenge of achieving human-like behavior in RL. We agree with the reviewer that VQVAE and RHC are not novel in themselves. However, to the best of our knowledge, this work is the first to combine macro actions learned from human demonstrations via VQVAE with RHC to achieve both high task performance and human-likeness across multiple RL algorithms.\n\nWhile previous human-like RL methods often rely on handcrafted behavioral cost functions, our approach offers a general framework that avoids manual design. We believe this represents a meaningful and sufficiently novel contribution, and a significant step forward for the human-like RL community.\n\n> Q1: Human evaluation settings …\n\nThere are 19 human evaluators in our human study. We will include this information in the revision. Thank you for pointing it out. We will clarify this in the revision. In addition, we follow the similar method used in [2] reporting 96% confidence intervals for both the Turing Test and the ranking test. Thank you again for raising this point.\n\n[2] Tirinzoni, Andrea, et al. \"Zero-shot whole-body humanoid control via behavioral foundation models.\" arXiv preprint arXiv:2504.11054 (2025).\n\n> Q2: Additional computational costs …\n\nSince we only have 25 trajectories per task, the VQVAE can be trained in a very short time. Specifically, training VQVAE takes only 3 to 4 minutes, whereas RL training requires several hours. While MAQ-based RL introduces some additional computational overhead, we conduct a comparison of inference time for each RL algorithm, with and without MAQ, as shown in the table below. Overall, the cost remains acceptable.\n\n||Vanilla RL (ms)|MAQ (ms)|Ratio (MAQ/Vanilla RL)|\n|-|-|-|-|\n|SAC/MAQ+SAC|1.31|1.35|1.03|\n|IQL/MAQ+IQL|0.81|1.12|1.39|\n|RLPD/MAQ+RLPD|0.28|0.81|2.93|\n\nIt is also worth noting that MAQ can offer faster inference efficiency, as it executes an entire macro action (e.g., 9 consecutive actions in the paper) with a single forward pass, whereas vanilla RL must forward at every timestep. In this case, MAQ-based RL can even achieve faster overall inference time than vanilla RL, especially when longer action sequences are used. We will include these results with a detailed explanation of the computational cost in the revision. Thank you for your suggestion."}]

Author rebuttal:

Meta-review: This paper addresses the challenge of developing reinforcement learning agents that exhibit human-like behavior while maintaining task performance. The authors formulate human-likeness as a trajectory optimization problem and propose Macro Action Quantization (MAQ), a framework that distills human demonstrations into discrete macro-actions using Conditional Vector Quantized Variational Autoencoders (VQVAE). The key innovation is learning fixed-length action sequences from human demonstrations and having RL agents operate in this macro-action space rather than primitive actions. The system is evaluated in four Adroit tasks (including opening door, hammering a nail, twirling a pen, moving a ball), where the generated behaviors are compared with human demonstrations using dynamic time warping and Wasserstein distances, success rate, as well as via human studies. Results show that the generated behaviors achieve higher similarity scores and success rates, while being less distinguishable from human demonstrations. 

The main strengths for this work are as follows: (1)  The paper addresses an important but underexplored area in RL - achieving human-like behavior rather than just optimal performance. This is particularly relevant for applications requiring human-AI interaction and trust.
(2) The authors provide both quantitative trajectory similarity metrics and qualitative human studies. The inclusion of Turing tests and human ranking evaluations strengthens the validity of their human-likeness claims. (3) MAQ demonstrates good generalization across different RL algorithms (IQL, SAC, RLPD) and can be easily integrated into existing methods, making it practically useful. (4) The paper includes appropriate baselines, ablation studies, and addresses key concerns raised during review through additional experiments. 

The main weaknesses of this work are as follows: (1) Experiments were confined to Adroit manipulation tasks. While the authors addressed this during rebuttal with Atari experiments, broader evaluation across diverse domains would strengthen generalizability claims.
(2) The approach relies on fixed-length action sequences (length 9), which may not capture the natural variability in human behavior segments. This constraint could limit applicability to tasks requiring variable-length behaviors. (3)  The paper lacks theoretical analysis of when and why the compositionality assumption of macro actions holds. The empirical success doesn't fully address the fundamental question of macro-action reusability. (4) While addressed in rebuttal, the additional computational costs of VQVAE training and inference deserve more thorough analysis in the main paper. (5) The submission misses analysis on critical hyperparameters including the minimum amount of human demonstrations required and the impact of codebook size K in the VQVAE framework, which was reported in the rebuttal. (6) There is a strong assumption of requiring human demonstrations specifically formatted as macro actions, which may severely limit the method's applicability and require domain-specific knowledge for macro action identification.

Reasons for acceptance: The work addresses an important gap: human-like behavior in RL is an underexplored but practically important problem. The combination of quantitative trajectory similarity improvements and positive human evaluation studies provides convincing evidence of the method's effectiveness. The technical approach is well-motivated, and the experimental methodology is appropriate, with good attention to potential confounds and alternative explanations. The framework's ability to integrate with existing RL algorithms and improve human-likeness across different methods demonstrates practical value. However, there are also a series of limitations in this version, including fixed-length macro actions, compositionality assumptions, and a lot of experiments are done in the rebuttal period instead of the submission, which affects the quality significantly. Although the thorough rebuttal addresses reviewer concerns, we can only accept this work after the necessary discussion and results are involved in the final version.
Decision: Accept (poster)