Unlock the Intermittent Control Ability of Model Free Reinforcement Learning

Authors: jiashun  liu
jianye  hao
xiaotian  hao
yi  ma
yan  zheng
yujing  hu
tangji  lv
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 05:57:04

Abstract: Intermittent control problems are common in real world. The interactions between the decision maker and the executor can be discontinuous (intermittent) due to various types of interruptions, e.g. unstable communication channel. Due to intermittent interaction, agents are unable to acquire the state sent by the executor and cannot transmit actions to the executor within a period of time step, i.e. bidirectional blockage, which may lead to inefficiencies of reinforcement learning policies and prevent the executors from completing the task. Such problem is not well studied in the RL community. In this paper, we model Intermittent control problem as an Intermittent Control Markov Decision Process, i.e agents are expected to generate action sequences corresponding to the unavailable states and transmit them before disabling interactions to ensure the smooth and effective motion of executors. However, directly generating multiple future actions in the original action space has unnatural motion issue and exploration difficulty. We propose **M**ulti-step **A**ction **R**epre**S**entation (**MARS**), which encodes a sequence of actions from the original action space to a compact and decodable latent space. Then based on the latent action sequence representation, the mainstream RL methods can be easily optimized to learn a smooth and efficient motion policy. Extensive experiments on simulation tasks and real-world robotic grasping tasks show that MARS significantly improves the learning efficiency and final performances compared with existing baselines.

Main paper: 
 introduction: 
 In recent years, the field of deep reinforcement learning (DRL) has witnessed striking empirical achievements in a variety of Markov Decision Process (MDP) problems [Mnih et al., 2013 , Kaufmann et al., 2023] and has been successfully applied to many real-time control tasks [Mahmood et al., 2018 , Lee et al., 2020] . In real-time control, "interaction" plays a crucial role. At each time step, the decision maker obtains observations from the environment and feeds actions back to the action executor through real-time interactions. Thus, in an ideal MDP setting, the interaction should be continuous. However, in many scenarios, interactions become intermittent due to limitations of realistic conditions [Jiang et al., 2021] or communication interruption (e.g., unstable network) [Dong et al., 2009] . Due to bidirectional communication blockage caused by intermittent interaction, agents (decision makers) are unable to acquire the state sent by the executor and cannot transmit actions to the executor within a period of time step (as shown in Figure 1 (top). The sparse observations resulting from this phenomenon can make normal step-by-step decisions unstable. Thus, directly deploying existing DRL algorithms could make the action executor abruptly stop (can not make decision when the current state is unavailable) or move abnormally when the environment suddenly changes [Sutton and Barto, 2018] .Figure 1 : Detail of Intermitted Control (top part). Two types of intermittent interactions (bottom part). For both cases, we can generate a sequence of actions in advance for the next c states based on the current state st to make the control more smooth and robust.Typically, there are two types of intermittent interactions. Illustrations are shown in Figure 1(bottom) . ❶ Fixed interaction interval: in numerous real-world robot scenarios, interacting with the environment is often timeconsuming and costly [Liu et al., 2020] . For example, before the robotic arm performs its next actions, it has to first halt its current operations, move to a designated position to capture an image, and then use a specific technique to extract features and provide them to the decision maker. To ensure reliable operation and stable movement, we usually set a fixed interaction interval for the robot arm [Jiang et al., 2021 , Bonarini, 2020] . ❷ Random interaction interval: unexpected interaction intervals may occur due to unstable communication channels and loss of wireless signals [Li et al., 2016] . For example, in real-time strategy games, the decision end remotely controls the terminal non-player characters (NPC) [Zheng et al., 2019] , the interaction interval between the decision end and NPC terminals may become random [Wong et al., 2021] due to above reasons, which may cause the NPC to be stuck and disconnected from the changed environment, reducing the player's experience.The essence of addressing the two aforementioned intermittent control tasks is identical, that is, to achieve intensive control under sparse interaction (or observation) to ensure the effective and smooth movement of the executor, ultimately leading to efficient task completion. To this end, we introduce Intermittent-MDP (refer to Sec.3 for further elaboration) to model the above two settings in a unified manner, that is, the agent is expected to decide on a sequence of actions based on the current state, covering a suitable number of time steps, to maintain smooth and efficient motion of the executor between the two interactions. The most direct approach would be to employ modelbased reinforcement learning (MBRL) methods with multi-step decision-making capabilities, such as Dreamer [Hafner et al., 2023] and TD-MPC [Hansen et al., 2022] . Regrettably, the form of multi-step decision makes the error of the dynamic model accumulate and then make the policies suboptimal. In addition, the high demand for high-quality data and the complexity of constructing dynamic models makes MBRL deployment costly in real-world scenarios [Janner et al., 2019] (detailed experimental analysis in Appendix C.1). Instead, we sought to unlock the multi-step decision-making ability of the model-free DRL approach, e.g. TD3 [Fujimoto et al., 2018] and PPO [Schulman et al., 2017] .The most simple method for model-free DRL to alleviate the intermittent interaction issue is using frameskip (also commonly known as 'action-repeat') [Kalyanakrishnan et al., 2021] , where the same action (usually the last action) is repeated during a fixed interval which is often used in Atari [Braylan et al., 2015] . However, longer frameskip will lead to mechanized motion, making it impossible for the agent to change actions at key states and thus resulting in suboptimal policies. Another way is to let the RL algorithms make up-front decisions (advance decisions) for future steps according to the current state or the received delayed state. Only actions for the correct time steps will be executed. Compared to frameskip, these methods can improve action diversity. However, directly increasing the horizon of decision making will lead to action space explosion and increase the difficulty of policy optimization [Chen et al., 2021] .In this paper, we propose Multi-step Action RepreSentation (MARS), which is the first plugin method for DRL algorithms to solve intermittent control tasks, significantly reducing the difficulty of multi-step policy training while ensuring the flexibility and diversity. The high level idea is shown in Figure 2 . MARS constructs a compact and decodable low dimensional latent action space for the original multi-step actions, the latent action dimension does not explode as the steps number increases. Based on this latent action space agent can learn a stable latent policy. The latent action selected by the policy can be reconstructed into original multi-step actions by the decoder. Specifically, MARS relies on a conditional Variational Auto-encoder (c-VAE) [Sohn et al., 2015] to construct the latent representation space for the associated multi-step actions. Intuitively, an efficient latent action space should have two characteristics: ❶ Decision space simplification: the combination number of long-sequence actions is huge, especially in the continuous action space, which can complicate the decision space and potentially result in suboptimal policies. We restrict the decision space to a relatively small subspace by introducing the concept of action transfer scale υ and taking it as the condition term of our VAE, i.e. using υ to implicit segment the latent space [Sohn et al., 2015] . ❷ Dynamic semantic smoothness, a characteristic of the original action space, which means latent actions that have similar impacts on the environment should be close together in latent spaces. Dynamic semantic smooth action space can retain the utility of adding Gaussian noise perturbations to the policy during training: While ensuring that the impacts of initial latent action chosen by the policy and latent action after disturbance on the environment is similar in high probability, some uncertainty is also retained that the disturbed action may explore rare actions that have unknown impacts on the environment, which helps maintain a balance between exploration and exploitation [Schwarzer et al., 2020] . We utilize unsupervised environmental dynamics to learn a predictive representation of dynamics for improving semantic smoothness. In practice, we adding an additional dynamic residual prediction module for VAE. Our contributions are summarized as follows: We propose the first plugin method and DRL framework for solving intermittent control tasks via multi-step action representation learning. We provide two modules to improve the effectiveness and smoothness of the learned multi-step latent space. Our method outperforms baselines on both realworld fixed interaction interval robotic control tasks and random interaction interval simulation control tasks.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Please refer to abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
background: 
 Markov Decision Process (MDP). A MDP can be represented as a tuple: (S, A, P, R, γ, T ), where S denotes the state set, A denotes an action set, P is the transition function: S × A × S → [0, 1] and R is the reward function: S × A → R. γ ∈ [0, 1) is a discount factor and T is the decision horizon.The goal is to optimize the agent's policy to maximize the expected discounted cumulative reward.Variational Auto-Encoder. The variational auto-encoder (VAE) is a directed graphical model with certain types of latent variables, such as Gaussian latent variables. A generative process of the VAE is as follows: a set of latent variable z is generated from the prior distribution p θ (z) and the data x is generated by the generative distribution p θ (x|z) conditioned on z : z ∼ p θ (z), x ∼ p θ (x|z). In general, parameter estimation of directed graphical models is often challenging due to intractable posterior inference. However, the parameters of the VAE can be estimated efficiently using the stochastic gradient variational Bayes (SGVB) framework, where the variational lower bound of the log-likelihood is used as a surrogate objective function. In this framework, a proposal distribution q θ (x|z), is introduced to approximate the true posterior p θ (x|z). MLPs are used to model the recognition and the generation models. Based on the Gaussian latent assumption, the first term of Eq.1 can be marginalized. The second term can be approximated by drawing samples z (l) (l = 1, ..., L) by the proposal distribution q θ (x|z). The empirical objective of the VAE is written as:EQUATION3 Intermittent Control Markov Decision Process (Intermittent-MDP)In this section, we introduce the Intermittent Control Markov Decision Process (Intermittent-MDP) to model both fixed and random interaction interval control problems. The objective of Intermittent-MDP is to train the policy to decide on an effective action sequence u based on the current state s, which should be longer than or equal to the maximum interaction interval c, thus ensuring smooth and efficient motion of executor between the two interactions.Definition of Intermittent-MDP. Intermittent-MDP (M) can be represented as a tuple: M ≜ ⟨S, U, R, K, µ 0 ⟩. S ∈ R n is the state space, U is the set of action sequence u that the policy can select. and µ 0 denotes the initial state distribution. The reward is defined as R, K is the multi-step transition function.Different from normal MDP in which the agent makes a decision a t according to the current state s t , intermittent-MDP requires the agent to generate an action sequence u t = {a t , ..., a t+c } according to the received state s t (a t is the single step action at timestep t). The cumulative rewards of executed actions can be received in the next interaction. Policy π takes current state to select action sequence u t = {a t , ..., a t+c }. When the interaction interval is present for j timesteps, the executor uses j single-step actions in the action sequence u to maintain the motion (j ∈ {0, 1, ..., c}). When the interaction is continuous, the executor uses the first single-step action in u each time (degenerated to the transition function P in normal MDP). Thus, the environment transition function K is defined as:EQUATIONIf the interval is random and we only know the maximum number of interval step c, Intermittent-MDP is represented as random Intermittent-MDP. When the interval time is fixed, Intermittent-MDP represents a fixed interaction interval problem, i.e., fixed Intermittent-MDP.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The limitations can be seen in Sec.6 of main paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We make sure to preserve anonymity.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
multi-step action representation: 
 In this section, we introduce Multi-step Action Representation (MARS), a novel framework that can map long action sequences into an efficient latent space. We hypothesize that multi-step actions reside on a homogeneous manifold. To learn the multi-step action representation, we employ VAE to constructing a compact and decodable latent representation space to approximate such a manifold.To align our latent action space with the two characteristics mentioned in Sec.1, i.e. ① Decision space simplification ② Dynamic semantic smoothness. In Sec.4.1, we introduce the concept of action transition scale and encode this as a conditional aid in VAE training to simplify the decision space of the policy. In Sec.4.2, we demonstrate how state residual prediction can be leveraged to smooth the semantics of dynamic transitions in latent action spaces, and in Sec.4.3, we illustrate the combination of MARS with DRL, as well as the extension of our approach to random intermittent-MDP.Although VAE has been proven to be able to construct a compact and effective latent action space for normal MDP with state s as the prior condition [Li et al., 2021] , due to the complex combinatorial nature of multi-step actions in Intermittent-MDP, the policy still struggles to explore the optimal actions in the vast action space. Therefore, to address this challenge, we introduce the concept of action transition scale υ as a conditional term in VAE, to constructs the multi-step latent action space Z. υ stands for a description of the motion style, it represents the accumulation of action change scales within each action sequence u. By determining υ, we can constrain the latent action z ∈ Z chosen by the policy within a related subspace [Sohn et al., 2015] in which all candidate action sequences have the same motion style in terms of υ, thus reducing the difficulty of exploration. We choose υ as a conditional term due to its task-specific nature. For instance, in robot scenarios, to ensure the balance of the robot and prevent joint damage, the scale of action changes is typically small. In some obstacle avoidance tasks, however, the magnitude of action changes can be significant when the agent encounters suddenly approaching obstacles. Once the policy roughly determines the appropriate action transfer scale for the current state, it can make more suitable decisions, making the policy more efficient and controllable in the complex action space. Notably υ does not need to be set manually but is selected adaptively by the policy through training, which is described in detail in Sec.4.3. We formulate υ as follows:υ ut = c-1 i=t ||a i+1 -a i || (c -1) × B .(3) c is the maximum interval and B denotes the upper limit of action change. The numerator part represents the total absolute difference between consecutive actions, used to assess the transition magnitude of u t . Eq.3 normalizes υ ut to [0, 1].Given an action sequence u t and the corresponding states s t:t+c , our encoder q ϕ (z ut |u t , s t:t+c , υ ut ) parameterized by ϕ takes s t:t+c and the action transition scale υ ut as conditions to build a multistep latent action space, and maps the action sequence u t into the latent variable z ut ∈ R d1 (d 1 denotes the dimension of z ut ). The decoder p ψ (û t |z ut , s t , υ ut ) parameterized by ψ then reconstructs the multi-step actions u t from z ut . We employ a Gaussian latent distribution N (µ z , σ z ) to model q ϕ (z ut |u t , s t:t+c , υ ut ) where µ z and σ z are the mean and standard deviation outputted by the encoder.The decoder decodes z ut ∼ N (µ zu t , σ zu t ) as following: ût = g ψ1 • p ψ0 (z ut , s t , υ ut ), g ψ1 is a fully-connected layer for reconstruction. • denotes cascade, i.e. the output of p ψ0 acts as the input of g ψ1 . We utilize cascaded heads because traditional parallel heads, employed for both reconstruction and state residual prediction, can interfere with optimizing individual objectives and impede the learning of the shared representation [Azabou et al., 2021] . p ψ0 denotes the shared layers of the decoder. ψ i∈{1,2} denote the parameters of the prediction networks. The loss function of our VAE is:EQUATION)where D is the buffer. The first term is the reconstruction loss (using MSE) of the action sequence, the last term is the Kullback Leibler divergence D KL between the variational posterior of latent representation z and the standard Gaussian prior. By using the reparameterization trick [Kingma and Welling, 2013] , ût is differentiable with respect to ψ and ϕ. For any latent variables z ut , it can be decoded into multi-step actions ût conveniently by the VAE decoder. That is, Encoder :zu t ∼ q ϕ (•|ut, st:t+c, υu t ) ∀ st:t+c, ut, υu t Decoder :ût = g ψ 1 • p ψ 0 (zu t ,In section 4.1, we introduce how to build a decodable representation space for multi-step actions. However, it is still inefficient to learn the policy and value functions in the latent action space learned by the above VAE.In comparison to the real-world action space, the current construction of our action space lacks a crucial attribute: dynamic semantic smoothness. This refers to the manifestation of differences in environmental impact through the Euclidean distance between points in the latent space, where closer points correspond to more similar impacts. To address this issue, we further apply an unsupervised learning loss based on environmental dynamics prediction to refine the multi-step action representation. Relevant proof can be found in [Whitney et al., 2019] . MARS captures the environmental dynamics by predicting the state transition residual. We use the state transition as a measure of environmental dynamic becuase: ❶ State transition does not require a per-step reward, it can be used in reward-agnostic pretraining. ❷ Building value equivalent models is more difficult since Q-values in the early stage of training are inaccurate [Wang et al., 2016] . In contrast, state transition is more reliable and accessible. ❸ The same reward or Q-value may correspond to different environmental dynamics, but the same environmental dynamics have the same reward or Q-value.Specifically, MARS predicts the residual difference between the state s t+c after the execution of u t and the current states t . As shown in the left of Figure 3 , h ψ2 is a subnetwork of the decoder. For any transition sample (s t , u t , s t+c ), we denote the state residual as δ st,st+1 = s t+c -s t and denote p state = h ψ2 • p ψ . The residual δst,st+c is predicted as:EQUATIONThe environmental transition prediction loss is defined as:EQUATIONAbove all, the ultimate training loss for the multi-step action representation is:EQUATION)where β is a hyper-parameter that controls the relative importance of the L dy and L V AE . L M ARS only depends on the environmental dynamic data which is reward-agnostic [Erraqabi et al., 2021 , Yarats et al., 2021] . During training, transitions stored in the buffer or offline dataset are utilized. Additionally, we observe that MARS exhibits insensitivity to data quality. Notably, data collected through random policies suffice for effective training the multi-step action representation.Algorithm 1 MARS-TD3Initialize actor π ω and critic networks Q θ1 , Q θ2 Initialize conditional VAE q ϕ , p ψ , buffer D.Stage 1: Build latent action space while not reaching warmup training times do Fill D with data generated by random policy or offline datasets Update q ϕ , p ψ using samples in D. ▷ Eq.8 end while Stage 2: Train the RL policy while t < policy training time do z t , υ t = π ω (with Gaussian noise)u t = g ψ1 • p ψ0 (z t , s, υ t ) Execute u t , observe r and new state s ′ Fill D with (s, u t , z t , υ t , r, s ′ ) update Q θ1 , Q θ2 , π ω ▷ Eq.9, Eq.10 end whileAs a plug-in method, MARS can be applied to any RL algorithm. It contains two types of actions: ❶ the encoded multi-step action z and ❷ the action transition scale υ. RL algorithms maximize the expected cumulative reward by selecting the optimal z t and υ t at s t . In this section, we apply MARS to a typical model-free RL method TD3 [Fujimoto et al., 2018] as an example. TD3 is a deterministic Actor-Critic algorithm. As illustrated in the right part of Figure 3 , with the learned transition-aware multi-step action representation, the actor network learns a latent policy π ω that outputs the latent actions according to current state s, i.e., [z t , υ t ] = π ω (s). z t and υ t respectively represent the action selected at time t from Z and υ constructed in the section 4.1. Then we obtain the corresponding multistep actions u t by decoding the latent action z t and υ t :u t = g ψ1 • p ψ0 (s, z t , υ t ).Two critic networks Q θ1 , Q θ2 are utilized which take the latent actions z t and υ t as inputs to approximate the value function Q πω more accurately. We train the critic network using the transition data (s, υ t , z t , r, s ′ ) sampled from the experience replay. To simplify notations, in this subsection, s is the current state. s ′ is the state perceived at the next interaction interval. The critic loss function is:EQUATIONWhere y = r +γ minj=1,2 Q θj (s ′ , π ω (s ′ )), ω denotes the target network parameters. The actor is updated according to the Deterministic Policy Gradient [Silver et al., 2014] :EQUATION)The overall algorithm MARS-TD3 is summarized in Algorithm 1, which contains two major stages: ① the warmup stage and ② the policy learning stage. In stage 1, MARS is trained using a prepared replay buffer D. The SC-VAE is updated by minimizing the VAE and the environmental dynamic prediction loss. Note that the proposed algorithm has no requirement on how the buffer D is prepared and here we simply use a random policy to gather the data. In stage 2, given an environment state, the latent policy outputs the latent action z t and the action transition scale υ t perturbed by the Gaussian exploration noise. Then, the latent action is decoded into the original multi-step actions by the decoder so as to interact with the environment. The newly collected transition sample is stored in the replay buffer D. After that, the latent policy is updated using the data sampled from D. The action representation model is also updated periodically in the second stage to make continual adjustments to the change of data distribution. The detailed network architectures are presented in appendix B.1.As for random interaction interval tasks, the interaction interval cannot be predicted, so we let the agent output an action sequence with the maximum length. To improve the training stability, we precisely record the absolute time step and the execution flag of each action, which makes the actions be executed in strict accordance with the time step order. When a new action sequence arrives at the action executor, the previous action sequence will be replaced, and the execution flag of the unexecuted actions will be set to False. The subsequent rewards will be attributed to the actions executed in the new action sequence. Thus, the actual reward stored in the experience replay D for each latent action is the sum of the executed action reward in the corresponding sequence. 
experiment: 
 We empirically evaluate MARS to answer the following research questions. RQ1: Performance in random interaction interval tasks. Can MARS significantly improve the performance in random Intermittent-MDP tasks, such as simulated remote NPC control tasks? RQ2: Performance in fixed interaction interval tasks. Can MARS significantly improve the performance in fixed Intermittent-MDP problems, such as real-world robot arm grasping tasks? RQ3: Generalization. Can MARS be seamlessly integrated into existing RL algorithms and improve their performance? RQ4: Ablation study. Do both the action transition scale and the state dynamic prediction contribute to optimizing the multi-step latent action space? How is the robustness of MARS?5.1 Random Intermittent-MDP Tasks (RQ1)Benchmarks. We select two types of control tasks to simulate the remote NPC control problem with random interaction intervals: (1) robot control and (2) navigation tasks. For robot control tasks, we select four typical openai Mujoco tasks with random interaction delays,i.e., Hopper, Ant, Walker, HalfCheetah. Mujoco is a well-known testbed and is widely used in reinforcement learning research [Brockman et al., 2016] . For navigation tasks, we used the medium and difficult maps of 2dmaze in D4RL [Fu et al., 2020] , where the agent's goal is to walk to the end of the maze under unstable interactions. Both types of tasks are modified to incorporate random interaction intervals, mimicking the real-world remote control scenarios [Chen and Wu, 2015] . In RTS games [Andersen et al., 2018] , the maximum interaction interval usually spans 0.5s to 0.7s, with the action execution time typically between 0.1s and 0.25s. Accordingly, we set the maximum interval to 10 time steps, requiring the policy to generate an effective action sequence a t:t+9 based on the received state s t .Baselines. To our knowledge, no specific solution exists for the Intermittent-MDP problem. Thus, we compare with three baseline methods.(1) Perfect-TD3: We train TD3 in the perfect environments with continuous interaction without interaction interval, i.e., normal MDP. It is the highest standard for evaluating MARS.(2) Frameskip-TD3: We combine the frameskip technique with TD3 (a common trick for multi-step decision) and apply it to the intermittent-MDP tasks.(3) Multistep-TD3:We modify TD3 to directly make decisions for the c future steps by outputting a concatenated action vector of c times of dimensionality.To mitigate implementation bias and ensure a comprehensive comparison, we implement all methods using the same architecture based on TD3 [Fujimoto et al., 2018] . For all tasks, we set the dimension of z t to 8 and the scaling parameter β to 5. We set the warm-up (stage 1) step to 400000 and 100000 for the Mujoco tasks and the navigation task respectively. Detailed parameter setting can be found in appendix B.2. The results in Figure 4 show that MARS-TD3 outperforms the other methods in all tasks, especially in the high-dimensional action control tasks (i.e., Mujoco). Compared with vanilla multstep-TD3, our method can significantly improve the performance of the RL algorithm in random Intermittent-MDP control problems by learning a more compact multistep action representation while avoiding the convergence difficulties caused by dimensional explosion. Note that MARS-TD3 can also achieve comparable performance with perfect-TD3 in most tasks. Step 1:Initializes the manipulator to the specified positionStep 2: moves to the side for observationStep 3: Select actions based on observationStep 4: Move the object to the boxStep 2: Move to the designated location for observation Task description.The experiment involves a 6-DoF robot arm performing grasping tasks within a 30 × 30 × 30cm 3 tabletop workspace. 15 rounds of experiments are conducted using our method and baselines. In each round, 6 objects are randomly selected and placed on the table. The robot observes the workspace with a single-depth image from a fixed side view. The viewpoint of the virtual camera points toward the workspace's origin at a radial distance r = 2l and an angle θ = π/3, l is the workspace's length. We follow the setting of traditional grasp tasks [Jiang et al., 2021] and set the total number of interactions to 6, meaning that the policy is expected to complete the grasp of all six targets in six intervals. Thus, the robot arm is allotted a single observation of the environment per object. As shown in Figure 5 , at each interaction interval, there are 4 steps to grasp an object. The robotic arm has to first acquire observations at the beginning of each grasping. Upon perceiving the observation, an action sequence a t:t+15 is executed to grasp the target object and then place the object in the target box. The time limit for grasping one object is set to 30s. The robot need to move to a predefined location for capturing image (images are transformed into a 512-dim vector). Action space is 3-dim2 , the reward function encourages policies to maximize the grasping success rate and motion smoothness.Baselines. Online RL demands extensive exploration, resulting in low sample efficiency. Besides, Online RL accidental random exploration may lead to robot arm damage. Thus, we opt for offline RL commonly utilized in robot scenarios, i.e. TD3+Behavior Cloning (TD3-BC) [Fujimoto and Gu, 2021] , to serve as the backbone. Methods: (1) Vanilla TD3-BC.(2) Multistep TD3-BC, same as Multistep-TD3 used in the section 5.1 (3) Dense observation + TD3-BC. An extra camera provides dense observation every 3 seconds, and the robot arm still has to move to the observing position at the start of each episode to eliminate differences caused by the observing motion.Evaluation metrics.(1) Grasp success rate (GSR), the ratio of successfully grasping the objects.(2) Declutter rate (DR), the average ratio of objects removed after successful grasping.(3) Motion smooth ratio (MSR): grasp time total time , evaluating whether the motion is smooth and natural (pauses and shaking can result in lower scores). All results are averaged over 15 simulation rounds. The overall comparison is presented in Table 1 .The results show that due to the sparse interaction, the vanilla TD3-BC fails to complete the task and frequently pauses or takes unnatural motions during the execution. While the performance of TD3-BC is significantly improved with additional dense observations, it remains unsatisfactory due to the lack of real-time feedback. Moreover, the use of extra cameras results in higher costs of the experiment. Although Multistep + TD3-BC helps smoothen the motion, it fails to effectively mitigate the instability caused by Intermittent-MDP. MARS addresses the limitation of the vanilla Multistep + TD3-BC, which makes TD3-BC complete the task with high quality. We provide the corresponding motion pattern for each method. To further verify the effectiveness of MARS combined with online RL on fixed interaction interval control task. The results in the Appendix C.2 show that our method outperforms the baselines on all four tasks.We further test MARS with popular RL algorithms on three random interaction interval tasks: Hopper, Walker, and Maze hard. We maintain consistent parameters for each method and implement them based on the publicly available codebase. We train each RL algorithm with dense interactions as baselines and compare them with MARS-enhanced methods under the random interaction interval setting. Results in Appendix C.3 shows that all methods can learn effective policies with the help of MARS and the converged performance is comparable to the perfect interaction settings.  
conclusion and limitation: 
 In this paper, we observe that previous DRL methods fail to learn effective policies in intermittent control scenarios because of the discontinue interaction. To improve the performance of DRL on such tasks, we propose Multi-step action representation (MARS) to construct a reliable multi-step latent action space. Based on this latent action space, DRL methods can make effective advance decisions to ensure the smoothness and efficiency of the agent's motion when it cannot interact with the environment. MARS outperforms baselines in a variety of fixed and random interaction intervals control tasks. Additionally, MARS has potential for improvement in represent extremely long action sequences, which we will address by identifying more powerful representation models, e.g. transformer based VAE, in the future. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This paper is only about promoting technological innovation and does not have social impactGuidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Ours paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original paper that produced the code package, and state which version of the asset is used (include a URL).Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: This paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Our paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Markov Decision Process (MDP). A standard MDP can be represented as a tuple: (S, A, P, R, γ, T ), where S denotes the state set, A denotes an action set, P is the transition function: S ×A×S → [0, 1] and R is the reward function: S × A → R. γ ∈ [0, 1) is a discount factor and T is the decision horizon. The target of the agent is to optimize its policy to maximize the expected discounted cumulative reward.Frameskip In general, parameter estimation of directed graphical models is often challenging due to intractable posterior inference. However, the parameters of the VAE can be estimated efficiently in the stochastic gradient variational Bayes (SGVB) framework, where the variational lower bound of the log-likelihood is used as a surrogate objective function. In this framework, a proposal distribution q θ (x|z), which is also known as a "recognition" model, is introduced to approximate the true posterior p θ (x|z). The multilayer perceptrons (MLPs) are used to model the recognition and the generation models. Assuming Gaussian latent variables, the first term of Equation A can be marginalized, while the second term is not. Instead, the second term can be approximated by drawing samples z (l) (l = 1, ..., L) by the recognition distribution q θ (x|z), and the empirical objective of the VAE with Gaussian latent variables is written as follows:EQUATIONB Experimental Details 256 and 256 hidden units with ReLU activation (except for the output layer) for the actor network for all algorithms. For DDPG the critic denotes the Q-network. For PPO, the critic denotes the V-network. All algorithms (TD3, DDPG, PPO) output two heads at the last layer of the actor network, one for latent action and another for dynamic transition potential.The structure of MARS is shown in Tab.3. We use element-wise product operation [Mahmood et al., 2018] and cascaded head structure [Fuchs et al., 2021] to our model [Ma et al., 2024 [Ma et al., , 2022]] .For all experiments, we use the raw state and reward from the environment, and no normalization or scaling is used. No regularization is used for the actor and the critic in all algorithms. An exploration noise sampled from N(0, 0.1) [Dong et al., 2009] is added to all baseline methods when selecting an action. The discounted factor is 0.99 and we use Adam Optimizer [Li et al., 2016] for all algorithms. Tab.4 shows the common hyperparameters of algorithms used in all our experiments. For PPO, the actor network and the critic network are updated every 2 and 10 episode respectively for all environments. The clip range of the PPO algorithm is set to 0.2 and we use GAE [Sutton and Barto, 2018] for a stable policy gradient. For DDPG, the actor network and the critic network is updated at every 1 environment step. For TD3, the critic network is updated every 1 environment step and the actor network is updated every 2 environment steps.The default latent action dim is 8, we set the KL weight in representation loss L M ARS as 0.5. Environment dynamic prediction loss weight β is 5 (default). The results in Table 5 show that the model-based reinforcement learning approach is significantly lower than our approach in all four scenarios, and even slightly worse than using TD3 for direct multi-step decision making. We analyze that this is because the errors caused by the mismatch (sub-optimal) of the dynamic model will accumulate due to multi-step decision-making, resulting in sub-optimal policy. Unfortunately, the training of dynamic models is data hungry (high cost), that is, a large amount of high-quality expert data is required to ensure the accuracy of the model shop, which is difficult to obtain, especially in real-world scenarios. We use the mainstream online reinforcement learning algorithm TD3 in combination with MARS and compare it with the baseline mentioned in Sec.5.1 in four tasks. We set the interval to 10 time steps, requiring the policy to generate an effective action sequence a t:t+9 based on the received state s t .For all tasks, we set the dimension of z t to 12 and the scaling parameter β to 4. We set the warm-up (stage 1) step to 300000 and 100000 for the Mujoco tasks and the navigation task respectively. The results in Figure 7 show that MARS-TD3 outperforms the other baselines in all fixed Intermittent-MDP tasks and achieve comparable performance with perfect-TD3 in most tasks. This further proves that MARS can effectively improve the effectiveness of Online DRL on fixed Intermi tasks.We test MARS with popular RL methods on three tasks: Hopper, Walker, and hardMaze. To make the experiment fair, we used the same parameters for all methods and implemented them based on public code. We use each RL algorithm to train on three tasks under the ideal setting and compare them with their corresponding improvement methods. To show the optimal score after the algorithm convergence, we train all the algorithm's 2000000 time steps. The results in Tab.6 show that all methods can learn effective policies with the help of MARS and perform similarly to their ideal settings. The differences in scores are mainly due to the variation in performance of the RL algorithms. In summary, MARS can be combined with different methods to provide a reliable action space for solving Intermittent-MDP as normal MDP with RL. The parameters of all methods are optimized by grid search. The results of applying MARS to popular RL algorithms on three random interaction interval tasks. The maximum interaction interval is set to 8. Each data in the table is in the following format: MARS-RL score | the score difference compared to the perfect dense interaction baseline. ↓ denotes the score of MARS lower than the dense interaction baseline. ↑ denotes the score of MARS is higher. All scores are averaged over 5 runs. We conducted two experiments to show how well the two mechanisms of MARS work together. Although the results of randomized Intermittent-MDP and fixed Intermittent-MDP are slightly different, the same conclusion can be derived: The green curves in Figure 8 demonstrate that the representation model with increased action transition scale is much better than the original VAE. This means that dynamic transition potential can create an latent action space by explicitly modeling the dependence between multi-step actions. The blue curves also show that VAE with state dynamic prediction is better than the original VAE because it can represent action sequences that have similar environmental effects at close locations. Finally, the red curves show that the two mechanisms work well together in MARS, and combining them improves representation ability.To further demonstrate the effectiveness of MARS in diverse intermittent control scenarios. For fixed interaction control tasks, we uniformly set the forbidden interaction duration and conducted four experiments on Hopper. The results in Figure 9 (a) show that MARS can solve most tasks effectively and still guarantee good scores at long intervals, but the effectiveness of MARS decreases significantly when the interval is too long (which is not common in real-world scenarios). We believe that this is because VAE is unable to effectively characterize excessively long sequences, leading to the failure of multi-step action space modeling. In addition, to observe the sensitivity of MARS to interaction intervals on random Intermittent-MDP tasks, we uniformly set the forbidden interaction duration and conducted four experiments on Hopper. The results in Figure 9 (b) show that in random Intermittent-MDP scenarios, MARS performs well in both short and medium-interval scenarios. However, convergence changes slowly in the very long interval scenario, and the score is only half that of the medium interval task. Because MARS's representational capabilities are not perfect for modeling long action sequences for extremely longspaced tasks (even if this setting rarely occurs in real-world scenarios). Therefore, in the future, we hope to find more suitable representation models to overcome this problem.The representation space dimension of VAE is an important hyperparameter. If the latent space dimension is too low, a large amount of original data information will be lost, resulting in invalid representation space. On the contrary, when the latent space dimension is too large, the calculation amount of the model will be increased. To verify the sensitivity of MARS to latent space dimensions, we test it on two tasks with different original action dimensions. We set up four sets of latent space dimensions for fixed Intermittent-MDP Hopper (interaction interval time step: 8, original action dimension: 3, so the action sequence dimension to be modeled is 24). The learning curve in Figure 10 (a) shows that for raw data of such high dimensions, when the latent space dimension is set too low, the latent space information will be lost, resulting in the convergence failure of reinforcement learning policies. On the contrary, too high a latent space dimension increases the complexity of reinforcement learning policy exploration.In addition, we set up four comparison experiments on the 2dmaze task with a lower dimension of the original action sequence (interaction interval time step: 4, original action dimension: 2, so the action sequence dimension to be modeled is 8). The experimental results in Figure 10 (b) show that the suboptimal policy can be learned when the latent space dimension is low, because the original data dimension is low. So the low-dimensional latent space loses less information. The score increases as the latent space dimension increases. However, when the latent space dimension is too high, the score will drop significantly, which is because of the exploration difficulties brought by high-dimensional latent space.In this section, we conduct some additional experimental results for a further study of MARS from different perspectives: We provide the exact number of samples used in the warm-up stage (i.e., stage 1 in Algorithm 1 in each environment in Tab.7. The number of warm-up environment steps is about 5% ∼ 10% of the total environment steps in our original experiments. Moreover, we also conducted some experiments to further reduce the number of samples used in the warm-up stage (at most 80% off). See the colored results in Tab.7. MARS can achieve comparable performance with < 3% samples of the total environment steps.The number of warm-up environment steps is about 5% ∼ 10% of the total environment steps in our original experiments. The number of warmup environment steps can be further reduced by at most 80% off (thus leading to < 3% of the total environment steps) while the comparable performance of our algorithm remains. NeurIPS Paper ChecklistThe action denotes the coordinates expected to be reached by the hand, the defualt underlying planning algorithm generates the motion trajectory via the expected coordinates). 


Decision: The paper addresses an important practical problem (intermittent loss of communication in  control systems) in a principled and interesting way. All reviewers are favourable and the authors did a great job during the rebuttal process of allaying concerns.

Reviews: review: paper_summary: paper_summary: This paper addresses the issue of intermittent control problems, common in real-world scenarios where interactions between decision-makers and executors are disrupted due to unstable communication channels. These disruptions lead to bidirectional blockages, preventing agents from acquiring state information and transmitting actions, thus reducing the efficiency of reinforcement learning (RL) policies. The paper models this problem as an Intermittent Control Markov Decision Process and proposes a solution called Multi-step Action Representation (MARS). MARS encodes a sequence of actions into a compact latent space, enabling RL methods to optimize smooth and efficient motion policies. The experiments demonstrate that MARS significantly enhances learning efficiency and performance in both simulation and real-world robotic tasks compared to existing baselines.
strength_weakness: strength_weakness: Some presentations, especially on notation use, are unclear. See my questions. Section 4 is unnecessarily long and consists of a lot of redundant text.
questions: questions: - Line 181: What's the upper limit of action change?
- Eq 3 defines how the action transition scale is computed, but in Figure 3, why does the policy need to output the action transition scale?
- You denote the reconstruction layer as $g_{\psi)1}$. What does "1" in the subscript mean? This is confusing since you use subscript to denote timestep as well. Similarly, what's the purpose "2" of $h_{\psi_2}$?
- Line 245: It's confusing to say "choosing optimal z" since you just sample z from a policy.
- It seems relevant to https://arxiv.org/pdf/2304.13705. How do you compare with MARS and this work?
- All the experiments in Figure 4 are conducted in the same time interval. How does the performance difference change over time interval?
limitations: limitations: Yes, it's discussed.
paper_summary: paper_summary: The paper introduces Multi-step Action RepreSentation (MARS) to address intermittent control problems in reinforcement learning. Intermittent control refers to situations where the interaction between the decision maker and the executor is discontinuous due to interruptions or communication issues. MARS encodes a sequence of actions into a compact and decodable latent space, allowing RL algorithms to optimize and learn smooth and efficient motion policies. Experiments are conducted on both simulated and real-world tasks, demonstrating that MARS significantly improves learning efficiency and performance compared to existing baselines.
strength_weakness: strength_weakness: * The comparisons in the experiment are with some simple and intuitive baselines. While the authors mention that no specific solution exists for this problem, it would still be helpful to discuss related work in other research fields.
* The experiments include both simulated and real-world tasks, but overall the number of tasks is a bit limited. It would be beneficial to include more tasks to validate the effectiveness of MARS.

Minors:
* Figure 6 is not adequately described, e.g., what the task is and what Vanilla_VAE stands for.
* Stage 2 in Algorithm 1 should be revised according to Line 276: The action representation model is also updated periodically in the second stage to make continual adjustments to the change of data distribution.
* Eq. (3): $\sum_{i=t}^{c-1} \rightarrow \sum_{i=t}^{c+t-1}$.
* Line 230: $\delta_{s_t,s_{t+1}} \rightarrow \delta_{s_t,s_{t+c}}$
* Line 231: $p_{state}=h_{\psi_2}\circ p_\psi \rightarrow p_{state}=h_{\psi_2}\circ p_{\psi_0}$
* Line 222: becuase $\rightarrow$ because
questions: questions: * Why does the input of the encoder $q_\phi$ include $s_{t:t+c}$? According to the description of the intermittent control task (Line 31: agents are unable to acquire the state sent by the executor), isn't $s_{t:t+c}$ inaccessible to the agent?
* Which variable is the Gaussian exploration noise added to, the decoded action sequence $u_t$, the latent variable $z_t$, or $v_t$?
limitations: limitations: The authors have stated in the Conclusion and Limitation section that representing long action sequences is a limitation and future direction of this work. However, there are no results suggesting that MARS suffers from long action sequences. It would be helpful to discuss more about this limitation.
paper_summary: paper_summary: This paper proposes a new method for RL problems that needs to propose a sequence of actions at each step due to latency of the environment. The method is straightforward in that it uses VAE to encode the action space of a consecutive of actions. Empirical results show that this simple technique improves the performance and sample efficiency.
strength_weakness: strength_weakness: The setting looks like the delay MDPs but is simpler as the observation is not delayed. The paper does not compare with methods for delay MDPs. The method is straightforward and easy to think of.
questions: questions: 1. Why define intermitten MDP separately while delayed MDP is widely adopted in the community?
2. How does the method compare to existing delay MDP methods?
limitations: limitations: See weakness.

Comments: title: Score Raised
Supplement for comparison experiments between our method and the delayed MDP methods
Re-optimize the response to the weakness
Re-optimize and simplify our rebuttal
Rebuttal by Authors
Rebuttal by Authors
Rebuttal by Authors
Rebuttal by Authors
comment: Thanks for the detailed explanation and comparison. I suggest to add these experiments in the main paper in the next version.
We enhanced the comparison experiment between the MARS and delayed MDP methods by increasing the number of seeds from 4 to 8. Furthermore, two new baselines are added to enhance the richness of the experiment.  The experimental results show that the advantage of our method is further improved as the number of seeds doubles.

*Baselines:* We append two newest baselines to the three existing SOTA delayed MDP methods (DCAC, SA, delayed Dreamer): BPQL [1], the latest Actor-critic-based continuous control algorithm for delayed feedback environments. AD-RL [2],  a SOTA method that utilizes auxiliary tasks with short delays to accelerate RL with long delays.

*Benchmarks:* We chose six difficult tasks. DeepMind Control tasks: Dog Run, Dog Trot, Dog Stand and Humanoid Walk. Besides, two robotic arm control tasks in MetaWorld: Sweep Into and Coffee Push are used. The environmental parameters and network hyperparameters remained consistent with the main experiment.

*Metrics:* We evaluate methods in terms of performance and Smoothness (whether the motion is coherent and stable, invalid jitter and stagnation will reduce the score, please refer to Sec.5.2 for details).

Table 1. Performance in newly added difficult tasks (smoothness (%)):
| Method      |Dog Run|Dog Trot| Dog Stand| Humanoid Walk|Sweep Into|Coffee Push|
| :-----------: | :-----------: | :------------: | :-----------: | :-----------: | :-----------: | :-----------: |
| **Ours**|$124.61\pm 24.71 (78)$|$592.42\pm 28.76 (88)$|$626.11\pm 18.36 (76)$|$120.82\pm 26.45 (85)$|$0.53\pm 0.04 (74)$|$0.42\pm 0.05(69)$|
| DCAC|$91.48\pm 16.75 (55)$|$411.06\pm 36.52 (76)$|$538.47\pm 22.73 (64)$|$101.63\pm 15.28 (46)$|$0.42\pm 0.08 (48)$|$0.23\pm 0.11 (51)$|
| BPQL|$95.37\pm 20.31 (62)$|$451.72\pm 36.67 (78)$|$526.13\pm 17.92 (66)$|$92.84\pm 22.51 (61)$|$0.47\pm 0.11 (53)$|$0.27\pm 0.05 (57)$|
| AD-RL|$88.26\pm 14.03 (64)$|$448.49\pm 24.72 (72)$|$471.82\pm 21.17 (51)$|$105.32\pm 15.25 (57)$|$0.39\pm 0.08 (45)$|$0.23\pm 0.07 (41)$|
| SA |$93.26\pm 28.14 (33)$|$384.26\pm 45.03 (42)$|$486.91\pm 10.51 (32)$|$78.21\pm 27.41 (47)$|$0.49\pm 0.07 (42)$|$0.33\pm 0.06 (34)$|
| delayed Dreamer |$95.28\pm 19.42 (46)$|$416.65\pm 24.18 (45)$|$526.07\pm 11.52 (38)$|$92.07\pm 13.59 (43)$|$0.48\pm 0.08 (36)$|$0.36\pm 0.03 (46)$|

Table 1 shows that our method performs better than Delayed MDP methods in almost all intermitted MDP control tasks while ensuring the smooth and coherent motion of the agent.  Besides, the advantage of our method is further improved as the number of seeds doubles.

[1]: Kim, Jangwon, et al. "Belief projection-based reinforcement learning for environments with delayed feedback."NuerIPS 2023.

[2]: Wu, Qingyuan, et al. "Boosting Long-Delayed Reinforcement Learning with Auxiliary Short-Delayed Task." ICML 2024.
Thanks for the clarification. I have no further questions and increased my rating.
I thank the authors for their detailed rebuttal. The additional experiments enrich and enhance the empirical results, and it would be beneficial to include them in the paper. I have updated my score accordingly.
Dear reviewer 5oZS, we sincerely apologize for the inconvenience caused by placing our important experimental results in the general PDF, which may have required additional time for you to locate them. To address this, We have strategically placed the experiment that should be added to the PDF onto the webpage (integrated with responses to each question) to enhance the readability. 

If you have any further questions or suggestions, please feel free to share them with us. Your feedback is invaluable and catalyzes enhancing our work.
Dear reviewer Bn1i, we sincerely apologize for the inconvenience caused by placing our important experimental results in the general PDF, which may have required additional time for you to locate them. To address this, We have strategically placed the experiment that should be added to the PDF onto the webpage (integrated with responses to each question) to enhance the readability.

If you have any further questions or suggestions, please feel free to share them with us. Your feedback is invaluable and catalyzes enhancing our work.
**Show your detailed plan of revision**

- In lines 164-180 of section 4.1 (Scale-conditioned Multi-step Action Encoding and Decoding), we condensed the repetitive introduction of the action transition scale. This streamlining is particularly evident in lines 173-178, where we illustrate with the example of robot motion, 
as we have already covered this concept in the Introduction.
- Section 4.3 (DRL with Multi-step Action Representation) is redundant. We streamlined the redundant training process explanation (lines 267-277) since the preceding two subsections are described in detail and a pseudo-code is used to assist readers in comprehension.

With the above reduction, the overall space of Section 4 is reduced by 3/4 pages, which just allows us to incorporate the newly added main experiment into the main text.

*If you have any further questions or suggestions, please feel free to share them with us. Your feedback is invaluable and catalyzes enhancing our work.*
We apologize for using the comment button to rebuttal and the long reply due to an incorrect understanding of the rule (moving the pdf supplement to comment). In the future, we commit to rigorously adhering to the submission order.

**Q1.1: A more convincing comparison is needed.** 

For this iteration, we raised the seed number to 8 and introduced six additional famous difficult scenarios (4 DeepMind Control scenarios: Dog Run, Dog Stand, Dog Trot, Humanoid Walk; 2 metaworld scenarios: Sweep Into, Coffee Push). interval step is $14$, mean advantage ratio: (Ours-ACT)/ACT

The findings presented in Table 1 indicate that doubling the number of seeds leads to a notable reduction in score variance for each method, resulting in improved and more stable performance. 
Our method stands out prominently in all scenarios except Dog Trot and Coffee Push. Our analysis suggests that the random intervals in our scenario may be a contributing factor that restricts the performance of ACT.

Table 1. Performance score of difficult tasks (Average of 8 runs):
| Method|ACT|Ours| Mean advantage ratio (%)|
| :-: | :-: | :-: | :-: |
| Dog Run|$74.93\pm 31.26$|$124.61\pm 24.71$|$65.64$|
| Dog Trot|$438.94\pm 36.27$|$574.12\pm 28.76$|$30.80$|
| Dog Stand|$561.68\pm 11.34$|$626.11\pm 18.36$|$11.47$|
| Humanoid Walk|$82.15\pm 19.34$|$108.82\pm 26.45$|$32.47$|
|Coffee Push|$0.44\pm 0.07$|$0.53\pm 0.04$|$20.45$|
|Sweep Into|$0.27\pm 0.03$|$0.39\pm 0.06$|$44.45$|

**Q1.2: discuss the "technical focus of the ACT"**

- Different from MARS, which is a lightweight auxiliary plug-in to improve the multi-step decision-making ability of the model-free RL methods, ACT directly constructs multi-step decision-making policy by imitation learning. 
- ACT uses Transformer-based Action Chunking and Temporal Ensemble **at the decision level** to ensure the stability of each step of the execution, i.e. obtains more accurate actions through the synthesis of $k$ decisions at the same time step.  In contrast, MARS focuses on developing lightweight plugins, utilizing MLP as the backbone, and incorporating action transition scale and state residual regularization terms to enhance the quality of action representation **at the VAE training level**.

**Q1.3: You can apply ACT model to RL setting.**

We successfully transitioned ACT into the RL setting by incorporating a momentum penalty into the standard reward and leveraging the Importance Experience Buffer.

Adapting ACT to the RL setting showed a slight improvement over the original ACT in most scenarios. However, there remains a significant gap between ACT and MARS. 

Table 2. Performance (Average of 8 runs):
| Method|Dog Run|Dog Trot| Dog Stand| Humanoid Walk|Sweep Into|Coffee Push|
| :-: | :-: | :-: | :-: | :-: | :-: | :-: |
| **MARS**|$124.61\pm 24.71$|$574.12\pm 28.76$|$626.11\pm 18.36$|$108.82\pm 26.45$|$0.53\pm 0.04$|$0.39\pm 0.06$|
| RL setting ACT |$68.93\pm 31.62$|$462.32\pm 21.08$|$541.73\pm 14.86$|$89.12\pm 23.21$|$0.46\pm 0.05$|$0.31\pm 0.06$|
| ACT |$74.93\pm 31.26$|$438.94\pm 36.27$|$561.68\pm 11.34$|$82.15\pm 19.34$|$0.44\pm 0.07$|$0.27\pm 0.03$|

**Q2**

We compare all methods under varied c settings on Walker2d-v2. 

Tab.3 shows that MARS exhibits minimal sensitivity to the length of the interval time step $c$, showcasing consistently outstanding performance. In the RL setting, ACT ranks second, but its performance decreases with longer interval steps. We aim to enhance ACT's online learning for extended intervals. original ACT struggles with stability in longer time steps. The other baselines are less effective, while frameskip shows better stability in longer interval scenarios.

Table 3. Performance  (Average of 8 runs):
| Method| Decision step $c=4$|$c=8$|$c=12$|$c=16$|
| :-: | :-: |:-: |:- |:-: |
| MARS |$5813\pm 126.52$|$5734\pm 203.05$|$5624\pm 115.31$|$5608\pm 142.67$|
| RL setting ACT |$4792\pm 206.15$|$5176\pm 163.29$|$4136\pm 227.91$|$4092\pm 322.84$|
| ACT |$4608.74\pm 148.92$|$4795.03\pm 163.28$|$3911.43\pm 215.53$|$3471.23\pm 186.31$|
|frameskip-TD3|$3235.79\pm 107.42$|$3378.47\pm 203.28$|$3755.14\pm 291.07$|$3165.62\pm 108.76$|
| multistep-TD3|$3766.31\pm 166.28$|$2904.01\pm 156.13$|$1007.43\pm 272.65$|$657.41\pm 75.85$|

**Q4**

RL policy requires the decoder of c-VAE to reconstruct the action sequence. During the reconstruction, the decoder must incorporate not only the latent action but also the suitable action transition scale (ATS) as a conditional term to ensure that the decoded action aligns with the current state.
To enable adaptive selection of ATS, which would be inefficient and costly to manually provide for each task or state, we leverage the adaptive decision-making capability of RL (following the hybrid control RL output style). This allows the policy to decide on actions within the latent space $z$ and allocate a separate decision head to select a number from ATS.

**Q6**

We use $h_\theta$ and $g_\mu$ to represent the two output heads in the updated version.
Please use the rebuttal button to respond. The rebuttal has a 6000-character limit, but the official comments do not. Using comments to respond is inappropriate since this increases the burden of reviewing and is unfair to the other authors. I will skim through the response since it's too long.

- Q1: From the comparison with ACT and MARS, I don't think you can draw a statistically significant conclusion that MARS is better than ACT since their mean scores are close and standard deviations are overlapped. 
  - Your response to "Focus" reads a bit weird. The first sentence talks about the goal of ACT paper. Yes, that's their goal, but I think in this rebuttal, you should discuss the "technical focus of the ACT model" instead of the goal of their paper. 
  - Training style: Got it.
  - Form of application & Technique: I think you can apply ACT model to RL setting. 

- Q2: If I remember correctly, c is the max time interval of an MDP. What not doing this experiment on the other baselines?

- Q3: Got it.

- Q4: Please shorten it.

- Q6: If you're talking about the output head in Figure 3, please consider the other subscript that represents the purpose of the output heads better.

- In the latest version, we streamlined the content of Section 4 to enhance its conciseness and clarity: I need to see your detailed plan of revision; otherwise, I don't think the next version will be concise.

I appreciate the author's additional experiments, but I will keep my rating since I still have lots of concern of letting this paper in (see my comments above).
## Part 3
### Weaknesses
**W1: The method is straightforward and easy to think of.**

We add a description of the novelty of our work. The Backbone of MARS is straightforward, a C-VAE, thereby enhancing convenience and plug-and-play capabilities from an application standpoint. However, at the practice level, we discovered that the original C-VAE could not effectively construct the semantic smooth latent space and the representation realization of long action sequences was poor, thereby constraining the optimization of the RL algorithm. Thus, the primary innovation of our method revolves around enhancing the capability of constructing the latent space of the original VAE through the development of more effective auxiliary techniques.

In particular, we introduce two innovative techniques for the original VAE: 1) the introduction of action transition scale (ATS) to dynamically restrict action decoding within an effective interval to ensure policy effectiveness (Sec 4.1), and 2) the incorporation of a state residual module (SR) to encourage points in the latent space with similar environmental impacts to be closer to each other (Sec 4.2), enhancing the overall model performance.

Experiments are conducted on Ant-v2 to assess the efficacy of the two key modules. The results in Tab. 7 indicate that RL optimization is more efficient with these modules. Additionally, Table 8 shows that the performance of MARS surpasses that of the original VAE in long-sequence decision scenarios.


Table 7. Performance (smoothness (%)) (Average of 3 runs):
| Method  |Ant-v2|
| :-: | :-: |
| VAE + ATS + SR (MARS) |$4354.61\pm 158.44$|
| VAE + SR |$4016.47\pm 213.06$|
| VAE + ATS |$4122.18\pm 65.37$|
| vanilla VAE|$3685.92\pm 362.72$|

Table 8. Performance in Walker2d  (Average of 4 runs):
| Method| Decision step $c=4$|$c=8$|$c=12$|$c=16$|
| :-: | :-: |:-: |:-: |:-: |
| VAE + ATS + SR (MARS) |$5309\pm 143.85$|$5283\pm 171.46$|$5309\pm 143.85$|$5194\pm 201.52$|
| vanilla VAE |$4623.51\pm 463.81$|$3941.21\pm 297.44$|$3806.14\pm 538.24$|$3612 .23\pm 635.42$|
## Part 2
**Q2: How does the method compare to existing delay MDP methods?**

Thanks for your suggestion. we add a comparison of our method with mainstream delayed MDP methods on multiple complex tasks from the perspective of multiple metrics in the new version.

*Baselines:* We select two recent SOTA methods in the Delayed MDP domain, i.e. DCAC [2] and State Augmentation (SA) [3] (Relax the Intermitted setting restrictions, allow such methods to additionally use dense rewards and delay priors at each step, and set the delay coefficient to the interaction interval). For a more comprehensive analysis, a recent model-based approach,i.e. delayed Dreamer[5], is also chosen. 

*Benchmarks:* We further select four more challenging DeepMind Control tasks focused on bionic robot locomotion: Dog Run, Dog Trot, Dog Stand and Humanoid Walk. DMC tasks demand coordinated movement from multi-joint bionic robots. Besides, two robotic arm control tasks in MetaWorld: Sweep Into and Coffee Push are used. The environmental parameters and network hyperparameters remained consistent with the main experiment.

*Metrics:* We evaluate methods in terms of performance and Smoothness (whether the motion is coherent and stable, invalid jitter and stagnation will reduce the score, please refer to Sec.5.2 for details).

Table 5. Performance in newly added difficult tasks (smoothness (%)) (Average of 4 runs):
| Method      |Dog Run|Dog Trot| Dog Stand| Humanoid Walk|Sweep Into|Coffee Push|
| :-----------: | :-----------: | :------------: | :-----------: | :-----------: | :-----------: | :-----------: |
| **Ours**|$124.61\pm 44.92 (75)$|$574.12\pm 28.76 (88)$|$614.03\pm 17.42 (73)$|$105.47\pm 35.83 (82)$|$0.51\pm 0.13 (68)$|$0.38\pm 0.06(63)$|
| DCAC|$96.87\pm 28.44 (53)$|$426.93\pm 50.48 (72)$|$562.64\pm 22.73 (64)$|$105.32\pm 29.16 (49)$|$0.44\pm 0.27 (41)$|$0.19\pm 0.13 (48)$|
| SA |$92.74\pm 51.06 (37)$|$385.67\pm 52.49 (39)$|$503.94\pm 14.86 (27)$|$75.66\pm 31.42 (45)$|$0.52\pm 0.21 (37)$|$0.34\pm 0.09 (39)$|
| delayed Dreamer |$95.31\pm 26.74 (46)$|$428.39\pm 46.23 (46)$|$526.07\pm 21.84 (25)$|$89.25\pm 27.41 (41)$|$0.56\pm 0.17 (32)$|$0.39\pm 0.04 (37)$|

Table 6. Performance in old tasks (smoothness (%)) (Average of 4 runs):
| Method  |Ant-v2|Walker2d-v2| HalfCHeetah-v2| Hopper-v2|
| :-: | :-: | :-: | :-: | :-: | 
| **Ours**|$4354.61\pm 158.44$ (78)|$5436.42\pm 217.83$ (82)|$6175.63\pm 273.95$ (76)|$2613.58\pm 177.96$ (83)|
| DCAC |$3279.82\pm 127.83$ (42)|$4892.18\pm 383.07$ (54)|$5811.51\pm 108.33$ (58)|$2684 .31\pm 238.27$ (63)|
| SA |$3492.53\pm 131.95$ (51)|$2584.18\pm 106.24$ (59)|$3281.45\pm 139.42$ (66)|$381.74\pm 53.67$ (71)|
| delayed Dreamer|$2408.25\pm 31.76$ (35)|$1529.26\pm 68.21$ (63)|$112.73\pm 17.82$ (68)|$1173.93\pm 78.28$ (78)|

Table 5,6 shows that our method performs better than Delayed MDP methods in almost all intermitted MDP control tasks while ensuring smooth and coherent motion of the agent. 

*P.S. If you know other delayed MDP methods that you would like us to compare but are not in the scope of our investigation, please let us know and we would be happy to include them in our experimental analysis.*

[1]: Du, Keliang, et al. "Random-Delay-Corrected Deep Reinforcement Learning Framework for Real-World Online Closed-Loop Network Automation." Applied Sciences 2022.

[2]: Ramstedt, Simon, et al. "Reinforcement learning with random delays." ICLR 2020.

[3]: Nath, Somjit, et al. "Revisiting state augmentation methods for reinforcement learning with stochastic delays." CIKM 2021.

[4]: Jiang, Zhenyu, et al. "Synergies between affordance and geometry: 6-dof grasp detection via implicit representations." Transactions on Robotics 2021.

[5]: Karamzade A, Kim K, Kalsi M, et al. "Reinforcement learning from delayed observations via world models." PMLR 2024.

**The Third part of the response is in the following comment. We are grateful for your thorough and conscientious review.**
## Part 2
**Q2: Why does the input of the encoder include $s_{t:t+c}$? According to the description of the intermittent control task (Line 31: agents are unable to acquire the state sent by the executor), isn't $s_{t:t+c}$ inaccessible to the agent?**

This question is quite detailed, and we provide further explanation in the updated version. During the policy training stage, only the decoder of the VAE is used to collaborate with the agent's policy training, and $s_{t:t+c}$ is not required in this phase. The encoder is only used in the pre-training stage. 

During the pre-training phase, the construction of the action space is self-supervised, meaning it is independent of policy learning （VAE only needs sufficient information about the environment). Thus, $s_{t:t+c}$ utilized by the encoder does not represent expert data gathered by the RL policy; instead, it is a randomly sampled augmented dataset (abundant but of low quality, please refer to Section 4.1 for detail).

We set up a set of experiments in a random Intermitted MDP Mujoco scenario to compare the effectiveness of expert data (collected by ppo) and randomly generated data (random policy) for VAE training. Experimental results show that for a fixed action latent space, randomly sampled state transitions contain richer states (sufficient state transition in the environment can be sampled) and the constructed space is more effective.

Table 3. Performance (Average of 4 runs):
| Method      | Ant-v2| 
| :-----------: | :-----------: |
| MARS with expert data|$5473\pm 219.33$|
| **MARS with random data** |$5908\pm 194.61$|
### Limitations
**L1: The authors have stated in the Conclusion and Limitation section that representing long action sequences is a limitation and future direction of this work. However, there are no results suggesting that MARS suffers from long action sequences. It would be helpful to discuss more about this limitation.**

In the appendix of the new version, we tested MARS with varying interval step $c$ in the Walker2d environment. The other environmental parameters and network hyperparameters remained consistent with the main experiment.

Table 4. Performance (Average of 4 runs):
| Method      | c=4|c=8|c=12|c=16|c=20|c=24|
| :-----------: | :-----------: |:-----------: |:-----------: |:-----------: |:-----------: |:-----------: |
| MARS|$5309\pm 143.85$|$5283\pm 171.46$|$5309\pm 143.85$|$5194\pm 201.52$|$4758\pm 106.73$|$4514\pm 377.25$|

The Above results reveal a diminishing performance of MARS as the decision step size increases beyond $c=20$. We attribute this trend to the limitations in representation capacity imposed by the MLP architecture in the VAE. In the future, we plan to investigate alternative effective networks like Transformers to enhance the construction capabilities of MARS within action spaces under very long interval step setting.
## Part 2
**Q4:Eq 3 defines how the action transition scale is computed, but in Figure 3, why does the policy need to output the action transition scale?.** 

This is an in-depth problem, and we emphasize it in the new version. In the second stage, the decoder is employed to decode the latent action selected by the policy, with the action transition scale (ATS) functioning as a condition term that dynamically adjusts to guide the decoder's generation process. This condition term helps constrain the latent variable within a smaller subspace to rectify any erroneous decisions made by the policy. Hence, ATS can be regarded as another decision space akin to the action latent space $z$. However, the ATS is not constructed through deep learning but according to our formulated approach in Sec. 4.1. To enable adaptive selection of ATS, which would be inefficient and costly to manually provide for each task or state, we leverage the adaptive decision-making capability of RL. This allows the policy to decide on actions within the latent space $z$ and allocate a separate decision head to select a number from ATS.

This output structure has been extensively validated in the area of hybrid action space control [1][2]. In the new version, we have included ablation experiments on Ant-v2 to demonstrate the viability of entrusting the ATS selection of RL policy.

The baseline method involves selecting appropriate ATS through pre-defined manual scripts without altering other modules. Results indicate that the RL policy can identify the suitable ATS and achieve commendable performance after a certain exploration step.

Table 4. Performance (Average of 3 runs):
| Method      |  training step = 50k|training step = 1m|training step = 2m|
| :-----------: | :-----------: | :-----------: | :-----------: | 
| Ours |$1962\pm 421.42$|$3243\pm 109.25$|$4183\pm 171.46$|
| Baseline |$2213\pm 311.97$|$3126\pm 128.36$|$4207\pm 136.61$|

**Q5: Line 245: It's confusing to say "choosing optimal z" since you just sample z from a policy.** 

Thanks for pointing out our imprecise presentation. In the new version, "optimal" is removed.

**Q6: You denote the reconstruction layer as $g_{\phi_1}$. What does "1" in the subscript mean? This is confusing since you use subscript to denote timestep as well. Similarly, what's the purpose "2" of $h_{\phi_2}$?** 

We apologize for any confusion caused by our presentation. We intend to use the number $i=\{1,2\}$ to represent the $i_{th}$ parallel output head (reconstruction layer) connected after the decoder. Redundant symbols are removed in the new version.
### Weakness
**W1: Section 4 is unnecessarily long and consists of a lot of redundant text.** 

 In the latest version, we streamlined the content of Section 4 to enhance its conciseness and clarity.

[1]: Li, Boyan, et al. "Hyar: Addressing discrete-continuous action reinforcement learning via hybrid action representation." ICLR 2022.

[2]: Fan, Zhou, et al. "Hybrid actor-critic reinforcement learning in parameterized action space." IJCAI 2019.