Unlock the Intermittent Control Ability of Model Free Reinforcement Learning
Authors: Jiashun Liu
Jianye HAO
Xiaotian Hao
Yi Ma
YAN ZHENG
Yujing Hu
Tangjie Lv
Venue: NeurIPS 2024 poster (2024)
Keywords: Deep Reinforcement Learning ; Representation Learning; Intermittent Control
Primary area: reinforcement_learning
TL;DR: We observe that previous DRL methods fail to learn effective policies in intermittent control scenarios because of the discontinue interaction and propose a plugin method for DRL to address  such problems.

Abstract: Intermittent control problems are common in real world. The interactions between the decision maker and the executor can be discontinuous (intermittent) due to various types of interruptions, e.g. unstable communication channel. Due to intermittent interaction, agents are unable to acquire the state sent by the executor and cannot transmit actions to the executor within a period of time step, i.e. bidirectional blockage, which may lead to inefficiencies of reinforcement learning policies and prevent the executors from completing the task. Such problem is not well studied in the RL community. In this paper, we model Intermittent control problem as an Intermittent Control Markov Decision Process, i.e agents are expected to generate action sequences corresponding to the unavailable states and transmit them before disabling interactions to ensure the smooth and effective motion of executors. However, directly generating multiple future actions in the original action space has unnatural motion issue and exploration difficulty. We propose **M**ulti-step **A**ction **R**epre**S**entation (**MARS**), which encodes a sequence of actions from the original action space to a compact and decodable latent space. Then based on the latent action sequence representation, the mainstream RL methods can be easily optimized to learn a smooth and efficient motion policy. Extensive experiments on simulation tasks and real-world robotic grasping tasks show that MARS significantly improves the learning efficiency and final performances compared with existing baselines.

Unlock the Intermittent Control Ability of Model Free Reinforcement Learning
Jiashun Liu, Jianye Hao'& Xiaotian Hao College of Intelligence and Computing Tianjin University China
Yi Ma
School of Computer and Information Technology Shanxi University China
YAN ZHENG
Yujing Hu & Tangji Lv FUXI Al Laboratory NetEase
Tianjin University China yanzheng@tju.c edu. cn
China
Abstract
Intermittent control problems are common in real world. The interactions between the decision maker and the executor can be discontinuous (intermittent) due various types of interruptions, e.g. unstable communication channel. Due to inter- mittent interaction, agents are unable to acquire the state sent by the executor and cannot transmit actions to the executor within period of time step. i.e. bidirec- tional blockage, which may lead inefficiencies reinforcement learning policies and prevent the executors from completing the task. Such problem is not well studied in the RI community. this paper, we model Intermittent Control Problem an Intermittent Control Markov Decision Process, i.e., agents are expected to generate action sequences corresponding them before disabling interactions to ensure the smooth and effective motion of executors. However, directly generating multiple future actions in the original action space has unnatural motion issue and exploration difficulty. We propose Multi-step Action RepreSentation (MARS), which encodes a sequence of actions from the original action space compact and decodable latent space. Then based on the latent action sequence representation, the mainstream RL methods can be easily optimized learn smooth and efficient motion policy. Extensive experi- ments on simulation tasks and real-world robotic grasping tasks show that MARS significantly improves the learning efficiency and final performances compared with existing baselines.
the unavailable states and transmit
1 Introduction
In recent years, the field of deep reinforcement learning (DRL) has witnessed striking empirical achievements in a variety of Markov Decision Process (MDP) problems [Mnih et al. 2013 Kaufmann et al.] 2023) and has been successfully applied to many real-time control tasks [Mahmood et al. 2018 Lee et al. 2020). In real-time control, "interaction" plays a crucial role. At each time step, the decision maker obtains observations from the environment and feeds actions back to the action executor through real-time interactions. Thus, an ideal MDP setting, the interaction should be continuous. However, in many scenarios, interactions become intermittent due to limitations of realistic conditions (Jiang
al. 2021 or communication interruption (e.g., unstable network) [Dong et al.) 2009). Due to
Cooresponneecce to: Jianye Hao <jianye.hao @ @tw..edu.cn.>>>
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

bidirectional communication blockage caused by intermittent interaction, agents (decision makers) are unable to acquire the state sent by the executor and cannot transmit actions to the executor within a period of time step (as shown in F (top). The sparse observations resulting from this phenomenon can make normal step-by-step decisions unstable. Thus, directly deploying existing DRI. algorithms could make the action executor abruptly stop (can not make decision when the current state is unavailable) or move abnormally when the environment suddenly changes [Sutton and Barto 2018). Typically, there are two types of intermittent interactions. Illustrations are shown in Fig- ure [bottom). O Fixed interaction interval: in numerous real-world robot scenarios, in-
Execution stagnation
Executor End
St
12
S113
Sr44
S
Sr.
aq
Interaction failure
teracting with the environment is often time-
Sra11
Sraz
Sra3
S
S116
consuming and costly (Liu et al. 2020]. For ex-
CCannot get current state/
Agent End
ample, before the robotic arm performs its next actions, it has to first halt its current operations, move to designated position to capture image, and then use a specific technique to ex- tract features and provide them to the decision maker. To ensure reliable operation and stable movement, we usually set a fixed interaction interval for the robot arm (Jing et al. [202 Bonarini 2020 @ Random interaction in- terval: unexpected interaction intervals may occur due to unstable communication channels and loss of wireless signals [Li et al.) [016]. For example, in real-time strategy games, the decision end remotely controls the terminal non-player characters (NPC) [Zheng et al. 2019], the interaction interval between the decision end and NPC terminals may become random [Wong et al. 2021] due to above reasons, which may cause the NPC to be stuck and disconnected from the changed environment, reducing the player's experience.
Intermitted Control Problem
Ideal smooth policy
Traditional polcy
Execution
Execution
Execu deree
Stop & vralll
Maximize Interval
Excraale
Interval
Time
Time
Constant interaction interval Random interaction interval Figure I: Detail of Intermitted Control (top part). Two types intermittent interactions (bottom part). For both cases, we can generate sequence of actions in advance for the next states based on the current state to make the control more smooth and robust.
The essence of addressing the two aforementioned intermittent control tasks is identical, that is, to achieve intensive control under sparse interaction (or observation) to ensure the effective and smooth movement of the executor, ultimately leading to efficient task completion. To this end, we introduce Intermittent-MDD (refer to Secß for further elaboration) to model the above two settings in unified manner, that is, the agent is expected to decide on sequence of actions based on the current state, covering a suitable number of time steps, to maintain smooth and efficient motion of the executor between the two interactions. The most direct approach would be to employ model- based reinforcement learning (MBRL) methods with multi-step decision-making capabilities, such as Dreamer (Hafner et al. 2023] and TD-MPC [Hansen et al. 2022). Regrettably, the form of multi-step decision makes the error of the dynamic model accumulate and then make the policies suboptimal. In addition, the high demand for high-quality data and the complexity of constructing dynamic models makes MBRL deployment costly in real-world scenarios [Janner et al. 2019] (detailed experimental analysis in Appendix C.I.. Instead, we sought to unlock the multi-step decision-making ability of the mrdel-free DRL approach, e.g. TD3 [Fujimoto et al.) 2018) and PPO [Schulman et al.) 2017).
The most simple method for model-free DRL to alleviate the intermittent interaction issue is using frameskip (also commonly known as 'action-repeat') ) [Kalyanakrishnan et al., 2021]. where the same action (usually the last action) is repeated during a fixed interval which is often used in Atari (Braylan) jet al. 2013). However, longer frameskip will lead to mechanized motion, making it impossible for the agent to change actions at key states and thus resulting in suboptimal policies. Another way is to let the RL algorithms make up-front decisions (advance decisions) for future steps according to the current state or the received delayed state. Only actions for the correct time steps will be executed. Compared to frameskip, these methods can improve action diversity. However, directly increasing the horizon of decision making will lead to action space explosion and increase the difficulty of policy optimization [Chn et al. 2021].
In this paper, we propose Multi-step Action RepreSentation (MARS), which is the first plugin method for DRL algorithms to solve intermittent control tasks, significantly reducing the difficulty of multi-step policy training while ensuring the flexibility and diversity. The high level idea is shown in Figure MARS constructs a compact and decodable low dimensional latent action

---

space for the original multi-step actions, the latent action dimension does not explode as the steps number increases. Based on this latent action space agent can learn stable latent policy. The latent action selected by the policy can be reconstructed into original multi-step actions by the decoder. Specifically, MARS relies on conditional Variational Auto-encoder (c-VAE) (Sohn et al.) 2015) to construct the latent representation space for the associated multi-step actions. Intuitively, an efficient latent action space should have two characteristics: the combination number of space, which can complicate the decision space and potentially result in suboptimal policies. We restrict the decision space to relatively small subspace by introducing the concept of action transfer scale v and taking it as the condition term of our VAE, i.e. using v to implicit segment the latent space (Sohn al., 2015 @ Dynamic semantic smoothness, characteristic of the original action space, which means latent actions that have similar impacts on the environment should be close together in latent spaces. Dynamic semantic smooth action space retain the utility of adding Gaussian noise perturbations to the policy during training: While ensuring that the impacts of initial latent action chosen by the policy and latent action after disturbance the environment is similar in high probability, some uncertainty is also retained that the disturbed action may explore rare actions that have unknown impacts on the environment, which helps maintain balance between exploration and exploitation [Schwarzer et al.. 2020 We utilize unsupervised environmental dynamics to learn a predictive representation of dynamics for improving semantic smoothness. In practice, we adding an additional dynamic residual prediction module for VAE.
Decision space simplification: actions is huge, especially in the continuous action
Our contributions are summarized as fol- lows: > We propose the first plugin method and DRL framework for solving intermittent control tasks via multi-step action represen- tation learning. > We provide two modules improve the effectiveness and smoothness of the learned multi-step latent space. Our method outperforms baselines on both real- world fixed interaction interval robotic con- trol tasks and random interaction interval simulation control tasks.
Internitiont interaction
Decode
Decision
Select Latent Action
Latent
Space
Original multi-abep. Action
Environment
Action trajectory
Figure 2: Conceptual overview of MARS.
2 Background
Markov Decision Process (MDP). A MDP can be represented tuple: (S,A,P, R, 7, where denotes the state set, A denotes an action set, P the transition function: S x A - [0, and RR is the reward function: S x A -> R. 7 [o, 1) is discount factor and T is the decision horizon. The goal is to optimize the agent's 's policy to maximize the expected discounted cumulative reward.
The variational auto-encoder (VAE) is directed graphical model with certain types of latent variables, such as Gaussian latent variables. generative process of the VAE is as follows: set of latent variable is generated from the prior distribution po(2) and the data is generated by the generative distribution pe (x] conditioned on 2 : ~ po(2), x ~ po(x)z) (z). In general, parameter estimation of directed graphical models is often challenging due intractable posterior inference. However, the parameters of the VAE can be estimated efficiently using the stochastic gradient variational Bayes (SGVB) framework, where the variational lower bound of the log- log-likelihood is used as a surrogate objective function. In this framework, a proposal distribution go (x)22) is introduced to approximate the true posterior (= (z]2)). MLPs are used model the recognition and the generation models. Based on the Gaussian latent assumption, the first term of Eq can be marginalized. The second term can be approximated by drawing samples 20) = 1,... L) by the proposal distribution qe (z)2). The empirical objective of the VAE is written as:
Variational Auto-Encoder.
Lv +) Σ (c)20)) KI((O([[=]IN((.II)
(1)
3

---

3 Intermittent Control Markov Decision Process (Intermitten-MDDP
In this section, we introduce the Intermittent Control Markov Decision Process (Intermittent-DDP to model both fixed and random interaction interval control problems. The objective of Intermittent- MDP is to train the policy to decide on an effective action sequence u based on the current state s, which should be longer than equal to the maximum interaction interval thus ensuring smooth and efficient motion of executor between the two interactions.
Definition of Intermittent-MDD. Intermittent-MDP (M) can be represented as a tuple: M _ (S,4, R. K. /4))- S E RR is the state space, 4 is the set of action sequence u that the policy can select. and μo denotes the initial state distribution. The reward is defined as R, KC is the multi-step transition function.
Different from normal MDP in which the agent makes a decision a according to the current state 8t. intermittent- MDDP requires the agent to generate action sequence u, {ac. .... at+eJ according to the received state sy (a is the single step action at timestep t). The cumulative rewards of executed actions can be received in the next interaction. Policy takes current state to select action sequence 24 = {a,, .... a(+c]. }- When the interaction interval is present for j timesteps, the executor uses j single-step actions in the action sequence
maintain the motion (j E {0, ...,,)))) When the interaction is continuous, the executor uses the first single-step action in u. each time (degenerated to the transition function in normal MDP). Thus, the environment transition function K. is defined as:
e)).
(((++5|81, un) = - P(ss11 a, (u,ls)) j E 10.1.....c c), ag E us.
(2)
If the interval is random and we only know the maximum number of interval step c, Intermittent- is represented as random Intermittent-DDI When the interval time is fixed, Intermittent-MDP represents a fixed interaction interval problem, i.e., fixed Internittent-MDD.
4 Multi-step Action Representation
In this section, we introduce Multi-step Action Representation (MARS), a novel framework that can map long action sequences into an efficient latent space. We hypothesize that multi-step actions reside on homogeneous manifold. To learn the multi-step action representation, we employ VAE to constructing a compact and decodable latent representation space approximate such a manifold.
To align our latent action space with the two characteristics mentioned in See I i.c. Decision space simplification @ Dynamic semantic smoothness. In Sec 4.1 we introduce the concept of action transition scale and encode this as a conditional aid in VAE training to simplify the decision space of the policy. In Sec we demonstrate how state residual prediction can be leveraged to smooth the semanties of dynamic transitions in latent action spaces, and in Sec we illustrate the combination of MARS with DRL, as well as the extension of our approach to random intermittent-MDP.
4.1 Scale-Conditioned Multi-step Action Encoding and Decoding
Although VAE has been proven to be able to construct a compact and effective latent action space for normal MDP with state the prior condition (Li et al. 2021], J, due the complex combinatorial nature of multi-step actions in Intermittent-MDPP the policy still struggles to explore the optimal actions in the vast action space. Therefore, to address this challenge, we introduce the concept of action transition scale v as a conditional term in VAF, to constructs the multi-step latent action space Z. V stands for description of the motion style, it represents the accumulation of action change scales within each action sequence u. By determining v. we can constrain the latent action E Z chosen by the policy within related subspace [Sohn al. 2015) in which all candidate action sequences have the same motion style in terms of v, thus reducing the difficulty of exploration. We choose a conditional term due to its tase-specific nature. For instance, in robot scenarios, to ensure the balance of the robot and prevent joint damage, the scale of action changes is typically small. some obstacle avoidance tasks, however, the magnitude of action changes can be significant when the agent encounters suddenly approaching obstacles. Once the policy roughly determines the appropriate action transfer scale for the current state, can make more suitable decisions, making the policy more efficient and controllable in the complex action space. Notably does not need to be

---

set manually but is selected adaptively by the policy through training, which is described in detail in Sec We formulate v as follows:
Eis || aill (c 1) B
(3)
C is the maximum interval and B denotes the upper limit of action change. The numerator part represents the total absolute difference between consecutive actions, used to assess the transition magnitude of un Eq 3 normalizes Uu, to [0, 1].
Given action sequence u, and the corresponding states St.t+e, our encoder (a Wey Stat+ee Vue) parameterized by Φ takes Stat+ and the action transition scale Uu, as conditions to build a multi- step latent action space, and maps the action sequence 4t into the latent variable Eu, E IRd (dj denotes the dimension of k..). The decoder P& (u St, Ua,) parameterized by ti then reconstructs the multi-step actions ue from Zu,- We employ a Gaussian latent distribution N(uz, Oz) to model (zu. [ue, St.t+c: Vu,) where μ4 and Og the mean and standard deviation outputted by the encoder. The decoder decodes Zu, ~ N N( oz..) as following: ûe 9v, o Pvy (Zug St, Un ). Ie, is a fully-connected layer for reconstruction. denotes cascade, i.e. the output of Pvo acts the input of Doh-- We utilize cascaded heads because traditional parallel heads, employed for both reconstruction and state residual prediction, can interfere with optimizing individual objectives and impede the learning of the shared representation [Azabou et al., 202T]. Peo denotes the shared layers of the decoder. (1,2) denote the parameters of the prediction networks. The loss function of our VAE is:
LVAE(ó, v/) = B...-DDargg [l]
uz 112 Dxz (go(-lue, Stitte, Va, [O,, D)],
(4)
where D is the buffer. The first term is the reconstruction loss (using MSE) of the action sequence, the last term is the Kullback Leibler divergence DEL between the variational posterior of latent representation 2 and the standard Gaussian prior. By using the reparameterization trick [Kingma and Welling 2013), i, is differentiable with respect to de and ø. For any latent variables 2 it can be decoded into multi-step actions ún conveniently by the VAE decoder. That is,
Encoder :Zu, q0(- [ee Sorttie Vue))
V Sese-ti: Mes Vus
(5)
Decoder :û gv. Pvo (Zug 86; Biee)
V 8, Rugy Vug
Stage
Stage Envirorime
4.2 Dynamic-Aware Multi-step Action Representation
i ag a.
Besc
In section 4.1 we introduce how to build a decodable representation repreaatnnn space for multi-step actions. However, it still inefficient to learn the policy and value functions in the latent ac- tion space learned by the above VAE. In comparison to the real-world ac- tion space, the current construction of our action space lacks crucial attribute: dynamic semantic smooth- ness. This refers to the manifestation of differences in environmental im- pact through the Euclidean distance between points in the latent space, where closer points correspond to more similar impacts. To address this issue, we further apply an unsupervised learning loss based on refine the multi-step action representation. Relevant proof can
il a. a. aee
h
9
PΦo
Sp. Vu:
Latent space
Pdo
condition
Zuyz
St
94
condition
44 Sample
Selecti,, from Z
Select u
St.tte U4 Uur
RL policy
(a) Build latent action space Z
(b) Combine with RL policy
Figure 3: Architecture MARS. 't' denotes different stops at different stage. Both 2,, and 2 refer to variables in Z, but they are used in different stages (the same goes for Vu. and vz))
environmental dynamics prediction be found in Whitney et al., 2019]. MARS captures the environmental dynamics by predicting the state transition residual. We use the state transition measure of environmental dynamic becuase:
State transition does not require a per-step reward, it can be used in reward- -wwrd-agnostic pretraining. Building value equivalent models is more difficult since Q-values the early stage of training are inaccurate [Wang et al. 2016). In contrast, state transition is more reliable and accessible. O The same reward or Q-value may correspond to different environmental dynamics, but the same environmental dynamics have the same reward or Q-value.
5

---

Specifically, MARS predicts the residual difference between the state St+c after the execution of ug and the current states,, As shown in the left of Figure [3] f, subnetwork of the decoder. For any transition sample (st, ue, St+te), we denote the state residual as 05,,8113 = Sttc 8, and denote
Potate haz o Pv- The residual 00, is predicted as:
Pstate (Zugs St; Vns))
V 80s Fuse Une-
(6)
The environmental transition prediction loss is defined as:
Lay (0, 4/) Me [10 de Ses
(7)
Above all, the ultimate training loss for the multi-step action representation is:
LMs v) = AV + (0,)))
(8)
where / is hyper-parameter that controls the relative importance of the Lay and LV LMARS only depends on the environmental dynamic data which is ewaadd-agnsstic [Erraqabi et al., 2202 Yarats al.] 22221. During training, transitions stored in the buffer offline dataset utilized. Additionally, we observe that MARS exhibits insensitivity to data quality. Notably, data collected
through random policies suffice for effective training the multi-step action representation.
4.3 DRL with Multi-step Action Representation
As a plug-in method, MARS can applied to any RL algorithm. It contains two types of actions: ! the encoded multi-step action and @ the action transition scale e. RL al- gorithms maximize the expected cumulative reward by selecting the optimal and v, 8t. In this section, we apply MARS to a typ- ical model-free RL method TD3 (Fujimoto iet al. 2018) as an example. TD3 is a deter- ministic Actor-Critic algorithm. As illustrated in the right part of Figure 3 with the learned transition-awanl multi-step action representa- tion, the actor network learns a latent policy Th that outputs the latent actions according current state s, i.c. [Zz, uy] = (s). and es respectively represent the action selected time from 2 and v constructed in the section 4.1 Then we obtain the corresponding multi- step actions ue by decoding the latent action 2, and Ve: vi = Iv1 Pvo (8, Ze, vt).
Algorithm 1 MARS-TDS
Initialize actor and critic networks Qo Qo₂ Initialize conditional VAE go Pv. buffer D.
Stage 1: Build latent action space
while not reaching warmup training times do
Fill with data generated by random pol-
icy offline datasets
Update go P using samples in D. & Eq end while
Stage 2: Train the RL policy
while < policy training time do
2t, = Tw (with Gaussian noise)
U4 = 9eh, o Peo Peo (2t, 8, 24)
Execute u4 observe r and new state s'
Fill with (s, uz 21, Ve, r.
update Qø, Quy Rw end while
Eq
Eq 10
Two critic networks Qø Qe are utilized which take the latent actions 2, and Vt as inputs to approximate the value function Qn. more accurately. We train the critic network using the transition data (s, U,, 21, r,s') s') sampled from the experience replay. To simplify notations, this subsection, s is the current state. s' is the state perceived at the next interaction interval. The critic loss function is:
(l,),) E,,,,,,, [(V- (s, 21, [ for Vi E (1.2).
Where y r min Qs, (s', (s'))) @ denotes the target network parameters. The actor updated ==1,2 according to the Deterministic Policy Gradient Siiverr et al.
2014]:
V [w. E, V (s) Qo, (s, Tu (s))
(10)
The overall algorithm MARS-TD3 is summarized Algorithm I which contains two major stages: @ the warmup stage and @ the policy learning stage. In stage 1, MARS is trained using prepared replay buffer D. The SC-VAE is updated by minimizing the VAE and the environmental dynamic prediction loss. Note that the proposed algorithm has requirement how the buffer D is prepared

---

LAND-13 123
Verfect DX
LOAG TEG
Pather
Perfect TD3
300-
140
60
30-
0/
150
12
Lio
L7
2
Hopper
(b) Ant
Mazz-medium
2408
Matoer TD3
12000
Purfec- TD3
Magowa 103
4008
9.00
300-
2008
9:00
200
100
2000
2.00
2.2
18
6.30
0.25
0.2
0.75
130
120
150
L7
2,50
(c) Walker
(d) HallCheetah
(f) Maze-hard
Figure 4: Comparisons of methods in simulated remote NPC control tasks with random interaction interval. The y-axis denote the environment steps and average episode reward. Curves and shades denote the mcan and the standard deviation over runs.
and here we simply use random policy to gather the data. In stage 2, given environment state, the latent policy outputs the latent action 2 and the action transition scale perturbed by the Gaussian exploration noise. Then, the latent action is decoded into the original multi-step actions by the decoder so as to interact with the environment. The newly collected transition sample is stored in the replay buffer D. After that, the latent policy updated using the data sampled from D. The action representation model is also updated periodically in the second stage to make continual adjustments to the change of data distribution. The detailed network architectures are presented in appendix [B.1]
As for random interaction interval tasks, the interaction interval cannot be predicted, so we let the agent output action sequence with the maximum length. To improve the training stability, we precisely record the absolute time step and the execution flag of each action, which makes the actions be executed in strict accordance with the time step order. When new action sequence arrives at the action executor, the previous action sequence will be replaced, and the execution flag of the unexecuted actions will be executed in the new action sequence. Thus, the actual reward stored in the experience replay D for each latent action is the sum of the executed action reward the corresponding sequence.
False. The subsequent rewards will be attributed to the actions
Experiment
We empirically evaluate MARS answer the following research questions. RQ1: Performance in random interaction interval tasks. Can MARS significantly improve the performance in random Intermittent-MDD tasks, such simulated remote NPC control tasks? RQ2: Performance in fixed interaction interval tasks. Can MARS significantly improve the performance in fixed Intermittent- MDP problems, such as real-world robot arm grasping tasks? RQ3: Generalization. Can MARS be seamlessly integrated into existing RL algorithms and improve their performance? RQ4: Ablation study. Do both the action transition scale and the state dynamic prediction contribute optimizing the multi-step latent action space? How is the robustness of MARS?
5.1 Random Intermittent-MDD Tasks (RQ1)
5.1.1 Experimental Setups
Benchmarks. We select two types of control tasks to simulate the remote NPC control problem with random interaction intervals: (1) robot control and (2) navigation tasks. For robot control tasks, we select four typical openai Mujoco tasks with random interaction delays,i.e., Hopper, Ant, Walker, HalfCheetah. Mujoco is well-known testbed and is widely used in reinforcement learning
7

---

research [Brockman et al. |016] For navigation tasks, we used the medium and difficult maps of 2dmaze in D4RL [Fu al., 2020 where the agent's goal is walk to the end of the maze under unstable interactions. Both types of tasks are modified to incorporate random interaction intervals, mimicking the real-world remote control scenarios [Chen and Wu, 2015]. RT'S games [Andersen] et al. 2018), the maximum interaction interval usually spans 0.5s to 0.7s, with the action execution time typically between 0.1s and 0.25s. Accordingly, we set the maximum interval to 10 time steps, requiring the policy to generate an effective action sequence dc:r499 based on the received state 8t.
problem. Thus, we compare with three baseline methods. (1) Perfect-TD3: We train TD3 in the perfect environments with continuous interaction without interaction interval, i.e., normal MDP. It the highest standard for evaluating MARS. (2) Frameskip-TD3: We combine the frameskip technique with TD3 (a common trick for multi-step decision) and apply it to the intermittent tasks. (3) Multistep-TDDS We modify TD3 to directly make decisions for the future steps by outputting concatenated action vector of times of dimensionality.
Baselines. To our knowledge, no specific solution exists for the Intermittent-Me
5.1.2 Performance of remote NPC control tasks
To mitigate implementation bias and ensure comprehensive comparison, implement all methods using the same architecture based on TD3 [Fujimoto et al. 20118). For all tasks, we set the dimension of 2, 8 and the scaling parameter
5. We set the warm-up (stage 1) step to 400000 and 100000 for the Mujoco tasks and the navigation task respectively. Detailed parameter setting can be found in appendix B.2 The results in Figure show that MARS-TD3 outperforms the other methods in all tasks, especially in the sigh-dimmnsional action control tasks (i.c., Mujoco). Compared with vanilla multstep-ID33 our method can significantly improve the performance of the RL algorithm in random Intermittent-DDD control problems by learning a more compact multistep action representation while avoiding the convergence difficulties caused by dimensional explosion. Note that MARS-TIL can also achieve comparable performance with perfect- -D3 in most tasks.
5.2 Fixed Intermittent-MDD Tasks (RQ2)
5.2.1 Experimental Setups
Task description.The experiment in- volves 6-DoF robot arm performing grasping tasks within a 30 x 30 x 30cm' tabletop workspace. 15 rounds of exper- iments are conducted using our method and baselines. In each round, 6 objects are randomly selected and placed on the table. The robot observes the workspace with a single-depth image from a fixed side view. The viewpoint of the virtual camera points toward the workspace's origin at a radial distance r 2/ and an angle 0 = π /3, I is the workspace's length. We follow the setting of traditional grasp tasks (Jiang et al. 202T] and set the total number of interactions to 6, meaning that the policy is expected to complete the grasp of all six targets in six intervals. Thus, the robot arm allotted a single observation of the environment per object. As shown in Figure 5] at each interaction interval, there are 4 steps to grasp an object. The robotic arm has to first acquire observations at the beginning of each grasping. Upon perceiving the observation, an action sequence a 15 is executed to grasp the target object and then place the object in the target box. The time limit for grasping one object is set to 30s. The robot need to move to predefined location for capturing image (images are transformed into 512-dim vector). Action space is 3 3-din the reward function encourages policies to maximize the grasping success rate and motion smoothness.
SABp Uiadinrrre Bree men rantiuaakor in th spacited prsttion
Sau Move casigared location vorko
Foceddo R.A.
Figure 5: A complete grasp process in each interaction interval.
Baselines. Online RL demands extensive exploration, resulting in low sample efficiency. Besides, Online RL. accidental random exploration may lead to robot arm damage. Thus, we opt for offline
'The action denotes the coordinates expected to be reached by the hand, the defualt underlying planning algorithm generates the motion trajectory via the expected coordinates).
8

---

RL commonly utilized in robot scenarios, i.e. TD3+Behavior Cloning (TD3-BC) [Fujimoto and Gu 202T]. to serve the backbone. Methods: (1) Vanilla TD3-BC. (2) Multistep TD3-BC, same as Multistep- DDD used in the section 5.1 (3) Dense observation TD3-BC. An extra camera provides dense observation every seconds, and the robot arm still has to move to the observing position at the start of each episode to eliminate differences caused by the observing motion.
Evaluation metrics. (I) Grasp success rate (GSR), the ratio of successfully grasping the objects. (2) Declutter rate (DR), the average ratio of objects removed after successful grasping. (3) Motion smooth ratio (MSR): grasp time
evaluating whether the motion is smooth and natural (pauses and lotal time
shaking can result in lower scores). All results are averaged over 15 simulation rounds.
5.2.2 Performance of real-world Robot Arm Grasping
The overall comparison is presented in Table I The results show that due to the sparse interac- tion, the vanilla TD3-BC fails to complete the task and frequently pauses or takes unnatural motions during the execution. While the per- formance of TD3-BC is significantly improved with additional dense observations, it remains unsatisfactory due to the lack of real-time feedback. Moreover, the use extra cameras results in higher costs of the experiment. Although Multistep TD33BC helps smoothen the motion, it fails to effectively mitigate the instability caused by Intermittent-MDD MARS addresses the limitation the vanilla Multistep TD3-BC, which makes TD3-BC complete the task with high quality. We provide the corresponding motion pattern for each method. To further verify the effectiveness of MARS combined with online RL on fixed interaction interval control task. The results the Appendix C.2 show that our method outperforms the baselines on all four tasks.
Method
MSR(%)
GSR(%)
DR(%)
MARS TD3-BC
70.8
74.2
82.9
Multistep + TD3-BC
70.5
61.5
69.4
Dense obs + TD3-BC
65.3
59.1
70.7
Vanilla TD3-BC
22.5
34.1
27.3
Table 1: Performance of the robot arm grasping task.
5.3 Generalization of MARS (RQ3)
We further test MARS with popular RL algorithms on three random interaction interval tasks: Hopper, Walker, and Maze hard. We maintain consistent parameters for each method and implement them based on the publicly available codebase. We train each RL algorithm with dense interactions as baselines and compare them with setting. Results in Appendix C.3 shows that all methods can learn effective policies with the help of MARS and the converged performance is comparable to the perfect interaction settings.
methods under the random interaction interval
5.4 Ablation Study and Visual Analysis (RQ4)
We conducted evaluations on the key compo- nents of MARS, i.e. action transition scale and state dynamic prediction. Figure o (a) shows that both components optimize the latent space and improve the learning efficiency of DRL poli- cies. MARS performs best when both modules are integrated.
16
125
220
comprehensive analysis is
Random Intemitemm MDP Hopper
(b) 20:2555 visualzators
Figure Ablation study, the curve and shade denote the mean and standard deviation the returns over runs. sualize the latent action representations. We color each action based on its impact the environment. Results show that actions with a similar impact, i.e., òs,, on the environment are clustered closely, which indicats that the learned action representation is dynamic smooth. Besides, results in Appendix C,3 improoe the robustness of MARS to different interaction interval settings. Results in Appendix C.6 show the influence of latent space dimensions MARS. Lastly, an analysis of self-supervised training steps can be found in Appendix C.7
shown in Appendix [C44 Figure 6 (b) uses
SNE [van der Maaten and Hinton 2008 to vi-
6 Conclusion and Limitation
In this paper, we observe that previous DRL. methods fail to learn effective policies in intermittent control scenarios because of the discontinue interaction. To improve the performance of DRL on

---

such tasks, we propose Mtep action representation (MARS) to construct a reliable multi-step latent action space. Based on this latent action space, DRL methods can make effective advance decisions to ensure the smoothness and efficiency of the agent's motion when it cannot interact with the environment. MARS outperforms baselines in a variety of fixed and random interaction intervals control tasks. Additionally, MARS has potential for improvement in represent extremely long action sequences, which we will address by identifying more powerful representation models, e.g. transformer based VAE, in the future.
Acknowledgement
This work is supported by the National Natural Science Foundation of China (Grant Nos. 62422605. 92370132, 62106172), the National Key R&D Program of China (Grant No. 2022ZD01 16402) and the Xiaomi Young Talents Program of Xiaomi Foundation.
References
Per-Ame Andersen, Morten Goodwin, and Ole-Christoffer Granmo. Deep rts: a game environ- ment for deep reinforcement learning in real-time strategy games. 2018 IEEE conference on computational intelligence and games (CIG), pages 1-8. IEEE, 2018.
Mehdi Azabou, Mohammad Gheshlaghi Azar, Ran Liu, Chi-Heng Lin, Erik C Johnson, Kiran Bhaskaran-Nait Max Dabagia, Bernardo Avila-Pires, Lindsey Kitchell, Keith Hengen, al. Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprint arXiv:2 10106, 2021.
Andrea Bonarini. Communication in human-robot interaction. Current Robotics Reports, 1:279-285, 2020.
Alexander Braylan, Mark Hollenbeck, Elliot Meyerson, and Risto Miikkulainen. Frame skip is a powerful parameter for learning to play atari. In AAAI-15 Workshop Learning for General Competency Video Games, 2015, URL http:t/t://nn.... cs. www.wsteaaa......... wwweedduubbbbbe Pbraylrrallnn
aaai :aai 15ws
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint 0rXiv:1606.015.0 2016.
Chen Chen, Hongyao Tang. Jianye Hao, Wulong Liu, and Zhaopeng Meng. Addressing action oscillations through learning policy inertia. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7020-7027, 2021.
Qian Chen and Dapeng Wu. Delay-ratendissortionn model for real-time video communication. IEEE Transactions Circuits and Systems for Video Technology, 1376-11373399999 2015.
Xihua Dong, Xiaochen Li, and Dapeng Wu. Analysis of packet error probability in delay con- strained communication over fading channels. In 2009 6th IEEE Consumer Communications and Networking Conference, pages 1-5, JEEE, 2009.
Akram Erraqabi, Mingde Zhao, Marlos C Machado, Yoshua Bengio, Sainbayar Sukhbaatar, Ludovic Denoyer, and Alessandro Lazaric. Exploration-driven representation learning in reinforcement learning. In /CML 202/ Workshop on Unsupervised Reinforcement Learning, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Florian Fuchs, Yunlong Song, Elia Kaufmann, Davide Scaramuzza, and Peter Dürr. Super-human per-
formance gran turismo sport using deep reinforcement learning. IEEE Robotics and Automation Letters, 6(3):4257-42666
2021.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 02000132-21443 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
10

---

Danijar Hafner. Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint wrXiv:2301.04100 2023.
Nicklas Hansen, Xiaolong Wang, and Hao Su. Temporal difference learning for model predictive control. arXiv preprint arXiv:2203 04955, 2022.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu. Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. arXiv preprint 2021.
Shivaram Kalyanakrishnan, Siddharth Aravindan, Vishwajeet Bagdawar, Varun Bhatt, Harshith Goka, Archit Gupta, Kalpesh Krishna, and Vihari Piratla. An analysis of frame-skipping in reinforcement learning. arXiv preprint arXiv:2102.03718, 2021.
Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and
Davide Scaramuzza. Champion- drone racing using deep reinforcement learning. Nature, 620 (7976))982-987,, 2023.
Diederik P Kingma and Max Welling. arXiv: 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47)xcabc5986, 2020.
Boyan Li, Hongyao Tang, Yan Zheng. Jianye Hao, Pengyi Li, Zhen Wang. Zhaopeng Meng, and Li Wang. Hyar: Addressing discrete-comtinuuus action reinforcement learning via hybrid action representation. arXiv preprint 2 109.05990, 2021.
Yuliang Li, Rui Miao, Changhoon Kim, and Minlan Yu. Lossradar: Fast detection of lost packets in data center networks. In Proceedings of the 12th International on Conference on Emerging Networking EXperiments and Technologies, CONEXT 16, page 481-495, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 978 1550342926. 2999609. URL ptttps://doi.org..tttt 01410019115129992111 2999609
doi: 10.1145/2999572.
Chuzhao Liu, Junyao Gao, Yuanzhen Bi, Xuanyang Shi, and Dingkui Tian. A multitasking-orrimne robot arm motion planning scheme based on deep reinforcement learning and twin synchro- control. Sensors, 20(12), 2020. ISSN 1424-8220.
doi: 10.3390//20123515. URL https
7/www.mdps www.. wtt.comm/12220 424 0222-2220//2/12/3555
Yue Ma, Yali Wang, Yue Wu, Ziyu Lyu, Siran Chen, Xiu Li, and Yu Qiao. Visual knowledge graph for human action reasoning in videos. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4132-4141, 2022.
Yue Ma, Yingqing He, Hongfa Wang, Andong Wang, Chenyang Qi, Chengfei Cai, Xiu Li, Zhifeng Li, Heung-Yern Yeung Shum, Wei Liu, et al. yourow-your-llick Open-domain regional image animation via short prompts. arXiv preprint wrXiv:2403.08263 2024.
A Rupam Mahmood, Dmytro Korenkevych, Brent J Komer, and James Bergstra. Setting up reinforcement learning task with a real-world robot. In 2018 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 4635-4640. IEEE, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1707007707.06347, 2017.
Max Schwarzer, Ankesh Anand, Rishab Goel, R Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations, arXiv preprint arXiv:2007.05929, 2020.
11

---

David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning, pages 387-395. Pmir, 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL https: httptttpppps//ppopreeeeeeeeeeeee neuraiiinnsnnnuiiiinne nwwrips. wttpscaaapppeeeleeppppeeeeeeeeee F1L//855222edaa33672255532222333 Papeer..ff
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learn- ing Research, 9(86):2579-200 2605, 2008. URL http://jmlr... org/ http:///ppp/ee/////////aaaaaaaaa html
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003. PMLR, 2016.
William Whitney, Rajat Agarwal, Kyunghyun Cho, and Abhinav Gupta. Dynamics-aware embeddings. arXiv preprint arXiv: 093577 2019.
Albert Wong, Chunyin Chiu, Gaétan Hains, James Behnke, Youry Khmelevsky, and Tyler Sutherland. Network latency classification for computer games. 2021 IEEE International Conference on Recent Advances in Systems Science and Engineering (RASSE), pages 1-6, 2021. doi: 10.1 109/ RASSE53195.2021.666 266968888...
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with prototypical representations. In International Conference on Machine Learning, pages 11920- 11931. PMLR, 2021.
Yan Zheng, Changjie Fan, Xiaofei Xie, Ting Su, Ma, Jianye Hao, Zhaopeng Meng, Y. Liu, Ruimin Shen, and Yingfeng Chen. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 772-784, 2019. URL pattps://apt: semanticscholar.orgg CorpusID:220694499
12

---

A Preliminaries
Markoy Decision Process (MDP). standard MDP can be represented as a tuple:
where S denotes the state set, A denotes an action set, P is the transition function: 8 1 and is the reward function: S A + I.. 7 [0, 1) is a discount factor and 7 is the decision horizon. The target of the agent is to optimize its policy to maximize the expected discounted cumulative reward.
Frameskip. Frame- skimping may be viewed as an instance of (partial) open-loop control, under which predetermined sequence of (possibly different) actions is executed without heed to intermediate states. Aiming to minimize sensing, KKlyyaakrishhan et al. [2021] proposes a framework for incorporating variable-length open-loop action sequences regular (closed-loop) control. The primary challenge in general open-loop control is that the number of action sequences of some given length d is exponential in d. Consequently, the main focus in the area on policies to prune corresponding data structures [Braylan et al. 2015). Since action repetition restricts itself to set of actions with size linear in d, allows for d itself be set much higher in practice. With frame-skipping the agent is only allowed to sense every d state: that is, if the agent has sensed a state 8, at time step >= 0, is oblivious to statess;41, S(+2; ..., St+d-1. and next only observes St4d-
The variational auto-encoder (VAE) is a directed graphical model with certain types of latent variables, such as Gaussian latent variables. A generative process of the VAE is as follows: a set of latent variable z is generated from the prior distribution po(2) and the data generated by the generative distribution pe (z]]2) conditioned on : po(z), x pe (z(z). In general, parameter estimation of directed graphical models is often challenging due to intractable posterior inference. However, the parameters of the VAE can be estimated efficiently in the stochastic gradient variational Bayes (SGVB) framework, where the variational lower bound of the log-likelihood is used surrogate objective function. In this framework, a proposal distribution (2(2 22,, which is also known as a "recognition" model, is introduced to approximate the true posterior Po(x)22. The multilayer perceptrons (MLPs) are used model the recognition and the generation models. Assuming Gaussian latent variables, the first term of Equation A can be marginalized, while the second term not. Instead, the second term can be approximated by drawing samples 2(0(l I,... L) by the recognition distribution (0(2),), and the empirical objective of the VAE with Gaussian latent variables is written as follows:
Variational Auto-encoder:
Lv AE v) = Σ (alz [xl K (
(11)
B Experimental Details
B.1 NETWORK STRUCTURE
Layer
Actor Network
Critic Network
Fully Connected
(state dim, 256)
(statedim dim latent space dim, 128)
Activation
ReLU (256, 128)
ReLU (256, 128)
Fully Connected
Activation
ReLU
ReLU
Fully Connected I128,latent space dim) and u dim Tanh
(128, 1)
Activation
None
Table Network Structures for DRI. Methods
Our codes are implemented with Python 3.7.9 and Torch 1.7.1. All experiments were run on a single NVIDIA GeForce GTX 3090 GPU. Each single training trial ranges from 4 hours to 17 hours, depending on the algorithms and environments. We will open source code in the near future.
Our TD3 is implemented with reference to gitggb. wtathub.com/sttee wwtp./////ff////aooa (TD3 source-code). DDPG and PPO are implemented with reference to https://gi whh th-pytorch For fair comparison, all the baseline meth- ods have the same network structure (except for the specific components of each algorithm) as our MARS- RD3- implementation. As shown in T we two-layer werw-forward neural network of
com/sweetice/
13

---

Model Component
layer
dimension
Fully Connected (encoding)
(R3 (56) (stae dim + u dim. 256)
Fully Connected (condition)
Element-wisc Product
ReLU (encoding). ReL.U(condition)
Fully Connected
(256. 256)
Conditional Encoder Network
Activation
ReLU
Fully Connected (mean)
(256, latent space dim)
Activation
None
Fully Connected (log std)
(256, latent space dim)
Activation
None
Fully Connected (latent)
(latent space 256)
Fully Connected (condition)
(stae dim +u dim, 256)
Element--wis Product
ReLU (encoding), ReL.Ucondiiion)
Fully Connected
(256, 256)
Activation
ReL.U
Conditional Devoder, Prediction Network
Fully Connected (v)
(256, action dynamic transition)
Activation
None
Fully Connected (reconstruction)
(256, multi-step action dim)
Activation
None
Fully Connected
(256,256)
Activation
ReLU
Fully Connected (prediction)
(256, dim)
Activation
None
Table 3: Network structures for the Multi-stop action representation (MARS).
256 and 256 hidden units with ReLU activation (except for the output layer) for the actor network for all algorithms. For DDPG the critic denotes the Q-network. For PPO, the critic denotes the V-network. All algorithms (TD3, DDPG, PPO) output two heads at the last layer of the actor network, one for latent action and another for dynamic transition potential.
The structure of MARS is shown in Tab 3
We element-wiss product operation Mahmood et al.
2018 and cascaded head structure Fuchs el al. 2021 to our model Ma et 2024 2022
B.2 Hyperparameter
For all experiments, we use the raw state and reward from the environment, and no normalization or scaling is used. No regularization is used for the actor and the critic in all algorithms. An exploration noise sampled from N(O, 0.1) [Dong et al., 2009) is added to all baseline methods when selecting an action. The discounted factor is 0.99 and use Adam Optimizer (Li et al. 2016) for all algorithms. Tab4 shows the common hyperparameters of algorithms used in all our experiments.
Hyperparameter
Frameskip-TDD TD3
Multistep-TD3
MARS-PPO
MARS-TD3
MARS DDPG
Actor I leaming Rate
1e
le
le
le
1r
le
Critic Learning Rate
le a
Representation Model Leaming Rate
None
None
None
1e-4
5e-3
5e-8
Discount Factor
0.99
0.99
0.99
0.99
0.99
0.99
Batch Size
128
128
128
128
128
128
Buffer Size
1c5
le5
1c5
1c5
1c5
lc5
Table 4: A comparison of common hyperparameter choices of algorithms. We use 'None' to denote the 'not applicable' situation.
B.3 Additional Implementation Details
For PPO, the actor network and the critic network are updated every 2 and 10 episode respectively for all environments. The clip range of the PPO algorithm set to 0.2 and we use GAE [Sutton and Barto 2018) for a stable policy gradient. For DDPG, the actor network and the critie network is updated at every environment step. For TD3, the critic network is updated every environment step and the actor network is updated every 2 environment steps.
The default latent action dim is 8, we set the KI. weight in representation loss LMAARS as 0.5. Environment dynamic prediction loss weight 3 is 5 (default).
14

---

C Additional experiment
C.1 Performance of model-based reinforcement learning algorithms on Intermittent-MDD
tasks.
Methods
Ant (fixed-Intermitten)
Hopper (fixed-Iniermitten)
Ant (random-- Internittennt)
Hopper (random-Iniermiitee))
TD-MPC
1795.4 375.6
1795.4 4: 21-18
1447.2 = 694.8
1073.6 +: 157.1
Dreamer-v2
1648.2 117.5
758.1 116.4
1064.7 691.8
974.7 = 201.8
11D3-multistep
2673.6 + 3316.8
1359.7 + 258.3
2796.4 + 264.1
1211.6 + 168.5
MARS-TDS
2572.9 248.1
3762.7 +: 371.4
3105.7 = 412.6
2647.9 +: 204.8
Table 5: Comparison hetween MBRL and MFRL in intermittent control tasks, average of
The results in Table Shoow that the model-based reinforcement learning approach is significantly lower than our approach in all four scenarios, and even slightly worse than using TD3 for direct multi-step decision making. We analyze that this is because the errors caused by the mismatch (sub-optimal) of the dynamic model will accumulate due to multi-step decision-making, resulting in sub-optimal policy. Unfortunately, the training of dynamic models is data hungry (high cost), that is, a large amount of high-quality expert data is required to ensure the accuracy of the model shop, which is difficult to obtain, especially in real-world scenarios.
C.2
Validation of the combination of MARS and online methods
300
Hopper
Walker
(d) HailCChetah
Figure 7: The performance of the methods on four simulated tasks. The curve and shade denote the mean and a standard deviation over 5 runs.
We use the mainstream online reinforcement learning algorithm TD3 in combination with MARS and compare it with the baseline mentioned in Sec 5.1 in four tasks. We set the interval to 10 time steps, requiring the policy to generate an effective action sequence Oc:+4 based on the received state 8t-
For all tasks, set the dimension of 2, 12 and the scaling parameter 3 to 4. We set the warm-up (stage I) step to 300000 and 100000 for the Mujoco tasks and the navigation task respectively. The results in Figure show that MARS-TDS outperforms the other baselines in all fixed Intermittent- MDP tasks and achieve comparable performance with perfect- TD3 in most tasks. This further proves that MARS can effectively improve the effectiveness of Online DRL fixed Intermi tasks.
C.3 Generalization of MARS
We test MARS with popular RL methods on three tasks: Hopper, Walker, and hardMaze. To make the experiment fair, used the same parameters for all methods and implemented them based on public code. We use each RL algorithm to train on three tasks under the ideal setting and compare them with their corresponding improvement methods. To show the optimal score after the algorithm convergence, we train all the algorithm's 's 2000000 time steps. The results in Tab show that all methods can learn effective policies with the help of MARS and perform similarly to their ideal settings. The differences in scores are mainly due to the variation in performance the RL algorithms. In summary, MARS can be combined with different methods to provide reliable action space for solving Intermittent-MIDP as normal MDP with RI..
C.4 Details of Ablation study
15

---

Benchmarks
MARS-PPO
MARS-DDPO
MARS- TD33
Maze hard
256
0.7 1
243
2.5 1
311
16.3 1
Hopper
2851.4
13.5
1815.6
184.3
3384
53.1 1
Walker
3831.2
285.1
1032.7
201.9
4
4821.6
427.6 t
Table 6: The parameters of all methods are optimized by grid search. The results of applying MARS to popular RL algorithms three random interaction interval tasks. The maximum interaction interval is set to 8. Each data in the table is in the following format: MARS-RL score the score difference compared the perfect dense interaction baseline. &. denotes score of MARS lower than the dense interaction baseline. denoles the score of MARS higher. All scores are averaged over 5 runs.
18
42 130
(a) Fixed Intermittent MDP scenarios
(b) Random Intermittent MDP scenarios
Figure 9: The curve and shade denote the mean and standard deviation
We conducted two exper- iments to show how well the two mechanisms of MARS work together. Al- though the results of ran- domized Intermittent-MDD and fixed Intermittent-MDD are slightly different, the same conclusion can be de- rived: The green curves in Figure g demonstrate that the representation model with increased action transition scale is much better than the original VAE. This means that dy- namic transition potential can create latent action space by explicitly modeling the dependence between multi-step actions. The blue curves also show that VAE with state dynamic prediction is better than the original VAE because it represent action sequences that have similar environmental effects at close locations. Finally, the red curves show that the two mechanisms work well together in MARS, and combining them improves representation ability.
1100
550
100
800
610
ond
025
sin
on
1/2
12s
15
64
06
64
(a) Fix Intermittent- MDD Walker
(b) Random Intermittent-We Walker
Figure 8: Details of ablation study. The curve and shade denote the mean and standard deviation over runs.
C.5 Validity verification of multi-style interaction intervals
To further demonstrate the effectiveness of MARS in diverse intermittent control scenarios. For fixed interaction control tasks, we uniformly set the forbidden interaction duration and conducted four experiments on Hopper. The results in Figure 9(a) show that MARS can solve most tasks effectively and still guarantee good scores long intervals, significantly when the interval is too long (which is not common in real-world scenarios). We believe that this because VAE unable to effectively characterize excessively long sequences, leading to the failure of multi-step action space modeling.
the effectiveness MARS decreases
16

---

Esteet
Latere dim
Lason din
Latan dim 1n
Latezz dizs
(a) Results Hopper
(b) Results MAZE
Figure 10: The curve and shade denote the mean and a standard deviation over 5 runs.
addition, to observe the sensitivity of MARS to interaction intervals on random Intermittent-MDD tasks, we uniformly set the forbidden interaction duration and conducted four experiments Hopper. The results in Figure Ib) show that in random Intermittent-MDP scenarios, MARS performs well in both short and medium-interve scenarios. However, convergence changes slowly in the very long interval scenario, and the score is only half that of the medium interval task. Because MARS's representational capabilities
not perfect for modeling long action sequences for extremely long- spaced tasks (even if this setting rarely occurs real-workd scenarios). Therefore, the future, we hope to find more suitable representation models overcome this problem.
C.6 The influence of Latent action space dimension on algorithm effect
The representation space dimension of VAE an important hyperparameter. the latent space dimension is too low, large amount of original data information will be lost, resulting in invalid representation space. On the contrary, when the latent space dimension too large, the calculation amount of the model will be increased. To verify the sensitivity of MARS to latent space dimensions, we test it on two tasks with different original action dimensions. We set up four sets latent space dimensions for fixed Intermittent-MDP Hopper (interaction interval time step: 8, original action dimension: 3, so the action sequence dimension be modeled 24). The learning curve in Figure 10 a) shows that for raw data of such high dimensions, when the latent space dimension is set too low, the latent space information will be lost, resulting in the convergence failure of reinforcement learning policies. On the contrary, too high latent space dimension increases the complexity of reinforcement learning policy exploration.
In addition, set up four comparison experiments the 2dmaze task with a lower dimension of the original action sequence (interaction interval time step: 4, original action dimension: 2, so the action sequence dimension to be modeled is 8). The experimental results in Figure b) show that the suboptimal policy can be learned when the latent space dimension is low, because the original data dimension is low. So the low-dimensional latent space loses less information. The score increases as the latent space dimension increases. However, when the latent space dimension is too high, the score will drop significantly, which is because of the exploration difficulties brought by high dimennsoini latent space.
C.7 The influence of environment steps of warmup stage
In this section, we conduct some additional experimental results for further study of MARS from different perspectives: We provide the exact number of samples used in the warm-up stage (i.e., stage 1 in Algorithm Tin in each environment in Tab The number of warm-up environment steps about 5% ~ 10% the total environment steps in our original experiments. Moreover, we also conducted some experiments to further reduce the number of samples used in the warm-up stage (at
17

---

most 80% off). See the colored results in T MARS can achieve comparable performance with 3% samples the total environment steps.
Conclusion: The number of warm-up environment steps is about 5% ~ 10% of the total environment steps in our original experiments. The number of warmup environment steps be further reduced by at most 80% off (thus leading to < 3% of the total environment steps) while the comparable performance of our algorithm remains.
Environment
Warm-up steps (original)
Warm-up steps (new)
Total Env. Steps
Hopper Ant Walker
3219.1)
100000(0.02) (0.02)086..4 3086.4)
5000000
400000(0.08) 4305.7)
100000(0.02) 4025.6)
5000000
400000(0.08 4961.3)
100000(0.02
4792.6)
5000000
HalfCheetah 400000(0.08 6593.2)
100000(0.02
6071.2)
5000000
2dmaze-medium 2dmaze-hard 10.000 327.6)
100000
30000(0.025) 118,5)
1200000
35000(0.0292/296.1
1200000
Table 7:
The exact number of samples used in warm-up stage training in different environments. The column of 'original' denotes what is done in experiments; the column of 'new' denotes additional experiments conduct with fewer warm-up samples (and propertionally fewer warm-up training). For
each entry x. environment steps
number of samples (environment steps). y denotes the percentage number of
stops during the training process' and z denotes the corresponding performance of
namberr of lotal MARS-TDS
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations can be seen in Sec.6 of main paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
Please refer to Sec.3 of the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
The code link can be found in abstret, and the hyperparameter setting can be found in Appendix B.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: The code link can be found in abstret, and the hyperparameter setting can be found Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed reproduce the results. See the NeurlPS code and data submission guidelines (https //nisscc/ppbbl httppu/ic/guieee/oooooeoooooeeee
ssionpolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The experimental detail can be found in Sec. 5.1 of the main paper and Appendix B.I. The hyperparameter setting can be found in Appendix B.2.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results are accompanied by error bars, confidence intervals. We detail the calculation of error bars, running steps and the number of seeds in the text description of each figure.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The Experiments Compute Resources can be found in Appendix B.I.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/////eeurrs..ccc///////ttta
Answer: [Yes]
Justification:
We make sure to preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper is only about promoting technological innovation and does not have social impact
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
22

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Ours paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package, and state which version of the asset is used (include a URL).
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com commdaaasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "lW1DtabJQJ", "reviewer": "Reviewer_TqM1", "summary": "This paper addresses the issue of intermittent control problems, common in real-world scenarios where interactions between decision-makers and executors are disrupted due to unstable communication channels. These disruptions lead to bidirectional blockages, preventing agents from acquiring state information and transmitting actions, thus reducing the efficiency of reinforcement learning (RL) policies. The paper models this problem as an Intermittent Control Markov Decision Process and proposes a solution called Multi-step Action Representation (MARS). MARS encodes a sequence of actions into a compact latent space, enabling RL methods to optimize smooth and efficient motion policies. The experiments demonstrate that MARS significantly enhances learning efficiency and performance in both simulation and real-world robotic tasks compared to existing baselines.", "questions": "- Line 181: What's the upper limit of action change?\n- Eq 3 defines how the action transition scale is computed, but in Figure 3, why does the policy need to output the action transition scale?\n- You denote the reconstruction layer as $g_{\\psi)1}$. What does \"1\" in the subscript mean? This is confusing since you use subscript to denote timestep as well. Similarly, what's the purpose \"2\" of $h_{\\psi_2}$?\n- Line 245: It's confusing to say \"choosing optimal z\" since you just sample z from a policy.\n- It seems relevant to https://arxiv.org/pdf/2304.13705. How do you compare with MARS and this work?\n- All the experiments in Figure 4 are conducted in the same time interval. How does the performance difference change over time interval?", "limitations": "Yes, it's discussed.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is, in general, well written, except for some confusing notation use. The problem studied in the paper is important but under-explored in RL literature, which in my opinion, makes this paper significant.", "weaknesses": "Some presentations, especially on notation use, are unclear. See my questions. Section 4 is unnecessarily long and consists of a lot of redundant text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your recognition of our paper's motivation, performance, and potential academic impact. Your positive feedback is highly encouraging. We improved our work with your valuable questions.\n\nIf you think the following response addresses your concerns, we would appreciate it if you could kindly consider raising the score.\n## Part 1\n### Questions\n**Q1: It seems relevant to \"Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware\" (https://arxiv.org/pdf/2304.13705). How do you compare with MARS and this work?**\n\nThanks for sharing this related research, the paper you provided further motivates us to improve our work. And in the new submission, we discussed this question in detail.\n\nThe difference between the two works (ours is referred to as MARS, and related work is referred to as ACT) :\n\n- **Focus:** ACT addresses cutting-edge challenges in robotics: Can learning enable low-cost and imprecise hardware to perform these fine manipulation tasks? However, our primary objective is to develop a plug-in module that enhances the RL algorithm's proficiency in Intermitted MDP tasks.\n\n- **Training style:** ACT is an end-to-end supervised training method, and MARS is an unsupervised training method. \n\n- **Form of application:** ACT, *a multi-step decision model*, primarily leverages the generative capability of C-VAE and depends on high-quality expert data to enhance the model's multi-step decision-making proficiency through imitation learning. MARS, *a multi-step action space construction model*, primarily utilizes the latent space construction capability of C-VAE to build the action space using low-quality random transition data. Subsequently, it aids in the training of RL-style training. Therefore, it is more suitable for scenarios where reinforcement learning excels.\n\n- **Technic:** ACT creatively introduces action chunking and temporal ensemble to address the compounding errors associated with imitation learning in a manner that aligns with pixel-to-action policies. MARS, on the other hand, assists in action space construction by introducing action transition scale and state residual guidance.\n\nAlthough there are significant differences between the two methods, ACT inspired us in two points:\n\n*We found ACT to be a dependable and valuable baseline, and we included it in the main experiments of the new submission.*\n- Based on the code provided in the paper, we migrated it to our setup. Initially, we utilized PPO to gather expert data for fully supervised training. We set the chunking number to the maximum interval for our task and configured the Temporal Ensemble to the recommended value in the paper,i.e. 4. Following the paper's suggestions, we trained using L1 loss.\n- Benchmarks: Consistent with the Mujoco tasks in the original version.\n- We find that ACT performs better than the original baselines we compare, but underperforms our method on most tasks.\n  \nTable 1. Performance score of random Intermitted MDP setting (Average of 4 runs):\n| Method  |Ant-v2|Walker2d-v2| HalfCHeetah-v2| Hopper-v2|\n| :-----------: | :-----------: | :------------: | :-----------: | :-----------: | \n| **Ours**|$4354.61\\pm 158.44$|$5436.42\\pm 217.83$|$6175.63\\pm 273.95$|$2613.58\\pm 177.96$|\n| ACT|$3279.82\\pm 127.83$|$4892.18\\pm 383.07$|$5811.51\\pm 108.33$|$2684 .31\\pm 238.27$|\n| frameskip TD3|$492.53\\pm 31.95$|$2584.18\\pm 106.24$|$3281.45\\pm 139.42$|$381.74\\pm 53.67$|\n| Multi-step TD3|$408.25\\pm 31.76$|$529.26\\pm 68.21$|$112.73\\pm 17.82$|$1173.93\\pm 78.28$|\n\n*ACT inspired us to employ a transformer architecture (similar to a BERT-like training style) instead of an MLP to construct the C-VAE. This transition is expected to enhance the representation capabilities of MARS in future work.*\n- We conducted a set of experiments on the Ant-v2, and we observed that the transformer-based MARS shows promise in enhancing RL algorithms and shows a more significant increase in representation ability when the *interval time step* $c$ becomes longer.\n    \nTable 2. Performance (Average of 4 runs):\n| Method      | c=4 | c=8 | c=12 |c=16 | c=20 |\n| :-----------: | :-----------: | :------------: | :-----------: |:-----------: |:-----------: |\n|Transformer based |$5281.771\\pm 231.59$|$5417.26\\pm 193.18$|$5513.47\\pm 337.52$|$5604.31\\pm 246.52$|$5337\\pm 114.23$|\n| MLP based|$5309\\pm 143.85$|$5283\\pm 171.46$|$5309\\pm 143.85$|$5194\\pm 201.52$|$4758\\pm 106.73$|\n\n**Q2: All the experiments in Figure 4 are conducted in the same time interval. How does the performance difference change over time interval?**\n\nThanks for your valuable advice. In the appendix of the new version, we tested MARS with varying interval step $c$ in the Walker2d environment. The other environmental parameters and network hyperparameters remained consistent with the main experiment.\n\nTable 3. Performance (Average of 4 runs):\n| Method      | c=4|c=8|c=12|c=16|c=20|c=24|\n| :-: | :-: |:-: |:-: |:-: |:-----------: |:-----------: |\n| MARS|$5309\\pm 143.85$|$5283\\pm 171.46$|$5309\\pm 143.85$|$5194\\pm 201.52$|$4758\\pm 106.73$|$4514\\pm 377.25$|\n\nThe Above results reveal a diminishing performance of MARS as the decision step size increases beyond $c=20$. We attribute this trend to the limitations in representation capacity imposed by the MLP architecture in the VAE. In the future, we plan to investigate alternative effective networks like Transformers to enhance the construction capabilities of MARS within action spaces under very long interval step setting.\n\n**Q3: Line 181: What's the upper limit of action change?** \n\nWe covered this in detail in the new version. The upper action limit $B$ varies according to each task. The semantics is the maximum scale that an action can change. For example, The range of actions in mujoco is $[-1,1]$, then $B=1- (-1)$.\n\n**The second part of the response is in the following comment. We are grateful for your thorough and conscientious reviewing.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8n4P2eZSUG", "reviewer": "Reviewer_5oZS", "summary": "The paper introduces Multi-step Action RepreSentation (MARS) to address intermittent control problems in reinforcement learning. Intermittent control refers to situations where the interaction between the decision maker and the executor is discontinuous due to interruptions or communication issues. MARS encodes a sequence of actions into a compact and decodable latent space, allowing RL algorithms to optimize and learn smooth and efficient motion policies. Experiments are conducted on both simulated and real-world tasks, demonstrating that MARS significantly improves learning efficiency and performance compared to existing baselines.", "questions": "* Why does the input of the encoder $q_\\phi$ include $s_{t:t+c}$? According to the description of the intermittent control task (Line 31: agents are unable to acquire the state sent by the executor), isn't $s_{t:t+c}$ inaccessible to the agent?\n* Which variable is the Gaussian exploration noise added to, the decoded action sequence $u_t$, the latent variable $z_t$, or $v_t$?", "limitations": "The authors have stated in the Conclusion and Limitation section that representing long action sequences is a limitation and future direction of this work. However, there are no results suggesting that MARS suffers from long action sequences. It would be helpful to discuss more about this limitation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper addresses an interesting problem of intermittent control in RL, which has not been broadly studied in the RL community.\n* The concepts of action sequence representation and action transition scale in MARS sound interesting and effective to me.\n* The paper provides a thorough explanation of the MARS method, including the encoding and decoding process, the use of action transition scale, and the state dynamic prediction. The experiments include both simulation tasks and real-world robotic grasping tasks and demonstrate the effectiveness of MARS in improving learning efficiency and performance.\n* The paper is easy to follow.", "weaknesses": "* The comparisons in the experiment are with some simple and intuitive baselines. While the authors mention that no specific solution exists for this problem, it would still be helpful to discuss related work in other research fields.\n* The experiments include both simulated and real-world tasks, but overall the number of tasks is a bit limited. It would be beneficial to include more tasks to validate the effectiveness of MARS.\n\nMinors:\n* Figure 6 is not adequately described, e.g., what the task is and what Vanilla_VAE stands for.\n* Stage 2 in Algorithm 1 should be revised according to Line 276: The action representation model is also updated periodically in the second stage to make continual adjustments to the change of data distribution.\n* Eq. (3): $\\sum_{i=t}^{c-1} \\rightarrow \\sum_{i=t}^{c+t-1}$.\n* Line 230: $\\delta_{s_t,s_{t+1}} \\rightarrow \\delta_{s_t,s_{t+c}}$\n* Line 231: $p_{state}=h_{\\psi_2}\\circ p_\\psi \\rightarrow p_{state}=h_{\\psi_2}\\circ p_{\\psi_0}$\n* Line 222: becuase $\\rightarrow$ because", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply thankful to you for recognizing the presentation and originality of our work; this positive feedback is greatly encouraging. Furthermore, your objective advice motivates us to further improve this work.\n\nIf you think the following response addresses your concerns, we would appreciate it if you could kindly consider raising the score.\n## Part 1\n### Weaknesses\n**W1：The comparisons in the experiment are with some simple and intuitive baselines. While the authors mention that no specific solution exists for this problem, it would still be helpful to discuss related work in other research fields. It would be beneficial to include more tasks to validate the effectiveness of MARS.**\n\nThanks for your valuable suggestions. We add multiple methods in similar fields as baselines and verify the effectiveness of all the methods in more complex scenarios than the original version.\n\n**Baselines:** We select the latest multi-step decision-making fully supervised method ACT [1] from the robotics learning area, which requires us to build an expert dataset for it via ppo in advance; and two recent SOTA methods in the Delayed MDP domain, i.e. DCAC [2], State Augmentation (SA) [3] and delayed Dreamer[4] (Relax the Intermitted setting restrictions, allow such methods to additionally use dense rewards and delay priors at each step, and set the delay coefficient to the interaction interval). \n\n**Benchmarks:** For simulation environments, we further select four more challenging DeepMind Control (DMC) tasks focused on bionic robot locomotion: Dog Run, Dog Trot, Dog Stand and Humanoid Walk. DMC tasks demand coordinated movement from multi-joint bionic robots. Besides, two robotic arm control tasks in MetaWorld: Sweep Into and Coffee Push are used. \n\n**Metrics:** We evaluate methods in terms of performance and Smoothness (whether the motion is coherent and stable, invalid jitter and stagnation will reduce the score, please refer to Sec.5.2 for detail).\n\nTable 1a. Performance score of random Intermitted MDP setting (smoothness (%)) (Average of 4 runs):\n| Method      |Dog Run|Dog Trot| Dog Stand| Humanoid Walk|Sweep Into|Coffee Push|\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| **Ours**|$124.61\\pm 44.92 (75)$|$574.12\\pm 28.76 (88)$|$614.03\\pm 17.42 (73)$|$105.47\\pm 35.83 (82)$|$0.51\\pm 0.13 (68)$|$0.38\\pm 0.06(63)$|\n| ACT|$108.37\\pm 36.51 (72)$|$476.95\\pm 32.37 (79)$|$607.94\\pm 20.56 (75)$|$110.71\\pm 16.46 (76)$|$0.47\\pm 0.07 (57)$|$0.21\\pm 0.03 (69)$|\n| DCAC|$96.87\\pm 28.44 (53)$|$426.93\\pm 50.48 (72)$|$562.64\\pm 22.73 (64)$|$105.32\\pm 29.16 (49)$|$0.44\\pm 0.27 (41)$|$0.19\\pm 0.13 (48)$|\n| SA |$92.74\\pm 51.06 (37)$|$385.67\\pm 52.49 (39)$|$503.94\\pm 14.86 (27)$|$75.66\\pm 31.42 (45)$|$0.52\\pm 0.21 (37)$|$0.34\\pm 0.09 (39)$|\n| delayed Dreamer |$95.31\\pm 26.74 (46)$|$428.39\\pm 46.23 (46)$|$526.07\\pm 21.84 (25)$|$89.25\\pm 27.41 (41)$|$0.56\\pm 0.17 (32)$|$0.39\\pm 0.04 (37)$|\n\nTable 1b. Performance score of fixed Intermitted MDP setting (smoothness (%)) (Average of 4 runs):\n| Method      |Dog Run|Dog Trot| Dog Stand| Humanoid Walk|Sweep Into|Coffee Push|\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| **Ours**|$162.52\\pm 64.43 (82)$|$593.73\\pm 23.13 (85)$|$622.68\\pm 26.07 (79)$|$121.32\\pm 52.16 (84)$|$0.64\\pm 0.07 (73)$|$0.42\\pm 0.11 (72)$|\n| ACT|$127.23\\pm 29.33 (76)$|$493.56\\pm 48.27 (87)$|$627.31\\pm 14.83 (76)$|$103.21\\pm 19.35 (78)$|$0.51\\pm 0.09 (69)$|$0.34\\pm 0.06 (64)$|\n| DCAC|$94.42\\pm 19.24 (58)$|$451.92\\pm 27.06 (79)$|$568.07\\pm 38.26 (74)$|$113.73\\pm 22.24 (54)$|$0.54\\pm 0.05 (48)$|$0.28\\pm 0.14 (52)$|\n| SA |$92.31\\pm 26.36 (34)$|$405.78\\pm 74.19 (43)$|$562.84\\pm 34.69 (31)$|$83.73\\pm 28.41 (55)$|$0.47\\pm 0.13 (42)$|$0.29\\pm 0.16 (44)$|\n| delayed Dreamer |$103.71\\pm 73.49 (51)$|$485.93\\pm 95.27 (56)$|$541.58\\pm 39.46 (36)$|$94.61\\pm 22.31 (52)$|$0.66\\pm 0.13 (46)$|$0.43\\pm 0.145 (48)$|\n\nAbove results show that our method performs better than other methods in almost all intermitted MDP control tasks while ensuring smooth and coherent motion of the agent. The effect of the supervised learning method ACT outperforms the delayed MDP methods, and the delayed MDP methods perform well in the robotic arm scene but cannot maintain motion smoothness and time efficiency.\n\n**W2: Minors in writing**\n\nWe apologize for any inconvenience caused by our rough writing. We refined our grammar, spelling, formulas, and notations.\n### Questions\n\n**Q1: Which variable is the Gaussian exploration noise added to, the decoded action sequence $u_t$, the latent variable $z_t$, or $\\upsilon_t$?**\n\nWe add Gaussian perturbations to $z_t$ in this paper. Additionally, we analyzed each of the above methods on a random Intermitted Mujoco task and added it to the new submission.\n\nTable 2. Performance (Average of 4 runs):\n| Method | Ant-v2|\n| :-: | :-: |\n| $u_t$ perturbation|$5886\\pm 242.68$|\n| $z_t$ perturbation |$5908\\pm 194.61$|\n| $\\upsilon_t$ perturbation |$5397\\pm 206.84$|\n\nThe above results indicate that adding noise to $u_t$ and $z_t$ yields comparable effects. Furthermore, experiments suggest that adding noise to $\\upsilon_t$ has no good impact.\n\n[1]: Zhao, Tony Z., et al. \"Learning fine-grained bimanual manipulation with low-cost hardware.\" RSS 2023.\n\n[2]: Ramstedt, Simon, et al. \"Reinforcement learning with random delays.\" ICLR 2020.\n\n[3]: Nath, Somjit, et al. \"Revisiting state augmentation methods for reinforcement learning with stochastic delays.\" CIKM 2021.\n\n[4]: Karamzade A, Kim K, Kalsi M, et al. \"Reinforcement learning from delayed observations via world models.\"  PMLR 2024.\n\n\n**The second part of the response is in the following comment. We are grateful for your thorough and conscientious reviewing.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r9MbwZipHM", "reviewer": "Reviewer_Bn1i", "summary": "This paper proposes a new method for RL problems that needs to propose a sequence of actions at each step due to latency of the environment. The method is straightforward in that it uses VAE to encode the action space of a consecutive of actions. Empirical results show that this simple technique improves the performance and sample efficiency.", "questions": "1. Why define intermitten MDP separately while delayed MDP is widely adopted in the community?\n2. How does the method compare to existing delay MDP methods?", "limitations": "See weakness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. The method is simple but the performance improvement is significant.", "weaknesses": "The setting looks like the delay MDPs but is simpler as the observation is not delayed. The paper does not compare with methods for delay MDPs. The method is straightforward and easy to think of.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to you for recognizing the importance of our research. Your suggestions inspired us to improve our work, we analyzed the difference between our setting (intermittent MDP) and Delayed-MDP through demo examples and detailed explanations in the revised version. Additionally, we compared our method with several SOTA delayed MDP methods on multiple complex tasks in the main experiment section (4 more robotic motion control tasks and 2 more manipulation tasks). \n\nIf you think the following response addresses your concerns, we would appreciate it if you could kindly consider raising the score.\n## Part 1\n### Questions\n**Q1: Compare Intermitted MDP and Delayed MDP: Why define intermittent MDP separately while delayed MDP is widely adopted in the community?**\n\nWe described our setting more clearly in the new version. Both the Intermittent MDP and the traditional Delayed MDP models aim to represent non-ideal (unstable) environments. However, the tasks differences that both focus on also make the following distinction：\n\n**Reward feedback:** Mostly, in delayed MDP, the reward associated with each state-action pair can be acquired independently (**dense reward**)[1]. However, Intermittent MDP regards reward feedback as the cumulative rewards of a sequence of actions (**sparse reward**), a depiction that aligns more realistically with scenarios in practice. \n- We compare the performance of the delayed MDP methods with our method on Ant-v2 when dense rewards are not available (simulate lost state by repeating the last reward in the sequence). The results show that our method is more effective in dealing with fuzzy reward feedback. We use two recent SOTA delayed MDP methods, i.e. DCAC[2] and State Augmentation(SA)[3], as representatives of this class of methods. The setup is the same as that in the main experiments.\n    \nTable 1. Performance (Average of 3 runs):\n| Method      | Intermittent Control Ant-v2|\n| :-: | :-: | \n| Ours |$4354.61\\pm 158.44$|\n| DCAC  |$3281.93\\pm 544.17$|\n| SA |$3446.21\\pm 262.59$|\n\n**Motion mode requirements (described in Fig.1):** The real-world tasks that Intermittent MDP targets typically always involve high-frequency operational demands, e.g. game NPC control and robot control[4]. Thus, **Intermittent MDP expects the action to be executed at every time step to guarantee smooth and stable motion. In contrast, delayed MDP does not incorporate this constraint and permits certain time steps to be non-execution[3].** For instance, in a scenario where a robot needs to continue walking despite a blocked interaction channel, it cannot afford to wait for the next state to be successfully transmitted before taking action. Delaying the decision in such cases could result in the movement abruptly halting within a specific timeframe, leading to a loss of balance and a potential fall. So the key in the above task is not solely on the efficiency of some particular executing actions. Rather, the emphasis lies on ensuring that each action smoothly transitions with its neighboring actions while maintaining validity. \n- We compared the motion smoothness between our method and delayed MDP methods in the Humanoid scenario,  Humanoid requires smooth fine-tuning at each time step to maintain the balance of the torso effectively. We use the Smoothness rate to measure the motion (whether the motion is coherent and stable, invalid jitter and stagnation will reduce the score, please refer to Sec.4.2 for details). The results show that MARS can make the movement of the executor smoother and more stable.\n\nTable 2. Performance (smoothness rate (%)) (Average of 3 runs):\n| Method| Intermittent Control Ant-v2|\n| :-: | :-: | \n| Ours |$4354.61\\pm 158.44$ (78)|\n| DCAC  |$3495.68\\pm 244.51$ (42)|\n| SA |$3108.29\\pm 194.35$ (51)|\n\n**The information used for decisions:** Delayed MDP methods typically involve accessing prior information, such as the time to be delayed or even intermediate states, to enhance decision-making [2]. Our constraint is more stringent, where the agent is only permitted to make advanced decisions based on the current state.\n- We compare the effect of the delayed MDP method without auxiliary information (simulate lost state by using zero mask) and our method, on the Intermitted control task. The results show that our method is somewhat more robust to sparse information.\n    \nTable 3. Performance (Average of 3 runs):\n| Method| Intermittent Control Ant-v2|\n| :-: | :-: | \n| Ours |$4354.61\\pm 158.44$|\n| DCAC  |$2795.68\\pm 485.83$|\n| SA |$3371.46\\pm 169.27$|\n\n**Decision step number per step:** The delay scenario addressed by delayed MDP involves brief durations of delay (statistics from [2] indicate that the majority of delays are less than one second), making the future decision action length relatively short in this context. In contrast, our setup necessitates accounting for instances where the communication channel may remain non-functional for an extended time due to channel breakdowns. Therefore, Intermittent MDP method must consider longer durations (in real scenarios, there could be intervals exceeding 5 seconds [4]) in our deliberations, i.e. Deciding on a lengthy action sequence in a single step.\n- We tested the performance with various single-step decision action sequence lengths for the corresponding methods of the two MDPS in the Walker2d scenario (DCAC is the easiest method to change to a multi-step decision mode in delayed MDP). Tab. 4 shows that MARS performs better in the long decision sequence setting. \n\nTable 4. Performance (Average of 3 runs):\n| Method| Decision step $c=4$|$c=8$|$c=12$|$c=16$|\n| :-: | :-: |:-: |:-: |:-: |\n| MARS|$5309\\pm 143.85$|$5283\\pm 171.46$|$5309\\pm 143.85$|$5194\\pm 201.52$|\n| DCAC|$5134\\pm 118.64$|$4796\\pm 208.72$|$2962\\pm 571.07$|$2836\\pm 485.11$|\n\n**Due to the word limit, the reference and second part of the response are in the following comment. We are grateful for your thorough and conscientious reviewing.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Response
We sincerely appreciate all reviewers for their meticulous assessment and valuable insights to our paper. Special thanks to all three reviewers for their thorough and meticulous review of our submission.

Please permit us to present the additional primary experiment results here. Taking into account the recommendations from reviewer Bn1i and reviewer 5oZS, We have conducted a comparison of the performance of our method with several **additional baselines** chosen from the Delayed MDP and robotic learning domains across **more challenging scenarios.**

### Question: How does the method compare to related works in other areas, e.g. delay MDP methods?

We add a comparison of our method with mainstream delayed MDP methods on multiple complex tasks from the perspective of multiple metrics in the new version.

**Baselines:** We select several recent SOTA methods in the Delayed MDP domain, i.e. DCAC [1] and State Augmentation (SA) [2] (Relax the Intermitted setting restrictions, allow such methods to additionally use dense rewards and delay priors at each step, and set the delay coefficient to the interaction interval). For a more comprehensive analysis, a recent model-based approach,i.e. delayed Dreamer [3], is also chosen. 

**Benchmarks:** We further select four more challenging DeepMind Control (DMC) tasks focused on bionic robot locomotion: Dog Run, Dog Trot, Dog Stand and Humanoid Walk. DMC tasks demand coordinated movement from multi-joint bionic robots. Besides, two robotic arm control tasks in MetaWorld: Sweep Into and Coffee Push are used. The environmental parameters and network hyperparameters remained consistent with the main experiment. For methods that require an expert dataset, we use the trained PPO to collect data in an ideal setting environment.

**Metrics:** We evaluate methods in terms of performance and smoothness (whether the motion is coherent and stable, invalid jitter and stagnation will reduce the score, please refer to Sec.5.2 for details).

Table 1. Performance in newly added difficult tasks (smoothness (%)) (Average of 4 runs):
| Method      |Dog Run|Dog Trot| Dog Stand| Humanoid Walk|Sweep Into|Coffee Push|
| :-----------: | :-----------: | :------------: | :-----------: | :-----------: | :-----------: | :-----------: |
| **Ours**|$124.61\pm 44.92 (75)$|$574.12\pm 28.76 (88)$|$614.03\pm 17.42 (73)$|$105.47\pm 35.83 (82)$|$0.51\pm 0.13 (68)$|$0.38\pm 0.06(63)$|
| DCAC|$96.87\pm 28.44 (53)$|$426.93\pm 50.48 (72)$|$562.64\pm 22.73 (64)$|$105.32\pm 29.16 (49)$|$0.44\pm 0.27 (41)$|$0.19\pm 0.13 (48)$|
| SA |$92.74\pm 51.06 (37)$|$385.67\pm 52.49 (39)$|$503.94\pm 14.86 (27)$|$75.66\pm 31.42 (45)$|$0.52\pm 0.21 (37)$|$0.34\pm 0.09 (39)$|
| delayed Dreamer |$95.31\pm 26.74 (46)$|$428.39\pm 46.23 (46)$|$526.07\pm 21.84 (25)$|$89.25\pm 27.41 (41)$|$0.56\pm 0.17 (32)$|$0.39\pm 0.04 (37)$|

Table 2. Performance in old tasks (smoothness (%)) (Average of 4 runs):
| Method  |Ant-v2|Walker2d-v2| HalfCHeetah-v2| Hopper-v2|
| :-----------: | :-----------: | :------------: | :-----------: | :-----------: | 
| **Ours**|$4354.61\pm 158.44$ (78)|$5436.42\pm 217.83$ (82)|$6175.63\pm 273.95$ (76)|$2613.58\pm 177.96$ (83)|
| DCAC |$3279.82\pm 127.83$ (42)|$4892.18\pm 383.07$ (54)|$5811.51\pm 108.33$ (58)|$2684 .31\pm 238.27$ (63)|
| SA |$492.53\pm 31.95$ (51)|$2584.18\pm 106.24$ (59)|$3281.45\pm 139.42$ (66)|$381.74\pm 53.67$ (71)|
| delayed Dreamer|$408.25\pm 31.76$ (35)|$529.26\pm 68.21$ (63)|$112.73\pm 17.82$ (68)|$1173.93\pm 78.28$ (78)|

Table 1,2 show that our method performs better than Delayed MDP methods in almost all intermitted MDP control tasks while ensuring smooth and coherent motion of the agent. 

*P.S. If there are any other Delayed MDP methods that you believe should be compared but fall outside the scope of our current investigation, please inform us, and we will gladly incorporate them into our experimental analysis.*


[1]: Ramstedt, Simon, et al. "Reinforcement learning with random delays." ICLR 2020.

[2]: Nath, Somjit, et al. "Revisiting state augmentation methods for reinforcement learning with stochastic delays." CIKM 2021.

[3]: Karamzade A, Kim K, Kalsi M, et al. "Reinforcement learning from delayed observations via world models." PMLR 2024.

Meta-review: The paper addresses an important practical problem (intermittent loss of communication in  control systems) in a principled and interesting way. All reviewers are favourable and the authors did a great job during the rebuttal process of allaying concerns.
Decision: Accept (poster)