Decision Transformer under Random Frame Dropping
Authors: Kaizhe Hu
Ray Chen Zheng
Yang Gao
Huazhe Xu
Venue: ICLR 2023 poster (2023)
Keywords: Decision Transformer
Reinforcement Learning
Frame Dropping

Abstract: Controlling agents remotely with deep reinforcement learning~(DRL) in the real world is yet to come. One crucial stepping stone is to devise RL algorithms that are robust in the face of dropped information from corrupted communication or malfunctioning sensors. Typical RL methods usually require considerable online interaction data that are costly and unsafe to collect in the real world. Furthermore, when applying to the frame dropping scenarios, they perform unsatisfactorily even with moderate drop rates. To address these issues, we propose  Decision Transformer under Random Frame Dropping~(DeFog), an offline RL algorithm that enables agents to act robustly in frame dropping scenarios without online interaction. DeFog first randomly masks out data in the offline datasets and explicitly adds the time span of frame dropping as inputs. After that, a finetuning stage on the same offline dataset with a higher mask rate would further boost the performance. Empirical results show that DeFog outperforms strong baselines under severe frame drop rates like 90\%, while maintaining similar returns under non-frame-dropping conditions in the regular MuJoCo control benchmarks and the Atari environments. Our approach offers a robust and deployable solution for controlling agents in real-world environments with limited or unreliable data.

Published as a conference paper at ICLR 2023
DECISION TRANSFORMER UNDER RANDOM FRAME DROPPING
Kaizhe Hu", Ray Chen Zheng' Tsinghua University, Shanghai Qi Zhi Institute hkz228mails htt.emaiissstiiuusa...aa edu.
Yang Gao, Huazhe Xu
Tsinghua Universtiy, Shanghai Al Lab, Shanghai Qi Zhi Institute
ABSTRACT
Controlling agents remotely with deep reinforcement learning (DRL) in the real world yet come. One crucial stepping stone is to devise RL algorithms that are robust in the face of dropped information from corrupted communication malfunctioning sensors. Typical RL methods usually require considerable online interaction data that are costly and unsafe to collect in the real world. Further- more, when applying the frame dropping scenarios, they perform unsatisfacto- rily even with moderate drop rates. To address these issues, propose Decision Transformer under Random Frame Dropping (DeFog), an offline RL algorithm that enables agents to act robustly in frame dropping scenarios without online interaction. DeFog first randomly masks out data in the offline datasets and ex- plicitly adds the time span of frame dropping as inputs. After that, a finetuning stage on the same offline dataset with a higher mask rate would further boost the performance. Empirical results show that DeFog outperforms strong baselines under severe frame drop rates like 90%, while maintaining similar returns under non-frame-dropping conditions in the regular MuJoCo control benchmarks and the Atari environments, Our approach offers a robust and deployable solution for controlling agents in real-world environments with limited unreliable data.
INTRODUCTION
Imagine you are piloting a drone on a mission to survey a remote forest. Suddenly, the images transmitted from the drone become heavily delayed even disappear temporarily due to poor com- munication. An experienced pilot would use their skill to stabilize the drone based on the last received frame until communication is restored.
In this paper, we aim to empower deep reinforcement learning (RL) algorithms with such abilities control re- mote agents. In many real world control tasks, the deci- sion makers are separate from the action executor (Saha & Dasgupta 2018), which introduce the risk of packet loss and delay during network communication. Further- more, sensors such cameras and IMUs sometimes prone to temporary malfunctioning, or limited by hard- restrictions, thus causing the observation be available at certain timesteps (Dulac-Aroil et al. (2021). These examples lead to the core challenge devising the desired algorithm: controlling the agents against frame dropping, i.e., a temporary loss of observations as well other information.
3500
DeFog
3000
AgAgent
Average Retum
1500
1000
500
8
Figure 1: RL performance in the Hopper-v3 environment under different frame drop rates.
"qqual contribution

---

Published as a conference paper at ICLR 2023
Figure [ illustrates how regular RL algorithm performs under different frame drop rates. Our findings indicate that RL agents trained in environments without frame dropping struggle to adapt to scenarios with high frame drop rates, highlighting the severity of this issue and the need to find solution for it.
This problem gradually attracts more attention recently: Nath al. (2021) adapt vanilla DQN al- gorithm to a randomly delayed markov decision process; Bouteiller et al. (2020) propose a method that modifies the classic Soft Actor-Critic algorithm (Haamoja al.) 2018) handle observation and action delay scenarios. In contrast to the frame-delay setting in previous works, we try to solve more challenging problem where the frames are permanently lost. Moreover, previous methods usually learn in an online frame dropping environment, which can be unsafe and costly.
In this paper, we introduce Decision Transformer under Random Frame Dropping (DeFog), an of- fline reinforcement learning algorithm that is robust to frame drops. The algorithm uses Decision Transformer architecture (Chen et al. (02T) to learn from randomly masked offline datasets, and includes an additional input that represents the duration of frame dropping. In continuous control tasks, DeFog can be further improved by finetuning its parameters, with the backbone of the Deci- sion Transformer held fixed.
We evaluate our method on continuous and discrete control tasks in MuJoCo and Atari game envi- ronments. In these environments, observations are dropped randomly before being to the agent. Empirical results show that DeFog significantly outperforms various baselines under frame dropping conditions, while maintaining performance that are comparable to the other offline RL methods in regular non-frame- dropping environments.
RELATED WORKS
2.1 CONTROL UNDER FRAME DROPPING AND DELAY
The loss or delay of observation and control is an essential problem in remote control tasks (Balemi Brunner 1992, Funda Paul 1991). In recent years, with the rise of cloud-edge computing systems, this problem has gained even more attention in various applications such as intelligent connected vehicles (Li et al. 2018) and UAV swarms (Bekkouche et al. 2018
When reinforcement learning is applied to such remote control tasks, robust RL algorithm is desired. Katsikopoulos & Engelbrecht (2003) first propose the formulation of the Random Delayed Markov Decision Process. Along with the new formulation, method is proposed to augment the observation space with the past actions. However, previous methods [Walsh et al. 2009 Schuitemal al. 2010) usually stack the delayed observations together, which leads to expanded observation space and requires a fixed delay duration as a hard threshold.
Hester & Stone (2013) propose predicting delayed states with random forest model, while Bouteiller et al. 2020) tackle random observation and action delays in a model-free manner by rela-
belling the past actions with the current policy to mitigate the off-policy problem. Nath et al.) (2021) build upon the Deep Q-Network (DQN) and propose a state augmentation approach to learn an agent that can handle frame drops. However, these methods typically assume a maximum delay span and are trained in online settings. Recently, Imai et al. (2021) train vision-guided quadrupedal robot to navigate in the wild against random observation delay by leveraging delay randomization. Our work shares the same intuition of the train-time frame masking approach, but we utilize Decision Transformer backbone with a novel frame drop interval embedding and a performance-impren finetuning technique.
2.2 TRANSFORMERS IN REINFORCEMENT LEARNING
Researchers recently formulate the decision making procedure in offline reinforcement learning as a sequence modeling problem using transformer models (Chen et al.) 2021 Janner et al.) (021) In contrast to the policy gradient and temporal difference methods, these works advocate the paradigm of treating reinforcement learning predicting actions from the observation sequence and the task specification. The Decision Trans- former model (Chen et al. 2021) takes the encoded reward-to-go. state, and action sequence as
supervised learning problem (Schmidhuber) 2019). directly
2

---

Published as a conference paper at ICLR 2023
input to predict the action for the next step, while the Trajectory Transformer (Janner et al., 2021 first discretizes each dimension of the input sequence, maps them to tokens, then predicts the fol- lowing action's tokens with beam search algorithm.
The concurrent occurrence of these works attracted much attention in the RL community for further improvement upon the transformers. [heng al. (2022) increases the model capacity and enables online finetuning of the Decision Transformer by changing the deterministic policy to a stochastic one and adding an entropy term encourage exploration. [Tang & Ha (202T) train transformer-basee agents that are robust to permutation of the input order. Apart from these works, various attempts have been made to improve transformers in multi-agent RL, meta RL, multi-task RL, and many
other fields (Meng et al. 2021 Xu et al. 20222 Lee et al. 2022
Reid et al. 2022).
3 METHOD
In this section, first describe the problem setup and then introduce Decision Transformer under Random Frame Dropping (DeFog), a flexible and powerful method to tackle sporadic dropping of the observation and the reward signals.
3.1 PROBLEM STATEMENT
In the environment with random frame dropping, the original state transitions of the underlying Markov Decision Process are broken; hence, the observed states and rewards follow a new transition process. Inspired by the Random Delay Markov Decision Process proposed by Bouteiller et al.
2020 we define the new decision process as Random Dropping Markov Decision Process:
Definition 1 (Random Dropping Markov Decision Process (RDMDP))
An RDMDP could be described as M = (S,A,R. D, Os, OR), where S,A are the state and action space, P (st+1] St, 04) is the state transition possibility, R(st, ae) is the reward function, ((so) ) is the initial state distribution. 'D is the Bernoulli Distribution of frame dropping. Os is the function that emits the observation, and OR the function that emits the cumulative rewards. In the frame dropping setting, we assume the cumulative reward RG Et T2 is observed instead of the immediate reward rt.
At each timestep t. a drop frame indicator E {0,1) is drawn from the distribution D, with d 1 indicating that the frame is dropped and d = O the opposite. The observed state & and the cumulative reward RG of the timestep are updated by
8; = (8t 8 d) = d 8 + (1 (1- -d)
(1)
R. OR,,, Re- d. R (1 1--d) d) Ri
The observation and the cumulative reward repeats the last observed one s; and R respectively if the current frame is dropped. If the current frame arrives normally, the observation and cumulative reward is updated. Note that the rewards are cumulated on the remote side, so the intermediate rewards obtained during the dropped frame are also added. Following the definition in the Decision Transformer where target return Rearget is set for the environment, we define the real and observed
reward-to-go as St Rarget RG and ÿ. Rurget R respectively.
The goal of DeFog is to extract useful information from the offline dataset so that the agent can act smoothly and safely in a frame-dropping environment.
3.2 DECISION TRANSFORMER UNDER RANDOM FRAME DROPPING
We choose Decision Transformer our backbone for its expressiveness and flexibility as an offline RL learner. To attack the random frame dropping problem, we adopt a three-pronged approach. First, we modify the offline dataset by randomly masking out observations and reward-to-gos during training, and dynamically adjust the ratio of the frames masked. Second, we provide a drop-span embedding that captures the duration of the dropped frames. Third, we further increase the robust- ness of the agent against higher frame dropping rates by finetuning the drop-span encoder and action predictor after the model is fully converged. A full illustration of our method is shown in Figure
3

---

Published as a conference paper at ICLR 2023
4113
Decision Transformer
Drop-Span Emb. ==0
P=0
==c
=
Timestep Emb.
9--11
Gc-1
9--1
8
d2
9t-1
8
Gc++1
ap++
Received
Dropped
Dropped
Received
Figure 2: Decision Transformer under Random Frame Dropping (DeFog). Reward-to -oogo and state are repeated from the previous steps if the current frame is dropped. Timestep and drop-span embeddings, indicating the timestep and number of consecutive frame drops, are added onto the encoded reward-to-go go and state before being sent the Decision Transformer backbone. Since actions are not dropped, only the timestep embeddings are added to the encoded actions. The DT backbone outputs the predicted action embeddings, which is passed through decoder obtain the predicted actions.
3.2. DECISION TRANSFORMER BACKBONE
The Decision Transformer (DT) takes in past K timesteps of reward-to-go It-K:t- observation St-K:t, and action Oę-K:t embedded tokens and predicts the next tokens in the same way as the Generative Pre-Training model (Radford et al. 2018). Let Φn O, and ø, denote the reward-to- go, state, and action encoders respectively, the input tokens obtained by first mapping the inputs to d' d-dimensional embedding space, then adding timestep embedding (t) to the tokens.
Let Ug., Us and No, denote the input tokens corresponding the reward-to-go, the observation, and the action of the timestep É respectively, and Eg.. Us, and Va, be their counterparts on the output token side. DT could be formalized as:
Ugs (g w()), Us. = (8())) + +((), ua. = (a(a:) w(I
(3)
Cge-K K Use- K Ca K. Ug Us. Ua, = DT Us- Ua.-kx-.... Ugg Uges Ua,)
(4)
Online decision transformer (ODT) by ZZhng et al. (2022) enables online finetuning of the decision transformer models. We adopt the ODT model architecture because it has larger model capacity. Following their work, we also omit the timestep embedding w() in the gym environments.
During training time, instead of directly predicting the action a,, we follow the setting of the ODT to predict a Gaussian distribution for action from the output token of the state token inputs.
To (at (ss) N (Jo (v.,): Eo (v,,))
(5)
The covariance matrix Eg assumed be diagonal, and the training target is minimize the negative log- likelihood for the model to produce the real action in the dataset T:
1
J(0) - B(axs))
log (ax Usa)
(6)
3.2.2 TRAIN-TIME FRAME DROPPING
To prepare the model for frame dropping, we manually mask out observation and reward-to-go from the dataset. During the training stage, we specify an empirical dropping distribution and periodically sample "drop-masks" from it. A drop-mask is binary vector of the same size as the
4

---

Published as a conference paper at ICLR 2023
dataset and serves the drop distribution D of RDMDP. If frame in the dataset is marked as "dropped" by the current drop-mask, the observation and reward-to-2 soard-to-go go of that frame are overwritten by the most recent non-dropped frame. We refer the time span between the current frame and the last non-dropped frame as the drop-span of that frame.
One key consideration for the training scheme is the distribution D of the drop-mask. A natural solution is to assume each frame has the same possibility pa be dropped, and the occurrence of dropped frame is independent. Under this assumption, the stochastic process of whether each frame dropped becomes Bernoulli process. Additionally, we guarantee that the first frame for each trajectory is not dropped. After a certain number of training steps, the drop-mask is re-sampled from D so that those dropped frames of the dataset could be used. We can also change 'D as the training proceeds, for example, to linearly increase the pa throughout training. However, we empirically find that usually, constant Pa is sufficient.
3.2.3 DROP-SPAN EMBEDDING
In a frame dropping scenario, the agent must deal with the missing observation and reward-to-gi tokens. Instead of dropping the corresponding tokens in the input sequence, repeat the last observation or reward-to-go token and explicitly add a drop-span embedding to those tokens apart from the original timestep embedding (t).
Let k denote the drop-span since the last observation of timestep t, the drop-span encoder & maps integer a d-dimensional token the same shape as the other observed input tokens. The model input with the drop-span embedding becomes:
U.S (6:) + (ke) w(t,, Ug. 6 + (kee) w(t), Ua, )a(ae) w(t)
(7)
Since the actions are decided and executed by the agent itself, they do not face the problem of frame dropping. The drop-span embedding is analogous to the timestep embedding in the Decision Transformer, but it bears the semantic meaning of how many frames are lost. Compared to the other indirect methods, the explicit use of drop-span embedding achieves better results. Detailed comparison could be found in Section 4.5
3.2.4 FREEZE-TRUNG FINETUNING
The combination of the train-time frame dropping and the drop-span embedding is effective in mak- ing model robust to dropped frames. However, we observed that in continuous control tasks, finetuning procedure can further improve performance in more challenging scenarios.
Inspired by recent progress prompt-tuning in natural language processing (Liu al. 2021 and computer vision (Jia et al. 2022). we propose a finetuning procedure called "freeze-truie finetuning" that freezes most of the model parameters during finetuning. The procedure involves finetuning the drop-span encoder y and the action predictor Ty after the model has converged.
During this stage, the training procedure is the same that of the entire model. We draw drop-masks from D to give the drop-span embeddings enough supervision, with the drop rate pa typically higher than in the main stage. While the number of training steps during this stage can be one-fifth of the main stage, the empirical results show that this procedure can improve the model's performance in higher dropping rates across multiple environments.
The whole training pipeline of our method could be found in Appendix A
4 EXPERIMENTAL RESULTS
In this section, we describe our experiment setup and analyze the results. We compare our method with state-of-the-art delay-aware reinforcement learning methods, as well as online and offline rein- forcement learning methods in multiple frame dropping settings. We first evaluate whether DeFog is able to maintain its performance as the drop rate pa increases. We then explore which key factors and design choices helped DeFog to achieve its performance. Finally, we provide insights why DeFog can accomplish control tasks under severe frame dropping conditions.
5

---

Published as a conference paper at ICLR 2023
4.1 EXPERIMENT SETUP
To comprehensively evaluate the performance of DeFog and baselines, we conduct experiments on three continuous control environments with proprioceptive state ipputs in the gym MuJoCo environ- ment (Todorov et al. 2012], as well as three discrete control environments with high-dimensional image inputs in the Atari games.
In each of the three MuloCo environments, we use D4RL (Fu et al. 2020) which contains offline datasets of three different levels: expert, medium, and medium-replay While the three Atari environments, we follow the Decision Transformer to train DQN agent's replay buffer (Agarwal et al. 2020). We train on seeds and average their results in test time. We leave the detailed description of the settings to Appendix B.2
average sampled dataset from
During ovaluation, we test the agents in an environment that has frame drop rates ranging from 0% to 90%. Results are shown by plotting the average return under 10 trials against test-time drop rate for different agents. The performance curve of our method is compared against various baselines:
Reinforcement learning with random delays (RLRD; Bouteiller et al., 2020). RLRD is a method proposed to train delay-robust agents by adding randomly sampled delay well as an action buffer to its observation space. RLRD has a maximum delay constraint and is not suited for discrete action tasks like Atari. In our frame dropping setting, modify RLRD by limiting the delay value in the augmented observation to its maximum even if frames are still dropped. We compare our method RLRD in the gym MuJoCo environment.
Twin-delayed DDPG (TD3; Fujimoto et al. 2018). We train an online expert RL agent under regular taa-ffrme-droppiin settings using TD3 for continuous control tasks. We note that TD3 also has the privilege to interact with the environment.
Decision transformer (DT; Chen et al. 2021). We train the vanilla DT using exactly the same offline datasets without the proposed components Section 3.2
Batch-constrained deep Q-learning (BCQ; Fujimoto et al. 2019). BCQ is an offine RI. method that aims reduce the extrapolation error offinne RL by encouraging the policy to visit states and actions similar to the dataset.
TD3 + Behavioral cloning (TD3+BC; Fujimoto & Gu. 2021). TD3+BC is built on top of TD3 to work offline by adding a behavior cloning term to the maximizing objective. Despite the simplicity, it is able to match the tfate-of-the-an performance.
Conservative Q-learning (CQL; Kumar et al., 2020). CQL state-of-the-att model-free method which tries to address the issue of over-estimation in offline RL by learning conser- vative Q function that lower-bounds the real one.
We leverage the implementation of Takuma Seno (202T) to train an offline agent BCQ, TD3+BC, and CQL. We note that the online methods such RLRD and TD3 are trained directly in the environment. Hence, their performance are invariable to different dataset types, and their curves are plotted repeatedly for comparison. We also include DeFog without finetuning version to evaluate the effectiveness of freeze-trume finetuning. Since TD3 cannot handle Trame-dropphl scenarios, we only plot it in the first of the figures for better illustration. For fair comparison, delay for RLRD is created by re-sending the dropped observations, which again has a probability Pa of being lost. For baselines of the discrete control tasks, we train the offline RL. agents until they reach the performance of DeFog under non- non-frame- frame- trume-dropping conditions, as we aim to evaluate how these methods preserve their performances under frame dropping settings.
assume the
4.2 EVALUATION IN THE CONTINUOUS CONTROL TASKS
We first evaluate our performance on three MuJoCo continuous control environments, namely Hopper-v3, HalfCheetah-v3, and Walker2d-v3, The results each dataset given in Figure
We find that DeFog is able to maintain performance under severe drop rates. For example, the performance of the finetuned version the Walker2d-Expert dataset barely decreases when the drop rate is as high as 80%. Meanwhile, the performance of the vanilla DT and TD3 agents come close to zero the drop rate exceeds 67%.

---

Published as a conference paper at ICLR 2023
d-Expert
Hopper-Expert
Halfcheetah-Expert
5000
15.00
12000
DT
Average Return
1000
Average Retur
1000
Average Reture
2,000
befopflours)
DeFogtOuns
2500
2000
2000
TD3
6000
BCQ
2000
1000
1500
4000
TD 103
1000
500
0.10
0.85
0.80
0.00
Drop
Drop
Drop
Talker2d-Mediiue
Hopper-Mediue
Melicheetah-Mediiee
1500
3000
Average Reburn
3000
Average Return
Average Return
4000
DeFogrtOuust
2000
2500
2000
BCQ
TD3PusBC
1500
1000
1000
0.00
0.10
0.23
0.75
0.80
0.85
0.00
0.00
0.10
0.25
0.80
0.85
000
0.10
0.25
0.00
Frame Drop Rate
Frame Drop Rate
Frame Drop Rate
Walker2d-Medium Medium-Replay
Hopper- Medi-m-Mdiuuieeee Replly
Halfchectah- Replly
1000
3000
5000
DT
1500
DeFoptDura
Average Return
3000
Average Return
Average Return
Defogrt[Ours
2500
2000
2000
AIND
2000
1500
2000
1000
1000
1000
500
000
8,10
6.00
0.85
0.00
9.19
0.80
0.85
0.80
0.00
8.1d
0.00
0.65
Drop
Drop
Drop
Figure 3: Performance on continuous control tasks. There are five baselines: a) TD3: the online TD3 agent, only included the Expert datasets for better scaling. b) DT: the offline Decision Transformer agent trained on the same dataset as DeFog. c): RLRD: the online RLRD agent that optimized deal with random frame delay. Since it's an online method there's no performance distinction between three kinds of datasets. d) BCQ, TD3PlusBC: other offline methods trained on the same dataset as DeFog. The full-fledged version of our method is indicated with DeFog/f (Ours), while the result of non-finetuned version is indicated with DeFog (Ours) for comparison.
By looking the starting point of the performance curves, note that DeFog can achieve the the vanilla DT agent in non-frame-dropping scenarios. Despite a high train-
same performance time drop rate of 80% applied to DeFog, none of them are negatively affected when tested with drop rate of 0%. As a comparison, the online RLRD method failed to achieve the same non-frame- dropping performance as other online baselines.
In the HalfCheetah-Exxe Expert setting, our method significantly outperforms the RLRD baseline with drop rates lower than 50%; however, in more extreme cases the RLRD takes over. RLRD, with its advantage of accessibility to the environment, was able to keep the performance better possibly due HalfCheetah-Fxppee
dataset's narrow distribution. In the medium and medium-replay datasets, DeFog is limited to data with less expertise, thus obtaining reduced average return, but the overall performance of DeFog comparable to that RLRD.
We can also find that freeze trunk finetuning effectively improves DeFog's performance in various settings. In all 9 settings, the finetuned agent obtains better or at least the same results with the non-finetuned ones. The finetuning is especially helpful in high drop rate scenarios shown in the Hloperr-ediig and Walker2d-Expe Expert settings. We highlight that the finetuning is done over the offline dataset without online interaction well.

---

Published as a conference paper at ICLR 2023
4.3 EVALUATION THE DISCRETE CONTROL TASKS
In this section, we evaluate our performance on three discrete control environments of Atari (Belle- mare et al. 2013): Qbert, Breakout, and Seaquest. Following the practice of the Decision Trans- former, we use 1% of a fully trained DQN agent's replay buffer for training. The results are shown in Figure We find that DeFog outperforms the DT, BCQ and CQL baselines. We also find that in some environments, the performance of DeFog outperforms the Decision Transformer even under non man-frameddroppigg conditions. We believe that these environments, using masked out datasets not only helps the agent to be more robust to frame dropping, but also makes the task more challeng- ing in the sense that the agent needs to understand the environment dynamics better to give action predictions, which helps the agent make better decisions even when the frame drop rate is zero.
Breakout-
Qbert-Expert-Replay
Seaptees-Experr-Replay
60
1200
pF
Aveeage Return
6000
Setur
50
Average Return
1000
Defug[Ours)
5000
800
4000
20
600
3000
2000
400
1000
0.10
0.95
0.85
0.90
0.90
0.10
Frame Drop Rate
Frame Drop
Frame Drop Rate
Figure 4: The performance in the discrete Atari game environments. DT is the offline Decision Transformer agent trained on the same dataset as DeFog. There are two other offline baselines: BCQ and CQL. Our method is indicated with DeFog (Ours).
4.4 VISUALIZED RESULTS
To gain a better understanding of our method, we visualize the results in the MuloCo environment.
Frame Received
Frame Dropped
Figure 5: Visualization results of DeFog HalfCheetah agent under 90% frame drop rate. In each frame, the orange cheetah the observed state &,, while the purple cheetah is the actual state 8t-
We first visualize the performance of an DeFog agent under a frame drop rate of 90% Figure 5 The HalfCheetah agent (blue) is able to act correctly even if the observation (semi-transparent yel- low) is stuck at steps ago. Once new observation comes in, the agent immediately adapts to the newest state and continues to perform series of correct actions.
Building on top the previous setting, we aim to exploit the capability of DeFog under extreme conditions by increasing the frame drop rate to 100%. In this way, the agent is only able to look the initial observation and needs to make the rest the decisions blindly. As shown Fig- ure 6 the HalfCheetah agent continues run smoothly for more than 24 frames, demonstrating the Transformer architecture's 's ability to infer from the contextual history. We conjecture that such phenomenon is analogous to how humans perform skill such as swinging tennis racket without thinking about the observations.
4.5 ABLATION STUDY
We conduct ablation study the drop-span embedding and freeze-trunk finetuning parts of DeFog.
8

---

Published as a conference paper at ICLR 2023
Frame Received
Frame Dropped
Figure 6: Visualization results of DeFog HalfCheetah agent under 100% frame drop rate. Only the very first observation received. This scenario explores how far DeFog can go without any observation.
For the drop-span embeddings, we implement an alternative method that implicitly embeds the drop span information. Concretely, at each time step i, if the current frame is dropped, change the corresponding timestep emhedding w(t) to the received one w(t - kz). Hence, for the implicit em- bedding method, we have Us, 6s (s;) ( kt), Ug. (g(g)) +w k2), Ua (a(ae) + (t) In this way, the agent can infer the drop-span from the timestep embedding. As shown in Figure 7 we see that the proposed explicit drop-span embedding outperforms the implicit embedding, show- ing the effectiveness and necessity of explicitly providing the drop-span information to DeFog
Now we ablate the freeze-trunk finetuning method by comparing the the same model with and with- out the finetuning stage. As shown in Figure 7 the finetuning outperforms the original model on all the continuous tasks. We believe that the performance gain in complex continuous control tasks is due to the crucial modules (i.e., action predictor and drop-span encoder) further adjusting them- selves after the Decision Transformer backbone has converged. provide more ablation studies in Appendix C
Walker2d-Expert
Hlopeer-Medium MMeium
HalfcheetahhMMdiun
5000
3000
we freeze DeFoglOurs)
Average Return
Werraa veragge Return
wit deip
3000
1300
2000
1000
1300
0.30
0.50
0.30
0.00
0.30
0.99
0.30
Frame Drop Rate
Frame Drop Rate
Frame Drop
Rate
Figure 7: Ablation study of the dpop-span embedding and the finetuning stage. "w/o freeze" means without freeze-trunk finetuning, and "w/o drop" stands for without drop-span embedding. We find that the proposed methods are effective in the continuous control tasks.
CONCLUSION
In this paper, we introduce DeFog, an algorithm based on Decision Transformer that addresses critical challenge real-world remote control tasks: frame dropping. DeFog simulates frame drop- ping by randomly masking out observations in offline datasets and embeds frame dropping timespan information explicitly into the model. Furthermore, we propose a freeze-trunk finetuning stage to improve robustness high frame drop rates in continuous tasks. Empirical results demonstrate that DeFog outperforms strong baselines on both continuous and discrete control benchmarks under se- vere frame dropping settings, with frame drop rates as high 90%. We also identify promising future direction for research to handle corrupted observations, such as blurred images or inaccurate velocities, and to deploy the approach real robot.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program on New Generation Artificial Intelligence" (Grant No. 202IAAA0150000, This work is also supported by grant from the Guoqiang In- stitute, Tsinghua University.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
S Balemi and UA Brunner. Supervision of discrete event systems with communication delays. In 1992 American Control Conference, pp. 2794 2798. IEEE, 1992.
Oussama Bekkouche, Tarik Taleb, and Miloud Bagaa. Uavs traffic control based on multi-access edge computing. In 2018 IEEE Global Communications Conference (GLOBECOM), pp. 1-6. IEEE, 2018.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Yann Bouteiller, Simon Ramstedt, Giovanni Beltrame, Christopher Pal, and Jonathan Binas. Rein- forcement learning with random delays. In International conference on learning representations, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097 2021.
Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real- woaldworld reinforcement learning: definitions, benchmarks and analysis. Machine Learning, 14199)2441,-2468, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Confer- Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pp. 201 20132-20145, 2021. URL 2021/hash/ a8 waw.a88666600000 2ac.aa520009000000022222022202 aaw.:/aaaiaeeeeeeaaeeeeeeeee Abstract. htm_
neur.ps.....rr.................. cttpaper/
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2052-2062, PMLR, 09-15 Jun 2019. URL https::/prooeeeingg..tt .m...
press/v97 htt.://uummooo111000 html
J Funda and RP Paul. Efficient control of robotic system for time-delayed environments. In Fifth International Conference on Advanced Robotics' Robots in Unstructured Environments, pp. 219-224. IEBE, 1991.
10

---

Published as a conference paper at ICLR 2023
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off Off-policc maximum entropy deep reinforcement learning with stochastic actor. In Jennifer Dy and An- dreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1861-1870. PMLR, 10-15 Jul 2018. URL htRthhhtt://poooeeeee
ngs ml. wttp./..eee///////aaaaea jab.b.a.B...... html
Todd Hester and Peter Stone. Texplore: real-time sample-efficient reinffrccenn learning for robots. Machine learning, 90(3):385 2013.
Chieko Sarah Imai, Minghao Zhang, Yuchen Zhang, Marcin Kierebinski, Ruihan Yang, Yuzhe Qin, and Xiaolong Wang. Vision-guided quadrupedal locomotion in the wild with multi-modal delay randomization. arXiv preprint 1101:::209.14449 2021.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203. 1tt.:22223..11119 12119, 2022.
Konstantinos Katsikopoulos and Sascha Engelbrecht. Markov decision processes with delays and asynchronous cost collection. IEEE transactions on automatic control, 48(4):568-574, 2003.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179: 1791119-19 2020.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transform- ers. arXiv preprint arXiv: 15241, 2022.
Yongfu Li, Chuancong Tang, Srinivas Peeta, and Yibing Wang. Nonlinear consensus- based con- nected vehicle platoon control incorporating car-following interactions and heterogeneous time delays. IEEE Transactions on Intelligent Transportation Systems, 20(6))2209-22 2219, 2018.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:21 10.00221 2021.
Linghui Meng, Muning Wen, Yaodong Yang. Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks. arXiv preprint arXiv:2112.02845 2021.
Somjit Nath, Mayank Baranwal, and Harshad Khadilkar. Revisiting state augmentation methods for reinforcement learning with stochastic delays. In Proceedings of the 30th ACM International Conference Information & Knowledge Management, pp. 1346-1355, 2021.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? arXiv preprint arXiv: 12112, 2022.
Olimpiya Saha and Prithviraj Dasgupta. A comprehensive survey of recent trends in cloud
robotics architectures and applications. Robotics, 7(3), 2018. ISSN 2218-6581. doi: 10.3390/ robotics7030047. URL htttppa::/wwww.e// http://ww/mppptp www.. com/ 0tth/2228666811111 20118-6581/7/3444
Juergen Schmidhuber. Reinforcement learning upside down: Don't 't predict rewards just map them to actions. arXiv preprint arXiv:1912.0287. 2019.
Erik Schuitema, Lucian Buşoniu, Robert Babuška, and Pieter Jonker. Control delay in reinforcement learning for real-time dynamic systems: A memoryless approach. In 2010 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 3226-3231. IEFE, 2010.

---

Published as a conference paper at ICLR 2023
Michita Imai Takuma Seno. d3rlpy: An offline deep reinforcement library. In NeurlPS 2021 Offline Reinforcement Learning Workshop. December 2021.
Yujin Tang and David Ha. The sensory neuron as a transformer: Permutation-invariiat neural networks for reinforcement learning. Advances in Neural Information Processing Systems, 34:
22574-22587, 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1026-5033. IEEE, 2012. doi: 10.1 11109/RRSS.2022.38610000
Thomas Walsh, Ali Nouri, Lihong Li, and Michael L. Littman. Learning and planning in envi- ronments with delayed feedback. Autonomous Agents and Multi-Agent Systems, 18(1):83-105, 2009.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Con- ference on Machine Learning, pp. 24631-24644 PMLR, 2022.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In Kamalika Chaudhuri, Stefanic Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Pro- ceedings of Machine Learning Research, pp. 177022227057 PMLR, 17-23 Jul 2022. www.iiagsss............eeeeeeeee
URL
https://proceedinggs mlr, wrw.sresppeeeeeee v1 62 / 2hengg222hh
12

---

Published as a conference paper at ICLR 2023
A ALGORITHM DETAILS
The overall algorithm of DeFog could be summarized in Algorithm T for the hyperparameters we use, please refer to Appendix B.2
Algorithm Decision Transformer under Random Frame Dropping (DeFog)
Require: update_interval interval finterval to update the drop-mask)
Require: n_ n_updates, n. tltransitions (number of updates, number transitions in the dataset)
Require: sample(S, n) (sample elements uniformly from S}
Require: cumcount (aancoun(fflgs)) {count cumulative number of O's before current position (inclusive)}
Require: train(rtg, obs, act, timestep, dropspan, freeze_trunk) {train the DT model]
Require: batch size, act buffer, obs buffer, rtg buffer, context length for freeze trunk (true, false), do
+ n_updates
while N 0
.M < update
if freeze_trume then
M + [M/5]
end if
drop_mask <- sample( false); n_transitions) (whether each frame is dropped)
dropspans <- cumcount(drop (anncoun(ddrop_mas)) (osddroppssk)) (duration of frames dropped)
while . 0
selected index + sample((() I,.,, trantiition,, batch-size) size)
timestep indlcteedindex
dropspan - dropspans/selocted index]
dropped index selected index dropspan
rigs + rlg. rtg.buufer[drropee petiee/dddroppe_dddexx: Lindex: dropped Lindex + context_length
observations + bbfffer (rrppeddindeex dropped Lindex context
actions + act_ cenuffer(selecte__nneee selected_index + context_ trains observations, actions, timestep, dropspan, freeze freeze.trunk)
M t -
end while
N +- N -
end while end for
B EXPERIMENT DETAILS
B.1 DATASETS AND SETUP
B.I.1 GYM MUJoCO
We use the D4RL dataset (Fu al. 2020) that contains data collected by SAC agents. There three datasets for each environment expert, medium, and medium-replay. The expert dataset is collected by fully-trained expert policy, while the medium dataset is collected by agent about half the performance of the expert. Medium-replay includes the trajectories in a medium agent's buffer, and is the most diverse dataset with the lowest average return. Details of the different datasets are provided in Table
B.1.2 ATARI
For Atari game environments, we use the DQN Replay Dataset (Agarwal et al. 2020). which is collected from the replay buffer of a DQN agent during training of these Atari games. Following the practice of the Decision Transformer, we only dataset, which is 500 thousand the 50 million transitions observed by an online DQN agent. We define three kinds of datasets for each game well expert, medium, and expert-replay. The expert
small portion of the dataset: 1% of the whole
13

---

Published as a conference paper at ICLR 2023
Dalasel
No. Trajectories
No. Timesteps
Average Returns
Best Returns
Halfchectah-Exxa Expert
1000
1000 000
10656,43
11252.04
Halfcheetah-Melle Medium
1000
1000 000
4770.33
5309 039
Halfcheetah-Mil Mediiuu Replay
202
202 000
3093.29
4985. 14
Hopper-Exper
1027
999 494
3511.36
3759.08
Hopper-Medium
2186
906
1422.06
3222.36
Hupper--Medium Replay
2041
402 000
467.3
3192.93
Walker2d- Expert
1000
999 214
4920.51
5011.69
Walker2d-
1190
999 995
2852.09
4226.94
Walker2d-Medinne Replay
1093
302 000
682.7
4132
Table 1: D4RL Gym MuJoCo Datasets Sizes and Returns
and the medium dataset are collected from the DQN agent's buffer during the final and medium training stages, while the expert-replay dataset is sampled evenly from the whole replay buffer.
B.2
HYPERPARAMETER SETTINGS
B.2.1
GYM MuJoCo
For the gym MuJoCo environment, we use the same model architecture as the Online Decision Transformer (Zheng et al. 2002).. While the Online Decision Transformer uses different training parameters for each environment, we keep most of the training parameters the same among different environments.
Hyperparameter
Value
Hyperparameter
Value
Number of layers
4
Learning
le-4
Number of attention heads
Weight decay
1e-3
Embedding dimension
512
Batch size
256
Training context length K
20
Total training steps
le5
Dropout probability
0.1
Finetune training steps
2e4
Activation function
ReLU
Learning rate warmup steps
le4
Gradient norm clip
0.25
Drop-mask update interval
100
(a) Architecture Parameters
(b) Training Parameters
Table 2: Common Parameters for Gym MuloCo
For each dataset, we specify target reward, and report the combination train time drop-rate and finetuning drop-rate. The environment and dataset related paramenters as follows:
Enviromment
Dataset
Target Reward
Training Drop Rate
Finetuning Drop Rate
HalfCheetah
Expert
12000
0.5
0.5
HalfCheetah
Medium
12000
0.8
0.8
HalfChectah
Medium Replay
12000
0.8
0.8
Hopper
Expert
4000
0.9
0.9
Hopper
Medium
4000
0.5
0.8
Hopper
Medium Replay
4000
0.8
0.8
Walker2d
Expert
5000
0,9
0.9
Walker2d
Medium
5000
0.8
0.8
Walker2d
Medium Replay
5000
0.8
0.8
Table 3: Dataset Specific Parameters for Gym MuJoCo
B.2.2 ATARI
For the Atari environments. the same model architecture the Decision Transformer since the Online Decision Transformer doesn't 't perform experiments on these environments. The hyper- parameters are follows:
14

---

Published as a conference paper at ICLR 2023
Hyperparameter
Value
Hyperparameter
Value
Number of layers
Learning rate
6c-4
Number of attention heads
Weight decay
0.1
Embedding dimension
64
Batch size
128
Training context length K
30
Total training steps
le5
Dropout probability
0.1
Finetune training steps
2c4
Activation function
ReLU
Learning rate warm up steps
le4
Gradient norm clip
1.0
Drop-mask update interval
1000
(a) Architechture Parameters
(b) Training Parameters
Table 4: Common Parameters for Atari
The environment and dataset related parameters given in Table Linear increasing frame drop rate means that the drop rate is linearly increased from the start to end values.
Environment
Dataset
Target Reward
Training Drop Rate
Finetuning Drop Rate
Qbert
Expert Replay
14000
0.4
0.5
Seaquest
Expert Replay
1150
0-0.8 Linear Increase
0.8
Breakout
Expert Replay
90
0-0.8 Linear Increase
0.8
Table 5: Dataset Specific Parameters for Atari
C SUPPLEMENTARY RESULTS
In this section, we present further experimental results on the different components and settings of the DeFog model. Since we want to show the change the agent's performance as the frame drop rate increases, the results are presented by the performance curves of the average return against the frame drop rate. To make the results more descriptive but not overwhelming, the three most representative curves are selected for most of the settings, while the descriptions and analyses of the results are based on all the settings.
C.I DECISION TRANSFORMER BACKBONE
Training non-Decision Transformer Model on Masked-out Datasets DeFog simulates the frame dropping scenario by using a masked dataset with frames intentionally hidden from the agent. This is tightly integrated with our drop-span embedding in the DeFog model, as the drop-span infor- mation must be supervised and conveyed into the hidden representations. To determine the Decision Transformer architecture's contribution conduct an experiment with the TD3+BC (Fujimoto & Gu 2021) baseline to train on a masked dataset. We masking rate of 50%, which on par with lower than that
DeFog's 's strong results in frame dropping scenarios,
in DeFog.
The results are shown in Figure 8 The TD3+BC trained with a masked dataset is able to perform slightly better than the normal TD3+BC agent under higher frame drop rates in the Walker2d and Hopper environments. However, it collapses in the HalfCheetah environment. Although the average return improves slightly using masked dataset, still nowhere close the performance of DeFog. We believe this shows that the use of a masked dataset alone is not enough for DeFog's achievement.
Reconstruction of Frames During Training The Decision Transformer architecture can issue three different types of tokens, corresponding to the next action, state, and reward-to-go respectively. While the authors of Decision Transformer only let the model predict the actions. it may be helpful to infer the actual state when the observation is dropped. With this motivation, we conduct experiments to see if letting the model predict the actual state reward-to-ge has positive impact the model's performance. We evaluate the influence of predicting state, the reward-to- to-ooo and both on all nine settings of the MuJoCo tasks and report the results for three of them in Figure 9
15

---

Published as a conference paper at ICLR 2023
Walker2 delpperdd-Expett
Hopper-Expert
Halfcheetah-Expert
5000
12000
DT
Average Return
Average Retum
2000
Average Return
TD3HuaeC
DefogetOur))
2500
3000
2000
8000
6000
2000
1500
1000
1000
500
2000
0.10
0.85
0.80
0.00
Drop
Drop
Drop
Figure 8: Ablation study on training a non-Decision Transformer based method with masked-out dataset. "TD3PlusBC" is TD3+BC trained a perfect uncorrupted dataset, while "TD3PlusBCMssked" denotes training a TD3+BC agent with masked-our dataset.
We find it somewhat surprising that the performance of the model deteriorates significantly in four of the nine environments (HalfChectah-Expenn SetCheetah-Medium-- Replay) when only state prediction is applied. Only predicting the reward-to- go apart from the actions doesn't 't hinder the performance as much; predicting both of them doesn't affect the performance in general. We suspect this is due to the lack of supervision on the reward signal, which is exacerbated when the model is forced to predict both the state and action signals. In the original setting, where only the actions were predicted, and in the last setting, where all three tokens were predicted, this type of imbalance isn't as pronounced.
Malleer2d-MeduumRRppaaa Walker2d-Expert, and
Walker2d-Meeium--ee
Walker2d-Expee
Halfcheetah- Replay
3500
4000
5000
5000
DeFagrOurs) Rig Predict
Average Return
3000
Average Return
Average Return
2500
3300
3000
2000
1500
2000
2000
1000
500
0.90
0.50
Drop Rate
Figure 9: Ablation study on dropping the action together with the observation and reward-to-go
C.2 TRAIN-TIME FRAME DROPPING
In this section, we examine more carefully on the train-tine time frame dropping scheme, specifically the interval for resampling the drop-mask, the placeholder for dropped frames, the random process for generating the drop-mase mask, and the content to drop from the observation.
Frame Dropping Mask DeFog periodically samples and updates drop-mask that decides which frames in the dataset are marked as dropped. By doing so, DeFog can take advantage of the full dataset and avoid overfitting the current un un-masked dataset. To further explore the learning ability of DeFog, we conduct the experiment where the drop-mask never updates. In this way, the dropped frames which take up 50% to 90% of the dataset are never seen by DeFog during training.
Results in Figure show that the performance of DeFog without update is similar to the original version using the full dataset and still outperforms other baselines, implying that DeFog is able to learn even when the dropped frames are never seen. We note that the performance degrades on the medium-replay datasets of Halfcheetah and Hopper environments. One potential reason is the relatively small volume these datasets. As shown in Table I while the expert and medium datasets contain around 1M timesteps of data, the medium replay datasets have only 200k-4001 In the case Halfcheetah-Medcae Reelay, the number of non-dropped steps is only 8% of the Biichhetaa-MMddiue dataset.
16

---

Published as a conference paper at ICLR 2023
Walker2d-EEperr
Hopper-Expert
Halfcheetah--Ea EEExpert
2500
Average Retur
4000
Average Retum
2000
Average Return
10000
DefogiOursi
2500
8000
1000
2000
8000
2000
1000
0.90
0.90
Drop Rate
Drop
Frame Drop
Figure 10: Ablation study on fixing the frame drop-mask. "Fix Drop Mask" indicates fixing the drop-mask throughout training.
Placeholder for Dropped Frames During train-time frame dropping, if a frame marked as dropped, DeFog follows simple and intuitive approach replace both the observation and the reward-to-go of that frame to the most recent non-dropped ones. We explore the following substitu- tions for the dropped frames:
Adding noise to the dropped frames. This could be interpreted as stimulating the evolution of the unknown real states. For each step, we sample from a Gaussian noise distribution which is estimated from all the changes between consecutive observations in the dataset. When frames are dropped successively, the Gaussian noises add up form new Gaussian distribution. We use scale factor of 0.1 and 0.5 to experiment the influence of the noise intensity.
Simply replacing the dropped frames with zeros.
Replacing the embedding those dropped tokens specific learnable [MASK] token. We trial on two settings, one where the dropped observation and dropped reward-to-go share the same [MASK] token, and the other where the two tokens are separate.
The results are presented in Figure and 12
Walker2d-Expert
Hopper-Expert
Halfcheetah-Expert
5000
07
Average Retum
4.00
Average Retumm
Average Return
Add hobe 2.5x
DaPogtOurt
8000
Add Note: 0.1x
3000
8000
2000
4000
1000
500
2020
0,00
0.10
0.25
8.32
6.60
8,30
0.00
0.30
0.80
0.89
0,80
0.00
0.10
Frame Drop Rate
Drop
Drop
Figure 1I: Ablation study on adding noise to those dropped frames. "Add Noise 0.1x" and "Add Noise 0.5x" denote a noise scaling factor 0.1 and 0.5, respectively.
The results show that for adding noises, neither the scaling factor of 0. nor 0.5 helps with DeFog's performance. We find that increasing the noise intensity simply makes performance worse. In datasets such as Hopper-Medium and Walker-Expert, the deterioration is more noticeable.
In the case where we replace the dropped frames with learnable [MASK] tokens, both settings have performance better than vanilla Decision Transformer but worse than DeFog. We do not find this result surprising as single learnable mask cannot carry enough information for all the dropped frames, while the previous frame that DeFog would be similar to the current dropped frame.
Finally, replacing dropped frames with zeros does not result in a much better performance than the vanilla Decision Transformer, as the zero token basically provides information. Interestingly, the zero-mane masked version performs better than the carnable-toker version. When using zero tokens for the dropped observation and seward-to-pe to-go, the transformer backbone receives nothing more than the drop-span embedding, which turns out to better convey the information needed for control than that when added with learnalle token.
17

---

Published as a conference paper at ICLR 2023
Walker2d-Experr
Hopper-Expert
Ha Haffcheetth-EEpert
5000
3500
DT
Average Return
4.00
Average Retumm
3000
Average Return
12000
Detag(Gara)
2300
8800
3000
2000
2000
1500
4000
1000
1000
500
2000
0.00
030
025
0.50
0.65
0,90
0.00
030
0.5
0.65
0.50
625
823
640
frame Drop Rate
Frame Drop Rate
Frame Drop Rate
Figure 12: Ablation study on replacing the dropped frames with different kinds of [MASK] tokens. "Separate Mask" denotes that the observation and the reward-to-go do not share the same [MASK] token, while "Shared Mask" indicates the opposite. The "Zero Mask" simply consists of all zeros.
Frame Dropping Process The binary sequence of whether each frame dropped can be viewed as random process. In DeFog, use fixed drop rate pa as the probability for any single frame to be dropped, which results of dropping processes, we conduct experiments the setting of frame dropping being Markov process. The probability of the next frame being dropped no longer a constant value pa, but instead follows the transition matrix:
Bernoulli process for dropping frames. To explore other kinds
P
I
P1
P1 P2
1
P2
The matrix could be interpreted in the follow manner: given the current frame is not dropped, the probability for the next frame be dropped pr; if the current frame is dropped, then the probability for the next frame being dropped P2. The reason for choosing a Markov process is it resembles the behavior in communication scenes where frames dropped chunk by chunk rather than frame by frame. If p1 = p2, then the situation degenerates to a Bernoulli process.
Soalkerd-MMedim-aee Replay
Seoper-MMediure uu-ley
Halfcheetah- Meniittth--Meuumm -eplayy
3000
3500
Average Return
3000
2500
Average Return
2500
8219:0.67
2000
7319:0.75
2000
1500
1000
1300
1000
500
500
8.25
0.30
0.00
0.30
0.25
0.99
Frame Drop Rate
Frame Drop Rate
Frame Drop Rate
Figure 13: Ablation study on using a Markov Process for frame dropping. "0.5" and "0.9" represent
using a Bernoulli Process of ps 0.5 and pa 0.9. "8219:0.67" denotes P1 = 0.2, P2 = 0.9. with steady distribution of frame dropping probability 0.67. Similarly, "7319:0.75" means p1 0.3, P2 = 0.9 with a steady distribution of frame dropping probability 0.75.
Our experimental results, given in Figure 13 show that when comparing the Markov dropping pro- cess to the Bernoulli one, the agent trained under Markov dropping process with drop probability P2 performs similar to that with Bernoulli dropping process under Pa. and this pattern is somewhat universal no matter what P1 is. We find this result to abide with the fact that frame dropping setting, the moments where frames change P1, we find that in general the less P1 is, the better the performance. This is not surprising decreasing P1 would imply that there are more timesteps of consecutive undropped frames where the agent can leverage and make better decision.
dropped affect the overall performance more. If fix P2 and
Dropping the Action In the training of DeFog, we drop the observation and the reward-to-go of the frames marked by the drop-mask, while remaining the action of those frames untouched. We
18

---

Published as a conference paper at ICLR 2023
perform extra experiment where the actions masked out alongside the state and reward-to-ge and results show that the performance is negatively affected as shown in Figure 14
Walker2d-Medium
Hopper-Medium
Halfcheetah-Medium
3500
5000
Defog(Cunn)
Avera rege Return
3000
Averege Retum
2000
Avera Return
DetogtOrep Action)
2500
2000
3300
1500
1200
2000
1000
500
1500
500
0.00
Frame Drop Rate
Frame Drop Rate
Frame Drop Rate
Figure 14: Ablation study dropping the action together with the observation and reward-to-go.
C.3 DROP-SPAN EMBEDDING AND FREEZE TRUNK FINETUNING
Explicit Drop-Span Encoder and Finetuning Figure 15 contains the full ablation results of Fig-
ure 7 showing the performance of the ablated models on all datasets.
Wal kalker2d-Expert
Hopper-Expert
Halfcheetah Expert
3500
12000
Average Return
4002
Return
3000
2580
0000
2000
4000
1003
500
Drop
Drop
Walker2d MilkeerddMeeium
Hopper-Medium
Haifcheetah- Metittah-Meiiummm
5000
DT
freze
Avarage Retur
3000
Roerage Return
DeTeg
2500
2000
3000
wil drop
2003
Average
2000
1000
1000
1000
500
500
Drop
Walker2d- Minker?d-Nedium-ee Medi-M--ellullaa Replay
Hopper-edium-RRgee Redium-Reppay
Halfcheetah Meaetttah-Meiu---- wediiM--ddlllaaaeee
4003
Auaraped 5
3000
21:00
Average Retum
4000
DeT Dunn
freeze
2000
Average
15:00
3000
1300
1000
2000
1000
500
030
Drop
Figure 15: Ablation results on the explicit drop-span encoder and freeze-trunk finetuning all 9 gym MuJoCo environments. The label "w/o freeze" stands for without freeze-trunk finetuning, while "w/o drop" denotes using the implicit embedding method.
The use of explicit drop-span embedding was able to improve performance over implicit embedding by a huge margin in datasets. For the other datasets, using implicit embedding all led to deterio- ration in performance leveraged by a DeFog agent is the relative rather than the absolute timestep when the last frame was observed. The longer the drop-span of the current frame, the less should be considered in tion prediction, and the action history could be a better reference for decision-makinn We conclude
well, though not significant. We believe this shows that the information
19

---

Published as a conference paper at ICLR 2023
that critical information like the drop-span needs to be explicitly given, and performance would be hindered even if the agent can work out the number by simple arithmetic.
Removing Drop-Span and Timestep Embeddings Both the explicit drop-span encoder and the implicit embedding try to convey the drop-span information to the agent. We also conduct experi- ments totally removing this piece of information, by using normal timestep embedding without any other kind of drop-span embedding. The agent no longer receives information on how many frames are dropped. Finally, we perform the experiment of removing the timestep embedding but keeping the drop-span embedding. As mentioned above, the explicit drop-span encoder only gives information on the relative time span of dropped frames, not the actual timestep.
Walker2d-Exper
Hopper-Expert
Halfcheetah-Expert
5000
D7 Detag(
Average Return
4200
3300
Average Return
10000
Remove Drop
2500
3000
2000
6000
2000
1500
4000
1300
100
2000
0.30
0.99
0.00
0.1
0.85
690
Frame Drop
Rate
Frame Drop Rate
Frame Drop Rate
Figure 16: Ablation study removing the drop-span information. "Remove Drop" denotes remov- ing the drop-span embedding without using implicit method, while keeping the timestep embedding; "Remove Time" indicates removing the timestep embedding, while keeping the explicit drop-span embedding.
The results are given in Figure [6 and the performance degraded upon both of the embeddings' removal. We find drop-span embedding to be the key factor in DeFog. Mcanwhile, the removal of the timestep embedding does not cause severe drop performance. Under non-frame-dis drooping conditions, the Online Decision Transformer also conducted the experiments of removing timestep embeddings and found that performance was not heavily affected. As suggested by [Zheng et al.) (2022), this could be due to the timestep information deduced from the reward-to-ge signal, making lack of timestep embedding no longer fatal.
Finetuning Individual Components DeFog currently finetunes the action predictor and the drop- span encoder. For a better understanding of the finetuning stage, as well as the functions of specific elements in the model, we conduct experiments finetuning these components separately. The results are given in Figure 17
Walker2d-Expcr
Hopper-Expert
Hallchhetahh-Epertt
5000
3500
Nwer age Return
4000
Retum
3000
Average Return
10000
2300
8000
(lecton
3300
2000
Meoth
1500
1300
1000
500
2000
0.00
0.50
0.85
0.50
0.90
0.25
0.80
0.65
0.00
6:25
0.85
9
frame Drop
Rate
frame Drop Rate
Frame Drop Rate
Figure 17: Ablation study on separately finetuning the components of DeFog. "DeFog" denotes not finetuning anything. "f/skipstep". "flaction"" "f/both" stand for finetuning the drop-span encoder, the action predictor, and both, respectively.
We find that only finetuning the drop-span encoder gives slightly better performance the Meeieee2ddMMiuuu-Rpppee
Walker2d- -xeker,ddEExpet, and MalfChheetah-Medium Mieplly datasets. While on finetuning both the drop-span encoder
the other datasets, for example the Menper--eddum--Roee and the action predictor led to better results. In general, none of the finetuning methods significantly outperform their counterparts. We believe this is understandable as the action predictor and the drop-span encoder are both key components the DeFog model.
20


Reviews:
[{"review_id": "qwkFnFoESZI", "reviewer": "Reviewer_dgXg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of learning with random frame dropping. They propose to augment the decision transformer by randomly masking out observations in the offline dataset and explicitly adding the timespan of frame dropping as input to the transformer. They demonstrate that their approach, called DeFog, outperforms two online RL baselines under significant frame-dropping in Atari and MuJoCo. ", "summary_of_the_review": "While this paper addresses a problem of practical interest and is quite well-written, the experimental setup has some significant flaws including unfair comparisons, so it is difficult to draw any useful insights in the current stage. I recommend the authors to take into account the received feedback and resubmit an improved version of this work to a future conference. ", "strength_and_weaknesses": "# Strengths\n1. well-motivated problem of some practical interests \n2. the paper is easy to read overall\n3. evaluation on two widely-used domains\n\n# Weaknesses\n### 1. the setting doesn't make sense to me and in light of this, the results are not surprising at all\n- In the intro, you motivate this work by saying that in the real-world, often times observations are dropped or corrupted. However, as far as I understand, you collect an \"oracle\" dataset where no observations are dropped (or corrupted). Then, you randomly corrupt them in order to learn a more robust algorithm. But the whole point of solving this realistic setting is that you cannot collect / don't have access to such perfect / uncorrupted trajectories, isn't it? To me it seems like you motivated your work with one setting but you proceed to solve a completely different setting with totally unrealistic assumptions for the problem you claim to solve. \n- It is not surprising at all that if you mask out observations you will learn a more robust policy than DT which learns with full observations so when observations are masked out it has no way of knowing how to deal with that. \n- I believe the setting you should be considering (as motivated by the intro) is one in which your offline dataset contains dropped frames but your goal is still to learn a robust policy from them, perhaps by randomly masking out other frames. \n- In addition, you seem to be changing the dropped frames as mentioned at the top of page 5 \"sample drop-mask periodically from it\". Again, this indicates that your method has access to oracle trajectories collected without any frame dropping and then you randomize which ones are dropped, which of course will make the method more robust. \n\n### 2. the comparisons seem unfair an not very relevant\n- First of all, I don't think the comparison between online RL methods like TD3 / RLRD and DT / DeFog is fair. DT and DeFog have access to the oracle trajectories without any dropped frames (which DeFog perturbs to make the policy more robust). If I understand correctly, TD3 and RLRD collect data online under the frame-dropping regime so they can never see the same trajectory with and without certain frames dropped in order to learn what it should do in that situation. In contrast, DeFog can randomly drop frames so it gets to see the oracle. Thus, this is an unfair comparison. To make it more fair, you need to basically collect the offline dataset used to train DT an DeFog under the same frame dropping setting / environment as used for training your online RL baselines. \n- In addition, DeFog has explicit access to the timestep and drop-span embedding. Do TD3 and RLRD have access to this information? There is no mention of this but if they don't, the comparison is unfair because DeFog again has privileged information. \n- In general, comparing with online RL is quite strange and hard to make a fair comparison because it collects its own data so the results will be highly dependant on the offline dataset used by DT / DeFog (even assuming that your offline data is collected under the same conditions as the online data). \n- Given the above, the paper is incomplete without more fair comparisons with strong offline RL baselines trained on the same datasets as DT / DeFog such as IQL, CQL, offline DQN, and even BC (behavioral cloning) or BC transformer. These experiments could shed more light into whether the benefits are coming from the transformer architecture, offline learning, masking approach, dataset or something else. It could be the case that this masking also help offline RL methods which would make your approach even more general. \n\n### 3. needs more baselines and ablations to better understand where the gains are coming from \n- I would also like to see ablations of DeFog that doesn't use the drop-span embedding at all, one that doesn't use the timestep embedding at all, and what that doesn't use any of them. This would help in better assessing the contribution of each part of the architecture. Otherwise, again it is difficult to know where the gains are coming from. \n- For the finetuning part, I would like to see ablations that only finetune the action predictor and only finetune the drop-span encoder. \n- It would also be interesting to see what happens if instead of repeated prior frames, you replace them with zeros or an average of the N previous frames, a random frame from the previous N, or the previous frame with some added noise. This would shed more light into that particular choice of the algorithm. Does repeating the frame help because it adds some noise or is using the prior frame more helpful than adding other types of noise?\n\n### 4. the setting and experimental setup isn't very clear, some important details are missing\n- please see the section on Clarity below\n\n\n# Minor\n1. Why doesn't finetuning help in Atari?\n2. Why doesn't your approach help as much in Atari?\n3. Why is the variance of DT so large on Atari?\n4. In Figure 7, it looks like removing the explicit drop embedding only hurts performance on Walker. Do you understand why that is? Does it mean it's not so important after all? In the text, you claim this is an important contribution, but the results seem mixed. I recommend running this ablation on more environments for a more robust conclusion (and try to understand why / when it helps).  \n5. How many seeds do you use?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ifgdeTGaNzA", "reviewer": "Reviewer_qcaZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targeted at an interesting and important problem: controlling the agents against frame dropping. Although there are some works aiming to solve that, they are either bottlenecked by delay threshold, or not built on the recent Decision Transformer backbone. This paper proposed a simple yet effective pre-training strategy with Offline Decision Transformer. That is, pre-training Decision Transformer with random frame dropping. This work is in line with well-known self-supervised learning methods such as Masked Language Modeling, etc. The decision robustness against frame dropping is enhanced with minimal modeling changes. The experiments on MuDojo and Atari environments (both continuous control and discrete control settings) demonstrated the effectiveness.", "summary_of_the_review": "This submission proposes frame-dropping to improve the robustness of decision transformers. However, the core idea is too similar to a previous work “The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning, NeurIPS 2021”. ", "strength_and_weaknesses": "# Strength\n\n- The problem is well defined, with clear mathematical formulation, i.e., Random Dropping Markov Decision Process.\n- This method is quite simple, with just changes on the input layer and training strategy from vanilla Offline Decision Transformer;\n- This method is quite effective, with convincing results on well-known platforms compared to strong baselines.\n\n# Weakness\n\n- This submission ignored a very related paper “The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning, NeurIPS 2021”. Although the NeurIPS paper does not consider dropping frames and the motivation is different, the effect of random replacement of the input signals is similar to frame dropping in my opinion.\n- The experiments are not enough in terms of discrete control tasks: (1) the baselines are strong enough and not representative; (2) the performance difference is not that noticeable compared to the offline Decision Transformer baseline; (3) DeFog outperforms offline Decision Transformer even if the frame drop rate is 0.00, which suggests that the random frame dropping strategy brings performance gains as well in scenarios without frame dropping problem. This lacks further discussion; (4) DeFog/f (with finetuning) is not discussed in this setting due to zero improvement. This lacks further discussion as well.\n- There are too few ablation experiments that only contain the part about drop-span embedding. The supplemental discussions about (at least) drop-mask distribution beyond Bernoulli Process would be interesting.\n- As for the ablation study: only drop-span embedding v.s. implicit embedding is not adequate. It is interesting to ask what would happen if we simply replace the dropped frame with a special token like [MASK] instead of with the last tokens.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eXgKIeC47Kl", "reviewer": "Reviewer_4AHJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Decision Transformer under Random Frame Dropping (DeFog). Defog is an simple yet effective algorithm based on DT that is designed to robust to frame dropping issue, which is one of the key challenges in real-world remote control applications. Defog simulates the frame dropping phenomenon  by randomly masking out the observations in offline datasets and embeds frame dropping timespan information explicitly to the model. Experiments are conducted on both continuous and discrete control tasks, i.e. MuJoCo and Atari. Empirical results show that Defog is robust to frame-dropping issue in some of the tasks, mainly on MuJoCo.", "summary_of_the_review": "This work is well motivated and the writing is easy to follow. I like it starts with the real-world challenges and then formulate the research problem based on it. The proposed approach is simple yet effective. Empirical results validate the robustness of Defog against frame-dropping issues in most of continuous control tasks when comparing with the other approaches. \nDespite only empirical comparisons, I like this paper also trying to show some insights underlying the technical approach, e.g. visualization analysis.\n\nHowever, the technical novelty is limited. The propose approach seems a simple extension on DT, specifically adding random mask to simulate the frame-dropping and a drop-span embedding to give explicit signals to the model. While the whole model architecture and the training objectives are all the same with DT. Empirically, Defog demonstrated its effectiveness on MuJoCo. While on Atari, seems it does not show much benefits when comparing with a vanilla DT.  Besides, no source codes are provided, so I can not validate its reproducibility.", "strength_and_weaknesses": "Pros:\nThis work is well motivated and the writing is easy to follow. I like it starts with the real-world challenges and then formulate the research problem based on it. \nThe proposed approach is simple yet effective. Empirical results validate the robustness of Defog against frame-dropping issues in most of continuous control tasks when comparing with the other approaches. \nDespite only empirical comparisons, I like this paper also trying to show some insights underlying the technical approach, e.g. visualization analysis.\n\nCons:\nMy main concern is the technical novelty. The propose approach seems a simple extension on DT, specifically adding random mask to simulate the frame-dropping and a drop-span embedding to give explicit signals to the model. While the whole model architecture and the training objectives are all the same with DT. \nAnother concern is that the drop-span embedding has a strong dependence on knowing how where and how many frames are dropped. However, in many real-world applications, such signals are unknown, in these cases the drop-span beddings limited the broad applications of Defog.\nEmpirically, Defog demonstrated its effectiveness on MuJoCo. While on Atari, seems it does not show much benefits when comparing with a vanilla DT.\n\nMinor:\nTypo Sec. 4.4 'DeFogin' -> 'DeFog in ....'\n\nSome questions:\n1) Would you give more analysis on the different observations on continuous tasks (Mojoco) vs, discrete tasks (Atari)?\n2) it is good see the ablation on the contribution of drop-span embedding and finetuning stage. While that would be good to give more insights on why and how such training regime can improve the robustness.\n3) I am curious what if Defog also randomly mask out actions. Because the current training gives the model ground actions sequentially, which maybe lead to trivial solutions as the model already seen the actions signals for next step prediction. In Fig 5 and Fig 6, we can see that the trained defog tends to give smooth actions which maybe adapted to the showing cases. But if an agent has drastic motion pattern, then naively insert smooth actions would collapse the agent. \n4) what if also ask the model to reconstruct the masked out states? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster