Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents Decision Transformer under Random Frame Dropping (DeFog). Defog is an simple yet effective algorithm based on DT that is designed to robust to frame dropping issue, which is one of the key challenges in real-world remote control applications. Defog simulates the frame dropping phenomenon  by randomly masking out the observations in offline datasets and embeds frame dropping timespan information explicitly to the model. Experiments are conducted on both continuous and discrete control tasks, i.e. MuJoCo and Atari. Empirical results show that Defog is robust to frame-dropping issue in some of the tasks, mainly on MuJoCo.
strength_and_weaknesses: Pros:
This work is well motivated and the writing is easy to follow. I like it starts with the real-world challenges and then formulate the research problem based on it. 
The proposed approach is simple yet effective. Empirical results validate the robustness of Defog against frame-dropping issues in most of continuous control tasks when comparing with the other approaches. 
Despite only empirical comparisons, I like this paper also trying to show some insights underlying the technical approach, e.g. visualization analysis.

Cons:
My main concern is the technical novelty. The propose approach seems a simple extension on DT, specifically adding random mask to simulate the frame-dropping and a drop-span embedding to give explicit signals to the model. While the whole model architecture and the training objectives are all the same with DT. 
Another concern is that the drop-span embedding has a strong dependence on knowing how where and how many frames are dropped. However, in many real-world applications, such signals are unknown, in these cases the drop-span beddings limited the broad applications of Defog.
Empirically, Defog demonstrated its effectiveness on MuJoCo. While on Atari, seems it does not show much benefits when comparing with a vanilla DT.

Minor:
Typo Sec. 4.4 'DeFogin' -> 'DeFog in ....'

Some questions:
1) Would you give more analysis on the different observations on continuous tasks (Mojoco) vs, discrete tasks (Atari)?
2) it is good see the ablation on the contribution of drop-span embedding and finetuning stage. While that would be good to give more insights on why and how such training regime can improve the robustness.
3) I am curious what if Defog also randomly mask out actions. Because the current training gives the model ground actions sequentially, which maybe lead to trivial solutions as the model already seen the actions signals for next step prediction. In Fig 5 and Fig 6, we can see that the trained defog tends to give smooth actions which maybe adapted to the showing cases. But if an agent has drastic motion pattern, then naively insert smooth actions would collapse the agent. 
4) what if also ask the model to reconstruct the masked out states? 
clarity,_quality,_novelty_and_reproducibility: This work is well motivated and the writing is easy to follow. I like it starts with the real-world challenges and then formulate the research problem based on it. The proposed approach is simple yet effective. Empirical results validate the robustness of Defog against frame-dropping issues in most of continuous control tasks when comparing with the other approaches. 
Despite only empirical comparisons, I like this paper also trying to show some insights underlying the technical approach, e.g. visualization analysis.

However, the technical novelty is limited. The propose approach seems a simple extension on DT, specifically adding random mask to simulate the frame-dropping and a drop-span embedding to give explicit signals to the model. While the whole model architecture and the training objectives are all the same with DT. Empirically, Defog demonstrated its effectiveness on MuJoCo. While on Atari, seems it does not show much benefits when comparing with a vanilla DT. 

Besides, no source codes are provided, so I can not validate its reproducibility.
summary_of_the_review: This work is well motivated and the writing is easy to follow. I like it starts with the real-world challenges and then formulate the research problem based on it. The proposed approach is simple yet effective. Empirical results validate the robustness of Defog against frame-dropping issues in most of continuous control tasks when comparing with the other approaches. 
Despite only empirical comparisons, I like this paper also trying to show some insights underlying the technical approach, e.g. visualization analysis.

However, the technical novelty is limited. The propose approach seems a simple extension on DT, specifically adding random mask to simulate the frame-dropping and a drop-span embedding to give explicit signals to the model. While the whole model architecture and the training objectives are all the same with DT. Empirically, Defog demonstrated its effectiveness on MuJoCo. While on Atari, seems it does not show much benefits when comparing with a vanilla DT.  Besides, no source codes are provided, so I can not validate its reproducibility.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper targeted at an interesting and important problem: controlling the agents against frame dropping. Although there are some works aiming to solve that, they are either bottlenecked by delay threshold, or not built on the recent Decision Transformer backbone. This paper proposed a simple yet effective pre-training strategy with Offline Decision Transformer. That is, pre-training Decision Transformer with random frame dropping. This work is in line with well-known self-supervised learning methods such as Masked Language Modeling, etc. The decision robustness against frame dropping is enhanced with minimal modeling changes. The experiments on MuDojo and Atari environments (both continuous control and discrete control settings) demonstrated the effectiveness.
strength_and_weaknesses: # Strength

- The problem is well defined, with clear mathematical formulation, i.e., Random Dropping Markov Decision Process.
- This method is quite simple, with just changes on the input layer and training strategy from vanilla Offline Decision Transformer;
- This method is quite effective, with convincing results on well-known platforms compared to strong baselines.

# Weakness

- This submission ignored a very related paper “The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning, NeurIPS 2021”. Although the NeurIPS paper does not consider dropping frames and the motivation is different, the effect of random replacement of the input signals is similar to frame dropping in my opinion.
- The experiments are not enough in terms of discrete control tasks: (1) the baselines are strong enough and not representative; (2) the performance difference is not that noticeable compared to the offline Decision Transformer baseline; (3) DeFog outperforms offline Decision Transformer even if the frame drop rate is 0.00, which suggests that the random frame dropping strategy brings performance gains as well in scenarios without frame dropping problem. This lacks further discussion; (4) DeFog/f (with finetuning) is not discussed in this setting due to zero improvement. This lacks further discussion as well.
- There are too few ablation experiments that only contain the part about drop-span embedding. The supplemental discussions about (at least) drop-mask distribution beyond Bernoulli Process would be interesting.
- As for the ablation study: only drop-span embedding v.s. implicit embedding is not adequate. It is interesting to ask what would happen if we simply replace the dropped frame with a special token like [MASK] instead of with the last tokens.
clarity,_quality,_novelty_and_reproducibility: # Novelty

The frame dropping is easy-to-implement and effective in some cases. I can expect that this simple trick would be used widely in the community

# Reproducibility

I have mixed feelings about the reproducibility: even though the frame dropping is easy to implement, I still hope the authors could release the complete experimental code for others to reproduce. 

# Requested Changes

- More baselines for discrete control task settings
- Analyzing why finetuning on the discrete control setting doesn't work
- Discussing the placeholder of dropped frames
summary_of_the_review: This submission proposes frame-dropping to improve the robustness of decision transformers. However, the core idea is too similar to a previous work “The Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement Learning, NeurIPS 2021”.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the problem of learning with random frame dropping. They propose to augment the decision transformer by randomly masking out observations in the offline dataset and explicitly adding the timespan of frame dropping as input to the transformer. They demonstrate that their approach, called DeFog, outperforms two online RL baselines under significant frame-dropping in Atari and MuJoCo. 
strength_and_weaknesses: # Strengths
1. well-motivated problem of some practical interests 
2. the paper is easy to read overall
3. evaluation on two widely-used domains

# Weaknesses
### 1. the setting doesn't make sense to me and in light of this, the results are not surprising at all
- In the intro, you motivate this work by saying that in the real-world, often times observations are dropped or corrupted. However, as far as I understand, you collect an "oracle" dataset where no observations are dropped (or corrupted). Then, you randomly corrupt them in order to learn a more robust algorithm. But the whole point of solving this realistic setting is that you cannot collect / don't have access to such perfect / uncorrupted trajectories, isn't it? To me it seems like you motivated your work with one setting but you proceed to solve a completely different setting with totally unrealistic assumptions for the problem you claim to solve. 
- It is not surprising at all that if you mask out observations you will learn a more robust policy than DT which learns with full observations so when observations are masked out it has no way of knowing how to deal with that. 
- I believe the setting you should be considering (as motivated by the intro) is one in which your offline dataset contains dropped frames but your goal is still to learn a robust policy from them, perhaps by randomly masking out other frames. 
- In addition, you seem to be changing the dropped frames as mentioned at the top of page 5 "sample drop-mask periodically from it". Again, this indicates that your method has access to oracle trajectories collected without any frame dropping and then you randomize which ones are dropped, which of course will make the method more robust. 

### 2. the comparisons seem unfair an not very relevant
- First of all, I don't think the comparison between online RL methods like TD3 / RLRD and DT / DeFog is fair. DT and DeFog have access to the oracle trajectories without any dropped frames (which DeFog perturbs to make the policy more robust). If I understand correctly, TD3 and RLRD collect data online under the frame-dropping regime so they can never see the same trajectory with and without certain frames dropped in order to learn what it should do in that situation. In contrast, DeFog can randomly drop frames so it gets to see the oracle. Thus, this is an unfair comparison. To make it more fair, you need to basically collect the offline dataset used to train DT an DeFog under the same frame dropping setting / environment as used for training your online RL baselines. 
- In addition, DeFog has explicit access to the timestep and drop-span embedding. Do TD3 and RLRD have access to this information? There is no mention of this but if they don't, the comparison is unfair because DeFog again has privileged information. 
- In general, comparing with online RL is quite strange and hard to make a fair comparison because it collects its own data so the results will be highly dependant on the offline dataset used by DT / DeFog (even assuming that your offline data is collected under the same conditions as the online data). 
- Given the above, the paper is incomplete without more fair comparisons with strong offline RL baselines trained on the same datasets as DT / DeFog such as IQL, CQL, offline DQN, and even BC (behavioral cloning) or BC transformer. These experiments could shed more light into whether the benefits are coming from the transformer architecture, offline learning, masking approach, dataset or something else. It could be the case that this masking also help offline RL methods which would make your approach even more general. 

### 3. needs more baselines and ablations to better understand where the gains are coming from 
- I would also like to see ablations of DeFog that doesn't use the drop-span embedding at all, one that doesn't use the timestep embedding at all, and what that doesn't use any of them. This would help in better assessing the contribution of each part of the architecture. Otherwise, again it is difficult to know where the gains are coming from. 
- For the finetuning part, I would like to see ablations that only finetune the action predictor and only finetune the drop-span encoder. 
- It would also be interesting to see what happens if instead of repeated prior frames, you replace them with zeros or an average of the N previous frames, a random frame from the previous N, or the previous frame with some added noise. This would shed more light into that particular choice of the algorithm. Does repeating the frame help because it adds some noise or is using the prior frame more helpful than adding other types of noise?

### 4. the setting and experimental setup isn't very clear, some important details are missing
- please see the section on Clarity below


# Minor
1. Why doesn't finetuning help in Atari?
2. Why doesn't your approach help as much in Atari?
3. Why is the variance of DT so large on Atari?
4. In Figure 7, it looks like removing the explicit drop embedding only hurts performance on Walker. Do you understand why that is? Does it mean it's not so important after all? In the text, you claim this is an important contribution, but the results seem mixed. I recommend running this ablation on more environments for a more robust conclusion (and try to understand why / when it helps).  
5. How many seeds do you use?
clarity,_quality,_novelty_and_reproducibility: # Clarity
The paper is fairly well-written but the clarity could be improved in certain places. Some important details about the experimental setup are missing.  

1. For example, it is not immediately clear whether the authors consider frame dropping during data collection, training, finetuning, at test time or a combination of these. I recommend adding more details about the precise setting in the introduction to avoid confusion. The experimental setup should be further clarified since it's not very clear. You seem to assume no frame dropping during data collection, but a changing distribution of frame dropping during training. Is this true? Do you keep the same percentage of frame dropping during training? Does this percentage change at test time or during the finetuning stage? Does evaluation happen online for all methods? What changes at test time, is it only the frame dropping percentage and is that fixed?

2. The authors mention making deep RL (DRL) agents robust to frame dropping in the abstract and introduction, but this is misleading since they don't propose a DRL algorithm, but rather an algorithm that learns from offline datasets which is a very different settings. Hence, those claims are incorrect. 

3. On page 2, you mention the drop-span embeddings but don't define them. I don't think this is a common term in the literature so I suggest defining it the first time you mention it to avoid confusion. 

4. Other details such as the number of runs used to compute the results in the plots are missing. This is an important detail so should be mentioned in the main paper rather than appendix. 

5. I found Figure 2 to be somewhat hard to read. I'm not sure why you chose to add those lines on top of the timestep and drop-span embeddings. If they don't serve a particular purpose, I suggest removing them to make it more readable.

6. I found the sentence on page 3 "the optimization goal of our method is to imitate the oracle policy that have access to the true state in a frame-dropping environment through learning from the trajectories generated by the oracle" quite confusing. It seems to be contradicting your motivation of learning under frame-rating which would imply that you don't have access to the oracle and perfect trajectories, but rather the data was collected under frame dropping scenarios. I don't know when this assumption is realistic since it's as if you're saying "we can't collect uncorrupted data but here we assume we have access to uncorrupted data". 

# Quality
As mentioned above, the paper needs more work in order to warrant acceptance at the ICLR conference. In particular, 1) the problem setting needs to be clarified, 2) the experimental setup needs to be realistic and ensure it is addressing the problem we care about rather than making unrealistic assumptions and leading to trivial conclusions, 3) more appropriate baselines need to be added for a fair comparison, 4) more ablations needs to be added for a better understanding of where the gains are coming from.  

# Novelty
The ideas in this paper are not particularly novel. The proposed approach is a combination of well-known ideas but this exact instantiation hasn't been proposed before as far as I know. The authors cite related work and places their contribution in the context of the broader literature. I would say novelty is not a major problem with this paper, so I wouldn't use this as a reason to reject it if the other (more important) issues are resolved. 

# Reproducibility
I didn't see any mention of the code or plans to open source it. While the paper contains reasonably details about the methods and experimental setup, this is not enough to easily reproduce their results. Do the authors plan to open source their code and if so, what is the timeline for that? This is an important factor in my decision. 

summary_of_the_review: While this paper addresses a problem of practical interest and is quite well-written, the experimental setup has some significant flaws including unfair comparisons, so it is difficult to draw any useful insights in the current stage. I recommend the authors to take into account the received feedback and resubmit an improved version of this work to a future conference.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: A very simple extension of Decision Transformer is proposed to provide robustify the model against frame dropping, which is well motivated in real-world remote control applications. The key idea is to simply to randomly mask out the observations in offline datasets and embed frame dropping timespan information explicitly to the model. MuJoCo and Atari results show the benefits relative to a comprehensive set of baselines.

Strengths: Simplicity and effectiveness of the proposal on the benchmarks tested. 

Weaknesses: Technical novelty seems quite low. The paper would be more convincing if other baselines are also reported with and without frame dropping during training for a more complete empirical evaluation. 


justification_for_why_not_higher_score: No strong champion for the paper, the reviews seem a very lukewarm in the evaluation of technical novelty. 
justification_for_why_not_lower_score: Robustness of decision transformer-based policies under observation corruption is an important practical problem. As such, the simple training techniques benchmarked in this paper maybe of interest to RL/Robotics communities.
Decision: Accept: poster