Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a new way to apply Decision-Transformer (DT) [1] to unseen tasks through the adapter layer initialized by a Hyper-network. They showed their method could adapt faster and better to unseen tasks with few parameter updates when comparing with previous works Prompt-DT [2], SiMPL [3], IA3 [4], and variations of their model. Especially their model can adapt even only with the demonstration of unseen tasks where expert action is unavailable.

strength_and_weaknesses: Strength
- This paper proposes a new way for DT to adapt the unseen tasks through additional neural network layers called an adapter layer. It shows better performance than baselines and model variations.
- Through designing, their adaptation can cover not just full trajectories but also demonstration data. Their model can adapt the unseen tasks with and without expert action.
- Their model is compared with many variations, such as without hyper-network initialization or training the adapter with DT simultaneously.

Weaknesses
- Some parts of this paper are not clear to me.
    - In section 3.4 equation (3), the convolutional network's input is unclear. Why did you concatenate $L_s(h^o)+L_t(h^o)+L_l(h^o)$ and $L_{\hat{r}}(h^o)+L_t(h^o)+L_l(h^o)$? Is it possible to use $L_s(h^o)+L_t(h^o)+L_l(h^o)+L_{\hat{r}}(h^o)$?
    - In section 4.2 HDB-IA3, what is the meaning that the models do not utilize position-wise rescaling? 
    - Table 2 is hard to understand. Does that mean the success rate? The average success rate when achieving to get a success episode? You mentioned that the data efficiency is based on the top-2 number of online rollout episodes until a success episode. Does it mean the number in the table is the average of the two number of online rollout episodes?
    - In section 5.2, relating to the unclarity of table 2, could you explain more about why PDT hardly improves success rates?
    - > First, HDT achieves similar training task performance compared with the pre-trained DT model as in Figure 3 (a), which shows that the hyper-network could distinguish the task-specific information of training tasks.
        - Is it because the multi-task DT can distinguish the task-specific information through a given state, action, and reward sequence?
- Typo in conclusion. grinned -> grind.
clarity,_quality,_novelty_and_reproducibility: Except for some parts, this paper is well-written and clearly explains and evaluates their model. Their model is novel, different from Prompt-DT or IA3 methods and outperforms them. Through the details about the hyperparameters, it looks reproducible.
summary_of_the_review: This paper proposes a new method for DT to adapt the unseen tasks. The way to do this is novel and shows better performance compared with previous works [2,3,4]. For me, some parts are unclear, which I hope to be updated, but overall, this paper is well-written and good to share in our community.


[1]Chen, Lili, et al. "Decision transformer: Reinforcement learning via sequence modeling." Advances in neural information processing systems 34 (2021): 15084-15097.

[2] Xu, Mengdi, et al. "Prompting decision transformer for few-shot policy generalization." International Conference on Machine Learning. PMLR, 2022.

[3] Nam, Taewook, et al. "Skill-based Meta-Reinforcement Learning." arXiv preprint arXiv:2204.11828 (2022).

[4] Liu, Haokun, et al. "Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning." arXiv preprint arXiv:2205.05638 (2022).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, authors study the important question of efficiently adapting decision transformers when having access to a handful of demonstrations for unseen tasks. They propose to augment the base DT with an adaptation module that can be controlled by a hyper-net, which can take the demonstration as inputs. The author conduct experiments on meta world and shows that their results are better than other baselines.
strength_and_weaknesses: Strength:
The problem is well motivated, and efficiently adapting large pertained decision transformers with a handful of observation is an important line of research. Also the LfO setting is interesting since in many real life robotic applications it’s hard to obtain state action pairs as demonstrations. The paper is also easy to follow. 

What I like most is the experiment design and the details. While at first sight, the proposed method seems very complicated, the design is justified in the ablation study. E.g., Fig 3 suggests that the hypernet is better than other simpler ways of adapting transformers like prompting (PDT) and IA3. It’s also nice to see the comparison with meta RL methods like SimPL, but it’s expected that SimPL is not as good as the proposed method, since SimPL does not take into the demonstration in the unseen tasks. It’s also interesting to see that the hyper net can encode the task-specific information as in sec5.3. While the paper only has one environment Metaworld, but meta45 is a pretty challenging task alone. I think this could be interesting for researchers studying decision transformers.

Weakness:
While I think metaworld is enough, more environments could be nicer, like Kitchen, which seems to be more widely adopted in the previous literature.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to read. I like the demo webpage with a clear summary of main ideas and performance.
summary_of_the_review: In general, I think it's a good paper and recommend acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new approach

Decision Transformers (DT) have demonstrated strong performances in offline reinforcement learning settings, but quickly adapting to unseen novel tasks remains
challenging. To address this challenge, we propose a new framework, called
Hyper-Decision Transformer (HDT), that can generalize to novel tasks from a
handful of demonstrations in a data- and parameter-efficient manner. To achieve
such a goal, we propose to augment the base DT with an adaptation module,
whose parameters are initialized by a hyper-network. When encountering unseen tasks, the hyper-network takes a handful of demonstrations as inputs and
initializes the adaptation module accordingly. This initialization enables HDT to
efficiently adapt to novel tasks by only fine-tuning the adaptation module. We validate HDT’s generalization capability on object manipulation tasks. We find that
with a single expert demonstration and fine-tuning only 0.5% of DT parameters,
HDT adapts faster to unseen tasks than fine-tuning the whole DT model. Finally,
we explore a more challenging setting where expert actions are not available, and
we show that HDT outperforms state-of-the-art baselines in terms of task success
rates by a large margin. Demos are available on our project page.
strength_and_weaknesses: # Strengths
- important problem of interests for the community
- promising initial results, the proposed approach seems to be quite effective 
- simple, intuitive approach
- some interesting insights on how the method scales with the number of training tasks and model size

# Weaknesses
### 1. evaluation on a single domain
- The method is evaluated only on the tasks from Meta World, a robotic manipulation domain. Hence, it is difficult to judge whether the results will generalize to other domains. I strongly recommend running experiments on a different benchmark such as Atari which is commonly used in the literature. This would also verify whether the method works with discrete action spaces and high-dimensional observations. 

### 2. evaluation on a setting created by the authors, no well-established external benchmark
- The authors seem to create their own train and test splits in Meta World. This seems strange since Meta World recommends a particular train and test split (e.g. MT10 or MT50) in order to ensure fair comparison across different papers. I strongly suggest running experiments on a pre-established setting so that your results can easily be compared with prior work (without having to re-implement or re-run them). You don't need to get SOTA results, just show how it compares with reasonable baselines like the ones you already include. Otherwise, there is a big question mark around why you created your own "benchmark" when a very similar one exists already and whether this was somehow carefully designed to make your approach look better.

### 3. limited number of baselines
- While you do have some transformer-based baselines I believe the method could greatly benefit from additional ones like BC, transformer-BC, and other offline RL methods like CQL or IQL. Such comparisons could help shed more light into whether the transformer architecture is crucial, the hypernetwork initialization, the adaptation layers, or the training objective. 

### 4. more analysis is needed
- It isn't clear how the methods compare with the given expert demonstrations on the new tasks. Do they learn to imitate the policy or do they learn a better policy than the given demonstration? I suggest comparing with the performance of the demonstration or policy from which the demonstration was collected. 

- If the environment is deterministic and the agent gets to see expert demonstrations, isn't the problem of learning to imitate it quite easy? What happens if there is more stochasticity in the environments or the given demonstration isn't optimal?

- When finetuning transformers, it is often the case that they forget the tasks they were trained on. It would be valuable to show the performance of your different methods on the tasks they were trained on after being finetuned on the downstream tasks. Are some of them better than the others at preserving previously learned skills?

### 5. missing some important details
- The paper seems to be missing some important details regarding the experimental setup. For example, it wasn't clear to me how the learning from observations setting works. At some point you mention that you condition on the expert observations while collecting online data. Does this assume the ability to reset the environment in any state / observation? If so, this is a big assumption that should be more clearly emphasized and discussed. how exactly are you using the expert observations in combination with online learning? 

- There are also some missing details regarding the expertise of the demonstrations at test time. Are these demonstrations coming from an an expert or how good are they? 

# Minor
- sometimes you refer to generalization to new tasks. however, you finetune your models, so i believe a better term would be transfer or adaptation to new tasks. 



clarity,_quality,_novelty_and_reproducibility: # Clarity
The paper is clear overall. However, it is missing some important details as explained above. 

# Quality 
The paper is well-motivated, proposes a sensible approach, and compares it with a number of different baselines and ablations on one domain, demonstrating promising results. However, I believe the empirical evaluation could be further improved by including evaluations on well-established benchmarks, other domains, and a few more relevant baselines, as detailed above. The paper could also benefit from more extensive analysis. 

# Novelty 
While the different parts of the approach aren't novel, this particular combination and application is, so I believe the work contains enough novelty for this to not be a factor against acceptance, assuming all the other more important issues are adequately addressed.

# Reproducibility
I couldn't find any mention of the code. While the method contains some details regarding the implementation and experimental setup, these are far from enough to easily reproduce the work. Do you have any plans of open sourcing the code? This is an important factor in my decision. 
summary_of_the_review: This paper proposes a new approach for finetuning on new tasks after offline training on a set of different tasks. While the authors present some promising initial results, the empirical evaluation requires more work to warrant acceptance at the ICLR conference. In particular, the generality of the approach cannot be assessed without evaluating it on multiple well-established benchmarks designed in prior work (rather than by the authors themselves). More baselines and analysis could also further strengthen the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: As I understand: For learning an agent across multiple RL tasks that can quickly adapt to new tasks, the authors combine several established paradigmas: 
- decision transformers as the base archictecture for the learning agent, trained from offline RL data across tasks; 
- adding additional "adaption" parameters (layers);
- these parameters are pre-trained using the Hyper net training (Alg 1) and then used as initalization;
- when a new task arrives, with demonstrator data, then first the adaptation layer is fine-tuned (from the above initializatin) on that new task's demonstrator data (while the base params are kept). This is a trade-off to be able to be adapt, while avoiding too much variance of the overall architecture upon new tasks (and forgetting old ones).

Some of these ideas come from NLP and are here tansferred to DT.

In the experiments, HDT outperforms some baselines and some questions are tried to be answered experimentally, e.g., if the hyper net learns to use the task-specific data (5.3).
strength_and_weaknesses: **Strengths:**

The overall problem - multi-task learning - and the choice of the interesting recent DT architecture as base, are well motivated.

The method combines several paradigms and seems to transfer insights from NLP Transformer models, which is nice.

The experiments seem to indicate, that the method works comparably well.


**Weaknesses and specific points for improvement:**

Many paradigmas are put together (meta, offline RL, IL, …), sometimes with more, sometimes with less motivation. For instance, why combine offline RL base-training with expert-data as task-specific info? Where does this scenario happen in practice (i.e., where do we have the base offline RL data plus a demonstrator for each task)? It is not entirely counterintuitive, but I didn't see the clear motivation. Maybe the main reason is that only this setting (i.e., data from a demonstrator in the new task) fits well into the quite restrictive DT format where the data has to be in the form of s/a/r-triplets. But how about just using offline RL data for the new task as well (or is there too little information content in non-expert data?). A stronger motivation would help here.

The wiriting clarity and understanding of the method needs to be improved, e.g.,
* The action variable $a$ in Alg 1/2 comes from nowhere. It needs to be related to the trajectories etc. Essentially, variables always have to be properly introduced (within a context/algo).
* I couldn't follow the argument in sec 5.3. Why again does Fig 3 (a) show that the hyper net in fact learns to use task-specific knowledge?
* Imo, DT is introduced a bit too briefly although it is at the very core and a quite new paradigm. I didn't understand how the reward-to-go is obatained but I guess this simply the retrospective sum since we are in an offline setting - OK but maybe clarify. How about this length K "lag" fed into the transofmer. Why not, as in classic RL, just feed the *single* current state (plus desired reward in the sense of the DT idea) and output an action? Is this to account for non-Markov, or for an agent memory, or for task info? There is an informal comment on this on p4, but it feels vauge - then again, maybe one just has to accept that these sort of approaches are less motivated by understanding and more by experiments.

Further comment on experiment: I'm a bit surprised about the very poor performance of some of the baslines in Table 2 (0 success). This can be a hint that they are not tuned properly enough.

In terms of novelty/significance, there seems to be some novelty, however, the paper is heavily based on "just" combining several well-established ideas.
clarity,_quality,_novelty_and_reproducibility: See also above.

Re. correctness: The evidence is mostly empirical; I did not check all details of the experiments. Overall, the experimental section is sensibly structured (several baselines, questions answered by comparison with other models, etc.); however, in the specifics, I did not find all details understandable, e.g., the point above (re. 5.3).
summary_of_the_review: The multi-task problem is well-motivated, the interesting recent DT approach is combined with further ideas to address it; the novelty exists but is limited and the writing and explanations need to be improved. Hence, I do not see a fundamental issue; however, the contribution seems rather limited.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper enables decision transformers to rapidly adapt to novel tasks with good sample and parameter efficiency.  By employing an adaptation module with a hyper-network to initialize its parameters, only fine-tuning of the module is needed for new tasks.  Experimental results on object manipulation shows the method can adapt quite efficiently.

The paper is well written and addresses an important problem with a novel technique.  Empirically, it outperforms the SOTA.  The writing can still be improved, and additional experiments can be beneficial in terms of more domains, more baselines, and external benchmark. 

justification_for_why_not_higher_score: The writing can still be improved, and additional experiments can be beneficial in terms of more domains, more baselines, and external benchmark. 
justification_for_why_not_lower_score: The paper is well written and addresses an important problem with a novel technique.  Empirically, it outperforms the SOTA.
Decision: Accept: poster