Hyper-Decision Transformer for Efficient Online Policy Adaptation
Authors: Mengdi Xu
Yuchen Lu
Yikang Shen
Shun Zhang
Ding Zhao
Chuang Gan
Venue: ICLR 2023 poster (2023)
Keywords: Offline Reinforcement Learning
One-shot Imitation Learning
Parameter-efficient Fine-tuning

Abstract: Decision Transformers (DT) have demonstrated strong performances in offline reinforcement learning settings, but quickly adapting to unseen novel tasks remains challenging. To address this challenge, we propose a new framework, called Hyper-Decision Transformer (HDT), that can generalize to novel tasks from a handful of demonstrations in a data- and parameter-efficient manner. To achieve such a goal, we propose to augment the base DT with an adaptation module, whose parameters are initialized by a hyper-network. When encountering unseen tasks, the hyper-network takes a handful of demonstrations as inputs and initializes the adaptation module accordingly. This initialization enables HDT to efficiently adapt to novel tasks by only fine-tuning the adaptation module. We validate HDT's generalization capability on object manipulation tasks. We find that with a single expert demonstration and fine-tuning only 0.5% of DT parameters, HDT adapts faster to unseen tasks than fine-tuning the whole DT model. Finally, we explore a more challenging setting where expert actions are not available, and we show that HDT outperforms state-of-the-art baselines in terms of task success rates by a large margin. Demos are available on our project page: https://sites.google.com/view/hdtforiclr2023/home.

Published as a conference paper at ICLR 2023
HYPER-DECISION TRANSFORMER FOR EFFICIENT ON- LINE POLICY ADAPTATION
Mengdi Xu', Yuchen Lu?, Yikang Shen?, Shun Zhang'. Ding Zhao' & Chuang Gan Carnegie Mellon University. 2 University of Montreal, Mila, MIT-IBM Watson Al Lab," UMass Amherst
ABSTRACT
Decision Transformers (DT) have demonstrated strong performances in offline re- inforcement learning settings, but quickly adapting to unseen novel tasks remains challenging. To address this challenge, Hyper-Decision Transformer (HDT), that can generalize to novel tasks from a handful of demonstrations in data- and parameter-efficient manner. To achieve such a goal, we propose to augment the base DT with an adaptation module, whose parameters are initialized by hyper-network. When encountering un- seen tasks, the hyper-network takes handful of demonstrations as inputs and initializes the adaptation module accordingly. This initialization enables HDT efficiently adapt to novel tasks by only tne-tuning the adaptation module. We val- idate HDT's generalization capability on object manipulation tasks. We find that with single expert demonstration and fine-tuning only 0.5% of DT parameters, HDT adapts faster to unseen tasks than fine-tuning the whole DT model. Finally, we explore more challenging setting where expert actions not available, and we show that HDT outperforms state-of of-the--rr art baselines terms of task success rates by large margin. Demos are available our project page.
propose new framework, called
INTRODUCTION
Building an autonomous agent capable of generalizing to novel tasks has been longstanding goal of artificial intelligence. Recently, large transformer models have shown strong generalization capa- bility language understanding when fine-tuned with limited data (Brown et al., 2020; Wei et al., 2021). Such success motivates researchers to apply transformer models to the regime of offline re- inforcement learning (RL) (Chen et al., 2021; Janner al., 2021). By scaling up the model size and leveraging large offline datasets from diverse training tasks, transformer models have shown be generalist agents successfully solving multiple games with single set of parameters (Reed et al., 2022; Lee et al., 2022). Despite the superior performance in the training set of tasks, directly deploying these pre-trained agents to novel unseen tasks would still lead suboptimal behaviors.
One solution is to leverage the handful of expert demonstrations from the unseen tasks to help policy adaptation, and this has been studied in the context of meta imitation learning (meta-IL) (Duan et al., 2017; Reed et al., 2022; Lee et al., 2022). In order to deal with the discrepancies between the training and testing tasks, these works focus on fine-tuning the whole policy model with either online rollouts from the test environments. However, with the advent of large pre-trained transformers, it is computationally expensive to fine-tune the whole models, and is unclear how perform policy adaptation efficiently (Figure (a)). We aim to fill this gap in this work by proposing a more parameter-efficie solution.
expert demonstrations
Moreover, previous work falls short in tasks only provide demonstrations without expert actions. This similar to the state-only imitation learning or Learning- -rooinn--roo--ooo Prroo-bbserattiin (LfO) settings (Torabi et al., 2019; Radosavovic et al., 2021), where expert actions are unavailable, and therefore we term this setting as meta Learning- frrmm- Observation (meta-LfO). As result, we aim develop more general method that can address both meta-IL and meta-LfO settings.
challenging yet realistic setting where the target
Project Page: https: /tt///atsssss/s.///////////e goog1 wwt.cooo/eewweeeccccccce c2 /home.

---

Published as a conference paper at ICLR 2023
(1)
Preteainnig with Divense Training Taoks
Efficient Adaptaaion to Unoren Target Tasks For-shut Denoo
(b]
Recent Context
Large Pretrained Transformce
Low-level
Actions
Massive parameters
Large Protrained Transformer
Pretraining
Demo.
Demo.
Adaptation
Encoder
Module
Adaptation
Tiny persinaters
Figure 1:
Efficient online policy adaptation of pre-trained transformer models with few-shot demon- adaptation module
strations. To facilitate data efficiency, we introduce a demonstration-conniiine that helps leverage prior knowledge in the demonstration and guide exploration. When adapting to novel tasks, we only fine-tune the adaptation module to maintain parameter efficiency.
The closest work to ours is Prompt-DT (Xu et al., 2022), which proposes to condition the model behavior in environments a few demonstrations as prompts. While the method is originally evaluated for meta-IL, the flexibility of the prompt design also allows this method to be useful for meta-LfO. However, find that Prompt-DT hardly generalizes to novel environments (as is shown empirically in Section 4), since the performance of the in-context learning paradigm, e.g., prompting, generally inferior to fine-tuning methods (Brown et al., 2020). The lack of efficient adaptation mechanism also makes Prompt-DT vulnerable to unexpected failures in unseen tasks.
In order achieve the strong performance of fine-tuning-bast methods as well maintain the efficiency of in-context learning methods, we propose Hlyee-Decisioon Transformer (HDT) for large pre-trained Decision Transformers (DT) (Chen al., 2021). HDT composes of three key compo- nents: (1) a multi-task pre- pre-trained DT model, (2) adapter layers that can be updated when solving novel unseen tasks, and (3) hyper- that outputs parameter initialization of the adapter layer based on demonstrations. The praitrd DT model encodes shared information across diverse training tasks and serves as a base policy. To mimic the performance of fine-tuning methods, we introduce an adapter layer with a bottleneck structure to each transformer block (Houlsby et al., 2019). The parameters the adapter layers can be updated to adapt to a new task. Moreover, it only adds small fraction of parameters to the base DT model. The adapter parameters are initialized by a single hyper-network conditioning on the demonstrations with or without actions. In such a way, the hyper-network extracts the task-specific information from demonstrations, which is encoded into the adapter's 's initialized parameters.
We evaluate HDT in both meta-IL (with actions) and meta-LfO (without actions) settings. In meta- II., the adapter module could directly fine-tune in a supervised manner as few-shot imitation learn- ing. In meta-LiO, the agent interacts with the environment and performs RL, while conditioning on the expert states. We conduct extensive experiments in the Meta- World benchmark Yu et al. (2020), which contains diverse manipulation tasks requiring fine-grind gripper control. We train HDT with 45 tasks and test its generalization capability in 5 testing tasks with unseen objects, seen objects with different reward functions. Our experiment results show that HDT demonstrates strong data and parameter efficiency when adapting to novel tasks.
We list our contributions as follows:
1. We propose Hyper-Decision Transformer (HDT), wansformer-based model which generalizes to novel unseen tasks maintaining strong data and parameter efficiency.
In the meta-IL setting with only one expert demonstration, HDT only fine-tunes a small fraction (0.5% of the model parameters and adapts faster than baselines that fine-tune the whole model, demonstrating strong parameter efficiency.
3. In the meta-LfO setting with only 20-80 online rollouts, HDT can sample successful episodes and therefore outperforms baselines by large margin in terms of success rates, demonstrating strong data efficiency.
2 RELATED WORK
Transformers in Policy Learning. Transformer (Vaswani et al., 2017) has shown success in natu- ral language tasks thanks to its strong mequence-modeling capacity. As generating policy in RI. is
2

---

Published as a conference paper at ICLR 2023
essentially sequence prediction problem, Chen et al. (2021) propose Decision Transformer (DT) that adapts the Transformer architecture to solve offline RL problems, achieving performances com- parable to traditional offline RL algorithms. This work also inspires adaptions of Transformer in different RL settings. Zheng et al. (2022) propose Online DT,, which enables DT to explore online and applies it in an online RL setting. Xu et al. (2022) use the prompt in Transformer provide task information, which makes the DT capable of achieving good performance in different tasks. Reed et al. (2022) also exploit the prompt in Transformer and train one Transformer model solve different games or tasks with different modalities. Lee et al. (2022) propose multi-game DT which shows strong performance in the Atari environment (Bellemare et al., 2013). Furuta al. (2021) identify that DT solves a hindsight information matching problem and proposed Generalized DT that can solve any hindsight information matching problem.
One-shot and Meta Imitation Learning. Traditional imitation learning considers single-task setting, where the expert trajectories are available for a task and the agent aims the learn an opti- mal policy for the same task. One-shot and meta-imitation learning aims to improve the sample efficiency of imitation learning by learning from a set of training tasks and requires the agent to generalize to different tasks. Duan et al. (2017) train one-shot imitator that can generate optimal policies for different tasks given the demonstrations of the corresponding task. Finn et al. (2017b) enable robot to learn new skills using only one demonstration with pixel-level inputs. James al. (2018) first learn task embeddings, and use the task embedding of the test task jointly with test task demonstrations to find the optimal control policy in robotic tasks.
Learning from Observations. When deployed in a new environment or tested on a new task, it may be challenging for an agent achieve good performance without any information about the task. Therefore, the agent can benefit from starting with imitating some expert trajectories (Argall al., 2009). Usually, expert trajectories consist of sequence of state, action pairs rolled out from an expert policy. However, assuming the availability of both states and actions can prevent the agent from learning from many resources (for example, online video demonstrations). So we consider a setting where such trajectories may only contain states and rewards rolled out from the expert policies. This is considered as learning from observations in the literature (Torabi et al., 2019). Prior work trains inverse dynamics model and uses it to predict actions based on states (Radosavovic et al., 2021) and enhances learning from observation methods by minimizing its gap to learning from demonstrations (Yang et al., 2019). Transformers are also applied in this problem achieve one-shot visual imitation learning (Dasari & Gupta, 2020).
Hyper-networks. The idea of using one network (the hyper-network) generate weights for an- other (the target network) is first introduced in Ha et al. (2016). In the continuing learning setting, Von Oswald et al. (2019) proposes generating the entire weights of target network while condi- tioned on the task information. In the robotic setting, Xian et al. (2021) considers a model-based setting and uses hyper-networks to generate the parameters of neural dynamics model according to the agent's 's interactions with the environment.
Parameter-efficiee Adaptation. When modern neural network models become increasingly large, generating or fine-tuning the full model becomes increasingly expensive. Lester et al. (2021) in- troduces "soft prompts" to alternate the output of pre-trained transformer model. Task- specifpe soft prompts enable the transformer model to output answers for the given task. Liu et al. (2022) proposes adding a few controlling vectors in the feedforward and the attention blocks to control the behavior of a pre-trained transformer model. Mahabadi et al. (2021) uses a hyper-network to gen- crate task-specific adaptation modules for transformer models from a task embedding. This method enables efficient adaptation by just learning the task embedding vector.
HYPER DECISION TRANSFORMER
To facilitate the adaptation of large-scale transformer agents, we propose former (HDT), Transformer-based architecture maintaining data and parameter efficiency during adaptation to novel unseen tasks. HDT consists three key modules as in Figure 2: a base model encoding shared knowledge across multiple tasks (Section 3.2), hyper-network representing task- specific meta-learner (Section 3.4), and an adaptation module updated to solve downstream unseen tasks (Section 3.3). After presenting the model details, summarize the training and adaptation algorithms in Section 3.5.
Decision Trans-
3

---

Published as a conference paper at ICLR 2023
3.1 PROBLEM FORMULATION: EFFICIENT ADAPTATION FROM OBSERVATIONS
We aim to improve the generalization capability of large transformee--assed agents with limited online interactions and computation budgets. Existing large wansformer-basee agents have demon- strated strong performances in training tasks, and may have sub-optimal performances when directly deployed in novel tasks (Reed et al., 2022; Lee et al., 2022; Xu et al., 2022). Although the discrep- ancy between the training and testing tasks has been widely studied in meta RL (Finn et al., 2017a; Duan et al., 2016), efficient adaptations of large transformer- agents are still non-trivial. Be- yond the data efficiency measured by the number of online rollouts, we further emphasize the pa- rameter efficiency measured by the number of parameters updated during online adaptation.
We formulate the environment M = (S, P, R. (u). S and A are the state and the action space, respectively. P is the transition probability and P S x A S + IR R is the reward function where R S + IR f is the initial state distribution. Following Xu et al. (2022); Torabi et al. (2019), we consider efficient adaption from demonstrations with a base policy pre-trained with set of diverse tasks. Formally, we assume access to a set of training tasks Trrasn Each task T, E Térain is accompanied by a large offline dataset Dj and few demonstrations P:: where i E [|Ttrain|]] collected with an (unknown) behavior policy, where h (5o. 4o, a, E A, r as the per-timestep reward, and H as the episode length.
Markov Decision Process (MDP) represented by 5-tuple
[]- The dataset D; contains trajectories h 8H, ag TH) with s; E S,
We consider the efficient adaptation problem in two settings; meta-imitation learning (meta-IL) and meta-learning from observations (meta-LfO). In meta-IL, the demonstration dataset P consists of full trajectories with expert actions, h = (8o, dg To, 88H, OH, ). In contrast, in meta-LfO,, the demonstration dataset P consists of sequences of state and reward pairs, ho = (80, We evaluate the generalization capability in a set of testing tasks Trestt each accompanied with a demonstration dataset P. Note that the testing tasks are not identical to the training tasks and may not follow the same task distribution as the training ones. For instance, in manipulation tasks, the agent may need to interact with unseen objects.
&H,TH) )-
3.2 DECISION TRANSFORMER (DT) AS THE PRE-TRAINED AGENT
Large transformer- beased agents are highly capable of learning multiple tasks (Lee et al., 2022; Reed et al., 2022). They follow offline RL settings and naturally cast RL as sequence modeling problems (Chen et al., 2021; Janner et al., 2021). By tasking a historical context as input, transformers could leverage more adequate information than the observations in the current timestep, especially when the dataset is diverse and the states are partially observable. In this work, we follow the formulation of DT (Chen al., 2021), which autoregressively generates actions based recent contexts. At each timestep 4, DT takes the most recent K K-step trajectory segment T as input, which contains states actions and rewards-to-go i ET=
(ft- ( K+1 at- ............. Pe, at).
DT predicts actions at heads, where action or is the action the agent predicts in state 8t. The model minimizes the mean squared error loss between the action predictions and the action targets. In contrast to behavior cloning (BC) methods, the action predictions in DT additionally condition on rewards-to-go and timesteps. The recent context contains both per-step and sequential interactions with environments, and thus encodes task-specific information. Indeed, in Lee et al. (2022), recent contexts could embed sufficient information for solving multiple games in the Atari environment (Bellemare et al., 2013), While in situations where a single state has different optimal actions for different tasks, additional task-specific information (eg. the demonstration in target tasks) injected into the model further helps improve the multi-task learning performance (Xu et al., 2022).
In this work, we pre-train DT model with datasets from multiple tasks the base agent. is worth noting that the base DT model could be replaced by any other pre-trained transformer agent.
3.3 ADAPTATION MODULE
To enable parameter-efffcieet model adaptation, insert task-specific adapter layers to transformer blocks. Adapter-basee fine-tume fine-tuning is part of pa parameter-efficienn fine-tuning of large language models and generalizes well in NLP benchmarks (Mahabadi et al., 2021). The adapter layer only contains
4

---

Published as a conference paper at ICLR 2023
Hyper-Decision Transformer
Transformer Block
Hyper-network
Adaatter
Adapter Purumetess
Adapter
FILM
Forword
Transformer Block
LayerNarm
Projectoey
Carsilaie
+
Transformer Block
GELU
Sell-Atteation
Haper
RYG
LayerNorm
Projection
LD
Figure 2: Model architecture of Hyper- Sepee-Deciiion Transformer (HDT), Similar DT. HDT takes recent contexts as input and outputs fine-grind actions. To encode task-specific information, HDT injects adapter layers into each decoder block. The adapter layer's parameters come from stand- alone hyper-network that takes both demonstrations without actions and the decoder's 's layer id.
small number of parameters compared with the base DT policy. When fine-tuning the down- stream unseen tasks, HDT only updates the adapter layers with the base DT model's parameters frozen. Such task-wise adaptation mechanism helps maintain parameter efficiency and avoid neg- ative interference where adapting to a new task could lead to decreasing performances in other tasks.
Considering that DT is decoder-only transformer model, we insert adapter module into each decoder block and place it before the final residual connection, as shown in Figure 2 (middle). Let the hidden state be X RRxd, where K is the context length and dy is the embedding dimension for each token. For each decoder block with layer id I, its adapter layer contains a down-projectin layer Dj xdv, d4, < dz, GELU nonlinearity, an up-projection layer U/ E Rd, and feature-wist wise linear modulation (FiLM) layer FiLM, (Perez et al., 2018). The bottleneck architec- ture constructed by the down- and up-projection layers helps reduce the adapter's parameters. The FiLM layer consists of weights "/2 & and feature-wise biases 31 E Rd= which help selectively manipulate features. Formally, the adapter layer transforms the hidden state X by
Ad) (GELU Di) O Br,
where @ is the Hadamard product. For each task, we initialize the adapter layer with a shared hyper-network across multiple tasks and encoder layers as described in Section 3.4.
3.4 HYPER-NETWORK
To promote data-efficient online adaptation, we propose to use shared hyper-network to generate the adapter layers' initial parameters. The hyper-network captures common information across tasks by multi-task training. It also encodes task-specific information for each task by taking the task demonstration without actions ht" as input, applicable for both meta-IL and meta-LfO settings. Given an unseen task, T E Teest and its accompanied h?, the trained hyper-network could generate good parameter initialization that facilitates quick online rollouts. Motivated by Mahabadi et al. (2021), HDT further utilizes a compact hyper-network for generating adapters of each block and takes the layer id as the additional input.
The hyper-network module consists of linear layers to get emheddings for each modality, 1D convolutional module Conv to get shared task-specific encoding 2 IR Hdd where H is the demonstration length, and set of linear layers to get adapter parameters. Formally, let the weights of linear embedding layers be L, E 1Rd,xd, LL E 1xd. L, E 1 L, 1 for the state, sowards-to-go. to-go, timesteps, and layer id, respectively. Similar to timesteps representing temporal positions, the layer id represents positional information of decoder blocks. Hence, we add layer id embedding to state, action, and reward-to-go tokens. There are four types of parameter-generation layers with weights, T.D Rdzdd xd: for down proje-pooectioonn layers, Ta E &Rdad, Xd, for up-projection layers, and L, E xds La E IB for FiLM layers. Formally, the hyper-network conducts the following operations.
Ut, Di,,, B1 Hyper-network((f..)) where
2 = Conv( concat( (h²) + (,")) + (h"),, L( + L((h)) + ))))))),
U: Luz, D, LD2, 71 L,2, 31 Lgz.
5

---

Published as a conference paper at ICLR 2023
Algorithm Hyper-network Training
Input: training tasks Ttrain HDTs with hyper-network parameters 6, training iterations N, offline dataset D {h (8o. do, ro, ,81, (HH, rn)), demonstrations P, per-task batch size
M, learning rate (rub
2: for 1 to N do for Each task T: E T do Sample M trajectory of length K from D, and demo. hg from P;
3:
4:
5:
Get minibatch B.M {(h, Ti.m )
6: Get a batch B = -1,111 T=
[T
7: ared HDT (h. Ta.m) V(h,, m.m) B
8:
0s Vo |18 Σ (AG.T,,)EBB arredy3 where refers to actions in Tt,m
Algorithm Efficient Policy Adaptation without Expert Actions (meta-L.f)
1: Input: testing task T E Trest, HDT, with adapter parameters v), online rollout budget Nepi.
one-shot demonstration without actions P, batch size M, learning rate ag, exploration rate
2: Initialize adapter parameters vé with trained heper-network
3: Initialize empty data buffer D = {0}
4: while episode number less than Napi do
5: Collect one episodic trajectory h with e-greedy
6: Relabel rewards-to-2e of h with actual rewards and append h to data buffer D
7: Sample M segments T of length
from D, and demo. h? from P
8: Get a batch B = {hh, Tm 3mm== m==1
9:
arred t HDT (h", Tm). Y(h (m)) Tm) EB
10:
V t V - av Ve LmggsmeeBl arrody? where a refers to actions in Tm
3.5 ALGORITHM
DT Pre-training. We first pre-train DT model with datasets across multiple tasks. We assume that the datasets do not contain trajectories collected in testing tasks. We defer the detailed training algorithm in the appendix in Algorithm
Training Hyper- We train the hyper-networks as shown in Algorithm 1. For notation sim- plicity, we denote hyper-network parameters ø. During hyper-network training, the pre- pre-trained transformer agent's parameters are frozen. HDT only updates hyper-network parameters & to make sure the hyper-network copes with the pre-trained agent well as extracts meaningful task specifcc information from demonstrations h To stabilize training, each gradient update á based on a large batch containing trajectory segments in each task. Following Chen al. (2021), minimize the mean-squared error loss between action predictions and targets.
Fine-tuning on Unseen Tasks. When fine-tuning in unseen target tasks, HDT first initializes adapter parameters (denoted as v)) with the pre- pre-trained hyper-network and then only updates the adapter parameters. When there only exist demonstrations without actions h" in the target task (meta-LfO), HDT initializes an empty data buffer and collects online rollouts with e-greedy as in Algorithm 2. Following Zheng al. (2022), updates adapters with the same mean-squared eтror loss as training hyper- heter-ss When there exist expert actions in the demonstration (meta-IL). HDT could omit the online rollout and directly fine-tune the adapter with the expert actions following Algorithm 4 in appendix Section A.
relabel the rewards-to-pe of the collected trajectories. HDT
EXPERIMENTAL SETUP
We conduct extensive experiments to answer the following questions:
Does HDT adapt unseen tasks while maintaining parameter and data efficiency?
How HDT compared with other prompt-finettune and parameter-efficient fine-tuning methods in the field of policy learning?
Does the hyper-network successfully encode task-specific information across tasks?
Does HDT's adaptivity scale with training tasks' diversity, the base DT policy's model size, and the bottleneck dimension?

---

Published as a conference paper at ICLR 2023
4.1 ENVIRONMENTS AND OFFLINE DATASETS
The Meta- World benchmark (Yu et al., 2020) contains table-top manipulation tasks requiring a Sawyer robot to interact with various objects. With different objects, such as drawer and win- dow, the robot needs to manipulate them based on the object's s affordance, leading to different reward functions. At each timestep, the Sawyer robot receives a 4-dimensional fine-grind action, represent- ing the 3D position movements of the end effector and the variation of gripper openness. We follow the Meta- World MLL5 benchmark. We use training set containing 45 tasks for pre-training DT and neper-networks. The testing set consists of 5 tasks involving unseen objects or seen objects but with different reward functions.
For each training task, we collect an offline dataset containing 1000 episodes with the rule-based script policy provided in Yu et al. (2020) and increase the data randomness by adding random noise to action commands. For each testing task, we collect one demonstration trajectory with the script policy, Note that the expert rule-based script policy is tailored to each task and has an average success rate of about 1.0 for each task in the training and testing sets. We set the episode length as 200. Each episode contains states, actions, and dense rewards at each timestep.
4.2 BASELINES
We compare our proposed HDT with six baselines answer the questions above. For each method, we measure the task performance in terms of the success rate in each testing task, the parameter efficiency according to the number of fine-tuned parameters during adaptation, and the data efficiency based on the top-2 number of online rollout episodes until success episode. All methods except for SiMPL utilize a base DT model with size [512, 4, 8), which are the embedding dimension, number of blocks, and number of heads, respectively.
PDT. Prompt Decision Transformer (PDT) (Xu et al., 2022) generates actions based both the recent context and pre-collected demonstrations in the target task. To ensure a fair comparison, omit the actions in the demonstration when training PDT with T During fine-tuning, we conduct prompt-tuning by updating the demonstration with the Adam optimizer. PDT helps reveal the difference between parameter-efficient tuning and prompt-tuning policy learning.
DT. We fine-tune the whole model parameters of the pre-trained DT model during online adapta- tion. DT helps show the data- and -fffcieer-ffccciecyy of HDT.
HDT-IA3. To show the effectiveness of the adapter layer in policy learning, we implement an- other parameter-effiiien fine-tuning method, HDT-IA3, motivated by Liu et al. (2022). HDT- IA3 insert weights to rescale the keys and values in self-attention and feed-forward layers. Both HDT--A3 and HDT do not utilize position-wise rescaling. In other words, HDTTIA3 trains hyper- network to generate rescaling weights shared by all positions in each block. During fine-tuning, only the rescaling weights are updated. We detail HDT-IA3's structure Section A.3.
SiMPL. Skill-based Meta RI. (SiMPL) (Nam et al., 2022) follows SPiRL's setting (Pertsch et al., 2020) and uses online rollouts with RL fine-tune on downstream unseen tasks. We pick this additional baseline for the meta-LiO, since this setting can be thought of as meta-RL with obser- vations. We aim show the improvements when having access expert observations.
HDT-Rand. To reveal the effect of the adapter initialization with pre-trained hyper-networks, we compare HDT with HDT-Rand which randomly initializes adapter parameters during fine-tuning. RaT-Rand only updates the adapter parameters similar to HDT.
5 RESULTS AND DISCUSSIONS
5.1 DOES HDT GENERALIZE TO UNSEEN TASKS WITH PARAMETER AND DATA EFFICIENCY?
When fine-tuning in unseen tasks with demonstrations containing no expert actions, HDT achieves significantly better performance than baselines in terms of success rate, parameter efficiency (Ta- ble 1). and rollout data efficiency (Table 2).
Success Rate. Without fine-tuning, HDT could only achieve a success rate of around 0.12 in testing tasks. In the meta-LfO setting, after collecting handful of episodes in each task, HDT demonstrates a significant improvement to a success rate of around 0.8, as shown in Table 1. In the door-lock and door-unlock tasks, where the object already shows up in the training tasks, HDT could fully
7

---

Published as a conference paper at ICLR 2023
[al ML45 Train
(b) ML45 Test (meta-iL)
[c] MLA5 Test (meta-LfO)
1.0
0.8
0.0
0.6
Sec0 Rotee
0.6
Sucress Rate
0.6-
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
1:00 2500 2000
4000
75
3.2 2000
75
200
HDT PDT
DT
HOT-AS
SIMPL
Figure 3: Qualitative results in WorldWorld benchmark. Each curve is averaged across 5 seeds. We show the training curves of our proposed HDT and baselines in (a), the adaptation performance with one-shot demonstration containing expert actions (meta-IL.) in (b), and adaption performance with a demonstration containing no expert actions (meta-LfO) in (c). When expert actions are unavail- able, HDT outperforms baselines by large margin.
Table I: Quantitative results on Meta- World ML45 benchmarks.
Model Sizes
Merld ML Peformances
Adaptation
Percentage
Train
Test (no-FT)
Test (meta-II., I shot)
Test (meta-I.fO)
HDT
69K
0.5%
0.89 0.00
0.12.1 0.01
0.93 0.10
0.80 : 0.16
PDT
6K
0.05%
0.88 0.00
0.06+ 0.05
0.04 + 0.07
0.09 = 0.01
DT
13M
100%
0.70 + 0.04
0.08 0.03
0.87 + 0.15
0.46 = 0.21
HDT-AA3
6K
0.05%
0.88 0.00
0.04 0.02
0.10 0.01
0.16 = 0.14
solve the tasks. In more challenging tasks where there exist unseen objects, including bin-picking, box-close, and hand-insert, HDT still outperforms baselines by a large margin, as in Table 2 and Figure 3. SiMPL, without utilizing the demonstrations, could achieve performance improvements while still underperforming our proposed HDT. Such observations show the benefit of leveraging prior information from demonstrations.
Parameter Efficiency. In meta-LiO, fine-tuning the whole parameters of the base DT model (de- noted as DT) comes third in terms of the average success rate (Figure 3 (c)). However, it requires a significantly larger amount of computation budget to fine-tune 13M parameters compared with our proposed HDT, which fine-tunes 69K parameters (0.05% of 13M) as in Table 1. In the simpler setting with expert actions (meta-IL), all baselines require no online rollouts and update with expert actions. In this case, HDT converges faster than fine-tuning DT as in Figure (b).
Data Efficiency. In meta-LiO, we set the online rollout budget as 4000 episodes. As in Table 2, HDT could sample successful episode in around 20 to 80 episodes, much smaller than the number of episodes required by baselines. Such a comparison shows that the adapter module initialized by the pre-trained hyper-network helps guide the exploration during online rollouts.
5.2 ADAPTER LAYERS V.S. OTHER EFFICIENT FINE-TUNING METHODS IN POLICY LEARNING
Prompt-tuning and parameter-efficient fine-tuning are popular paradigms in tuning pre-trained large language models Liu et al. (2021). We are interested in the effectiveness of methods both regimes when tuning pre-trained transformer agents. We treat PDT the representative prompt- tuning baseline and HDT-IA3 as another baseline in the regime of parameter-efficient fine-tuning, We show that HDT outperforms both terms of adaptation performances in both meta-IL II and meta- LfO settings. Although PDT could sample successful episodes quickly during online adaptation (Table 2), purely updating the prompt input hardly improves success rates. We observe similar trends when fine-tuning with expert actions (Figure 3 (b)). HDT-IA3 could sample success episodes but may require a larger number of rollouts than PDT and HDT. As in Figure (c), fine-tuning HDT-IA3 could result in success rate improvements, which is much less than our proposed HDT.
5.3
DOES THE HYPER-NETWORK ENCODE TASK-IPECIFIC INFORMATION?
HDT relies on the hyper-network to extract task-specific information and utilizes the information by generating the adapter's parameters. We hope to investigate whether the hyper-network encodes meaningful task speciffc details. We visualize the adapter's parameters initialized by the hyper- network for each task in Figure The detailed visualization process is deferred to Section B.2. Figure 7 shows that the adapters' parameters for different tasks could be distinguished from each
8

---

Published as a conference paper at ICLR 2023
Per-task Quantitative results without expert actions (meta-LfO). For each testing task, we present the success rate averages across episodes and measure the data efficiency based on the average of the smallest and the second smallest numbers of online rollout episodes until collecting success episode. The symbol "x" means that no successful episodes are sampled during online rollouts. We highlight the best performances.
Table 2:
bin-picking
box-closc
hand-insert
door-lock
docr-unlock
success rate
data eff.
Suicess rate
data eff.
success
data eff.
success
data eff.
success
data eff.
0.60 0.49
0.80
0.34
0.60
1.00
1.00
PDT
0.00 1: 0.00
40
0.06
2 0.12
20
0.00 2 0.00
20
0.59
+ 0.48
20
0.00 1 0.00
170
0.16
0.37
0.37
0.16
HDTHAA3
0.00 1: 0.00
190
0.00
1 0.00
950
0.00
2 0.00
0.00
+ 0.38
0.75
1: 0.25
20
HDT Rand
0.00
0.07
0.00
2600
HDT HDT--malll--e train
0.00 1 0.00
440
0.00 2 0.00
440
0.83 : 0.29
800
0.17 + 0.29
1220
0.67 + 0.58
20
Da Taseett [1
[diTo: (netee
150
100
17%
179
100
200
Figure 4: Ablation results to show the effect of training tasks and model size. Decreasing the adapter's bottleneck hidden size would slow down the convergence when there are expert actions as in (a), and cause significant performance drop when no expert actions in (b). Similar trends are observed with decreased base DT's model size as in (c) and (d). With 10 training tasks, HDT-small- train underperforms HDT.
other, showing that the hyper-network indeed extracts task-specific information. To support the argument, we further visualize the environment rollouts in the 5 testing tasks in Section B.I. Com- pared with HDT-Rand, which randomly initializes adapter layers, HDT has much better adaptation performance with a higher success rate and strong data efficiency (Table 2 and Figure 4).
5.4 DOES HDT'S PERFORMANCE SCALE WITH TRAINING TASKS AND MODEL SIZES?
Lee et al. (2022) show that the data diversity and model size are crucial when training large transformer- baasdd models in the multi-game setting. We hope to understand whether similar trends show up when training HDT. The ablation results are summarized Table and Figure 4. To un- derstand the effect of training task diversity, we compare HDT with HDT-small-train which trains the hyper-network and the base DT model with 10 training tasks. In meta-LfO, HDT-small train underperforms HDT in terms of success rates and efficiency when sampling successful episodes. Such results show the importance of training task diversity. It is worth noting that with a smaller DT model size, the transformer agent could hardly sample successful episodes in meta-LfO and thus no improvements during online adaptation (Figure 4 (d)). The performances also drop with smaller hidden sizes of the adapter layers. With expert actions in meta-IL, smaller DT model size and bottleneck dimension lead to slower convergence.
6 CONCLUSION
We propose Syper-Decision Transformer (HDT), a transformer-basee agent that generalizes to un- seen novel tasks with strong data and parameter efficiency. HDT fine-tunes the adapter layers in- troduced to each transformer block during fine-tuning, which only occupies 0.5% parameters of the pre-trained transformer agent. We show that in the Meta-Worke benchmark containing fine-grind manipulation tasks, HDT converges faster than fine-tuning the overall transformer agent with expert actions. Moreover, HDT demonstrates strong data efficiency by initializing adapter layers' param- eters with a hyper-network pre-trained with diverse tasks. When expert actions are unavailable, HDT outperforms baselines by a large margin in terms of success rates. We attribute the strong performance to good initializations of the adapter layers, which help HDT achieve successful online rollouts quickly. We hope that this work will motivate future research on how optimally fine- tune large transformer models to solve downstream novel tasks. Interesting future directions include scaling HDT up to handle embodied AI tasks with nigh-dimensional egocentric image inputs.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This project was supported by the DARPA MCS program, MIT-IBM Watson AI Lab, National Science Foundation under grant CAREER CNS-2047454, and gift funding from MERL, Cisco, and Amazon.
REFERENCES
Brenna D Argall, Sonia Chernova, Manuela Veloso, and Brett Browning. survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-455 483, 2009.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013. 3,4
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901, 2020. 1,
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 14:15084-15097, 2021. 2. 3, 6
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. arXiv:2011.05970, 2020. 3
arXiv preprint
Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:/611.02777 2016.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. Advances in neural information processing systems, 30, 2017. 1,
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135. PMLR, 2017a. 4
Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot visual imi- tation learning via meta- meta-learning. In Conference on robot learning, pp. 357-368, PMLR, 2017b.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020. 17
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021. 17
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline
hindsight information matching. arXiv preprint arXiv:2111.10366 2021. 3
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: 2016.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffcient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Stephen James, Michael Bloesch, and Andrew J Davison. Task-embedded control networks for few-shot imitation learning. In Conference robot learning, pp. 783-795. PMLR, 2018. 3
10

---

Published as a conference paper at ICLR 2023
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. Advances in neural information processing systems, 34:1273-1221 2021. 1, 4
lyy Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:21
2021. 17
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 1111179--111 1191, 2020. 17
Kuang- Huei Lec, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transform- ers. arXiv preprint arXiv:2205. 15241, 2022. 1,3,4, 4,9
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-effiiient prompt tuning. arXiv preprint (tti::20040869,,, 2021. 3
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter- efffccett fine-tuning is better and cheaper than in-context learn- ing. arXiv preprint arXiv:2205.050m 2022. 7, 14
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13586, 2021. 8
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2106.0444 04489, 2021. 3, 4,5
Taewook Nam, Shao-Hua Sun, Karl Pertsch, Sung Ju Hwang, and Joseph J Lim. Skill-based meta- reinforcement learning. arXiv preprint 12XX::2244.11226 2022.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018. 5
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010. 1944, 2020. 7
Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. State-only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7865-7871. IEEE, 2021. 1,3
Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175
2022. 1,3,
Michita Imai Takuma Seno. d3rlpy: An offline deep reinforcement library. In NeurlPS 2021 Offline Reinforcement Learning Workshop. December 2021. 17
Faraz. Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from obser- vation. arXiv preprint arXiv: 2019. 3,
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Johannes Von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint arXiv:1906.0005 00695, 2019. 3
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. arXiv preprint 1rXiv:2109.0/66 2021. 1
11

---

Published as a conference paper at ICLR 2023
Zhou Xian, Shamit Lal, Hsiao- Yu Tung, Emmanouil Antonios Platanios, and Katerina Fragkiadaki. Hyperdynamics: Meta-learning object and agent dynamics with hypernetworss. arXiv preprint arXiv:2103.09439, 2021. 3
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Con- ference on Machine Learning, pp. 24631-24644 PMLR, 2022. 2, 4.7
Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. Ad- vances neural information processing systems, 32, 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Neta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100, PMLR, 2020. 2,
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint
arXiv:2202.05607, 2022, 3,6 6
12

---

Published as a conference paper at ICLR 2023
A ADDITIONAL ALGORITHM DESCRIPTIONS
We present the algorithms for pre-training DT models in Section A.1, efficient adaptation HDT when there exist expert actions in Section A.2, the model architecture of HDT-IA3 in Section A.3.
A.1 DT PRE-TRAINING
We pre-train a base DT model with large offline datasets collected from training tasks. Each gradient update of the DT model considers information from each training task.
Algorithm DT pre-training
1: Input: training tasks T training task size T = [Ttroin], DTa, training iterations N, offline dataset D. demonstrations P. per-task batch size M, learning rate og
2: for = 1 to N do
3:
for Each task T. TTtran do
4:
Sample M trajectory of length K from D;
5:
Get minibatch B,M (Tr.m))) m==
Get a batch B = (BMM Tirain apred DT (Ti.m) ((T.)) B
6:
1==1
7:
8:
LMSE [ Enimper EB (a apredy3
9:
i - f - VoLMSE
A.2 EFFICIENT ADAPTION WITH EXPERT ACTIONS (META-IL)
With expert actions, the transformer model does not need to conduct online rollouts and updates by fine-tuning with the expert actions.
Algorithm 4 Efficient Policy Adaptation with expert actions (meta-IL)
1: Input: testing task T E Tlest, HDT, with adapter parameters v/, one-shot demonstration with actions P, batch size M, learning rate Ag. iteration N 2: Initialize adapter parameters d with trained neper-network
3: for n n====1 to do
4: Sample M segments of length K from P, and demo. ho by omitting the actions
5: Get a batch
{(h, : Tm }
mmml
6: aored HDDT (ho, Tm). Vhe (m)) B
7:
1b v 1 at V [BT E Engg aPredy2
A.3 HDT-IA3
Transformer Block
Hyper-network
Foed Forward
Concoltion
LayerNorm
Self-Attention
Laver ID
RIG
State
Timeslap
LayerNorm
Figure 5: Model Architecture of HDTT-IA.
We show the model architecture of HDT-IA3 in Figure Compared with our proposed HDT, HDT- IA3 has hyper-network that takes the same encoding of demonstrations as HDT but outputs IA3
13

---

Published as a conference paper at ICLR 2023
parameters instead. The parameter efficient fine-tuning method, IA3, proposed by (Liu al., 2022), adds position-win -wise scaling weights to the self-attention and activations between feedforward layers. Considering that the transformer block in DT only contains feedforward layer, rescale the outputs of the feedforward layer instead. Moreover, to make fair comparison, we use one set of IA3 parameters for all positions, similar to the position-agnostic rescaling of HDT.
Formally, the IA3 parameters include 4 E [Rd=, ly E R E R The key The the self-attention module rescaled by la and lu respectively. The feedforward output is rescaled by lff-
and the value u of
A.4 MODEL SIZES
In this section, we describe the model size of different methods to help compare parameter efficiency.
Table 3: Detailed Model sizes for experiments with a large base DT model.
Base model
Hypernet
Adaptation
fine-tune type
Percentage
HDT
13M
2.3M
69K
Adapter layer parameters
0.5%
HDT-IA3
13M
0.2M
6K
TA3 parameters
0.05%
PDT
13M
6K
demonstrations
0.05%
DT
13M
13M
full model parameters
100%
Table 4: Model sizes when diffeernnt base DT models and adapter's bottleneck sizes.
Embedding
blocks
heads
bottleneck size
total
DT
updated
HDT
512
4
8
16
15.5M
13.1M
69K
HDT-medium
128
4
2.0M
1.3M
26K
HDT-small
128
3
1
16
1.4M
0.7M
13K
HDT-hidden-8
512
8
14.4M
13.1M
37K
HDT-hidden-4
512
4
8
13.9M
13.1M
20K
HDTT-hidden-2
512
4
8
13.6M
13.1M
12K
A.5 HYPER- PITER-PARAMETERS
Table 5: Hyperparameters for D'T-related models
Hyperparameters
Value
K (length of context r)
20
demonstration length training batch size for each task M pre-training iterations
200
16
4000
number of gradient updates in each iteration
10
number of evaluation episodes for each task learning rate og, Otg. Osg- learning rate decay weight activation
10
le-4
le-4
GELU
online rollout budget Nepi
4000
online rollout budget in each training iteration fine-tuning iterations exploration
20
200
0.2
SiMPL learning rate
5e-5
14

---

Published as a conference paper at ICLR 2023
B ADDITIONAL RESULTS IN META- WORLD BENCHMARK
B. THE EFFECT OF -EETORKS THROUGH ENVIRONMENT VISUALIZATION
We aim show the quality of the adapter's parameter initialization by visualizing the environment rollouts. The comparisons between our proposed HDT and the baseline HDT-Rand show that initial- izing the adapter layer with hyper-networks provides a strong prior helping guide online exploration.
C
(a) Bin picking task RaT robot directly goes to target positions interacting the
(b) The Bin-picking task with HDT. the robot tries to grasp the bin but falls to do sinze it doses the gripper toc fast.
C
(e) The door lock task with HDT-Rand. The robot fais manipu the lock but direcily door handle.
GAC
idi door-lokk task with HDT robot successt ally manipulates lock and the door open.
del The hanc--irtttt task with HDT-Rand The rebet fails meve the object and directly goes to the target position.
hand-insrtt
object target
object midway.
Figure 6: Environment Visualization. We provide key screenshots of three testing environment roll- outs to compare the initialization our proposed HDT and HDT-Rand, which randomly initialize the adapter layers.
B.2 T-SNE VISUALIZATION OF ADAPTERS' PARAMETERS INITIALIZED BY THE EYPER-NETWORK
To show whether the hyper-network extract task-specific information, visualize the adapters' pa- rameters initialized by the hyper-network in Figure 7. For each task, we randomly sample 100 expert demonstrations without actions, which serves input to the hyper-network, and collect 100 samples of the adapters' parameters as the output of the hyper-network. Considering that the adapters con- tain around 69k parameters which form a high-dimensional vector, we first reduce the dimension to 1000 via principle component analysis (PCA). We then project the PCA results 2-dimensional space using t-Distributed Stochastic Neighbor Embedding (t-SNE). We use sklearn to conduct PCA and t-SNE. We change the perplexity to 10, considering that samples. All the other hyperparameters except the perplexity are the same as the default parameters for PCA and t-SNE.
relatively small number of
In Figure 7, we show the transformed 2D features for both the training and testing tasks. The training tasks are labeled with black text and circle markers. The testing tasks labeled with red text and
15

---

Published as a conference paper at ICLR 2023
Pagee close
p.d saaee
Lig pushis
gpeep-9
hamphjorka
back-
spot open
plble sidee back didaminn www.nnnnnnssoooo cbress side
dige back
Bitton p prosopppon
Uiistckingg
De
box-choeev2
shelppaace
opens
bample ded--sidew
-50
crawer-opin--g
put
Aygata open 22g
doaslodkv22
Jophcllose-2
bundkroan 142
-00
30
350
Figure 7: T-SNE visualization of the initialized adapter layers based on pre-trained hyper-network. The cross marker and red text represent testing tasks. The dot marker and black text represent training tasks.
cross markers. We can see that the adapters' parameters for different tasks could be distinguished from each other, showing that the hyper-network indeed extracts task-specific information.
B.3 TESTING PERFORMANCES
We provide the quantitative per-task results in the testing tasks in Table 6 and Table 7. More concretely, Table 6 shows the results for HDT with different base DT model sizes and adapter hidden sizes when there exist no expert actions. Table shows the results for methods when the expert actions are available.
Table Per-task Quantitative results for ablation studies without expert actions (meta-LrO).
bin-picking
box-close rale rollouts
hand-insert
door-lock success rale rollouts
does-unlock
rate rollouts
rollouts
success rale rollouts
MiDT-medium
0.00 + 0.00
0.00 0.00
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
HDT-small
0.00
0.00 0.00
+
30
HDT HDT-hidden-8
0.25 : 0.43
1170
0,13 0.22
240
0.00
0.00
3160
0.38 + 0.41
40
0.50 : 0.50
2140
HDTThhddee-4
0.00 = 0.00
0.25 0.43
1180
0.00
0.00
0.00 + 0.00
x
0.00 + 0.00
3020
HDT-hidden-2
0.00 0.00
0.13
0.22
3710
0.00
0.00
0.00
180
16

---

Published as a conference paper at ICLR 2023
Table Per-task Quantitative results with expert actions (meta-IL).
bin-picking
box-close
hand-insert
door-lock
door-unlock
HDT
1.00 + 0.00
0.63 + 0.41
1.00 :: 0.00
1.00 + 0.00
0.98 + 0.04
PDT
0.25 1 0.43
0.25 0.17
0.00
1 0.00
0.33 1 0.41
0,50 0.50
DT
1 0.47
0.80 0.28 0.28
1.00
= 0.00
1.00 + 0.00
1.00 0.00
HDT-IA3
0.00 0.00
0.00 0.00
0.00
0.00 = 0.00
0.50 0.12
HDTT-medium
0.67 0.47
0.17
0.24
1.00 0.00
1.00 + 0.00
1.00 0.00
HDDT-small
0.16 + 0.24
0.29 0.24
0.00 + 0.00
0.83 + 0.24
0.17 + 0.24
HDT-hidden-&
1.00 1 0.00
0.28 1 0.22
1.00 1 0.25
0,00 1 0.00
1.00 1: 0.00
HDT-hhiden-4
0.00 + 0.00
0.38
+ 0.31
0.84 = 0.27
1.00 = 0.00
0.69 + 0.42
HDDT-hidden-2
0.00 0.00
0.16 0.27
0.00 0.00
1.00
0.00
0.69 0.23
B.4 ADDITIONAL BASELINES ON META-WORLD WITH 45 TRAINING TASKS
We train baselines on the training tasks of the Meta- World ML45 benchmark. We implement CQL Kumar et al. (2020), IQL Kostrikov et al. (2021), BC and TD3BC Fujimoto & Gu (2021) based on the d3ripy package Takuma Seno (2021). The main results are Figure 8. Note that CQL and IQL do not incorporate offline demonstrations when interacting with the environments. Thus is expected that CQL and IQL would perform worse than our proposed HDT.
From Figure 8 and Table 8, we can that in the meta-IL IL setting, BC and IQL could have perfor- mance improvement but still underperform HDT. In the meta-LfO setting, CQL and IQL are trained in online manner and underperform HDT. It is also worth noting that all four baselines here will suffer from the forgetting problem since all the model parameters are modified.
(meta-1L)
(meta-LLO
COL
BC
COL
IOL
TD38C
0.8
Success Batt
0.6
0.6
Success
03.4
0.4
0.2
25
50
100
325
150
200
25
75
100
125
150
175
200
Triining Iteration
Training Iteration
Figure 8: Qualitative results in Meta-World ML45 benchmark
Table 8: Quantitative results of baselines on Meta- Meta-World ML45 benchmarks.
Baselines in Meta- World ML45
Train
Test (no-FT)
Test (meta-IL, 1 shot)
Test (meta-LfO)
HDT
0.891 0.00
0.12# 0.01
0.93 - 0.10
0.80 0.16
BC
0.70 0.01
0.10 0.06
0.51= 0.22
TD3+BC
0.53 1 0.00
0.00 1 0.00
0,12: 0.13
CQL
0.06 = 0.01
0.04 + 0.07
0.32= 0.19
0.53+ 0.21
IQL.
0.62 + 0.00
0.13 + 0.18
0.53+ 0.20
0.31+ 0.17
B.5 ADDITIONAL EXPERIMENTS ON D4RL POINTMAZE ENVIRONMENTS
We hope to show the performance of HDT in another domain beyond manipulation tasks and thus add another set of locomotion tasks based on D4RL's pointmaze environment Fu al. (2020). We train all the methods with 50 training tasks and test in 10 testing tasks. The detailed description of the environment and tasks deferred to Section C.2.
The main results for the pointmaze environment are in Figure 9 and Table 9. We can see that HDT still outperforms baselines in terms of adaptation capability to unseen tasks in both meta-IL -L and meta-Lon Lfo settings. The results in the navigation domain help validate the main results presented in the original submission.
17

---

Published as a conference paper at ICLR 2023
(4) Pointmize Train
Paintmaze Test (metailL)
Pointmaze Test (metaiLFO))
1.0
0.5
0.8
0.8
0.6-
6.4
Suraes Rabe
0.6
0.4
C4
0.4
0.7
0.0
500
10.0
1500
2000
2500
3000
25
50
75
100
175
150
175
200
01
25
50
75
100
17%
150
175
200
Toining
Thiniing Rerstion
Taining
1107
PDT
DT
HDT-K3
Figure 9: Qualitative results in Pointmaze benchmark
Table 9: Quantitative results on Pointmaze environments.
Pointmaze Peformances
Train
Test (no-FT)
Test (meta-IL, shot)
Test (meta-LiO)
HDT
0.97 0.00
0.73 0.00
0.83 0.10
0.89 0.08
0.95 0.00
0.54 0.20
0.10
0.94 = 0.00
0.03
0.46 = 0.21
0.79 0.16
HDT-IA3
0.97 0.00
0.55
0.37 0.16
0.33 0.15
C ENVIRONMENT DETAILS
C.I META- WORLD EXPERIMENTS
C.I.1 SUCCESS CRITERION
We utilize the success signal feedback from the environment to calculate the success score, which is the sum of the success signals within one episode. To avoid false positive success identifications, we define successful episode if the success score is larger than 5 and the total episode return is larger than 300.
C.2 POINTMAZE EXPERIMENTS
C.2.1 DATA GENERATION
For each task in the training set, we use the expert rule-based controller provided the D4RL. package to generate 100 episodes. We use the medium-maze the maze layout. For each testing task, we also use the expert controller provided but only generate one episode.
C.2.2 SUCCESS CRITERION
To avoid false positive success identifications, we define a successful episode if the accumulated sparse reward is larger than 5.
C.2.3 TRAINING AND TESTING TASKS
Each task corresponds to different start and goal positions.
The list of 50 training tasks:
Task index: 0, start: [3,3], goal: [1,1]
Task index: 7, start: [3,3], goal: [2,5]
Task index: I, start: [3,3], goal: [1,2]
Task index: 8, start: [3,3], goal: [2,6]
Task index: 2, start: [3,3], goal: [1.5]
Task index: 9, start: [3,3], goal: [3,2]
Task index: 3, start: [3,3], goal: [1,6]
Task index: 10, start: [3,3], goal: [3,4]
Task index: 4, start: [3,3], goal: [2,1]
Task index: 11, start: [3.3], goal: [4,1]
Task index: 5, start: [3,3], goal: [2,2]
Task index: 12, start: [3,3], goal: [4,2]
Task index: 6, start: [3,3], goal: [2,4]
Task index: 13, start: [3,3], goal: [4,4]
18

---

Published as a conference paper at ICLR 2023
Task index: 14, start: [3,3], goal: [4,5]
Task index: 32, start: [4,4], goal: [2,5]
Task index: 15, start: [3,3], goal: [4,6]
Task index: 33, start: [4,4], goal: [2,6]
Task index: 16, start: [3,3], goal: [5,1]
Task index: 34, start: [4,4], goal: [3,2]
Task index: 17, start: [3,3], goal: [5,3]
Task index: 35, start: [4,4], goal: [3,3]
Task index: 18, start: [3,3], goal: [5,4]
Task index: 36, start: [4,4], goal: [3,4]
Task index: 19, start: [3,3], goal: [5,6]
Task index: 37, start: [4,4], goal: [4,1]
Task index: 20, start: [3,3], goal: [6,1]
Task index: 38, start: [4,4], goal: [4,2]
Task index: 21, start: [3,3], goal: [6,2]
Task index: 39, start: [4.4], goal: [4,5]
Task index: 22, start: [3,3], goal: [6,3]
Task index: 40, start: [4,4], goal: [4,6]
Task index: 23, start: [3,3], goal: [6.5]
Task index: 41, start: [4,4], goal: [5,1]
Task index: 24, start: [3,3], goal: [6,6]
Task index: 42, start: [4,4], goal: [5,3]
Task index: 25, start: [4,4], goal: [1,1]
Task index: 43, start: [4,4], goal: [5,4]
Task index: 26, start: [4,4], goal: [1,2]
Task index: 44, start: [4,4], goal: [5,6]
Task index: 27, start: [4,4], goal: [1,5]
Task index: 45, start: [4,4], goal: [6,1]
Task index: 28, start: [4,4], goal: [1,6]
Task index: 46, start: [4,4], goal: [6,2]
Task index: 29, start: [4,4], goal: [2,1]
Task index: 47, start: [4,4], goal: [6,3]
Task index: 30, start: [4,4], goal: [2,2]
Task index: 48, start: [4.4], goal: [6,5]
Task index: 31, start: [4,4], goal: [2,4]
Task index: 49, start: [4,4], goal: [6,6]
The list 10 testing tasks:
Task index: 50, start: [3,2], goal: [1,1]
Task index: 55, start: [3,4], goal: [2,2]
Task index: 51, start: [3,2], goal: [4,4]
Task index: 56, start: [4,5], goal: [6,5]
Task index: 52, start: [3,2], goal: [2,5]
Task index: 57, start: [4,5], goal: [3,2]
Task index: 53, start: [3,4], goal: [4,6]
Task index: 58, start: [4,5], goal: [5,3]
Task index: 54, start: [3,4], goal: [1,6]
Task index: 59, start: [5,4], goal: [3,2]
19


Reviews:
[{"review_id": "w0Y557rrgC", "reviewer": "Reviewer_aTrS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new approach\n\nDecision Transformers (DT) have demonstrated strong performances in offline reinforcement learning settings, but quickly adapting to unseen novel tasks remains\nchallenging. To address this challenge, we propose a new framework, called\nHyper-Decision Transformer (HDT), that can generalize to novel tasks from a\nhandful of demonstrations in a data- and parameter-efficient manner. To achieve\nsuch a goal, we propose to augment the base DT with an adaptation module,\nwhose parameters are initialized by a hyper-network. When encountering unseen tasks, the hyper-network takes a handful of demonstrations as inputs and\ninitializes the adaptation module accordingly. This initialization enables HDT to\nefficiently adapt to novel tasks by only fine-tuning the adaptation module. We validate HDT’s generalization capability on object manipulation tasks. We find that\nwith a single expert demonstration and fine-tuning only 0.5% of DT parameters,\nHDT adapts faster to unseen tasks than fine-tuning the whole DT model. Finally,\nwe explore a more challenging setting where expert actions are not available, and\nwe show that HDT outperforms state-of-the-art baselines in terms of task success\nrates by a large margin. Demos are available on our project page.", "summary_of_the_review": "This paper proposes a new approach for finetuning on new tasks after offline training on a set of different tasks. While the authors present some promising initial results, the empirical evaluation requires more work to warrant acceptance at the ICLR conference. In particular, the generality of the approach cannot be assessed without evaluating it on multiple well-established benchmarks designed in prior work (rather than by the authors themselves). More baselines and analysis could also further strengthen the paper. ", "strength_and_weaknesses": "# Strengths\n- important problem of interests for the community\n- promising initial results, the proposed approach seems to be quite effective \n- simple, intuitive approach\n- some interesting insights on how the method scales with the number of training tasks and model size\n\n# Weaknesses\n### 1. evaluation on a single domain\n- The method is evaluated only on the tasks from Meta World, a robotic manipulation domain. Hence, it is difficult to judge whether the results will generalize to other domains. I strongly recommend running experiments on a different benchmark such as Atari which is commonly used in the literature. This would also verify whether the method works with discrete action spaces and high-dimensional observations. \n\n### 2. evaluation on a setting created by the authors, no well-established external benchmark\n- The authors seem to create their own train and test splits in Meta World. This seems strange since Meta World recommends a particular train and test split (e.g. MT10 or MT50) in order to ensure fair comparison across different papers. I strongly suggest running experiments on a pre-established setting so that your results can easily be compared with prior work (without having to re-implement or re-run them). You don't need to get SOTA results, just show how it compares with reasonable baselines like the ones you already include. Otherwise, there is a big question mark around why you created your own \"benchmark\" when a very similar one exists already and whether this was somehow carefully designed to make your approach look better.\n\n### 3. limited number of baselines\n- While you do have some transformer-based baselines I believe the method could greatly benefit from additional ones like BC, transformer-BC, and other offline RL methods like CQL or IQL. Such comparisons could help shed more light into whether the transformer architecture is crucial, the hypernetwork initialization, the adaptation layers, or the training objective. \n\n### 4. more analysis is needed\n- It isn't clear how the methods compare with the given expert demonstrations on the new tasks. Do they learn to imitate the policy or do they learn a better policy than the given demonstration? I suggest comparing with the performance of the demonstration or policy from which the demonstration was collected. \n\n- If the environment is deterministic and the agent gets to see expert demonstrations, isn't the problem of learning to imitate it quite easy? What happens if there is more stochasticity in the environments or the given demonstration isn't optimal?\n\n- When finetuning transformers, it is often the case that they forget the tasks they were trained on. It would be valuable to show the performance of your different methods on the tasks they were trained on after being finetuned on the downstream tasks. Are some of them better than the others at preserving previously learned skills?\n\n### 5. missing some important details\n- The paper seems to be missing some important details regarding the experimental setup. For example, it wasn't clear to me how the learning from observations setting works. At some point you mention that you condition on the expert observations while collecting online data. Does this assume the ability to reset the environment in any state / observation? If so, this is a big assumption that should be more clearly emphasized and discussed. how exactly are you using the expert observations in combination with online learning? \n\n- There are also some missing details regarding the expertise of the demonstrations at test time. Are these demonstrations coming from an an expert or how good are they? \n\n# Minor\n- sometimes you refer to generalization to new tasks. however, you finetune your models, so i believe a better term would be transfer or adaptation to new tasks. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "n4BWI4D5po", "reviewer": "Reviewer_9MRj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new way to apply Decision-Transformer (DT) [1] to unseen tasks through the adapter layer initialized by a Hyper-network. They showed their method could adapt faster and better to unseen tasks with few parameter updates when comparing with previous works Prompt-DT [2], SiMPL [3], IA3 [4], and variations of their model. Especially their model can adapt even only with the demonstration of unseen tasks where expert action is unavailable.\n", "summary_of_the_review": "This paper proposes a new method for DT to adapt the unseen tasks. The way to do this is novel and shows better performance compared with previous works [2,3,4]. For me, some parts are unclear, which I hope to be updated, but overall, this paper is well-written and good to share in our community.\n\n\n[1]Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" Advances in neural information processing systems 34 (2021): 15084-15097.\n\n[2] Xu, Mengdi, et al. \"Prompting decision transformer for few-shot policy generalization.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Nam, Taewook, et al. \"Skill-based Meta-Reinforcement Learning.\" arXiv preprint arXiv:2204.11828 (2022).\n\n[4] Liu, Haokun, et al. \"Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.\" arXiv preprint arXiv:2205.05638 (2022).", "strength_and_weaknesses": "Strength\n- This paper proposes a new way for DT to adapt the unseen tasks through additional neural network layers called an adapter layer. It shows better performance than baselines and model variations.\n- Through designing, their adaptation can cover not just full trajectories but also demonstration data. Their model can adapt the unseen tasks with and without expert action.\n- Their model is compared with many variations, such as without hyper-network initialization or training the adapter with DT simultaneously.\n\nWeaknesses\n- Some parts of this paper are not clear to me.\n    - In section 3.4 equation (3), the convolutional network's input is unclear. Why did you concatenate $L_s(h^o)+L_t(h^o)+L_l(h^o)$ and $L_{\\hat{r}}(h^o)+L_t(h^o)+L_l(h^o)$? Is it possible to use $L_s(h^o)+L_t(h^o)+L_l(h^o)+L_{\\hat{r}}(h^o)$?\n    - In section 4.2 HDB-IA3, what is the meaning that the models do not utilize position-wise rescaling? \n    - Table 2 is hard to understand. Does that mean the success rate? The average success rate when achieving to get a success episode? You mentioned that the data efficiency is based on the top-2 number of online rollout episodes until a success episode. Does it mean the number in the table is the average of the two number of online rollout episodes?\n    - In section 5.2, relating to the unclarity of table 2, could you explain more about why PDT hardly improves success rates?\n    - > First, HDT achieves similar training task performance compared with the pre-trained DT model as in Figure 3 (a), which shows that the hyper-network could distinguish the task-specific information of training tasks.\n        - Is it because the multi-task DT can distinguish the task-specific information through a given state, action, and reward sequence?\n- Typo in conclusion. grinned -> grind.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OIKff_RF4SW", "reviewer": "Reviewer_k7sT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "As I understand: For learning an agent across multiple RL tasks that can quickly adapt to new tasks, the authors combine several established paradigmas: \n- decision transformers as the base archictecture for the learning agent, trained from offline RL data across tasks; \n- adding additional \"adaption\" parameters (layers);\n- these parameters are pre-trained using the Hyper net training (Alg 1) and then used as initalization;\n- when a new task arrives, with demonstrator data, then first the adaptation layer is fine-tuned (from the above initializatin) on that new task's demonstrator data (while the base params are kept). This is a trade-off to be able to be adapt, while avoiding too much variance of the overall architecture upon new tasks (and forgetting old ones).\n\nSome of these ideas come from NLP and are here tansferred to DT.\n\nIn the experiments, HDT outperforms some baselines and some questions are tried to be answered experimentally, e.g., if the hyper net learns to use the task-specific data (5.3).", "summary_of_the_review": "The multi-task problem is well-motivated, the interesting recent DT approach is combined with further ideas to address it; the novelty exists but is limited and the writing and explanations need to be improved. Hence, I do not see a fundamental issue; however, the contribution seems rather limited.", "strength_and_weaknesses": "**Strengths:**\n\nThe overall problem - multi-task learning - and the choice of the interesting recent DT architecture as base, are well motivated.\n\nThe method combines several paradigms and seems to transfer insights from NLP Transformer models, which is nice.\n\nThe experiments seem to indicate, that the method works comparably well.\n\n\n**Weaknesses and specific points for improvement:**\n\nMany paradigmas are put together (meta, offline RL, IL, …), sometimes with more, sometimes with less motivation. For instance, why combine offline RL base-training with expert-data as task-specific info? Where does this scenario happen in practice (i.e., where do we have the base offline RL data plus a demonstrator for each task)? It is not entirely counterintuitive, but I didn't see the clear motivation. Maybe the main reason is that only this setting (i.e., data from a demonstrator in the new task) fits well into the quite restrictive DT format where the data has to be in the form of s/a/r-triplets. But how about just using offline RL data for the new task as well (or is there too little information content in non-expert data?). A stronger motivation would help here.\n\nThe wiriting clarity and understanding of the method needs to be improved, e.g.,\n* The action variable $a$ in Alg 1/2 comes from nowhere. It needs to be related to the trajectories etc. Essentially, variables always have to be properly introduced (within a context/algo).\n* I couldn't follow the argument in sec 5.3. Why again does Fig 3 (a) show that the hyper net in fact learns to use task-specific knowledge?\n* Imo, DT is introduced a bit too briefly although it is at the very core and a quite new paradigm. I didn't understand how the reward-to-go is obatained but I guess this simply the retrospective sum since we are in an offline setting - OK but maybe clarify. How about this length K \"lag\" fed into the transofmer. Why not, as in classic RL, just feed the *single* current state (plus desired reward in the sense of the DT idea) and output an action? Is this to account for non-Markov, or for an agent memory, or for task info? There is an informal comment on this on p4, but it feels vauge - then again, maybe one just has to accept that these sort of approaches are less motivated by understanding and more by experiments.\n\nFurther comment on experiment: I'm a bit surprised about the very poor performance of some of the baslines in Table 2 (0 success). This can be a hint that they are not tuned properly enough.\n\nIn terms of novelty/significance, there seems to be some novelty, however, the paper is heavily based on \"just\" combining several well-established ideas.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3dnNI5CU16", "reviewer": "Reviewer_FfBQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors study the important question of efficiently adapting decision transformers when having access to a handful of demonstrations for unseen tasks. They propose to augment the base DT with an adaptation module that can be controlled by a hyper-net, which can take the demonstration as inputs. The author conduct experiments on meta world and shows that their results are better than other baselines.", "summary_of_the_review": "In general, I think it's a good paper and recommend acceptance.", "strength_and_weaknesses": "Strength:\nThe problem is well motivated, and efficiently adapting large pertained decision transformers with a handful of observation is an important line of research. Also the LfO setting is interesting since in many real life robotic applications it’s hard to obtain state action pairs as demonstrations. The paper is also easy to follow. \n\nWhat I like most is the experiment design and the details. While at first sight, the proposed method seems very complicated, the design is justified in the ablation study. E.g., Fig 3 suggests that the hypernet is better than other simpler ways of adapting transformers like prompting (PDT) and IA3. It’s also nice to see the comparison with meta RL methods like SimPL, but it’s expected that SimPL is not as good as the proposed method, since SimPL does not take into the demonstration in the unseen tasks. It’s also interesting to see that the hyper net can encode the task-specific information as in sec5.3. While the paper only has one environment Metaworld, but meta45 is a pretty challenging task alone. I think this could be interesting for researchers studying decision transformers.\n\nWeakness:\nWhile I think metaworld is enough, more environments could be nicer, like Kitchen, which seems to be more widely adopted in the previous literature.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster