Towards Controllable Policy through Goal-Masked Transformers
Authors: Xinyao Niu
Tong Sang
Yuchen Sun
Xiangjun Wang
Venue: Submitted to ICLR 2023 (2023)

Abstract: Offline goal-conditioned supervised learning (GCSL) can learn to achieve various goals from purely offline datasets without reward information, enhancing control over the policy. However, we argue that learning a composite policy switchable among different goals seamlessly should be an essential task for obtaining a controllable policy. This feature should be learnable if the dataset contains enough data about such switches. Unfortunately, most existing datasets either partially or entirely lack such switching demonstrations. Current GCSL approaches that use hindsight information concentrate primarily on reachability at the state or return level. They might not work as expected when the goal is changed within an episode. To this end, we present Goal-Masked Transformers (GMT), an efficient GCSL algorithm based on transformers with goal masking. GMT makes use of trajectory-level hindsight information, which is automatically gathered and can be adjusted for various statistics of interest. Due to the autoregressive nature of GMT, we can change the goal and control the policy at any time. We empirically evaluate GMT on MuJoCo continuous control benchmarks and Atari discrete control games with image states to compare GMT against baselines. We illustrate that GMT can infer the missing switching processes from the given dataset and thus switch smoothly among different goals. As a result, GMT demonstrates its ability to control policy and succeeds on all the tasks with low variance, while existing GCSL works can hardly succeed in goal-switching.

Under review as a conference paper at ICLR 2023
TOWARDS CONTROLLABLE POLICY THROUGH GOAL- MASKED TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
Offline goal-conditioned supervised learning (GCSL) can learn to achieve various goals from purely offline datasets without reward information, enhancing control over the policy. However, we argue that learning composite policy switchable among different goals seamlessly should be an essential task for obtaining a con- trollable policy. This feature should be learnable if the dataset contains enough data about such switches. Unfortunately, most existing datasets either partially entirely lack such switching demonstrations. Current GCSL approaches that use hindsight information concentrate primarily on reachability at the state or re- turn level. They might not work as expected when the goal is changed within an episode. To this end, we present Goal-Masked Transformers (GMT), an efficient GCSL algorithm based on transformers with goal masking. GMT makes use of trajectory-level hindsight information, which is automatically gathered and can be adjusted for various statisties of interest. Due the autoregressive nature of GMT, we can change the goal and control the policy at any time. We empirically eval- uate GMT on MuJoCo continuous control benchmarks and Atari discrete control games with image states to compare GMT against baselines. We illustrate that GMT can infer the missing switching processes from the given dataset and thus switch smoothly among different goals. As result, GMT demonstrates its ability to control policy and succeeds on all the tasks with low variance, while existing GCSL works can hardly succeed in goal-switchin!!
INTRODUCTION
Runners can control and adjust their pace in a marathon by switching comfortably between various poses for different goals. Similarly, agents can also acquire such switching ability through rein- forcement learning (RL) or imitation learning (IL). This process generally requires environments that can start with arbitrary pose states, carefully tuned rewards, massive offline demonstrations. However, these critical things are notoriously challenging to obtain. In comparison, by knowing the pace of each running stance, human can easily switch between various poses to control speed without learning such switching processes intentionally.
From another perspective, we try to formulate this problem (GCSL) (Ghosh et al. 2019) problem given a fixed amount of offline dataset: Considering pose or pace goal, can agents learn composite policy that can switch among these goals interchange- ably over the dataset? We refer to this problem as the goal-switching problem. Since the distribution of initial states shifts between the training and evaluation, this problem might face the covariate shift issue. Instead of fixed set of states in the training set, any state might be the start of the switched goal during evaluation, resulting in agents not knowing how to achieve the goal. The goal-switching has widespread adoption in practical applications. The control of robots to transfer to a different skill while performing another skill is essential in robotics. In the game field, it can induce immersive experiences by managing the performance and strategies of Al bots according to the game progress.
coal-conditioned supervised learning
Recent works (Ghosh et al. 2019 Ding et al. 2019: Furuta et al. 2021 Eysenbach et al., 2022 Reed et al. 2022) on GCSL mainly focus on learning how to achieve arbitrary goals. They tend to use either state (Emmons et al. 2021) or return-to-go (Chen et al. 202 1 as goals. Models that use states
'The code will be available as soon possihle.

---

Under review as a conference paper at ICLR 2023
Action Loss
Goal Loss
Causal Transformer
Pos Emb
Token Emb
9--1
Gp-1
9:
[mask]
Figure 1: Overview of GMT architecture.
as goals can be called state-conditioned models. These models generate actions based targeting future states or slices successful demonstration. There are also models conditioning on return-to- go the total rewards an agent can receive from the current step until the end of an episode. These methods typically use the "relabeling" strategy to excavate a variety of goal signals by bootstrapping any of the aforementioned goals from fixed dataset. However, each method from above has some fatal flaws that make them result in poor performance on the goal-switching problem. By setting return-to-go goals, switching can only happen by tweaking normalized return-oogoo, which is neither intuitive nor efficient. State-conditioned methods can reach an arbitrary state in theory. However, they tend to model the problem as a pure MDP, which leads to poor performance or failure when demonstrations of transiting between two states are missing from the dataset. We argue that the essence of the goal-switching problem is to enable the model to learn unseen transitions. Model- based approaches may be one remedy, which allows for unseen transitions by planning across latent space (Jiang al. 2022) or an explicitly learned world model (Micheli et al. 2022). These methods require additional environmental model learning with higher learning complexity.
This paper presents Goal-Masked Transformers (GMT), an efficient GCSL, algorithm that neither necessitates explicit world model learning nor successful demonstrations to solve the goal-switching problem. In particular, we employ a causal transformer (Radford et al. 2019) to autoregressively describe trajectories with transitions consisting of goals, states, and actions. With such setting, the goal can be changed at any moment, releasing the full potential of the policy control. However, since the goals are the same in each transition during training, the model tends neglect the change of goals, leading to a goal-switching failure during evaluation. In order to compel the model to put more effort into goals, information with a [mask] token. As result, we observe the promising outcome that agents can smoothly switch from one goal another. Figure presents an overview of GMT.
introduce masking mechanism with a probability of replacing the goal
Similar to previous GCSL works, we apply the "relabeling" strategy to increase the diversity and coverage of goals. Additionally, there are various expressions of goals and numerous approaches to achieving them. Thus, we propose a simple yet effective approach that automatically aggregates and clusters the offline data into several goals accordance with the statistics of interests.
In summary, our main contributions are follows:
We draw attention to the goal-switching problem and argue that doing so is an essential step toward controllable policy. The goal-switching problem requires that agents should adapt to goal changes within one episode. is an exceedingly challenging generalization problem, especially when training on limited datasets.
We propose Goal-Masked Transformers (GMT), family of goal-conditioned algorithms based on causal transformers with a goal-masking mechanism and hindsight information to achieve controllable policy. Through experiments, we demonstrate that GMT possesses soil-swit capabilities that are not present in the current GCSL algorithms.
We introduce unsupervised approach to cluster trajectories into multiple goals from the datasets without any goal information. Empirically, we find that this approach improves the stability and efficiency of the switching process.
2

---

Under review as a conference paper at ICLR 2023
2
METHOD
The purpose of GoaleMasked Transformers (GMT) is to train a controllable policy that can comfort- ably switch among goals over the demonstration {(s), ag. ... >)=-o where s of indicate state and action at timestep t in the ith of N trajectories respectively. To enable goal-switching, we design trajectories T; with transitions made up of goal, state, and action, i.e.,
(go= so: %b gi s), ai.-.., ST. ST, -1, -1.
where gf denotes goal at timestep t the i-th trajectory and T; indicates the length of the trajectory T5 We define the goal as statistics of interests, for example, reward, speed, number of blocks that break in a game, the dynamics of robots and strength of game bots. These statistics are generally not provided in the demonstration but are implicitly included in the data. The statistics can be linked to each trajectory by some kind of mapping function. Namely, we can mathematically formulate the goal as follows.
g = J dijs .1, ii, -1
where e indicates the statistics of an interest and g indicates the goal. Given the statistics of a specific interest e, the function f can map the trajectory into the corresponding goal.
2.1 TRAINING DATASET
In order to demonstrate the ability of GMT in different areas, we consider two applications to eval- uate GMT. We train our model on both tasks by utilizing offline trajectory data collected from RL agents, either are trajectories rollouts during training or collected by using training checkpoints with a certain level of skills to inference.
D4RL We conduct MuJoCo (Tooorvv et al. 2012) locomotion environments with continuous control from the D4RL (Fu et al. (2200) benchmark, including HalfCheetah, Hopper, Walker, and Ant. The locomotion task is the fundamental task in the field of robotics and we can show the con- trol of the movement speed (along the x-axis) by switching target poses. We combine the Medium and Medium-Fxper datasets provided in the D4RL benchmark as the training dataset for each lo- comotion environment except for HalfCheetah. On the HalfCheetah task, we also add the Random dataset where trajectories are totally generated by a random policy.
Atari We demonstrate that GMT can also be applied to games, such as video games. The RLUn- plugged (Gulcchre et al.) 2020 dataset provides trajectories of numerous Atari games (Bellemare) ct al., 2013) collected from the training progress of a DQN (Mnih et al., 2013) agent. The dataset covers various demonstrations, including agent behavior all learning stages. We train GMT over the Breakout dataset to show that GMT can switch between different levels of policies according the changes in goal. The method should be able apply to any game in theory, but due to the computational limitation, we only randomly select one game to show the idea.
The trajectories in both the public datasets contain states, actions, and rewards. However, we focus on goals instead of directly utilizing the reward information. Commonly in practice, the reward information hard obtain in sparse rewards settings or requires sophisticated designs of reward functions. Therefore, we leave reward information and use the "relabeling" strategy to extract goal information from the trajectories based on any statisties of interest to construct new trajectories with transitions consisting of goals, states, and actions.
straightforward strategy to relabel the trajectory with goals is manually group statistics of in- terest by hand. However, we observe that too few classes may lead to a high variance policy as extremely dissimilar demonstrations are grouped. Whereas having too many classes can hinder per- formance because the model struggles to distinguish similar trajectories. There might eventually be relatively few demonstrations that are rarely learnable. Thus, we require a novel relabeling approach that automate this process in efficient and effective way.
Here we propose an unsupervised manner to aggregate and label trajectories for the aforementioned offline datasets. We first perform k-means clustering (Lloyd 1982) along the statisties of interest (c.g. we are interested in the speed of locomotion on the x-axIs in MuJoCo) with a range of k. and then caleulate the silhouette score (Rousseeuw 1987) of each cluster. As there are now goodness
3

---

Under review as a conference paper at ICLR 2023
measurements of clustering with different k we sample le based on the normalized silhouette score from the top 85th percentile.
We uniformly sample 2000 trajectories to train models for each Atari game. For MuJoCo tasks, the specific numbers of trajectories in the dataset are listed in Appendix A.2.I Morcover, we do not require any extra dataset balancing filtering since the major purpose of this research is to show
that our model can switch between multiple goals rather than the magnitude of the rewards that agents will ultimately receive.
2.2 ARCHITECTURE
GMT accepts input in the form of a transition consisting of goal, state, and action token. The
tokenization scheme used for GMT is described in Appendix A.1.1 To generate the final model input, we apply three parameterized embedding functions f(; (7 f. f(: (g) The embedding functions carry out various operations depending the modality of the token to enable efficient learning from the multi-modal input sequence. A learned vector embedding space is used to store tokens associated with discrete- or continuous-valued observations or actions for any timestep. A single Convolutional Neural Network (CNN) is used to embed tokens from images at any timestep. Learnable position encodings are added for all tokens based on their global token position within their corresponding trajectories.
Under these circumstances, try to formulate this problem sequence modeling problem. Due to the nature of the given sequence, we use an ordinary causal transformer or decoder-only trans- former Vaswwani et al. 2017) as our main architecture. We feed 3K tokens worth of the most recent K timesteps (goal, state, action for each timestep) into GMT, where K represents the context length of GMT. In general, longer context length allows the model trace back longer in history but may require a more competent model and can be harder to train. A LayerNorm (Ba et al. 2016) is added before feeding the tokens to GMT.
Loss Given goals Jgt, states sgt and actions aca, GMT predicts the action a, å, and calculate the difference between å and a,. For discrete (goals and actions for Atari games) and continuous values (actions for MuJoCo tasks), GMT respectively uses cross entropy and mean square error (MSE) as the loss. When only the supervisory signal the action is there, the relationship between goals, states, and actions is frequently neglected. This situation is particularly evident in the case of goals because goals share the same value along the trajectory. In order to improve the situation, we add auxiliary goal loss force the model to focus goal information. Therefore, the total loss C is a weighted sum of the action loss La and goal loss Eg. namely, L = C. + aL, where is the scalar weight to balance two terms. We set 1 our all experiments. worth noting that there is no loss for the first goal as it is always given to the model.
2.3 GOAL MASKING
Surprisingly, the naive goal prediction task hardly contributes to policy learning. The model only requires the goal information to predict the goal label at the carly stage of the trajectory. When reaching several goals, the model can actually predict the goal label from the states and actions as the state information may differ significantly across different goals. The model tends to ignore the goal label in the later phase of the training data and only learns the relationship between states and actions. The goal prediction task, therefore, becomes a goal memorization task.
Thus, instead of always using the actual label when predicting the goal, we have probability of p to replace the true goal with a trainable token, [mask]. We expect the model to learn and enforce the relationship between goals, states, and actions instead of only memorizing the previous goals. We propose token masking that replaces each goal token with [mask] token with a probability of p. but only performs within a context ? All the masking operations are dynamic, meaning the mask is rampldd for each training iteration. We conduct experiments to compare different p (See
We also propose trujectory masking that replaces goal tokens in single trajectory with a probability of p. However, we find that controllable policy. We analyze the reasons Section 2.3
this method still struggles with the goal-switching problem and cannot obtain
4

---

Under review as a conference paper at ICLR 2023
Table I: Success rate of GMT and other baselines different MuloCo task
HalfCheetah
Hepper
Walker
2
2
D
0
1-1
1-+
0
1-+3
masking)
15/15
15/15
15/15
15/15
14/15
14/15
13/15
GMTigraj. maskinng)
3/15
0/115
0/15
15/15
1/15
14/15
3/15
14/15
2/15
GM masking)
0/15
0/15
0/15
15/15
0/15
14/15
0/15
14/15
0/15
15/15
Prompi DI Xirea
6/15
2/15
4/15
1/15
3/15
1/15
0/15
2/15
3/15
Decision Transforect
2021
5/15
14/15
3/15
9/15
5/16
8/15
4/16
1/15
1/15
Section 3.2.1 and eventually find that token masking with = 0.5 works the best. We set it as the default for our method.
EXPERIMENTS
We use a unified architecture for both Atari games and MuJoCo tasks (See Appendix A.1.2 for the implementation details and A.1.3 for deployments). In this section, we mainly demonstrate the results of MuJoCo tasks and analyze how GMT works through a series of ablation studies. We leave the analysis of Atari games in the Appendix (see (see)..2.2) A.2.2)
3.1 RESULTS
In this section, we use both step-based graphs and task success rates to evaluate the ability of GMT. Notice that higher variance may not necessarily indicate destructive behaviors in step-based illus- trations. We focus on trajectory-level alignments, meaning different trajectories with the same goal can eventually reach similar final results through different paths. Thus, all the following step-based plots are specifically used to demonstrate consistent differences between various goals. The success rate is summarized as tables for each task.
We also summarize the success rate of each MuJoCo task for different models in | We define each task success if the model successfully achieve the first goal and consequently switch to new goal, where the difference in statistics interests is within 10%.
3.1.1 McJoCo
Srantep
(a) HalfCheetah
(b) Hopper
(c) Walker
(d) Ant
Dine dng
Smrolap
smalaspe
Limeulago
(e) HalfCheetah (500)
(l) Hopper (250)
Walker (350)
(h) Ant (400)
Figure 2: GMT with non-switching (a-d) and switching (e-h) tasks. Numbers in brackets in the switching task represent the timestep that goal switching happened. The number of lines represents the number of goals for each task, which is obtained by our automatic clustering method. We also experiment with the effect generating more goals, see A.2.7 for detailed results.

---

Under review as a conference paper at ICLR 2023
We use four MuJoCo environments (HalfCheetah,, Hopper, Walker, and Ant) major testbed to evaluate GMT. In environments, our statistic of interest is the speed on the x-axis (which is also the reward of the environments each step the default setting).
All figures in this section (unless otherwise mentioned) use concrete lines to represent the mean and the same color with higher transparency around lines to indicate the standard deviation of the statistics. All the mean and variance are obtained by running over the model 15 times with 15 fixed random seeds. We show that GMT is capable of learning imitate different trajectories with sufficient data in Table We also demonstrate step rewards received across different goal labels and tasks in Figure
As depicted in Figure we use t-SNE (Hinton & Roweis 2002) to visualize the distribution of states for trajectories with differennt goals. We only demonstrate the results of HalfCheetah here, full results including other environments can be interpreted similarly and can be found in Appendix A.2.4 From the figure, some states' overlap can be observed as many of these states may start from similar points cross similar middle states when reaching desired goals. To mathematically model the problem, the naive goal prediction task becomes to model the probability of Po(g: [8x; Ons 9x), where we use the footnote K to represent indices in the context of our model. However, when states are distinct enough, which is very common in the later simulation of the MuJoCo task, the goal prediction task tends to degenerate to (g (xx also explains why trajectory masking does not work well.
ax) as previous goals no longer necessary. This
0 1
timesteps
800
700
400
300
(a) Each goal without switching
(b) Goal switching from to
Figure 3: Left: The graph represents the state distribution of HalfCheetah for the non-switching task. Legends the top left corner indicate the goal label. We can observe some overlaps between states on the distributions, which reasonable as there are overlaps between states the environments when reaching different goals from similar set of starting points. Right: We visualize the trace of the goal-switching task in terms of the state distribution. The brighter the color, the later the timestep. demonstrates how states initially distributed around goal 0 and gradually diffuse the region goal 1.
3.1.2 ALTERNATIVE ARCHITECTURES
Here we would like to discuss other model architectures and auxiliary tasks for achieving goal switching without demonstration. We initially try to use RvS(states as goals) (Emmons et al.) (02T)) to achieve arbitrary goal reachability. Referring to Emmons et al. (202I), we use two layer multi- layer perceptron (MLP) with hidden size of 1024 for this experiment. We also try to use causal transformer as the backbone, which is the same architecture as ours but replaces the goal label. In this case, the state goal is tokenized following the tokenization scheme for other continuous values. However, we not find our architecture to provide an apparent performance boost but only less sensitive to the hyperparameters. Thus, only report the original results that

---

Under review as a conference paper at ICLR 2023
MLP. During training, sample an arbitrary state from the training dataset timestep T t, and sample another state that is in the same trajectory at timestep T t. We keep doing this process to train the model until convergence. To aid better strategy learning, tasks. Inspired by [Schwarzer et al. (2021). we find that predicting the distance between the state and goal and trajectory labeling prediction works the best. However, by carefully tuning the model, we find that the state-conditioned model can hardly distinguish similar states. It can have decent performance in some tasks but poorly react to the switch task overall. As illustrated in Figure 4 the model can learn different trajectories with a slightly higher variance compared to our model. It also demonstrates how the model performs in goal switching setup. As illustrated, many ooal-switching are unsuccessful, showing that this architecture is not capable of this task.
also add some auxiliary
(a) HalfCheetah
(b) Hopper
(c) Walker
(d) Ant
Smartep
Sivectppe
Tirettape
(e) HalfCheetah
(I) Hopper
(g)
Walker
(h) Ant
Figure 4: States as goals (a-d) non-switching task (e-h) switching task
We also consider other transformer-based architectures (Prompt-DT) Xu et al. 2022) and De- cision Transformer Chen et al. (2021). However, Prompt-DT requires sampling few states from the 'successful' demonstration as prompts guide the model, which impossible as the successful demonstration of goal switching is absent from the dataset. Similarly, Decision Transformer requires tweaking directly on the normalized reward, which also difficult. Hence, few adjustments are made to their original architectures to suit our task. When composing prompts for Prompt-DT, replace the old prompt that is toward the old goal and sample states from the new goal label once switched. For the Decision Transformer, the switch point but subtract the already received return along the way. As illustrated, prompt-DT suffers from all tasks. Whereas Decision Transformer has some good performance when switching to better goals, it does not show competitive performance against GMT. Eventually, we find that the demonstration of the prompt is changed in the middle of the evaluation for the purpose of goal switching, the model starts to confuse and failed in most of the tests.
replace the return-to-goal with the new target
3.2 ABLATION
3.2. MASKING RATIO
niva dus
Mww ww
420
300
123
222
Binectaos
Smadteps
(a) masking ratio 0.3
(b) masking ratio 0.7
Figure 5: Ablation study of masking ratio in HalfCheetah
7

---

Under review as a conference paper at ICLR 2023
Goal masking plays pivotal role in GMT. In this section, investigate whether the result is sitive to the masking ratio. We observe that the masking ratio has no effect on non- soitswng tasks, which is reasonable. By both increasing decreasing the masking ratio (as shown in Figure [5), can observe that various trajectories are no longer switchable to other trajectories or significantly increase in variance. For example, switching from goal label 2 to I actually fails when the masking ratio reduces to 0.3. Also, the models cannot switch to trajectories that are labeled with 0 when the masking ratio increase 0.7. Other tasks demonstrate very similar trends, and show their results in Appendix A.2.5
3.2.2 CONTEXT LENGTH
Srrslap
tmesaap
(a) context length 50
(b) context length 150
Figure 6: Ablation study of context length in HalfCheetah
Previous studies (Chen et al. 2021 Xu al. 2022) found that transformer-based RI. algorithms are sensitive to context length. Thus, we demonstrate how context length affects the performance of GMT. As result, find that context length only has limited effect non-switching tasks. However, too short a context length can cause switching failure, and too long context length can significantly increase the switch delay and cause higher variance. Other tasks demonstrate remarkably similar trends, and we show their results in Appendix A.2.6
3.2.3 SMOOTHNESS OF TRANSITIONS
(a) Switch between models or switch with short context
(b) Switch through GMT
Figure 7: Two sequences of frames that renders how HalfCheetah's pose change under different model setup. Every frame on each figure sampled from every three consecutive frames of environments. We use two red rectangles to highlight four keyframes where the change happened. As illustrated, goal (pose) switching in (a) is not smooth, showing some strange poses that can hardly be observed at any other time. Whereas, the pose in (b) does not obviously seem to be affected by the goal switching, demonstrating smoother transition and better control.
One intuitive idea is whether we could complete goal-switchin tasks by switching between different models. Specifically, we train multiple models for different goals, and then simply switch between different models when goal switching is initiated. We find that the model can switch between dif- ferent goals but in a different way. figuurrjillussrates that the models actually restart the process by acting arbitrarily until the model collapses to the state that the model has seen in the training dataset (likely the starting state), which indicates that the model is actually doing things like resetting the environment and then attempting to reach the target state by starting from scratch. Similar trajecto- ries can be also found GMT with very short context length (i.e. context length K 1). However, GMT performs much smoother transitions between distinct goals, indicating that GMT has barely
8

---

Under review as a conference paper at ICLR 2023
been affected by the distribution shifting. We believe that either the distribution shifting a lack of historical context is to blame.
4 RELATED WORK
Goal-conditioned tasks are introduced to reach the desired goal and improve policy control. There has been a lot of research (Andrychowicz et al.) 2017 Plappert et af.] 2018 Chane-Sane et al,) 2021 Chen et al. [021]] Furuta et al. [021] Ma et al., (022) on how goal-conditioned approaches can be more effectively generalized to achieve multiple goals or even unseen goals. This work argues that goal-switching can enable better generalizability, especially in the supervised learning setting. Through our method, models can handle unseen transitions, where both goal and state may be observed, but the combination is never seen. Goal-switchng can be learned in theory via goal-conditioned reinforcement learning (GCRL) (Liu et al. 2022) by carefully tuning rewards, or goal-conditioned supervised learning (GCSL) (Ding al. 2019) by feeding sufficient demonstra- tions. However, in practice, these methods are resource intensive. On the other hand, our proposed approach can infer the switching process between arbitrary goals from an imperfect dataset.
Goal-conditioned tasks can also be formulated as sequence modeling problem (Chen et al. 2021 Xu al. 2022 Reed al. 2022 Furuta al. (02T). Such sequence models becoming popular in the RL community recently. New research is being done in variety of areas. including multi-
agents (Wen et al.) (222), planning (Jiang et al., 2022), online learning (Zheng et al. [2022). and offline learning (Chen et al., (021). We follow recent work (Chen et al. 2021: Furuta et al. 2021) that models the full trajectory of RI. using GPT-2-stylle Transformers, treating as sequence the application circumstance, there various approaches to
modeling problem. Depending modeling trajectories. Our formulation enables sequence modeling to achieve a controllable policy by controlling each step's goal without any reward information. The Categorical Decision Trans- former (CDT) proposed in Furuta et al. (202T) is the most similar to our architecture. However, CDT only reformulates the goal learning from regression problem (as in Decision Transformer) to a classification problem, which is substantially different than what we are trying to achieve.
CONCLUSION
We present a crucial problem towards controllable policy: the goal-switching problem. This problem is challenging for training on fixed datasets, which typically do not cover all the switching processes encountered during deployment. We argue that the existing GCSL methods are not designed to solve such an out-of outoof-distribution (OOD) problem. This paper proposes goal-maske simple but effective GCSL method based on causal transformers. More specifically, introduce masking mechanism that forces the model to learn generatively about the connection between goal, propose unsupervised manner to relabel goals from the dataset
transformers (GMT),
state, and action. Additionally, based on the statics of interest, which would reduce the volatility while switching between goals. We empirically show that GMT can achieve goal-switching with different modal inputs and analyze the impact of various factors on performance.
GMT is goal-conditioned supervised learning algorithm, which still suffers from the problems of supervised learning, such as heavily relying on the dataset's 's quality. Switching between goals can be done smoothly the dataset quality is good, but the switching process may have a higher variance if the dataset quality is terrible. Meanwhile, high-quality dataset allows GMT to learn better and more complex policies. GMT is not limited to the GCSL paradigm and can be easily extended to offline RL or GCRL in conjunction with existing works.
Furthermore, GMT does not learn the goal-switching controller that determines when to change the goal. We emphasize that the problem GMT is trying solve is the ability to learn goal-switching from imperfect datasets. This dramatically enhances the controllability of the policy and enriches human- comante interaction (HCI). At the same time, GMT makes it straightforward to text tokens as goal tokens, significantly improving the potential of HCI by fusing them with natural language. We leave these as future works.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Picter Abbeel, and Wojeiech Zaremba. Hindsight experience re- play. Advances in neural information processing systems, 30, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 177.:060770044000 2016.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Elliot Chane-Sane, Cordelia Schmid, and Ivan Laptev. Goal-conditioeed reinforcement learning with imagined subgoals. In International Conference on Machine Learning, pp. 1430-1440.
PMLR, 2021.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Picter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5084-15097, 2021.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phiclipp. Goal-conditioned imitation learning. Advances in neural information processing systems, 32, 2019.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline supervised learning? arXiv preprint arXiv:2112. 10751, 2021.
Benjamin Eysenbach, Soumith Udatha, Sergey Levine, and Ruslan Salakhutdinov. Imitating past successes can be very suboptimal. arXiv preprint 2xXiv:2206.03378,, 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Hiroki Furuta, Yutaka Matsuo, and Shixiang Shane Gu. Generalized decision transformer for offline hindsight information matching. arXiv preprint arXiv:2111.10366 2021.
Dibya Ghosh, Abhishek Gupta, Ashwin Reddy, Justin Fu, Coline Devin, Benjamin Eysenbach, and Sergey Levine. Learning to reach goals via iterated supervised learning. arXiv preprint 1XXiv:9912.00888 2019.
Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zoina, Rishabh Agarwal, Josh Merel, Daniel Mankowitz, Cosmin Paduraru, et al. RI unplugged: A suite of benchmarks for offline reinforcement learning. Advances in Neural Information Process- ing Systems, 33:7248-7259, 2020.
Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. Advances in neural informa- tion processing systems, 15, 2002.
Zhengyao Jiang, Tianjun Zhang, Michael Janner, Yueying Li, Tim Rocktäschel, Edward Grefen- stette, and Yuandong Tian. Efficient planning in a compact latent action space. arXiv preprint arXiv:2208. 10291, 2022.
Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transform- ers. arXiv preprint arXiv:2205.15 15241, 2022.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.08// 2022.
Stuart Lloyd. Least squares quantization pem. IEEE transactions on information theory, 28(2): 129-137, 1982.
Yecheng Jason Ma, Jason Yan, Dinesh Jayaraman, and Osbert Bastani. How far i'll go: Offline goal- conditioned reinforcement learning via --avvantege regression. arXiv preprint arXiv:2206.03000 2022.
10

---

Under review as a conference paper at ICLR 2023
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers sample efficient world models. arXiv preprint Stti::2209.005888 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:13/2.5002. 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et el. Human-level control through deep reinforcement learning. nature, 51 518(7540):529-333 2015.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforce- ment learning: Challenging roboties environments and request for research. arXiv preprint arXiv: 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06175 2022.
Peter Rousseeuw, Silhouettes: a graphical aid to the interpretation and validation of cluster analy- sis. Journal of computational and applied mathematics, 20:53-65, 1987.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Advances Neural Information Processing Systems, 1686 2021.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5023 IEEE, 2012. doi: 10.1109/RROS 2.116.666619
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is sequence modeling problem. arXiv preprint arXiv:2205.14990 14953,, 2022.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In International Con- ference on Machine Learning, pp. 24631-24645. PMLR, 2022.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint NXXv:2202.00607,, 2022.

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 MODEL
A.1.1 TOKENIZATION
The trajectories may include data from multiple modalities. We transform data into flat se- quence of tokens in order to enable the processing of this multi-modal data. For GMT, we apply the following tokenization scheme to produce better results.
Continuous values (e.g. joint positions locomotion tasks) are normalized to a standard normal distribution. The mean and variance that are used for normalization are calculated separately on each dataset.
Discrete values (e.g. Atari button presses) used as integer tokens directly without further processing.
Images (e.g. Atari raw observations) are normalized between [0, 1] for each pixel (e.g.. divided by 255 for Atari games). Inspired by Multi-Game Decision Transformers (MGDT) (Lee et al., 2022), we experiment with splitting images into small patches, but find drastically reduces the performance due to the much shorter receptive fields of the model and therefore we abandon the change.
Data is transformed into tokens, which are sorted in the order of goal, state, and action. Each agent episode is comprised of multiple agent timesteps in time order.
A. 1.2
IMPLEMENTATION DETAILS
Table 2: Hyper-parameters
Env.
Hidden size
Blocks
Heads
Context length
MuloCo 128
2
100
Atari
768
12
50
We a unified architecture for both Atari games and MuJoCo tasks, the hyper-parameters listed in Table 2 Blocks represent the number of multi-head self-attention (Vaswani et al.) (017) blocks we have for the model. During training. we only calculate the action loss and goal loss. Regardless of the goal embedding provided to the model (cither true token or [mask] token), we calculate the loss towards the true label described in Section 2.2 Training takes approximately an hour on a single NVIDIA A100 GPU for each MuJoCo task and approximately two days for each Atari game on eight NVIDIA A100 GPUs.
A.1.3 DEPLOYMENT
During deployment, we first select goal label described in the previous section. The first obser- vation is then produced by the environment and tokenized before being added to the sequence. GMT autoregressively samples the action vector one token a time. This action is sent the environment which steps and yields a new observation. The above procedure repeats. Each transition is made up of a goal, state, and action. context window of K tokens. The goal can be changed at any time during deployment, simply by feeding the model with the desired goal token.
past observations and actions are constantly visible GMT in its
Motivated by MGDT, we introduce inference-time probability- method, called expert action inference, to sample actions in Atari games. Rather than always selecting the action with the highest logits (by using argmax), sample the action with the top 9 quantile. Then, based on the normalized logits of selected actions. In practice, we find = 0.6 to work the best. As otherwise mentioned, this action sampling strategy by default in Atari games.
sample actions
12

---

Under review as a conference paper at ICLR 2023
It is important note that all the methods introduced in this section only applied in the inference procedure of GMT. During training, the goals in each trajectory remain the same and cannot be altered while training. Expert action inference is likewise only used during evaluation for Atari.
A.2 EXPERIMENTS
A.2.1 MuJoCo
Table 3: Mujoco dataset statistics v.s. evaluation performance without any switching for each dataset label. gl stands for goal label.
HallCheetah
Hopper
Walker
Ant
gl
gl 0
0
gi
Dataset
10670
230
124
1.2
5141
Evaluation
0.31 1:0
4940 +:
10747
119
1626
285
S3H = 678
2861 +:
4911 1:30
4067
363
5203 ::
Traj. Num
500.00
363.00
300.00
300.00
300,00
158.00
300.00
195.0
300.00
iillsstrtee that our model can learn the representation of trajectories with different labels. However, there are two exceptions which are gl for HalfCheetah and gl for Ant. We realize that there is noticeable gap between the mean value for these tasks. By further investigating, we find that this may be caused by differences in reward calculations between the environment that perform data collection and evaluations. We decide to live with this difference and use this mean as the new standard to determine whether or not the switch task successful.
Table 3
A.2.2 ATARI
We select one Atari game (Breakout) to demonstrate that GMT can also be applied to video games with image inputs and complex policies. For each Atari game, the chosen statistics of interest is the reward. We perform various data augmentations for Atari games during training, including random crop and random rotation.
We cluster the Breakout into 11 goals. Figure 10 shows the trajectories sampled from each goal label in the Breakout game. We can observe several patterns across different trajectory labels. For example, goals and 10 tend to focus on breaking the middle part, however, and tend to break evenly without too much preference. Additionally, the agent tends to break more blocks on the top as the value of the goal label increase. This is a reasonable behavior as breaking those blocks on the top tends give higher rewards. It is possible to reduce this variance by clustering trajectory using the normalized score which only considers the sign of the score as in previous work (Mnih et al., 2015). However, this change is not adopted because the normalized reward in this scenario depends on how many blocks the agent breaks, which is not the statistic of interest for this experiment.
As there are too many combinations for switching, we illustrate few here (specifically between 2, 5, and 10) to evaluate the ability of GMT. As demonstrated in Figure g we expect the total reward of the trajectory sits around 23 0.3 + 74 0.7 57.5 when switching from 2 to 10 at timestep 300, and sits around 74 x 0.3 + 23 x 0.7 38.3 when starting with 10 and end with 2 under the same procedure. We use 0.3 and 0.7 as each part generally takes 30% and 70% of the entire trajectories, respectively. By using a similar method, we can also calculate the expected return when switching between 2 and 5. They are 35.1 and 25.9, for goal - 5 and 5 - 2, respectively. We observe that switches are successful across all the illustrated examples, and the final episodic reward is close to our expectations. We also found that the expectation calculation tends to overestimate the total reward that agent can receive when switching from better performance goal to the worse one (e.g. 10 - as we demonstrated). Unlike the MuloCo task, this is a reasonable phenomenon in Breakout the block closer the top has high marks, which Thus, switching to a better strategy in the later game tends to receive more rewards. This can also be observed from the left line plot, in which the gradient of the lines is much high in later games.
only be hit the later game.
13

---

Under review as a conference paper at ICLR 2023
before saitch
after switch
2-10
50
2.5
reward
reward
10
1200
2-5
5-30
timesteps
goal label
Figure 8: Left: The rewards trends within an episode that the agent gets for different goal labels with without switching. We dash lines to represent cases without switching references. All switches are happened at timestep 300 by gradually replacing the old goal with the new goal to provide agents with some context before and after the switch. For better illustration, all lines smoothed by using an exponential running average with smoothed parameter 3 = 0.97. Right: We try to emphasize the total reward before and after the reward changed in this figure. Bars with multiple colors represent the total reward received in different stages, whereas bars with only one color (blue) mean that goals consistent across the episode without changes.
(a) Switch every 100 steps
(b) Switch every 200 steps
(c) Switch every 333 steps
Figure 9: GMT switch more than once within an episode
A.2.3 MORE SWITCH WITHIN ONE EPISODE
clear that our modal is capable of switching more than once within an episode as shown in 9 When GMT switches every 100 steps, 15/15 complete the first switch. However, only 3/15 still on the track at the end (success all 9 switches). When GMT switches every 200 steps, 13/15 can finally made it (success all 4 switches). When GMT switches every 300 steps, all attempts succeed (success all 4 switches). Apparently, the more frequent the switch, rate the switch happens before the finish of another switch. This is because all middle transitions are 'guessed' by our model. Keep staying and operating in such state space will eventually lead to unrecoverable error. However, model still have decent performance when the interval is reasonable.
higher variance and failure
14

---

Under review as a conference paper at ICLR 2023
RAB
ROM
MON
RON
RO4
BAR
ROS
RET
RAN
Figure 10: Breakout trajectory samples. From top to down are goals labeled from 0 to 10.
15

---

Under review as a conference paper at ICLR 2023
A.2.4 STATE DISTRIBUTIONS FOR MORE ENVIRONMENTS
timesteps
timesteos
30
900
900
20
10
700
700
500
10
600
400
400
-10
300
-10
700
100
220
100
-30
-20
20
-30
(a) Goal switching from 0 to
(b) Goal switching from to
Figure 11: State distribution in HalfCheetah
low
high
900
50
700
500
400
-30
200
100
-100 -100 -80 -50
(a) Each goal without switching
(b) Goal switching from to
Figure 12: State distribution in Walker
16

---

Under review as a conference paper at ICLR 2023
100
0
100
tmesteps
50
300
200
-00
200
-100
-100
(a) Each goal without switching
(b) Goal switching from to
Figure 13: State distribution Hopper
0 1
60
900
700
600
500
-70
400
400
200
-60
100
-80
-80
(a) Each goal without switching
(b) Goal switching from to
Figure 14: State distribution in Ant
17

---

Under review as a conference paper at ICLR 2023
A.2.5
MASKING RATIO FOR MORE ENVIRONMENTS
Diewar day
600
620
202
200
400
400
200
6.0
timesteps
timestops
timestaps
(a) Ant
(b) Hopper
(c) Walker
Figure 15: Ablation study masking ratio with 0.3 on three different tasks with switching
Aed
stop
620 200
200
600
200
600
timesteps
timesteps
timesteps
(a) Ant
(b) Hopper
Walker
Figure 16: Ablation study of masking ratio with 0.7 on three different tasks with switching
A.2.6 CONTEXT LENGTH FOR MORE ENVIRONMENTS
Wiredops
Biresteps
Tinectess
context length
(b) context length 150
Figure 17: Ablation study of context length Hopper
18

---

Under review as a conference paper at ICLR 2023
010 readd
Ornoleps
Unveslere
context length 50
(b) context length 150
Figure 18: Ablation study of context length in Ant
(a) context length 50
(b) context length 150
Figure 19: Ablation study of context length in Walker
A.2.7
FINER-GRAINED GOAL CLUSTERING
12
step reward
step reward
8
wwww
200
400
600
800
0
200
400
600
800
timesteps
timestep
Figure 20: Walker with goal labels
Figure HalfCheetah with 6 goal labels
Both Figure 20 and 21 shows that too many class labels will eventually result in goals that are unable to distinguish between each other. Many of the trajectories overlap with each other in both enviropments.
19


Reviews:
[{"review_id": "qKEZtYHADN", "reviewer": "Reviewer_S5ck", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the problem of goal conditioned reinforcement learning, where the authors propose a method to train goal conditioned policies via supervised learning using Transformers. Specifically, the authors treat the transitions of state, action and goal in a trajectory as a sequence, and train a Transformer network with causal attention to predict the future action and goals in an autoregressive way. In addition, in order to make the model pay more attention to goals, the authors also introduce the strategy of goal masking during training, where the ground truth goals are replaced with a mask token with certain probability.\n\nThe authors evaluate the proposed method in Gym locomotion and Atari tasks, and the results show that the proposed method is able to perform the correct task condition on given goals.\n", "summary_of_the_review": "Given the concerns about novelty and empirical evaluations, I cannot recommend acceptance of the paper in its current state.\n", "strength_and_weaknesses": "I think this paper presents an interesting idea of training goal conditioned policies. However, I do have some important concerns about the proposed method.\n\n\nI’m not convinced about the novelty of the proposed method. The proposed method is essential decision transformers [1] applied to goal conditioned reinforcement learning, where the return-to-go is replaced by the goal in the transitions, where the goal can be considered a special case of rewards. Hence the only novelty for the proposed method is the application of masking in goals, which I do not believe is enough contribution.\n\nMy major concern about this paper is that the authors have not included any comparison of the proposed method to previously proposed baselines. Goal or return conditioned RL is a relatively mature subfield in RL with many prior works. Just to name a few, I believe the specific tasks solved in the paper can also be solved by prior methods such as Decision Transformers [1], Trajectory Transformers [2], HER [3], GCSL [4], RCP [5] and RvS [6]. Therefore, it is impossible to verify the empirical performance of the proposed method without comparing to some of these baselines.\n\n\n### References\n[1] Chen, Lili, et al. \"Decision transformer: Reinforcement learning via sequence modeling.\" Advances in neural information processing systems 34 (2021): 15084-15097.\n\n[2] Janner, Michael, Qiyang Li, and Sergey Levine. \"Reinforcement learning as one big sequence modeling problem.\" ICML 2021 Workshop on Unsupervised Reinforcement Learning. 2021.\n\n[3] Andrychowicz, Marcin, et al. \"Hindsight experience replay.\" Advances in neural information processing systems 30 (2017).\n\n[4] Ghosh, Dibya, et al. \"Learning to reach goals via iterated supervised learning.\" arXiv preprint arXiv:1912.06088 (2019).\n\n[5] Kumar, Aviral, Xue Bin Peng, and Sergey Levine. \"Reward-conditioned policies.\" arXiv preprint arXiv:1912.13465 (2019).\n\n[6] Emmons, Scott, et al. \"RvS: What is Essential for Offline RL via Supervised Learning?.\" arXiv preprint arXiv:2112.10751 (2021).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OofJ1XylI_H", "reviewer": "Reviewer_HqN1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The goal of this paper is to try to learn a goal-conditioned policy which is robust to goal-switching, i.e. when the goal given to the policy changes in the middle of the episode. The proposed approach uses a causal transformer, and proposes masking the goal randomly to ensure that the model (GMT) is able to predict the goal from action and state information. The paper presents results on MuJoCo and Atari environments. ", "summary_of_the_review": "Overall, this paper presents an interesting direction and a novel approach that provides an initial result towards this direction, but design choices and explanations of the approach are unclear. Additionally, this approach needs a more thorough set of evaluations (see strengths and weaknesses sections). ", "strength_and_weaknesses": "Strengths: \n\n- To my knowledge, the idea of goal masking in GCSL-style methods is novel\n- The method has interesting promise and good applications \n\nWeaknesses\n\nIn my opinion, a major weakness is the lack of detail around why certain design decisions are made about this approach. For example, I think more discussion is needed as to why goal switching is a difficult problem with the current framework. Wouldn't since we learn a policy pi(a_t | g_t, a_t) via hindsight relabelling, shouldn't the policy automatically react well to changing g_t. Another detail missing is how exactly the training data is selected, how the goals are chosen. Couldn't we think of goal-switching as two runs of shorter length? \n\nSecondly, I think more evaluations are needed. There are no comparisons to baseline approaches (Decision Transformer, standard GCSL, etc). It is important to show how these methods perform when the goal is switched in the middle of the episode. The metric used for the Mujoco tasks is the episode reward, but for goal-conditioned frameworks, the success rate is what is important. Can the policy reach the goal? It is unclear why clustering is needed, adds a strong assumption that we need to know the full state space, which will not be true when scaling to more difficult settings, thus I think this approach should be tried on random (not clustered goals). Ablations such as no-masking, are also missing. \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E1TeMlJkfR", "reviewer": "Reviewer_33J4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies offline goal-conditioned reinforcement learning problems and proposes a novel offline goal-conditioned supervised learning method, Goal-Masked Transformers (GMT), which applies random goal-masking and predict goals as an auxiliary tasks. The learned policy could successfully switch the goals during a single episode (e.g. goal 0 --> goal 1) in both MuJoCo or Atari environments", "summary_of_the_review": "This paper lacks the definition of goals, proper comparison against relevant baselines, and novelty compared to Decision Transformer or Generalized Decision Transformer. Considering those aspects, I vote for rejection.", "strength_and_weaknesses": "### Strength\n- The writing is easy to follow.\n- The learned transformer policy seems to smoothly switch the goal during a single episode, compared to multiple models or shot context (Figure 7).\n\n\n### Weaknesses\n- There are no baselines for comparison. Decision Transformer [1], Categorical Decision Transformer or Bi-directional Decision Transformer [2],  Prompt Decision Transformer [3], GoFAR [4], WGCSL [5], and LiPS [6] can be relevant methods to be compared (of course, not the all but more is better).\n- The goals in MuJoCo and goals in Atari are quite unclear. In MuJoCo, goals seem to stand for the discrete label that is clustered with k-means (based on x-velocity in the trajectory). In Atari, goals seem to stand for the returns in the trajectory. As far as I know, those abstract quantities are often not considered as a goal in the existing goal-based RL literature.\n- The goal switching is limited to once in the episode. Also, the number of goals is limited to 3 (0, 1, 2 in MuJoCo, 2, 5, 10 in Atari).\n\n\n[1] https://arxiv.org/abs/2106.01345\n\n[2] https://arxiv.org/abs/2111.10364\n\n[3] https://arxiv.org/abs/2206.13499\n\n[4] https://arxiv.org/abs/2206.03023\n\n[5] https://arxiv.org/abs/2202.04478\n\n[6] https://arxiv.org/abs/2012.03548\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject