Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Proposes an approach to using a single bidirectional transformer to accomplish many different reinforcement learning tasks, including behavior cloning, (sub)goal conditioning, and conditioning on properties more generally. It accomplishes this by creating a sequence of out of the (state, action, properties) for all timesteps, then either (1) training on a single type of mask, (2) training across many masking styles drawn from fixed set, and (3) randomly masking positions in the sequence. (2) and (3) can be finetuned in single task fashion. They evaluate on a toy gridworld and the mujoco 2d maze tasks. They find that Uni[MASK] performs better across both given a short truncated sequence, but it underperforms a GPT model for longer sequences. They attribute this to difficulties in training BERT models.
strengths_and_weaknesses: 
	- originality
		- Concurrent work suggests that the core ideas of this paper are in the air (see: foundation posterior, multigame decision transformer, The UL2 paper on unifying masking schemes).
	- quality
		- The paper is well executed. It has simple experiments which verify the core claims.
		- My main concern is that the length 10 context experiments underperforming. Would like to see more study of this instead of alluding to claims that BERT has this known failure mode.
	- clarity
		- The paper is well written and a pleasant read. It adequately cites prior work, and it does a great job of explaining and illustrating the core ideas.
	- significance
		- The paper is a solid contribution to the literature. Overall, its core findings and technique align with concurrent and prior work.
questions: 
	1. Clarify which task is shown in table 1. The grid world or maze?
	2. Consider expanding on what position vs. timestep encoding is in the body of the paper (line 231)?
	3. The "bert has no mouth and must speak" paper was withdrawn
	4. Take a look at XLNet and NADE style masking, which would allow combining benefits of GPT and BERT
	5. One possibile reason the longer sequence is failing is because it is never exposed to many masked positions in a row at the end of the sequence. Randomized decoding as in xlnet might help here, or applying more structured masking (such as blocking out spans instead of ). See the uni
		1. Perhaps training on both multitask and 
	6. What happens if you train on both multitask and random masking? 
	7. Potential improvements
		1. Vary the sequence lengths instead of holding fixed
			1. This is more natural for GPT style models, but can be done with XLNet/NADE approaches as well
			2. How might Uni[MASK] handle the full length 200 timestep sequences instead of truncating to last 5/10?
		2. Ideally would have more complicated tasks than the two presented
		3. Consider studying how 
	8. Relevant concurrent work: foundation posterior, which applies a similar approach to inference in probabilistic programs (i.e. generalized bayesian networks) in Stan.
	9. nits
		1. works -> work in line 300
limitations: There are no major concerns with this paper
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents the Uni[MASK] framework, which generalizes masked language modeling (in natural language processing) to more general sequence decision problems. Given actions, states and optional property tokens (such as the return-to-go), multiple tasks such as behavioral cloning, reward-conditional offline RL, forward dynamics, etc., may be represented by specific masking schemes. The authors propose to train a model by using a randomized masking scheme (instead of single-task only or with a fixed set of tasks). Such a Uni[Mask] model may also be fine-tuned on a given task afterwards. A pre-trained Uni[MASK] model often performs comparably to single-task training, and fine-tuning further improves results.

[Post author response] I updated the rating I assigned to this paper, in particular because of the correction to the originally misleading discussion.

strengths_and_weaknesses: **Strengths**

The proposed approach neatly formalizes multiple tasks under a general masking framework.

Training a Uni[MASK] model, then finetuning it on a given task, generally leads to better results than single-task training only.

The approach is tested in both discrete (MiniGrid) and continuous (Maze2D) environments.

**Weaknesses**

The discussion around line 200 is misleading. In figure 7, multi-task performs worse than single-task (although sometimes negligibly), which doesn't support H1.

The distinction between BERT-like and GPT-like models should be described more clearly (see question below).

The MiniGrid environment appears to be very simple. I would be interested in seeing how well the approach scales with larger grids.
questions: **Questions**

Most important question: Could you please clarify exactly what you mean by "BERT" architecture vs "GPT" architecture, and how they interact with the masking schemes you propose? In NLP, BERT uses a transformer encoder (self-attention) with some masked inputs whereas GPT uses a transformer decoder (causal self-attention, i.e. not looking at the future).

L160+: Should there be a reward for shorter paths? Otherwise, there could be useless 

L190: Why no test data?

L212: Do you mean figure 15 in the appendix? In general, figures referred in the main text should also be within the paper itself.

**Suggestion**

Please move the description of random masking (end of appendix 1) and appendix 2 to the main text. This is too important to leave out.

**Other**

L235: This points to the wrong appendix.
limitations: Some limitations are discussed.
review_title: null

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This submission proposed a new framework, Uni[MASK], which performs unified inference in sequential decision problems, using different masking schemes. 

The authors demonstrated that randomly sampling masking schemes help the bidirectional transformer model being able to do behavior cloning, rewarding conditioning, dynamics modeling and more.

The experiments are first on grid-world and then extended to Mujoco-physics Maze2D environment. The comparison between single-task, multi-task, random-task, with finetune are through and solid, integrating the theory with practice, and quite convincing. The experiments in Mujoco-physics Maze2D environment shows that Uni[MASK] framework outperforms baselines, such as Feedforward Neural Network and Decision Transformer.

I fully endorse that this submission is a wonderful work: well designed experiments, diplomatic and sound illustration and the novelty of unifying decision making through Uni[MASK] and bidirectional models.
strengths_and_weaknesses: Strengths:
* Uni[MASK] unifies inference task in sequential decision problems.
* The illustration of motivation, theory and experiments design is convincing and sound, written and presented very well.
* The authors demonstrate how randomly sampling masking schemes at training time produces a single multi-inference-task model that can do behavior cloning, reward-conditioning, dynamics modeling and more.
* The authors test how training on many tasks affects single-task performance.
* The authors show how fine-tuning models trained with random masking consistently outperforms single-task models.
* The proposed Hypothesis H1 ( {multi-task, random-mask, finetune} > single-task) and H2 ( {random-mask > multi-task} ) make sense intuitively, as "H1 test whether models indeed learn richer representations by training on multiple inference tasks" and H2 test "whether training on all possible tasks by randomly sampling maskings at training time is better than selecting a set of specific maskings"
* The results from well designed experiments strongly support hypothesis H1 and H2.
* The authors extensively analyze how different training regimes (combination of single-task, multi-task, random mask, fintune) affect performance. The analysis is very convincing to me.
* The authors also make experiments comparison with GPT-like architectures, and the analysis is also very convincing: "We find that while using GPT seems to yield similar performance to BERT for context length five, using GPT seems to give an advantage for longer sequence lengths," and "This suggests that if one were able to use a GPT architecture and train it with random masking and fine-tuning, it might be possible to get the best of both worlds."

Weakness:
* Relative short context lengths, and the experiments is mainly on the grid world (4 * 4), if the grid size increases, it might further improve this work.

questions: From line 285 to line 2899, "A clear avenue of future work would therefore be to get the “best of both worlds”: long sequences and benefits of random-mask pre-training by using a GPT-like architectures, with our random-mask and finetune training regimes. This which would require finding ways to make GPT act like a bidirectional model – for which recent methods in NLP might offer a useful starting point". In the rebuttal period, may I know the opinion from the authors, why making GPT act like a bidirectional model is important in the decision making scenario? The answers to this question would make me (maybe other readers) understand this paper better.
limitations: * Relative short context lengths. Experiments with longer context lengths would make this submission more convincing.
* The experiments are mainly on the grid world (grid size is 4* 4). If the grid sizes increase more, this work would become more persuasive.
review_title: null

Meta-review: metareview: This paper extends masked language modeling to other sequential decision making problems. The idea is simple (which is a plus), the experiments are thorough, and the results are convincing. All reviewers agreed this is a good paper. I recommend acceptance.
confidence: Certain
Decision: Accept