Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors proposed a technique to improve zero-shot and few-shot capabilities for Large language models (LLM). Specifically, when performing the next token prediction task, the proposed method will randomly select past tokens masked out to fine-tune the LLMs. Experimental results show that the proposed FCM improves PaLM in both zero and few-shot tasks.
strength_and_weaknesses: Strength:
1. The proposed method is simple and easy to reproduce.
2. Experimental results show the proposed method can effectively improve the LLMs in both zero- and few-shot tasks.

Weaknesses:
1. This method can be seen as a variation of schedule sampling (Bengio et al., 2015) or a simplified version of UniLM (Dong et al., 2019). The novelty of this paper is limited.
2. On the other hand,  the authors should give more analysis about why this method can work in LLMs. The training set of pre-training models contains a lot of noise data. I wonder if the proposed method is necessary.


Bengio et al., 2015. Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks.
Dong et al., 2019. Unified language model pre-training for natural language understanding and generation.
clarity,_quality,_novelty_and_reproducibility: This clarity of this paper is clear, and the method mentioned in this paper is easy to reproduce.  The novelty of this paper is limited.
summary_of_the_review: The novelty of this paper is limited.  Methods similar to this paper have been widely used in previous work. On the other hand,  the authors should give more analysis about why this method can work in LLMs.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel, simple, scalable and efficient pre-training approach for causal, decoder-only language models. The proposed approach is called Forgetful Causal Masking (FCM) and it randomly masks out past tokens. FCM pre-training is applied to PaLM to demonstrate significant improvement in the 0-shot performance on the SuperGLUE benchmark. FCM is also tested on a variety of 0-shot and few-shot tasks - PIQA, ARC, OpenBookQA, Winograd, WinoGrande, ANLI, StoryCloze, and LAMBADA tasks to demonstrate consistent performance improvement across such diverse tasks. 

strength_and_weaknesses: ## Strengths
- Simple and scalable approach: no extra parameters or computations are added to pre-training using causal LM objectives.
- Extensive experimentation
- Performance improvements (modest) on a wide variety of tasks

## Weaknesses

### Training 
- __(W.0)__ Kindly share the computational resources used, memory requirements and the training time.

### Results and Analysis
- __(W.1)__ In Table 1, along with the mean of the 3 random realizations, the spread should be provided as well, given that many of the performance numbers are within a single percentage point. 
- __(W.2)__: The performance of FCM appears to be better than PaLM, but not significantly and consistently so. For significance, I’m using a criterion of at least 1% in absolute terms. It will be good if the authors can shed some light on the following:
- __(W.2.a)__ __0-shot__: In 8/19 tasks, the performance improvement is < 1%.  
- __(W.2.b)__ __1-shot__: In 9/ 19 tasks, the performance improvement is < 1% and 2/19 times it’s worse. 
- __(W.2.c)__ __few-shot__: In 4/ 19 tasks, the performance improvement is < 1% and 4/19 times it’s worse. 
- __(W.2.d)__ Scaling trends: If we look at the consistency of performance improvement as the number of shots increase, there are very few tasks, on which FCM consistently stays better than PaLM by at least a percentage point. On the NLU tasks, FCM seems to lose its advantage as the number of shots increase, sometimes becoming much worse.
- __(W.3)__ __Classification finetuning__: results on SuperGLUE dev set only show significant improvement on 4 out of 8 tasks, with majority of the gains observed on WSC and CB. The authors should discuss the variability in performance gains.
- __(W.4)__ __Scalability, 0-shot__: In Table 4, FCM 1B results seem better than FCM 8B on CB, RTE, and WiC tasks. Kindly discuss. 
- __(W.5)__ __Scalability, k-shot__: In Table 5, the performance of both PaLM and FCM sometimes, and more often for FCM, degrades as the number of shots increases. Kindly discuss.
- __(W.6)__ __Ablation of mask ratio__: Irrespective of the average across all the tasks, FCM[0,0.15] is better only on 2 out of 8 tasks in the 0-shot setting and 3 of 8 tasks in the 1-shot setting. This shows that the choice of mask ratio is task dependent. Kindly discuss. 
- __(W.7)__ __Comparison with Dropout__: Contrary to the claim of dropout hurting FCM performance, it improves in 3 of 8 tasks. It improves performance by 10.2 absolute % points for the MultiRC task. It seems that the benefit of dropout is task dependent.

clarity,_quality,_novelty_and_reproducibility: **Clarity**: The ideas are presented reasonably clearly and in a structured manner.

**Quality**: The quality of the paper is mostly good, with some minor issues pointed out below which are easily fixable.
- some grammatical mistakes and typos.
- Appropriate use of capitalization in the ‘References’ section.

**Novelty**: 
- Low on technical novelty. The simple, novel, scalable pre-training masking tactic seems to have a broad minor to moderate impact on a variety of tasks. I expect this to become a part of the pretraining toolbox for causal LLMs.

**Reproducibility**:  
- Should be reproducible (subject to availability of computational resources).

summary_of_the_review: The paper proposes a simple and scalable pre-training tactic - of causal random masking for causal language modeling. It doesn’t require additional parameters or computations. Extensive experimental evaluation is performed. While the performance gains seem modest to moderate, and somewhat inconsistent, they are largely positive across a large range of tasks. Though the technical novelty is limited, I expect the proposed mechanism to become a part of the pretraining toolbox for causal LLMs.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple method of randomly masking past tokens during causal language modeling that boosts zero-shot capabilities  and fine-tuning results by a non-trivial margin. The problem motivation is capturing the best of both decoder and encoder LMs. While previous methods achieve this as well, the authors assert that the proposed method in this paper does not add any additional computational cost. 
strength_and_weaknesses: Strengths
1. The method is very simple and well explained in the method section. Experimental details are also well explained.
2. Different ablations of which factor of the proposed method contribute to the performance enhancement are provided.
3. Proposed method boosts performance in the zero-shot setting as well as when fine-tuned to perform the downstream task. 

Weaknesses
1. While the proposed method does not add any computational cost in theory, since previous tokens are randomly masked out, won't there be an engineering overhead since the previous hidden states cannot be cached which would have remained unchanged if the previous token weren't masked out otherwise? 
clarity,_quality,_novelty_and_reproducibility: It would be great if the authors could elaborate more on the disadvantage of randomly making out previous tokens in terms of the actual engineering implementation. It would be great if the authors could clarify if there were any misunderstanding. 

Besides this part, the paper is well-written and easy to follow, with clear explanations of the technical parts. The experimental set-up is well explained, and the results are also well portrayed with convincing results, explanations, and ablations. 


summary_of_the_review: The authors propose a very simple methodology of simply masking out random tokens during causal language modeling which proves to be effective at enhancing both zero-shot capabilities and the downstream fine-tuning results. I highly recommend this paper be published at this conference. However, it would be great if the authors could elaborate more on the question of the proposed method in terms of the engineering implementation aspect.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to randomly mask past tokens in casual language models. 

strength_and_weaknesses: *Strength*:
This paper is well written and easy to understand.

*Weaknesses*:

The biggest weakness of this paper is ignoring related research papers and lack of comparison. 
[1] presents comprehensive study about different training objectives in zero-shot priming and finetuning. They compared 6 different training objectives in table 1 (https://arxiv.org/pdf/2205.11726.pdf): NXTUNI, NXTPRE, MSKUNI, MSKBI, HYBUNI, HYBPRE. Casual language modeling such as GPT models belong to NXTUNI. They controlled hyper-parameters and training data to be same and studied these six different objectives. The most relevant method to this paper is HYBUNI (casual masking objective). [2] is the representative work of causal masking objective. This paper presents this new causal masking objective without comparing it with the most relevant casual masking method (CM3). 

In addition, this paper should carry out apple-to-apple comparisons with other models. For example, it needs to control model sizes (model parameters), batch size, hyper-parameters, training tokens, training devices, to be exactly the same. And control Total train compute (flops) to be similar. Therefore, the qualified apple-to-apple comparison is only PaLM 1B v.s. FCM 1B (in table 3), PaLM 128M v.s. FCM 128M (in table 4). For others, PaLM 8B used different training data with FCM 8B, and T5 used different model parameters. 

*References*

[1]. Artetxe, Mikel, et al. "On the Role of Bidirectionality in Language Model Pre-Training." arXiv preprint arXiv:2205.11726 (2022).

[2]. Aghajanyan, Armen, et al. "Cm3: A causal masked multimodal model of the internet." arXiv preprint arXiv:2201.07520 (2022).
clarity,_quality,_novelty_and_reproducibility: *Quality*:

Due to lack of comparisons with relevant methods, I don’t think this paper has a high quality.

*Novelty*:

The casual masking method proposed by this paper is quite similar to [2] and HYBUNI in [1]. [2] also adds some masks on the basis of casual language modeling. The difference is that [2] puts the masked tokens to the end of sentence, so the masked places can not only attend to previous tokens but also gather information from future tokens. Compared to the FCM in this paper, I think the performance of causal masking methods from [2] might be better. But due to lack of comparison, I cannot see the performance differences.

*Reproducibility*:
easy to reproduce. 

*References*

[1]. Artetxe, Mikel, et al. "On the Role of Bidirectionality in Language Model Pre-Training." arXiv preprint arXiv:2205.11726 (2022).

[2]. Aghajanyan, Armen, et al. "Cm3: A causal masked multimodal model of the internet." arXiv preprint arXiv:2201.07520 (2022).
summary_of_the_review: This paper presents a new casual masking objective, and carries out apple-to-apple comparisons with PaLM 128M, PaLM 1B. In comparison with this paper, [1] studied 6 different training objectives (including the most relevant casual masking method) and carried out comparisons with these six training objectives with 5 different scales: 125M, 355M, 1.3B, 2.7B, 6.7B. Due to seriously ignore related articles and casual masking method from [2] is pretty similar as this paper (approach of [2] might be better), I choose to reject this article.


*References*

[1]. Artetxe, Mikel, et al. "On the Role of Bidirectionality in Language Model Pre-Training." arXiv preprint arXiv:2205.11726 (2022).

[2]. Aghajanyan, Armen, et al. "Cm3: A causal masked multimodal model of the internet." arXiv preprint arXiv:2201.07520 (2022).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This submission proposed a simple yet effective pre-training approach, Forgetful Causal Masking (FCM), for causal, decoder-only Large language models (LLM). FCM randomly masks out past tokens during causal language modeling in next token prediction task. Experimental results show that the proposed FCM improves PaLM consistently in both zero and few-shot tasks on diverse benchmarks including SuperGLUE, PIQA, ARC, OpenBookQA, Winograd, WinoGrande, ANLI, StoryCloze, and LAMBADA.  It is popular area trying to capture the best of both decoder and encoder LMs. The authors demonstrate that the proposed method does not add any additional computational cost.

Strength of the submission:
1. The proposed method is simple and easy to reproduce. No extra parameters or computations are added to LLM, which allows FCM to be applied to broader use cases.
2. Experimental results also show the proposed method effectively improve the LLMs in both zero- and few-shot tasks. Experiments are done on a variety of popular benchmarks in zero shot and few shot tasks. The consistent improvement and simple techniques make it almost no doubt the benefit of FCM on causal LLM is generalizable. 
3. Several ablations to explore which which factor of FCM contributes to the performance gain.

Weakness/missing of the submission:
1. Several reviewers raise the concern about technical novelty as the method is relatively simple. Novelty is a relatively subjective metric, and simple but effective method is still valuable. However, in a highly competitive venue such as ICLR, we do hope the accepted paper can enlight the community with future research direction and demystify why the method works or not works, beyond showing promising performance results. The submission proposes a simple method that improves performance consistently across diverse benchmarks. It is convincing that the proposed method is effective. However, the reviewers prefer to see more discussion about what makes the simple method work (not only ablation showing how each component contributes to the final results, but also what's the intution behind the simple design, what's the conjecture why the approach is effective, any patterns the authors observed in the experiments that can/can't be applied to other tasks or type of LLMs, any theory or experiments that can support the conjecture, etc.). With such discussion, the impact of the submission will be signifcantly upleveled.

The authors and several reviewers have disagreement about if FCM is similar to existing methods such as CM3/HYBUNI. The area chair believes that the difference between proposed method and existing ones is often subjective but less decisive. It's more important to provide empirical evidence supporting efficacy of the overall method, insight and conjecture why the method works, and theory/reasoning/experiments verify the conjecture.
justification_for_why_not_higher_score: This is a hard decision. The current submission reads more like a methodology paper with solid performance gain. The reviewers hope to see more in-depth discussion about the insight and conjecture why the proposed method works, and theory/reasoning/experiments verify the conjecture. Such discussion will entertain more ICLR participants and uplevel the impact of this submission.
Decision: Reject