Kati8/ICLR_2023_2025 huggingface dataset:
authors: Xiang Li
authors: Xin Jiang
authors: Xuezhi Fang
authors: Yiqun Yao
authors: Xuying Meng
authors: Aixin Sun
authors: Yequan Wang
reviews: 
Comments: The paper propose FreeLM as a finetuning-free method for downstream tasks. Compared with pretraining then fine-tuning paradigm, FreeLM fuses the task data into pretraining data and is iteratively trained in the pretraining stage. On several classification tasks, FreeLM is shown to be better than much larger GPT-3.
Strengths: - Injecting task bias into pretraining stage is an interesting idea.
- The paper is mostly well written, with intensive experiments and analysis on various aspects of the proposed method.
Weaknesses: - **Need more in-depth analysis regarding the influence on NLG tasks**: As the author mentioned that FreeLM has a higher perplexity than GPT-2, it would be more informative to provide a numerical comparison on NLG benchmarks instead of only showing a few cases.
- **Missing comparison with multitask fine-tuning**: Table 4 does not include the results of the multitask fine-tuning (https://openreview.net/forum?id=gEZrGCozdqR) baseline, which is also learning-free when applied to other tasks. It would be important to compare with multitask fine-tuning results and analyze the generalization ability compared with FreeLM.
- **Scalability**: In-context learning is another learning-free method when performing downstream tasks and the performance of which gains a lot after scaling. However, there is a lack of comparison with in-context learning when model scales up.
Recommendation: 3: reject, not good enough
Reviewer confidence: 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
Soundness/correctness: 2 fair
Clarity: 3 good
reviews: 
Comments: In this paper, the authors introduce a novel approach to pretrain a decode-only language model, eliminating the need for subsequent fine-tuning. Drawing on prompt-based learning principles, their model, FreeLM, excels in a variety of downstream tasks. Impressively, it surpasses the performance of larger models like GPT-3 and InstructGPT on the GLUE benchmark, despite having substantially fewer parameters.
Strengths: 1. By merging pretraining and finetuning into one unified stage, the authors present a potentially more efficient training approach.
 
2. Remarkably, despite its smaller size, FreeLM surpasses larger models like GPT-3 and InstructGPT in GLUE's downstream tasks.
Weaknesses: 1. While the "teacher iterative" method is an interesting take on prompt-based learning, merging pretraining and prompt-finetuning into a single step may not be as novel as it seems. A clear distinction and justification for this fusion could strengthen the paper.

2. The choice of the GLUE benchmark for evaluation does not make sense to me for a few reasons:

    1. GLUE is typically used to assess single-task, supervised learning models, mainly encoder-only (BERT-like) or sometimes encoder-decoder LMs (T5-like). However, decoder-only LMs, like FreeLM, may not perform as well on these tasks. For instance, decoder-only LMs generally score around 80 on MNLI, while state-of-the-art encoder-only models can achieve scores over 90. Even BERT-Base can achieve an accuracy of 84.
  
    2. The paper's approach to evaluating "generalization to unseen datasets" does not follow standard practices. A more established benchmark for zero-shot generalization of LMs on classification tasks is T0-Eval (https://arxiv.org/pdf/2110.08207.pdf), which includes 62 training datasets and 12 evaluation datasets.

    3. It is more appropriate to assess decoder-only LLMs on generative tasks, such as MathQA, HumanEval, MBPP, or OpenBookQA, rather than GLUE.

3. The comparison between FreeLM, GPT-3, and InstructGPT may not be entirely fair. A better approach would be to create a pretraining-finetuning baseline or to fine-tune an open-source LM like LLaMa on the GLUE benchmark for comparison.

4. Lastly, the paper's writing could be improved for clarity and readability. The language should be fluent and succinct, rather than coming across as if it was translated from another language.
Recommendation: 1: strong reject
Reviewer confidence: 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
Soundness/correctness: 1 poor
Clarity: 1 poor
reviews: 
Comments: The paper unifies the NLP tasks and proposes a fine-tuning-free (though not training-free) method for training a language model, claiming better performance than GPT-3 on various tasks (a point I view with skepticism).
Strengths: - The paper unified the NLP tasks, which could be helpful for research to evaluate LLM (but I believe [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) already did a good job now)
- I like the figures.
Weaknesses: Minor Weakness:
- I take issue with the term "fine-tuning-free" for the language model, as it suggests a misconception that the proposed method requires no training. I acknowledge that the community lacks a clear definition of "fine-tuning," commonly understood as "continuous training" on a pre-trained model. Nonetheless, the proposed language model also undergoes continuous training based on GPT-2, contradicting the "fine-tuning-free" label. Furthermore, even if you are starting training from scratch, calling it a "fine-tuning-free" language model is misleading if it indeed involves parameter updates.

Major Weaknesses:
- The paper uses many terms to embellish the method, such as "teacher signal" and "language signal," but in reality, the approach is a "co-training" scheme that trains on both target tasks and unsupervised English data. The paper argues that "pre-training and fine-tuning is not an optimal paradigm" and suggests "training them together." I do not recognize the benefits here. Is it to prevent catastrophic forgetting? If that's the case, it requires verification within the paper. Overall, this paper lacks novelty and fails to demonstrate the effectiveness of the proposed method.
- I assume one of the motivations behind "co-training" is to enhance generation ability. However, the authors did not substantiate this with perplexity scores. Some examples are unconvincing. More evidence is needed to support it, and if other methods are considered unreliable, human evaluation is required.
- The paper asserts that their proposed LM achieved SOTA, surpassing GPT-3, which is not accurate. Even straightforward fine-tuning on these tasks with GPT-2 outperforms Free-LM. Would it then be plausible for the authors to assert that "fine-tuning GPT-2 on these target data achieves SOTA and is superior to GPT-3"? The authors should explicitly state the limitations in this section. By the way, the authors also need to show details how they design the prompt for GPT-3, an improper prompt also has big impact on the performance.
Recommendation: 3: reject, not good enough
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 1 poor
Clarity: 3 good