Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper trains a new 130B parameter model with the GLM architecture/objective, with contributions in automatically stabilizing spikes with existing techniques, focussing on keeping inference costs and requirements low and therefore the model accessible to a large number of people. It is an achievement to beat GPT3 numbers and in a few cases PaLM numbers.
strength_and_weaknesses: Strengths:
 - Impressive set of techniques put together for stability and model quality (DeepNorm initialization, GLM instead of CausalLM objective, Embedding Layer Gradient Shrink)
 - Impressive performance on measured benchmarks esp compared to GPT3 & in a few cases PaLM

Weaknesses
 - Would have loved to see some ablations, especially over the first few ~100B tokens as to which technique contributes how much to performance, either within GLM-130B or implemented across models, i.e. for example with T5 style models the GLM objective would be essentially T5’s denoising task + prefixLM. Otherwise it is not clear what technique is responsible for how much of the gains.
 - More evaluations: The evaluation section is severely limited
   - It could have evaluated fine-tuning over a few tasks (Ex: PaLM has both few-shot numbers and fine-tuning numbers for a few benchmarks like SuperGlue and TyDiQA) — this is understandably partly a resource issue.
   - More extensive few-shot evaluations on say SuperGlue, Winograd style tasks, CBQA style tasks, CommonSense reasoning, Chain of Thought prompting etc — Since inference is both fast and cheap this shouldn’t be too much of an issue and would increase my confidence of this method’s applicability and more importantly in the released checkpoint.
   - Not evaluating on tasks with fixed labels (like NLI) seems like severely and unnecessarily restrictive, i.e. I don’t think Xian et al 2018’s advice applies here, one can/should do NLI tasks with encoding the label in the prompt, while not giving an example (for zero-shot) or giving a few examples of each class (in a few-shot setting) and evaluating. Since MultiTask Instruction Pre-training (MIP) is happening here, I think it is fair to only exclude tasks that are substantially similar in the actual example content, if the task type seen is similar (say classification with the same labels) this should be called out and reported separately.

clarity,_quality,_novelty_and_reproducibility: Points of Clarification (did not impact the score):

p1 / Introduction - “However, both GPT-3 (and other 100B-scale ones)—the model itself—and how it can be trained, have been thus far not available to the public.”

It might be better that this be reframed for clarity, I think all the models describe how they are trained (i.e. methodology in their paper), two of these have weights released (OPT 175B on request and Bloom 176B on github here - https://huggingface.co/bigscience/bloom/tree/main), and atleast Bloom’s script is at https://github.com/bigscience-workshop/bigscience/tree/master/train/tr11-176B-ml#readme

p2 / Introduction — “associated with significantly less bias” any causal explanation?

p2 / Introduction — “unique property of GLM architecture” “INT4 quantization introduces very limited performance losses” : It is not clear that ability to do INT4 comes directly from GLM architecture, it could also be the other changes like training objective (MASK + gMASK) or initialization choices like DeepNorm etc or a combination of both. Fig 5 does explain why INT4 works for GLM-130B and not BLOOM, but the claim that this is due to the GLM architecture seems strong, accordingly “unique property” seems overly strong and unsubstantiated.

p2 / Introduction — “associated with significantly less bias and generation toxicity”: Would the authors make any causal explanation as to why this ought to be the case, i.e. the major contribution seems to be pre-training mechanics, I cannot see how “less bias” would follow, this might be empirically true, but lacking a causal explanation as to why this would be so (for example: data cleanup or filtering) this can only be taken as an incidental fact, rather than a recipe to have less bias.
p4 / Section 2.2 — “Self-Supervised Blank Infilling”: This section is slightly confusingly framed, as written it led me to believe that both [MASK] and [gMASK] are in the **same sequence** — it wasn’t until section 2 ending on p5 where it became clear that these are two separate tasks (due to the lengths being different in MASK and gMASK). I would advice to split the section “Self-Supervised Blank Infilling” into two, one for MASK, another for gMASK, or at least call out them separately within the Self-Supervised section.


Minor comments and typos (that did not impact the score):

 - p1 / Abstract - “disconvergence”, maybe write “divergence” since that’s the commonly used term? Or do you want to convey that disconvergence = opposite of convergence, so could be either divergence (loss steadily increasing) or NaN or just unstable loss?
 - p1 / Introduction - “capabilities suddenly arouse”, s/arouse/arise or arose.
   - Same comment as above on p9 / Sec 6 “arouse as models” → “arise as models”
   - p9 / Sec 7 Point 4 “arises” — however this point is poorly written
 - p1 / Introduction - “pioneers the studies of”, s/studies/study
 - p1 / Introduction - “we come to realize”, reframe as “we came to realize” or “we have come to realize”
 - p1 / Abstract & p2 / Introduction - “most ever affordable”, “most affordable” would suffice here.
 - p5 / Sec 3 - “We spend months” change to “We spent months”
 - p5 / Sec 3 - “in sacrifice of a significant penalty on model performance”, this is quite unclear, please reframe
 - p6 / Sec 3 - “first used in the multi-modal transformer CogView”, maybe just cite the cogview paper right there, sometimes paper title does NOT contain the name of the model, so the reference will be hard to find.
 - p8 / Sec 5.3 - “Here is out intuitive attempt” → “Here is our intuitive attempt” ; “we come up with” → “we came up with”
 - p8 / Sec 5.3 “if we ever got a chance to continu pre-training GLM-130B” - I would suggest summarizing under “future work”
 - p9 / Sec 6 - “despite many emerged 100B” cut out emerged
 - p9 / Sec 6 - “it has been concentrated on” reframe perhaps
 - p9 / Sec 7 - “architecture alternative, in addition to GPTs” →  “architecture alternative to GPTs” you are pointing out that GLM is an alternate way of self-supervision. I would rather frame it as an alternate “objective” instead.
 - p9 / Sec 7 Point 6 - “gradient to it’s 0.1” - ill framed sentence
 - p9 / Sec 7 Point 7 - “unique” is a strong word for here
 - p10 / Ethics - “And to estimate and better collaborate” no need for estimate
 - p10 / Reproducibility - PaLM citation is wrong, Paperno 2016 is mentioned instead of the PaLM citation.
 - p10 / Reproducibility - “We have paid great exertion”, better way to frame would be “We paid great effort”

summary_of_the_review: Weak Accept owing to a severely limited evaluation section and methodology, the work itself pulls a lot of techniques together to get a stably pre-trained O(100B) model with GLM objective and stability techniques which is very valuable to get broad recognition.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: It's always hard to summarize papers that report on such a big project, but fundamentally this is a tech report of training an English+Chinese LLM with techniques that improve on the state of the art for pretraining methodology and training stability.

GLM-130 is concurrent with OPT and BLOOM, two other GPT-3-class LLMs that were released openly this year. It has d_model=12288 (same as GPT-3, OPT, and BLOOM) and 130B parameters and was trained on 200B tokens each of English and Chinese, including English NLU tasks formatted for multitask instruction pretraining to increase zero-shot performance on other, unseen English and Chinese tasks. It used a GLM objective that mixes span-by-span span-filling and prefix-LM, enabling use of bidirectional attention at inference time to further improve zero- and few-shot performance.

It consistently outperforms GPT-3, OPT, and BLOOM on English despite using fewer parameters and less compute (also less English data than GPT-3/OPT), and it sometimes also outperforms the much larger PaLM. It also consistently outperforms ERNIE TITAN 3 on Chinese, despite fewer parameters, less compute, and less Chinese data. As it can run in both bidirectional and unidirectional mode at inference time, ablating this shows that bidirectional attention contributes greatly to its outperformance. It also exhibits less bias/toxicity than other LLMs.

In terms of optimization methodology, it uses relatively standard techniques including AdamW with LR warmup and cosine decay, and also uses early batch size warmup to a relatively large 8.65M token total batch size. For training stability, it uses post-LN with depth-based initialization ("DeepNorm"), downscaled gradient in embeddings ("Embedding Gradient Shrink"), gradient clipping, and dropout. Either the DeepNorm or something else about the model makes the weight distribution especially narrow, enabling near-lossless int4 weight quantization for small-footprint inference.

In terms of training efficiency, it achieves about 32.5% MFU at steady state [135e12/312e12 * 3/4 for remat] and 25.1% MFU end to end [(130e9*400e9*6)/(96*8*312e12*60*60*24*60)] using 8 pipeline stages with a PipeDream-Flush (aka 1F1B) schedule, 4-way tensor parallelism, and 24-way (2-way intranode and 12-way internode) data parallelism. The authors also provide an efficient FasterTransformer-based inference setup that supports consumer GPUs.
strength_and_weaknesses: Strengths:
- Very careful attention to training stability
- Comprehensive details
- Accessibility to smaller scale researchers
- Open source
- General openness even when admitting something could have been done better (e.g. "We later found the instructions to implement two-dimensional RoPE from its author’s blog https://kexue.fm/archives/8397, but our training has proceeded for weeks.")

Weaknesses:
- The work chooses optimization hparams like initial LR and LR schedule relatively unsystematically.
- They use dropout without explaining why, even though there's no repeated data in the dataset
- The precision discussion mentions avoiding bf16 because of fp32 gradient accumulation, but fp16 or even bf16 accumulation is typically fine in practice.
- There isn't really an explanation of why GLM-130 has narrower weight distribution than BLOOM (my guess is it's related to DeepNorm)?
- The paper doesn't explicitly state that the Pile test set was held out of the training set, but I'm assuming it was.
- Figure 7 leaves out PaLM 1- and 5-shot BBL, which are much better than 0-shot, and Chinchilla (which was only evaluated at 5-shot). Those are reasonable choices, but I'd prefer to err on the side of including more data points. Table 4 also leaves out PaLM and Chinchilla 5-shot.
- no explanation of how 135 TFLOPS was computed (in particular, it probably includes remat but this isn't stated). Would recommend adopting MFU/HFU from PaLM paper.
clarity,_quality,_novelty_and_reproducibility: GLM-130B has very clear architecture and training details, relative to some other LLM papers. While it also has relatively few truly novel architectural or training components, this is reasonable as large-scale work isn't the time to introduce those. It also has unprecedented reproducibility for an LLM training paper, due to releasing just about every important component in open source.

What appear to be the two key advances relative to other contemporary LLMs (bidirectional attention and instruction pretraining) are concurrent with other very recent work (e.g. UL2 and Flan from Google) so it could be helpful to contextualize with those.
summary_of_the_review: I think this really represents the gold standard in reproducibility for LLM publications, as well as the most capable open-source English or Chinese language model in existence.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces GLM-130B, a bilingual English and Chinese LLM. The model is trained for 400B tokens (half on English  and half on Chinese). The model follows a bidirectional attention architecture with the self-supervised blank infilling objective. They also include a small percentage of MIP tokens. Int4 quantization enables inference on a diverse set of platforms. Various model instabilities and mitigation strategies are discussed.

The model, training log and inference through free APIs are provided and an effort is made to make the entire work reproducible.

Detailed comments:
Table 4: You state that the reason for PaLM's 1-shot ability being so much better than GLM is the diverse dataset. Could you elaborate a bit more?
Section 6, subsection name "Transferring" --> consider using a better subsection name.
strength_and_weaknesses: Strengths:
- The model is more reproducible than other similar sized counterparts.
- The model's performance is competitive on the benchmarks that are evaluated on and outperforms large models like GPT-3 and PaLM.
- Quantization for inference and mitigation strategies like deepnorm and embedding shrinkage for training instability are useful for the community.

Weaknesses:
-  I would have liked to see more English tasks evaluated on. The current set of results on the LaMBADA dataset a lot. I would like to see results on more GPT-3 tasks or SuperGLUE tasks for English.
- More results at different scales and projection of trends or emerging capabilities as scale increases would have been interesting. The current set of results are missing this.
clarity,_quality,_novelty_and_reproducibility: Clarity:
Paper is clearly written and easy to follow.

Quality:
The work is high quality and various aspects of building the model are studied well.

Novelty:
One could argue that the paper lacks some novelty as they do not comprehensively produce new techniques that have been ablated at a smaller scale that have shown to produce improvements. But I understand that this is hard to do at such scale since they only get one shot at training the model. Smaller scale experiments with some of the new techniques could have helped here.

Reproducibility:
Overall they take good care to make the model, training log, evaluation sets are reproducible.
summary_of_the_review: Overall, the paper is clearly written and details the building of the GLM-130B bilingual LLM. The model is competitive with other similar sized models and care has been taken to make the model set up reproducible. The evaluation suite for English could have been more comprehensive to substantiate the claims a bit more.  I would have also liked to see more experiments at a smaller scale and studies of emerging capabilities of the model at scale. Since this is a bilingual model, studying how much transfer or interference happens between these two languages would have been interesting.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces and open-sources one of the largest 100B bilingual pre-trained language model GLM-130B. It covers a detailed pre-training process (training strategies, design choices) and the resulting model outperforms several large language models on English and Chinese zero-shot understanding benchmarks. The author also investigates the efficient inference of the GLM and show the efficacy of using Int4 quantization. 

strength_and_weaknesses: GLM-130B: AN OPEN BILINGUAL PRE-TRAINED MODEL
*Paper Summary
This paper introduces and open-sources one of the largest 100B bilingual pre-trained language model GLM-130B. It covers a detailed pre-training process (training strategies, design choices) and the resulting model outperforms several large language models on English and Chinese zero-shot understanding benchmarks. The author also investigates the efficient inference of the GLM and show the efficacy of using Int4 quantization. 

Summary Of Strengths
- The paper is well-written and clearly presented;
- It detailed the efforts made during the pre-training process for the GLM-130B, making it very valuable for practitioners in the large language model pre-training area; (though several design choices might not be feasible to ablate at that scale.). 
- The resulting model was tested across benchmarks, and showed surprising robustness towards quantization, making it more accessible for academia. 


Summary Of Weaknesses
- Why the bi-lingual GLM performs a lot better in LAMBADA as shown in Figure 2? Is this also true for MMLU, Pile, and NLG tasks, any intuition about this?
- Why not compare to OPT-175B models for quantization? 
- Intuition why the few-shot performance of GLM-130B in Table 4. falls behind other methods? Also, will the quantized GLM-130B yield similar few-shot performance?
- Which GPT-3 175B is the GLM-130M compared to? Is it a fairer comparison to the InstructGPT variants?
- How will the author attribute the performance gain (Bi-directional modeling [1], Multi-Task Instruction Pre-Training [2], or the Bilingual corpus and model architecture)?
- It is not surprising that the Bi-directional model will outperform the Uni-directional model in understanding tasks. How are the generation tasks performance of GLM-130B compared to uni-directional counterparts?


[1] Tay, Yi, et al. "Transcending Scaling Laws with 0.1% Extra Compute." arXiv preprint arXiv:2210.11399 (2022).

[2] Chung, Hyung Won, et al. "Scaling Instruction-Finetuned Language Models." arXiv preprint arXiv:2210.11416 (2022).
clarity,_quality,_novelty_and_reproducibility: - The paper is well-written and clearly presented;
- The model is open-sourced; 
- The design choice and detailed engineering efforts 
- More analysis behind the surging performance will be very valuable to add, for example:
    - Any emergent abilities of the model (in terms of scale and in terms of the amount of data) as shown in [3]. 
    - Why the Bilingual model outperforms the Uni-lingual model by a large margin. 
    - Few-shot performance of quantized GLM-130B; 

summary_of_the_review: It is not surprising that a large-scale bi-directional language model will outperform the causal counterpart on a range of zero-shot language understanding tasks. The detailed engineering efforts and design choices make this paper valuable, though my main concerns are what factors make the performant GLM-130B unclear, as well as NLG / few-shot performance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: All reviews liked the paper which includes comprehensive experiments and details. The reviewers also liked the amount of engineering effort behind this paper. Major plus for open source! 

From reading the reviews and paper, I think this is a pretty neat technical report that could be accepted to ICLR. 

Therefore, I recommend acceptance. 
justification_for_why_not_higher_score: This paper surely deserves to be accepted, but I don't think it's very interesting for an oral since it's a technical report. I would prioritize more interesting papers for oral/spotlight.


justification_for_why_not_lower_score: This paper is a good one that should be accepted.
Decision: Accept: poster