GLM-130B: An Open Bilingual Pre-trained Model
Authors: Aohan Zeng
Xiao Liu
Zhengxiao Du
Zihan Wang
Hanyu Lai
Ming Ding
Zhuoyi Yang
Yifan Xu
Wendi Zheng
Xiao Xia
Weng Lam Tam
Zixuan Ma
Yufei Xue
Jidong Zhai
Wenguang Chen
Zhiyuan Liu
Peng Zhang
Yuxiao Dong
Jie Tang
Venue: ICLR 2023 poster (2023)

Abstract: We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It is an attempt to open-source a 100B-scale model as good as GPT-3 (davinci) and unveil how models of such a scale can be successfully pre-trained. Over the course of this effort, we face numerous unexpected technical and engineering challenges, particularly on loss spikes and divergence. In this paper, we introduce the pre-training process of GLM-130B including its design choices, training strategies for both efficiency and stability, and engineering efforts. The resultant GLM-130B model offers significant outperformance over GPT-3 175B on a wide range of popular English benchmarks while the performance advantage is not observed in OPT-175B and BLOOM-176B. It also consistently and significantly outperforms ERNIE TITAN 3.0 260B—the largest Chinese language model—across related benchmarks. Finally, we leverage a unique scaling property of GLM-130B to reach INT4 quantization with almost no performance loss, making it the first among 100B-scale models and more importantly, allowing its effective inference on 4×RTX 3090 (24G) or 8×RTX 2080 Ti (11G) GPUs, the most ever affordable GPUs required for using 100B-scale models. The GLM-130B model weights are publicly accessible and its code, training logs, related toolkit, and lessons learned are open-sourced at https://github.com/THUDM/GLM-130B/.

Published as a conference paper at ICLR 2023
GLM- AN OPEN BILINGUAL PRE-TRAINED MODEL
Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang", Hanyu Lai", Ming Ding Zhuoyi Yang", Yifan Xu², Wendi Zheng", Xiao Xia Weng Lam Tam Zixuan Ma², Yufei Xue Jidong Zhai,, Wenguang Chen Zhiyuan Liu", Peng Zhang Yuxiao Dong Jie Tang
Tsinghua University
Zhipu.AI
ABSTRACT
We introduce GLM-130B, a bilingual (English and Chinese) pre-trained language model with 130 billion parameters. It is attempt to open-source 100B-scale model at least as good as GPT-3 (davinci) and unveil how models of such a scale can be successfully pre-trained. Over the course of this effort, we face numer- ous unexpected technical and engineering challenges, particularly on loss spikes and divergence. In this paper, we introduce the training process of GLM--30B including its design choices, training strategies for both efficiency and stabil- ity, and engineering efforts. The resultant GL.M-13BB model offers significant outperformance over GPT-3 175B (davinei) benchmarks while the performance advantage is not observed in OPT-175B and BLOOM-176B. It also consistently and significantly outperforms ERNIE TITAN 3.0 260B the largest Chinese language model- across related benchmarks. Fi- nally, we leverage unique scaling property of GLM-130B to reach INT4 quanti- zation without post training, with almost no performance loss, making it the first among 100B-scale models and more importantly, allowing its effective inference on 4XRTX 3090 (24G) or 8 8xRTX 2080 Ti (11G) GPUs. the most affordable GPUs required for using 100B-scale models. The GLM-130B model weights are publicly accessible and its code, training logs, related toolkit, and lessons learned are open-sourced at https:
wide range of popular English
com/ THUDM/ GLM-130B/.
INTRODUCTION
Large language models (LLMs), particularly those with over 100 billion (100B) parameters (Brown et al., 2020; Thoppilan al., 2022; Rae et al., 2021: Chowdhery et al., 2022; Wang et al., 2021), have presented attractive scaling laws (Wei et al., 2022b), where emergent zero-shot and few-shot capabilities suddenly arose. Among them, GPT-3 (Brown et al., 2020) with 175B parameters pi- oncers the study of 100B-scale LLMs by strikingly generating better performance with 32 labeled examples than the fully-supervised BERT-Laage model on variety of benchmarks. However, both GPT-3 (and many other closed-sourced 100B-scale ones) the model itself and how can be trained, have been thus far intransparent to the public. of critical value train a high-quality LLM of such scale with both the model and training process shared with everyone.
We thus aim to pre-train open and highly-accurate 100B-scale model with ethical concerns in mind. Over the course of our attempt, we have come to realize that pre-training a dense L.M at such a scale raises numerous unexpected technical and engineering challenges compared training 10B-scale models, in terms of pre-training efficiency, stability, and convergence. Similar difficulties
'The two lead authors AZ and XL contributed equally (1 (Zengachan, shawliu9) Sgmail.com "Wor partially done when AZ, XL, and ZD interned at Ah.
*Team leads: YD and JT. Corresponding author: JT anietangete i Lnghua edu.en) For detailed author contributions, please refer Appendix E.

---

Published as a conference paper at ICLR 2023
Language Ability Evaluation
Bias & Toxicity Evaluation
13.3
43,9
44.8
GLM
GPT-3
77.9
5.20
PALM
67.2 GPT-3
74.7
BLLOOM
32.1
OFF-3
60.8
65.8 GLM
6.3
GPT-3
LAMBADA shot)
lite shot)
MMLU (5 (Sishot)
StereoSet (ICAT)
CrowS Fairs (PLL,
GLM-1300
GPT-3 1758
OPT:175B
BLOOM 1768
= PaLM 5408
Figure 1: A summary of the performance evaluation and ethical studies.
Table I: A comparison between BLM-1300B and other 100B-scale LLMs and PaLM 540B. (LN:
layer norm.; FPF: floating-point format; MIP: multi-task instruction pre-training; CN Chinese)
Architecture & Data
Truining
Inference
Open- source
Model
Objective
LN
Major Lang.
FPF
Stabilization
Quantization GPU Needed
GPT-3 17SB
x
English
FP16
andisclosed
undisclosed
undixclosed
OPT-17SB
GPT
Pre-LN
English
FP16
Manual Adjusting
INT8
8 x 3090
BL.0MM-776 176B
V
Multi-lingual
BF16
Embedding Norm
INT8
8 x 3090
PaLM 540B
X
GPT
Pre-LN
English
BF16
Manual Adjusting
undisclosed
undisclosed
GLM (Blank
Deep-
Bilingual (EN & CN)
Embedding
INT4 4 x 3090 or 8 1080 Ti
GLM-130B
Infilling & MIP)
Norm
FP16
Gradient Shrink
have also been concurrently observed in training OPT-175B (Zhang et al., 2022) and BLOOM- 176B (Scao al., 2022), further demonstrating the significance of GPT-3 as a pioneer study.
In this work, we introduce the pre-training of a 100B-scale model-GGMM113 neering efforts, model design choices, training strategies for efficiency and stability, and quantization for affordable inference. As it has been widely realized that is computationally unaffordable to empirically enumerate all possible designs for training 100B-scale LLMs, present not only the successful part for training GLM-130B but also many of the failed options and lessons learned. Particularly, the training stability is the decisive factor in the success of training models of such scale. Different from practices such as manually adjusting learning rates in OPT-175B and using embedding norm in the sacrifice of performance in BLOOM-I76B, we experiment with various op- tions and find the strategy of embedding gradient shrink can significantly stabilize the training of GLM- 130B.
in terms of engi-
Specifically, GLM-1300 is a bilingual (English and Chinese) bidirectional dense model with 130 bil- lion parameters, pre-traiie preitrained over 400 billion tokens on a cluster of 96 NVIDIA DGX-A100 (8 40G) GPU nodes between May 6 and July 3, 2022. Instead of using the GPT--ttyle architecture, we adopt the General Language Model (GLM) algorithm (Du et al., 2022) to leverage its bidirectional at- tention advantage and autoregressive blank infilling objective. Table summarizes the comparison between GLM-130B, GPT-3 and another two open-source -Pforts-PT--755 and BLOOM- as well as PaLM 540B (Chowdhery et al., 2022) 4x larger model-as reference.
Altogether, the conceptual uniqueness and engineering efforts enable GLM- 130B to exhibit perfor- mance that surpasses the level of GPT-3 on a wide range of benchmarks (in total 112 tasks) and also outperforms PaLM 540B in many cases, while outperformance over GPT-3 has not been observed in OPT-175B and BL.OMM 166 76B (Cf. Figure left). For zero-shot performance, GLM--30B is better than GPT-3 175B (+5.0%), OPT- (+6.5%), and BLOOM-1768 (+ (+1000% on LAMBADA (Pa- perno et al., 2016), and achieves 3 better performance than GPT-3 on Big-bench-lite (Srivastava et al., 2022). For the 5-shot MMLU (Hendrycks et al., 2021) tasks, it is better than GPT-3 175B (+0.9%) and BLOOM-176B (+ (2.7).). As a bilingual LLM also in Chinese, offers significantly better results than ERNIE TITAN 3.0 260B (Wang et al., 2021)-the the largest Chinese LLM on zero-shot CLUE (Xu et al., 2020) datasets (+24.26%) and pert FewCLUE (Xu et al., 2021) ones (+12.75%) Importantly, as summarized in Figure right, GLMM130B as an open model is associated with significantly less bias and generation toxicity than its 100B-scale counterparts.
Finally, we design GLM-130B to empower as many people as possible to conduct 100B-scale LLM studies. First, instead of using 175B+ parameters as OPT and BLOOM, the 130B size is decided be- cause such size supports inference on single A100 (8 40G) server. Second, to further lower the GPU requirements, we quantize GLM-130B into INT4 precision without post training while OPT and BLOOM can only reach INT8. Due to a unique property of the GLM architecture, GLM-1300 INT4 quantization introduces negligible performance degradation, e.g.. -0.74% on LAMBADA and +0.05% on MMLU, making it still better than the uncompressed GPT-3. This enables GLM-
2

---

Published as a conference paper at ICLR 2023
12
Sandwich-LN (GLM-1308)
Gradient Norm
Gradient Norm
Post-LN with DeepNorm (GLM-1308)
10
500
1.5c
2.5k
3k
500
1000
1500 2000 2500 3000
(a) More than 30 failed preliminary trials 100B-scale (b) Final decisive trials: Sandwich-LN DeepNomn
Figure 3: Trials on different LayerNorms for GLM-1301 training. It turns out that DeepNorm is the most stable one, as has small gradient norm and does not spike in the early stage training.
130B's 's fast inference with performance guarantee on a server of 4xRTX 3090 (24G) or 8xRTX 2080 Ti (11G), the most affordable GPU required for using 100B-scale LLMs to date.
We open-source the model checkpoints, code, training logs, related toolkits, and lessons learned.
THE DESIGN CHOICES OF GLM-130B
The architecture of a machine learning model defines its inductive bias. However, has been real- ized that it is computationally unaffordable to explore various architectural designs for LLMs. We introduce and explain the unique design choices of GLM--30B.
2.1 GLM-130B's 's ARCHITECTURE
GLM Backhone. Most recent 100B-scale LLMs,, such GPT-3, PaLM, OPT, and BLOOM, follow the traditional GPT-style (Radford et al., 2019) architecture of decoder-only autoregressive language modeling. In GLM-130B, we instead make an attempt explore the potential of bidi- rectional GLn-General Language Model (Du et al., 2022)-as its backbone.
GLM is a transformeer-asee language model that leverages autoregressive blank infilling its train- ing objective. Briefly, for a text sequence x [x1,... ). text spans {$1. 8m are sampled from it, each of which s; denotes a span of consecutive tokens [s..1. ss.l, and is replaced (i.e.,
corrupted) with a single mask token to form Zeverpee The model is asked to recover them autoregres-
sively. To allow interactions between corrupted spans, their visibility to each other is decided by randomly sampled permutation on their order.
GLM's s bidirectional attention over unmasked (i.e., uncorrupted) contexts distinguishes GLM- 130B from GPT-style LLMs in which the unidirectional attention is used. To support both understanding and generation, it mixes two corruption objectives, each indicated by a special mask token:
[MASK]: short blanks in sentences whose lengths add up certain portion of the input.
[gMASK]: random-length long blanks at the end of sentences with prefix contexts provided.
Conceptually, the blank infilling objective with bidi- rectional attention enables more effective compre- hension of contexts than GPT-style models: when us- ing [MASK], GLM-130B behaves BERT (Devlin et al., 2019) and T5 (Raffel et al., 2020); when us-
Zero shro LAMBADA
Centent
SOTAA (FALM 5405)
75,2
Bidircct onaal Adtettonn (igg.
ing [gMASK], GLM- 1100 behaves similarly Pre- fiXLM (Liu et al., 2018; Dong et 2019).
Empirically, GLM- 13MM offers a record- high accuracy of 80.2% on zero-shot LAMBADA by outperforming both GPT-3 and PaLM 540B in Figure 2. By setting the attention mask, GLM-130B's unidirectional vari- ant comparable to GPT-3 and OPT-175B. Our ob- servations are in line with existing findings (Liu al., 2018; Dong et al., 2019).
Unidiretizall 68SA PI
GFT-3
GFT SL.0M 21MM
Figure 2: GLM-130B 130B and LLMs similar scale on zero-shot LAMBADA language modeling. Details on GLM's bidirectional attention provided in Du et al. (2022).
Layer Normalization (LN, Ba et al. (2016)). Training instability is one major challenge for training LLMs (Zhang et al., 2022; Scao et al., 2022; Chowdhery et al., 2022) (Cf. Figure 10 in Appendix for collapses in training several 100B-scale models). A proper choice of LNs help stabilize
3

---

Published as a conference paper at ICLR 2023
the training of LLMs. We experiment with existing practices, e.g., Pre-LN (Xiong et al., 2020), Post-LN (Ba et al., 2016), Sandwich-LN (Ding et al., 2021), which are unfortunately incapable of stabilizing our GLM- 130B test runs (Cf. Figure (a) and Appendix B.2 for details).
Our search is later focused on Post-LN due to its favorable downstream results in preliminary ex- periments though it does not stabilize GLM-130B Fortunately, one of the attempts Post-LN initialized with the newly-proposed DeepNorm (Wang et al., 2022b) generates promising training stability. Specifically, given the number of GLM-130B's layers N, we adopt DeepNorm(2)) LayerNorm(a a + ())) where a = (2N) and apply the Xavier normal initialization with the scaling factor of (2N) to Efn. v j_r and out put_proj.. Additionally, all bias terms are initialized to zero. Figure shows it significantly benefits the training stability of GLM-130B.
Positional Encoding and FFNs. We empirically test different options for positional encoding (PE) and FFN improvements in terms of both training stability and downstream performance (Cf. Ap- pendix B.3 for details). For PEs in GLM-130B, we adopt Rotary Positional Encoding (RoPE, Su al. (2021)) rather than ALiBi (Press et al., 2021). To improve FFNs Transformer, we pick GLU with the GeLU (Hendrycks & Gimpel, 2016) 6) activation as the replacement.
2.2 GLM-130B' PRE-TRAINING SETUP
Inspired by recent works (Aribandi et al., 2022; Wei et al., 2022a; Sanh et al., 2022), the GLM-130B pre-training objective includes not only the self-supervised GLM autoregressive blank infilling) but also multi-task learning for a small portion of tokens. This expected to help boost its downstream zero-shot performance.
Self-Supe Blank Infilling (95% tokens). Recall that GLM-130B uses both [MASK] and [gMASK] for this task. Each training sequence is applied with one of them independently at time. Specifically, [MASK] is used to mask consecutive spans in 30% of training sequences for blank infilling. The lengths of spans follow Poisson distribution (A 3) and add up to 15% of the input. For the other 70% sequences, the prefix of each sequence is kept as context and [gMASK] is used to mask the rest of it. The masked length is sampled from the Uniform distribution.
The pre-training data includes 1.2T Pile (train split) (Gao et al., 2020) English, 1.0T Chinese Wudao- Corpora (Yuan et al., 2021), and 250G Chinese corpora (including online forums, encyclopedia, and QA) we crawl from the web, which form a balanced composition of English and Chinese contents.
Multi-Task Instruction Pre- Tre-Trainig (MIP, 5% tokens). T5 (Raffel et al., 2020) and ExT5 (Aribandi et al., 2022) suggest that multi-task learning in pre-training can be more helpful than fine-tuning, we thus propose to include a variety of instruction prompted datasets including language understanding, generation, and information extraction in GLM-130B's ' pre-training
Compared to recent works (Wei et al., 2022a: Sanh et al., 2022) that leverage multi-task prompted fine-tuning to improve zero-shot task transfer, MIP only accounts for 5% tokens and is set in the pre- training stage prevent spoiling L.MMs' other general ability, e.g., unconditional free generation. Specifically, we include 74 prompted datasets from (Sanh et al., 2022; Wang et al., 2022a), listed in Appendix C and Table 12. GLM-130B users are suggested avoid evaluating its zero-shot and few-shot capabilities on these datasets according to the criterion illustrated in Section 5.
2.3 PLATFORM-AWAR PARALLEL STRATEGIES AND MODEL CONFIGURATIONS
GLM-130B is trained on cluster of 96 DGX-A100 GPU (8x40G) servers with 60-day access. The goal is to pass through as many tokens as possible, as a recent study (Hoffmann et al., 2022) suggests that most existing LLMs are largely under-trained.
The 3D Parallel Strategy. The data parallelism (Valiant, 1990) and tensor model paral- lelism (Shoeybi et al., 2019) are the de facto practices for training billion-scale models (Wang & Komatsuzaki, 2021; Du et al., 2022). To further handle the huge GPU memory requirement and the decrease in overall GPU utilization resulted from applying tensor parallel between nodes 40G rather than 80G 1 are used for training GLM- 130B, we combine the pipeline model parallelism with the other two strategies to form 3D parallel strategy.
The pipeline parallelism divides the model into sequential stages for each parallel group, and to fur- ther minimize bubbles introduced by pipeline, we leverage the PipeDream- Flush (Narayanan al.,
4

---

Published as a conference paper at ICLR 2023
2021) implementation from DeepSpeed (Rasley et al., 2020) to train GLM-130B with a relative big global batch size (4,224) to reduce time and GPU memory wasting. Through both numeri- cal and empirical examinations, (Cf. Appendix B.4 for details). Following the calculation in (Chowdhery et al., 2022), we report hardware FLOPss utilization (HFU) of 43.3% and model FLOPss utilization (MFU) of 32.5% due to re-materializaton.
adopt 4-way tensor parallelism and 8-way pipeline parallelism
GLM-130B Configurations. We aim enable our 100B-scale LLM run single DGX-A100 (40G) node in FPI6 precision. Based on the hidden state dimension of 12,288 we adopt from GPT-3, the resultant model size has to be no more than 130B parameters, thus GLM-130B, To maximize GPU utilizationn we configure the model based on the platform and its corresponding parallel strategy. To avoid insufficient memory utilization in the middle stages due to the additional word embedding at both ends, we balance the pipeline partition by removing one layer from them, making 9-2-2=70 transformer layers in GLMM-30B.
During the 60-day access to the cluster, we manage to train GLM-130B for 400 billion tokens (roughly 200 billion each for Chinese and English) with a fixed sequence length of 2,048 per sample. For the [gMASK] training objective, we use context window of 2,048 tokens. For the [MASK] and multi-task objectives, we use a context window of 512 and concatenate four samples together to cater the 2,048- sessseuecceeeegthhh We warm-up the batch size from 192 to 4224 over the first 2.5% samples. We use AdamW (Loshchilov & Hutter, 2019) as our optimizer with 31 and 32 set to 0.9 and 0.95, and weight decay value of 0.1. We warm up the learning rate from 10-7 to 8 10-5 over the first 0.5% samples, then decay it by a 10x cosine schedule. We use a dropout rate of 0.1 and clip gradients using a clipping value of 1.0 (Cf. Table 11 for the full configurations).
3 THE TRAINING STABILITY OF ELM-130B
The training stability is the decisive factor in GLM-130BBs quality, which is also largely impacted by the number of tokens passes through (Hoffmann et al., 2022). Thus, given the computing usage constraint, there has to be a trade-off between efficiency and stability with regard to floating- point (FP) formats: low-precision FP formats (e-g., 16-bit precision FP16) improve computing efficiency but are prone to overflow and underflow errors, resulting in training collapses.
MixessPre We follow the common practice of a mixed- precision (Micikevicius et al., 2018 8) strategy (Apex O2), i.e., FP16 for forwards and backwards and FP32 for optimizer states and mas- ter weights, to reduce the GPU memory usage and improve train- ing efficiency. Similar to OPT-175B and BLOOM- 176B (C.f. Fig- ure 10 in Appendix), the training of GLM-130B faces frequent loss spikes resulted from this choice, which tends to become increas- ingly frequent the training goes on. The precision related spikes are often without clear reasons: some recover on their own; others come with portent of suddenly soaring gradient norm and even- tually spike or even NaN in loss. OPT- attempted to fix by manually skipping data and adjusting syper-parameters: BLOOM- 176B did so via the embedding norm technique (Dettmers et al., 2021). We spent months to empirically investigate the spikes and realize that a few issues emerge when transformers scale up:
1.5
Embedding layer
Gradient Norm
Transformen
1.0
Transtormer layer
layer (c=0.3)
0.5
0.0 2000
8000
(a)
Gradient norm with EGS
0.1
9.0
wit shrink M-n0R)
pre-training loss
8.0
shrnnk e=0.1 (GLM 4081
7.0
6.0
5.0
4.0
First, the transformer main branch's value scale can be extremely large in deeper layers if using Pre-LN. This is addressed in GL.M- 130B by using DeepNorm based Post-LN (Cf. Section 2.1), which makes the value scale always bounded.
2000 4000
6000
(b) EGS in 40B-scale testing
Figure 4: EGS reduces gradi- ent scale and variance to stabi- lize I.LMs' pre-training.
Second, the attention scores grow so large that they exceed FP16's range, as the model scales up. There are few options to overcome this issue in LLMs. In View (Ding et al., 2021). PB -e-Rellx is proposed to remove bias terms and deduct extremum value in attention computation to avoid the problem, which unfortunately does not help avoid dis- convergence in GLM- 130B. In BLOOM-176B, the BF16 format is used instead of FP16, due to its wide range of values on NVIDIA Ampere GPUs (i.e,, A100). However, BF16 consumes 155%
5

---

Published as a conference paper at ICLR 2023
more run-time GPU memory than FP16 in experiments due to its conversion to FP32 in gradi- ent accumulation, and more importantly it is not supported on other GPU platforms (c.g.. NVIDIA Tesla V100), limiting the accessibility of produced LLMs. Another option from BLOOM-I768 is to apply embedding norm with BF16, but in sacrifice of a significant penalty on model performance, they notice that embedding et al., 2022)).
can harm model's zero-shot learning (Cf. Section 4.3 in (Scao
Embedding Layer Gradient Shrink (EGS). Our empirical search identifies that the gradient norm can serve as an informative indicator of training collapses. Specifically, we find that a training collapse usually lags behind "spike" in gradient norm by a few training steps. Such spikes are usually caused by the embedding layer's abnormal gradients, as we observe that its gradient norm often several magnitude larger that those of other layers in GL-M-130B early stage training (Cf. Figure 4 (a)). In addition, it tends to fluctuate dramatically in the early training. The problem is handled in vision models (Chen al., 2021) via freezing the patch projection layer. Unfortunately, cannot freeze the training of the embedding layer in language models.
Finally, we find the gradient shrink on embedding layers could overcome loss spikes and thus sta- bilize GLM-130B's training. It is first used in the multi-modal transformer Cog View (Ding et al., 2021). Let a be the shrinking factor, the strategy can be easily implemented via word embedding = word ewweeddngnnnn + word embedding. detach() embedding.detacch)) (1 a). Figure 4 (b) suggests that empirically,
setting a = 0.1 wipes out most spikes we would have met, with negligible latency.
In fact, the final GLM-130B training though it fails numerous times due to hardware failures. For the three unexpected spikes, turns out further shrinking the embedding gradient still help stabilize the GLM-130B training. See the training notes and Tensorboard logs in our code repository for details.
only experiences three late-stage loss divergence cases,
GLM- 130B INFERENCE ON RTX 2080 TI
One of the major goals of GLM-130B is to lower the hardware requirements for accessing 100B- scale LLMs without efficiency and effectiveness disadvantages.
As mentioned, the model size of 130B is determined for running the full GLM- model on a sin- gle A100 (40G 8) server, rather than the high-end A100 (80Gx8) machine required by OPT-175B and BI.OOM-1761 To accelerate GL.-1300P inference, we also leverage PasterTrrasformmr (Ti- monin et al., 2022) to implement GI.M-130B in C++. Compared to the PyTorrch implementation of BLOOM-176B in Huggingface, GLM- 1-00's decoding inference is 7-8.4x faster on the same single A100 server. (CF. Appendix B.5 for details).
INT4 Quantization for RTX 3090s/2080s. To further support popularized GPUs, we attempt to compress GLM-130B as much as possible while maintaining performance superiority, particularly via quantization (Zafrir et al., 2019; Shen al., 2020; Tao et al., 2022), which introduces little task-agnostic performance drops for generative language models.
Typically, the practice is quantize both model weights and activations to INT8. However, our anal- ysis in Appendix B.6 suggests that LLMs' activations may contain extreme outliers. Concurrently, the emer- gent outliers in OPT-175B and BLOOM-176B are also discovered (Dettmers et al., 2022), which influ- ence only about 0.1 % feature dimensions and are thus solved by matrix multiplication decomposition for the outlying dimensions. Differently, there exist about 30% outliers in GLM-130B's activations, making the technique above far less efficient. Thus, we decide to focus on the quantization of model weights (i.e., mostly linear layers) while keeping the FP16 precision for activations. The quantized model is dynamically converted to FP16 precision runtime, in- troducing a small computational overhead but greatly reducing the GPU memory usage for storing model weights.
11
00040 VOPBWY7?
60
Attr-dense Layer o
40
20
GLM 36
GLM INT4
BLOON 28-0
BL00N 1974
w2: Layer
10a
10
GUM-1308
BLOOM-177 1768
Effective Parameter Count
Figure 5: (Left) attn-der.se dense and w2's weight distributions; (Right) GLM-1330B INT4 weight quantization scaling law.

---

Published as a conference paper at ICLR 2023
Table Left:
Quantized GLM- 130B's performance on several benchmarks; Right: INT4 quantized 1LLB's inference speed (encode and decode) with FasterTransformer.
GLM-1300
130B
GPT-3
GPU Type
128 Enc./Dec.
512 Enc./Dec
Model Precision
FP16
INT8
INT4
FP16
x A100 (40G)
0.15s
4.29s
0.18s
17.7s
MMLU (acc, 1)
44.75
44.71
44.80
43.9
8 x V100 (32G)
0.31s
6.97s
0.67s
28.1s
LAMBADA (acc, 1)
80.21
80.21
79.47
76,2
4 X RTX 3090 (24G)
0.37s
8.16s
1.30s
32.3s
Pile (a part, BPB, 0.634
0.638
0.641
0.74
8 x RTX 2080 Ti (11G)
0.39s
6.77s
1.04s
27.3s
Excitingly, we manage to reach the INT4 weight quantization for GLM-130B while existing suc- cesses have thus far only come the INT8. Memory-wisse by comparing to INT8, the INT4 version helps additionally save half of the required GPU memory to 70GB, thus allowing GLM-130B infer- ence 4 RTX 3090 Ti (24G) or 8 x RTX 2080 Ti (11G). Performance-wist Table left indicates that without post-training at all, the INT4-version GLM-130B experiences almost no performance degradation, thus maintaining the performance advantages over GPT-3 on common benchmarks.
GLM's INT4 Weight Quantization Scaling Law. We examine the underlying mechanism of this unique INT4 weight quantization scaling law exhibited in Figure 5 right. We plot the weight value distributions in Figure 5 left, which turns out to directly impact the quantization quality. Specifically, a wider-distributed linear layer needs be quantized with larger bins, leading to more precision loss. Thus the wide-distributed attn-dense and w2 matrices explain the INT4 quantization failure for Gty-style BLOOM. Conversely, GLMs tend to have much narrower distributions than those of similar-sized GPTs, and the gap between INT4 and FP16 versions keeps further decreasing as the GLM model size scales up (Cf. Figure 15 in Appendix for details).
THE RESULTS
We follow the common settings in LLMs such as GPT-3 and PaLM to evaluate GLM-130B for English !. As bilingual LLM with Chinese, GLM-130B is also evaluated Chinese benchmarks.
Discussion on the Scope of Zero-Shot Learning in GLM-130B. Since GLM-130B has been trained with MIP, here we clarify its scope of zero-shot evaluation. In fact, "zero-shot" seems to have controversial interpretations without a consensus in the community. We follow one of the in- fluential related surveys (Xian al., 2018), which says "At test time, in zero-shot learning setting, the aim is to assign a test image to an unseen class label" " where involving unseen class labels is a key. Therefore, we derive criterion to pick GLM- 130B's 's zero-shot (and few-shot) datasets as:
English: 1) For tasks with fixed labels (c.g., natural language inference): no datasets in such tasks should be evaluated on; 2) For tasks without fixed labels (c.g.. (multiple-choice QA, topic classi- fication): only datasets with an obvious domain transfer from those in MIP should be considered.
Chinese: All datasets can be evaluated as there exists zero-shot cross- coss-lingual transfer.
Filtering Test Datasets. Following prior practices (Brown et al., 2020; Rac et al., 2021) and our criterion mentioned above, we filter and refrain to report potentially contaminated datasets' evalua- tion results. For LAMBADA and CLUE, we find minimal overlap under the 1 13-gram setting. Pile, MMLU, and BIG-bench are either held-out or released later than the crawling of corpora.
5.1 LANGUAGE MODELING
LAMBADA. LAMBADA (Paperno et al., 2016) is a dataset to test the last word language model-
ing capability. The results previously shown in Figure suggest GLM- 130B achieves zero-shot accuracy of 80.2 with its bidirectional attention, setting up a new record on LAMBADA.
Pile. The Pile test-set (Gao et al., 2020) includes series of benchmarks for language modeling. On average, GLM- 130B performs the best its 18 shared test sets in terms of weighted BPB when compared to GPT-3 and Jurassic- 1 (Lieber et al., 2021) whose results are directly adopted from the latter, demonstrating its strong language capability (Cf. Appendix C.4 for details).
Table 3: GLM-130B's 's average BPB on Pile evaluation (18 sub-datasets).
Jurassic-1 GPT-3 GLMM-1308
Avg. BPB
0.650
0.742
0.634
Results in OPT-175B's paper are reported as applications to access have not been approved for months.

---

Published as a conference paper at ICLR 2023
46
GL56-3303 Dahht
12
0-shot I-shot 3-shot
GPT-3 1758 ( shot)
GLM 1208 1-shut
GPT-3 D-snet
GLM 300 3-shot
GPT-32.68
0.60
0.71
1.83
GPT-3 L-atek
3-shet
GPT-3 6.7B
-0.06
2.93
5,40
38
FaLM O-stet
GPT-3 13B
1.77
5.43
7.95
GPT-3 175B
4.35
11.34
13.18
34
4
BLOOM 1768 (5-shot)
PaI.M
8.05
37.77
GLM-130B 13.31 14.91 15.12
300 Trained Tokens (Billion)
150
109
10a
1020
10
Effective Parameter Count
Figure 6: GLM-130B MMLU
Figure 7: BIG-bench-lite evalua- Table 4: Details on BIG-
(57 tasks) along training steps.
tion (24 tasks) across scales.
bench-lite (24 tasks).
5.2 MASSIVE MULTITASK LANGUAGE UNDERSTANDING (MMLU)
MMLU (Hendrycks et al., 2021) is diverse benchmark including 57 multi-choice question an- swering tasks concerning human knowledge ranging from sigh-school-level to expert-level. It is released after the crawling of Pile and serves as an ideal test-bed for LLMs' few-shot learning. The GPT-3 result is adopted from MMLU and BLOOM-I760 176B is tested by using the same prompts as GLM-130B's (Cf. Appendix C.6 and Table for details).
GLM-130B's few-shot (5-shot) performance on MMLU approaches GPT-3 (43.9) after viewing about 300B tokens in Figure 6. It continues moving up as the training proceeds, achieving an accuracy of 44.8 when the training has to end (i.e., viewing 400B tokens in total). This aligns with the observation (Hoffmann et al., 2022) that most existing LLMs are far from adequately trained.
5.3 BEYOND THE IMITATION GAME BENCHMARR (BIG-BENCII)
BIG-bench (Srivastava et al., 2022) benchmarks challenging tasks concerning models' ability on reasoning, knowledge, and commonsense. Given evaluating on its 150 tasks is time-consuming for LLMs. we report the CEG-bench-lite an official 24-task sub-collection for now. Observed from Figure 7 and Table GLM- outperforms GPT-3 175B and even PaLM 540B (4x larger) in zero-shot setting. This is probably owing to GI-0-33'ss bidirectional context attention and MIP, which has been proved improve zero-shot results unseen tasks (Wei et al., 2022a; Sanh al., 2022). As the number of shots increases, GLMM130B'S performance keeps going up, maintaining its outperformance over GPT-3 (Cf. Appendix C.5 and Table for details on each model and task).
Limitations and Discussions, In the experiments above, we observe that GLM-130B' performance growth (13.31 to 15. 12) with the increase of few-shot samples is not as significant as GPT-3's (4.35 to 13.18). Here is our intuitive attempt to understand the phenomenon.
First, the bidirectional nature of GLM- 130B could lead to strong zero-shot performance (as is indi- cated in zero-shot language modeling), thus getting closer to the few-shot "upper-bound" for models of similar scale (i.e., 100B-scale) than unidirectional LLMs. Second, it may be also attributed to a deficit of existing MIP paradigms (Wei et al., 2022a; Sanh et al., 2022), which only involve zero-shot prediction in the training and will be likely to bias GLM-130B for stronger zero-shot learning but relatively weaker in-context few-shot performance. To correct the bias, potential solution we came up with would be to employ MIP with varied shots of in-context samples rather than only zero-shot samples.
Finally, despite almost the same GPT architecture as GPT-3, PaLM 540B's s relative growth with few- shot in-context learning substantially more significant than GPT-3's. We conjecture this further acceleration in performance growth is source of PaLM's high-quality and diverse private-collecte training corpora. By combining our experiences with (Hoffmann al., 2022)'s insights, we came to realize that better architectures, better data, and more training FLOPS should be further invested.
5.4 CHINESE LANGUAGE UNDERSTANDING EVALUATION (CLUE)
We evaluate GLM-130B's Chinese zeoo performance established Chinese NLP benchmarks, CLUE (Xu et al., 2020) and FewCLUE (Xu et al., Notee that we do not include any Chinese downstream tasks in MIP. To date, we have finished testing on part of the two benchmarks, including
8

---

Published as a conference paper at ICLR 2023
Acc. EM
60
28.8
27.5
20/1
27/4
77.5
23.9
77.0
77AL
74.7
21-2
59.0
Titan-2688
40
52.8
515
26.3
46.6
20
29.5
164
EPASTMT
OONLIFC
BUSTM
CHID.FC CLUEWSC-FC
C3
W5C1.1
CMNLI
DRCD
OCNLI50K
AFQMC
CMRC2018
Figure 8: GLM-130B and ERNIE Titan 3.0 260B evaluated on zero-shot CLUE and FewCLUE.
7 CLUE and 5 FewCLUE datasets (Cr. Appendix C.7 for details). We compare GLM-130B to the largest existing Chinese monolingual language model-thh the 260B ERNIE Titan 3.0 (Wang et al., 2021). We follow its setting report zero-shot results dev datasets. GLM- 130B consistently outperforms ERNIE Titan 3.0 across 12 tasks (Cf. Figure 8). Interestingly, GLM-1 1-10 performs at least 260% better than ERNIE due to GLM-130B s pre-training objective that naturally resonates to abstractive MRC's form.
abstractive MRC datasets (DRCD and CMRC2018)) possibly
RELATED WORK
In this section, we review related work to GL.M-100 on topics of pre-training, transferring, and inference of pre-trained LLMs (Qiu et al., 2020; Bommasani al., 2021).
Pre-Training. Vanilla language modeling refers to decoder-only autoregressive models (e.g., GPT (Radford et al., 2018)), but it also recognizes any forms of self-supervised objectives on texts. Recently, transformer- based (Vaswani et al., 2017) language models present a fascinating scaling law: new abilities (Wei et al., 2022b) arise models scale up, from 1.5B (Radford et al., 2019), 10B-scale language models (Raffel et al., 2020; Shoeybi et al., 2019; Black et al., 2022), to 100B- scale GPT-3 (Brown al., 2020). Later, despite many 100B-scale LLMs (Lieber et al., 2021: Thop- pilan et al., 2022; Rae et al., 2021; Smith et al., 2022; Chowdhery et al., 2022: Wu et al., 2021; Zeng al., 2021; Wang et al., 2021) in both English and Chinese, they are not available public only accessible via limited APIs. The closeness of LLMs severely stymies its development. efforts, along with recent ElutherAL, OPT- 175B (Zhang al., 2022), and BLOOM- (Scao al., 2022), aim to offer high-quality open-sourced LLMs to our community.
HLM-130B'S
Transferring. Though fine-tuning has been a de facto way for transfer learning, the evaluation for LLMs has been focused on prompting and in-context learning due to their tremendous sizes (Brown al., 2020; Liu al., 2021a). Nevertheless, some recent attempts has been on parameter-efficien learning language models (Houlsby al., 2019) and prompt tuning (i.e., P-tuning, Li & Liang (2021); Liu et al. (2021b); Lester et (2021); Liu et al. (2022)). For now we do not focus on them and will leave the comprehensive testing of them on GLM-130B in future study.
Inference. Most public-accessihle L.MS nowadays are providing their services via limited APIs. this work, important part of our endeavor has been on LLMs' efficient and fast inference. Related work may include distillation (Sanh et al., 2019; Jiao et al., 2020; Wang et al., 2020), quantiza- tion (Zafrir et al., 2019; Shen et al., 2020; Tao et al., 2022), and pruning (Michel al., 2019; Fan et al., 2019). Very recent work (Dettmers et al., 2022) shows that LLMs such as OPT-175B and BLOOM-176B
be quantized work, we demonstrate GLM's scaling law for INT4 weight quantization, which allows GLM-130B inference on as few as 4 RTX 3090 (24G) GPUs or 8 RTX 2080 Ti (11G) GPUs.
bit due special distribution of outlier dimensions. In this
7 CONCLUSION AND LESSONS
We introduce GLM- bilingual pre-trained language model that aims to facilitate open and inclusive LLM research. GLM-130B* technical and engineering undertakings generate insight into LLMs' architectures, pre-training objectives, training stability and efficiency, and affordable infer- ence. Altogether, contributes to the high quality of GLM-130P in terms of both language perfor- mance 112 tasks and ethical results bias and toxicity benchmarks. Our experiences of both success and failure are condensed into the lessons for training 100B-scale LLMs, attached in the Appendix B. B.10.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This research was supported by Natural Science Foundation of China (NSFC) 61825602, 62276148 and Zhipu.AI. We thank all our collaborators and partners from the Knowledge Engineering Group (KEG), Parallel Architecture & Compiler technology of Mobile, Accelerated, and Networked sys- tems Group (PACMAN), Natural Language Processing Group (THUNLP) at Tsinghua University, and Zhipu.AI.
ETHICS STATEMENT
We hereby acknowledge that all of the co-authors of this work are aware of the provided ICLR Code of Ethics and honor the code of conduct. This work introduces an open-source Large Language Model (LLM), which could be used to generate synthetic text for harmful applications, such tele- marketing fraud, political propaganda, and personal harassment as is discussed in (Weidinger et al., 2021; Sheng et al., 2021; Dev et al., 2021). We do not anticipate any hazardous outputs, especially towards vulnerable and historically disadvantaged groups of peoples, after using the model.
And to better collaborate with our community to prevent and ultimately eliminate the risks techni- cally, we make the following crucial open efforts in this work:
Open-Sourced LLMs for Ethical Risk Study. While some people think that restricting the access of LLMs can prevent such harmful applications, we argue that promoting LLM inclusivity can lead to better defense against potential harms caused by LLMs. Currently, only governments and large corporations can afford the considerable costs of pre- pre-training LLMs. There is no guarantee that organizations having the the substantial financial resources will not do harm using a LLM. Without access to such LLMs, individuals cannot even realize the role of LLMS in the harm.
Conversely, releasing an open LLM can provide access and transparency to all the researchers and promote the research reduce the potential harm of LLMs, like algorithms identify the synthetic text Gehrmann et al. (2019). Also, it is known that LLMS can suffer from problems in fairness, bias, privacy, and truthfulness Zhang al. (2021); Lin et al. (2022); Liang et al. (2021); Bender et al. (2021). An open LLM can reveal the model parameters and internal states corresponding to specific inputs instead of providing APIs black-box models. In conclusion, researchers can conduct analysis of LLMs' flaws in depth and propose improved algorithms to solve the problems.
Ethical Evaluation and Improvements. We also evaluate our model over wide range of English ethical evaluation benchmarks, including bias measurement (Nadeem et al., 2021; Nangia et al., 2020), hate speech detection (Mollas al., 2020), and toxic generation estimation (Gehman et al., 2020). Notwithstanding their deficiency (Blodgett et al., 2021; Jacobs & Wallach, 2021), these datasets serve as a meaningful initial step towards an open quantitative evaluation LLMs.
Our evaluation implies that our algorithm designs, especially the bilingual pre-training of a LLM, can significantly mitigate the biases and toxicity an LLM may present while keeping its strong language performance compared to other LLMs (Brown et al., 2020; Zhang et al., 2022) trained with monolingual English corpora (Cf. Appendix A for more details).
REPRODUCIBILITY
Compared to mainstream closed-sourced LLMs including GPT-3 175B(Brown et al., 2020), PaLM 540B (Chowdhery et al., 2022), Gopher (Rae et al., 2021), Chinchilla (Hoffmann et al., 2022), LaMDA (Thoppilan al., 2022), FLAN (Wei et al., 2022a), and many others, GLMM-130B is open- sourced and devotes to promote openness and inclusivity in LLM research from the very beginning.
We have paid great effort to ensure the reproducibility of our evaluation. For pre-training section, despite the unaffordable costs it needs to reproduce at present, we still make our best efforts to dis- close the code, details, and the whole process of GLM-1 pre-training. Our endeavor to allow GLM-130B inference few popularized GPUs such as 3090/2080 Ti also aligns with the repro- ducibility undertaking, as it allows most academic researchers to reproduce GLM-130B- 's results on their offline machines. We also provide free APIs for individual users to test GLM-130B ability.
10

---

Published as a conference paper at ICLR 2023
Pre-Training. We provide the complete training notes, Tensorboard logs, and code for our pre- training in our repository (Cf. Abstract). The pre-training eyperparaamter and cluster configu- ration are provided in Section 2.3 and Table 11. The training corpora composition and details for Multi-task Instruction Pre-training are provided in Section 2.2 and Appendix C.I and C.2.
Evaluation. We organize all the evaluation, including language benchmarks (LAMBADA, Pile, MMLU, BIG-bench, CLUE, and FewCLUE) and ethical benchmarks (CrowS--Parss, StereoSet, ETHOS, RealToxicPrompts), into one-command-to-rum bash scripts in our code repository. Data processing details for language modeling benchmarks are provided in Section 5. and Appendix C.4, for MMLU are provided in Section 5.2 and Appendix C.6, for BIG-bench are provided in Section 5.3 and Appendix C.5, for CLUE and FewCLUE are provided in 5.4. For all ethical evaluation, please refer Appendix for details.
REFERENCES
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami AI-Rfou. Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguis- fics: Human Language Technologies, pp. 3554-3565, 2021.
Vamsi Aribandi, Yi Tay. Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q Tran, Dara Bahri, Jianmo Ni, al. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference on Learning Representations, 2022.
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victo- ria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasumuru, al. Efficient large scale language modeling with mixtures of experts. arXiv preprint arXiv:2112. 10684, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450. 2016.
Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Ab- heesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Févry, et al. Promptsource: An integrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 93-104, 2022.
Emily M. Bender, Timnit Gebru, Angelina McMillan Majoor, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In FAccT '21: 2021 ACM Con- ference on Fairness, Accountability, and Transparency, Virtual Event / Toronto, Canada, March 3-10, 2021, pp. 610-623. ACM, 2021.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 1533-1544, 2013.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, al. Piqa: Reasoning about physical com- monsense natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Sidney Black, Stella Biderman. Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He. Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-20b: An open-source autoregressive language model. In Proceedings of BigScience Episode # 5-Workshop on Chal- lenges & Perspectives in Creating Large Language Models, pp. 95 136, 2022.
Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, and Hanna Wallach. Stereotyping norwegian salmon: An inventory of pitfalls in fairness benchmark datasets. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1004-1015, 2021.
11

---

Published as a conference paper at ICLR 2023
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXivv2188.07258 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,, 2020.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 6491- 6506. Association for Computational Linguistics, 2021.
Xavier Carreras and Lluís Mårquez. Introduction to the conll-2005 shared task: Semantic role labeling. In CoNLL, pp. 152-164, 2005.
Thiago Castro Ferreira, Claire Gardent, Nikolai Ilinkkh, Chris van der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. The 2020 bilingual, bi-directional WebNLG+ shared task: Overview and evaluation results (WebNLG+ 2020). In Proceedings of the 3rd Inter- national Workshop on Natural Langnage Generation from the Semantic Web (WebNLG+), pp. 55-76, Dublin, Ireland (Virtual), 12 2020. Association for Computational Linguistics. URL https: aaclanthhllgyyyyoc org/ 2020 C. webnlg-1.7.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self- supervupe vision transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp- 9640-9649, 2021.
Ke-Li Chiu and Rohan Alexander.
Detecting hate speech with gpt-3. arXiv preprint
arXiv:2103. 12407, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.088 02311, 2022.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1803.05457 2018
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2978-2988, 2019.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization. arXiv preprint (exxi:22100028000 2021.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers at scale. arXiv preprint urXiv:2208.07339, 2022.
Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, J. M. Phillips, and Kai Wei Chang. Harms of gender exclusivity and challenges in non-binary representation in language technologies. ArXiv, abs/2108. 12084, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4188 2019.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to- eext-to-image generation via transformers. Advances in Neural Information Processing Systems, 34: 1099882222833 2021.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre- pre-traimin training for natural language understanding and generation. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2023
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 02.3200335, 2022.
Ondřej Dušek, David M. Howcroft, and Verena Rieser. Semantic noise matters for neural natural language generation. In Proceedings of the 12th International Conference on Natural Language Generation, pp. 421-426. Tokyo. Japan, Gioober-November 2019. Association for Computa- tional Linguistics. doi: 10. N..1866//1/W11--6663 URL wwwps:ttt//ttc...t. / a www.lanhhhlloooooooooo ogy org/W1 /W1 -9652.
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. T-rex: A large scale alignment of natural language with knowledge base triples. In Proceedings of the Eleventh International Conference Language Resources and Evaluation (LREC 2018), 2018.
Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Kumar Goyal, Peter Ku, and Dilek Hakkani- Multiwoz 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In LREC, 2020.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth demand with structured dropout. arXiv preprint arXiv:1909 1556, 2019.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00022 2020.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Realtoxic- ityprompts: Evaluating Neural Toxic Degeneration in Language Models. dblp://jourals/hbbpp 2020.
Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. GLTR: Statistical detection and vi- sualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 111-116, Florence, Italy, July 2019, As- sociation for Computational Linguistics.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Aremu Anuoluwapo, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna Clinciu, Dipanjan Das, Kaustubh D Dhole, et al. The gem benchmark: Natural language generation, its evaluation and metrics. GEM 2021, pp. 96, 2021.
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant, Did aristotle laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9:346-361, 2021.
Peter Hase, Mona T. Diab, Asli Celikyilmaz, Xian Li, Zornitsa Kozareva, Veselin Stoyanov, Mohit Bansal, and Srinivasan Iyer. Do language models have beliefs? methods for detecting, updating, and visualizing model beliefs. CoRR, abs/21 11.111...6665 13654, 2021.
Ruining Hc, Anirudh Ravula, Bhargav Kanagal, and Joshua Ainslie. Realformer: residual attention. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 929_ 2021.
Transformer likes
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv:1606.08415, 2016.
arXiv preprint
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob
Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza
Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
13

---

Published as a conference paper at ICLR 2023
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Lcrge-scale pre- training for text-to-video generation via transformers. arXiv preprint TrXiv:2205.18805 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hyoukloong Lec, Jiquan Ngiam, Quoc V Le, Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32, 2019.
Abigail Z Jacobs and Hanna Wallach. Measurement and fairness. In Proceedings the conference on fairness, accountability, and transparency, pp. 375-385, 2021.
the 2021 ACM
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1166341744 2020.
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zeutlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1601-1611, 2017.
Paul R Kingsbury and Martha Palmer. From treebank propbank. Citeseer.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Lingnistics, 7:453-466, 2019.
Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. Quantifying the carbon emissions of machine learning. CoRR, abs/1910.09700, 2019.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-effiiien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3045-3059, 2021.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. Thir- leenth international conference on the principles of knowledge representation and reasoning. 2012.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers),
4582-4597, 2021.
Xiangyang Li, Yu Xia, Xiang Long, Zheng Li, and Sujian Li. Exploring text-transformers in aaai 2021 shared task: Covid-19 fake news detection in english. In CONSTRAINT@@AAA, 2021.
Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. Towards under- standing and mitigating social biases in language models. In Proceedings of the 38th Interna- tional Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Leurning Research, pp. 6565-6576. PMLR, 2021.
Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. Jurassic-1: Technical details and evalua- tion. White Paper. A121 Labs, 2021.
Chin-Yes Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004, Association for Computational Linguis- tics. URL https: //t////aaa//ttaaa loy org wrgg//0000000 1
14

---

Published as a conference paper at ICLR 2023
Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252 Dublin, Ireland, May 2022. Association for Computational Linguistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv22107.35880 2021a.
Peter Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. Generating wikipedia by summarizing long sequences. In International Conference Learning Representations, 2018.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint arXiv:2103 2021b.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable fine-tuning across scales and tasks. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? Advances in neural information processing systems, 32, 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct elec- tricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018.
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. Memory- based model editing scale. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 15817-15831. PMLR, 2022.
Ioannis Mollas, Zoe Chrysopoulou, Stamatis Karlos, and Grigorios Tsoumakas. Ethos: an online hate speech detection dataset. arXiv preprint arXiv:2006. 2022200000822,, 2020.
Moin Nadeem, Anna Bethke, and Siva Reddy. Stereoset: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume Long Papers), pp. 5356-5371, 2021.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman. Crows-pairs: A challenge dataset for measuring social biases masked language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1953-1967, 2020.
Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memory-efficient Hipeline-paraller don training. In International Conference on Machine Learning, pp. 7937-7947. PMLR, 2021.
Tomoko Ohta, Yuka Tateisi, and Jin-Dong Kim. The genia corpus: An annotated research abstract corpus in molecular biology domain. In HLT, pp. 82-86, 2002.
15

---

Published as a conference paper at ICLR 2023
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoe- Quae--Qaan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 1525-1534, 2016.
David A. Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. CoRR, abs/2104. 10350, 2021.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. Towards robust linguistic analysis using ontonotes. In CoNLL, pp. 152, 2013.
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: input length extrapolation. In International Conference on Learning Representations, 2021.
Attention with linear biases enables
Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. Learning compact metrics for MT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 751-762. Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 1211865/1/2021.mmmllp https: / /acl antho ogy. org/ 2021 emnlp-ma 58.
URL
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre-trained models for natural language processing: survey. Science China Technological Sciences, 63(10): 1872-1897, 2020.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing with unsupervised learning. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXi::21 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning. pp.
PMLR, 2021.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 3505 3506, 2020.
Sebastian Riedel, Limin Yao, and Andrew McCallum. Modeling relations and their mentions with- out labeled text. In ECML-PKDD, pp. 148-163, 2010.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 5418-5426, 2020.
Dan Roth and Wen-tau Yih. A linear programming formulation for global inference in natural language tasks. In HLT-NAACI, pp. 1-8, 2004.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In NAACI-HIT (2), 2018.
16

---

Published as a conference paper at ICLR 2023
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo-opes, Burcu Karagol Ayan, Tim Salimans, et al. Pho- torealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Information Processing Systems.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction the con1l-2003 shared task: Language- independent named entity recognition. In HLT-NAACL, pp. 142-147, 2003.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint (eXiiv/9900.00 108, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representa- fions, 2022.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint 2rXiv:2211.05100 2022.
Timo Schick, Sahana Udupa, and Hinrich Schütze. Sell-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp. Transactions of the Association for Computational Linguistics, 9:1408-1424, 2021.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. MLSUM: The multilingual summarization corpus. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8051 3051- Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 10.18866/v12200 wttp:///v2222.ccom wttp:/00...mmmmmmaam mainl--m.....77.. URL https: /awaaaaannnnntoyoaaaaaaaaaaaolaa org/ 2020 amanp-mainn...mmmmmmmmmmammmmm 6.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8815-8821, 2020.
Emily Sheng, Kai-Wei Chang, P. Natarajan, and Nanyun Peng. Societal biases in language genera- tion: Progress and challenges. In ACI., 2021.
Sam Shleifer, Jason Weston, and Myle Ott. Normformer: Improved transformer pretraining with extra normalization. arXiv preprint arXiv:21 2021.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model par- allelism. arXiv preprint 1xXiv:1909 htt.::/090000033... 2019.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. Using deep- speed and megatron to train megatron-turing nlg 530b, large-scale generative language model. arXiv preprint arXiv:2201.119900 2022.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Barriga-Alonse al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.044 046615 2022.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp- 3645-3651 Association for Computational Linguistics, 2019.
Jianlin Su. Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864 2021.
17

---

Published as a conference paper at ICLR 2023
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensequa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4149-4158, 2019.
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. 4821 4836, 2022.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201 S.Xiv:2201.0823:: 082399, 2022.
Denis Timonin, Bo Yang Hsueh, and Vinh Nguyen. Accelerated inference for large transformer models using nvidia triton inference server. NVIDIA blog. 2022.
Leslie G Valiant, A bridging model for parallel computation. Communications of the ACM, 33(8): 103-111, 1990.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
David Wadden, Ulme Wennberg, Yi Luan, and Hannaneh Hajishirzi. Entity, relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP),, pp. 5784 5789, 2019.
C. Walker and Linguistic Data Consortium. ACE 2005 Multilingual Training Corpus. Linguistic Data Consortium, 2005. ISBN 9781585633760.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. SuperGLUE: A Stickier Benchmark for General-Purpose Lan- guage Understanding Systems. In NeurlPS 2019, pp. 3261-3275, 2019.
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model. https:: /ttp://ggtttttttt www.iil.cooo/iiiigggg/e
z www.sshttaasfoooooo jax, May 2021.
Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. Deepstruct: Pretraining of language models for structure prediction. In Findings of the Association for Com- putational Linguistics: ACL 2022. pp. 803-823, 2022a.
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. Deepnet: Scaling transformers 1,000 layers. arXiv preprint 2rXiv:2003.00555 2022b.
Shuohuan Wang, Yu Sun, Yang Xiang, Zhihua Wu, Siyu Ding, Weibao Gong, Shikun Feng, Jun- yuan Shang, Yanbin Zhao, Chao Pang, al. Ernie 3.0 titan: Exploring larger-scale knowledge en- hanced pre-training for language understanding and generation. arXiv preprint arXiv:2 /2. 12731, 2021.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neu- ral Information Processing Systems, 33:5776-5788, 2020.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Rationale- augmented ensembles in language models. arXiv preprint urXiv:2207.00777 2022c
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu. Brian Lester, Nan Du, An- drew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference Learning Representations, 2022a.
18

---

Published as a conference paper at ICLR 2023
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.00 07682, 2022b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022c.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprini urXiv:2112.04359, 2021.
Shaohua Wu, Xudong Zhao, Tong Yu, Rongguo Zhang, Chong Shen, Hongli Liu, Feng Li, Hong Zhu, Jiangang Luo, Liang Xu, al. Yuan 1.0: Large-scale pre-trained language model in zero- shot and few-shot learning, arXiv preprint arXiv:2110.04755, 2021.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251 12226, 2018.
Ruibin Xiong. Yunchang Yang, Di He, Kai Zheng. Shuxin Zheng, Chen Xing. Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533 PMLR, 2020.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao,, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. Clue: A chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics. pp. 4762-4772, 2020.
I Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Huilin Xu, Hu Yuan, Guoao Wei, Xiang Pan, Xin Tian, Libo Qin, et al. Fewclue: A chinese few-shot learning evaluation benchmark. arXiv preprint Stti:2200..04998, 2021.
Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding. Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang. and Jie Tang. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. Al Open, 2:65-68, 2021.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-Neurlle Edition (EMC2-NIPS), pp. 36-39. IEEF, 2019.
Wei Zeng. Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang. Xin Jiang, ZhenZhang Yang. Kaisheng Wang. Xiaoda Zhang, et al. Pangu-'a: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint arXiv:2104.12369, 2021.
Chiyuan Zhang, Daphne Ippolito. Katherine Lee, Matthew Jagielski, Florian Tramèr, and Nicholas Carlini. Counterfactual memorization in neural language models. CoRR, abs/21 129388 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position- aware attention and supervised data improve slot filling. In EMNLP, pp. 35-45, 2017.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. "going on a vacation" takes longer than "going for walk": A study of temporal commonsense understanding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3363-3369, 2019.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. Modifying memories in transformer models. CoRR, abs/2012.00366 2020.
19

---

Published as a conference paper at ICLR 2023
Part I
Appendix
Table of Contents
Ethics: Evaluation on Biases and Toxicity
21
Bias Measurement: CrowS-Pait
21
A.2 Bias Mcasurement: StercoSet
.re.c...........................
21
A.3 Hate Speech Detection: ETHOS
22
A.4 Toxic Genearation: Real PealToxiiProopts
22
B Technical Details
23
B.1
Tokenization
23
B.2
Layer Normalization
24
B.3
Positional Encoding and
Network
24
B.4
Pipeline Parallel Analysis
25
B.5
Inference Acceleration
....aaiioo......................
27
B.6
Activation Outlier Analysis
................................
27
B.7
Weight Quantization
28
Quantization settings
28
Ablation on Contribution Attribution
29
B.10 Lessons Learned
30
C Dataset and Evaluation Details
32
C.I Multi-task Instruction Pre-training (MIP)
32
C.2 Data and prompts in MIP for DeepStruct
32
C,3 Result Sources for GPT-3, BLO0MM-166, and OPT-175B
39
C.4 Pile Test-set Evaluation
39
C.5 BIG-bench-lic lite Evaluation
40
C.6 MMLU Evaluation
40
C.7 Chinese Language Understanding Evaluation
40
C.8 Natural Language Generation
41
C.9 Winograd Stynld Tasks
43
C.10 Closed-book Question Answering
43
C.I1 Commonsense Reasoning
C.12 Fixed Label Datasets: Case Study in Natural Language Inference
44
C.13 SuperGLUE
44
C.14 Chain-of-Thouuee Prompting
45
D Scaling and Emergent Abilities in GLM-130B
46
E
Contributions
52
E.1
Preparation
52
E.2
Model Training
52
E.3
Post Training
52
E.4
Project Management.
52
E.5
Computation Sponsor
52
Brief History GLM-133B
Broader Impact
55
G.1 Impact on Al Research
55
G.2 Impact Individual Developers and Small Companies
55
20

---

Published as a conference paper at ICLR 2023
G.3 Social Impact
55
H Environmental Impact
56
A ETHICS: EVALUATION ON BIASES AND TOXICITY
Albeit LLMs' strong abilities in language and beyond, which could bring substantial welfare to human beings, they can potentially produce toxic and illegal contents for evil use (Weidinger al., 2021: Sheng et al., 2021; Dev et al., 2021; Bommasani et al., 2021). In GLM-130B, before granting model weight to applicants. in the model license we demand them agree that they will use for any deeds that may be harmful to society and human beings.
Additionally, from a technical perspective, we argue that we must also understand LLMs' toxic and biased behaviors and ultimately eliminate them. This aligns with our commitment to "LLM Inclusivity", as is necessary to include more people the open-sourced LLM research to facilitate the process. Moreover, if an LLM is shown to be good at identifying toxic and biased content, techniques such as self-diagnoses (Schick al., 2021) can help to reduce the harmful generation in a self-consistent post-processing procedure. Therefore, 130B over a variety of related benchmarks to shed light on the challenging topic. Despite their limitations (Blodgett et al., 2021; Jacobs & Wallach, 2021) which should be addressed in future work, they still serve as a good start to arouse the community's 's awareness of the problem.
initial step, evaluate GLM-
A.1 BIAS MEASUREMENT: CROWS-PAIRS
Table 5: CrowS-Pain (Nangia al., 2020) Bias Measurement. The lower scores the better.
CrowS-Pairs (Nangia al., 2020), or namely Crowdsourced Stereotype Pairs benchmark, is widely used for measuring biases for masked language models. It collects 1508 examples with nine different conventional biases and adopts a probing-based approach to compare the pseudo- log-likeling log-likelihood of pair of stereotypical and anti- stereotypical sentences. Since GLM-130B is pre-trained with autoregressive blanking infill- ing, CrowS-Pairs evaluation is directly appli- cable. We compare the GPT-3 Davinci and OPT-175B's results CrowS-Pairs reported in (Zhang et al., 2022) with GLM-130B.
Category
GPT-3 OPT-175B GL.M1330 1300
Gender
62.6
65.7
55.7
Religion
73.3
68.6
73.3
Race/Color
64.7
68.6
58.5
Sexual orientation
76.2
78.6
60.7
Age
64.4
67.8
63.2
Nationality
61.6
62.9
64.1
Disability
76.7
76.7
71.6
Physical appearance
74.6
76.2
74.6
Sociocconomic status
73.8
76.2
70.9
Overall
67.2
69.5
65.8
Our results presented in Table 5. GL.M-130B shows fewer biases on almost all kinds of stereo- types except for religion and nationality. We speculate that it is because GLM-130B is bilingual pre-trained LLM that learns the semantics for certain content from both English and Chinese cor- pora. Since CrowsS-Pain Commission's 's list?, the bias distributions in two different cultures and languages may be different and consequently reconcile social biases in GLM-130B on benchmark originally designed for English- society. We think this is an interesting finding, as multi-lingual pre-training may help LLMs to present less harmful biases for better fairness. Finally, also admit that GLM- 130B may in turn presents some special Chinese biases which currently lack testing benchmarks and require considerable future efforts to detect and prevent.
stereotypes mainly draw from the US Equal Employment Opportunities
A.2 BIAS MEASUREMENT: STEREOSET
Another widely used bias and stereotype evaluation benchmark is StereoSet (Nadeem et al., 2021), which is also adopted in (Lieber et al.. 2021; Artetxe et al., 2021; Zhang al., 2022). To balance the evaluation between bias detecting and language modeling quality. StereoSet reports a series of metries including Language Modeling Scores (LMM),, Stereotype Score (SS), and Idealized Context Association Test Score (ICAT) as an overall averaged metric. For example, given the premise "She
httpps: htttpss:/ww.eee.. wttp:://wwwwwwweg///////ee bi ted- wed-empldeeeeeeeeeeeeeeeeeeeee-e 1mmployeet-epmm wiw.eiieeptciiiccccc
21

---

Published as a conference paper at ICLR 2023
is the twin's mother". StereoSet provides three candidate hypothesis: 1) "the water is deep", 2) "she is a lazy, unkind person", and 3) "she is a kind, caring woman". The first option servers as a distractor to test models' language capability and calculate LMS; the second and third statements are anti-stereotypical and stereotypical respectively and used for calculating SS. A widely-adopted technique here is calibrate the likelihood of an option according to its length (Lieber al., 2021; Zhang et al., 2022), as the distractor term is particularly short.
Following (Zhang al., 2022), we normalize scores over tokens rather than characters (Lieber al., 2021) to yield model predictions for calculating the metrics. The results are shown in Table 6. As we observe, GLM-130B exceedingly outperforms GPT-3 Davinci and OPT-175B on all metrics. Such results accurately align with our discoveries in language modeling experiments and CrowS-Paan bias evaluation, that GLM-130B has a high quality in both language modeling and social fairness. Table 6: StereoSet (Nadeem et al., 2021) Bias Measurement with LMS (1), SS (J), and ICAT (1).
Profession
Gender
Religion
Race
Overall
Category
LMS
SS
ICAT
LMS
SS
ICAT
LMS
ICAT
LMS
SS
ICAT
LMS
SS
ICAT
GPT-3
78.4
63.4
57.5
75.6
66.5
50.6
80.8
59.0
66.3
77.0
57.4
65.7
77.6
60.8
60.8
OPT-175B
74.1
62.6
55.4
74.0
63.6
53.8
84.0
59.0
68.9
74.9
56.8
64.8
74.8
59.9
60.0
GLM-130B
86.5
59.6
69.9
83.9
63.5
61.2
91.0
53.5
84.6
85.7
54.1
78.7
86.0
57.3
73.5
A.3 HATE SPEECH DETECTION: ETHOS
Social media corpus may contain hate speeches, and to investigate to what extent LLMs know and help to identify them is crucial. We adopt the ETHOS dataset originally proposed in (Mollas et al., 2020) to detect sexism and racism speech on zero-shot or few-shot datasets created by (Chiu & Alexander, 2021). GPT-3 Davinei (a public-accessible variant of GPT-3 175B) and OPT 175B are also tested on the benchmark (whose results are reported in (Zhang et al., 2022)). For binary clas- sification including Zero-shot, One-shot, and Few-shot (binary) (which answers "yes" or "no"), we report binary Fl; for multiclass classification (which answers "yes", "no", or "neither"), we report micro FI. We adopt almost the same prompts as (Chiu Alexander, 2021), except aligning the Few-shot (binary) prompt the form used in One-shot and adding the word "Classi ciattiiiiiiii iinn before the colon in the original Few-shot (multiclass) prompt.
Results are shown in Table 7. We find that GLM-130B outperforms two other LLMs among four different settings. On one hand, GLM- 130B's pre-training over unsupervised diverse corpora from online forums and social media in- cluding sections such as "hackernews", "stack- exchange", and "pile oc" can endow our model with the background knowledge to identify those speeches. On the other hand, the MIP training may also improve GLM-130B's zero-shot and few-shot capabilities.
Table 7: ETHOS (Mollas et al., 2020) Hate speech detection. "(bi)" and "(mul)" denote bi- nary and multiclass classification respectively. All scores are Fl and the higher the better.
GPT-3
OPT-175B
GLMM-13B
Zero-shot One-shot
62.8
66.7
68.8
61.6
71.3
79.1
Few-shot (bi)
35.4
75.9
79.7
Few-shot (mul)
67.2
81.2
85.8
A.4 TOxIC GENEARATION: REALTOXICPROMPTS
GLM--1508
halds mobabity conerusstonn
0,23
GPT-1 Devincl
Evaluating the toxicity of generation by given prompts an important part of a model's safe deployment. We
0.20
evaluate the toxic generation of GLM-130B on the Re- alToxicsrompts (Gehman et al., 2020) dataset. Fol- lowing its settings, we use nucleus sampling (p- = 0.9)
2.15
0.30
to generate 25 continuations for each of the 10K ran- dom sampled prompts, limiting the maximum gener- ated length to 128 tokens. Then we report the mean toxicity probabilities of 25 continuations evaluated by Perspective API3. In order to make a fair comparison
8.05
0.0
0.2
0.4
0.6
0.8
1.0
Prompt Texicht Probability (Blnned)
Figure 9:
RealToxicPrompps (Gehman al., 2020) evaluation. Lower continua- tion toxicity probability is better.
Phttps:/ www.pperspecte.. veapi ccm/
22

---

Published as a conference paper at ICLR 2023
Empirical Learning
Im-oss--raining/in tokens
In-loss-traininglm loss vs tokens
1.0e-4
Re-load and adjust the learnng rate after collapse
7.5
Learning Rate
0.80
6.5
the reSt 1048
0.6e-4
5.5
0.4e4
3.5
1768
2.5
0.0c-4
(a) OPT 175B's s experiments
(b) BLOOM 176B's experiments
Ui Iriics Enice-tsreginiie
-500
230
2.56
3/2
(c) GLM 130B's experiments
(d) GLM 130B's real training
Figure 10: Handling training collapses and instability is the first priority when training LLMs.
under different tokenization methods, we only report the toxicity score of the first complete sentence of continuation as found that the score returned by the Perspective API seems increase with sentence length.
Results are shown in Figure Generally, as the toxicity of the given prompt increases, the toxicity probability of the continuation increases accordingly in both models. Compared to GPT-3 Davinci, GLM-130B has lower toxicity rate in all cases, indicating that GLM-130B is less prone to gener- ating toxic content.
B TECHNICAL DETAILS
In this section, we introduce additional details about the technical issues we have identified and solved throughout the GLM-130B training. Along with concurrent open-source LLM efforts, we believe that those published details could serve as great cornerstones to future LLM training.
B. TOKENIZATION
For the tokenization of the corpus, we implement text tokenizer based on the package icetk with several adjustments. As an image-text unified tokenizer, the vocabulary size of icetk is 150000. The first 20000 tokens are image tokens and the rest are text tokens. The text tokenizer of icetk is formulated and trained by sentencepiece', English and Chinese contents. We divide tokens recognized by the tokenizer into four categories. The common tokens assigned from 20.20000 to No. consisting of punctuations, numbers and spaces free of extended definition. No.20100 to No.83822 are English tokens and No.83823 to
25GB bilingual corpus equally distributed with
23

---

Published as a conference paper at ICLR 2023
No.14565: Chinese tokens. Tokens after 145655 other special tokens including concate- nated punctuations and pieces from other languages, etc.
During our implementation, We ignore the first 20000 image tokens and simply utilize the latter 130000 intended for text tokenization. we disable the ignoring of linebreak to tokenize the line- break mark An into No. 20004 token <n> On the basis of inherent tokens, add special tokens [MASK] and [gMASK] for model prediction. We also add special tokens <sop>, <eop>, <eos> for sentence and passage separation.
B.2 LAYER NORMALIZATION
Here we briefly introduce the history of layer normalization in language modeling problems, and how its variants perform in recent LLMs including our experiments for them on GLM-130B.
Post-LN (Vaswani et al., 2017). Post-LN is jointly proposed with the transformer architecture and is placed between the residual blocks. It then adopted by BERT (Devlin et al., 2019) for bidirectional language model pre-training. Nevertheless, Post-LN was later accused of transformers' slow and vulnerable converging (Xiong et al., 2020) and the Pre-LN emerged as a substitute.
Pre-LN (Xiong et al., 2020). On the contrary, Pre-LN is located in the residual blocks reduce exploding gradients and becomes dominant in existing language models, including recent LLMs. However, OPT- 175B (Zhang al., 2022), BLOOM (Scao al., 2022), and rext-to-image model CogView Ding et al. (2021) later observe that Pre-LN still unable to handle the vulnerable training when models scale up 100B meet multi-modal data. This is also justified in GLM-130B preliminary experiments, where Pre-LN consistently crashes in its early stage training.
Additionally, another problem rooted in Pre-LN transformers that it may harm the model perfor- mance after tuning compared to Post-LN. This is observed in (He et al., 2021).
Sandwich-LN (Ding et al., 2021). As remedy, top of Pre-LN, CogView (later in Norm- former (Shleifer et al., 2021)) develops Sandwich-LN which appends extra normalization to the end of each residual branch. Accompanied with PB-Relax Prrcision-Bottlennee niques, they stabilize the training of 4-billion ext-to-image generation model. Despite its superi- ority over Pre-LN, sadly Sandwich-LNy is also proved collapse in GLM-130B training: let alone the potential consequent weaker tuning performance caused by its Pre-LN nature.
Relaxation) tech-
B.3 POSITIONAL. ENCODING AND FEED-FORWARD NETWORK
Positional Encoding Vanilla transformer adopts absolute (or sinuous) position encoding, and is later evolved into relative positional encoding (Dai et al.. 2019). Relative PEs can capture word relevance better than absolute positional encoding. Rotary Positional Embedding (RoPE) (Su et al., 2021) is relative position encoding implemented in the form of absolute position encoding, and its core idea is shown in the following equation.
RRmg)) (Rak)
R R,, = 9 ST,,, Rm-mk
(1)
The product of at position m and & at position n is related to their distance n - m, which reflects the relativity of the position encoding. The definition of R. in the above equation is
m01 sin m81
sin m8;
0
0
cos m6,
0
0
m02
sin mQ
0
Rd
0
0
sin m022
cos m0,
(2)
0
cos m0a/2
sin m0ay2
0
0
0
sin m0a/2
cos m0a/2
To allow its value to decay as the distance increases, 0 takes the value
-26-11
0 = 0, = 10000
i E 1,2, ...
(3)
24

---

Published as a conference paper at ICLR 2023
A wwo-dimensional absolute position encoding method is proposed in vanilla GLM for modeling both intra- and inter-span position information. In GLM-130B, different from the two-dimensional positional encoding used in vanilla GLM, we turn back to conventional mee-imensional positional encoding. However, we originally thought that two-dimensional form cannot be directly applied to RoPES. As substitute plan, in GL.M-1303 130B we simply remove the second dimension used in the original GLM as we find that the unidirectional attention mask sub-matrices for [MASK] generation indicate the token order as well. This observation results in our transforming GLM-130B positional encoding into one me--imensioonl one according to the following strategies:
For sequences corrupted by short spans, we discard the recond-dimensional position encoding. For sequences corrupted by a long span at the end, we change the positional ids to one-dimensional 0, 1,.... 5 1, and generated tokens will just prolong the first-dimensional positional encoding from the last context token $ - 1.
Feed-forward Network Some recent efforts to improve transformer architecture have been on the FFN. including replacing it with GLU (adopted in PaLM). Research shows that using GLU can improve model performance, which is consistent with our experimental results (Cr. Table 8). Specifically, we use GLU with the GeLU (Hendrycks & Gimpel, 2016) activation. as
FFNGeGLU (z: Wi, V, W2) = (GeI.U(x:11) @ vV W₂
(4)
In order to keep the same parameter as the vanilla FFN, the feed-forward size ditn (which is usually 4dh, where du is the hidden dimension) is reduced to hd the V is additionally introduced.
Ablation Study on PE and FFN In order to validate our PE and FFN choices, test them our experiments by pre-training GLMBase (110M) over random 50G Chinese and English mixed corpus. We compare absolute PE with two recent popular relative PE variants, RoPE (Chowdhery et al., 2022) and ALiBi (Press al., 2021). For FFN, we compare vanilla FFN with Gate Lin- ear Unit with GeLU activations. Results from Table 8 show that both ALiBi and RoPE improve perplexity on the test set, and the improvement is more significant with RoPE while using GeGLU can further improve the model's performance.
Table Ablation Study for PE and FFN on GLMBase
Model
Test PPL
GLMBas
24.58
ALiBi
24.14
+ROPE
22.95
+ RRoPE GeGLU
22.31
B.4 PIPELINE PARALLEL ANALYSIS
In pipeline parallelism, each stage consists of three operations (Cf. Figure 11(a)): forward (de- noted as F), backward (denoted as B), and optimizer step (denoted as U). However, naive sequential pipeline implementation leads to an unbearable amount of bubbles. The improved Gpipe (Huang et al., 2019) (Cf. Figure 11(b)) strategy reduces bubbles drastically via splitting data into micro- batches; the more micro-batches there are, the more stages can compute simultancously in an itera- tion. The recent PipeDream-FFuu Flush (Narayanan et al., 2021) (Cf. Figure (c)) additionally optimizes the GPU memory usage by interweaving forward and backward from different stages to reduce for- ward activation's memory occupation.
We analyze the bubble share in GLM- ' pre-training by assuming that the number of pipeline segments is p, the number of micro-batches is m, and the time for forward and backward per micro- batch are ty and tg. In ideal case, forward and backward take tadeal = m(t tt.). But in practice, the default pipeline delivery strategy causes p forward propagation and p 1 backward propagation bubbles, respectively, for a total time of /bubble = (p - () + 4b), so that the bubble occupancy is
tbubble
p
bubble-ratio
(5)
lideal + baubble
m
For larger numbers of micro-batches, the bubble percentage will be reduced to acceptable level. In particular, experiments in GPipe Huang et al. (2019) show that when m 2 4p, the total percentage
'e later found the instructions to implement two-dimensional RoPE from its author's blog hotps: //kexue. fm/ arrchiises... 8397, but our training has proceeded for weeks.
25

---

Published as a conference paper at ICLR 2023
Forward
Backward
Optimizer Stop
Fo
U,
U
GPU 2
Fb
B,
Up
U
Fo
Up
U
Bubble time
GPU 0
Bg
Up
U
Time
(a) Naive pipeline implementation, which can be extremely inefficient.
Forward
Backward
Optimizer Step
GPU 3
F
F2
Fg
F
F
Fg
Fy
Bg
B,
8,
B;
8,
Bg
B,
U:
GPU
F
F
Fg
F4
F
F
Bg
B,
8
8g
8,
B,
By
U
F
F
B
8,
B
B
Ug
G
F
F
B
B
B
B
U
Time
(b) GPipe (Huang al., 2019) implementation.
Forward
Backward
Optimizer
GPU 3
Fn
Bg
B,
F,
F2
8,
F,
B
F6
B
Fg
B,
GPU
Fo
F,
F2
Fg
B,
B,
B2
F4
8
F2
B,
F6
B,
F,
8,
8,
Ug
GPU
F2
5,
B,
G
F.
B
F,
F
B
Fo
B
B,
6,
Time
(c) Pipedream (Narayanan et al., 2021) implementation (used in GLM-130B).
Figure 11: Different pipeline strategies and their conceptual comparison.
of pipeline bubble time is reduced to negligible level due to the forward recomputation technique in backpropagation that allows some overlap in computational communication, thus showing that the bubbles introduced in parallel by the pipeline model do not seriously deplete the training efficiency.
In general, in order to make full use of the hardware, parallel groups consisting of multiple nodes and try to the full memory of each node. In this case, we can freely adjust the ratio of pipeline model parallelism and tensor model parallelism. Since data parallelism hardly affects the computation time. assume that the scale of data parallelism is = 1, the total number nodes is n, the scale of tensor model parallelism f, and the scale of pipeline model parallelism is p. and satisfies t p. the bubble share in this case is
common to place models into model
n/l m + n/t
bubble-ratio
(6)
From the above equation, we can see that increasing the size of tensor parallelism will further reduce the bubble ratio. However, the tensor parallelism scale cannot be increased indefinitely, which would lead to reduction in computational granularity and greatly increase the communication cost across a certain threshold. Therefore, we can conclude that the size of tensor model parallelism should increase slowly as the model size increases, but not more than the number of graphics cards in a single machine. In the training of GLM- MLMM-10B, the experiments show that the optimal tensor parallelism scale is 4 and does not scale up to the scale of 8 in the DGX-A100 system. The other parameters are m = 176, p = 8, and the bubble share is calculated to be only 3,8%, which is sufficient demonstrate the efficiency of pipeline model parallelism.
26

---

Published as a conference paper at ICLR 2023
Table 9: Decoding speed in our real trials between BLOOM-176B (Scao et al., 2022) (from Hug- gingface Transformers) and GLM- 1-100''ss implementation in 16-bit precision with 8 x A100 (80G).
Decode Tokens
128
512
1024
2048
BLOOM-1700 76B
36,76s
137.91s
287.938 39.81s (x7.2)
631.81s 89.88s (x7.0)
GLM 130B
4.40s (x8,4)
18.77s (x7.3)
3500
sarted hodde state dimensions
3500
12
bidden stats cinvesiors
3000
2000
2500
2500
2000
2000
1500
1500
1000
1000
500
500
0-
300
200
500
100
200
500
position the
position the
Figure 12: Distribution outliers in GLM-130B's activations. The vertical axis denotes the hidden state dimensions (4,096 rather than 12.288 this is a parallel segment), and the horizontal denotes tokens in input sentence. Using 128x 128 2D histogram to get a better view the distribution of outliers. The figure the right swaps some of the vertical coordinates that it can be clearly seen that the outlier occur about 30% of its dimensions.
B.5 INFERENCE ACCELERATION
A model's plain PyTorch implementation is easy to read and run, but it can be intolerably slow for LLMs. Based on NVIDIA's FasterTransformer we spend two months implementing GLM-130B into C++ to speed up inference, including the following main optimizations:
Optimize time-costing operations such as GeGLU, Layer Normalization, and SoftMax.
Reduce the number GPU kernel calls (e.g., fuse Multihead/Attentonn into one computation ker-
nel).
Specify the algorithm of the best performance when calling cuBLAS.
Improve the computing efficiency by transposing the model parameters in advance.
Use half2 in FP16 computation to double the half's access bandwidth and computing throughput.
We currently pack up the full FasterTransformer implementation for GLM-130B into a plug-and- play docker image for users' convenience, and we are still working on adapting it to our Pytorch implementation by only changing one line of code. A comparison between our speeding up GLM- 130B implementation and the so far default available BLOOM- 176B implementation in Hugging- face Transformers is shown in Table 9. Our implementation for GLM-130B can be 7.0 8.4 times faster than BLOOM- Pytorch implementation. The exertion to accelerate LLM for tolerable response speed could be extremely crucial to popularization.
B.6 ACTIVATION OUTLIER ANALYSIS
As is described in prior sections, GLM-130B's weight can be quantized into INT4 to drastically cut down parameter redundancy in the inference. However, also find that GLMM-130B' activations (i.e., hidden states between layers) cannot be properly quantized, as they contain value outliers as is also suggested in concurrent literature (Dettmers et al., 2022).
What is special in GLM-130B is that 30% of its dimensions may present value outliers (Cf. Fig- 12), while other GPT-based LLMs (e.g., OPT-175B and BLOOM 176B) only has very few outlying dimensions (Dettmers et al., 2022). Therefore, the solution to decompose matrix multipli-
27

---

Published as a conference paper at ICLR 2023
cation for Sigher-precision computation in outlying dimensions proposed in (Dettmers et al., 2022) not applicable to GLM-130B.
We study whether these outliers be ignored in LLM quantiza- tion, and the answer is interestingly "no". These values can be sev- eral orders of magnitude larger than ordinary activation values (Cf. Figure 13). While most values (accounts for 99.98% dimensions in hidden state) stay less them 6, those outlying dimensions can reach 50 or even over 100. They are speculated to be some important clues for GLM-1330 and potentially other LLMs to memorize some fixed world or language knowledge, and thus removing or omitting them in quantization can lead significant performance degradation.
Figure 13: GLM-130B's activation outliers' absolute value scale.
B.7 WEIGHT QUANTIZATION
B.7.1 PRELIMINARIES
Absmax Quantization is symmetric quantization that a range of (-absmax(1)) absmax(2)) is mapped to [-(2h 1), g for
absmax(x)
S 2b
(7)
= round(r/sa)
(8)
where 8, is the scaling factor, 3ig is the quantization result and b is the bit width.
Zeropoint Quantization is an asymmetric quantization that range of [min(x), max(x)] is mapped (2 1), 2 1].
82 ma) min(x)
(9)
2 - 2
2- round(min (z)) 82) +2--1
(10)
Tq round(I/s..
(11)
where is the zero point.
Col/Row-wise Quantization Using single scaling factor for the weight matrix often leads to more quantization errors because one single outlier leads to decrease in the quantization precision of all other elements. A common workaround is group the weight matrix by rows by columns, with each group being quantized separately and having independent scaling factors.
B.8 QUANTIZATION SETTINGS
Our goal is to save GPU memory as much as possible without hurting model performance. prac- tice, we only quantize linear layers, which take up most of the transformer parameters, and leave input/output embedding, layer normalization, and bias terms unchanged. At the quantization pre- cision of INT4, two INT4 weights are compressed into one INT8 weight for saving GPU memory usage. Absmax quantization is adopted since we found enough to maintain model performance, and it is more computationally efficient than zeropoint quantization. During inference, only quan- tized weights are stored in GPU memory, the FP16 weights for linear layers will be dequantized at runtime.
B.8.1 QUANTIZATION RESULTS SCALES
GLM models at 10M to 1OB scale are from GLM's original paper(Du et al., 2022). Although the architecture of smaller scale GLMs are not the same GLM-130B, we believe that the training objective is the key factor for quantization. Table 10 shows the performance of GLM and BLOOM family models different scales the LAMBADA dataset with different quantization methods. Almost all models maintain performance at INT8 precision. In general, GLM maintains better performance than BLOOM INT4 precision as it scales.
28

---

Published as a conference paper at ICLR 2023
Table 10: Accuracy on LAMBADA dataset for GLM and BLOOM family at 100M 176B scales across different quantization precision.
BLOOM-560M BLOOM-IBI BLOOM-3B BLOOM-7B BLOOM-176B
Original
31.40%
40.68%
48.30%
54.91%
64.37%
Absmax INT8, col-wise
26, 12%
40.69%
48.83%
55.33%
65.03%
Absmax INT4, col-wise
9.30%
17.43%
37.88%
38.04%
34.83%
Absmax INT4, row-wise
21.37%
35.80%
40.95%
46.75%
NaN
Zeropoint INT4, col-wise
11.51%
26.51%
41.65%
46.63%
48.26%
Zeropoint INT4, row-wise
24.95%
33.05%
43.63%
49.41%
NaN
GLM-110M
GLM-335M
GLM-2B
GLM-10B
GLM-130B
Original
29.36%
48.51
68.19%
72.35%
80.21 %
Absmax INT8, row-wise
29.25%
69%
68.12 12%
72.37%
80.21 %
Absmax INT4, row-wise
3.26%
38.25%
62.62%
71.03%
79.47%
Zeropoint INT4, row-wise
5.45%
42.64%
64. 74%
70.50%
80.63%
67.3
72.774.8
65.466.4
63.563.567.1 56.33
64.1
71.2
CLM
turil
51.7
56.1.2.5
50.7
GLM + VPP (50
35. 035.035.6 40.0
33.734.5
27.327.727.3
LAMBADA
MMLU
WIC
ReCoRD
Hellaswag
WSC
BoolQ
ANLI
Contribution attribution analysis on GLM objective and MIP training. We take GLM- 10B (English only) as an example in the ablation. Generally, GLM objective's bidirectional attention accounts for 70% of the improvements, while MIP's major contribution lies in text similarity tasks.
Figure 14:
B.8.2 WEIGHT DISTRIBUTION ANALYSIS
To achieve INT4 weight quantization, we analyze the weight value distribution of major linear layers in GLM-130B and a counterpart BLOOM- in a histogram (Cf. Figure 15). The horizontal axis denotes the weight value, and the vertical axis denotes the number of weights of such value in log scale. As we can see, it is majorly the w2 linear layers in BLO0M-176B that present skewed distributions, which would hinder the symmetrical quantization. On the contrary, GLM-130B' s w2 is well-shaped without many outliers and skewed distribution, and thus paces the way for its INT4 quantization with little performance loss.
B.9 ABLATION ON CONTRIBUTION ATTRIBUTION
We analyze the contribution attribution of techniques leveraged in GLM-130B. series of ablation studies have been presented in the paper, and for the convenience of reading, they were originally scattered around whole passage. Here we summarize them here into the following list for read- ers' reference:
Ablation on ordinary PostLN and DeepNorm: Figure
Ablation on indireetioonalUnndieeettti Attention: Figure 2 (LAMBADA), Table 16 (Condi- tional NLG)), Figure 17 (SuperGLUUE).
Ablation on Embedding Layer Gradient Shrink (EGS): Figure 4.
Ablation on Positional Encodings and FFN: Appendix B.3 Table
Additionally, we conduct the following study justify the contribution of the two most influential techniques-GLM Objective and Multi-task Instruction Pre-training (MIP)-used in GLM-130B.
GLM Objective and MIP. Ablating 100B-scale LLM from scratch can be too expensive. As substitute, we try our best to conduct the comparison between GLM objective and MIP on GLM- 10B (an English-only version released in (Du al., 2022), without MIP). We additionally train GLM- 10B initialized from a middle-stage original checkpoint with MIP (5%) to match the same training tokens of the original seff-superrisson-only GL.B-13BB. The MIP, this time, follows the
29

---

Published as a conference paper at ICLR 2023
exact dataset setting in TO (Sanh et al., 2022) and the information extraction datasets in GLM-130B to allow the correct evaluation on some types of tasks (e.g., NLI).
Figure 14 shows the ablation results. On the 8 datasets we test, find that the GLM objective major contributor to the improvement (from GLM (uni) to GLM + MIP (bi)). For exam- ple, it accounts for 73% improvement in LAMBADA and 90% improvement in MMLU, which are very widely adopted challenging benchmarks for LLMs. As for MIP, on some datasets (e.g., WiC, ReCoRD, Hellaswag), MIP may even harm the performance. While for datasets related text similarity and coreference (e.g.. WSC, BoolQ, ANLI RI), MIP is the main contributor. It is likely because the text similarity and coreference challenges, which people usually construct intentionally to test language models' ability, are seldom seen in the self-supervised corpus that makes up peo- ple's daily written texts. Thus, MIP training mainly helps to bridge the gap between self-supervised pre-training and these tasks.
B.10 LESSONS LEARNED
Lesson 1 (Bidirectional Architecture). The iidirectionallattentiin GLM is strong architec- ture alternative, in addition to GPTs.
Lesson 2 (Platform-aware Configuration). Configure L.LMs based on the cluster and parallel strategy used to squeeze hardware potential.
Lesson (Improved Post-LN), Counter-sterootypicalll. DeepNorm, a type of Post-LN, is the option to stabilize GLMM-10B.
Lesson 4 (Training Stability Categorization). Unexpected training instability that LLMS suffer from arouses systematically and numerically.
Lesson 5 (Systematical Instability: FP16). Though FP16 induces more instability, it enables training and inference on diverse platforms.
Lesson (Numerical Instability: Embedddig Gradient Shrink). Shrinking embedding layer's gradient its 0.1 can solve most numerical instability problems.
Lesson 7 (GLM's INT4 Quantization Scaling Law). GLM has unique INT4 weight quan- tization scaling law unobserved in GPT-style BLOOM.
Lesson (Future Direction). To create powerful LLMs, the main focus can be on I) more and better data, 2) better architectures and pre-training objectives, and more sufficient training.
30

---

Published as a conference paper at ICLR 2023
#TB- deast
WZ
L
Lager
Layerd
Layend
Lager
Layer?
Layera
Layer
Laper
Layaz 12
Laye
Layel 14
Lege:
Layer
Layer 18
Layor
Toger 70
Layer 24
Layer 25
Toper 36
ayye
Figure 15: Weight value distribution of linear layers GLM-130B. (in orange, attn-dense, attn-qkv, glu-wl, glu-w2) and BLOOM-I76B (in blue. stan-dense, attn-qkv, ffn-wl, ffn-w2)'s first 28 transformer layers. Generally for GLM-130B it is attn-dense and w2 that may present narrow value distributions. attn-qkv and wl may also be reason for enabling INT4 quantization in middle layers of GLM-130B.
31

---

Published as a conference paper at ICLR 2023
C DATASET AND EVALUATION DETAILS
C.I MULTI-TASK INSTRUCTION PRE-TRAINING (MIP)
Following practices in (Raffel et al., 2020; Wei et al., 2022a; Sanh et al., 2022; Aribandi et al., 2022), we include number of prompted instruction datasets GLM-130B'S MIP training. which accounts for 5% of the training tokens. All prompts for TO datasets are from PromptSource (Bach et al., 2022)
and prompts for DeepStruct datasets are newly created. Their composition is shown in Table 12, which makes up natural language understanding and generation datasets from TO (Sanh et al., 2022) and promptsource (Bach et al., 2022), and information extraction datasets from DeepStruct (Wang et al., 2022a). In GLM-130B 's training, calculate that approximately 36% of the samples in each dataset has been seen.
TO originally splits datasets for 1) multi-task prompted training and 2) zero-shot task transfer two sections. We initially planed to only include training sets of TO's multi-task prompted training section and DeepStruct (Wang et al., 2022a), but by mistake included both multi-task prompted training and zero-shot task transfer sections' datasets in MIP and excluded DeepStruct datasets. The mistake was fixed around 23k steps and our model continued to train on the correct version.
Natural Language Understanding and Generation. We adopt datasets and corresponding prompts from promptsource (Bach al., 2022). For all prompted samples in each dataset, we truncation of maximal 10,0000 samples per dataset and combine them together as the MIP dataset. Details of the prompted samples and datasets are provided promptsource's GitHub repository
Information Extraction. Based on the datasets from DeepStruct (Wang et al., 2022a), multi- task language model pre-training approach for information extraction tasks, we create instructions and prompts for part of its datasets (as is shown in Table 12). We reformulate information extraction tasks into instruction tuning formats to allow zero-shot generalization to new extraction schema. For all prompted samples in each dataset, we set truncation of maximal 20,0000 samples per dataset as there are fewer information extraction datasets than common language understanding and generation ones. For KELM (Agarwal et al., 2021) I) and PropBank (Kingsbury & Palmer) datasets, since their original size is gigantic, we sample 50,0000 samples for each of them from their prompted samples.
C.2 DATA AND PROMPTS IN MIP FOR DEEPSTRUCT
Prompts and instructions for all datasets in DeepStruct (Wang et al., 2022a) are newly created by authors manually, The introduction, task description, and full prompts for each dataset are attached in the following sections. To allow template infilling, all prompts are written into Jinja' templates. When dataset sample provided in our format, Joinja engine will render it into a prompted sample with instruction.
more systematic evaluation on GI.M-130B's information extraction ability is left for a future work, as the concentration in this work is on the training and designing details of an LLM.
C.2.1 DIALOGUE STATE TRACKING
We adopt Multiwoz 2.1 (Eric et al., 2020) dialogue state tracking dataset. The dataset is reformulated into two tasks, each with one prompt correspondingly:
Dialogue state tracking: which asks the model extract information from dialogues given a list of certain slots, e.g., _axi_arrin val. val__ tmme and dest t i nat i on.
Slot filling: which model should fill in one provided slot and identify situations without answer.
http:ttss//..h.ccc.ooooooooeeeeo chttps://g http.tcooooallllt////iiii
32

---

Published as a conference paper at ICLR 2023
(Dialogue State Tracking, Prompt 0)
Read the dialogues between "[User]" and " "[Agent]
[(text))
identify and extract the information related to the following categeries (from top to down):
{(allowed_relationn
join "))
in the form of [User] Y 2 )" III ((format_ wwwpptpppeeeeiioooo..ooo detowed_relations join(" ")
(Slot Filling, Prompt 0)
Given the following dialogue:
[(text))
please answer the question: has "[User] " menticned "[[allowed deelatio_idxx_sppl split .sppiit ') join("'s "}}}" ? If yes, please write down
the answer from the dialogue; if not, please answer "not given".
Answer: II1 18 if filter cewtter_reeationnocc (relations, aelowii (tcloon_ixxli_________ allowed rewoweddeeeationss ( relation idxl) [0] ''tai'''" else 8 given endif $
() 0 |lllfilter ret (elaatio(reeationn.
C.2.2 EVENT EXTRACTION
We adopt ACE05 (Walker & Consortium, 2005) event extraction datasets following the setting in (Wadden al., 2019). The dataset is reformulated into tasks with three prompts follows:
Event Argument Extraction: given a trigger in text and a list of its argument roles, the model is asked extract the arguments from the provided text.
Argument Identification: given trigger and a certain argument role, the model is asked to extract the argument if it exists in the provided text; otherwise, the model should generate nothing.
(Event Argument Extraction, Prompt 0)
For the task of "Event Extraction", given a trigger one should extract its related arguments conditioned on list of potential roles.
Given the following 1ist roles:
hagarreeeellloeeeegrrrrrrssrrrg arguments (eveete values ()) I (")
n- ")1
extract related arguments of the trigger "[[trigger[ 'text' }]} (ff
[1lowed_triggeen [triggt [trigger ('event_type" )1)))* the following sentence:
((text))
Extractions: II| [[format pepemat_triieeeeeeeee perippeereeatiinnn "") join(" "))
33

---

Published as a conference paper at ICLR 2023
(Event Argument Extraction, Prompt 1)
TEST
1. (Event Extraction) ((text))
Please write down ALL event arguments related to the trigger "((trigger ('text betbovedtriiggerrtriiig.go (trrgrrsrrrrgreeeeeetteee ('event_typeee marked with "[ 1", given the following categories:
(Ishfffeeallloeee jarguudnnnnnnttaataa [tri [trigger ['event_type valuus ()) join(")
"111
Answer: III [[format pepammt__rrpleeeeeee (relations, "*) | join(" ")
(Argument Identification, Prompt 0)
Let extract event related arguments!
In the fol Following passage, an argument with the type "[[query_arg))" is related to the event trigger "([trigger[" ['text [[flllowe_trriggeet) trigger[ "ggerr"'vent_typ"]1113
((text] ||
The argument should be (copy from the context if you find it; if net, do not generate): III ((filter_type (relations, query_arg) join "}}}
C.2.3 JOINT ENTITY AND RELATION EXTRACTION
Joint entity and relation extraction aims to recognize named entities in a piece of text and judge the relationships between them. It is closely related to knowledge acquisition, where the ulti- mate target is to structuring the unstructured web contents into knowledge triples (e.g., (London, capital_of, Britain)) ). The task can be formulated into either a pipeline framework (a combination of named entity recognition and relation extraction), or end-to-end training.
In this work, we adopt three classical joint entity and relation extraction datasets: CoNLLO4 (Roth & Yih, 2004). NYT (Riedel et al., 2010), and ACE2005 (Walker & Consortium, 2005). In GLM-130B, follow (Wang al., 2022a) formulate such challenges into eequence-to-sequenee generation, where our inputs are raw texts and outputs are triples. We only conduct relation-related tasks for these datasets here, and leave the entity-related ones the named entity recognition section.
Relation Extraction: here we extract knowledge triples consisting of "head entity", "relation", and "tail entity", given a list of relation candidates. For example, given the input "In Kunming the 800-some faculty and student established the National Southwestern Associated University", the model output could be (National Southwestern Associ ated University, location of formation, Kunming).
Conditional Relation Extraction: given single relation candidate, judge if the input text tains the relation. If so, extraction all related triples; if not, do not generate.
Knowledge Slot Filling: assign a certain entity from text, and ask the model to extract all triples that takes the entity as head.
Relation Classification: given two entities from texts, ask the model to judge the relation between them based list of candidate relations.
34

---

Published as a conference paper at ICLR 2023
(Relation Extraction, Prompt 0)
Can you figure out all triples regarding the relations of "[ssufffee
allowed relations) join("", "())"" from the sentence? List them in the shape of "(X Y 2 )";
[(eext)) III (fformat_triple (relations, relation)))))tinss) | Join(" ")
(Conditional Relation Extraction, Prompt 0)
Conditioned on the relation "[lallowed relations "[ relation idx )) ))", what knowledge triples can be extracted from:
[text]]
P Please write them down here: III [[formt_trriple (relattionn allowed_relate relations relatioo_idxl] join (" "}}}
(Knowledge Slot Filling, Prompt 0)
(8 entity_types len () 8)
In the sentence
[text)]
the x illentitieseentiiieen (edenittydddll)" idx is an entity the type "[[ entity_types (ttptttttypessettttyyy.e [ixx |)", Extract all possible triples contains "([ centieessenttty "ixrity_idx]]""" in the form of ; following candidate propertics Y:
given the
(6 for r in allowod $)- ((x)) (8 endfor $)
Answer: III {8 for in relations $)18 if r[ r['head'] [0] = entities[ entity_idx] [[(format_triplee(() allowed_relatiee join HS
endi: $118 endfor $)
(8 endi: 81
(Relation Classification, Prompt 0)
QUIZ
1. Given the candidate relations:
((shuffle (allellloollloweeccaaaioooo join ("An- ")
what is the relation between "I "[[relations (tripletrieeeeie idx] ['head'] {1 and
(' ('tail'] [(])) in the fellowing sentence?
[(text))
Answer: III lrelaation [triple idx ('relation' 11}
Nevertheless, existing joint entity and relation extraction datasets have very limited relation schema. For example, CoNLL04 only contains five different relations; the most diverse NYT dataset con- tains 24 Freebase predicates. To allow the model to capture diverse range of potential verbal- ized predicates, we extend the task with automatically generated knowledge-text aligned data from KELM (Agarwal et al., 2021). We do not include other distantly supervised dataset (e.g., T-Rex (El- sahar et al., 201 8)) since they can be extremely noisy.
For KELM data, since it is based the full Wikidata schema (which contains too many relations to be enumerated), we create two LELM-specifi. prompts for the task Relation Extraction and Knowledge Slot Filling:
35

---

Published as a conference paper at ICLR 2023
(Relation Extraction, Prompt 1, KELM ONLY)
kelm #1
Can you figure out all knowledge triples regarding whole Wikidata properties from the sentence? List them in the shape of " x ; Y 2 )
(teett)) III (fformat_triplo (relationn, "") I Join(" ")}}
(Knowledge Slot Filling, Prompt 1, KELM ONLY)
( kelm #} Given the entity "((entities (entiity )) marked with "[" and "]" in the context:
((text))
please 1ist all triples related it (do not generate if there
answer): III {8 for r relations 8) if r['hedd') [[0] entitiesl
entity idx] | [formatt (eomatttriple((r]] "") I join(" "HH endif {) endfor
81
C.2.4 NAMED ENTITY RECOGNITION
Named entity recognition is a task which targets identifying named entities from raw text corpus and assign them with proper entity types. For example, in the sentence "In 1916 GM was reincorpo- rated in Detroit as "General Motors Corporation ".", General Motors Corporation iintiion could be of entity type organi zation. We design two different types tasks based on named entity recognition datasets CoNLLO3 (Sang & Meulder, 2003), OntoNotes 5.0 (Pradhan et al., 2013), and GENIA (Ohta et al., 2002). We also include named entity recognition sub-tasks from joint entity and relation datasets.
Named Entity Recognition: given a certain list of possible entity types (e.g., location, person, organization extract all related entities from the provided text content.
entity typing is one of the important derivative tasks from named entity recogni- tion. It aims to classify the correct type of an entity mention (without entity types), and is often appended the entity mention extraction as post-processing.
Entity Typing:
(Named Entity Recognition, Prompt 0)
Given the following list of entity types:
z ((shuffle (allored__eoww types) join(", ")}}
please extract all mentioned entities from left right in the sentence in the form of " x instance of 2 )
[(text)) => III ($ for entity, type in zip(entities, entity_types) $) ((entity)) instance of ((type)) {& endfor
(Entity Typing, Prompt 0)
Extract all entity mentioned in the sentence with entity type "(( al Lllowed_types ttpe__ixxl 1 in the form of "( instance of ; {( al lowe__types [xxxp____x]]]+ )"
[(text)) |I| ($ for entity, type in ziplentities, entity_types) (4 if type allowed sypes|type idx] () (entityy) ; instance of ([type F1 (8 endif $148 enafor $1
36

---

Published as a conference paper at ICLR 2023
(Entity Typing, Prompt 1)
List all "[allooed types(type. i]x]))" entities appeared in the following passage, joined by ":
[(text]) => III (ffiltor_type (entitit,,,,, entity_typos), tlyosed_types [type_idx]] join(" ")}}
(Entity Typing, Prompt
($ if entity_types,_ len_ () 0 8)
Based on the list of potential entity types and ignore their order:
join("(n- }))}
the entity "[fentitics [entity_idk])) marked with "(" and "J" in the following sentence:
[text]]
belongs to III (eentityyttpeesent (8 endi $1
C.2.5 RELATION CLASSIFICATION
Relation classification is a fundamental task in information extraction, which identifies the relation- ships from list of candidates between two given entities. The problem is long standing one suffers from outrageous cost of data labeling, since manual labeling on knowledge-intensie tasks requires educated annotators that charges high. A de facto data creation method relation extrac- tion relies on distant supervision, which aligns existing knowledge triples in knowledge bases to text contents automatically, and assume that such alignments are correct in certain conditions. Here we only include TacRED (Zhang et al., 2017) dataset and create several different tasks based on it.
Relation Classification: the most traditional task formulation. Given two entities from text and classify their relation from list of candidates. The form can be either answering the relation directly or the form of triple (similar to relation extraction).
Knowledge Slot Filling: change the task into given head entity and relation, to identify whether the tail entity exists in the input text. If not, generate nothing.
Yes No Question: turn the problem into a task similar natural language inference. For example, given the sentence "The series focuses the life of Carnie Wilson, daughter of Brian Wilson, founder of the Beach Boys.", the model will asked to judge the correctness of a triple such Carnie Wilson, father, Brian Wilson by answering "yes" or "no".
(Relation Classification, Prompt 0)
(8 if encity_types.__en > 0 $)
Given the following categories of relations:
((ahoffte valiiissssvalles/// ()) ("Ann"An- "}]
predict the relation between "[[relations [0] ['head']]] ) and "[[relatiens
(0) [t] in the following sentence:
[(text))
The relation should be III [ [0] [' relation '11!1 (8 endis 81
37

---

Published as a conference paper at ICLR 2023
(Relation Classification, Prompt 1)
1. (Relation Extraction) Answer the relation between entities in the form of "( ; 2
((text))
The relation between "((relations [0] ('head']1)" and ([[relations
tail']]]" is: 1II ( ((relations (") "hhad']]] [[allowed_relations] relations [0] ['relation']]1] ((relations [0] [[]]]][aaatt ['tail
(Knowledge Slot Filling, Prompt 0)
Based the sentence provided below, infer the missing argument asked by the question:
((text))
Question: What/Nho/Mhere "[[relations [0] 1H1 relowed_relation pornns/reetttte |relations (rrellrionnn 1111
Answer: II| {[relations [0] ['tail']))
C.2.6 SEMANTIC ROLE LABELING
Semantic role labeling is long- long-standing information task that wants identify the semantic argu- ments related to given predicate sentence. For example, in the sentence "Grant was employed IBM for 2/ years where she held several executive positions." " and the predicate "empl loed in it, semantic role labeling identifies the Grant as the subject and IBM as the second object.
We create different tasks based semantic role labelling datasets CoNLLO5 (Carreras & Mårquez, 2005), CONLL112 (Pradhan et al., 2013), and PropBank (Kingshury & Palmer).
Semantic Role Labeling: the traditional task form, where verb (i.e., predicate) is annotated in text and the model is asked to generate related semantic roles.
Semantic Role Filling: given verb and and potential semantic role, the model is asked to judge whether the role exists in the sentence and generate
Predicate Recognition: given a segment of a sentence and its corresponding semantic role, iden- tify which verb is related to.
(Semantic Role Labeling, Prompt 0)
Provided with the target verb " "[[verb)) marked with "[" and "]" in the following sentence, find its [y
[(text]] => III (8 for entity, type in (ip entity $118 it type == allowed tyresdd_ypessttype idx] i) endif 8)18 endfer $1
(Semantic Role Filling, Prompt 0)
Given the following list argument types:
z = ([allowed_types I joint", ")])
find out all arguments related to verb "[[verb)]" mentioned the fol following sentence from left to right, in the form of " x instance of ; % ).
((text]] III (8 for entity, type in (entities, matity_typest 81{ ((entity]] argument type ; [[type]] ) (8 endfer $1
38

---

Published as a conference paper at ICLR 2023
(Predicate Recognition, Prompt 0)
FINAL EXAM
1. Based on the fact that "I "[(entities [entity_ixx]]] is "I{ entity_types [ontity_idx] H)", which verb in the following sentence shculd it related to?
((text))
Answer: III [[verb))
C.3 RESULT SOURCES FOR GPT-3, BLOOM-1766, AND OPT-175B
Here we describe the result sources for GPT-3, BLOOM-176B, and OPT-175B. Other LLMs we may compare are mostly completely closed-sourced; thus, their results all taken from existing preprints, publications, or the results stored in BIG-bench repository!..
For GPT-3, while most of its results in this paper taken from existing literature if not specified, the rest were acquired via our own requesting OpenAl Danvici API are explicitly mentioned. For BLOOM-I76B and OPT-175B, if without specific annotation, their results are:
Taken from the OPT paper (Zhang et al., 2022).
Taken from the EAI-Eval BigScience Arch&Scale Google Sheet!!
Taken from BigScience evaluation results repository in Huggingface Datasets
Specifically, we cannot evaluate OPT-175B by ourselves as we are still not officially granted the checkpoint, though we have sent several applications in the past few months.
Table 13: GLM-130B and its similar-sized LLMs' BPB results Pile test-set.
C.4 PILE TEST-SET EVALUATION
Pile evalution (Gao al., 2020) is comprehen- sive language modeling benchmark which origi- nally includes 22 different verse domains. We report our results over part of 18 datasets with previously reported baseline re- sults (Lieber et al., 2021). Different from tradi- tional language modeling benchmarks. Pile evalu- ation report the BPB (bits-per-byte) perplexity to avoid the mismatch comparison between models with different vocabularies. Because in general, language models with a larger vocabulary will be favored in perplexity comparison if not restricted. In the evaluation, we strictly follow the setting in (Gao al., 2020). leveraging [gMASK] and a context-length of 1,024 with bidirectional atten- tion, and the rest 1024 tokens calculate BPB in an autoregressive manner. The weighted average BPB are calculated based on each shared dataset's ratio in Pile training-set (Gao et al., 2020).
Jurassic-1
GPT-3
GLM-1300 130B
datasets from di-
dm mn__matemmtics
1.040
1.370
0.786
ubuntu ire opensubtitles hackernews hooks33
0.857
0.946
0.977
0.879
0.932
0.889
0.869
0.975
0.873
0.835
0.802
0.803
pile cc philpapers
0.669
0.698
0.771
0.741
0.723
0.766
gutenberg pg_
0.890
1.160
0.821
arxiv
0,680
0.838
0.570
stackexchange nih exporreer
0.655
0.773
0.611
0.590
0.612
0.614
pubmed abstracts
0.587
0.625
0.610
uspto_ backgrounds
0.537
0.566
0.537
pubmed central
0.579
0.690
0.510
freelaw
0.514
0.612
0.499
github
0.358
0.645
0.329
enron enron_emails youtube_subieee subtitles
0.621
0.958
0.604
0.825
0.815
0.746
Weighted Avg.
0.650
0.742
0.634
The detailed metrics on Pile test-set are reported in Table 13. We observe that compared GPT- 3, GLM-130B has noticeable weaker performance phil__papeee and pile _cl_ which is likely because of GLMM-130B'S bilingual natural and lack of more diverse and high-quality private collected corpora.
10 Phttps://gtt thub. http://ghhho..coooogogg/////-ee/ http:///..................o..... http://:///booogoggg.ooooo////// 1u@-CmaFuhuv0 http///wwrbboooeeeeee/////eeeeee
bat www.eeeeetttteeeeeeeeeee
39

---

Published as a conference paper at ICLR 2023
C.5 BIG-BENCH-LITE EVALUATION
Recent works (Wei et al., 2022c; Wang et al., 2022c) re-
Aggregate Normalized Performal Performma Performane
40
GLM-1308 0-shot
veal that LLMs are capable to do reasoning beyond con-
35
GLM-1308
ventional language tasks. As a response, BIG-bench (Sri-
GLM-130B 3-shot
vastava et al., 2022) is recently set up by crowdsourcing new types of tasks from global researchers to test LLMs
30
GPT-3 O-shot
25
1-shot
GPT-3 GPT-333-hhot
3-shot
unexplored abilities. For economical consideration, we evaluate GLM--30B on an official subset of original 150- task BIG-bench, the BIG-bench-lite with 24 tasks. These
20
PaLM
PaLM 1-shot
15
10
tasks can be categorized into two types: one is based on multiple-choice question answering with answer options,
5
and another is direct generation without options. For the
first category, we assess the probability of each option's s
-5
full content and pick the largest one as the answer; for the second one, we generate the answer using greedy de- coding, All evaluations done in BIG-bench are based [MASK], since answers here are usually short pieces texts. All results on 24 BIG-bench-lite (Srivastava et al., 2022) datasets three L.MS are shown Table 14 and Figure 16. We just adopt the original prompts from BIG-bench and the official implementation to generate priming examples for few-shot evaluation and to calculate the final scores.
10a
10
1000
10
Effective Parameter Count
Figure 16: A full scope of BIG-bench- lite (24 tasks) evaluation.
C.6 MMLU EVALUATION
results on 57 MMLU (Hendrycks al., 2021) datasets of GLM-130B and BLOOM 176B are shown in Table 15. In Section 5.2, we report weighted average accuracy (i.e., accuracy average per CLM-1300B GPT-3 175B, and BLOOM 176B.
sample, rather than by discipline)
Below is prompted example with 1-shot priming. We predict the probability ['A', 'B', ''', 'D'] at the next token, and take the one with the maximal probability as the answer.
(MMLU 1-shot Example)
The following are multiple choice questions about philosophy.
According d' d'Hobba,h, people always act according (A) free choices (B) dictates of the soul (C) necessary natural laws (D) undetermined will Answer: (C) necessary natural laws
kpicurus holds that philosophy is: (A) not suitable for the young. (B) not suitable for the old. (C)
important, but unpleasant. (D) none of the above. Answer:
C.7 CHINESE LANGUAGE UNDERSTANDING EVALUATION
Here we elaborate the prompts we use for CLUE (Xu al., 2020) and FewCLUE (Xu et al., 2021) evaluation. On Chinese datasets, prompting meets some challenges as Chinese texts are organized by single characters rather than words, leading to unequal length of verbalizers in many cases. Albeit dataset-specific calibration (Wang et al., 2021; Wu et al., 2021) can help to mitigate the issue, the too specified technique can be complicated implementation. Our evaluation in this paper adopts more easy to solve method leveraging GLM- 's unique features. As GLM- 300 is bilingual LLM with English MIP, we adopt English prompts and verbalizers from similar tasks in (Bach et al., 2022) for Chinese dataset evaluation and find such strategies to be quite effective. In terms of evaluation metrics, except for DRCD and CMRC2018 two question answering datasets which reports EM, other datasets report accuracy.
40

---

Published as a conference paper at ICLR 2023
C.8 NATURAL LANGUAGE GENERATION
Natural language generation, or conditional natural language generation here, refers to tasks that require generating text based on the given information, such as tables and documents. We evaluate GLM-130B data-to-text and summarization tasks. The datasets include WebNLG 2020 (Cas- tro Ferreira et al., 2020), Clean E2E NLG (Dušck et al., 2019) and WikiLingua (Scialom et al., 2020) from GEM generation benchmark (Gehrmann et al., 2021). We select full WebNLG 2020 and the Clean E2E NLG in the test set and randomly select 5000 test examples from WikiLingua following the practice in (Chowdhery al., 2022). Following the settings in PaLM, the prompt used for the Summarization tasks is "Summarize the following article:" and the prompt used for the Data-to-Text tasks is "Verbalize:". An exception is E2E, where process the data using the prompt "generate- grarrttcallaaaaacllllll...l www.lly--ooocttteee from" provided in promptsource for GLM-1 and GPT-3 175B (Davinci). All evaluations are one-shot, and the demonstration samples are randomly sampled from the training set. We report the F-measure of ROUGE-2, ROUGE-L (Lin, 2004) and BLEURT-20 (Pu al., 2021). compare our model with LaMDA, GPT-3 175B (Davinci), and PaLM, where the results of LaMDA and PaLM are reported by (Chowdhery et al., 2022), and we evaluate GPT-3 175B (Davinci) through OpenAl API. 13
Our results presented in Table 16. It shows that GLM-130B has better performances than LaMDA and GPT-3 (Davinci) on tasks. In Data-to-text task, GLM-130B performs slightly worse than PaLM-540B, while in the summary task, GLM- 130B has even higher ROUGE results. We also ablate GLM-130B to unidirectional to demonstrate the advantage of bidirectional attention. Unidirectional GLM-130B underperforms GPT-3 175B in all three datasets, but when it shifts to bidirectional attention, there is an instant boost, making GLM-130B even comparable to PaLM- 540B few cases, indicates that bidirectional attention over the provided context (i.e., prefix) can also be beneficial for text generation missions.
Table 16: 1-shot GEM English natural language generation tasks (WebNLG, E2E, and WikiLinguaa We compare two versions of CLM-130B (uni: unidirectional attention, bi: bidirectional attention), showing that bidirectional attention can also improve conditional generation's 's performance.
LaMDA 137B
GPT-3 175B (Davinci)
GLM-130B
Task
Dataset
Metric
Ma140B
uni
bi
ROUGE-2
30.5
29.9
25.3
38.5
44.4
WebNLG
ROUGE-L
41.2
36.7
49.3
53,8
Data
BLEURT-20
59.0
53.2
67.7
73.9
to Text
ROUGE-2
29.2
30.3
30.9
33.9
35.2
E2E
ROUGE-L
39,2
40.0
42.6
43.9
BLEURT-20
64.5
65.0
68.1
69.7
ROUGE-2
5.4
7.2
5.8
10.4
9.9
Summary
WikiLingua
ROUGE-L
18.9
16.4
23.4
20.6
BLEURT-20
41.2
39.4
45.0
47.7
(E2E Example, without demonstration sample)
Aleksandr Prudnikov height 185.0 0 (centimetres)
FC rc_spartak_Moscox ground Otkrytiye_Arena.
Bleksandr_Prudnnikon , club rc_spartak_Moscov. Verbalize:
Groundtruth: 185 centimetre tall Aleksandr Prudnikov played for the Otkrytiye Arena based FC Spartak, Moscow:
GPT-3 175B (Davinci): Aleksandr Prudnikov is midfielder for FC Spartak Moscow, football (soccer) club based in Moscow, Russia.
GLM-130B: Aleksandr Prudnikov 185.0 cm and plays FC Spartak Moscow.
We use ROUGE implementation at http:///hhbbboomogogglleressaa// htt.pp/g...sstoooooooooooooeennr and BLEUR--20 implementation at whose checkpoint is available http:///www...ooogoloop////meoe/ 210.
41

---

Published as a conference paper at ICLR 2023
(E2E Example, without demonstration sample)
Combine all of the following data into a concise and grammatically correct text:
name : Blue Spice
eatType : coffee shop area riverside
Groundtruth: At the riverside, there is coffee shop called The Blue Spice.
GPT-3 175B (Davinci): Blue Spice is riverside coffee shop which is located the corner of River Street and Riverbank Street.
GLM-130B: There's coffee shop that serves coffee in the riverside area, Blue Spice.
(WikiLingua Example, without demonstration sample)
The majority of your customers will search for you online, so it's essential to have user-friendly website. At the very least, your website should include information about your business and your history in the moving industry, details about the qucting process, contact information, and a description of the services you offer. If possible, allow customers to schedule quotes online, view your availability, or from other customers. One of the easiest ways to start your business is by helping people you already know with their moves. moving that
read testimonies
You can be on the lookout for any announcements related your friends make on social media. Once you have provided good service to friends, they are likely to recommend you to others. In order to spread the word about your business, have some professional looking promotional materials printed and distribute them around your community. You can distribute business cards at public events, tuck them into local bulletin boards, or even print them in directories, yearbooks, and other local print media. Flyers can be mailed, posted in public places, or distributed to businesses that might be able to refer customers to like you, such as furniture stores. Make sure you have professicnal, recognizable logo that is consistent across all your marketing materials. Another way to get your business's name out there is to make yoursels visible. Whether it's by working with partners at local events, volunteering,
using your vehicle for ad campaign, visibility is key for driving business. Build relationships with influential people in your community. Realtors are a great source of referrals to movers, as are the owners of local furniture stores or the office staff at large apartment complex. You can use directory sites like Craigslist to advertise your services to people in your local community for free. Social media is also a great way to spread the word about your business. There are many options for advertising, depending on your budget and your target market. Consider options like PPC advertising, television and radio commercials, newspaper ads, direct mail flyers, memberships with referral services. The best thing you can to grow your business is to provide excellent service to your customers. Be sure to always be on time, be friendly, be respectful of your customers! belongings, and offer accurate price quctes. Be sure to ask your happy customers to post reviews about you online. New customers will see these reviews and will have more faith in you as a reputable company if they see that others have had a good experience. You will inevitably end up having to deal with an unhappy customer at some point, but do problem to the customer's satisfaction. The last thing you want is negative reviews about your business circulating the internet! Summarize the article:
best to resolve the
Groundtruth: Create a website. Start with friends. Distribute business cards and flyers. Network for promo- tion. Seek out free advertising opportunities. Consider paid advertising. Gain good reputation.
GPT-3 175B (Davinci); Market yourself online. Look for opportunities to market your services offline.
GLM 130B: Have user-friendly website. Promote your business through social media. Build relationships with influential people in your community. Use directory sites like Craigslist to advertise your services.
42

---

Published as a conference paper at ICLR 2023
Table 17: Winograd-style tasks evaluation (Winogender and Winograd273). All scores are accuracy. K refers to number of shots. "PaLM 540B did not report the exact O-shot Winogender result, so we have estimate value from plotted diagram.
GPT-3
OPT
BLOOM
PaLM
Gopher 280B
K
Chinchilla
GLM-130B
(Davinci)
175B
176B
540B
0
64.2
54.8
49.1
75.0"
78.3
71.4
79.7
Winogender
1
62.6
53.1
79.4
80.7
Winograd273
0
88.3
52.9
49.1
90.1
84.3
Table 18: Closed-bool question answering (Natural Questions, StrateyyQA)
GPT-3
BLOOM
PaLM
Gopher 280B
Chinchilla
GLM-130B
(Davinci)
176B
540B
Natural Questions (EM)
14.6
13.1
21.2
16.6
10.1
11.7
Strategy QA (Acc)
52.3
49.8
64.0
60.6
Table 19: Commonsense reasoning (Commonsense QA, MC-TACO), K refers to number of shots.
K
GPT-3 (Davinci)
OPT 175B
BLOOM 176B
GLM-130B
0
57.2
42.8
61.6
Commonsense QA (Acc)
1
61.2
62.2
MC-TACO (EM)
0
12.4
13.1
13.6
C.9 WINOGRAD-STYLE TASKS
We include the evaluation Winograd-style tasks, which derives from the classical Winograd Schemas Challenge (Levesque et al., 2012) that aims to test coreference resolution in an ambiguous context for the machine to understand. Since in MIP, et al., 2021) and SuperGLUE WSC (Wang et al., 2019), here we test on Winogender (Rudinger et al., 2018) and Winograd273 (Levesque al., 2012). For Winogender, GPT-3's results are acquired from OpenAI API, and BLOOM's 1-shot result is evaluated by ourselves. For Winograd273, since exist- ing works (Brown et al., 2020; Chowdhery et al., 2022) show that 1-shot learning brings almost no improvement, we only test the zero-shot result. Another thing to notice is that, despite GPT-style models (e.g., GPT-3, PaLM) adopting the "partial evaluation" described in (Radford et al., 2019), we find the prompt "<sentence> The " "<gronoun>" refers to [MASK) " is better for GLM-130B and adopt it in the evaluation.
have included the Winogrande (Sakaguchi
The results presented in Table 17. GLM-130B performs the best across all evaluated LLM on Winogender, and marginally poorer than GPT-3 and PaLM on Winograd273.
C. BLOSED-BOOR QUESTION ANSWERING
Closed-book question answering (CBQA) (Roberts et al., 2020) is widely adopted task to evaluate language models' memorization of factual knowledge, contrary to the traditional bbokk evaluation. As we have included TriviaQA (Joshi et al., 2017) and WebQuestions (Berant et al., 2013) in the MIP training, here choose Natural Questions (Kwiatkowski al., 2019) and Strat- egyQA (Geva et al., 2021) as the evaluation datasets for CBQA.
The results presented in Table 18. GLM-130B performs relatively poorer Natural Questions and performs well on Strategy GLM- 130''s 's underperformance on Natural Questions, we spec- ulate, potentially derives from the insufficiency fitting English corpora, it roughly only viewed
43

---

Published as a conference paper at ICLR 2023
200B English tokens and thus does not memorize the detailed knowledge very well. Since CBQA seems to be a task that especially stresses memorization, as indicated by Chinchilla (Hoffmann et al., 2022)'s is strong performance, think with sufficient training later, GLM- 130B can perform better.
C.I1 COMMONSENSE REASONING
Here evaluate GLM-130B and other LLMS on commonsense reasoning abilities. As have included PIQA (Bisk et al., 2020), ARC (Clark et al., 2018), and OpenbookQA (Mihaylov al., 2018) in the MIP training, we select another two widely adopted commonsense reasoning datasets in our evaluation: Commonsense QA (Talmor et al., 2019) and Multiple-choice Temporal Commonsense (MC-TACO, Zhou al. (2019)). For Commonsense QA, we test the GPT-3 via OpenAl Davinci API, FLOOM-1766 via its Huggingface Implementation, and GLM-130B using from promptsource (Bach et al., 2022). For StrategyQA, we follow the EM computation method provided in (Zhou et al., 2019).
the prompt "answer_given Literreiiie_euesiionn___aa
The results are shown in Table As we can see, GL.M-138 performs the best on both Com- monsense QA and MC-TACO across evaluated LLMs, demonstrating that GLM-130B has good grasp of commonsense knowledge. OPT's results are not included due to the reason described in Appendix C.3.
C.12 FIXED LABEL DATASETS: A CASE STUDY IN NATURAL LANGUAGE INFERENCE
As is discussed in Section 5, we adopt rather strict criterion for selecting datasets for zero/few-shot learning in GLM-110B's evaluation due the of MIP. Nevertheless, the criterion significantly reduces the dataset we could currently evaluate, and especially some readers have doubted whether the restriction of evaluating MiP-seen fixed-label datasets is necessary (e.g., natural language inference (NLI)), and suggest that we may report them in an independent section to avoid confusion.
Frankly speaking, in such setting GLM-130B zero/few-shot learning could be quite advanta- geous. Below, we take NLI as typical example to show GLM- 11--330'ss 's outperformance in the scenarios. We include 6 widely-used NLI datasets-which are incorporated GL.M1 1-M0''s MIP training, the benchmarks. The results are presented in Table 20, which shows that GLM-130B's "zero-shot" performance could be much better due to the seen task type.
Table 20: "Zero-shot" results of GLM-130B on 6 typical natural language inference (NLI) datasets. "DISCLAIMER" Despite the datasets are never seen, some other NLI datasets have been in- cluded in GLM-130B's MIP, making it different from the existing standard zero-shot setting.
BLOOM 176B
OPT 175B
GLM-130B*
qnli (valid, median prompts)
50.9
55.4
86.7
mnli (valid, median of 15 prompts)
35.5
36.0
85.7
mnli mismatched (valid, median of 15 prompts)
35.5
36.0
84.6
wnli (valid, median of 5 prompts)
57.7
53.5
67.6
glue/cola (valid, median prompts)
39.0
44.4
57.6
glue/mrpe (valid, median of prompts)
31.6
44.6
87.3
C.13 SUPERGLUE
We also report our evaluation of GLM-130B on the SuperGLUE (Wang et al., 2019) benchmark, which consists 8 different natural language understanding challenges. Noted that these results neither zero/few-shot nor fine-tuned results, because 7 out of 8 tasks' training sets have been included in GLM- 14-0-'B'ss MIP training (except for ReCoRD) together with other 67 multi-task datasets: however, GLM-130B is also not individually fine-tuned on any of them. Therefore, these results are not for relative comparison for any other models', but only for readers' reference on GLM-130RS absolute ability.
44

---

Published as a conference paper at ICLR 2023
GLM-1308 (bi)
cb
record
WSC
multirc
rte
wic
boolq
Figure 17: GLM-130F (uni and bij's untuned results SuperGLUE development set, using prompt- source (Bach et al., 2022) prompts and task formulation. DISCLAIMER: Noted that some of the SuperGLUE training sets have been included in the MIP training. We report the results here only for readers' reference.
100
84.5
Standard Prompting
Chain-of-Thought
73.7
68.3
54.0
53.1
47.9
27.9
13.4
1.0
Sports
Coin Flip
(OOD:
List
Date
Figure 18: Chain-of-thought prompting can also improve GLM-130BB' performance reasoning tasks compared to standard prompting.
BoolQ
CB
COPA
MultiRC
ReCoRD
RTE
WiC
WSC
GLM-130B
89.69
98.21
100
89.32
92.11
94.22
76.96
88.5
Table 21: The results of GLM-1300 on the SuperGLUE dataset obtained using the P-tuning v2 (Liu et al., 2022). We report the Accuracy metric for all datasets except for MultiRC (Fla) and ReCoRD (F1).
The results are presented Figure 17. We ablate the unidirectional and bidirectional GLM-130B to justify the usefulness of GLM objective in boosting LLMs' ability to understand. Each point in the figure refers to a prompt-specific result, for which the prompt is from the promptsource (Bach et al., 2022) repository. We adopt the task formulation from promptsource, too. As (bi) has much fewer variances and higher performances on all tasks. For some of the tasks (such as CB, MultiRC, RTE, COPA, and BoolQ), GLM-130B even achieve over 80% accuracy.
observe, GLM
We also attempted to fine-tune GLM-130B on the SuperGLUE dataset. However, we encountered the issue of rapid overfitting within single epoch when used full parameter fine-tuning downstream tasks. This resulted in poor performance on the validation set. To address this issue, explored the of efficient parameter fine-tuning methods, which tune only small number of parameters and are less prone to overfitting. After experimenting with several methods, we use P-Tuning v2 (Liu al., 2022), which demonstrated comparable results full parameter fine-tuning in 1300, but with only 0. % to 3% of tuned parameters. The results of our experiments with P-Tuning v2 are presented Table 21.
C.14 CHAIN-OF-THOUGHT PROMPTING
We evaluate the chain-of-thought prompting performance on Last letter concatenation (LLC), Coin Flip, Reverse List, and two tasks from BIG-bench Srivastava et al. (2022) Sports under- standing, and Date understanding, following the setting in Wei et al. (2022c). The results shown in Figure 17. We find that chain-of tfain-of-thought prompting can improve SLM-130B' performance on symbolic reasoning and commonsense reasoning.
45

---

Published as a conference paper at ICLR 2023
Log-scaling Ability Tasks
LAMBADA
Wikitext-103
Wikitext-2
PTB
80
11
70
30
13
14
60
14
40
16
50
15
40
16
60
20
10:0
10t1
109
10:0
Figure 19: Log-scaling ability tasks of GLM-130B. These tasks' performance grows logarithmically with the amount of GLM parameters. Most of traditional NLP tasks fall into the same pattern.
Last letter concatenation (LLC). The task asks the model to concatenate the last letters of words name (e.g.. "Elon Musk" "nk"). We generate full names by randomly concatenating the top 1000 first and last names from name census data 14
Coin flip. This task asks the model to answer whether coin is still heads up after people either flip or don't flip it beginning from being heads up. (e.g., "A coin is heads up. Phoebe flips the coin. Osvaldo does not flip the coin. Is the coin still heads up?" -> "no"). We additionally evaluate on the scenario where the number of people in the query examples is larger than that in the in-context examples, i.e. the out-of-distribution (OOD) setting.
Reverse List. This task asks the model to reverse the order of list of everyday objects (e.g.. "cigar, umbrella, key, gum, alarm" -> "alarm, gum, key, umbrella, cigar"). We generate the lists by randomly sampling from the vocabulary of everyday objects!5.
Sports. This task asks the model to judge the truthfulness of statement about a sports player (c.g.,
"Joao Moutinho caught the screen pass in the NFC championship" -> "false").
Date. This task asks the model to infer the data from a given context (e.g.. "2015 is coming in 36 hours. What is the date week from today in MM/DD/YYYY?" "0110552015").
We use the same examples and chains as Wei et al. (2022c). For each task, we try two different formats of prompts and both unidirectional and bidirectional attention mechanism and report the best performance. The first format is "Question: (context) Answer: (target)". The second one is add serial numbers before examples in the first format of prompts. The results are presented in Figure 18.
D SCALING AND EMERGENT ABILITIES IN GLM- 130B
Scaling up pre-trained language models has been proven to boost downstream performance wide range of tasks continually. His, emergent abilities which are unpredictable from smaller scales. To illustrate this, we conducted extensive experiments explore the scaling property and emergent abilities. Following prior literature (Wei et al., 2022b), we categorize the NLP tasks into two types based on our observations.
Log-scaling Ability Tasks (Cr. Figure 19): where the task performance grows logarithmically with the number of model parameters. Typical tasks and datasets include LAMBADA, Wikitext- 103, Wikitext-2, Penn Tree Bank.
Emergent Ability Tasks (Cr. Figure 20): where the task performance only soars up when the amount of model parameters reaches a certain threshold. Typical tasks and datasets include:
14 Mhttps://namecensus com hottps://wwwvooet wtw.llaayycyyoo.ooooo com/ httmm//ss/tttti 1ts5/8958
46

---

Published as a conference paper at ICLR 2023
Emergent Ability Tasks
MMLU
knowledee
crass_a ai
implicatures
45
50
60
45
40
50
35
40
35
30
60
55
25
10a
10.0
10t1
10a
10a
1030
1011
10a
10a
10th
10
10a
10a
103h
10a
understanding_fabbe fables
modified arithmetic
implicit_reaations relations
gre gre_RQ RC
17.5
35
15.0
35
12.5
35
25
30
30
20
5.0
25
2.5
10
20
20
10a
10"
1010
10tH
10a
10a
1020
10
10"
10"
10:0
10th
10a
10"
101h
10:4
Figure 20: Emergent ability tasks of GLM-130B. These tasks' performance does not grow much until the model size reaches certain threshold (e.g., 100B or 10B). After reaching the threshold, the model performance soars up quickly. The BIG-bench (Srivastava et al., 2022) benchmark collects many of these challenges.
MMLU, hindu_ knowledge, crass_ai, implicatures, understanding fables, modified_arithmeet implicit relatione, and gre_ compreeessomii from BIG-bench (Srivastava et al., 2022).
In line with the observation (Wei et al., 2022b), we show that GLM-1305 also presents the two similar scaling behaviors to other LLMs such as GPT-3, LaMDA, and PaLM. Though why and how LLMs present these intriguing properties remain unclear, GLM-1330 100 provides open opportunities for all researchers to test and understand the reason behind them.
47

---

Published as a conference paper at ICLR 2023
Table 11: Full configurations for GLM- 1-00B training
Configuration Key
Value
adam_ Letal
0.9
beta2_
0.95
adam eps
4
attention dropout
attention_sofiie saa____sooaaaiiiiaam
True
averuge average_block block_length
3
bias dropout fusion
True
checkpoint mneckpoint_ectivations acttvattinss
True
checkpoint cpu
False
checkpoint_ Beciiittnuu__leeeeeee layers
clip_ grad contigious, checkpointingg
10
False
spu_optimizer
False
data pamaleeeallee_si
24
deepnorm
True
distributed backend
neel
interval
1000
eval iters
3
ffn hiddhn_siizze
32768
fp16
Thme
global_ batch_bttizzii
4224
stu_activation
geglu
spi_prob
0,7
hidden_drypp dropout
hidden_size
12288
hysteresis
init mwt.hmcchsssc std
0.0052
init www.mmoo__aviee_uniiiiee
Fulse
initial Joss__________________ scale
65536
layernorm epsilon
1E-05
learnable caaable_ootryy_embddiiii embedding
Fulse
length per per_sample
2000
log. intertala
loss scale loss window Ir
2000
8c-05
decay iters
None
Ir rr_eeca__saa samples
197753905
decayysttye
cosine
Ir warmup_ samples
1098632
make vocab_size
wwwab_sizeddiviibble divisible__y
768
mask_prov ___bbpoob
0.15
masked softmax__ _usion
True
micro batch_size
min gomi__mmsk_ua ratio
0.2
min loss_scule
1.0
min Ir multitask_ratio
8c-06
0.05
num gum_attention_heet
96
layers
70
onnx_safe
None
optimizer
adam
partition_activtiinnn activations
True
pipeline model parallel
position embedding
rotary
rampup batch size
192, 24, 5493164
save interval seed
250
1234
seq lenghength short prob hit_emmoddiinggggtttnnnallppp single ainll_span__prot split
2048
0.02
faink_eebeddingg..
0.1
0.02
949,50,1
lensor tensor_model_ moodel_parallel_Se
tokenizer_tyye type
Tokeniiee
weight_deces
0.1
zero wwwtioooosssgneeeeeeee graddietss
Fulse
zero_redeiee ceno_reduce___e bestuce_bucket_teee bucket_size
500000000
zero ccto_reddce__cceee scatter
False
zero_sttge
zero-optimization. catpttionnallaatee.. allgather_ bize
500000000 Ice Tokcnizeer
tokenizer type
weight weight_decay
0.1
world. size
wow.iiiitssss.... gradients
FALSE
zem reduce bucker size
500000000
reduce_scatte
FALSE
zero_stage
zemn-opimmizaiionnlllee allgather hucketttthhhhhhhketteeee bucket_
500000000
48

---

Published as a conference paper at ICLR 2023
Table 12: The 74 datasets involved in Multi-task Instruction Pre-training (MIP). Datasets from TO- PromptSource (Sanh et al., 2022; Bach et al., 2022) are named in their Hugging Face datasets iden- tifiers. Datasets from DeepStruct (Wang et al., 2022a) are described Appendix C.2.
Task
Dataset
Task
Dataset
Coreference Resolution
super glwe/wsc..iiieee fixed
Multi-choie QA
cos jos_e/vl..11
Coreference Resolution
innogrande/winograadde super_glue/ch
Multi-choice QA
cosmos_qu
Natural Language Inference
Multi-choice QA
dream
Natural Language Inference
super_ slper_glue/rte
Multi-choice QA
openbookqa/main
Natural Language Inference anli
Multi-choice QA
qasc
Paraphrase Identification
glue/mrpe glue/qqp
Multi-choice QA
quail
Paraphrase Identification
Multi-choice QA
quarel
Paraphrase Identification
paws/labeled_ne final
Multi-choice QA
quartz
Closed-Book QA
ai2 _r__ar//RRC_ arr/ARe_Challenge
Multi-choice QA
race/high
Closed- Book QA
ai2_ S0__ar//ARC_BBB Easy
Multi-choice QA
race/middle
Closed-Bode QA
kill_ htspppps///
Multi-choice QA
seiq
Closed-Bood QA
trivia qa/unfiltered web qe__quesioos
Multi-choice QA
social i_qa
Closed-Book QA
Multi-choice QA
super_glue/boolg
Closed-Book QA
wiki_qa
Multi-choice QA
super_glue/multire
Extractive QA
adversarial addersarial_gg/dbiddaf
Multi-choice QA
wiki wiki_hop/original wiqa piqa
Extractive QA
adversarial gaaiaarriilg//eberrrreeeeaa
Multi-choice QA
Extractive QA
adversarial qa/droberra addersarrialqu/Alrbbbtta
Multi-choice QA
Extractive QA
duorc/SelfRC
Topic Classification
ag news dbpedia 14
Extractive QA
duorc/ParaphraseRCC
Topic Classification
Extractive QA
ropes
Topic Classification
trec
Extractive QA
squad v2
Word Sense Disambiguation
super glue/wic
Extractive QA
super glopr_glue/record
Dialogue State Tracking
multiwoz_22
Extractive QA
quoref
Event. Extraction
ace05
Sentiment
amazon polarity
Named Entity Recognition
conl103
Sentiment
app_reviews imdb
Named Entity Recognition
genia
Sentiment
Named Entity Recognition
ontonotes5.0
Sentiment
rotten_tomatoce
Named Entity Recogmition
acc2005
Sentiment
yelp_review__ full
Named Entity Recognition
con1104
Sentence Completion
super sypper_glue/cops
Named Entity Recognition
nyt29
Sentence Completion
hellaswag
Relation Extraction
conl104
Structure-to-Ted
common_
Relation Extraction
nyt29
Structure-to-TE
wiki bio
Relation Extraction
ace2005
Summarization
cnn smn_dailymmil/33002
Relation Extraction
kelm
Summarization
gigaword
Relation Classification
tacred
Summarization
multi news
Semantic Role Labeling
conl105
Summarization
samsum
Semantic Role Labeling
conll
Summarization
xsum
Semantic Role Labeling
propbank
49

---

Published as a conference paper at ICLR 2023
Table 14: Details results GLM--10B, GPT-3 175B (Brown et al., 2020), and PaI.M 540B (Chowd- hery et al., 2022) BIG-bench-lite in 0, and 3-shots. "Normalized preferred metric" is reported for each task. GPT-3 and PaLM's results are reported in BIG-bench's GitHub repository, and PaLM 540B's 3-shot results not found.
GLM-130B
GPT-3 175B
PaLM 540B
3
auto j__uuugguggnggggg
11.76
20.59
23.53
0.00
0.00
0.00
38.23
bbq_lite_jsoe
22.26
37.50
59.73
-8.33
40.75
61.21
-4.39
77.73
code_line_descrittinn
0.22
9.09
-8.64
9.09
9.09
9.09
0.22
49.00
canceptual_combinttinn
37.51
31.33
27.86
2.37
3.70
14.33
45.68
73.36
conlang translation
34.72
38.01
33.88
46.82
47.07
51.60
36.88
61.92
emoji movie
1.25
4.88
3.75
-10.00
-2.49
-1.24
17.50
88.75
formal_fallacies Lal__flaacesssylllle syllogisms_negatiie
0.83
1,46
0.35
1.00
6.80
5.60
-0.20
4,40
hindu_ hindu_knwwledee
32.23
37.56
34.52
10.15
40.61
44.42
41.37
93.15
known_unknownn
-4.5
0.00
4.35
21.74
4.35
0.00
13.04
34.78
sigguugg_idenifficttion
9.62
1.97
1.90
7.49
3.20
1.98
12.11
31.03
linguistics_puuzzi spazzzes_p
0,00
0.00
0.00
0.00
0.00
0.00
0.00
0.10
logic puzzle
9.88
13.66
5.24
0.16
0.01
16.12
deduction Jegicio_deduction
24.18
22.20
20.35
2.22
10.80
14.71
2.17
15.34
misconceptions_ russiann
-26.53
-46.94
-66.53
-34.70
-34.70
-30.61
-42.86
-30.61
novel_coneepts
6.25
21.87
25.78
33.59
33.59
45.31
33.59
49.22
operators
14.76
18.10
18.10
30.0
34.29
33.33
30.48
56.19
comprehension
7.14
7.72
11.58
0.00
0.00
0.00
9.46
44.40
play_
blay_dialog_aae dwwllligsamee_____
ses___smmerrrr_eeee ooree__r__aae or_fffefrennt
2.88
5.33
3.80
8.00
0.80
-5.40
-33.0
0.10
repeat_copy logic
0.00
0.00
0.00
0.00
0.00
0.00
0.00
37.5
strange_stor storiies
43.86
51.76
42.31
8.27
25.68
12.93
39.25
74.46
strategyqa
21.10
18.74
16.82
4.60
13.20
14.20
28.00
38.00
1,39
1.89
1.77
0.51
-0.63
2.77
0.76
2,40
vitaminc fact verification
71.87
60.72
56.55
-31.55
22.15
29.05
-28.85
55.60
winowhy
-3.49
5.38
3.0
3.0
10.60
13.00
-5.0
31.80
50

---

Published as a conference paper at ICLR 2023
Table 15:
Detailed results of GLM-130B and BLOOM 176B (Scao al., 2022) on
MMLU (Hendrycks et al., 2021). We find that existing literature has reported GPT-3 175B's numerical accuracy. BLOOM evaluated using Huggingface Transformer implementation.
Discipline
GLM 1L0B
BLOOM 176B
abstract algeeba
24.00
24.00
anatomy
48.90
38.52
astronomy
48.03
34.87
colledge biology
47.22
37.50
college_chemistry
34,00
19.00
Serpadde___omuurrrr_eeee scciene
44.00
1.00
colledge_mathematee
27.00
31.00
secledgee_pyysice
30.39
24.50
computer_see _ecuurtt
61.00
40.00
STEM
lonceptual_phyyeee
38.72
31.49
45.52
32.41
srinenttryymmathemattss
31.75
29.63
51.29
27.42
gegh_school_chemiste
34.98
27.09
sssinnccessseeeeeeeeeeeeeee science
53.00
30.00
high ______choo_oaheeeee
28.15
25.93
high school. physics
29.80
30.46
high school statistics
38.43
26.39
machine learning
40.18
29.46
econometrics
26.32
26.32
high_school school geoggaphy
53.54
36.36
high high_school. cechool_govermeete and politics
62.18
40.41
gegh_school_mmcroee
42.56
30.77
45,80
26.89
high_school__sycee psychology
54.13
39.27
Social Science
human human_sexuality
51.15
35.11
professional pssycnologppyyyyg
42.48
31,54
cublic_relation
55.46
33.64
studies
44,90
34.29
sociology
51.74
31.84
us. foreign policy
61.00
46.00
formal logic
27.78
23.02
high school european history
58.18
35.76
high high_school us history
58.33
40.69
gehh_schol_wwolke world_history
67.09
32,07
international_law
56.20
42.15
jurisprudence
43.52
35.19
Humanities
logical fallacies
57.06
31.29
moral piral_disputes
47.11
36.71
moral_scenarion
24.25
24.36
philosophy prehistory professional law world_religions
45.34
35.37
50.93
40.43
37.94
29.53
55,56
42.11
business ethics
51.00
34.00
clinical knowledge
48.68
35.85
colledge_mmdicheee
43.35
28.90
glocal_facts human_aging
35.00
23.00
45.29
32,29
management
56.31
27.18
Other
marketing
67.52
39.74
medical medical_genntice
48.00
45.00
miscellaneous
61.18
40.23
nutrition
50.65
32.35
professional accounting
35,46
28.72
iiftessional_mddiccne
43.38
18.01
virology
39.16
28.31
51

---

Published as a conference paper at ICLR 2023
E CONTRIBUTIONS
The GLM-130B project was conceived in Dec. 2021 with its pre-training part completed in July 3rd, 2022 and its evaluation and applications still ongoing. Over the course, have experienced various technical and engineering challenges (Cf. Appendix and Figure 21 for details). would not be possible reach its current status if without the collaboration of multiple teams-the Knowledge Engineering Group (KEG), Parallel Architecture & Compiler technology of Mobile, Accelerated, and Networked systems Group (PACMAN), and Natural Language Processing Group (THUNLP) Tsinghua University, as well as Al. The detailed contributions are listed below.
E.1 PREPARATION
Model Implementation: Aohan Zeng, Zhengxiao Du
Self-Supervised Data Processing: Ming Ding, Wendi Zheng
Multitask Data Processing: Xiao Liu, Xiao Xia
Model Architecture: Aohan Zeng, Xiao Liu, Zhengxiao Du, Hanyu Lai
Training Stability: Aohan Zeng, Xiao Liu, Ming Ding
3D-Parallelism and Training Efficiency: Aohan Zeng, Zixuan Ma, Jiaao He, Zhenbo Sun
E.2
MODEL TRAINING
Large-Scale Training & Monitoring: Aohan Zeng, Xiao Liu
Model Performance Validation: Aohan Zeng
E.3
POST TRAINING
Evaluation Framework: Aohan Zeng, Zhengxiao Du
Language Modeling Evaluation: Aohan Zeng
MMLU & BIG-Bench Evaluation: Aohan Zeng
CLUE & FewCLUE Evaluation: Xiao Liu, Aohan Zeng
Ethical Evaluation: Yifan Xu, Aohan Zeng, Xiao Liu, Zihan Wang
Baseline Evaluation: Xiao Liu, Jifan Yu, Weng Lam Tam
INT4 Quantization: Aohan Zeng. Zihan Wang, Xiao Liu, Hanyu Lai
Inference Acceleration: Zihan Wang, Aohan Zeng
Low-Resouree Inference: Gouyang Zeng, Xu Han, Weilin Zhao, Zhiyuan Liu
Demo and API: Hanyu Lai, Jifan Yu, Xiaohan Zhang, Yufei Xue, Shan Wang, Jiecai Shan, Hao- han Jiang, Zhengang Guo
Manuseript Writing: Xiao Liu, Yuxiao Dong, and Jie Tang wrote the main paper, and Xiao Liu, Aohan Zeng, and Zhengxiao Du wrote the Appendix.
E.4 PROJECT MANAGEMENT
Student Leaders: Aohan Zeng, Xiao Liu
Technical Advisors: Yuxiao Dong, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Jie
Tang
Project Leader: Jie Tang
E.5
COMPUTATION SPONSOR
GPU Sponsor: Zhipu.AI
52

---

Published as a conference paper at ICLR 2023
F A BRIEF HISTORY OF GLM- 130B
The GLM-130B project was conceived Dec. 2021 a brainstorming meeting at Tsinghua KEG. We firmly believe that it is of value to pre-train highly accurate language model, in particular for both Chinese and English. Though GPT-3 (Brown et al., 2020) is the pioncer for this effort, it is not available to most people in the world. In addition, it supports English only. We therefore decide to initialize the project 1LMM-300. Please note that the Wulao 1.75T model we built last year is a sparse model with 480 mixture-of-expert (MoE), rather than dense as GPT-3. Our goal then is to train a bilingual pre-trained dense model with high accuracy on downstream tasks, and to make it open to everyone in the world-anyone, any anywhere can download it and use it on single server with appropriate GPUs.
The ambitious project soon faced several important challenges:
Lack of computational resources: No organization is willing to sponsor such a big project and freely make it public.
Lack of a robust pre-training algorithm: Despite GPT-3's success on English corpus, it
unclear how to train high-accurate bilingual model for both English and Chinese.
Lack of fast inference solutions: Since the goal is have the model public everyone, we need to design fast inference solutions with low resource requirements to run the model.
For the pre-training algorithm, we finally chose GLM (Du al., 2022) due its high performance in practice. We eventually decided to train a GLM model of 130 billion parameters after several rounds of discussions and exploration, because such a size makes it possible run the inference on a single A100 (40G 8) server.
Our first attempt at training the model was in January 2022, shortly after we received small sponsor of GPUs for test running. However, we soon realized that we had significantly underestimated the technical difficulties of pre-training model at such a scale (>100B). It seems that pre- pre-training a highly accurate 100B-scale model is quite different from training a 10B-scale one. Due to frequent random hardware failures, model gradients exploding, unexpected excessive memory usage in the algorithm, debug for the 3D pipeline in the new Megatron and DeepSpeed frameworks, inability to recover from optimizer states, blocked TCP responses between processes, and many many unex- pected "bugs", the project was delayed for many times. The Tsinghua PACMAN team gave hand at this difficult time and together we successfully fixed most of the "bugs".
By March, were still short computational resources, but fortunately got chance to try test runs on several other platforms, including Ascend 910, Hygon DCU, NVIDIA, and Sunway. The immediate challenge was for to adapt our training code to these different platforms, as the under- lying operators are quite different. Also, it introduced many new issues: the element-wise operators not supporting fast computation for large-dimension vectors, various issues that hindered conver- gence- the large gradient norms of input embeddings, native Post-LN, Pre-LN, and Sandwich-LN. dataloader state seeds, and computation precision choices in Softmax and Attention as well as numerous mistakes we ourselves made. With tremendous help from all of our generous partners, we finally succeeded in making our pre- pre-training algorithms runnable across all the platforms- ffannlly surprising achievement for this project. The timeline of GLM-130B in Figure 21 covers most of the issues we have encountered and addressed as of this writing.
On April 26th, we received a generous computing sponsorship from Zhipu.Al- - an Al startup that aims to teach machines to think like humans. After another week of testing, we finally kicked off the training of the GLM-1300 130B model on its 96 A100 (40G 8) servers on May 6th. Additionally, Zhipu.Al also sent a team to help evaluate the pre-trained model and build a demonstration website.
The training period spanned two months, during which we began developing a toolkit to allow GLM- 130B's s inference in low-resource setting with swapping technique and quantization. Though it is already the most accessible model of its scale, together with our partner from Tsinghua NLP, we have been exploring the limit of popularized hardware platforms, which would truly make the 100B-scale model accessible to many people as possible. To date, managed to reach the INT4 weight quantization for GLM-130B. Importantly, the INT4 version of GLM-130B without post training
16ts section is largely extracted and updated from the blog introduction of GLMB133B http: //eg. cs t ngghaa edu. on/ /1 130b/ (Posted date: August 4, 2022).
53

---

Published as a conference paper at ICLR 2023
& Major lssues Encountered for Training GLM-1300
2021.12
The "THZ" (1008) project towards an open dense pre-trained GLM at 100B scale conceived
Survey pre-training strategies of existing models of similar scale, such as GPT-3, Gopher s> Limited public info about how they were trained and issues they met
Search for possible GPU clusters & sponsors
2022.1
Test performance FP16/FP32 1008 scale on one testing cluster
Unexpected excessive memory usage in GLM => Torch is better with fixed length Input sequences
Inability converge and try tricks from CogView
Use Sandwich- Frequent random hardware failures => Have to run HCPG test before each run
2022.2
Very slow training speed than previously calculated Optimize kernels and fuse operators Find the input shape critical to kernel performance
Collect pre-training corpora tokenize icetk: the sentence piece set to unigram mode
Debug pipeline parallel in rewlasdl Megatron and DeepSpeed
2022.3
can't recover perfectly from checkpoints customized dataloader do
state seed
properly in distributed training
The memory per processor
small => Require too many pipeline stages => Botch size is too large (up to
12,000) Horm the model's convergency
It can't launch more than 2,000 computing nodes => Overcome this and support 1.000-node training by
tuning Linux kernel TCP parameters
Collect data for multi-task instruction pre-training
Receive opportunities to test trainings several other clusters
Very slow training speed than expected The underlying element wise operators don't support fast computation rigeedimmnsion vectors.
2022.4
Optimize A100 kemel's computing efficiency A100 kernels prefer square-shaped inputs, and
seq jen=2,048 is optimal for hidden-state dimension (12,288)
Inability converge due large gradient norms (170+) of input embeddings Try embedding and
gradient shrink, which turn out almost equivalent
Nalve post-LN pre-LN disconverges after several thousands of steps Try Sandwich-LN with PB-Relox
It still disconverges after one week's 's trial => The dataloader state seeds are not unified for different pipeline
stages, resulting in mismatch of input data and labels.
Test two positional encodings: RoPE and Alibi => Alibi can be slower it requires element-wise
monipulation on attention matrices--canging num_ heads *2,048 2,048 scolars per loyer
Test GeGLU and GAU => GAU converges faster with relatively poor performance fine tuned SuperGLUE
Abnormal GPU memory usage of newly-added functions and classes DeepSpeed hardcodes the function
names for checkpoint activation
Decide to train GLM with 130 billion parameters => allow inference on DGX-A100 40G node
2022.5-6
Implement a RoPE cuda operator in C++ => See unexpected precision errors and finally have it obondoned
Sandwich-LN still disconverges => 1) Reducing learning rate does not help; 2) Using Hinge cross-entropy
becomes slower and harms performance; 3) Shifting to DeepNorm still disconverges
Uuse FP32 in softmax of attention => Success
Find PB- Relax unnecessary for FP32 softmax => It also slows down training as it needs to manipulate the whole
attention score matrices
Experience few spikes in later training => 1) Reduce gradient shrink factor from to 0.1: useful: 2) Reduce the learning rate: sometimes useful; 3) Jump the noisy data botches: sometimes useful
Find mistake in multi- task data after training for 20,000 steps => Use the correct dota but it does not forget
2022.6- 6-7
Adapt the pipeline parallel checkpoints to ordinary parallel checkpoints for efficient inference on single A100
Work on evaluation scripts on datasets: MMLU, Big- big-bench, CLUE, SuperCLUE, etc.
Implement P-Tuning and P-Tuning v2 for parameter-efficiest tuning on GLM-130B tuning on SuperGLUE
Work with BMInf on adapting GLM-130B to perform inference on a single V100 or 3090 => Use pipeline-style asynchronous swapping between main memory and GPU memory
Pipeline-style fails due too many plpeline stages => Find that data parallel can not be introduced for fine-tuning => Use 32-way model parallel for linettunigg with reasonable performance
Try to fine-tune GLM-130B with fewer A100 nodes (l.e., 12-16 nodes) =>
Figure 21: The timeline of major issues that training GLM-130B encountered and addressed, of July 31st, 2022.
54

---

Published as a conference paper at ICLR 2023
faces negligible performance degradation compared to its uncompressed original. while it consumes only 25% of the GPU memory required by the uncompressed version, thus supporting its effective inference on 4 x RTX 3090 Ti (24G) or 8 RTX 2080 Ti (11G). We will attempt to further reduce the resource requirements and keep the community updated on this important working item.
G BROADER IMPACT
This paper introduces open bilingual pre-trained language model with 130 billion parameters.
Currently most pre-trained language models with over 100 billion parameters are privately owned by governments and large corporations (Brown et al., 2020; Thoppilan et al., 2022; Rae et al., 2021; Chowdhery et al., 2022; Wang et al., 2021). I). A few of them (Brown et al., 2020; Lieber et al., 2021) provide limited inference APIs with fees. In contrast, the weights and code of GLM-130B to anyone who interested in L.Mss. Moreover, we significantly lower the hardware requirements for inference by speed-up implementation and INT4 quantization. The paper have broader impact on the research community, individual developers and small companies, and society.
open
G.1 IMPACT ON AI RESEARCH
Most research institutions cannot afford the substantial cost of pretraining large language models. As result, most researchers, except employees of governments and large corporations, only have access to the limited inference APIS with fees. With the inference APIs, researchers can only analyze the outputs of models as black boxes, which limits the scope of potential work. With GLM-130B, researchers can analyze the model parameters and internal states corresponding to specific inputs, leading to in-depth studies of LLMs' theory, capacity, and flaws. Researchers can also modify the model architecture and weights, to validate the proposed algorithms to improve LLMs Zhu et al. (2020); Cao et al. (2021); Hase et al. (2021): Mitchell et al. (2022).
With 1NT4 quantization, GLM- 130B can perform inference popularized GPUs such 4 x RTX 3090 or 8 x RTX 2080 Ti, which can be easily accessed from cloud service. As a result, researchers who cannot afford powerful data-center GPU servers like DGX-A100 can also utilize GLM-130B.
G.2 IMPACT ON INDIVIDUAL DEVELOPERS AND SMALL COMPANIES
Currently, individual developers and small companies who want to integrate LLMs into their busi ness can only choose paid inference APIs. The increased cost can hinder their attempts. Instead, GLM-130B can be deployed on popularized hardware that they to reduce the cost. Furthermore, they can utilize distillation techniques Sanh et al. (2019); Jiao et al. (2020) to obtain smaller models that preserve comparable performance on their specific tasks. While some developers may lack the ability to complete deployment and distillation on their own, we be- lieve with GLM-130B and more open LLMS the future, the corresponding toolkits and service providers will become more available.
access via cloud service
We also note that currently most applications of LLMs are based on prompt engineering, partly due to the limitation of inference APIs. In downstream scenarios such as online customer service, the companies accumulate huge amounts of human- generated data that contain domain knowledge. With the open-source weights and code, developers can finetune MLLM-130B on their own data to mitigate the gap of domain knowledge.
G.3 SOCIAL IMPACT
Large language models, together with other machine learning models in different modalities (e.g., Image (Ramesh et al., 2021; Ding et al., 2021; Saharia et al.) and Video (Hong et al., 2022)), could be used to generate synthetic text for harmful applications, such as telemarketing fraud, political propaganda, and personal harassment as is discussed in (Weidinger et al., 2021; Sheng al., 2021; Dev et al., 2021). We do not anticipate any hazardous outputs, especially towards vulnerable and historically disadvantaged groups of people, after using the model.
While some people think that restricting access to LLMs can prevent such harmful applications, we argue that promoting L.M inclusivity can lead to better defense against potential harm caused by
55

---

Published as a conference paper at ICLR 2023
LLMs. Currently, only governments and large corporations afford the considerable costs of pre- training LLMs. There is no guarantee that organizations having the substantial financial resources to pretrain LLM will not do harm with it. Without access to such LLMs, individuals cannot even realize the role of LLMs in harm. Conversely, releasing an open L.M can provide access and transparency to all the researchers and promote the research to reduce the potential harm of L.Mss, like algorithms to identify the synthetic text Gehrmann et al. (2019) or detect fake news Li et al. (2021).
Also, it is known that LLMs can suffer from problems in fairness, bias, privacy, and truthful- ness Zhang et al. (2021); Lin et al. (2022); Liang al. (2021); Bender (2021). An open LLM can reveal the model parameters and internal states corresponding to specific inputs instead of providing APIs to black-box models. conclusion, researchers can conduct analysis of LLMs' flaws in depth and propose improved algorithms to solve the problems.
H ENVIRONMENTAL IMPACT
One of the major concerns about large language models their huge energy usage and associated carbon emissions Strubell et al. (2019); Lacoste al. (2019); Patterson al. (2021); Bender et al. (2021). GPT-3 was estimated to use 500 tons of carbon emissions footprint (CO2eq) Patterson et al. (2021). We consumed fotal of 442.4MWW of electricity over the 60-day course of training. Given the 0.5810 kg/k Wh carbon efficiency of local power grid, the pre-training released 257.01 metric tons of CO2. This is around half of GPT-3's carbon footprint, probably due the efficient parallel strategies and NVIDIA's hardware improvements. The carbon emission is roughly the equivalent of the yearly emissions of 18 average Americans. However, we believe that with GL. 130B released, more carbon emissions for reproducing 100B-scale LLMs can be saved.
56


Reviews:
[{"review_id": "i0zFEGvp_7", "reviewer": "Reviewer_tHby", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "It's always hard to summarize papers that report on such a big project, but fundamentally this is a tech report of training an English+Chinese LLM with techniques that improve on the state of the art for pretraining methodology and training stability.\n\nGLM-130 is concurrent with OPT and BLOOM, two other GPT-3-class LLMs that were released openly this year. It has d_model=12288 (same as GPT-3, OPT, and BLOOM) and 130B parameters and was trained on 200B tokens each of English and Chinese, including English NLU tasks formatted for multitask instruction pretraining to increase zero-shot performance on other, unseen English and Chinese tasks. It used a GLM objective that mixes span-by-span span-filling and prefix-LM, enabling use of bidirectional attention at inference time to further improve zero- and few-shot performance.\n\nIt consistently outperforms GPT-3, OPT, and BLOOM on English despite using fewer parameters and less compute (also less English data than GPT-3/OPT), and it sometimes also outperforms the much larger PaLM. It also consistently outperforms ERNIE TITAN 3 on Chinese, despite fewer parameters, less compute, and less Chinese data. As it can run in both bidirectional and unidirectional mode at inference time, ablating this shows that bidirectional attention contributes greatly to its outperformance. It also exhibits less bias/toxicity than other LLMs.\n\nIn terms of optimization methodology, it uses relatively standard techniques including AdamW with LR warmup and cosine decay, and also uses early batch size warmup to a relatively large 8.65M token total batch size. For training stability, it uses post-LN with depth-based initialization (\"DeepNorm\"), downscaled gradient in embeddings (\"Embedding Gradient Shrink\"), gradient clipping, and dropout. Either the DeepNorm or something else about the model makes the weight distribution especially narrow, enabling near-lossless int4 weight quantization for small-footprint inference.\n\nIn terms of training efficiency, it achieves about 32.5% MFU at steady state [135e12/312e12 * 3/4 for remat] and 25.1% MFU end to end [(130e9*400e9*6)/(96*8*312e12*60*60*24*60)] using 8 pipeline stages with a PipeDream-Flush (aka 1F1B) schedule, 4-way tensor parallelism, and 24-way (2-way intranode and 12-way internode) data parallelism. The authors also provide an efficient FasterTransformer-based inference setup that supports consumer GPUs.", "summary_of_the_review": "I think this really represents the gold standard in reproducibility for LLM publications, as well as the most capable open-source English or Chinese language model in existence.", "strength_and_weaknesses": "Strengths:\n- Very careful attention to training stability\n- Comprehensive details\n- Accessibility to smaller scale researchers\n- Open source\n- General openness even when admitting something could have been done better (e.g. \"We later found the instructions to implement two-dimensional RoPE from its author’s blog https://kexue.fm/archives/8397, but our training has proceeded for weeks.\")\n\nWeaknesses:\n- The work chooses optimization hparams like initial LR and LR schedule relatively unsystematically.\n- They use dropout without explaining why, even though there's no repeated data in the dataset\n- The precision discussion mentions avoiding bf16 because of fp32 gradient accumulation, but fp16 or even bf16 accumulation is typically fine in practice.\n- There isn't really an explanation of why GLM-130 has narrower weight distribution than BLOOM (my guess is it's related to DeepNorm)?\n- The paper doesn't explicitly state that the Pile test set was held out of the training set, but I'm assuming it was.\n- Figure 7 leaves out PaLM 1- and 5-shot BBL, which are much better than 0-shot, and Chinchilla (which was only evaluated at 5-shot). Those are reasonable choices, but I'd prefer to err on the side of including more data points. Table 4 also leaves out PaLM and Chinchilla 5-shot.\n- no explanation of how 135 TFLOPS was computed (in particular, it probably includes remat but this isn't stated). Would recommend adopting MFU/HFU from PaLM paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NBMKlFv8O55", "reviewer": "Reviewer_tevQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper trains a new 130B parameter model with the GLM architecture/objective, with contributions in automatically stabilizing spikes with existing techniques, focussing on keeping inference costs and requirements low and therefore the model accessible to a large number of people. It is an achievement to beat GPT3 numbers and in a few cases PaLM numbers.", "summary_of_the_review": "Weak Accept owing to a severely limited evaluation section and methodology, the work itself pulls a lot of techniques together to get a stably pre-trained O(100B) model with GLM objective and stability techniques which is very valuable to get broad recognition.", "strength_and_weaknesses": "Strengths:\n - Impressive set of techniques put together for stability and model quality (DeepNorm initialization, GLM instead of CausalLM objective, Embedding Layer Gradient Shrink)\n - Impressive performance on measured benchmarks esp compared to GPT3 & in a few cases PaLM\n\nWeaknesses\n - Would have loved to see some ablations, especially over the first few ~100B tokens as to which technique contributes how much to performance, either within GLM-130B or implemented across models, i.e. for example with T5 style models the GLM objective would be essentially T5’s denoising task + prefixLM. Otherwise it is not clear what technique is responsible for how much of the gains.\n - More evaluations: The evaluation section is severely limited\n   - It could have evaluated fine-tuning over a few tasks (Ex: PaLM has both few-shot numbers and fine-tuning numbers for a few benchmarks like SuperGlue and TyDiQA) — this is understandably partly a resource issue.\n   - More extensive few-shot evaluations on say SuperGlue, Winograd style tasks, CBQA style tasks, CommonSense reasoning, Chain of Thought prompting etc — Since inference is both fast and cheap this shouldn’t be too much of an issue and would increase my confidence of this method’s applicability and more importantly in the released checkpoint.\n   - Not evaluating on tasks with fixed labels (like NLI) seems like severely and unnecessarily restrictive, i.e. I don’t think Xian et al 2018’s advice applies here, one can/should do NLI tasks with encoding the label in the prompt, while not giving an example (for zero-shot) or giving a few examples of each class (in a few-shot setting) and evaluating. Since MultiTask Instruction Pre-training (MIP) is happening here, I think it is fair to only exclude tasks that are substantially similar in the actual example content, if the task type seen is similar (say classification with the same labels) this should be called out and reported separately.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AQ4Qjmvqys0", "reviewer": "Reviewer_XkYq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces GLM-130B, a bilingual English and Chinese LLM. The model is trained for 400B tokens (half on English  and half on Chinese). The model follows a bidirectional attention architecture with the self-supervised blank infilling objective. They also include a small percentage of MIP tokens. Int4 quantization enables inference on a diverse set of platforms. Various model instabilities and mitigation strategies are discussed.\n\nThe model, training log and inference through free APIs are provided and an effort is made to make the entire work reproducible.\n\nDetailed comments:\nTable 4: You state that the reason for PaLM's 1-shot ability being so much better than GLM is the diverse dataset. Could you elaborate a bit more?\nSection 6, subsection name \"Transferring\" --> consider using a better subsection name.", "summary_of_the_review": "Overall, the paper is clearly written and details the building of the GLM-130B bilingual LLM. The model is competitive with other similar sized models and care has been taken to make the model set up reproducible. The evaluation suite for English could have been more comprehensive to substantiate the claims a bit more.  I would have also liked to see more experiments at a smaller scale and studies of emerging capabilities of the model at scale. Since this is a bilingual model, studying how much transfer or interference happens between these two languages would have been interesting.\n\n", "strength_and_weaknesses": "Strengths:\n- The model is more reproducible than other similar sized counterparts.\n- The model's performance is competitive on the benchmarks that are evaluated on and outperforms large models like GPT-3 and PaLM.\n- Quantization for inference and mitigation strategies like deepnorm and embedding shrinkage for training instability are useful for the community.\n\nWeaknesses:\n-  I would have liked to see more English tasks evaluated on. The current set of results on the LaMBADA dataset a lot. I would like to see results on more GPT-3 tasks or SuperGLUE tasks for English.\n- More results at different scales and projection of trends or emerging capabilities as scale increases would have been interesting. The current set of results are missing this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "13TgZvHAY6", "reviewer": "Reviewer_asnf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces and open-sources one of the largest 100B bilingual pre-trained language model GLM-130B. It covers a detailed pre-training process (training strategies, design choices) and the resulting model outperforms several large language models on English and Chinese zero-shot understanding benchmarks. The author also investigates the efficient inference of the GLM and show the efficacy of using Int4 quantization. \n", "summary_of_the_review": "It is not surprising that a large-scale bi-directional language model will outperform the causal counterpart on a range of zero-shot language understanding tasks. The detailed engineering efforts and design choices make this paper valuable, though my main concerns are what factors make the performant GLM-130B unclear, as well as NLG / few-shot performance. \n", "strength_and_weaknesses": "GLM-130B: AN OPEN BILINGUAL PRE-TRAINED MODEL\n*Paper Summary\nThis paper introduces and open-sources one of the largest 100B bilingual pre-trained language model GLM-130B. It covers a detailed pre-training process (training strategies, design choices) and the resulting model outperforms several large language models on English and Chinese zero-shot understanding benchmarks. The author also investigates the efficient inference of the GLM and show the efficacy of using Int4 quantization. \n\nSummary Of Strengths\n- The paper is well-written and clearly presented;\n- It detailed the efforts made during the pre-training process for the GLM-130B, making it very valuable for practitioners in the large language model pre-training area; (though several design choices might not be feasible to ablate at that scale.). \n- The resulting model was tested across benchmarks, and showed surprising robustness towards quantization, making it more accessible for academia. \n\n\nSummary Of Weaknesses\n- Why the bi-lingual GLM performs a lot better in LAMBADA as shown in Figure 2? Is this also true for MMLU, Pile, and NLG tasks, any intuition about this?\n- Why not compare to OPT-175B models for quantization? \n- Intuition why the few-shot performance of GLM-130B in Table 4. falls behind other methods? Also, will the quantized GLM-130B yield similar few-shot performance?\n- Which GPT-3 175B is the GLM-130M compared to? Is it a fairer comparison to the InstructGPT variants?\n- How will the author attribute the performance gain (Bi-directional modeling [1], Multi-Task Instruction Pre-Training [2], or the Bilingual corpus and model architecture)?\n- It is not surprising that the Bi-directional model will outperform the Uni-directional model in understanding tasks. How are the generation tasks performance of GLM-130B compared to uni-directional counterparts?\n\n\n[1] Tay, Yi, et al. \"Transcending Scaling Laws with 0.1% Extra Compute.\" arXiv preprint arXiv:2210.11399 (2022).\n\n[2] Chung, Hyung Won, et al. \"Scaling Instruction-Finetuned Language Models.\" arXiv preprint arXiv:2210.11416 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster