Binary and Ternary Natural Language Generation
Year: 2023

Abstract:
Ternary and binary neural networks enable multiplication-free computation and promise multiple orders of magnitude efficiency gains over full-precision networks if implemented on specialized hardware. However, since both the parameter and the output space are highly discretized, such networks have proven very difficult to optimize. The difficulties are compounded for the class of transformer text generation models due to the sensitivity of the attention operation to quantization and the noise-compounding effects of autoregressive decoding in the high-cardinality output space. We approach the problem with a mix of statistics-based quantization for the weights and elastic quantization of the activations and demonstrate the first ternary and binary transformer models on the downstream tasks of summarization and machine translation. Our ternary BART base achieves an R1 score of 41 on the CNN/DailyMail benchmark, which is merely 3.9 points behind the full model while being 16x more efficient. Our binary model, while less accurate, achieves a highly nontrivial score of 35.6. For machine translation, we achieved BLEU scores of 21.7 and 17.6 on the WMT16 En-Ro benchmark, compared with a full precision mBART model score of 26.8. We also compare our approach in the 8-bit activation setting, where our ternary and even binary weight models can match or outperform the best existing 8-bit weight models in the literature. Our code and models are available at: https://github.com/facebookresearch/ Ternary_Binary_Transformer.

heading: 1 Introduction
text: Generative pre-trained transformers (Brown et al., 2020; Lewis et al., 2020; Radford et al., 2018) have emerged as powerful and generic tools, driving breakthroughs not only in language understanding but the field of AI in general. These models owe
∗Equal contribution
their success mainly to their seemingly infinite ability to scale to ever-larger data and model sizes. Unfortunately, such scaling comes at the cost of large computational requirements, putting extensively large generative transformers out of reach of all but the most resource-rich institutions. Even moderately sized pre-trained transformers have limited applications due to their size and computational cost. Making generative transformers more efficient is imperative for widening their use to more devices and practical applications.
In this work, we explore making generative pretrained transformers more efficient via the quantization of their weights and activations. Quantizing the weights of a neural network is useful for compression and allows the model to be stored more efficiently. However, compression alone does not reduce computation costs since the network’s activations need to be computed in full precision. Quantizing both weights and activations allows computation to be performed with lower precision, potentially leading to significant efficiency gains depending on the quantization level and hardware implementation. Quantizing neural networks have a long history, and multiple works have attempted to quantize pre-trained transformers at various quantization levels (Shen et al., 2020; Zhang et al., 2020; Liu et al., 2022; Qin et al., 2021). Most of this work focuses on encoder-only models (mainly BERT) for sentence and token classification tasks. Quantizing text generation models has generally been regarded as a more difficult task (Behnke et al., 2021; Tao et al., 2022) due to the large output vocabulary and sequential decoding. Recent work has tackled this problem, though only for mild quantization levels (down to 8-bit activations) and with mixed success.
In contrast, we are interested in very low-bit quantization, down to ternary and even binary weights and activations. In order to achieve this, we combine and unify best practices for weight and activation quantization and present a frame-
65
work that uses gradient-matching quantization for weights and elastic quantization for activations. We apply our method to natural language generation tasks and, for the first time, demonstrate low-bit generative transformers of competitive accuracy. Our ternary (weight and activation) model lags a full-precision BART (Lewis et al., 2020) model by only 4 points in ROUGE on the XSUM summarization dataset. In contrast, our model with ternary weights and 8-bit activations comes within 1 point and even outperforms comparable state-of-the-art models with 8-bit weights. We also demonstrate a fully binary (weights and activations) model. While not as competitive, it is able to achieve a highly non-trivial ROUGE-1 score of 31.7.
Our results also extend to machine translation models. On the WMT16 En-Ro benchmark, we quantize an mBART model to extend the ternaryweight 8-bit activation SoTA by 1.2 points while demonstrating fully ternary and fully binary translation models for the first time.
We summarize our contributions as follows: • We propose a novel combination of statisticsbased weight quantization with learning-based activation quantization, which enables stably training transformer encoder-decoder models to converge in the fully ternary/binary settings, which was not previously possible. • We significantly improve the state-of-the-art text generation models in the 8-bit activation and ternary/binary weight settings while setting the first non-trivial baselines for the fully ternary and fully binary settings.

heading: 2 Method
text: In this section, we first introduce the previous practices in binarization and ternarization. Then, we introduce a unified statistic-based weight binarization / ternarization method that can alleviate the gradient mismatch issue and enhance the quantized weights entropy. Lastly, we analyze the difference between weight quantization and activation quantization and propose an elastic ternarization method for activations. We abbreviate our method as TBT, short for “Ternary / Binary Transformer”.

heading: 2.1 Preliminary

heading: 2.1.1 Ternarization
text: Ternary neural networks, where real values are quantized to three levels, are first introduced in (Li et al., 2016). Thus, these values can be repre-
sented in 2 bits, leading to a 16× reduction in size and computation. Moreover, the computations can be calculated multiplication-free, leading to even further computation gains on suitable hardware. The recent work integrates the ternarization algorithm in natural language models for quantizing the weights and activations in classification tasks (Zhang et al., 2020) and ternarizing the weight (8- bit activations are used) in generative models (Li et al., 2022; Tao et al., 2022). The general formula (Li et al., 2016) for ternarization is as follows:
XiT =    −αT , if XiR < −∆ 0, if −∆ ⩽ XiR ⩽ ∆ +αT , if X i R > ∆
(1)
∆ = 0.7 · ||XR||l1
nXR (2)
αT =
∑ iX
i R · 1|XiR|>∆∑ i 1|XiR|>∆
(3)
Here XT denotes the ternary weights/activations, and XR represents their real-valued counterparts. nXR denotes the total number of elements in the tensor. ∆ is the ternary threshold, and αT is the scaling factor that minimizes l2-loss between XT and XR.

heading: 2.1.2 Binarization
text: The neural network binarization denotes representing the weights and/or activation with bi-level values. It is first proposed in BNN (Courbariaux et al., 2016) and has evolved in the follow-up works (Rastegari et al., 2016; Liu et al., 2018). Rastegari et al. (2016) formulates binarization as: XiB = αB ·Sign(XiR) = { −αB , if XiR < 0 +αB , if X i R ⩾ 0 (4)
αB = ||XR||l1 nXR
(5)
Here XB can represent binary weights or binary activations. αB denotes the scaling-factor that minimize the l2 loss between XR and αB ·Sign(XR).
The acceleration and compression effect of ternary/binary neural networks is significant. By representing the weights and activations with {−1, 0, 1}, the network enjoys ∼16× memory saving compared to its 32-bit floating-point counterpart. When further binarize the weights and activations to only 1-bit (i.e., {−1, 1}), up to 32×
model-size reduction and 58× speedup on CPUs have been achieved (Rastegari et al., 2016), where the matrix multiplication operations are replaced with light-weighted bitwise XNOR operations.
Despite its appealing characteristics, naively binarizing or ternarizing the transformer model for natural language generation results in several accuracy drops or even a total failure in training. It has been observed that the attention layers of the transformer network are difficult to quantize to low bits. Also, the auto-regressive decoding tends to accumulate errors due to quantization. Given the nature of generative language networks that require highprecision output, quantizing both the activations and weights in these models to extreme bit values is non-trivial and has not been explored before.

heading: 2.2 Stats-based max-entropy isometric weight quantization
text: We propose a statistics-based method for weight binarization/ternarization. Particularly, this novel quantization method considers maximizing the entropy of the quantized weights and reducing the gradient mismatch in the backward pass. Previous works (Courbariaux et al., 2016; Bai et al., 2021b; Zhang et al., 2020) are mainly focused on minimizing the l2 loss between the quantized weights and the real-valued weights to find the optimal quantization scheme,
α∗ = argmin ||αŴQ −WR||l2 (6)
where ŴQ denotes binary/ternary weights and α∗ denotes the optimal scaling factor calculated. Despite the broad application and great success of the classic quantization scheme, we found that merely minimizing the l2 loss neglects several critical but intractable issues in ultra-low-bit weight quantization: (1) The information entropy of the quantized weights is not considered. Eq. 1 and Eq. 4 calculate the quantized weights to minimize the distance to the real-valued weights, which could lead to imbalanced quantized weight distribution and harm the quantized weights representation capacity. (2) The quantization function Eq. 1 and Eq. 4 are not isometric, meaning that it does not consider the magnitude consistency between the quantized weights and real-valued weights, while we find that magnitude consistency contributes significantly to accurate gradient estimation.
Considering the above two limitations in previous solutions, we are motivated to design a novel
quantization function that enhances information entropy and reduces gradient mismatch. To boost the weights representation capability, in information theory, more information is preserved when the quantized weights contain higher entropy:
max pi
H = −pi log(pi), s.t. N∑
i=1
pi = 1 (7)
with pi denoting the proportion of real-valued weights being quantized to ith quantization level in total N levels. Eq. 7 can be easily solved with a Lagrange multiplier, and the optimal p∗i = 1 N , i ∈ {1, 2, . . . , N}, suggesting the best quantization scheme to preserve maximum information entropy is to distribute the real-valued weights in all quantization levels as evenly as possible.
For reducing the gradient mismatch, as suggested by the previous binarization work (Liu et al., 2020b), the magnitude difference between the quantized weight and the real-valued weight will greatly influence the gradient scale and a mismatch in magnitude will be amplified in back-propagation and cause gradient vanishing or explosion during training. Thus it is important to ensure the magnitude of real-valued weights and quantized weights are consistent.
Combining two requirements discussed above, we proposed max-entropy isometric weight quantization. In ternarization, it is formulated as
WiT = αT⌊Clip( WiR − µT
αT ,−1, 1)⌉
where µT = WR,
αT = 4 3 · ||WR − µT ||l1
nWR
(8)
Where WT and WR refer to the ternary weights and real-valued weights, respectively. The rounding function ⌊·⌉ and Clip(·) function quantize weights to {−1, 0, 1}. µT is the mean of realvalued weights and nWR denotes the number of weights in the weight matrix. Scaling factor α is calculated from the weight statistics and follows the entropy rule to scale the real-valued weight WR to be evenly distributed in quantization levels. In the ternary case, the weights are quantized to {−αT , 0, αT}. When the real-valued weights are initialized as uniformly and symmetrically distributed (He et al., 2015; Glorot and Bengio, 2010), the scaling factor αT will distribute
WiR α T
to [−1.5, 1.5], such that the output ternary weights
will have near uniform distribution in three ternary levels. Meanwhile, Eq. 8 is an isometric mapping where the real-valued weights are scaled by 1α
T to
near [-1, 1] and time αT to scale back after quantization. In this way, the magnitude is preserved.
Correspondingly, in the binary case we have,
WiB = αB · Sign( WiR − µB
αB )
where µB = WR,
αB = ||WR − µB ||l1
nWR
(9)
Here WB denotes the binary weights, where substracting the average µB makes the realvalued weight zero-centered before binarization and thus encourages an even distribution in binarized weights. Then the scaling factor αB matches the magnitude between real-valued and binary weights. Particularly, in Eq. 9, WiB = αB · Sign( WiR−µB
α B ) = αB · Sign(WiR − µB), we explicitly include the αB in the denominator to keep the binarization function isometric and the gradients w.r.t. weights can be calculated straight-
forwardly as:
∂WiB ∂WiR STE≈ 1 |W i R −µ B
α B
|<1 (10)
STE is abbreviated for straight-through estimator (Bengio et al., 2013), which replaces the nondifferentiable Sign function with Clip function in the backward pass. We show that the proposed maxentropy isometric weight quantization improves the accuracy of weight binarization / ternarization by 6.0 / 11.53 RougeL scores on the CNN/DailyMail benchmark, respectively. More details can be found in Sec. 3.2.

heading: 2.3 Learning-based activation quantization
text: In contrast to neural network weights that are stored on the disk, activations are calculated on-the-fly. The distribution of activations in a particular layer depends on the network weights as well as the corresponding input sequence, and thus varies from batch to batch. In order to have the quantization function better capture the underlying activation distribution, we propose learning-based activation quantization.
Inspired by BiT (Liu et al., 2022), we divide the activation layers into two categories: the activation
layers with non-negative values (XR ∈ R+), i.e., Softmax/ReLU layer outputs and the rest of the layers with both positive and negative activations (XR ∈ R). We binarize / ternarize the first activation category (XR∈R+) to {0, α} / {0, α, 2α}, and symmetrically quantize the later activation category (XR ∈ R) to {−α, α} and {−α, 0, α} in binary and ternary cases respectively. In this way, the activation distribution matches the original fullprecision activations and thus reduces the quantization error. Further, we learn to scale the real-valued activations to better fit quantization thresholds, and this learnable scaling factor can be updated endto-end with the gradients from the network loss to better account for overall network optimization.
In the ternary case, we propose the elastic ternarization function formulated as,
XiT = αTX̂ i T
=    αT⌊Clip( XiR α T , 0, 2)⌉, if XR∈R+
αT⌊Clip( X′iR α T ,−1, 1)⌉, if XR∈R
(11)
where XR and XT denote real-valued and ternary activations, respectively. To keep the formula concise, we set X′R = XR −XR, denoting the zeromean real-valued activations. αT is the scaling factor. Different from the weight quantization, the scaling factor in Eq. 11 is learned with the gradient update. We follow the practice in (Zhou et al., 2016; Esser et al., 2019) to calculate the gradients with straight-through estimation (STE) bypassing the non-differentiable rounding function:
∂XiT ∂αT STE≈    X̂iT − XiR α T ·10⩽XiR⩽2αT , if XR∈R+
X̂iT − X′iR α T ·1|X′iR|⩽αT , if XR∈R
(12)
The learnable scaling factor can dynamically adapt to different activation distributions and improve the ternarization accuracy. In the binary case, it is formulated as.
XiB = αBX̂ i B
=    αB⌊Clip( XiR α B , 0, 1)⌉, if XR∈R+
αB · Sign( X′iR α B ), if XR∈R
(13)
Here XB denotes the binary activations.
Correspondingly, the gradients w.r.t. the scaling factor α can be easily calculated as
∂XiB ∂αB STE≈    X̂iB− XiR α B ·10⩽XiR⩽αB , if XR∈R+
Sign(X′iR), if XR∈R
(14)
We demonstrate that with the learning-based activation quantization method and statistics-based weight quantization scheme, the proposed TBT for the first time is able to quantize the BART model for natural language generation tasks to ternary and even binary weights and activations, and achieve reasonable accuracy on summarization and translation benchmarks.

heading: 3 Experiments
text: In this section, we evaluate the effectiveness of our low-bit quantization scheme for natural language generative model on text summarization benchmarks: CNN/DailyMail (Nallapati et al., 2016) and XSUM (Narayan et al., 2018). We additionally experiment on the machine translation task with mBART on WMT16 English-Romanian (En-Ro) dataset (Bojar et al., 2016a).

heading: 3.1 Experimental settings
text: We follow recent work (Li et al., 2022) in training the quantized network with initialization and knowledge distillation from a full-precision pretrained model. Specifically, we use the BARTbase (Lewis et al., 2019) as our full-precision baseline for summarization tasks and mBARTlarge (Liu et al., 2020a) for the translation task. We train the quantized models for 20 epochs on 8 GPUs with a batch size of 128 and a learning rate of 2.5e-4 for 8-bit activation models and 5e-4 for binary and ternary activation models.

heading: 3.2 Summarization
text: For the summarization task, we adopt the following benchmarks:
The XSUM dataset (Narayan et al., 2018) consists of 226k documents sampled from the online news website of BBC, together with short, one sentence summaries. Since the summaries are very short, abstractive methods tend to do better on this dataset.
CNN/DailyMail (Nallapati et al., 2016) is another news summarization benchmark, with longer documents (~30 sentences) and longer, multisentence summaries. The dataset contains close to 300k document-summary pairs.
We use BART-base model (Lewis et al., 2019), which is an English-only encoder-decoder transformer with 140 million parameters. We compare using the standard ROUGE-{1,2,l} metrics for this task.
For the ternary weights and 8-bit activations setting, we compare with two state-of-the-art methods QuantBart (Tao et al., 2022) and DQ-BART (Li et al., 2022). For the fully ternary setting, and the binary quantization experiments, there is no prior art. Therefore we provide a naive quantization baseline, using popular implementations from previous work (Li et al., 2016; Courbariaux et al., 2016), and adapt the binary and ternary methods proposed for the BERT models (Bai et al., 2021b; Qin et al., 2021; Zhang et al., 2020) to BART.
Our main results are summarized in Table 1. In the ternary weights and 8-bit activations setting, TBT improves previous SoTA by up to 2.3 points in ROUGE score on XSUM, and up to 0.5 points on CNN/DailyMail. Both improvements are significant.
Further quantizing weights to binary, while keeping activations at 8-bit, we are still able to achieve a ROUGE-L score of 33.3 on XSUM, which is 0.8 points higher than the previous ternary SoTA (DQBART), and comparable on CNN/DailyMail. This is the first demonstration of a binary-weight generative transformer model of competitive accuracy to our knowledge. Additionally, TBT binary weight BART model achieves 1.2 points higher ROUGE score on CNN compared with the SoTA pruning method with the same compressed model size.
Moving on to ternary and binary activations, there is no prior art, and previous implementations fail to produce meaningful results. Our method, on the other hand, achieves ROUGE-L scores of 29.1 and 38.3 on XSUM and CNN/DailyMail in the fully ternary setting, which are 6.6 and 3.8 points behind the full-precision baseline respectively. Our fully binary (weights and activations) model has a wider gap at 10.4 and 8.9 points, however still manages to produce highly non-trivial output at ROUGE-L scores of 25.3 and 33.2 points for XSUM and CNN/DailyMail.

heading: 3.3 Machine translation
text: We also evaluate our model on machine translation. We adopt the En-Ro benchmark from the
WMT’16 shared task (Bojar et al., 2016b) to be compatible with previous work. Our base model is an mBART-large model (Liu et al., 2020a), a 680 million parameter multi-lingual encoder-decoder transformer pre-trained on 25 languages.
Table 2 shows our results. In the ternary weight setting with 8-bit activations, we improve the previous SoTA by 1.2 points, achieving 24.63 BLEU. Remarkably our binary weight model also outperforms the previous ternary weight SoTA by almost a full point. It scores 24.3 BLEU – only 1.5 points behind a full mBART model while being 16× smaller.
In the fully ternary and binary settings, where previous methods failed to converge, TBT models are able to reach practical levels of performance, with ternary TBT mBART achieving 21.7 BLEU, and TBT binary mBART at 17.59.

heading: 3.4 Ablations
text: As stated earlier, our main proposed modeling improvement is a combination of two methods:
statistics-based quantization for the weights, and learning-based quantization for the activations. We ablate the contribution of these methods and present the results in Table 3.
The results clearly show that while each method can give moderate gains by itself over the baseline, these improvements are not sufficient by themselves to produce meaningful results. None of the ablated models can achieve an R2 score above 1.5. It’s only the combination of the two, which together stabilize the training and result in good convergence for fully ternary and binary models.

heading: 3.5 Sequence length analysis
text: In language generation tasks, the error compounding issue in the recursive decoder generation process will largely amplify the quantization error or even lead to divergent results, and thus is an harsh factor to test the robustness of a quantization method. The average generated sequence length indicates whether the quantized model can overcome the compounding error and generate reasonable length of text.
In Table 4 we compare the generated sequence length between the proposed method and the baseline method (i.e., TWN (Li et al., 2016) for ternary, BWN (Courbariaux et al., 2016) for binary). Our method successfully produces summarizations with comparable length as the full-precision model on XSUM benchmark, even when both weights and activations are binarized.
Compared to XSUM dataset, for which the document are summarized to only one sentence, CNN/DailyMail is more challenging because it allows longer summary. We can clearly see that, the text generate with our 8-bit activation models can maintain near the similar average length as the full-precision BART model, while the binary and ternary activation models deviate moderately. In contrast, the baseline method is only able to derive
reasonable summarization with 2-bit weight 8-bit activations and fails at lower bit-width, showing the difficult natural of the language generation tasks.

heading: 3.6 Visualization
text: To further understand the effectiveness of the proposed method, we visualize weight and activation histograms in the BART model ternarized with the baseline method and the proposed method in Fig. 2.
Both the baseline method and our method use per-row weight ternarization, and thus a tensor tensor will have #row of scaling factors. As we can see in Fig. 2 (b) and (g), the proposed method allows the weights to be more evenly distributed in three ternarization levels, which can allow higher information entropy in quantized weights, as discussed in Sec. 2.2. Additionally, we calculate the quantized weight distribution entropy (i.e., Eq. 7) in 96 fully-connected layers in the BART-base model and found that the proposed TBT method achieves consistently higher entropy in quantized weights than the baseline method in all the layers. Further, an interesting phenomenon we can see in Fig. 2 (a) (e) is that ternary weights in a baseline model are very close to the Gaussian distribution, in contrast, weights ternarized with TBT are capturing a more sophisticated distribution. This phenomenon implies that the proposed method helps the weights learn more informative patterns and thus better satisfy the high demand for language generation tasks.
For activation quantization, it is evident that the
attention layer and the SoftMax output only contain the positive activations (XR ∈ R+). If simply ternarized to {−α, 0, α}, the ternary activations will waste one representative level (Fig. 2(d)) and therefore lead to lower accuracy. Instead, the proposed method uses a two-set ternarization method that ternarizes the non-negative activation layer (XR ∈ R+) to {0, α, 2α}, and learns the scaling factor α to better fit the underlying real-valued distribution. This ternarization method greatly reduces information loss and enhances the final accuracy.

heading: 4 Related Work
text: Quantization has long been studied to make neural networks more efficient (see (Hubara et al., 2017) for a survey). Due to the popularity of BERT, numerous works have studied quantization for transformer models, starting with 8-bit quantization (Zafrir et al., 2019; Fan et al., 2020), and progressing to 4-bit (Shen et al., 2020; Zadeh et al., 2020), ternary (Zhang et al., 2020) and binary Bai et al. (2021b); Qin et al. (2021); Liu et al. (2022). All of these works have focused on the encoderonly setting.
In the generative setting, Prato et al. (2019); Behnke et al. (2021) demonstrate quantized models for machine translation, and Fan et al. (2020); Bai et al. (2021a) for language modeling, though only for moderate quantization levels (4-8 bits). Most recently, Tao et al. (2022) and Li et al. (2022) pushed weight quantization down to 2 bits (with
8-bit activation quantization) and evaluated on language modeling and summarization. However, our method outperforms these works substantially, while also demonstrating accurate generative transformers with both weights and activations quantized to 2-bit and even 1-bit for the first time.

heading: 5 Conclusion
text: We have demonstrated high accuracy ternary and binary natural language generation models based on a pre-trained transformer encoder-decoder backbone. Quantizing both the weights and the activations of the network allow these models to run on special-purpose hardware using binary and ternary arithmetic, which doesn’t require multiplication modules. Therefore our results promise multiple orders of magnitude gains in efficiency while running these models, and can drastically expand the use cases of such models beyond just high end gpu servers. We are especially excited about the implications of our results for larger text generation models such as GPT-3 (Brown et al., 2020). These models have both demonstrated impressive capabilities, while also presenting enormous scaling and computational challenges. Low-bit quantization is a promising approach to mitigate some of these issues. Whether our approach will scale to these models is an open problem and an exciting future research direction.

heading: 6 Limitations
text: We conduct experiments on public datasets of finite sentence length, while generalizability to extremely long sequences or even streaming data has not been verified. Furthermore, the generalizability of the proposed quantization method to other tasks, including computer vision or speech recognition, remains to be tested. In addition, binarization and ternarization require bit-packing to have actual memory savings and dedicated hardware support for real-time acceleration, which is more of a hardware implementation aspect and not studied in this paper.

heading: 7 Ethics Statement
text: We affirm that we contribute to society, avoid harm, and are honest and trustworthy. We respect previous work and appropriately cite the methods and datasets we are using. All data we use is public and no private data is involved. There is some potential risk if the translation technique is maliciously
used by a third party and thus we are committed to maintaining the compression techniques we have developed and the general summarization/machine translation techniques used correctly without incurring any form of discrimination.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 6
3 A2. Did you discuss any potential risks of your work? Section 7
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? section 4
3 B1. Did you cite the creators of artifacts you used? section 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? section 4
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? section 4
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? section 3
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? section 3
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. section 3
C 3 Did you run computational experiments? section 3
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? section 3
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? section 3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? section 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? section 3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Whether our approach will scale to these models is an open problem and an exciting future research direction