SlowGuess/NAACL_2022_OCR huggingface dataset:
Mengjie Zhao† Fei Mi‡ Yasheng Wang‡ Minglei Li*
Xin Jiang‡ Qun Liu‡ Hinrich Schütze†
†CIS, LMU Munich ‡Huawei Noah's Ark Lab *Huawei Technologies Co., Ltd.
mzhao@cis.lmu.de,{mifei2, wangyasheng, jiang.xin, qun.liu}@huawei.com
Vast efforts have been devoted to creating high-performance few-shot learners, i.e., large-scale pretrained language models (PLMs) that perform well with little downstream task training data. Training PLMs has incurred significant cost, but utilizing the few-shot learners is still challenging due to their enormous size. This work focuses on a crucial question: How to make effective use of these few-shot learners? We propose LMTurk, a novel approach that treats few-shot learners as crowdsourcing workers. The rationale is that crowdsourcing workers are in fact few-shot learners: They are shown a few illustrative examples to learn about a task and then start annotating. LMTurk employs few-shot learners built upon PLMs as workers. We show that the resulting annotations can be utilized to train models that solve the task well and are small enough to be deployable in practical scenarios. Active learning is integrated into LMTurk to reduce the amount of queries made to PLMs, minimizing the computational cost of running PLM inference passes. Altogether, LMTurk is an important step towards making effective use of current PLMs.<sup>1</sup>
Equipped with prolific linguistic features (Liu et al., 2019; Tenney et al., 2019; Belinkov and Glass, 2019; Rogers et al., 2020) and rich world knowledge (Petroni et al., 2019; Poerner et al., 2020; Kassner et al., 2021), large-scale pretrained language models (PLMs) have been shown to be versatile: They are now basic building blocks (Bommasani et al., 2021) of systems solving diverse NLP tasks in many languages (Wang et al., 2018, 2019; Hu et al., 2020; Xu et al., 2020; Khashabi et al., 2021; Park et al., 2021; Adelani et al., 2021).
Recent work shows that PLMs are effective few-shot learners (Brown et al., 2020; Schick and Schütze, 2021b; Gao et al., 2021; Tam et al., 2021)
Figure 1: LMTurk overview; best viewed in color. We few-shot adapt PLMs to task  $\mathcal{T}$  (left) and then use them as crowdsourcing workers in active learning. We show that these PLM workers are effective in training a small model  $S$  through a customized active learning loop (right). LMTurk is a novel way to take advantage of large-scale PLMs: It creates models small enough to be deployed in resource-limited real-world settings.
Converting a PLM to LMTurker with few-shot gold data  $\mathcal{G}$  of task 7.
Small model  $S$  predicts unlabelled data  $\mathcal{U}$
Select data  $\mathcal{D}$  from  $\mathcal{U}$  with active learning.
LMTurkers  $\mathcal{A}$  annotate and aggregate labels of  $\mathcal{D}$ .
Training a new small model  $s$
through priming (Brown et al., 2020; Tsimpoukelli et al., 2021) or prompting (Li and Liang, 2021; Liu et al., 2021b; Lester et al., 2021; Zhao and Schütze, 2021). Developing few-shot learners is crucial because current NLP systems require much more data than humans (Yin et al., 2020). Few-shot learners tend to perform well; however, they still fall behind systems trained with abundant data. Furthermore, the enormous size of PLMs hinders their deployment in practice. For example, it is challenging to fit the 11 billion T5-XXL (Raffel et al., 2020) model on a single regular GPU.
Our goal in this paper is to devise methods that make more effective use of current few-shot learners. This is crucial because an increasing number
<sup>1</sup>Resources are available at: github.com/lmturk
Findings of the Association for Computational Linguistics: NAACL 2022, pages 675 - 692
of gigantic few-shot learners are trained; how to use them effectively is thus an important question. In particular, we want an alternative to hard-to-deploy huge models. At the same time, we want to take full advantage of the PLMs' strengths: Their versatility ensures wide applicability across tasks; their vast store of knowledge about language and the world (learned in pretraining) manifests in the data efficiency of few-shot learners, reducing labor and time consumption in data annotation.
In this work, we propose LMTurk, Language Model as mechanical Turk. Our basic idea (see Figure 1) is that, for an NLP task  $\mathcal{T}$ , we treat few-shot learners as non-expert workers, resembling crowdsourcing workers that annotate resources for human language technology. We are inspired by the fact that we can view a crowdsourcing worker as a type of few-shot learner: A few examples demonstrating  $\mathcal{T}$  teach her enough about  $\mathcal{T}$  to conduct effective annotation. For example, Snow et al. (2008) train workers with a few examples of annotating emotion; He et al. (2015) conduct short training sessions for workers before annotation; Lee et al. (2021) train workers with learning curricula.
Snow et al. (2008) pioneered crowdsourcing in NLP (Howe et al., 2006; Howe, 2008), motivated by the high cost of TreeBank annotation (Marcus et al., 1993; Miller et al., 1993). Crowdsourcing organizes human workers over the Web to annotate data. Workers need not be experts to be effective, resulting in reduced per-label cost. Active learning (Hachey et al., 2005; Felder and Brent, 2009) can be incorporated (Laws et al., 2011) to further decrease annotation cost, by lowering the number of labels to be annotated. LMTurk treats PLM-based few-shot learners as non-expert workers that produce training sets, which are then used to train a small machine learning model  $S$  specialized for  $\mathcal{T}$ . This scenario is analogous to active learning. We achieve two benefits: (i) low annotation cost because humans only need to annotate a few shots of data; (ii) solving practical NLP tasks with small models that are more real-world deployable.
LMTurk resonates with Laws et al. (2011)'s earlier idea of combining crowdsourcing and active learning. They consider human workers as "noisy annotators" while we explore the utilization of modern NLP few-shot learners (built upon machine learning models) as workers - which have the advantage of being free, instantly interactive, fast, responsive, and non-stopping.
Our contributions: (i) We propose LMTurk, a method that uses few-shot learners as crowdsourcing workers. Figure 1 shows the overview of LMTurk. (ii) We vary an array of important design choices, identifying strengths and weaknesses of LMTurk. (iii) Unlike much work on active learning in a synthetic oracle setting, we develop methods for handling the varying quality of annotation that does not come from an oracle. (iv) We extensively evaluate LMTurk on five datasets, showing that LMTurk can guide a small model  $S$  to progressively improve on  $T$ .  $S$  can then be deployed in practical scenarios. (v) This is the first work showing that few-shot learners give rise to effective NLP models through crowdsourcing and active learning with the benefits of low annotation cost and practical deployability.
Few-shot learners in NLP. Significant progress has been made in developing (Devlin et al., 2019; Peters et al., 2018; Yang et al., 2019; Brown et al., 2020), understanding (Liu et al., 2019; Tenney et al., 2019; Belinkov and Glass, 2019; Hewitt and Liang, 2019; Hewitt and Manning, 2019; Zhao et al., 2020a; Rogers et al., 2020), and utilizing (Houlsby et al., 2019; Zhao et al., 2020b; Brown et al., 2020; Li and Liang, 2021; Schick and Schütze, 2021a; Lester et al., 2021; Mi et al., 2021a) PLMs. Brown et al. (2020), Schick and Schütze (2021a), and Liu et al. (2021b) show that PLMs can serve as data-efficient few-shot learners, through priming or prompting (Liu et al., 2021a). For example, GPT3 achieves near state-of-the-art performance on COPA (Roemmele et al., 2011) with only 32 annotated data.
However, little to no work discusses or explores the actual practical utility of these few-shot learners. We aim to develop effective methods of utilizing them in practical scenarios.
Crowdsourcing has a long history in human language technology (Alonso et al., 2008; Callison-Burch, 2009; Trautmann et al., 2020); specialized workshops were organized (Callison-Burch and Dredze, 2010; Paun and Hovy, 2019). It has numerous applications (Yuen et al., 2011), but we focus on its application as voting systems. To reduce per-label cost, crowdsourcing organizes non-expert human workers distributed across the Web for annotation, instead of employing linguistic experts (Jamison and Gurevych, 2015; Bhardwaj et al., 2019;
Nangia et al., 2021). Snow et al. (2008) show that averaging ten crowdsourced labels matches an expert-level label for recognizing textual entailment (Dagan et al., 2006). Paun et al. (2018) show that incorporating structure in annotation models is important. Measuring label disagreements is also crucial (Dumitrache et al., 2021).
LMTurk utilizes NLP few-shot learners as non-expert workers. The few-shot training data can be viewed as the examples shown to humans before annotating. The process is free, fast, responsive, and non-stopping.
Active learning (AL; Cohn et al. (1996); Settles (2009)) strives to reduce the number of examples to be annotated via identifying informative examples with acquisition functions. Settles and Craven (2008) evaluate AL algorithms for sequence labeling. Zhang et al. (2017); Shen et al. (2017); Siddhant and Lipton (2018) apply AL to deep neural networks. Simpson and Gurevych (2018) devise a scalable Bayesian preference learning method for identifying convincing arguments. Lee et al. (2020) propose to consider user feedback in AL systems. Ein-Dor et al. (2020) explore AL for BERT. Schröder and Niekler (2020) review text classification with AL. Liang et al. (2020); Margatina et al. (2021) integrate contrastive learning into AL. Zhang and Plank (2021) identify examples with datamap (Swayamdipta et al., 2020).
We incorporate AL in LMTurk to reduce the amount of examples to be annotated by PLMs, reducing the computational cost of running several inference passes. This contributes to a more environmentally friendly (Strubell et al., 2019; Schwartz et al., 2020; Patterson et al., 2021) scenario.
Perhaps closest to our work, Yoo et al. (2021) conduct data augmentation via priming GPT3 and Wang et al. (2021) mix human- and GPT3-annotated data, focusing on cost analysis. GPT3 is utilized in a Language-Model-as-a-Service form by OpenAI, which is not free. Also, strategies of priming GPT3 may not generalize well to other PLMs. For example, priming strategies have to adapt to GPT3's maximum sequence length. However, maximum sequence length - as a hyperparameter - could vary across PLMs. In this work, we prompt publicly available free PLMs. This also makes the process more flexible; for example, the PLM can be updated with gradient descent.
3 LMTurk
3.1 Training few-shot learners
We first adapt a PLM to task  $\mathcal{T}$  with a few-shot human-labeled gold dataset  $\mathcal{G} = \{\mathcal{G}_{train};\mathcal{G}_{dev}\}$  of  $\mathcal{T}$ . This procedure mimics one of the initial but crucial steps in crowdsourcing: A few example annotations are shown to the workers, demonstrating  $\mathcal{T}$ ; workers learn about the task and then start annotating (Snow et al., 2008; He et al., 2015; Roit et al., 2020; Trautmann et al., 2020; Lee et al., 2021).
We achieve this adaptation through P-Tuning (Liu et al., 2021b). Taking movie review classification as an example, the goal is to associate a binary label  $y$  from  $\{-1, +1\}$  to an input sentence  $\mathbf{x} = (x_{1}, \dots, x_{n})$  where  $x_{i}$  refers to a token. Unlike finetuning and its variants (Devlin et al., 2019; Houlsby et al., 2019; Zhao et al., 2020b) that train a classifier head, P-Tuning reformulates a sentence into a cloze-style query; the PLM is then requested to respond to the query with an answer selected from a list of candidates. Concretely, an input pair
$$
(x, y) = \text {" w a t c h i n g i t l e a v e s y o u g i d d y . " , - 1)}
$$
is reformulated to:
$$
\left[ \mathrm {v} \right] \text {w a t c h i n g i t l e a v e s y u g d y . I t} \underline {{\underline {{\mathrm {i s}}}}} [ \text {M A S K} ] \quad .
$$
in which the underlined tokens are prompting words that give the model a hint about  $\mathcal{T}$ . “[v]” – whose trainable embedding vector is randomly initialized – is a prompting token injecting extra free parameters. The PLM is then requested to pick a word from {"bad", "good"} to fill in the position of "[MASK]". A mapping {"bad" → -1, "good" → +1} is used to transform the selected answer to a label such that standard evaluation measures like accuracy can be computed. Prompting has been shown to effectively adapt a PLM to  $\mathcal{T}$  with only a few annotations; see (Liu et al., 2021a) for a comprehensive review of prompting. We refer to a PLM adapted to  $\mathcal{T}$  as an LMTurker  $A$ .
We select prompting words and mappings based on the small development set  $\mathcal{G}_{dev}$ . §4.2 provides details on prompting and datasets.
3.2 Aggregating annotations
Individual workers are subject to annotation biases (Snow et al., 2008); therefore, crowdsourcing often collects labels from several workers (Yuen et al., 2011) for an example  $\mathbf{x}$  and then aggregates them for quality control (Alonso et al., 2008). It is straightforward to obtain a group of LMTurkers
$^{2}$ https://beta.openai.com/pricing
$\mathcal{A} = \{A_1,\dots,A_k\}$ , by adapting the PLM to  $\mathcal{T}$  with  $k$  different prompts. A querying sentence  $\mathbf{x}$  is then annotated by every LMTurker, resulting in a list of labels  $\mathbf{y} = [y_{1},\dots,y_{k}]$ . We evaluate different methods aggregating  $\mathbf{y}$  to a single label  $\hat{y}$ .
BestWorker. Among the  $k$  LMTurkers, we pick the one performing best on the dev set  $\mathcal{G}_{dev}$ .
MajorityVoting. We select the most frequent label in  $\mathbf{y} = [y_1, \dots, y_k]$  as  $\hat{y}$ .
To estimate an LMTurker's confidence on label  $y_{i}$ , we compare the logits<sup>3</sup> computed by the PLM:
$$
y _ {i} = \arg \max  (\operatorname {l o g i t} (y ^ {1}), \dots , \operatorname {l o g i t} (y ^ {N})),
$$
where  $N$  refers to the label set size, e.g.,  $N = 2$  for  $y$  from  $\{-1, +1\}$ . We then can evaluate several methods of aggregating annotations according to PLM logits.
LogitVoting. We average the logits from all  $k$  LMTurkers  $\{A_1, \dots, A_k\}$  to compute  $\hat{y}$ :
$$
\hat {y} = \arg \max  \left(\frac {1}{k} \sum_ {i = 1} ^ {k} \operatorname {l o g i t} \left(y _ {i} ^ {1}\right),..., \frac {1}{k} \sum_ {i = 1} ^ {k} \operatorname {l o g i t} \left(y _ {i} ^ {N}\right)\right).
$$
WeightedLogitVoting. We use LMTurkers' performance on  $\mathcal{G}_{dev}$  to weight their logits and then aggregate the predictions:
$$
\begin{array}{l} \hat {y} = \arg \max  \left(\sum_ {i = 1} ^ {k} w _ {i} \operatorname {l o g i t} \left(y _ {i} ^ {1}\right), \dots , \sum_ {i = 1} ^ {k} w _ {i} \operatorname {l o g i t} \left(y _ {i} ^ {N}\right)\right) \\ w _ {i} = f (A _ {i}, \mathcal {G} _ {d e v}) / \sum_ {i = 1} ^ {k} f (A _ {i}, \mathcal {G} _ {d e v}) \\ \end{array}
$$
where  $f(A_{i},\mathcal{G}_{dev})$  is the performance of the ith LMTurker  $A_{i}$  on  $\mathcal{G}_{dev}$
We collect and aggregate annotations from five LMTurkers, i.e., we use  $k = 5$  in our experiments.
3.3 Training a small model  $\mathcal{S}$
After adapting LMTurkers to  $\mathcal{T}$  through prompting with the few-shot gold dataset  $\mathcal{G}$ , we next train a small model  $S$  specialized to solve  $\mathcal{T}$ . Though large PLMs are versatile and strong performers, training and inference are faster and more efficient for small models: They are more deployable in resource-restricted scenarios, e.g., on edge devices (Jiao et al., 2020).
We mimic pool-based active learning (AL; Settles (2009)) to train  $\mathcal{S}$ . The motivation is to avoid frequent querying of LMTurkers  $\mathcal{A}$  because energy and time consumption of PLM inference is costly when the number of queries and  $|\mathcal{A}|$  are large.
Concretely, pool-based AL assumes a large collection of unlabeled data  $\mathcal{U} = \{\mathbf{x}_1,\dots ,\mathbf{x}_M\}$  for  $\mathcal{T}$ .
$\mathcal{S}$  is first trained with  $\mathcal{G} = \{\mathcal{G}_{train};\mathcal{G}_{dev}\}$ . After that, a group of examples  $\mathcal{B}$  from  $\mathcal{U}$  is sampled (c.f. §3.3.1), which LMTurkers annotate. Next, the annotated and aggregated examples  $\mathcal{B}'$  are concatenated with  $\mathcal{G}$  to train  $\mathcal{S}$ . The procedure is repeated iteratively, such that the training data for  $\mathcal{S}$  keeps expanding. We denote as  $\mathcal{S}^j$  the model trained after the  $j$ th iteration. Note that  $\mathcal{S}$  is trained from scratch in each iteration (Cohn et al., 1994).
3.3.1 AL acquisition function
At the beginning of the  $j$ th iteration, a straightforward strategy of sampling  $\mathcal{B}$  from  $\mathcal{U}$  is random sampling. AL promises to select a more informative  $\mathcal{B}$  such that the trained  $S^j$  performs better, under the same budget. These strategies – or acquisition functions – rely on  $S^{j-1}$ , i.e.,  $\mathcal{S}$  from the previous iteration:  $S^{j-1}$  is employed to infer  $\mathcal{U}$  to obtain labels and logits  $\mathcal{P}^{j-1} = \{(y_1, \mathbf{c}_1), \dots, (y_M, \mathbf{c}_M)\}$ ; each  $\mathbf{c}_i$  contains the logits of the  $N$  labels;  $y_i = \arg \max(\mathbf{c}_i)$ . We explore two common AL acquisition functions: Entropy (Roy and McCallum, 2001) and LeastConfident (Lewis and Gale, 1994).
Entropy selects from  $\mathcal{P}^{j-1}$  examples with the largest prediction entropy, computed using  $\mathbf{c}$ . Large entropy of an example  $\mathbf{x}$  implies that  $S^{j-1}$  is unsure about which label to select;  $\mathbf{x}$  is then a query made to LMTurkers to obtain its label  $\hat{y}$ .  $(\mathbf{x}, \hat{y})$  is subsequently added to  $\mathcal{G}_{train}$  for training  $S^j$ .
LeastConfident selects from  $\mathcal{P}^{j - 1}$  examples for which the maximum logit in  $\mathbf{c}$  is the smallest. Selected examples are then annotated and added to  $\mathcal{G}_{train}$  for training  $S^j$
Our AL setup is fairly standard, both in terms of acquisition functions and iterative enlargement by new sampled data  $\mathcal{B}$  at iteration  $j$  labeled by  $S^{j - 1}$ .
3.3.2 Considering annotation quality
As in any realistic AL scenario, annotations are not perfect: LMTurkers do not score perfectly on  $\mathcal{T}$ . As a result, annotation quality of LMTurkers needs to be taken into consideration before training  $S^j$ . Denoting the training data of  $S^j$  as  $\mathcal{D}^j$ , we explore a strategy of processing  $\mathcal{D}^j$ , based on LMTurker logits  $l$ .
InstanceTresholding. We preserve examples  $(\mathbf{x},\hat{y},\mathbf{l})\in \mathcal{D}^j$  for which entropy computed on  $\mathbf{l}$  is smallest.  $\mathcal{G}^{train}$  is always preserved because it is human-labeled gold data. Note that this is different from the strategy of sampling  $\mathcal{B}$ , where we select from  $\mathcal{P}^{j - 1}$  examples to which  $S^{j - 1}$  is most unsure
3Calibration can be conducted to further improve the estimation (Guo et al., 2017). We leave this to future work.
(computed with c). We evaluate the effectiveness of processing  $\mathcal{D}^j$  before training  $S^j$  in §5.6.
3.4 Summary of LMTurk
LMTurk can be viewed as intermediate between self training (Yarowsky, 1995; Abney, 2004; Lee et al., 2013; Mi et al., 2021b) and AL. Unlike self training, LMTurk employs external models provide labels to  $S$ . Different from the artificial setup used in many AL experiments, the provided labels do not have oracle quality; so  $S$  must use the annotations more carefully. We next conduct experiments investigating the effectiveness of LMTurk.
4 Datasets and Setup
We evaluate LMTurk on five datasets: Binary (SST2) and fine-grained (five classes) sentiment classification (SST5) with the Stanford Sentiment TreeBank (Socher et al., 2013); news article topic classification with the AG's News Corpus (AGNews; Zhang et al. (2015)); recognizing textual entailment (RTE; Dagan et al. (2006)); assessing linguistic acceptability (CoLA; Warstadt et al. (2019)). Appendix §A reports dataset statistics. SST2/SST5 and AGNews are widely used in crowdsourcing and AL (Laws et al., 2011; Ein-Dor et al., 2020; Margatina et al., 2021; Zhang and Plank, 2021). RTE and CoLA assess the models' ability to understand textual entailment and linguistic phenomena - as opposed to text categorization. We report Matthew's correlation coefficient for CoLA and accuracy for the others (Wang et al., 2018).
Few-shot datasets. Recall LMTurk uses a small human-annotated dataset  $\mathcal{G} = \{\mathcal{G}_{train};\mathcal{G}_{dev}\}$ . Denoting  $n$  as the number of shots per class, we sample  $\mathcal{G}_{train}^n$  and  $\mathcal{G}_{dev}^n$  for each of  $n\in \{8,16,32\}$ . For SST2, RTE, and CoLA, we use the train and dev sets of GLUE (Wang et al., 2018);  $\mathcal{G}_{train}^n$  and  $\mathcal{G}_{dev}^n$  are sampled from the train set; the dev set is used as the test set. For SST5 and AGNews, we use the official datasets;  $\mathcal{G}_{train}^n (\mathcal{G}_{dev}^n)$  is sampled from the train (dev) set; we report performance on the test set. We repeat the sampling process with three random seeds.
4.2 Training setup
Brown et al. (2020) show that large model size is
Motivated by Wang et al. (2017), we also investigate the effectiveness of weighting training examples. However, we do not observe noticeable improvements of task performance. We list more details in Appendix §E.
Schick and Schütze (2021a,b)	Gao et al. (2021)	Ours
SST2	n/a	93.0±0.6	93.08±0.62
SST5	n/a	49.5±1.7	46.70±0.93
RTE	69.8	71.1±5.3	70.88±1.70
AGN.	86.3±0.0	n/a	87.71±0.07
CoLA	n/a	21.8±15.9	19.71±1.89
Table 1: LMTurkers achieve comparable few-shot performance with the literature. We refer to  $PET$  results in Schick and Schütze (2021a,b) and results of Prompt-based  $FT$  (auto) + demonstrations in Gao et al. (2021).
necessary for strong few-shot performance. We use ALBERT-XXLarge-v2 (Lan et al., 2020) – of size 223M parameters – as our large PLM, which is adapted to be an LMTurker  $A$  of  $\mathcal{T}$  with  $\mathcal{G}$ . With parameter reuse, ALBERT-XXLarge-v2 outperforms larger models like the 334M BERT-large (Devlin et al., 2019). In contrast,  $S$  must be small to be deployable in practical scenarios. We use TinyBERT-General-4L-312D (Jiao et al., 2020), which has 14.5M parameters.
We train – with prompting – the large PLM with  $\mathcal{G}$  for 100 batch steps using batch size 16, AdamW (Loshchilov and Hutter, 2019) and learning rate 5e-4 with linear decay. We prompt the large PLM five times to obtain five LMTurkers; Appendix §C shows prompting details. At each iteration, we finetune  $\mathcal{S}$  for 20 epochs using batch size 32, Adam (Kingma and Ba, 2015) and learning rate 5e-5. Each experiment is run with three different random seeds. We use PyTorch (Paszke et al., 2019) and HuggingFace (Wolf et al., 2020).
5.1 Few-shot performance (non-iterative)
We compare few-shot performance of LMTurkers and the small model  $\mathcal{S}$  when only  $\mathcal{G}$  is used. LMTurker performance is comparable to prior work (Schick and Schütze, 2021a,b; Gao et al., 2021) as shown in Table 1.
Figure 2 compares performance of LMTurkers and  $S$ . Appendix §B Table 3 reports numeric values. LMTurkers perform clearly better than  $S$  on CoLA, SST5, AGNews, and SST2; e.g., for SST2, for train/dev size 16, LMTurker accuracy is  $93.08\%$  vs.  $75.83\%$  for  $S$ . LMTurkers' superiority over  $S$  on RTE is modest. As an inference task, RTE is more challenging than classification (e.g., AGNews). We hypothesize that current few-shot learners require more data than  $\mathcal{G}^{32}$  to process difficult tasks better than  $S$ . Scaling up to even larger PLMs is also a promising direction (Brown et al., 2020;
Figure 2: Few-shot test set performance of LMTurkers and  $\mathcal{S}$ . We use the few-shot gold datasets  $\mathcal{G}^8$  (top),  $\mathcal{G}^{16}$  (middle), and  $\mathcal{G}^{32}$  (bottom).
Lester et al., 2021).