Branch-GAN: Improving Text Generation with (not so) Large Language Models
Authors: Fredrik Carlsson
Johan Broberg
Erik Hillbom
Magnus Sahlgren
Joakim Nivre
Venue: ICLR 2024 poster (2024)
Keywords: GAN
GANN
Text Generation
NLP
NLU
Deep Learning
Machine Learning
Primary area: generative models
TL;DR: A computationally efficient GAN approach for sequential data, evaluated specifically for text generation.

Abstract: The current advancements in open domain text generation have been spearheaded by Transformer-based large language models. Leveraging efficient parallelization and vast training datasets, these models achieve unparalleled text generation capabilities. Even so, current models are known to suffer from  deficiencies such as repetitive texts, looping issues, and lack of robustness. While adversarial training through generative adversarial networks (GAN) is a proposed solution, earlier research in this direction has predominantly focused on older architectures, or narrow tasks. As a result, this approach is not yet compatible with modern language models for open-ended text generation, leading to diminished interest within the broader research community. We propose a computationally efficient GAN approach for sequential data that utilizes the parallelization capabilities of Transformer models. Our method revolves around generating multiple branching sequences from each training sample, while also incorporating the typical next-step prediction loss on the original data. In this way, we achieve a dense reward and loss signal for both the generator and the discriminator, resulting in a stable training dynamic. We apply our training method to pre-trained language models, using data from their original training set but less than 0.01% of the available data.  A comprehensive human evaluation shows that our method significantly improves the quality of texts generated by the model while avoiding the previously reported sparsity problems of GAN approaches. Even our smaller models outperform larger original baseline models with more than 16 times the number of parameters. Finally, we corroborate previous claims that perplexity on held-out data is not a sufficient metric for measuring the quality of generated texts.

Published as a conference paper at ICLR 2024
BRANCH-GAN: IMPROVING TEXT GENERATION WITH (NOT SO) LARGE LANGUAGE MODELS
Fredrik Carlsson* Johan Broberg'
Erik Hillbom"
Magnus Sahlgren
Joakim Nivre*
"RISE Research Institutes of Sweden
AI Sweden
Correspondence: fredrik.carlsson@e @ri.e
ABSTRACT
The current advancements in open domain text generation have been spearheaded by Transformer-basee large language models. Leveraging efficient parallelization and vast training datasets, these models achieve unparalleled text generation ca- pabilities. Even so, current models are known suffer from deficiencies such repetitive texts, looping issues, and lack of robustness. While adversarial train- ing through generative adversarial networks (GAN) is a proposed solution, ear- lier research in this direction has predominantly focused on older architectures, or narrow tasks. As a result, this approach is not yet compatible with modern language models for open-ended text generation, leading to diminished interest within the broader research community. We propose a computationally efficient GAN approach for sequential data that utilizes the parallelization capabilities of Transformer models. Our method revolves around generating multiple branching sequences from each training sample, while also incorporating the typical next- step prediction loss on the original data. In this way, we achieve a dense reward and loss signal for both the generator and the discriminator, resulting in stable training dynamic. We apply our training method to pre-trained language models, using data from their original training set but less than 0.01% of the available data. A comprehensive human evaluation shows that our method significantly im- proves the quality of texts generated by the model while avoiding the previously reported sparsity problems of GAN approaches. Even our smaller models out- perform larger original baseline models with more than 16 times the number parameters. Finally, corroborate previous claims that perplexity held-out data not a sufficient metric for measuring the quality of generated texts.
INTRODUCTION
The recent rapid advancements in AI have led to text generation capabilities that, according to some,
result in human quality texts (Bubeck et al., (203)). This advancement has primarily been propelled
by the foundational role of Transformer-basee large language models (LLMs) (Radford et al. 2019
Brown al. 2020 Chowdhery al.) [022 OpenAl 2023 Penedo et al. 2023 Touvron et al. 2023a bj Biderman et al. 2023). The Transformer architecture (Vaswani et al.) 2017) processes sequential data in parallel. in contrast the previously popular RNNs (Rumelhart al. 1986
Hochreiter & Schmidhuber 1997). This computational efficiency has enabled the scaling of both model parameters and training data, leading to enhanced performance and emergent capabilities.
However, various sources report deficiencies in modern LLMs, such as repetitive texts, looping is-
sues, and lack of robustness (Fu al. [02T] Holtzman al. 20200 Wang et al. 2022). One hypoth-
esis attributes these issues to exposure bias (Bengio et al. 2015 Ranzato et al. 2016), which high-
lights the disparity between next token prediction during training, and multi-step generation during inference. Since each generation step risks being erroneous, inference is susceptible to accumulat- ing errors that progressively drift the model away from the training distribution. Although heuristics and sampling strategies reduce the likelihood of erroneous individual generation steps (Keskar et al., 2019 Holtzman et al. 2020), they address symptoms rather than the underlying root cause.
Theoretically, training language models via generative adversarial networks (GANs) (Goodfellow) et al., 22200) provides an elegant solution to exposure bias. Training with multi-step generation of- fers the generator learning signal more akin to inference, from which it can learn correct after

---

Published as a conference paper at ICLR 2024
any misstep during generation. Unfortunately, GANs come with their own set of challenges (Cac- cia et al.] 2020). Multi-step generation increases the computational burden and the volatile training dynamic can lead to non-convergence and model collapse. Earlier work has addressed training dynamic issues using methods like dense rewards (Shi et al.] 2018 Semennuta et al. 2019 de Mas- son d'Autume al., (219) and incorporating next-token prediction, also known as maximum like- lihood estimation (MLE). However, this line of research has, to the best of our knowledge, focused mainly on RNNS, and is hence incompatible with the computational parallelism of Transformers.
We propose the computationally efficient Branch-GAN, which utilizes the parallelization capabil- ities of the Transformer architecture. Our method is based on generating multiple branching se- quences from each training sample, which are processed in parallel for both the generator and the discriminator. This parallelization enables massively dense rewards for both the generated and orig- inal texts, resulting in an unproblematic and stable training dynamic. According to human evalua- tion, LLMM utilizing the Branch-GAN method generate significantly better texts on average. Finally, note strong correlation between human preferences and automatic metrics such as TTR, with Branch-GAN improving on these metrics without an explicit incentive.
Our main contributions are the following:
A new method for training Transformers in GAN setup, which gives higher quality in text
generation with fewer model parameters
A dataset with manual rankings of text quality containing more than 10,000 annotations
An exploration of automatic metries that correlate well with human perceived text quality Implementation at: Caauub.com/Freddee Fraaa//aaeeeeaalllaannaa Saaallan/Brancc--G Branch-GAN
2 RELATED WORK
In the realm of GANs for language modeling, most methods were developed prior to the widespread adoption of pre-trained Transformers and therefore use RNNs (Yu et al.] 2017 Che et al., [017]] Lin et al., 2017; Guo et al. 2018 Shi et al.) 2018 de Masson d' d'Autume al. 2019). As a result,
they cannot compete with the generative capabilities of modern LLMs. More recently Scialom et al.
2020 , Wu al. (02T)) D. Scialom et al. 20021) Lamprier et (2022) trained language GANs or cooperative language GANs with pre- trained Transformers architetures, using models such as GPT- 2 (Radford al. 2019) as generators and RoBERTa (Zhuang et al.)
202I] as discriminators, or with
encoder-decoder models such as T5 (Raffel et al. 2020) and BART (Lewis et al. 2019). While these advancements have pushed the boundaries of language GANs, many of the challenges historically associated with GANs for language modeling still remain relevant today.
A critical issue in the adaptation of GANs for sequential data is the sparsity of the loss signal. In contrast to traditional next-token prediction models, where loss is computed at each token, the straightforward application of GANs in language modeling typically yields loss only for fully gen- erated sequences. To address this challenge, some methods leverage RNN-based discriminators to process sequences incrementally, thereby providing token-level loss feedback (Shi et al. 2018 Se- meniuta al. [2019] de Masson d' d'Autume al.) 2019). Alternatively, work such as Lin et al. (2017) modifies the discriminator's 's classification task to a ranking task, allowing them to rescale rewards from different samples in order offer more informative signal for the generator.
Additionally, the inherent instability of GAN training is further exacerbated by the gradient esti- mation step of reinforcement learning (RL) (Sutton & Barto, 2018). Though some methods train language GANs from scratch (Che et al. 2017 de Masson d' d'Autume et al. 2019). the majority of methods resort to MLE pretraiming as a foundational step (Yu et al. |017]] Shi al.
2018 Nie al., Guo et al. 2018, Sccialom et al. 2020 2021). Approaches such as Che et al. (2017) and Guo et al. 2018) combine MLE and adversarial training to effectively stabilize the learning process.
2019
Other ways to address the instability issues is by averaging rewards across multiple generated sam- ples, as demonstrated by Yu et al. (2017), or by employing a moving average of rewards over time steps (de Masson d' d'Autume et al. 2019 In different vein, both Shi et al. (2018) and Wu et al. (0021) recast text generation as an inverse reinforcement learning problem in an effort to increase stability. Their objective is to learn reward function that rationalizes an "expert policy" responsible for generating the training data. Yet another approach uses the decoder in a cooperative decoding scheme to produce more realistic samples (Scialom al. 2021 Lamprier al. 2022)
2

---

Published as a conference paper at ICLR 2024
MLE Loss
RL Loss
Generator Original
2
N-3
N-2
N-1
N
2
N-2
3
N-1
1
Attention Targets
Generaled
Generaled
2
N-3
N-2
N-1
Pass
Pass 2
Pass 3
Figure 1: The Branch-GAN generator outputs multiple branching sequences from the original text (left) and trained using a combination of standard MLE loss on the original text and RL loss on branching sequences (right).
METHOD
The key idea behind Branch-GAN is to efficiently generate multiple short continuations that branch out from an original text sequence, and supply a dense loss signal for both the original and generated sequences. The generator is trained via both MLE the original sequence and adversarial signal from the discriminator for the generated sequences. Simultaneously, the discriminator is trained to better distinguish original sequences from generated sequences.
Our method relies on the auto-regressive Transformer architecture for both the generator and the discriminator. As visualized in Figure [] we apply branching self-attention schema partly pro- cess multiple branching sequences in each forward pass. This schema allows both the generator and the discriminator efficiently compute independently generated sequences, and results dense learning signal for both models.
In the following sections, S represents an original text sequence in the training data with length N. We let G denote a generated sequence with length d where d is referred to as depth. Finally, we use subscript notation refer token in a sequence, meaning that S, refers to the rith token in S.
3.1 GENERATOR
During training we randomly sample K unique starting points in the range [1, N 1] for each batch, determining the start indices for our generated sequences {G', G G*}, From an initial forward pass over S greedily pick the first token for each generated sequence, but block the actual continuation token in S. This means that if the first starting index K = n, then G / Sa+1-
Following this, perform d forward passes, greedily selecting and appending each generated sequence. During these passes, each token self-attends to all previous tokens in the same generated sequence, addition all tokens in the original S that lie before its start index. Thus, the n:th token in G attends to {S} SK:} + ...
token to
In this way, each forward pass creates K new tokens. After d forward passes we end up with the original sequence together with K generated branching sequences consisting of d tokens each. The generator trained via causal language modelling loss over S, and via an adversarial RL loss for tokens in the generated sequences (Section 3.3
3.2 DISCRIMINATOR
The discriminator processes data in a fashion similar to the generator (Section 3.1 but with an addi- tional forward pass for the final generation step. This means that an original text S and K branching sequences with depth requires + 1 forward passes, regardless the number of sequences K. A visualization of the discriminator and its attention is available in Appendix
At each time-step n, the discriminator provides three sealar predictions D,. I, and B,, where D is the diseriminative prediction of the with token, while I, and B, are value head predictions for the n+1 time-step. All three these predictions are passed through a logistic function and thus fall in the range [0, 1).
3

---

Published as a conference paper at ICLR 2024
The discriminative prediction will, due to the sequential nature of the task, be influenced by its pretext. For example, if the discriminator at a point identifies a sequence as 'fake', it will influence subsequent predictions negatively. Thus, the generator RL loss is derived using the discriminator's value heads. The value head prediction I estimates D=+1 when the next token is from S. And B, estimates 1 if the next token is from G. The RL loss described in Equation T
During training, we can only assign labels to value heads when the correct information is available in the next time step. Thus, Ia is assigned labels at steps found within the original sequence - while B, is assigned labels during the generated sequences Gı.d-I, and the original time-steps S where n + 1 K, indicating the start of a generated sequence at the subsequent step.
3.3 GAN SETUP
As is common with GANs, the generator is trained to produce G to fool the discriminator, and the discriminator is trained to better distinguish between S and G. The discriminator utilizes binary cross-entropy for both the prediction Dn and the value head predictions, leading the discriminate loss LD and the value head loss Lv. The final discriminator loss is the weighted sum of CD+Ly *a.
The generator receives a normal MLE loss on predictions for S, along with RL loss for generated time-steps in G that do not exceed the position N. Time-steps in G are classified into one of four categories in accordance with Equation [] and given a loss LG accordingly. The final generator loss is the mean loss over all time-steps, for both S and G.
0
if 4-1 S B if 4-1 S Dr
log(p(())
LG
D. Di-Bi, log(p(())
1.-1
if B S Dr Th
(I)
log(1 p(t))
if D, B
EXPERIMENTS AND ANALYSIS
In this section, evaluate text generation using models trained with Branch-GAN, well as several baseline models. We begin with a large-scale human evaluation of text quality (Section explore automatic metrics that correlate with human quality judgments (Section 4.3) and use these metrics in an automatic model evaluation (Section 4.4). We conclude with a robustness evaluation (Section 4.5) and an ablation study focusing on the hyperparameters depth and sparsity (Section 4.6
4.1 EXPERIMENTAL SETUP
Although a necessity, our training starts from pre-trained language models, where both the gen- erator and the discriminator are initialized from the same checkpoint. We use checkpoints from the Pythia model suite (Biderman al. 2023), which were trained on the full Pile dataset (Gao al. 2020) using MLE training in accordance with the current LLM scaling laws (Kaplan et al. 2020).
In order to make our models comparable to the existing Pythia models we train exclusively on data from the Pile dataset. Our training data consists of 100k randomly selected sequences of length 128, tokenized using the Pythia tokenizer. As the full deduplicated version of the Pile contains about 207B tokens and we use 12.8M tokens, less than 0.01 % of the Pile data is used during our training.
We train using a branch sequence depth of d=16 and K=32 branches per sample. This is performed with a batch size of 8, for 4 epochs over the training data, resulting in 50k optimizer updates, where of the tokens come from generated sequences. The discriminator loss weight is set to a = 0.2.
Generating text requires the specification of a sampling method and its hyperparameters. Corrobo- rating earlier work, we found baseline models to generate more fluent texts when using top-p sam- pling (Holtzman et al. 2020) coupled with repetition penalty. Thus, Pythia and LLama-2 7B use top-p=0.95 sampling, unless otherwise stated. Branch-GAt models instead use greedy decoding (unless otherwise stated). since that is what is used during training. Models marked with an asterisk (*), use repetition penalty of 1.2 as proposed by Keskar et al. (2019), and all models utilize a beam size of 8. Results and analyses of additional sampling methods are available in Appendix H.2
4

---

Published as a conference paper at ICLR 2024
Table 1: Human evaluation results for models in comparison original texts (Wiki, Fiction) and to Branch-GAN 1B* (greedy) (W IB-B, Fict of texts that are judged to better than as good as the competing candidate.
Numbers represent the percentage
Model
Wiki Wiki
Fiction
Fi
Strong Baselines
LLama-2 7B (top-p)
32.0
50.7
38.7
43.3
GPT-4
81.0
88.3
73.3
86.7
Comparative Baselines
Pythia 410M (top-p)
10.7
13.0
6.7
11.3
Pythia IB* (top-p)
15.7
23.3
11.7
16.0
Pythia 6.9B* (greedy)
14.0
16.7
22.7
38.0
Pythia 6.9B* (top-p)
24.0
38.7
27.3
45.7
Our Models
Branch-GAN 410M (greedy)
49.7
58.7
51.7
68.0
Branch-GAN 410M (greedy)
48.3
57.3
71.0
70.7
Branch-GAN IBx (top-p)
48.7
65.7
50.3
59.3
Branch-GAN 1B*
(greedy)
68.7
76.0
4.2 HUMAN EVALUATION
To thoroughly evaluate text quality, we conduct rigorous evaluation study, gathering over 10k hu- man annotations from verified English speakers. To ensure high-quality annotations, each annotator was independently verified and hired as freelancer and fairly compensated. More information regarding the annotation process and the resulting dataset is available Appendix B
The evaluation is centered around deciding the preferred continuation to a given context, and posed two different scenarios. In the first scenario, one continuation always the original text from the corresponding context. In the second scenario, one continuation always come from Branch-GAN IB (greedy). The annotators' task is to determine the preferred continuation, or classify them as equally good.
Each trained model is evaluated on both scenarios and on contexts from two different datasets: Wikipedia (Merity al. 2017) and Fictional Stories ? dataset with a length of 128 tokens, of which 32 are used as context and 96 as continuation. Each model generates continuations all contexts and gather annotations per continuation compar- ison.
We randomly select 100 samples from each
Results in Table [ display the percentage of times that generated texts either preferred to judged as equally good as the other candidate. Branch-GAN outperforms all baselines with sig- nificant margins, with the exception of GPT-4. Both Branch-GAN and the Pythia models perform better on the fictional contexts compared to the more factual Wikipedia. The opposite is true for GPT-4, which is only just as good Branch-GAN fictional stories. Applying repetition penalty to Branch-GAN only improves its score on fictional stories.
hhttp://www..ppwor.ccomm

---

Published as a conference paper at ICLR 2024
Table 2: Fine-grained human evaluation for subset of models when compared against original
texts. All Pythia models use top-p sampling; all Branch-GAN models use greedy decoding.
Model
W iiki
Fiction Ties
Wins
Tíes Baselines
Losses
Wins
Losses
Pythia IB*
9.3
6.7
84.0
4.7
7.0
88.3
Pythia 6.9B*
9.3
14.7
76,0
10.7
16.7
72.7
GPT-4
41.3
39.7
19.0
37.3
30.7
32.0
Our Models
Branch- GANN 410M*
14.7
36.3
49.0
18.3
52.7
29.0
Branch-GAn 1B*
11.7
54.3
34.0
18.3
57.7
24.0
Table 2 denotes the fine-grained annotations for models when compared to the original texts. Branch-GAN IB scores over 50% ties for both datasets, while the Pythia models never tie more than 20%. GPT-4 achieves considerably more wins than other models, but also over 30% losses on the Fiction dataset. These losses may be due to annotators' personal preference in style. More fine-grained results are available in Appendix E
4.3 HUMAN PREFERENCES AND AUTOMATIC METRICS
While the human evaluation shows that adversarial training improves the models' ability to generate texts accordance with human preferences, does not tell what properties are characteristic of such texts. As starting point to finding automatic evaluation metrics that correlate with perceived text quality we investigate two properties: Type-Token Ratio (TTR) and cross-entropy of Pythia 6.9B. The idea is that TTR can capture the repetitiveness of a text, while cross-entropy informs us about the flow of the text.
We calculate these characteristics for all generated texts that received full annotation consensus, and compare them to the characteristics annotation is considered good if it is either preferred deemed equally good the alternative continuation. As detailed in Appendix B 56% the samples had agreement from all three annota- tors.
the original human written texts. As Section 4.2 an
As seen in Figure generated texts with similar TTR and cross-entropy as that of human written texts far more likely receive positive annotations. There exists virtually texts unanimously rated as good with a TTR < 0.5. The span of cross-entropy for good texts a bit wider, although still clearly centered around the mean for human texts. These results strengthen our intuition that characteristics of generated texts should be similar to those of human written texts, and that such characteristics can partially captured using combination of metrics like TTR and eross- entso-.
"Cross-entropy was preferred over perplexity, it casier visualize.
Original
Consensus Bad
Consensus Good
0.08
0.08
Density
0.06
Density
0.06
0.04
0.04
0.02
0.02
0.00
0.00
0.2
0.4
0.6
0.8
1.0
0.0
0.5
1.0
15
2.0
2.5 3.0 3.5 4.0
TTR
6.9b Cross Cross-Entropy
Figure 2: Distributions of TTR and Cross-entropy for Pythia 6.9B, for generated texts with annota- tion consensus. Dotted lines indicate the mean TTR and Cross-entropy of the original texts.

---

Published as a conference paper at ICLR 2024
Table Automatic evaluation of models using perplexity original text (Org PPL). perplexity of Pythia 6.9B* on generated texts (6.9b PPL), type-token ratio (TTR), and Self-BLEU (S BLEU).
Model
Org PPL 6.9b PPL
TTR S - BLEU
Ground Truth
Original Texts
13.08
71.8
2.4
Comparative Baselines
Pythia 1B
20.34
2.0
36.5
4.0
Pythia 6.9B*
13.08
1.96
45.6
3.9
Our Models
Branch-Gaae GAN IB
51.47
9.78
70.0
1.5
Branch- GAN IB
51.47
10.38
73.8
1.4
Strong Baselines
LLama 7B GPT-4
10.34
3.81
54.2
1.3
15.4
74.9
2.0
4.4 AUTOMATIC EVALUATION
Working from the findings in Section 4.3 we proceed to investigate the characteristics of generated from individual models. In addition measuring TTR and perplexity, we also measure the Self-BIEUU score, along with the model's perplexity on the original text continuations. The Self- BLEU score measure of diversity between all generated texts, where lower indicates more diversity. Further details and information these metrics can be found in Appendix
Table 3 we note that Branch-GAN displays a much higher perplexity for the original texts than the baseline models. Texts generated by the Pythia models exhibit significantly lower perplexity and TTR compared to the original texts. In contrast, texts produced by Branch-GAN demonstrate TTR and perplexity scores that more aligned with the original texts, with the repetition penalty elevating both scores. As indicated by Franch-GANT lower Self-BLEU score, texts from Branch- GAN more diverse than other baselines, and even the original texts. Interestingly, GPT-4 has both higher perplexity and lower diversity than Branch-GAN, while matching the TTR value of both Branch- GAN and the original texts. By contrast, the LLama model has high diversity, intermediate TTR and low perplexity, making it more similar to the Pythia models except for diversity.
4.5 ROBUSTNESS
To quantify the robustness effects of our adversarial training, we investigate the models' ability to generate texts from varying levels of noisy contexts. We create noise via character repetitions, where randomly selected characters are repeated before tokenization. For these experiments we report how TTR and perplexity according to Pythia 6.9B's change for the generated texts, noise in the context increases.
As with previous experiments the context length is set to 32 tokens (after noise is introduced), and the generated texts are 96 tokens. This done for the combined 100 Wikipedia contexts and 100 Fictional Stories contexts that used in Section 4.4
The results in Figure 3 demonstrate that the original Pythia models are susceptible to noise, as the mean TTR drops roughly 0.3 units for small levels of noise. Branch-GAN, although experienc- ing an early drop in TTR of roughly 0.1 units, remains fairly stable up until 16 character repetitions. Similarly, the perplexity of texts generated by the Branch-GAN models remain stable for up to 16 character repetitions. After this, Tranch-GAN generations not using a repetition penalty start degrading.

---

Published as a conference paper at ICLR 2024
Branch-GAN 1B
Branch-GAN 1B"
Pythia
Pythia 6.9
Original
0.7
12
6.9b Perplexity
0.6
10
0.5
8
0.4
6
0.3
4
0.2
0.1
4
8
16
32
64
0
1
4
16
32
64
Character Repetitions
Character Repetitions
Figure 3: Effects on generated texts when character repetitions are injected into the context.
4.6 ABLATION STUDY FOR DEPTH AND SPARSITY
Our final experiment explores the effects of training with different generation depths d, and different numbers of branch sequences K. For each configuration we train a Branch-GAN 410m for 4 epochs using the setup described in Section [.1] Using repetition penalty of 1.2, each model then generates 200 continuations, each 96 tokens long. for the combined Wikipedia and fiction dataset described in Section 4.4
The first experimental setup keeps the depth = 16 fixed while K changes. The second keeps the K 32 number of branch sequences fixed, while d changes. Finally, since increasing K can be done without increasing the number of forward passes, we keep the number generated tokens K d 512 fixed d changes.
The results in compensated for by increasing the number of branches. However, find diminishing returns for TTR when increasing the depth beyond 8, especially when using repetition penalty. Interestingly, decreasing the number of branches to (K 16, d 16), even (K 8, d 16), performs nearly as well as (K = 32, = 16). Thus, using fewer branches could be source for future computational efficiency improvements.
that a certain depth is required to achieve good TTR, and cannot be
Branch- 410m
Branch-GAN 410m"
Original Fixed Tokens
Branches
Depth
6.9b Perplexity
K=11
K-8
K:16
K:24
K:32
K-32
K232
K=32
K:32
K=32
K=512
K=128
K:64
K:48
K:32
d-16
d-16
C-16
d-16
6-15
d-1
d-4
d-8
d-12
d-16
d-1
d-4
d-5
d-12
d-16
32
33
d-15
6-15
Figure 4: Results from training Branch-GAN 410 for 4 epochs, varying the number of branch quences K and the depth level d. The first row shows Pythia 6,9b's perplexity on the generated texts. The second row shows the TTR of the generated texts.
8

---

Published as a conference paper at ICLR 2024
5 DISCUSSION & CONCLUSION
This paper has introduced Branch-GAN, an adversarial learning setup utilizing the parallelization capabilities of Transformers. Although method is applicable sequential data in general, focus on text generation via L.Ms.. To this end we have conducted a human evaluation study to properly evaluate the textual quality of our method and existing baselines.
Our results clearly indicate that the text generation capabilities of LLMs can be significantly en- hanced by incorporating adversarial training. Our Branch-GAN models outperform LLMs that more than 10 times larger, without training on any new data, alluding to possibly greater gains when applied to larger models.
Interestingly, the Branch-GAN models display comparatively poor perplexity on held-out texts, while generating texts that match TTR and perplexity of texts written by humans. It thus appears that the adversarial training method favors properties that correspond to human preferences at the expense of minimizing perplexity on held-out datasets, something we further analyze Appendix However, we argue that any such conclusions require further evidence as our training experi- ments are performed on very small datasets. Additionally, we note that although the model is only trained on branching sequences of length 16, the beneficial effect generalizes generation of longer sequences (up to length 96 in our experiments).
E.S
The Branch-GAN method opens up for many interesting experiments, such exploring different sampling methods during training, modifying the relative size of the generator and discriminator, dynamically changing the generation depth, and fine-tuning towards specific downstream tasks. Ad- ditionally, we hypothesize that incorporating adversarial learning could drastically increase the data efficiency of LLMs, and encourage further exploration in this direction.
Another line of future research concerns the characterization of text quality using automatic metrics, which have only started explore in this paper using TTR, Self-BIEUU and cross-entropy. In addition to their use in automatic evaluation, such metrics could potentially also be incorporated into training objectives.
Finally, it is worth noting that many of our training configurations were constrained by our lim- ited computational resources. Therefore, we also encourage further research to investigate the ef- fects of scaling up our proposed method. This includes using larger foundation models and more training data, increasing the generational depth, and training models without relying on additional pre-training.
ETHICS STATEMENT
The research reported in this paper involves an extensive human evaluation. the interest fairness as well as data quality, annotators were hired as freelancers through Upwork and paid 7.5-10 USD for each batch 100 examples, which corresponds to an hourly rate 5-10 USD. All datasets used publicly available, as specified in Sections 4.1 and 4.2 For more information about the Pile,
refer to the published data sheet (Biderman et al. 2022).
REPRODUCIBILITY STATEMENT
The code used for training and evaluation, along with the human annotations underlying the evalua- tion Section are available at wtw.://cooooreedddaaaaaaaaaaaaaa GANN
All pretrained LLMs used in this research are publicly available, with the exception of GPT-4, which we have nevertheless included in the interest of comparing to the commercial state of the art, even though these results may not be reproducible and evaluation are likewise publicly available. Finally, the hyperparameter settings are described in Appendix to facilitate reproduction of our results.
later date. All datasets used for further training
hhtps:/wwww.ppwor.ccom

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The research presented in this paper was supported by the Swedish Research Council (grant no. 2022-02909) and by a donation from Meta.
REFERENCES
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence
prediction with recurrent neural networks. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 28. Cur- ran Associates, Inc., 2015. URL https: htttt:::/pproceedingg... www.wnnnri.... cc/pppr files/ paper/2015/F1 / 995 (96766656669999999 (tt.:///8555...3.044444447333310
Stella Biderman, Kieran Bicheno, and Leo Gao. Datasheet for the Pile. arXiv preprint 2200//0001b222200111
arXiv:2201.0731, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal. Pythia: A suite for analyzing large language models across training and scaling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardtt Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 2397-2430. PMLR, 23-29 Jul 2023. URL https: http:://proccedingg
mmrr. mm
press/v202/bidernme
html
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Hemighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL https:: https::/ppxoceeednggss..tt newrppss. cctpspccppppppeeeeeeeeeeeee fwwes//aaer//eeeeeeee EL2e/255cccccc000000011111111111
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712. 2023. URL
httts:://rxxv..om wttp://www..gg/////////3/3 12712
Massimo Caccia, Lucas Caccia, William Fedus, Hugo Larochelle, Joelle Pineau, and Laurent Char- lin. Language gans falling short. In 8th International Conference on Learning Representa- tions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net, 2020. URL ht
net wwtp:/etttttettee..e wwr.mriiiiiiooo.... BJggaavviE A
Tong Che, Yanran Li, Ruixiang Zhang. R Devon Hjelm, Wenjie Li, Yangqiu Song, and Yoshua Bengio. Maximum-likelinn augmented discrete generative adversarial networks. arXiv preprint arXiw:1702.07983, 2017. URL/tttps://arxii http:/aawww..rrrabara/aaaaaaaa//
07983
Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. DialogSum: A real-life scenario dialogue summarization dataset. In Findings of the Association for Computational Linguisties: ACL-
IJCNLP 2021, pp. 5062-5074, Online, August 2021. Association for Computational Linguis- tics. doi: 10. 10.18653/ findings-acl.449, URL htt.httss//cclaamhhcoogg... org.2200
Findings-acl. 449
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. URL
https: wwc.iivorrgg.ooooo org 2204 02311
10

---

Published as a conference paper at ICLR 2024
Cyprien de Masson d' d'Auumme, Shakir Mohamed, Mihaela Rosca, and Jack Rae. Training language GANs from scratch. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bur E. Fox, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL /ttpo////etttsppprp/rreeeeooeeg/ new htwp://nsnninii...eeeeeeeeeeeeee ccpppprr files/
Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi. A theoretical analysis of the repetition problem in text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 12848 11856,, 2021.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: An 800gb dataset of di- verse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. URL https: ///axiv..rq/abs//1100..1
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Sky/lonn00.cco
io/
OpenWebbeettorpps 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11) 133914413,114, 2020.
Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, and Jun Wang. Long text generation via adversarial training with leaked information. In Proceedings of the AAAI conference artificial intelligence, volume 32, pp. 5141-5148, 2018.
Felix Hamborg, Norman Meuschke, Corinna Breitinger, and Bela Gipp. news-please: A generic news crawler and extractor. In Proceedings of the 15th International Symposium of Information Science, pp. 218-223, March 2017. doi: 40.5281/2emodo.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL htRL:ttts::///eeeeeele net//forum?dd=ryyooyyyyr
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836.. 2020. URL httpt:///aaat www.aaaivvvo....... wtwpxii.org//b///00000
08361
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher: CTRL: A conditional Transformer language model for controllable generation. arXiv preprint arXiv:1909.05858, 2019.
05858
Sylvain Lamprier, Thomas Scialom, Antoine Chaffin, Vincent Claveau, Ewa Kijak, Jacopo Staiano, and Benjamin Piwowarski. Generative cooperative networks for natural language generation. In International Conference Machine Learning, pp. 11891-11905. PMLR, 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.11 33461, 2019. URL/attps://arxiv wwwpiii.rrgaab...o. 11000.1990..333
Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, and Ming-ting Sun. Adversarial ranking for language generation. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Gamett (eds.), Advances in Neural Information
Pwciiionnoooeeeeeee Processing Systems, volume 30. wwwr wwtp:rippppppppppp...pppppppp pdf
Curran Associates, Inc.. 2017. URL ht http:////otpp///ooooeccccc.ooooa
fi (1aeppppr/0277111222772555555330

---

Published as a conference paper at ICLR 2024
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference on Learning Representations. ICLR 2017. Toulon, France, April 24-26. 2017, Conference Track Proceedings. (peenReview.net,2011 2017. URLhetps: //openrevies www.wwwwww.. nwt. d=By 72udxe
Weili Nic, Nina Narodytska, and Ankit Patel. RelGAN: Relational generative adversarial net- works for text generation. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https:
//open/eviww.net//eeeeeeeee umm ddrJedd3R5tm
OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
URL
https:
/arxiv.org//bb//233.. 087744
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp- 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis- tics. doi: 10.3115/1073303.10733333 URL
wrgp:o0-....//0020000 1040
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon Ilm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.011 16, 2023. UtL/tttss://at //app:::://axxivii/ www..woo...ggooo....o...ooo..o.. /twg//a///////ao/////////o//aaa/ abs 22066 01116
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. 2019. URL https:
hwt.b.ppenai..coo/tttteee-eeeeee insaperrissed_mult.ttasss.. teatulleestttt_eeerrrrr.r...
models/ /asss//agggggeee models are
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(5):5485-5511, 2020.
Marc'Aurelis Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojeiech Zaremba. Sequence level train- ing with recurrent neural networks. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. http: http://aaxxvvoorro abs 011111111111...6111
David E. Rumelhart, Geoffrey E. Hinton, and Ronald Williams. Learning internal representations by error propagation. In David E. Rumelhart and James I.. Mcclelland (eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume I: Foundations, pp. 318- 362. MIT Press, Cambridge, MA, 1986.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier. Benjamin Piwowarski, and Jacopo Taming language GANs with cautious sampling strategies. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 18978-18989, Curran Associates, Inc., newrips cc wwtpe/pppeeeeeeeeeeeeeeeeeeeeeee 2020. URL httpss:/proceedd 99e www.22....611100ee9992222 2eeee//beeeeeee5ccccacaa-eeeeeee Pcder files//apprr/2207
Staiano. ColdGANs:
Thomas Scialom, Paul-Alexis Dray, Jacopo Staiano, Sylvain Lamprier, and Benjamin Piwowarski. To beam not to beam: That is a question of cooperation for language GANs. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 26555-266577. Curran Associates, Inc., 2021. URL
Ei file/dfoeEe666000000fffeeeed22f3
Stanislau Semeniuta, Aliaksei Severyn, and Sylvain Gelly. On accurate evaluation of GANs for language generation. arXiv preprint arXiv:1806.04936, 2019. URL httlt:ttp///// //arriv....ii..a/.a wtw.iiiorgg////...........
abs/ / 1806. 946
12

---

Published as a conference paper at ICLR 2024
Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Toward diverse text generation with inverse reinforcement learning. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 4361-4367 2018.
Sandeep Subramanian, Sai Rajeswar, Francis Dutil, Chris Pal, and Aaron Courville. Adversarial generation of natural language. In Proceedings of the 2nd Workshop on Representation Learning NorN pp. 241 241-251, Vancouver, Canada, August 2017. Association for Computational Linguis-
tics. doi: 1//11/5///111112000//
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a. URL arxiv.
htt5/ba/23300 13971
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09286 2023b. URL argiv.org//
abs/2307) 09288
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http::://proeedddnssseeerrririre
filess/paper//0177
Xuezhi Wang. Haohan Wang. and Diyi Yang. Measure and improve robustness in nlp models: A survey. In Proceedings of the 2022 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies, pp. 4569-4586. Association for Computational Linguistics Lingusssis (ACL), 2022.
R. J. Williams. Simple statistical
algorithms for connectionist reinforcement
learning. Machine Learning. 8:229 22299225,,111 1992.
Qingyang Wu, Lei Li, and Zhou Yu. Textgail: Generative adversarial imitation learning for text generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14067-14075, 2021.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. Proceedings of the AAAI conference on artificial intelligence, volume 31, pp. 2852-2858, 2017.
Yaoming Zhu, Sidi La,, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st international ACM SIGIR conference on research development in information retrieval, pp. 1097-1100, 2018.
Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. robustly optimized BERT pre-training approach with post-training. In Proceedings of the 20th Chinese National Conference Computational Linguistics, pp. 2218-122,, 2021.
13

---

Published as a conference paper at ICLR 2024
Original
Generated
Discriminator Original
2
...
N-3
N-2
N-1
N
2
N-2
3
N-1
N
1
1
4
1
1
1
1
1
1
Attention Targets
2
2
Geniraatd
Generated
Pass
Pass
Pass 3
Pass 4
Figure 5: The Branch-GAN discriminator.
A DISCRIMINATOR
As described in Section 3.2 The discriminator processes data in a manner similar to that of the Generator. The difference is that it requires additional step for the final generation pass, as is visualized Figure 5 The Discriminator makes three predictions at each timestep, as depicted in O These predictions used construct loss signal, incorporating the discriminator's current hypotheses about subsequent predictions, as illustrated in 6
A.1 DUAL VALUE HEAD LOSS
The intuition behind the Branch- GAN RL loss is derived from the three distinct sections formed by I, and B,, See below.. In Section of the loss, D, receives higher discriminate score than expected score from human-written text -- Having successfully fooled the discriminator, generator is encouraged to simply maximize the log probability of that token. In Section 2 of the loss, D,, surpasses the expected score for generated text B but falls short of nn-1- In this sce- nario, the log loss is linearly scaled based on its position between Ln-1 and B In Section 3 of the loss, Dn receives a score that is lower than the expected score for generated text. As the discrim- inator is confident that the token is fake, the generator is directed minimize that token probability. In the corner case that the value heads misalign, and I,-1 S Bn- the loss is simply discarded. Empirical findings indicate that such misalignments occur extremely rarely during training.
A.2 OTHER LOSS FUNCTIONS
Early stages of development experimented with more typical loss functions such as REINFORCE Williams [1922). This led to an unstable GAN training dynamic, that often caused a generator col- lapse. Switching to the dual value head loss mitigated this issue, and since switching zero collapses has been encountered. However, as this was early stages of the research further work is needed to properly evaluate different loss functions in this GAN scenario.
1 Section 1
Log( (n))
D
I B n
D
B
D
Section 2
mM
n-
I. Log( p(n))
D.B.
N-1
N
Bud
Section 3
Discriminator
Log(1 -p())
Figure 6: Visualisation of the Diseriminator predictions and the RL loss.
14

---

Published as a conference paper at ICLR 2024
B ANNOTATORS AND COLLECTED DATASET
Wikipedia and FictionStories
Description
vs. Original
vs. Branch-GAN 1B*
Both
Dataset samples
2000
1800
3800
Annotators
9
9
Models
10
10
10
Samples per model
100
100
100
Annotators per model
3
3
Evaluated model preferred (%)
18.3
24.1
21.1
Evaluated equally good (%)
23.1
23.3
23.2
Original/Branch-GAA IB preferred (%)
58.6
52.6
55.7
Complete agreement" (%)
39.65
38.2
38.9
Partial agreement" (%)
48.0
48.9
48.4
Complete disagreement (%)
12.35
12.9
12.6
Agreement reliability (Fleiss' Kappa)
0.22
0.26
0.24
Complete agreement: All three annotators provided the label.
Partial agreement: Two annotators provided same label, and one differed.
Complete disagreement: Each annotator provided a different label.
Table 4: Statistics on all the collected annotations. The first column shows statistics for compar- isons between models and original texts and the second for comparisons between our 1B parameter Branch-GAN model using repetition penalty of 1.2. The last column shows the statistics for all comparisons.
All hired annotators were provided with the following instruction:
"Using only your personal preference and no tools such as Google or ChalGPT. Given an initial text snippet and two potential confinuations, decide which of the continuations you prefer. Each text and continuation spans at most a couple of sentences.
Sometines the decision is straight forward, one alternative might be very bad/repetitive. Other times you might be unable to pick one. You can then claim they are of equal quality."
15

---

Published as a conference paper at ICLR 2024
C TRAINING DETAILS
Hyperparameter
Value
General
Max Sequence Length
128
Epochs
4
Generator
Optimizer
Adam
Learning rate (start)
10 10-8
Learning rate warmup steps
1000
Learning rate (end)
10
Branches K
32
Depth d
16
Sampling strategy
Top-k
k (in Top-k)
50
Section weight (RL Advantage)
Section weight 2 (RL Advantage)
I
Section weight (RL Advantage)
1.5
Discriminator
Optimizer
Adam
Learning rate
10 10-4
Value head loss weight o
0.2
Table 5: Hyperparameters for Branch-GAN
C.I COMPUTATIONAL COST
The computational burden of the Branch-GAi training highly dependent on the training hyper- parameters. Each increase in depth infers an additional forward pass per sample, whilst the number of branching sequences affects directly affects the memory load. On top of this both the generator and discriminator need to be trained.
The big upside to Branch-GAN its data efficiency, seeing that the our results were attained with less than 0.01% % of the pre-training data. Thus, although our experiments were executed with neuight-forwand Python PyTorch, Branch-GAN 1B comfortably completed epochs under 40 hours. The hardware for this training single DGX machine with 8 NVIDIA 1100SXX44400GB GPUs.
16

---

Published as a conference paper at ICLR 2024
D AUTOMATIC EVALUATION METRICS
D.1 PERPLEXITY
When evaluating perplexity of models we always provide 32-tokens long contexts from 100 ran- domly selected samples from both the Wikipedia and Fictional Stories datasets. For each context the model subsequently generates a sequence of tokens. We compute perplexity exclusively on these 96 generated tokens and average the results for all samples.
D.2 TTR
The type-token ratio (TTR) is the number of unique (sub)words divided by the length of the gen- erated sequence. TTR is a commonly used measure of lexical richness in the language acquisition literature. The unnormalized TTR has been criticized for not being comparable across texts of dif- ferent lengths. This a problem in case, since all sequences are of the same length.
D.3 BLEF
We compute the Self-BLBU metric as described by Zhu et al. (2018), using 32-token long contexts from both the Wikipedia and Fictional Stories datasets. For our evaluation, we randomly select 100 samples from each dataset, leading to a total of 200 samples. Each sample is then used to generate sequence 96 tokens. The reported Self-BLEU scores represent the average computed over these 200 generated sequences.
D.4 CHARACTER REPETITION
When distorting input texts we do so by repeating randomly selected characters in the contexts before tokenization. Any character in the context string may selected but we prevent characters from being repeated more than ones. Since repeated characters affect the tokenized sequence length (usually making it longer) we truncate the distorted sequence to 32 tokens, the same lengths as the original context. Below is an example of an original context and three different levels of distortions with one, two and four repetitions.
Original Wikipedia Context:
Jive Records without the band's authorization or consent. Back 2 Base X peaked at No. 12 on the Independent Albums chart, and at No. 154 on the Billboard 200. Allmusic's 's Rob Theakston wrote
Number of repetitions:
Jive Records without the band's authorization consentt Back 2 Base X peaked No. 12 on the Independent Albums chart, and at No. 154 on the Billboard 200. Allmusic's 's Rob Theakston wrote
Number of repetitions: 2
Jive Records withoutt the band's authorization or consentt. Back 2 Base X peaked at No. 12 on the Independent Albums chart, and at No. 154 the Billboard 200. Allmusic's Rob Theakston wrote
Number of repetitions:
Jive Records withoutt the band's authorization consentt. Back BasSe peaked No. 12 on the Independent Albums chart, and at No.. 154 on the Billboard 200. Allmusic's 's Rob Theakston wrote
17

---

Published as a conference paper at ICLR 2024
E ADDITIONAL EXPERIMENTS & RESULTS
E.1 FINE-GRAINED HUMAN EVALUATION
Table shows the fine-grained annotations for models compared to original texts. Table 7 shows the fine-grat fineggrineed annotations for models when compared against texts generated by Branch-Gin GAN 1B*
Table 6: Fine-grained human evaluation when compared against original texts. All Pythia models and LLama-2 use top-p sampling; all Branch-GAN models use greedy decoding.
Model
Wiki
Fiction Ties
Wins
Ties
Losses
Wins
Losses
Baselines
Pythia 410*
6.7
4.7
88.7
3.3
3.3
93.3
Pythia IB*
9.3
6.7
84.0
4.7
7.0
88.3
Pythia 6.9
9.3
14.7
76.0
10.7
16.7
72.7
LLama-2 7B*
22.3
9.7
68.0
34.3
4.3
61.3
GPT-4
41.3
39.7
19.0
37.3
30.7
32.0
Our Models
Branch-GAl 410M*
14.7
36.3
49.0
18.3
52.7
29.0
Branch-GAN 1B"
11.7
54.3
34.0
18.3
57.7
24.0
Table 7: Fine-grained human evaluation when compared against texts generated by Branch-GAN IB Pythia models and LLama-2 use top-p sampling; Branch-GAN model use greedy decoding.
Model
W Wiki
Fiction Ties
Wins
Ties
Losses
Wins
Losses
Baselines
Pythia 410*
5.0
8.0
87.0
6.7
4.7
88.7
Pythia IB*
10.3
13.0
76.7
12.0
6.3
81.7
Pythia 6.9B*
19.0
19.7
61.3
10.7
35.0
54.3
LLama-2 7B*
40.3
10.3
49.3
30.3
13.0
56.7
GPT-4
55.3
33.0
11.7
52.3
34.3
13.3
Our Modely
Branch-GAN 410M
28.0
29.3
42.7
17.3
53.3
29.3
E.2 ADDITIONAL SAMPLING METHODS
We limited hyperparameter testing for the baseline models with human evaluation, starting with configurations recommended in previous studies. For Pythia 6.9b, we assessed three setups: repe- tition penalty 1.2 with greedy decoding Keskar et al. (2019); repetition penalty 1.2 with top-p=0.9 sampling (Holtzman et al.] 2020); and no repetition penalty with greedy decoding. Table 8 shows that repetition penalty and top-p sampling yields the most favorable results.
Table 8: Human evaluation results for different sampling settings for Pythia 6.9B in comparison to original texts (Wiki, Fiction) and to Branch-GAN IB* (Wik-iGG-iB Fic(BG- Numbers represent the percentage of texts judged to be better than as good as the alternative candidate.
Model
Wiki
W ik BG-IB*
Fiction
BB-I
Pythia 6.98B (greedy)
24.4
20.4
25.3
Pythia 6.9 (greedy)
14.0
16.7
28.3
38.0
Pythia 6.9B* (top-2)
24.0
38.7
27.4
45.7
18

---

Published as a conference paper at ICLR 2024
E.3 ADDITIONAL DATASETS
To complement the human evaluation performed on Wikipedia and Fictional Stories, we supply automatic evaluation additional datasets. These datasets selected cover wide range of different domains and consists of:
et al. (2017). Dialogsum Chen et al. (202T).
Legal Summaries Medical Transcription 5 and Open Web Gokaslan & Cohen 2019
Similar to all previous tasks, each dataset is chunked into into segments of 128 tokens, from which sample 100. These are then split into tokens pretext and 96 tokens continuations. Each model generates 96 continuation tokens, from which calculate the perplexity and TTR on these. This allows us compare the metrics of the generated texts against that of the original continuations.
Table the perplexity on these texts according to Pythia 6.9B. For each model we show Pythia 6.9B's perplexity on the original texts minus the perplexity on the generated texts. For all datasets, the Branch-GAN models clearly generate texts with perplexity closer to that of the original texts. The biggest difference are for the News and Wikipedia texts, and the smallest difference is for the dialog and fiction texts. For both Branch-GAN models top-p sampling yields a lower difference.
Table 10 shows the TTR, and for each model we show original TTR minus the TTR of generated texts. Again, the Branch-GAN models generate texts that have TTR significantly closer to that the original texts. Interestingly. the biggest difference in TTR are for Dialogsum and Fiction texts, and the smallest difference is for news texts. This is almost completely reversed ordering compared to the perplexity results. However, just as with the perplexity results, Branch- GAN models using top-p sampling resulted in slightly better results.
Table 9: Perplexity Scores According to Pythia 6.9B
Model
Legal
OpenWeb
Dialogsum
Medical
News
Wikitext
Fiction
Original Perplexity
Original Texts
8.67
9.41
7.25
11.52
14.03
14.79
11.37
Difference In Perplexity Compared To Original
Pythia 410m (g)
7.12
7.56
6.02
9.95
12.20
12.89
9.66
Pythia 410m* (g)
7.03
7.26
5.98
9.69
11.81
12.66
9.44
Pythia 410m* (s)
6,85
7.09
5.94
9.59
11.53
12.48
9.20
Pythia IB (g)
7.09
7.74
6.05
10.00
12.12
12.90
9.63
Pythia IB* (g)
7.00
7.38
6.01
9.81
11.80
12.65
9,43
Pythia 1B* (8)
6.98
7.21
5,94
9.62
11.65
12.57
9,22
Br-GAN 410m (g)
3.36
1,41
4.11
4.82
5.56
5.68
1.61
Br-GAN 410m* (g)
1.84
-0.42
4.03
3.31
4.55
4.56
0.45
Br-GAN 410m (8)
0.39
2.26
3,95
3,55
0.03
Br-GAN IB (g)
1.82
1.63
2.98
4.86
5.36
5.24
1.37
Br-GAN 1B* (g)
1.34
1.20
1.20
4.93
5.59
4.82
0.58
Br-GAN 1B* (S)
0.53
1.70
-0.05
4.01
4.85
4.34
0.07
19

---

Published as a conference paper at ICLR 2024
Table 10: Type Token Ratio
Model
Legal
Web
Dialogsum Original TTR
Medical
News
Wikitext
Fiction
Original Texts
66.2
74.2
61.8
68.8
73.4
71.8
71.8
Difference TTR Compared Original
Pythia 410m (g)
37.9
43.0
39.9
44.8
43.2
42.8
49.9
Pythia 410m* (g)
34,8
34.0
39.0
38.3
34.4
38.3
44.3
Pythia 410m*
28.9
33.5
37.8
36.9
30.3
36.0
40.6
Pythia 1B (g)
33.2
42.2
38.9
42.3
39.2
37.8
46.1
Pythia 1B* (g)
29.5
38.7
30.5
40.2
Pythia 1B* (s)
28.5
30.3
37.8
30.9
28.1
30.9
37.0
Pythia 6.9B (g)
28.9
30.7
36.4
40.3
33.2
33.4
38.15
Pythia 6.9B* (g)
24.1
20.0
37.3
33.2
25.1
25.0
32.0
Pythia 6.9
24.2
19.4
34.6
19.6
30.3
Br-GAN 410m (g)
15.2
10.6
26.4
12.6
11.5
9.8
8.0
Br-GAN 410m (g)
5.7
-42
21.4
2.1
5.4
1.5
2.1
Br-GAN 410m (8)
4,9
3.2
16.4
1.9
2.4
0.6
1.0
Br-GAN (g)
0.6
3,4
14.4
5.7
1.8
Br-GAN IB (g)
-3,6
-1.4
2.5
0.7
-0.1
-1.9
-2.1
Br-GAN IB (s)
-4.1
-0.6
-2.5
-2.3
-0.0
-3.3
-1.7
20

---

Published as a conference paper at ICLR 2024
0.300
1.0
Pythla 430m
0.18
1.0
Pythia
8.175
0.26
BBELUU
1.0
B-BLEU
0.24
1.0
0.125
6.100
0.50
0.66
0.50
0.08
0.31
2322
0:21
0.54
0.55
0.50
0.57
6.93
039
0.55
1.:4
2:15
0.9
0.7
Negative BLEU
Negative BLEU
Figure 7: Quality and Diversity scores for different temperatures. Temperature 1.0 as was used during all other experiments is marked for each model
E.4 TEMPERATURE SWEEP
Caccia et al. 2020) found that although GAN training can perceivably increase textual quality, this may be for specific temperature, and at the cost of diversity. Therefore they propose experiment that evaluates quality and diversity over a sweep of different temperatures using automatic metrics.
For each temperature the model generates 10k texts the contexts of a corpus. Quality is measured via the BLEU overlap (Papineni et al. 2002) of the generated texts and texts from the test set. Diversity is measured using self BLUF (Section D). The proposed experiment is analyzed using two-dimensional scatter plot over the two metrics, where the quality metric is inverted. Thus the plot origin is the ideal for both metrics.
We use the EMNLP News-20217 corpus from Subramanian et al. (017). Similar to the original work we also explored temperatures in range of [0.9, 4.0,. Finally, we note that all evaluation was performed using the original published code
Figure Socure7conmee contains the results of this experiment, with temperature 1.0 marked as this is the temper- ature that was used during all other experiments. The results clearly indicate the both the Pythia models generate texts that overlap more with the test sets. The Pythia models are also able to find temperatures that yield similar quality to that of Branch-GAN's GAN's best quality, but with higher diver- sity.
Interestingly, the quality measurement of this experiment radically contradicts the results attained from our human evaluation study. As clearly demonstrated in Section 4.2] Branch-GAN with a temperature of 1.0 generates texts with a higher quality, according to humans. However, the diversity metries for temperature 1.0 of both models are aligned with our results Section 4.4
E.4.1 AUTOMATIC METRICS FOR OPEN-ENDED TEXT GENERATION
We fully support the importance of evaluation over different sampling temperatures, but we argue against the use of BLEU as automatic evaluation metric for open-ended text generation. The discrepancy between our human study and the BLEU scores corroborates this stance.
Considering the vast amount of possible texts that of "high quality", the quality of open-ended text generation is not easily captured by overlap metrics. These metrics have been shown to correlate well with human judgment on constrained tasks such translation image captioning. However, to the best of our knowledge, no study has demonstrated their correlation on open-ended text gener- ation.
Admittedly, this is problem for the field of open-ended text generation as human evaluation is both timely and expensive. Until further work has progressed on automatic metrics we are thus unable to properly evaluate large ranges of temperatures. Conclusively, we cannot disprove that there exists sampling strategy where the original Pythia models outperform our Branch-GAN models.
21

---

Published as a conference paper at ICLR 2024
E.5
NEGATIVE LOG-LIKELIHOOD
Table 11: Average NLL Scores for Pythia and Branch-GAN Models
Generated
Original Fiction +
Model
Good Texts 4
Bad Texts
Wiki
The Pile 4
Pythia 410m
3.10
2.20
3,42
3.35
2.56
Pythia IB
2.86
2.06
3.17
3.20
2.38
Pythia 6.9B
2.75
1.87
2.80
2.86
2.10
Branch-GAN 410m
3.57
3.10
4.14
4.03
3.15
Branch-GAl GAN IB
3.27
3.20
4.13
4.10
3.10
To accompany the automatic evaluation Section [..4 we provide the models negative log- --llkkiihhodd scores for different datasets. For each dataset select 1000 texts, and the sequence length of each text is 128 tokens. The datasets fall into two categories: Original and Generated.
The original texts are simply texts taken from the test set of the corresponding dataset. The gen- crated texts, from any possible model, that received either all positive annotations, or all negative annotations. These are the same used in Section 4.3 Thus the generated texts are split into Good and Bad.
From the results in Table II is clear that the original Pythia models achieve lower NLL score across all datasets. However, this also includes the bad texts, that demonstrated the lowest NLL score. This is likely due to all the repetitions within the bad texts, making them more predictable. In contrast, the Branch-GAN has a higher overall NLL score, but the bad texts are not much likelier than the good texts.
E.5.1 INTERPRETATION & DISCUSSION
The original Pythia models are trained to predict the next token given a context. However, even though repetitive text may easily predictable, such predictability is not necessarily the desired generative behaviour. We argue that the ultimate goal of good generator is not to predict the most likely next token given a context. Instead, it should produce continuations that appeal to the reader. In cases where the context is repetitive and of low quality, continuation that can break this predictable pattern and enhance the overall quality is preferred.
The Branch-GAN models have attained their loss signal differently, they try produce sequences indistinguishable from real data, regardless of whether the previous generated token was good or bad. This opens up for the possibility of course correction during the training where, after repeti- tion is encountered, it might still be beneficial to generate something different. This would explain why the Branch-GAN models exhibits a higher NLL score for the bad texts, although they are easily predictable.
Whether these loss functions differ only in intuition, or actually lead theoretically and practically distinct generators, is beyond the scope of this paper. However, as seen in Section 4.2 Branch-G.i can generate texts preferred by human evaluators, despite performing worse on terms of PPL and NLL on original data. We acknowledge that these are preliminary results and strongly encourage further investigations into this direction research.
22

---

Published as a conference paper at ICLR 2024
F EXAMPLE OUTPUTS
F.1 BRANCH-GAN 410M GREEDY REPPENALTY: 1.2 BEAMSIZE: 8
song recorded by english-irish boy band one direction for their second studio album, take me home 2012) ). it was
Wikipedia Context:
kiss you
Generation: written and sung by. the lyries were inspired from a poem written by geoffrey o o'reily in. it tells story of love gone wrong between a father and a son and how the son forgivingly accepts his father as his real father, while simultaneously punishing him for not accepting his true father as his real father, in order to maintain the illusion of father and son as one and the same, as evidenced in the song lyrics and imagery. contents
Wikipedia Context:
out a t-34 at tugok, also called morisil, e company never reached its blocking
position. strong north korean force surprised and delivered
Generation: a surprise counterattack, which resulted in heavy fighting along the entire front line. heavy fighting continued along the entire front line, foreing enemy units reorganizing and leading to heavy casualties among the troops on both flanks, as well as among the artillery units along with tanks and armored personnel carriers. in addition heavy fighting and shelling by ground forces and air units, there were reports that at one time or another, at least two enemy tank battal
Wikipedia Context: with the increased number of civilian casualties compared with the world war i. winston churchill agreed to ware's proposal that the commission also maintain a record of commonwealth civilian
Generation: casualties, to enable the committee to take steps to prevent similar tragedies from occurring again in future. following the publication of this report, number of members of parliament (mps) raised concerns regarding the committee's handling. amongst others, david dimble mp, who chaired a committee meeting at which it considered evidence relating to the number, location and circumstances of civilian casualties during the first world war said: is deeply regrettable that, following the publication
to assume that it was this artificial underworld that such work as was necessary to the comfort of the daylight race was done? the notion was
Fiction Context:
Generation: repugnant to my sense of humanity; and besides repugnant to my convictions as to the moral principles by which it ought to be conducted, there seemed to me a fundamentally untenable inference to be drawn therefrom. for it seems to me, first of all-and this arises from my reading of shakespeare- of supposing that there was nothing in the underworld but what belonged to the humane order
playing with the other girls and lads, she had fled into the wood, though never so far as this. it was pleasant be hidden and Generation: yet to be seen at the same time as all these young fellows; for it gave one the opportunity of watching what was going on around them, while the same moment hearing what they said and doing what they did, without being overheard. this made one feel more kindly towards them all, for it gave them something to whisper about, while at the same moment hearing what they said and doing what they did without being overheard.
Fiction Context:
Fiction Context:
they came, and then, pointing to the bronze pedestal, i tried to intimate my wish to open it. but at my first gesture towards
Generation: it i saw only the shadowy outline and dim light of the figure beneath, for which reason i felt& a pang so keen that i began to doubt whether it had really moved at all, or whether it only spun round my brain like spinning wheel, and whirling round and round incessant circles, like some gigantic spinning wheel which awaits its completion. but as i gazed at that figure beneath my hand seemed to me
23

---

Published as a conference paper at ICLR 2024
F.2 BRANCH-GAN IB GREEDY REPPENALTY: 1.2 BEAMSIZE: 8
Wikipedia Context: in mid-november, the jma briefly tracked a weak tropical depression near wake island. the agency also briefly tracked tropical depression off the coast of vietnam on Generation: "new year's eve and again early on new year's day. over the next several days the jma tracked several other tropical storms as well as several mid-level hurricanes. most of the storms tracked well away from land and did no direct threat to u.s. cities. however, two tropical storms, one mid-level hurricane, one tropical storm near the lesser antilles, two hurricanes near the us west coast, storm in mexico,
Wikipedia Context:
experts disagree about the number of species in the genus attalea ( (broadly defined), ). in 1965, dutch taxonomist jan gerard wessels Generation: made the surprising claim that there were only two attalea species. he cited evidence from a variety of sources to substantiate his claim. these sources included fieldwork in the swartz mountains in the u.s. ( (wheee wessels also made field trips part of his fieldwork ), letters written by attalea to her sister elinor (elinor wasn't 't a writer, so elinor wrote these letters) ), fieldwork by
Wikipedia Context: upon drafting new charter for manila in june 1901, the americans made official what had long been tacit : that the city of manila was not intramuro
Generation: and would henceforth be treated as if it were another philippine province, like cebu or zamboanga. 1902, president mckinley issued executive order no. 10, formal recognition of manila's s extraterritorial jurisdiction. the following year, president woodrow wilson formally recognized manila by proclamation. 1908, congress ratified the treaty whereby the philippines ceded all of their mineral and agricultural rights to the united states in exchange for $15,000,000.
yawned and went on along the cliff, and they followed. then he stopped and went back. it was a splendid night, beset
Fiction Context:
Generation: only by clouds that moved restlessly across the sky, if the very bed itself was restless, shifting from one side of the room to the other, as if the mattress beneath him quaked from time to time. but he lay still, eyes closed, letting his head fall back that the top of it rested against the stone wall of the cave, just above the level of his
Fiction Context:
the conditions of underground life, and as happy in their way, as the overworld people were to theirs. seemed to me, the
Generation: wisest course would be for them all to submit to the will, or at least to abide by it-to believe it, and act upon it, as a universal law, for, surely, if they did so, there would be no room left for error, for there would be but one standard of truth the will itself! and this being so, what could be wiser than obeying it, thinking it right, and doing it
day." the she-bear did not answer, but there came a measured crunching sound out of the darkness. she had turned up a bone Generation: and was tearing it apart, smearing the white meat all about with her demented gleam. "ha!" she said. "now what was that noise?" and with that, a huge shadow overtook her, blotting out the moon and all the other stars in the firmament. it stood over them like an angry god. then suddenly there was flapping sound, as though great many bats had come out of the
Fiction Context:
24

---

Published as a conference paper at ICLR 2024
F.3 PYTHIA 410M
Top-P: 0.95 REPPENALTY: 1.2 BEAMSIZE: 8
Wikipedia Context: representative of the jewish consciousness. rosenfeld noted that while holocaust literature often either soars to expostulation and apostrophe ", sinks to Generation: paltry, self-congratulatory tone ", the same can be said of much of the literature of the post-holocaust period. rosenfeld also noted that the holocaust literature of the post-holocaust period was characterized by a great deal of self-congratulation, as well as a great deal of self-deprecation. for example, rosenfeld noted that the holocaust literature of the post-holocaust period was charac- terized by great
Wikipedia Context: mastodonsaurus and other similar animals were referred to as labyrinthodonts, named like labyrinthodon for teeth that were highly folded in cross section
Generation: references category:late cretaceous dinosaurs of north america cate- gorylabyyrinhodonns category:fossils the united states category:fosiils canada category:fossils of mexico sategory:fossic of south america category: of the united states category:fossils of mexico sategory:fossis of canada category:fossils of the united states
Wikipedia Context: on july 9, pagasa classified a system as tropical depression falcon, off the west coast of the philippines. the jma also briefly initiated advis
Generation: ories on the system. on july 10, pagasa issued tropical cyclone watch for the area of the storm. on july 11, pagasa issued a tropical cyclone watch for the area of the storm. on july 12, pagasa issued tropical cyclone watch for the area of the storm. on july 13, pagasa issued a tropical cyclone watch for the area of the storm
Fiction Context: club, and suddenly he perceived the occasion and, rolling over, began to struggle to his feet. he stood but clumsily. he supported
Generation: himself with one hand on the wall, and with the other he clutched at the window-sil "who you?" he cried. "who are you?" "it is i," said the man. "who are you?" "it is i," said the man. "who are you?" "it is i," said the man. "who are you?" "it
Fiction Context: in the same grey covering. then perceived, standing strange and gaunt in the centre the hall, what was clearly the lower part of a Generation: woman's 's body, and knew that it was the body of woman. did not know who she was, but knew that it was the body of woman. was the body of woman, and was the body of woman, and it was the body of woman, and it was the body of woman, and it was the body of a woman, and was the body a woman, and it was
Fiction Context: a time my brain went stagnant. presently got up and came through the passage here, limping, because my heel was still painful, and
Generation: couldn't 't stand it any longer. "i'm sorry," said, "but can't stand any longer. i'm going to get out of here." "you can't I get out of here," she said. "yes i can," said. "no, you can't," she said. "yes can," said. "no, you can't she
25

---

Published as a conference paper at ICLR 2024
F.4 PYTHIA IB ToP-P: P: 0.95 REPPENALTY: 1.2 BEAMSIZE: 8
Wikipedia Context: hünigen, the battles at ostrach and stockach, and the northern swiss campaign of 1799 that included battles at winterthur and zürich
Generation: as a result of his service in the war of the austrian succession (1740-48), he was made knight of the order of the golden fleece. references external links category:1743 births category:1800 1802 deaths category. generals category:austrian military personnel of the war of the austrian succession category.reciiiets of the order of the golden fleece category:
Wikipedia Context: from his base tel aviv, holocaust survivor emmanuel lieber directs a group of jewish nazi hunters in search of adolf hitler. lieber believes that the former
Generation: nazi leader is hiding in plain sight in the united states. lieber's film is based on the true story of the hunt for adolf hitler in the united states during world war ii. in the film, lieber travels to the u.s. holocaust memorial museum in washington, d.c. to meet with holocaust survivors and descendants of those who participated in the hunt for adolf hitler. lieber's film will be shown
Wikipedia Context: following the release of the album, stone sour went on to promote for over year; releasing five singles and touring in several regions, including the united Generation: states, canada, mexico, and the united kingdom. after the release of the album, stone sour went on to promote it for over year, releasing five singles and touring in several regions, including the united states, canada, mexico, and the united kingdom. after the release of the album, stone sour went on to promote it for over a year: releasing five singles and touring in several regions, including the united states, canada, mexico,
Fiction Context: then came the bats, and the star that was like ugh-lomi crept out of its blue hiding-place in the west. she called to Generation: him, but he did not answer, and she went on calling, but he did not answer, and she went on calling, but he did answer, and she went on calling, but he did answer, and she went on calling, but he did not answer, and she went on calling, but he did not answer, and she went on calling, but he did not answer, and she went on calling. but he did not answer
looks more like the first half of an animal than anything else in the world," he said. "fore-legs and no hind."
Fiction Context:
Generation: what do you mean by fore-legs?" asked. "the fore-leg is the part of the body that is closest to the ground, and the hind-leg is the part of the body that is farthest from the ground." "what do you call the hind-leg?" asked. "the hind-leg is the part of the body that farthest from the ground, and the fore-leg
Fiction Context: some months, and he was thinking of "reducing the figure," when he made singular discovery. at that time his health was very had
Generation: and he was obliged give up his profession, which he had been practising for more than twenty years, and to return to his old home, where he had lived for the last ten years. he was then about fifty-five years of age, and he was man of great strength of body, but his mind was feeble, and he was subject to many diseases. he had been in the army, and had served
26

---

Published as a conference paper at ICLR 2024
F.5 LLAMA2 7B
TOP-P: 0.95 REPPENALTY: 1.2 BEAMSIZE: 8
Wikipedia Context: the irish republican army (ra had been inactive militarily since the end of the border campaign in 1962. it was low in both personnel and Generation: was not until 1969 that the provisional ira (pira) began a campaign of bombings and shootings against the british army, the royal ulster constabulary (ruc) and protestant civilians in northern ireland. in 1970, the pira bombed the old bailey in london, killing one person and injuring 200. in 1972, the pi
the player, as donald, ventures through a variety of side-scrolling levels. generally, each level is divided into an overland part and dungeon Generation: part. in the overland part, the player controls donald from a top-down perspective, while in the dungeon part, the player controls donald from side-scrolling perspective. in the overland part, the player's goal to reach the end the level, while in the dungeon part, the player's goal is reach the end of the dungeon while collecting items and defeating enemies. ## reception reception agg
Wikipedia Context:
during the withdraw, however, a company ran into sizable north korean force and had scattered in the ensuing fight. most of the company, including its commander Generation: 's officers were killed or wounded, and the survivors found themselves cut off from the rest of the battalion. on the morning of june 25, 1950, the 24th infantry division's 2nd battalion, 34th infantry regiment, commanded by It. col. charles b. smith, was ordered to relieve the beleaguered company of the 19th infantry regiment.
Wikipedia Context:
Fiction Context:
somehow, to feel any humanity in the things. only my disinclination to leave weena, and persuasion that i
Generation: did should never find her again, prevented me from doing so. it was about two oloko in the afternoon when returned to the spot where i had left weena. she was there. i wandered about for some time, but could find no trace of her. at last came to the conclusion that she must have wandered away in search of food, and that had better return to the hut. when i reached the
Fiction Context:
sunlit city of delight he knew of in japan, half-way about the world. she loved the dream, but she feared the leap;
Generation: she loved the man, but she feared the leap; she loved the leap, but she feared the man. she loved the dream, but she feared the leap; she loved the man, but she feared the leap; she loved the leap, but she feared the man. she loved the dream, but she feared the leap; she loved the man, but she feared the leap; she loved the leap, but she feared the
Fiction Context:
began to put my interpretation upon the things had seen, and as it shaped itself something in this way.
to me that evening, my interpretation
Generation: it was the middle of the afternoon, and i was sitting on the edge of the porch, with my feet dangling in the grass, and my elbows on my knees, and my chin in the hollow of my hands, and my eyes on the ground, watching the ants as they came out of the wood-pit and trailed off into the sweet-sie sweet-smelling grass, watching the squirrels chase each other up and down the
27

---

Published as a conference paper at ICLR 2024
F.6 GPT-4
Wikipedia Context: spacey was mendes" first choice for the role of lester, though dreamworks had urged the director consider better-known actors; similarly, the Generation: studio suggested a number of high-profile actresses for the part of carolyn, lester's dissatisfied wife. however, the director strongly believed in his casting choices and fought for spacey and annette bening, whom he felt embodied the characters perfectly. mendes felt that spacey had the ability to portray lester's complex personality, showcasing the diametrically opposed facets of his character the comical and the tragically flawed. spacey's effortless versatility as an actor de-cl
osbert was no longer archdeacon by 1158, as his successor is attested by that point. osbert, however, continued to call himself
Wikipedia Context:
Generation: archdeacon" through the years. this wasn't due to delusions or an inability to accept his reality, but rather as means of maintaining a significant influence within the clergy and his surrounding community. his knowledge, experience, and wisdom were still highly regarded and sought after, and his prof-prrcl title seemed carry sense of endearment rather than mockery. despite losing his official title, osbert remained an integral part of the church; he assisted in monumental projects
Wikipedia Context: new york state route 31b ( ny 31b) was state highway in central new york in the united states. it served connector between ny 31
Generation: and ny 96 in the town of clifton springs. the highway was designated in 1949 and ran for approximately 6.6 miles through the rural landscapes of ontario county. the term "31b" was not indicative of a sequential order in the new york state route system, but instead detailed the nature of the route branch of ny 31. the highway started at intersection with ny 31 in the hamlet of gypsum and moved east into cl
Fiction Context: but his chief thought was of the crystal. he approached that topic in a gingerly manner, because he knew mrs. cave's peculiarities. he
Generation: had dealt with her in the past and remembered how fiercely she guarded the family heirlooms, how her eyes would sparkle with fierce territoriality whenever anyone expressed the slightest interest in them. mrs. cave, a solid woman of uncertain age with a hairline receding much faster than her fiery spirit, could become a lioness if she perceived threat to her treasure. but this time, it was different. this time, he needed the crystal, not just as a curiosity
Fiction Context: why, too, if the eloi were masters, could they not restore the machine to me? and why were they so terribly afraid of the Generation: dark? these questions tormented me as i sat among them, overwhelmed by their alienness. the eloi, in all their gentility and grace, held deep-rooted terror of the night. observed sense of dread creeping into their eyes as the sun began its descent, their laughter and merriment getting replaced by silence. not a single one of them dared to move without flicker of terror on their beautiful, fragile faces. couldn't understand
been speculating upon the destinies of our race, until have hatched this fiction. treat my assertion of its truth as mere
Fiction Context:
Generation: figment of my overactive imagination, an effusion of my persistent curiosity about the trajectory of humanity. yet beneath all my declarations lies an abysmal concern. in the grand purview of human history, could it not be possible that my fantasy may morph into reality? could not be that we, an entity constantly evolving and pushing the boundaries of our knowledge, unknowingly sowing the seeds of these imagined futures with every choice we make? my conjured world
28


Reviews:
[{"review_id": "Wwn0id3STj", "reviewer": "Reviewer_RVL4", "summary": "This paper introduces a new strategy for training language models using a combination of adversarial and MLE style losses. Historically, Language GANs have been hard to get to work well---successful methods have mainly resorted to primarily relying on MLE losses with a minor amount of adversarial fine-tuning. This work tries to alleviate some of the optimization issues for discrete language GANs by using a efficient training strategy that simultaneously optimizes multiple generations that branch off of a supervised sequence at various points in time. Combined with a MLE + dense rewards criterion, the paper shows that an effective language models can be learned.", "questions": "- I'm not clear about Table 4 in the appendix --- which model is being evaluated? I would have thought Branch-GAN, but don't understand why it would be compared \"vs. Branch-GAN 1B*\". \n- I'm also curious about win-rate _between_ models (it's noted in Sec. 5 that GPT-4 is consistently preferred over Branch-GAN, but I could only find comparisons to original texts in the paper).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly written, and the approach is simple while also appearing to be very successful. The results are also well-evaluated (at least for the specific text-continuation setting that is focused on). More broadly, I think that the paper introduces an interesting efficient batching technique for training language models with reinforcement learning --- that can possibly be applicable to other types of rewards beyond GANs (e.g., such as RLHF models). \n\nThe dataset of manual text quality rankings also seems like a valuable contribution if released, and the analysis on correlation (or lack thereof) of automatic metrics with human judgements could be of interest to practitioners.", "weaknesses": "Though the human evaluation on text completion is good, I'm really quite curious to see how the Branch-GAN performs on tasks that are more divergent from the pre-training setup, such as prompting tasks. \n\nI also find the loss function in Eq. (1) to be poorly motivated. Why choose this variant, over say REINFORCE/PPO with a baseline/advantage function? (I would also at least move more of the description of the loss in A.1 to the main text.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vJhRQCaygT", "reviewer": "Reviewer_Hqg8", "summary": "This paper presents Branch-GAN, a method for improving text generation with Transformers using generative adversarial networks (GANs). Branch-GAN generates multiple branching sequences from each training sample, and trains the generator and the discriminator in parallel. The paper shows that Branch-GAN models produce higher quality texts than large language models (LLMs) and other baselines, according to human evaluation and automatic metrics. The paper also explores the effects of noise, depth, and sparsity on text generation, and discusses the potential applications and limitations of Branch-GAN.", "questions": "How well does Branch-GAN generalize to other domains or tasks, such as summarization, dialogue, or question answering? What are the challenges or limitations of applying Branch-GAN to these scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Novel method**: A novel method for training Transformers in a GAN setup, which improves text generation quality with fewer model parameters and less data. I really like this idea to branch the sequence for efficient discrimination.\n2. **Human evaluation**: A large-scale human evaluation study that shows that Branch-GAN models generate significantly better texts than LLMs and other baselines.\n3. **Robustness**: An analysis of the robustness of Branch-GAN models to noisy contexts, showing that they can generate texts with stable TTR and perplexity.", "weaknesses": "1. **Lack of comparison**: The paper does not compare Branch-GAN with other language GANs or cooperative language GANs that use pre-trained Transformers.\n2. **Limited evaluation**: My main concern is that the paper only evaluates text generation on two domains (Wikipedia and Fictional Stories) and does not test the generalization ability of Branch-GAN on other domains or tasks, such as summarization, dialogue, or question answering.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "62nhTIJvdI", "reviewer": "Reviewer_MWH7", "summary": "The paper presents Branch-GAN, a novel method that improves text generation quality in large language models (LLMs) by using adversarial training. It overcomes the deficiencies of current models such as repetitive texts and looping issues by generating multiple branching sequences from each training sample, providing dense signals for both generator and discriminator. This leads to stable training dynamics and high-quality text generation, using only a fraction of the training data typically required. Human evaluations confirmed that Branch-GAN outperforms much larger baseline models and has robustness against noisy inputs. The paper concludes that adversarial training enhances LLMs' capabilities and suggests further exploration into adversarial learning for increased data efficiency and application to larger models.", "questions": "See Weaknesses)", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Advantages:\n\n - Higher Quality Text Generation: Branch-GAN can generate texts of significantly better quality on average compared to other models, as confirmed by human evaluations.\n - Data-Efficient Fine-Pretraining: The method achieves this enhanced performance while utilizing less than 0.01% of the original training data, indicating exceptional data efficiency. \n - Stable Training Dynamics: By generating multiple branching sequences from each training sample and processing them in parallel, Branch-GAN provides a dense and stable learning signal, leading to more stable training dynamics compared to existing language GANs that leverage sequence-level rewards (e.g. SeqGAN, RankGAN etc). I especially like the idea of using three prediction heads such that it implicitly incorporates the benefits of scheduled sampling, which is an algorithm that is known to practically help the model produce better outputs, despite some criticism from How (not) to train your generative model [https://arxiv.org/abs/1511.05101 ].", "weaknesses": "Disadvantages:\n\n - Confusing Illustrations: While many of the important elements have been shown in Fig1, I would really appreciate if the authors can considering showing some technical details of the discriminator in Fig 1 too. With the current illustration it's really difficult to know enough details about how the dense adversarial rewards are constructed, which are actually the most novel part of the proposed method. (The efficient pretext encoding reusing trick is good, but it's not what I think that differentiates BranchGAN from other language GANs)\n\n - Lacking Mode Collapse Detection and/or Diversity Measure [Addressed during Rebuttal]: According to some previous criticism against language GANs from Language GANs Falling Short [https://arxiv.org/abs/1811.02549 ], most previous language GANs that have shown effectiveness in producing more coherent output sequences generally can be fragile against the comparison of simply annealing the temperature of language models. In other words, it could be the case that BranchGAN, too, sacrifices diversity/mode coverage in exchange for precision/quality. A proper study on this is needed to respond to such challenges. **This is dangerously needing attention** because the remarkably increased PPL of the BranchGAN-generated samples by Pythia 6.9b and remarkably increased generation quality reminds me of some previous language GANs that also over-claim themselves to be able to generate high-quality samples, eventually turned out to be rather trading-off between quality and diversity and had zero improvement when considering both aspects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ls688g6Lxg", "reviewer": "Reviewer_rZ5m", "summary": "The paper proposed an adversarial training strategy to enhance the generation ability of LLMs, it generated multiple external branches along the input sequence and add RL loss over these branches. The manual evaluation on 10k+ annotations is conducted to demonstrate the effectiveness of the proposed method.", "questions": "1.For different generation tasks (such as Wiki, Fiction generation), was the proposed method tuned separately? In other words, did the current version support multi-task tuning?\n2.For sequence S with length N, K unique starting points is randomly sampled from [1, N-1] as branches. As each generated branch consists of d (e.g., 16) tokens, if the sampled starting point is N-1, is the corresponding generated branch easy to be recognized than that corresponding to other points? Or should we sample the starting point from the range [1, N - d]?\n3.For each branch, if the termination token was generated, was the branch still generated until its length = d? \n4.How about the training complexity w.r.t. branch number and depth of generation?\n5.For robustness analysis, how to repeat the random selected characters? It is better to give an example. Is the noise sensitive to the position of selected characters?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.A GAN-style training schema is proposed based on benefit of parallel computation of Transformer, which generates multiple branches and results in dense learning signals, this improves the model generation ability without any new training data.\n2.Extensive human evaluation and analysis to demonstrate the effectiveness the method.", "weaknesses": "1.Both generator and discriminator need to be tuned. This results in expensive computation and memory consumption, especially for large LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)