SlowGuess/NAACL_2022_OCR huggingface dataset:
Text Length Control By editing the length of the RST encoding, we gain strong control of the length of the generated text. Here we fix the key phrases and vary the length of the RST context (i.e., number of EDUs) from 2 to 14 to demonstrate the increasing length of generated text. Results are depicted in Figure 5. We believe that our method provides a more natural way to control text length when compared to the heuristic methods of fine tuning text generation hyper-parameters such as repetition penalty, length penalty and minimum length.
RST Tree Edit Distance Here we extend the Tree Edit Distance to RST Trees to evaluate how well RSTGen is able generate text that adheres to the RST encoding passed to it. Specifically, we investigate how well this model performs as the
Figure 5: Text Length Control: RSTGen exhibits the ability to implicitly control the length of generated text by controlling the size of the RST structure provided as context. In this figure we present the average word count (left  $y$ -axis) and average sentence count (right  $y$ -axis) against the length of the RST encoding passed to RSTGen.
RST structure becomes increasingly long. For this experiment, we pass an RST Encoding and a set of keywords to our model and generate a text  $y$ . We then extract an RST tree from the generated text  $y$  using an RST parser. An edit includes the following operations: Changing/Removing the relation or nuclearity label of the node at position  $l$  with cost 1; Changing the position of a node to its sibling node with cost 1; and Deleting/Inserting a blank node at position  $l$  with cost 3. For a tree of size  $s$  we use a normalising factor of  $3 \times s$ , the cost of creating the tree. Our normalising factor does allow distances over 1. We observe from Figure 6 that generated text is able to adhere to correct RST structure in terms of RST node positions relatively well. However, when considering nuclearity and relation, he inaccuracy of RST structure grows significantly.
6 Evaluation on Argument Generation and Story Generation
We evaluate RSTGen for the tasks of argument generation and story generation. These tasks require our model to output coherent and cohesive text.
We describe the datasets used for fine-tuning our model, baseline models and the methods used to ensure fair comparison. Training detail and hyperparameter setting can be found in Appendix C.
Datasets We use two different datasets for argument generation and story generation, respectively.
Figure 6: Normalised RST Tree Edit Distance: This figure shows the similarity between the RST structure of RSTGen output text and the RST encoding passed as input to RSTGen. We extend Tree Edit Distance to RST Trees to calculate three metrics based on combinations of node position, node nuclearity label and node relation label: (1) Simple Structure: Tree distance including only node positions; (2) Complex Structure: Tree distance including node positions and nuclearity; (3) Complete Structure: Tree distance including node position, nuclearity and relation.
CMV Dataset We use the argument generation dataset first introduced in (Hua et al., 2021). This dataset contains pairs of claims and counter-arguments extracted from titles and responses, respectively, from the Reddit ChangeMyView (CMV) forum. This dataset uses posts dated in 2019, which prevents overlap with our training set for the RST Annotated Dataset. The goal of this task is to generate the counter-argument given the initial claim. Writing Prompts Dataset We use the Story generation dataset created by Fan et al. (2018a) and utilised in (Ji and Huang, 2021). This dataset contains pairs of prompts and stories extracted from the WritingPrompts subReddit. The prompt is an introduction to a story that must be extended. We limit each story to be up to 270 tokens.
Baselines For argument generation, we evaluate the following baselines:
CTRL. (Keskar et al., 2019) uses text prompts to control the style of output. The trigger r/changemyview style output we prepend the text 'Opinion Title: ' to the opening statement. This model is approximately 10 times larger than RSTGen and we assume that it achieves performance gains primarily through its larger size.
PAIR (Planning and Iterative Refinement). Hua and Wang (2020) devised a two-step generation process. The first step uses a fine-tuned BERT to
Model	Control Method	Grammar ↑	Redundancy ↑	Focus ↑	Structure and Coherence ↑
CTRL	Text prompt	0.84	-0.16	-0.12	-0.76
PAIR	Keywords + positions	0.79	0.00	-0.02	-0.78
DYPLOC	Categorised keywords	0.73	-0.10	-0.03	-0.79
RSTGen	RST + PAIR Planner	0.81	-0.03	-0.03	-0.77
RSTGen	RST + DYPLOC Planner	0.83	-0.02	-0.04	-0.75
RSTGen	RSTpredicted+ PAIR Planner	0.75	-0.05	-0.06	-0.88
RSTGen	RSTpredicted+ DYPLOC Planner	0.76	-0.06	-0.05	-0.86
RSTGen w/o R.A.A.	RST + PAIR Planner	0.71	-0.04	-0.08	-0.81
RSTGen w/o R.A.A.	RST + DYPLOC Planner	0.65	-0.06	-0.09	-0.80
Table 3: Argument Generation Evaluation: the performance of baseline models and our RST frameworks with different control methods. The shortform R.A.A refers to the RST-aware attention described in Section 4.4.
allocate a position to each pre-defined keyword, in order to create a template for the text to be generated. Then a fine-tuned BART model is used to fill the template.
DYPLOC (Dynamic Planning of Content using Mixed Language Models). Hua et al. (2021) proposed first categorising the keywords into concepts and entities. A BERT-based Planner is used to expand this set of concepts and entities, while another is used to generate the claim. The claim provides a brief summary of the extended answer. Four separate BART Encoders and a BART Decoder are used to convert the expanded set of concepts and entities into an argumentative text.
For both PAIR and DYPLOC, we use their gold standard plan as input to the model, this follows the headline results given in both papers. In pursuit of fair testing, we convert the gold standard plans of PAIR and DYPLOC to two RST encodings. More concretely, for PAIR, we convert its gold standard template, consisting of a set of phrases and their corresponding starting positions in the text to be generated, into a RST encoding. These phrases become the keyphrases for RSTGen. The RST node position for each key phrase can be determined by parsing the true RST structure of the text. For DYPLOC, first we collect the gold standard set of concepts/entities and claims, and filter out any repeated words phrases. Then, the RST node position for each word/phrase can be determined in a similar manner to PAIR.
For Writing Prompts, we compare our proposed approach with the following models:
DiscoDVT (Ji and Huang, 2021). To the best of our knowledge it is the only other work that uses RST to improve the long form text generation ability of a language model. It has three modules: an RST planner, an encoder and a decoder. These mod
ules are combined to form a VAE structure wherein the RST planner produces a sequence of discrete hidden vectors representing the sequence of RST relations between EDUs in their text. Then at each generation step, sequential hidden vectors are used to guide the text generation process. For fair comparison we do not pass keyphrases to RSTGen.
RSTGen Ablations. We evaluate ablations of RSTGen to validate the two significance features of our proposed RSTGen framework. Firstly, we remove the RST-Aware Attentions (R.A.A.), after which we remove RST Nuclearity, then Relation, and finally node positions.
Metrics We use two sets of metrics. For the Argument Generation experiments, we use the GRUEN (Zhu and Bhat, 2020) set of metrics, which measure four important features of text: Grammaticality, Non-redundancy, Focus and Structure/Coherence. A final metric combines these four features, and has been shown to correlate strongly with human reviewers. For the Story Generation experiments, we use BLEU (Papineni et al., 2002) focusing on the  $n$ -gram precision; Distinct- $n$  (Li et al., 2016), which takes the proportion of distinct  $n$ -grams relative to all generated  $n$ -grams, thus producing a non-reference-based measure for diversity; the GRUEN Structure and Coherence metric (G-SC); and MS-Jaccard (Alihosseini et al., 2019) which uses the Jaccard Index to measure how similar the distribution of  $n$ -grams is between two sets of text.
6.2 Evaluation Results
This section presents the performance of RSTGen against baseline models and discusses the performance of RSTGen ablations.
Argument Generation Table 3 shows that our proposed RSTGen using the DYPLOC Planner performs on par with CTRL in terms of Grammar despite using only one tenth parameters of CTRL. It outperforms all the baselines in Structure and Coherence.
Using automatically predicted RST trees as input, we observe degraded performance of RSTGen results. We believe that this can be attributed to our Markovian RST sampling methodology of sampling a child node dependent on its direct parent node. This may produce RST Trees with unrealistic structures. Nevertheless, it still outperforms CTRL in Redundancy and Focus, implying that the combination of RST relations, nuclearity and key phrases provide strong content guidance.
RSTGen without the RST-aware attention (R.A.A.) experiences a drop in performance across all metrics, especially Grammar and Focus. During the generation of longer sequences, using R.A.A. ensures that the same key phrase or RST information does not influence adjacent elementary discourse units, leading to more diversity and less repetition in generated text. We posit that the removal of R.A.A. exacerbates the reduced performance of RSTGen for longer texts, a trait that was exemplified in Figure 6.
Story Generation Table 4 shows that the R.A.A. variants of our model perform on par with the baseline. We observe that the removal of the R.A.A. causes a significant drop in performance, specifically Distinct-4, confirming our findings from the Argument Generation. As the RST relations carries information on the semantics of text, we observe that its removal has a significant effect on the similarity based metrics of BLEU-1, Distinct-4 and MS-Jaccard 3.
Model	BLEU-1 ↑	Distinct-4 ↑	G-S&C ↑	MS-Jaccard 3 ↑
DiscoDVT	24.10	84.66	-0.73	34.76
RSTGen (no keyphrase)	24.01	84.85	-0.71	34.51
- R.A.A.	23.20	82.12	-0.75	33.21
- RST relation	22.28	81.09	-0.77	33.79
- RST nuclearity	22.29	81.12	-0.77	33.78
- RST positions	22.04	80.91	-0.75	33.73
Table 4: Story Generation Evaluation: Results for the RSTGen and Baselines on the Writing Prompts story generation dataset. RSTGen performs competitively with DiscoDVT.
We present a novel controlled text generation framework, RSTGen, which uses fine-control over a Rhetorical Structure Theory based context as a means to improve the coherence and cohesion of generated text. We also leverage the structural information presented by RST to propose an rst aware attention scheme, ensuring that the model attends to the correct information during long form generation. Through investigation of RSTGen's open generation text, we showed that our approach can exhibit a high level of interpretable fine-control over syntactic, semantic and structural features of text.
We acknowledge that our proposed model may be susceptible to learning harmful biases present in the dataset. In and of itself this has the potential to harm minorities, marginalised communities and project stigmas present in society. Further, we recognise that our efforts to improve coherence, cohesion and control might be misused to author offensive or fictitious content. Therefore, we advocate for morally correct and responsible practices in the case of real-world application.
This work was funded by the UK Engineering and Physical Sciences Research Council (grant no. EP/T017112/1, EP/V048597/1). YH is supported by a Turing AI Fellowship funded by the UK Research and Innovation (grant no. EP/V020579/1).
Danial Alihosseini, Ehsan Montahaei, and Mahdieh Soleymani Baghshah. 2019. Jointly measuring diversity and quality in text generation models. Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation.
Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. 2019. Constrained decoding for neural NLG from compositional representations in task-oriented dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 831-844, Florence, Italy. Association for Computational Linguistics.
Lynn Carlson, Daniel Marcu, and Mary Ellen Okuowski. 2003. Building a Discourse-Tagged Corpus in the Framework of Rhetorical Structure Theory, pages 85-112. Springer Netherlands, Dordrecht.
Patricia L. Carrell. 1982. Cohesion is not coherence. TESOL Quarterly, 16(4):479-488.
Jonathan P. Chang, Caleb Chiam, Liye Fu, Andrew Z. Wang, Justine Zhang, and Cristian Danescu-Niculescu-Mizil. 2020. Convokit: A toolkit for the analysis of conversations.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. A multi-task approach for disentangling syntax and semantics in sentence representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2453-2464, Minneapolis, Minnesota. Association for Computational Linguistics.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019. Plug and play language models: A simple approach to controlled text generation.
Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376-380, Baltimore, Maryland, USA. Association for Computational Linguistics.
Wenyu Du, Zhouhan Lin, Yikang Shen, Timothy J. O'Donnell, Yoshua Bengio, and Yue Zhang. 2020. Exploiting syntactic structure for better language modeling: A syntactic distance approach. In ACL.
Ondrej Dusek and Filip Jurcicek. 2016. Sequenceto-sequence generation for spoken dialogue via deep syntax trees and strings. arXiv preprint arXiv:1606.05491.
William Falcon et al. 2019. Pytorch lightning. GitHub. Note: https://github.com/PyTorchLightning/pytorch-lightning, 3:6.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018a. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898, Melbourne, Australia. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018b. Hierarchical neural story generation. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
Angela Fan, Mike Lewis, and Yann Dauphin. 2019. Strategies for structuring story generation. arXiv preprint arXiv:1902.01109.
Vanessa Wei Feng and Graeme Hirst. 2012. Text-level discourse parsing with rich linguistic features. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 60-68, Jeju Island, Korea. Association for Computational Linguistics.
Tanya Goyal and Greg Durrett. 2020. Neural syntactic preordering for controlled paraphrase generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 238-252, Online. Association for Computational Linguistics.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR.
Xinyu Hua, Ashwin Sreevatsa, and Lu Wang. 2021. DYPLOC: Dynamic planning of content using mixed language models for text generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6408-6423, Online. Association for Computational Linguistics.
Xinyu Hua and Lu Wang. 2019. Sentence-level content planning and style specification for neural text generation. arXiv preprint arXiv:1909.00734.
Xinyu Hua and Lu Wang. 2020. PAIR: Planning and iterative refinement in pre-trained transformers for long text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 781-793, Online. Association for Computational Linguistics.
Haozhe Ji and Minlie Huang. 2021. Discodvt: Generating long text with discourse-aware discrete variational transformer.
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Yinghao Li, Rui Feng, Isaac Rehg, and Chao Zhang. 2020. Transformer-based neural text generation with syntactic guidance.
William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text - Interdisciplinary Journal for the Study of Discourse, 8(3):243-281.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2267-2277, Minneapolis, Minnesota. Association for Computational Linguistics.
Xuan-Phi Nguyen, Shafiq Joty, Steven C. H. Hoi, and Richard Socher. 2020. Tree-structured attention with hierarchical accumulation.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners - gpt2.
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron C. Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. ArXiv, abs/1810.09536.
Maite Taboada and William C. Mann. 2006. Applications of rhetorical structure theory. Discourse Studies, 8(4):567-588.
Stephen Waller. 2015. Cohesion is still not coherence, so what is? English Teaching In China, (6):31-35.
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. 2019a. Topic-guided variational autoencoders for text generation. arXiv preprint arXiv:1903.07137.
Yaushian Wang, Hung-Yi Lee, and Yun-Nung Chen. 2019b. Tree transformer: Integrating tree structures into self-attention. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1061-1070, Hong Kong, China. Association for Computational Linguistics.
Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018. Toward fast and accurate neural discourse segmentation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 962-967, Brussels, Belgium. Association for Computational Linguistics.
Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, PeiHao Su, David Vandyke, and Steve Young. 2015. Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1711-1721, Lisbon, Portugal. Association for Computational Linguistics.
Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2253-2263, Copenhagen, Denmark. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019. Huggingface's transformers: State-of-the-art natural language processing.
Lili Yao, Nanyun Peng, Weischedel Ralph, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019. Plan-and-write: Towards better automatic storytelling. In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19).
Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019. Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models. arXiv preprint arXiv:1902.08858.
Wanzheng Zhu and Suma Bhat. 2020. GRUEN for evaluating linguistic quality of generated text. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 94-108, Online. Association for Computational Linguistics.
A RST Schemas
We list in Table A1 the RST relations and nucleus that our proposed RSTGen framework utilises. We also provide a schema in Table A2 explaining the meaning of Nuclearity labels used by RSTGen.
B Argument and Story Generation Dataset Details
For both the argument generation and story generation experiments our datasets originate from the respective paper's of models in the baseline study. We do not make any alterations to the datasets, but instead simply use it as provided. We refer the reader to and Hua et al. (2021) for the argument generation dataset (i.e., the CMV dataset) and Fan et al. (2018b) for further details regarding the story generation dataset (i.e., the Writing Prompt dataset).
Code The code used to train and evaluate our models can be downloaded from https://github.com/Rilwan-A/RSTGen. As well as code, the github will contain links to the RST annotated versions of the CMV dataset and Writing Prompts dataset. Access to the full RST Annotated Reddit dataset can be gained upon request.
Repositories The RSTGen models were extended from pretrained models in Huggingface's Transformers repository (Wolf et al., 2019). RSTGen is initialised using the GPT2-base model with approximately 124M parameters. The neural RST Predictor was initialised using the BART-base model. We used Pytorch-Lightning (Falcon et al., 2019) for all our training scripts.
Hardware For fine-tuning the RSTGen model on the RST Annotated Datasetset, we used 2 GeForce RTX 3090 (24GB). For the Argument Generation Tasks and the Story Generation Task, the RSTGen variants and RST Neural Sampler are fine-tuned using 1 GeForce RTX 3090 (24GB). All training was done using mixed Precision (FP16) to improve memory efficiency.
Fine-tuning For fine-tuning all variations of RSTGen, we used the Adafactor optimiser with the following parameter settings: scale parameter  $=$  False, relative step  $=$  True, warmup init  $=$  True, learning rate  $=$  None, weight decay  $= 0.01$ . Due to the high computational expense required, we did
not perform extensive hyper-parameter tuning for our RSTGen models.