SlowGuess/NAACL_2022_OCR huggingface dataset:
RSTGen: Imbuing Fine-Grained Interpretable Control into Long-Form Text Generators
Rilwan A. Adewoyin $^{1,4}$ , Ritabrata Dutta $^{2}$ , Yulan He $^{1,3}$
$^{2}$ Department of Statistics, University of Warwick, UK
3The Alan Turing Institute, UK
$^{4}$ Department of Computer Science and Engineering,
Southern University of Science and Technology, China
{rilwan.adewoyin,ritabrata.dutta,yulan.he}@warwick.ac.uk
In this paper, we study the task of improving the cohesion and coherence of long-form text generated by language models. To this end, we propose RSTGen, a framework that utilises Rhetorical Structure Theory (RST), a classical language theory, to control the discourse structure, semantics and topics of generated text. Firstly, we demonstrate our model's ability to control structural discourse and semantic features of generated text in open generation evaluation. Then we experiment on the two challenging long-form text tasks of argument generation and story generation. Evaluation using automated metrics and a metric with high correlation to human evaluation, shows that our model performs competitively against existing models, while offering significantly more controls over generated text than alternative methods.
Controllable text generation has attracted much attention in recent years. Generating coherent and cohesive long-form text with controllable attributes is particularly challenging due to the relatively complex syntactic and semantic structures involved compared to short-text generation. Long-form text generation can find applications in automatic argumentation, motivational speech, and opinionated writing, to name a few.
A popular approach to controllable text generation is to design prompts, which control high-level linguistic features (i.e., text style and sentiment) by prefixing simple context to the input of a language model. These non-granular methods are often too coarse to allow different parts of the text to have diverse or contrasting features. Furthermore, they tend to focus on a single linguistic feature of text, meaning that extra frameworks such as PlugandPlay (Dathathri et al., 2019) are required to control multiple linguistic features.
To achieve improved content control, researchers have coupled a content planning module with a text generation module (Hua and Wang, 2020; Hua et al., 2021), in which the content planning module firstly generates a set of keyphrases and their corresponding positions in sentences and the generation module fills in the gaps. To control syntactic structures of text, recent works have proposed a neuro-symbolic approach leveraging Dependency Structure Theory (DST) (Shen et al., 2019; Du et al., 2020). Enforcing specific structures over the generated text has been proven to be useful in increasing coherence and cohesion of short-form text. However, a relatively large amount of DST information is required to encode comparatively short texts. Thus, they are unsuitable for use in long-form text generation tasks.
Similar to DST, Rhetorical Structure Theory (RST) (Mann and Thompson, 1988) is a classical tree-based interpretation of natural language. Whereas DST is most useful for intra-sentence interpretation, RST is most useful for inter-sentence interpretation of language. We propose a neurosymbolic framework that can imbue an RST understanding of text to existing language models. More specifically, our framework (1) allows more fine-grained control of syntax, semantics and text structure; (2) utilises a well-defined rhetorical structure of language, thus offering better interpretability; (3) can be directly integrated into existing pre-trained language models such as GPT-2 (Radford et al., 2019) and BART (Lewis et al., 2019).
We evaluate our proposed framework on two tasks: argument generation and story generation. We show that our proposed framework improves upon existing content control approaches on automatic evaluation metrics including BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014) and generates better text in terms of grammar and coherence measure. We further demonstrate our model's ability to generate text
Human Language Technologies, pages 1822 - 1835
with control over syntactic, semantic and discourse features. Our main contributions can be summarised below:
- We propose a novel framework to imbue the RST information into pre-trained language models.
- We develop the first neurosymbolic framework that provides interpretable fine-control over syntax, semantics, discourse structure, topical keywords and keyword positions.
- We demonstrate the superiority of our proposed framework over existing planning and control methods for two long-form text generation tasks.
We discuss several lines of research in controllable text generation.
Prompts For prompt-based control, a context is presupended to a language model input. The presupended item is related to the type of desired output. This method has been used to manipulate the syntax (Dušek and Jurčićek, 2016; Goyal and Durrett, 2020) and semantics (Wen et al., 2015; Chen et al., 2019) of the output. Alternatively, the presupended item can provide semantic control in order to cover a given topic (Wang et al., 2019a), mention specified entities (Fan et al., 2019), display a certain attribute (Hu et al., 2017; Balakrishnan et al., 2019), or even exhibit a style of text (Keskar et al., 2019). These methods suffer from the inability to exert fine-control, that is, a change in any one of the input prompts will change the whole generated text. Furthermore, all these works utilise non-expansive features for their prompts, which prevents them from making iterative improvements to existing generated text.
Content Planning Despite the impressive progress made in many generation tasks, earlier neural models are known to produce low-quality content (Wiseman et al., 2017), often with low relevance, and poor discourse structure (Zhao et al., 2019). Content-based planning approaches were added into neural systems to enhance content relevance (Moryossef et al., 2019; Yao et al., 2019; Hua and Wang, 2019). Hua et al. (2021) extended previous content planning approaches by dividing the content plan into different types
of information; 'Entities', 'Claim' and 'Concepts'. An alternative planning approach focuses on where the key phrases should be placed in the generated text in order to improve coherence (Hua and Wang, 2020). The text generator is then conditioned on the provided key phrases and their respective positions in text. Similar to our approach, these methods allow users varying levels of custom control by manual augmentation of the planning module output.
Syntactic or Discourse Control Various works utilised syntactic parse trees with the transformer structure to gain syntactic control or improve interpretability on short-form text generation tasks (Li et al., 2020; Nguyen et al., 2020). With a focus on improving long-form text generation, Ji and Huang (2021) used a Variational Autoencoder (VAE) structure to model RST discourse relations between successive elementary discourse units (spans of text).
We build upon their approach by using a more expressive binary tree formalisation of RST. This formalisation extends the modelling of sequential elementary discourse units by also modelling nuclearity and relationship between non sequential discourse units.
Extending on previous works, Wang et al. (2019b) attempted to couple tree structures and transformers. We instead embed the tree structure of RST into transformers through constrained attention in a separate RST embedding dimension.
3 Rhetorical Structure Theory
Rhetorical Structure Theory (RST) provides a formal structure for interpreting language based on relations between parts of text. Each structure is defined by three sets of features: the binary tree structuring of Discourse Units (EDU)s, the nuclearity between sibling DUs, and the relationship between sibling DUs. An example RST tree using the schema provided by (Mann and Thompson, 1988) as shown in Figure 1.
Binary Tree Structuring of Discourse Units To form a binary discourse tree  $R$  from text  $t$ , the text must be successively divided into smaller subsets of text. In this manner, node 0 represents the full text, with subsequent nodes representing subtexts, as in Figure 1. The text at each node is called a Discourse Unit, while the nodes at the leaves of the tree are Elementary Discourse Units (EDUs).
Figure 1: RST example: An example text (top) interpreted through RST schema. The text is broken down into a binary tree with zero-indexed numbered node positions. Relations exist between the two sibling nodes of a parent node. Each pair of sibling nodes, have a coupled nuclearity label. Nodes with no children, are Elementary Discourse Units (EDUs).
Sibling Nodes - Relations: A parent node is a non-terminal node which has two child nodes, referred to as siblings. Each parent nodes' children, has a RST relation  $r_i, i \in \mathbb{N}_{<20}$  describing the syntactic purpose of each child node relative to the other. In Figure 1, the RST relations are presented in blue texts.
Sibling Nodes - Nuclearity: Associated with each pair of sibling node's relation is a joint nuclearity labelling,  $n_j, j \in \mathbb{N}_{< 5}$ . This nuclearity explains the role of each sibling node's discourse unit relative to the other sibling node. Each joint nuclearity labelling between pair of sibling nodes must include at least one Nuclei.2
Formally, we define a binary RST Tree  $R$  as a collection of its parent nodes  $\{\pmb{v}_l^{r,n}\}$ , where  $\pmb{v}_l^{r,n}$  is a parent node at position  $l$ , with a RST relation  $r$  and a Nuclearity  $n$  between its two children. For example,  $v_2^{\text{Conditional},SN}$  represents node 2 in Figure 1. We refer to the group of relations in an RST Tree,  $R$ , as  $R^r$ . Similarly, the group of Nuclearity values is referred to as  $R^n$ .
4 RSTGen: RST-Dependent Long-form Text Generator
We propose a long-form text generation approach, call RSTGen, which provides control over semantics and structure of text, with the aim of improving coherence and cohesion of the generated text. Coherence is concerned with how well constituent
1Table A1 in the Appendix details the 19 possible RST Relations.
2Table A2 in the Appendix details the 4 possible RST Nuclearity values.
Figure 2: RSTGen Structure: Our RST controlled text generation framework can be built upon any pretrained language model. We input RST tree structure information as  $R$ , comprising the Nuclearities  $R^n$ , RST Relations  $R^r$  and RST node positions  $R^l$ , and key phrase information at EDUs as  $K$ , comprising the key phrases  $K^w$  and the RST node positions of the key phrases,  $K^l$ . The discourse structure of the output text  $y$  is controlled by  $R^n$  and  $R^l$ . The syntax of  $y$  is controlled by  $R^n$ ,  $R^l$  and  $K^l$ . The semantics is controlled by  $R^r$  and  $K^w$ . Our framework creates a prefix-embedding from the encodings for  $R$  and  $K$ , after which text is generated as a continuation.
sentences/chunks follow on from the previous sentence/chunk. This can be interpreted as the topic of each subsequent sentence/chunk being relevant to the previous sentence/chunk (Carrell, 1982; Waller, 2015). Cohesion describes the way in which text is tied together by linguistic devices such as Therefore..., However..., In addition to (Waller, 2015). This can be interpreted as how smooth the model transitions between the types of sentences. With an RST interpretation, cohesion is loosely related to the RST relations between different nodes in the binary tree, while coherence is related to the RST relations and key phrases at EDUs.
Task formulation Our input consists of (1) an encoding for a binary RST Tree  $R$ , (2) a set of keyphrases information  $K$  that are relevant to the text. The RST encoding is formed as sets of three pieces of information for each parent node in the binary RST Tree. These include Nuclearity  $R^n$ , RST Relation  $R^r$  and RST node position  $R^l$ . The keyphrase information contains two pieces of information. These are  $K^w$  and  $K^l$ , the key phrases and the RST node positions of the key phrases. The model has been trained to work with varying levels of specification for the context information.
Partial Context Provision In practice, we do not expect users to provide the full RST Tree or key phrases since often only a subset of features of the context information may be known
or needed. For example, in Figure 2, the goal is to produce a text contrasting the innovative ability of Microsoft and Tesla. A user only needs to provide partial information, e.g., the keywords,  $K^{w} = [BMW, Tesla, innovation]$  and the RST relation,  $R^{r} \supset ['Contrast']$ . Our model will be able to automatically predict an RST tree following a conditional sampling method we have designed in Section 4.3.
While a user will often want to specify the key phrases themselves, we could also employ automated methods to create an expanded set. For example, existing Planning Modules (Hua and Wang, 2020; Hua et al., 2021) can be used to generate a set of keywords based on some prior information.
In what follows, we describe our proposed RST-Gen in more details.
4.1 Tokenisation
While our RST controlled text generation framework can be built upon any pre-trained language model, we use GPT as an example here. The GPT Tokeniser is used to tokenise the target text  $y$  and key phrases  $K^w$  to a set of word tokens. Two new tokens are added to the tokeniser: '  $<\text{rst}>$ ' and '  $<\text{kp}>$ '. The former is prepended to the RST tree  $R$ , and the latter is prepended to each key phrase.
4.2 Embedding Layers
Our framework requires three additional embedding layers to facilitate the encoding of the RST position, RST nuclearity and RST relation information presented in  $R^l$ ,  $R^n$  and  $R^r$ . These embedding layers are designed to produce vectors that match the size  $v$  of hidden vectors in the base model.
We use a RST relation Embedding Matrix  $W_{r} \in \mathbb{R}^{19 \times 768}$  to encode  $R^r$  as there are 18 possible RST relations and a pad token, and an RST Nuclearity Embedding matrix  $W_{n} \in \mathbb{R}^{4 \times 768}$  to encode  $R^n$ , which consists of 3 possible nuclearity labels and a pad token.
To embed RST node position encodings  $R^l$ , we create a novel embedding layer designed to capture the positional relationships of nodes in a binary tree, in a space efficient manner. The intent is to explicitly capture the relationship between a node and its ancestor nodes. Our embedding layer features a non-trainable embedding matrix  $W_{pe} \in \mathbb{R}^{\max_{\text{rst\_node}} \times \text{tree\_depth}}$ , a trainable feed forward layer  $W_{pff} \in \mathbb{R}^{\text{tree\_depth} \times 768}$  and a Gelu activation layer  $f_g(\cdot)$ . In our experiments
max rst_node = 4094 and max_tree_depth = 12.
The  $i$ -th column in the non-trainable embedding matrix  $W_{pe}$  is a vector representing the position of node  $i$ , in terms of a sequence of Lefts (L) and Rights (R) required to travel from the root node 0 to node  $i$ . Left is encoded as  $-0.05$  and Right as 0.05. For example, the vector at node position 5 in  $W_{pe}$  in the example RST tree shown in Figure 1 is encoded as  $[0.05, -0.05, 0, 0,\dots]$  with the remaining 0s representing padding values up to the length max_tree_depth.
It is important to note that in our RST Tree encoding  $R$ , a parent node  $\pmb{v}_i$  is labelled with the relation  $r_i$  and nuclearity  $n_i$  connecting its children. As such our encodings for a  $R^r, R^l$  and  $R^n$  do not include the leaf nodes with no children, but still represent  $R$ . This allows our encodings, for a Binary RST Tree with  $N$  nodes to be sequences of length  $\lceil N/2 \rceil$ .
4.3 RST Predictor
We use an RST Predictor to predict the relation and nuclearity of a child node conditional on the RST relation, nuclearity and position of their parent node. For example we predict the left child node  $\pmb{v}_{2l + 1}^{r_i,n_j}$ , by modelling the following conditional distribution  $\pmb{v}_{2l + 1}^{r_i,n_j} \sim p(r,n|l_{\mathrm{parent}},r_{\mathrm{parent}},n_{\mathrm{parent}},\mathbb{1}(\pmb{v}_{2l + 1} = \mathrm{left~child})$ . A full RST Tree can be predicted by iteratively repeating this one-step sampling.
We propose a neural sampler method to estimate the conditional distribution  $p(\cdot|\cdot)$ . It is trained on the RST Trees observed in the training set of the RST Annotated Dataset introduced in Section 6. Our neural RST Sampler, depicted in Figure 3, uses the BART (Lewis et al., 2019) model. The encoder takes a prompt as input. The decoder is reinitialised and reduced to two layers. The decoder takes four inputs: (1) The parent nodes' relation  $r$ ; (2) The parent node's nuclearity  $n$ ; (3) The parent node's RST position  $l$ ; (4) A vector  $b$  indicating whether the target node is a left child node or right child node. This vector  $b$  is calculated as the difference between the node position embeddings for the parent node and the target child node encodings.
We use the Embedding Layers structures from our RSTGen model to encode the inputs (1-4) to the Encoder. The BART decoder contains two classification heads, to predict relation and nuclearity.
Two approaches can be used to guide our RST
Figure 3: Neural RST Sampler: Our Neural RST Sampler can be used to iteratively predict an RST Tree, conditioned on a prompt which is passed to the Encoder. The Decoder takes as input the summation of embeddings for the parent' nodes relation  $r$ , nuclearity  $n$  and position  $l$ . Further we also pass in  $b$  which represents whether the target child node is a left or right child.
predictor to produce more relevant RST Trees. Firstly, the training set can be limited to a specific domain, such as argumentative text, to introduce bias the prediction towards a specific linguistic style. Secondly, during inference the predictive distribution can be augmented, to ensure specific structural properties are achieved. In our experiments, we use this to predict RST Trees with a specific size. We have extended this to other structural features such as specifying RST relation frequency, and position.
4.4 RST-Aware Attention (R.A.A.)
To better encode the RST tree structure in language model training and to improve the coherence of text by reducing the occurrence of hallucination in long-form generated text, we propose a novel attention scheme, called RST-aware attention. In particular, to generate a word token at position  $i$ , we create dynamic attention masks  $m_{i}$  allowing the hidden state  $h_i$  to focus only on structurally relevant hidden representations  $h_j$ ,  $j \neq i$ . The hidden representation  $h_j$  is structurally relevant to  $h_i$  if  $h_j$ 's associated RST node is an ancestor of the RST node associated to  $h_i$ . The RST-aware attention is described in Algorithm 1.
In the above process, we need to first detect the RST node position of the word token to be generated. We do this in a sequential manner with the help of an EDU splitter (Wang et al., 2018), which can detect EDU boundaries in text. At the start of text generation, we set the initial EDU node position as the leftmost leaf node in the RST tree and
then proceed to generate the first token. Afterwards, we use the EDU splitter to detect whether an EDU boundary has occurred. If no boundary is detected, we continue with the generation of the next word token; otherwise, we infer the next EDU node position as the second leftmost leaf node in the RST tree. The above process is repeated until the 'end of sequence' token is generated or until the final child EDU node is generated. In practice we use heuristics to avoid the need to perform an EDU boundary detection at each generation step.
5 Open Generation Evaluation
We first train a model using our RST annotated dataset described below. Then we analyse the model's ability to control semantic, syntactic and text structure.
Subreddit	% of Training Instances
r/CasualConversation	41.7
r/changemyview	19.1
r/DebateReligion	15.8
r/PoliticalDiscussion	9.62
r/relationship_advice	7.94
Other Subreddits	6.84
Total	965,411 samples
Table 1: Statistics of RST Annotated Dataset: The majority of our dataset is sourced from 5 subreddits which exhibit a tendency to produce longer and more complex texts.
RST Annotated Dataset We use the ConvoKit API (Chang et al., 2020) to collect a total of over 965k text examples from Reddit to use as the training set. Table 1 provides a detail regarding the division of the dataset between different subreddits. The following subreddits comprise the majority of our dataset, DebateReligion, RelationshipAdvice, Politics and ChangeMyView. These subreddits contain many long texts with opinions or persuasive
Input
Key phrases:	“good mouse alternative” and “none”
Text prompt:	“It would be”
RST structure:	{Relation: r1, Nuclearity: SN, Node position: 0}, {Relation: r2, Nuclearity: NS, Node position: 2}
Input Relations (r1, r2)	Generated Text
Attribution, Explanation	It would be a good mouse alternative, because none of us are going to be able to see.
Background, Evaluation	It would be a good mouse alternative, since none of us have ever seen one.
Background, Condition	It would be a good mouse alternative if none of them were good.
Contrast	It would be a good mouse alternative, but none of them are as good as a good one.
Table 2: Short Text Semantic control: Examples of text generated by varying the RST relations. Words highlighted in red in the generated text indicate the corresponding underlined RST input relations.
style of writing. Further, we choose large samples from the subreddit CasualConversation to complement the limited range of language present in the former subreddits. Each sample contains a reddit post, its RST tree generated using the best statistical RST parser (Feng and Hirst, 2012) and keyphrases extracted from the post using the PageRank inspired algorithm, TextRank (Mihalcea and Tarau, 2004). The keyphrase extraction process is detailed in Appendix D.
Semantic Control In Table 2 we show the generated short text conditional on various RST relations. For all four examples the same key phrases of "a good mouse alternative" and "none of this" were passed to the model. Furthermore, we provide the first three words "It would be" to the decoder as a text prompt. The text is generated by varying the RST relations,  $r_1$  and  $r_2$ . We can observe that the generated text varies depending on the desired RST relations input to the model.
Structural Discourse Control Here we use a reconstruction style test to evaluate the ability of our model to include the target relations in the generated text at the correct position. To do this, we allow our model to generate text  $\hat{y}$  using the RST tree  $T$  as the conditioning factor. We then use an RST parser to generate a RST tree  $\hat{T}$  from the generated text  $\hat{y}$ . For each relation  $r$ , we calculate a recall score as the proportion of nodes with relation  $r$  in  $T$  that also occur in  $\hat{T}$  at the same position  $l$ . The results are shown in Figure 4. We include results for different lengths of text, measured by elementary discourse units.
We observe that our model achieves strong levels of control for the following relations: Attribution, Background, Condition, Contrast, Elaboration, Joint, Manner-Means, and Temporal. We believe that the weakened performance on Cause
Figure 4: Structural Discourse Control: RST relation and position recall scores calculated for each RST relation from the generated text with its lengths measured by the number of Elementary Discourse Units in the range of  $\{4,8,12,16,20\}$ .
and Comparison is due to their respective similarity to Attribution and Contrast. We omit topic change since our datasets contains texts mostly constrained to a single topic.