StoryTrans: Non-Parallel Story Author-Style Transfer with Discourse Representations and Content Enhancing
Year: 2023

Abstract:
Non-parallel text style transfer is an important task in natural language generation. However, previous studies concentrate on the token or sentence level, such as sentence sentiment and formality transfer, but neglect long style transfer at the discourse level. Long texts usually involve more complicated author linguistic preferences such as discourse structures than sentences. In this paper, we formulate the task of non-parallel story author-style transfer, which requires transferring an input story into a specified author style while maintaining source semantics. To tackle this problem, we propose a generation model, named StoryTrans, which leverages discourse representations to capture source content information and transfer them to target styles with learnable style embeddings. We use an additional training objective to disentangle stylistic features from the learned discourse representation to prevent the model from degenerating to an auto-encoder. Moreover, to enhance content preservation, we design a mask-and-fill framework to explicitly fuse style-specific keywords of source texts into generation. Furthermore, we constructed new datasets for this task in Chinese and English, respectively. Extensive experiments show that our model outperforms strong baselines in overall performance of style transfer and content preservation.

heading: 1 Introduction
text: Text style transfer aims to endow a text with a different style while keeping its main semantic content unaltered. It has a wide range of applications, such as formality transfer (Jain et al., 2019), sentiment transfer (Shen et al., 2017) and author-style imitation (Tikhonov and Yamshchikov, 2018).
Due to the lack of parallel corpora, recent works mainly focus on unsupervised transfer by selfreconstruction. Current methods proposed to dis-
∗Equal contribution. †Corresponding author
entangle styles from contents by removing stylistic tokens from inputs explicitly (Huang et al., 2021) or reducing stylistic features from token-level hidden representations of inputs implicitly (Lee et al., 2021). This line of work has impressive performance on single-sentence sentiment and formality transfer. However, it is yet not investigated to transfer author styles of long texts such as stories, manifesting in the author’s linguistic choices at the lexical, syntactic, and discourse levels.
In this paper, we present the first study on story author-style transfer, which aims to rewrite a story incorporating source content and the target author style. The first challenge of this task lies in imitation of author’s linguistic choices at the discourse level, such as narrative techniques (e.g., brief or detailed writing). As exemplified in Table 1, the generation text for the JinYong (JY)1 style not only rewrites some tokens to the martial arts style (e.g., “白云” /“white cloud” to “白光一闪” /“light flashing”) but also adds additional events in detail and
1JinYong is a Chinese martial arts novelist.
14803
enrich the storyline (e.g., the red highlights). In contrast to the transfer of token-level features like formality, it is more difficult to capture the intersentence relations correlated with author styles and disentangle them from contents. The second challenge is that the author styles tend to be highly associated with specific writing topics. Therefore, it is hard to transfer these style-specific contents to another style. For example, the topic “talented man” hardly shows up in the novels of JY, leading to the low content preservation of such contents, as shown in the orange text in Table 1.
To alleviate the above issues, we propose a generation framework, named StoryTrans, which learns discourse representations from source texts and then combines these representations with learnable style embeddings to generate texts of target styles. Furthermore, we propose a new training objective to reduce stylistic features from the discourse representations, which aims to pull the representations derived from different texts close in the latent space. To enhance content preservation, we separate the generation process into two stages, which first transfers the source text with the style-specific content keywords masked and then generates the whole text by imposing these keywords explicitly.
To support the evaluation of the proposed task, we collect new datasets in Chinese and English based on existing story corpora.2 We conduct extensive experiments to transfer fairy tales (in Chinese) or everyday stories (in English) to typical author styles, respectively. Automatic evaluation results show that our model achieves a better overall performance in style control and content preservation than strong baselines. The manual evaluation also confirms the efficacy of our model. We summarize the key contributions of this work as follows: I. To the best of our knowledge, we present the first study on story author style transfer. We construct new Chinese and English datasets for this task. II. We propose a new generation model named StoryTrans to tackle the new task, which implements content-style disentanglement and stylization based on discourse representations, then enhances content preservation by explicitly incorporating stylespecific keywords. III. Extensive experiments show that our model outperforms baselines in the overall performance of style transfer accuracy and content preservation.
2The codes and data are available at https://github. com/Xuekai-Zhu/storytrans_public

heading: 2 Related Work

heading: 2.1 Style Transfer
text: Recent studies concentrated mainly on token-level style transfer of single sentences, such as formality or sentiment transfer. We categorize these studies into three following paradigms.
The first paradigm built a style transfer system without explicit disentanglement of style and content. This line of work used additional style signals or a multi-generator structure to control the style. Dai et al. (2019) added an extra style embedding in input for manipulating the style of texts. Yi et al. (2020) proposed a style instance encoding method for learning more discriminative and expressive style embeddings. The learnable style embedding is a flexible yet effective approach to providing style signals. Such a design helps better preserve source content. Syed et al. (2020) randomly dropped the input words, then reconstructed input for each author separately, which obtained multiple author-specific generators. The multi-generator structure is effective but also resource-consuming. However, this paradigm incurs unsatisfactory style transfer accuracy without explicit disentanglement.
The second paradigm disentangled the content and style explicitly in latent space, then combined the target style signal. Zhu et al. (2021) diluted sentence-level information in style representations. John et al. (2019) incorporated style prediction and adversarial objectives for disentangling. Lee et al. (2021) removed style information of each token with reverse attention score (Bahdanau et al., 2015) , which is estimated by a pre-trained style classifier. This paradigm utilizes adversarial loss functions or a pre-trained estimator for disentanglement. And experiment results indicate that explicit disentanglement leads to satisfactory style transfer accuracy but poor content preservation.
The final paradigm views style as localized features of tokens in a sentence, which locates styledependent words and replaces the target-style ones. Xu et al. (2018) employed an attention mechanism to identify style tokens and filter out such tokens. Wu et al. (2019) utilized a two-stage framework to mask all sentimental tokens and then infill them. Huang et al. (2021) aligned words of input and reference to achieve token-level transfer. To sum up, this paradigm maintains all word-level information, but it is hard to apply to the scenarios where styles are expressed beyond token level, e.g., author style.
Absorbing ideas from paradigm 1 and 2, we
apply explicit disentanglement by pulling close discourse representations, which is formulated into disentanglement loss. Furthermore, we design a fusion module to stylize the discourse representation.

heading: 2.2 High-Level Representation
text: Prior works captured the hierarchical structure of natural language texts by learning high-level representations. Li et al. (2015) and Zhang et al. (2019) proposed to learn hierarchical embedding representations by reconstructing masked version of sentences or paragraphs. Reimers and Gurevych (2019) derived semantical sentence embeddings by fine-tuning BERT (Devlin et al., 2019) on downstream tasks. Lee et al. (2020); Guan et al. (2021b) inserted special tokens for each sentence and devised several pre-training tasks to learn sentencelevel representations. We are inspired to use a sentence order prediction task to learn high-level discourse representations.

heading: 2.3 Long Text Generation
text: In order to generate coherent long texts, recent studies usually decomposed generation into multiple stages. Fan et al. (2018); Yao et al. (2019) generated a premise, then transformed it into a passage. Tan et al. (2021) first produced domain-specific content keywords and then progressively refines them into complete passages. Borrowing these ideas , we adopted a mask-and-fill framework to enhance content preservation in text style transfer.

heading: 3 Methodology

heading: 3.1 Task Definition and Model Overview
text: We formulate the story author-style transfer task as follows: assuming that S is the set of all author-styles, given a multi-sentence input x = (x1, x2, · · · , xT ) of T tokens and its author-style label s ∈ S, the model should generate a multisentence text with a specified author-style ŝ ∈ S while keeping the main semantics of x.
As illustrated in Figure 1, we split the generation process into two stages. We first identify stylespecific keywords k = (k1, k2, · · · , kl) from x, and then mask them with special tokens ⟨mask⟩. We denote the resulting masked version of x as xm = (xm1 , x m 2 , · · · , xmT ). In the first generation stage, we perform discourse representation transfer on xm. In the second stage, we complete the masked tokens in the output of the first stage conditioned on k in a style-unrelated manner.
Due to the lack of parallel data, typical style transfer models tend to optimize the selfreconstruction loss with the same inputs and outputs (Xiao et al., 2021; Lee et al., 2021). Obviously, training with only the self-reconstruction loss will make the model easily ignore the target style signals and simply repeat the source inputs. Therefore, in the first stage, we devise an additional training objective, to disentangle stylistic features from intermediate discourse representations {ri}ni=1, where n is the number of sentences. Then, we fused these style-independent discourse representations with the target style ŝ as a discourse-
level guidance for the subsequent generation of the transferred text. As for discourse representations learning, we employ a sentence order prediction loss to capture inter-sentence discourse dependencies. And we use a style classifier loss to control the style of generated texts (Lee et al., 2021). In summary, the fist-stage model is trained using the following loss function:
L1 = Lself + λ1Ldis + λ2Lsop + λ3Lstyle, (1)
where λ1, λ2 and λ3 are adjustable hyperparameters. Lself , Ldis, Lsop and Lstyle are the self-reconstruction loss, the disentanglement loss, the sequence order prediction loss and the style classifier loss, respectively. Figure 2 shows the workflow of learning objects.
In the second stage, we use a denoising autoencoder (DAE) loss to train another encoderdecoder model for reconstructing x:
L2 = − T∑
t=1
logP (xt|x<t, {ki}li=1,xm). (2)
This stage is unrelated to author styles, and helps achieve better content preservation.

heading: 3.2 Discourse Representations Transfer
text: As described in Figure 2, we propose to learn discourse representations, and then reconstruct the texts from discourse representations. And we perform the disentanglement and stylizing operation based on discourse representations.
Discourse Representations Supposing that xm consists of n sentences, we insert a special token ⟨Sen⟩ at the end of each sentence in xm (Reimers and Gurevych, 2019; Lee et al., 2020; Guan et al., 2021b). Let rn denote the hidden state of the encoder at the position of the n-th special token, {ri}ni=1 = Encoder(xm). And zn is the output of
the fusion module corresponding to rn. Previous studies have demonstrated that correcting the order of shuffled sentences is a simple but effective way to learn meaningful discourse representations (Lee et al., 2020). As shown in Figure 1, we feed zn into a pointer network (Gong et al., 2016) to predict orders. During training, we shuffled the original sentence order and feed the perturbed text into the encoder for calculating Lsop.
Fusion Module To provide signals of transfer direction, we concatenate the learned discourse representations {ri}ni=1 with the style embedding s and fuse them using a multi-head attention layer, as illustrated in Figure 1. To capture discourselevel features of texts with different author-styles, we set each style embedding to a vector with the same dimension as ri. Formally, we derive the style-aware discourse representations {zi}n+1i=1 as follows:
{zi}n+1i=1 = MHA(Q = K = V = {s ∥ {ri}ni=1}), (3)
where MHA is the multi-head attention layer, Q/K/V is the corresponding query/key/value, ∥ is the concatenation operation. Then, the decoder gets access to {zi}n+1i=1 through the cross-attention layer, which serve as a discourse-level guidance for generating the transferred texts. Then, we feed {zi}n+1i=1 into the decoder.
Pointer Network Following Logeswaran et al. (2018); Lee et al. (2020), we use a pointer network to predict the original orders of the shuffled sentences. The each position probability of sentence order is formulated as follows:
pi = softmax({zi}ni=1WzTi ), (4)
where pi is predicted probabilities of sentence i, W is a trainable parameter.

heading: 3.3 First-Stage Training Objectives
text: Self-Reconstruction Loss We formulate selfreconstruction loss as follows:
Lself = − T∑
t=1
logP (xmt |xm<t, {ri}ni=1, s), (5)
where s is the learnable embedding of s. During inference, we replace s with the embedding of the target style ŝ (i.e., ŝ), to achieve the style transfer.
Disentanglement Loss We disentangle the style and content on discourse representations. Inspired by prior studies on structuring latent spaces (Gao et al., 2019; Zhu et al., 2021), we devise an additional loss function Ldis to pull close discourse representations from different examples in the same mini-batch, corresponding to different author styles. Ldis and Lself work as adversarial losses and lead the model to achieve a balance between content preservation and style transfer. We derive Ldis as follows:
Ldis = 1
2b
b∑
i=1
b∑
j=1
∥ r̄i − r̄j ∥22, (6)
r̄ = 1
n
n∑
i=1
ri (7)
where b is the size of mini-batch.
Sentence Order Prediction Loss We formulate Lsop as the cross-entropy loss between the golden and predicted orders as follows:
Lsop = − 1
n
n∑
i=1
oi log(pi), (8)
where oi is a one-hot ground-truth vector of correct sentence position, and pi is predicted probabilities.
Style Classifier Loss We expect the transferred text to be of the target style. Hence we train a style classifier to derive the style transfer loss as follows:
Lstyle = −Ex̂m∼Decoder[logPC(s|x̂m)], (9)
where PC is the conditional distribution over styles defined by the classifier. We train the classifier on the whole training set with the standard crossentropy loss. Then, we freeze the weights of style classifier for computing Lstyle. On the other hand, we follow Lee et al. (2021); Dai et al. (2019) to use soft sampling to allow gradient back-propagation.

heading: 3.4 Content Preservation Enhancing
text: As aforementioned, author styles have a strong correlation with contents. It is difficult to transfer such style-specific contents to other styles directly. Since we train the model in an auto-encoder manner, it will have no idea how to transfer those content representations that have never seen other style embeddings during training. To address the issue, we propose to mask the style-specific keywords in the source text and perform style transfer on the
masked text in the first generation stage. Then, we fill the masked tokens in the second stage.
We follow Xiao et al. (2021) to use a frequencybased method to identify the style-specific keywords. Specifically, we extract style-specific keywords by (1) obtaining the top-10 words with the highest TF-IDF scores from each corpus, (2) retaining only people’s names, place names, and proper nouns, (3) and filtering out those words with a high frequency in all corpora3. We denote the resulting word set as Ds for the corpus with the style s. We extract the style-specific keywords k from the text x by selecting the words that are in Ds. We detail above operation and explain it in Appendix A.
In the second stage, we train another model to fill the mask tokens in outputs of the first stage conditioned on the identified style-specific keywords in source inputs. During training, we concatenate the keywords in k with a special token ⟨Key⟩ and feed them into the encoder paired with xm, as shown in Figure 1. The training object is formulated as Equation 2. During inference, the decoder generates the transferred text x̂ conditioned on the output of the first stage x̂m in an auto-regressive manner.

heading: 4 Experiments

heading: 4.1 Datasets
text: We construct stylized story datasets in Chinese and English, respectively. The Chinese dataset consists of three styles of texts, including fairy tales from LOT (Guan et al., 2021a), LuXun (LX), and JinYong (JY). Specifically, LuXun writes realism novels while JinYong focuses on martial arts novels. These texts of different styles have a gap in lexical, syntactic, and semantic levels. Samples of different styles are detailed in Appendix C.
In our experiments, we aim to transfer a fairy tale to the LX or JY style. The English dataset consists
3We set those words appearing in at least 10% samples in a corpus as high-frequency words.
of two styles of texts, including everyday stories from ROCStories (Mostafazadeh et al., 2016) and fragments from Shakespeare’s plays. We expect to transfer a five-sentence everyday story into the Shakespeare style. The statistics of datasets are shown in Table 2. The more details are described in Appendix B.

heading: 4.2 Implementation
text: We take LongLMBASE (Guan et al., 2021a) and T5BASE (Raffel et al., 2020) as the backbone model of both generation stages for Chinese and English experiments, respectively. Furthermore, the fusion module and pointer network consist of two and one layers of randomly initialized bidirectional Transformer blocks (Vaswani et al., 2017), respectively. We conduct experiments on one RTX 6000 GPU. In addition, we build the style classifier based on the encoder of LongLMBASE and T5BASE for Chinese and English, respectively.
We set λ1/λ2/λ3 in Equation 1 to 1/1/1, the batch size to 4, the learning rate to 5e-5, the maximum sequence length of the encoder and decoder to 512 for both generation stages in the Chinese experiments. And the hyper-parameters for English experiments are the same except that λ1/λ2/λ3 are set to 0.5/0.5/0.5 and the learning rate to 2.5e-5. More implementation details are presented in Appendix D.

heading: 4.3 Baselines
text: Since no previous studies have focused on story author-style transfer, we build several baselines by adapting short-text style transfer models. For a fair comparison, we initialize all baselines using the same pre-trained parameters as our model. Specifically, we adopt the following baselines: Style Transformer: It adds an extra style embedding and a discriminator to provide style transfer rewards without disentangling content from styles (Dai et al., 2019). StyleLM: This baseline generates the target text conditioned on the given style token and corrupted version of the original text (Syed et al., 2020). Reverse Attention: It inserts a reverse attention module on the last layer of the encoder, which aims to negate the style information from the hidden states of the encoder (Lee et al., 2021).

heading: 4.4 Automatic Evaluation
text: Evaluation Metrics Previous works evaluate style transfer systems mainly from three aspects
including style transfer accuracy, content preservation, and sentence fluency. A good style transfer system needs to balance the contradiction between content preservation and transfer accuracy (Zhu et al., 2021; Niu and Bansal, 2018). We use a joint metric to evaluate the overall performance of models. On the other hand, previous studies usually use perplexity (PPL) of a pre-trained language model. However, in our experiments, we found that the PPL of model outputs is lower than human-written texts, suggesting that PPL is not reliable for evaluating the quality of stories. Therefore, we evaluate the fluency through manual evaluation.
Specifically, we adopt the following automatic metrics: (1) Style Transfer Accuracy: We use two variants of style transfer accuracy following Krishna et al. (2021), absolute accuracy (a-Acc) and relative accuracy (r-Acc). We train a style classifier and regard the classifier score as the a-Acc. And r-Acc is a binary value to indicate whether the style classifier score the output higher than the input (1/0 for a higher/lower score). We train the classifier by fine-tuning the encoder of LongLMBASE and T5BASE on the Chinese and English training set, respectively. The classifier achieves a 99.6% and 99.41% accuracy on the Chinese and English test sets, respectively. (2) Content Preservation: We use BLEU-n (n=1,2) (Papineni et al., 2002) and BERTScore (BS) (Zhang* et al., 2020) between generated and input texts to measure their lexical and semantic similarity, respectively. And we report recall (BS-R), precision (BS-P) and F1 score (BS-F1) for BS. (3) Overall: We use the geometric mean of a-ACC and BLEU/BS-F1 score (BL-Overall/BS-Overall) to assess the overall performance of models (Krishna et al., 2020; Lee et al., 2021).
Results on the Chinese Dataset We show the overall performance and individual metrics results in Table 3. In terms of overall performance, StoryTrans outperforms baselines, illustrating that StoryTrans can achieve a better balance between style transfer and content preservation.
In terms of style accuracy, StoryTrans achieves the best style transfer accuracy (a-Acc) in LX and comparable performance in JY. The bad performance of baselines indicates the necessity to perform explicit disentanglement beyond the token level. In addition, manual inspection shows that Style Transformer tends to copy the input, accounting for the highest BLEU score and BERTScore.
This means Style Transformer only takes the target style signals as noise, which may result from the stylistic features existing in the contents. StyleLM and Reverse Attention get better transfer accuracy than Style Transformer by removing such stylistic features from the contents. Moreover, Reverse Attention obtains better style accuracy but worse content preservation than StyleLM. Therefore, reweighting hidden states allows better control over style than deleting input words explicitly.
In terms of content preservation, StoryTrans outperforms Reverse Attention. Additionally, StyleLM achieves better performance in content preservation, benefiting from inputting noisy versions of golden texts. But without disentanglement, it can’t strip style information. This leads to a lower overall performance than StoryTrans. As for Style Transformer, the results demonstrate that only an attention-based model hardly removes style features in overwhelming tokens information, leading to degenerate into an auto-encoder.
Results on the English Dataset Similarly, StoryTrans achieves the best overall performance on the English dataset, showing its effectiveness and generalization. And StoryTrans outperforms baselines significantly in terms of style transfer accuracy. As for content preservation, Style Transformer and Re-
verse Attention degenerate into an auto-encoder, and tend to copy the input even more than their performance on the Chinese dataset.
Results on Ablation Study As shown in Table 4, we observe a significant drop in transfer accuracy without Ldis or Lstyle. Ldis works by disentangling stylistic features from the discourse representations, while Lstyle exerts direct supervision on styles of generated texts. Without Lsop, model can hardly capture discourse-level information and keeps more source tokens, leading to higher BLEU scores and lower accuracy. When removing the second stage, the lowers BLEU scores show the benefit of the mask-and-fill framework for content preservation.

heading: 4.5 Manual Evaluation
text: We randomly sampled 100 fairy tales from the Chinese dataset, and obtained 800 generated texts from StoryTrans and three baseline models. Then, we hire three Chinese native speakers to evaluate in three aspects including style transfer accuracy (Sty.), content preservation (Con.) and coherence (Coh.). We ask the annotators to judge each aspect from 1 (the worst) to 3 (the best).
As illustrated in Table 6, our StoryTrans received
the highest style accuracy and modest performance in content preservation and coherence. More details and analysis are presented in Appendix G.

heading: 4.6 Case Study
text: Table 5 shows the cases generated by StoryTrans and the best baseline. StyleLM inserts many unrelated sentences, which overwhelm the original content and impact the coherence, further leading to the content loss of sentences 3 and 4. On the contrary, StoryTrans supplement several short phrases or plots (e.g., “纵身跃起” /“hurriedly jumped up”) to enrich the storyline and maintain the main content. Furthermore, StoryTrans can rewrite most sentences with the target style and maintain source semantics. In addition, StyleLM tends to discard the source entities and use words which is specific in the target style (e.g., “郭靖” /“Guo Jing”), while StoryTrans dose not, suggesting the necessity of the mask-and-fill framework.

heading: 4.7 Stylistic Feature Visualization
text: We follow Syed et al. (2020) to define several stylistic features and visualize the features of the golden texts and generated texts on the Chinese test set. The stylistic features include the type and number of punctuation marks, the number of sentences, and
the number of words. As shown in Figure 3, the texts generated by Reverse Attention and StyleLM have similar stylistic features to source texts. In contrast, StoryTrans can better capture different stylistic features and transfer source texts to specified styles. More details are in Appendix F.

heading: 5 Conclusion
text: In this paper, we present the first study for story author-style transfer and analyze the difficulties of this task. Accordingly, we propose a novel generation model, which explicitly disentangles the style information from high-level text representations to improve the style transfer accuracy, and achieve better content preservation by injecting style-specific contents. Automatic evaluations show StoryTrans outperform baselines on the overall performance. Further analysis shows StoryTrans has a better ability to capture linguistic features for style transfer.
Limitations
In style transfer, content preservation and style transfer are adversarial. Long texts have richer contents and more abstract stylistic features. We also notice that content preservation is the main disadvantage of StoryTrans in automatics evaluation results. Case studies also indicate that StoryTrans can maintain some entities and the relations between entities. However, strong discourse-level style transfer ability endangered content preservation. In contrast, baselines such as Style Transformer have better content preservation but hardly transfer the style. We believe that StoryTrans is still a good starting point for this important and challenging task.
During preliminary experiments, we also manually inspected multiple author styles besides Shakespeare, such as Mark Twain. However, we found that their styles are not as obvious as Shakespeare, as shown in the following example. Therefore, we only selected authors with relatively distinct personal styles for our transfer experiments. In future work, we will expand our research and choose more authors with distinct styles for style transfer. For example, the style distinction between the following examples is not readily apparent.
• Everyday story in our datatset: Ashley wanted to be a unicorn for Halloween. She looked all over for a unicorn costume. She wasn’t able to find one.
• "A Double Barrelled Detective Story" by Mark Twain: You will go and find him. I have known his hiding-place for eleven years; it cost me five years and more of inquiry.
Ethics Statement
We perform English and Chinese experiments on public datasets and corpora. Specifically, English datasets come from ROCstories and Project Gutenberg. Moreover, Chinese datasets include the LOT dataset and public corpora of JY and LX. Automatic and manual evaluation demonstrate that our model outperforms strong baselines on both Chinese and English datasets. In addition, our model can be easily applied to different languages by substituting specific pre-trained language models.
As for manual evaluation, we hired three native Chinese speakers as annotators to evaluate generated texts and did not ask about personal privacy or collect the personal information of annotators. We pay 1.8 yuan (RMB) per sample in compliance with Chinese wage standards. Considering it would cost an average of 1 minute for an annotator to score a sample, the payment is reasonable.

heading: Acknowledgments
text: This work was supported by the NSFC projects (Key project with No. 61936010 ). This work was also supported by the Guoqiang Institute of Tsinghua University, with Grant No. 2020GQG0005.

heading: A Style-Specific Contents
text: We detail how we extract style-specific contents and explain how they are used from the following three aspects:
What do we mean by “style-specific content”? We refer to "style-specific content" as those mainly used in texts with specific styles and should be retained after style transfer. For example, "Harry Potter" and "Horcrux" are style-specific since they are used only in J.K. Rowling-style stories. When transferring J.K. Rowling-style stories to
other styles, style-specific tokens shouldn’t be changed. However, existing models tend to drop style-specific tokens since they are not trained to learn these tokens conditioned on other styles.
How do we extract “style-specific contents”? We extract style-specific contents by (1) obtaining top10 salient tokens using TF-IDF, (2) reserving only people names (e.g., "Harry Potter"), place names (e.g., "London"), and proper nouns (e.g., "Horcrux"), and (3) filtering out high-frequency tokens in all corpus (e.g., "London") since these tokens can be learned conditioned on every style. We regard the remaining tokens as style-specific contents.
As mentioned before, we employ the TF-IDF algorithm on the corpus to obtain rough style-specific contents for different styles, respectively. The reason for using TF-IDF: it is necessary to ensure that the extracted tokens are salient to the story plots. We extract style-specific tokens from the salient tokens using the second and third steps. Then, we use a part-of-speech tagging toolkit (e.g., NLTK) to identify function words and prepositions to retain people’s names, place names, and proper nouns. Note that the frequency is an empirical value observed from datasets. However, the TF-IDF algorithm chooses the important words corresponding to the special style based on word frequency. There may be some style-unrelated words that are important to the content. Therefore, we need to filter out style-unrelated words. Concretely, we use Jieba4/NLTK(Bird et al., 2009) to collect the word frequency for Chinese and English datasets, respectively. Moreover, we regard the words possessing a high frequency in all styles corpus as style-unrelated words. Specifically, We set tokens appearing in 10% samples in the dataset as highfrequency words. Then we filter out these words to obtain style-specific contents. The frequency value needs to be reset to apply the method to other datasets.
How are the “style-specific contents” used? One challenge of long-text style transfer is transferring discourse-level author style while preserving the main characters and storylines. It’s difficult for existing models to transfer style-specific contents since they are not trained to learn these tokens conditioned on other styles. Therefore, we extract "style-specific contents" before style transferring and replace them with the special token "<Mask>".
4https://github.com/fxsjy/jieba
Then, the "style-specific contents" will be filled in the second stage, as shown in Figure 1.

heading: B Data Pre-Processing
text: Due to lack of stylized author datasets, we collected several authors’ corpus to construct new datasets. As for Chinese, we extracted paragraphs from 21 novels of LuXun (LX) and 15 novels of JinYong (JY), and fairy tales collected by Guan et al. (2021a). On the other hand, the English dataset consists of everyday stories from ROCStories (Mostafazadeh et al., 2016) and fragments from Shakespeare’s plays. Each fragment of Shakespeare’s plays comprises multiple consecutive sentences and as long as samples in ROCStories. We collect the Shakespeare-style texts from the Shakespeare corpus in Project Gutenberg5 under the Project Gutenberg License6. We use Jieba/NLTK (Bird et al., 2009) for word tokenization for the Chinese/English dataset in data pro-processing. In addition, these data are public corpora, and we also check the information for anonymization.
Regarding to limitation of modern language models, the length of samples is also limited. We set the max length as 384 and 90 for Chinese and English, respectively. Each sample has 4 sentences at least. We choose above length to balance the data length of different styles. Additionally, we filtered the texts which are too long to generate or too short to unveil author writing style. As Figure 4 shows, texts in the Chinese dataset spans a diverse range of length.

heading: C Different Style Samples
text: In process of constructing datasets, we try to collect different author corpus who have a gap in writing styles. As shown in Table 8, the JY-style texts mostly describe martial arts actions and construct interesting plots, while the LX-style texts focus on realism with profound descriptive and critical significance. And the fairy tales differ from these texts in terms of topical and discourse features. In the English datasets, the Shakespeare-style texts are flamboyant and contain elaborate metaphors and ingenious ideas, which the everyday stories are written in plain language and without rhetoric.
5https://www.gutenberg.org 6https://www.gutenberg.org/policy/license.html

heading: D More Implementation Details
text: In terms of selecting pre-trained model, LongLMbase and T5base are the generic base model for the Chinese and English generation,
respectively. To optimize the models for these specific languages, we have fine-tuned them using different hyperparameter values (λ1/2/3). These values were determined based on the performance
observed on a validation set, which was created by pre-extracting 5% of the training data for this purpose.

heading: E More Ablation Study Results
text: To explore the effect of the proposed component, we also conduct more ablation studies on Chinese datasets. As shown in Table 9, the ablation of Ldis leads to better style accuracy, which show the different trends comparing with English dataset. We conjecture that Ldis aims to maintain the content and reduces style information. Without Ldis, the powerful Lstyle leads the StoryTrans to degenerate to style conditional language model. Furthermore, the ablation of Lstyle also confirms the powerful ability of style control as in previous paper. And we find that when removing Lsop, the model loses the ability to transfer at the discourse level and has only learned token-level copy.

heading: F Style Analysis of Transferred Texts
text: In order to investigate whether our StoryTrans indeed rephrase the expression of texts, we employ surface elements of text to show author writing
styles. And the surface element are associated with statistical observations. For example, the small average length of sentences show the author preference to write a short sentence, and more question marks indicate the author accustomed to using questions. To this end, we use the number of (1) commas, (2) colons, (3) sentences in a paragraph, (4) question mark (5) left quotation mark, (6) right quotation mark, and (7) average number of words in a sentence to quantify surface elements into a 7 dimension vector. Then we leverage the t-SNE to visualize the golden texts and transferred texts. As shown in Figure 3, different style distribute separately across the style space. This proves JY, LX and fairy tale in Chinese dataset have a gap in writing style. And Figure 5 shows the transferred texts fall in golden texts in style space, indicating StoryTrans successfully transferred the writing style.

heading: G More Details of Manual Evaluation
text: In addition to automatic evaluation, we conduct manual evaluation on generated texts. As mentioned before, we require the annotators to score each aspect from 1 (the worst) to 3 (the best). As for payment, we pay 1.8 yuan (RMB) per sample in compliance with Chinese wage standards. Our annotators consist of undergraduate students who are experienced in reading texts written in the styles of the respective authors (JY and LX). To ensure they fully understand the evaluation metrics, we conducted case analyses with them. Our scoring rubric assigns 1, 2, or 3 points to the transferred text based on the proportion of sentences meeting the following criteria (1/3, 2/3, or 3/3):
• Style Accuracy: whether the transferred text conforms to the corresponding style.
• Content Preservation: whether the source content, such as character names, are retained.
• Coherence: whether the sentences in the transferred text are semantically connected.
And we compute the final score of each text by averaging the scores of three annotators.
As illustrated in manual evaluation, we observe that the results mainly conform with the automatic evaluation. Our StoryTrans obtained the highest score on the style accuracy in both transferred directions by a sign test compared to the other baselines, showing its stable ability of style control. Moreover, in terms of content preservation, the score
of StoryTrans is comparable with StyleLM and slightly higher than Reverse Attention, demonstrating that StoryTrans can keep the main semantics of input. In terms of coherence, the score of StoryTrans is also comparable with baselines, showing some room for improvement. As discussed before, Style Transformer tends to copy the input, leading to the highest performance in content preservation and coherence. In summary, human evaluation depicts the strength of StoryTrans not only on style control but also on overall performance, indicating a balance of these metrics.

heading: H More Case Studies
text: We show more cases in Table 7. Comparing source text with Style Transformer, Style Transformer copies the input and only changes little tokens. This result also confirms with highest BLEU and BERTScore in automatic results. Like StyleLM, Reverse Attention also incorporates some target author content into generated texts. However, Reverse Attention inserts too much content that overwhelms original plots. Furthermore, some critical entities (e.g., character name, “柯里教授” /“Professor Curry” → “柯镇恶” /“Ke Zhen’e”) are revised to the similar word on in target author corpus. To maintain the story coherence, these important entities should stay the same. In summary, the token-level transfer may destroy the essential plots and damage the coherence.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
3 A2. Did you discuss any potential risks of your work? 6
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4.2
3 B1. Did you cite the creators of artifacts you used? 4.2
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Appendix B
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Appendix B
3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Appendix B
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 4.1 and Appendix B
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We counted the details of the dataset and discussed the details in Section 4.1
C 3 Did you run computational experiments? 4.4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.2
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? 4.5
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? 4.5
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? 4.5 and Appendix F
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Appendix B and Appendix F
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. We do not submit the protocol to an ethics review board because our country has not yet established an ethical committee at the national level.
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? 4.5 and Appendix F

heading: Future Work
text: In future work, we will expand our research and choose more authors with distinct styles for style transfer. For example, the style distinction between the following examples is not readily apparent. • Everyday story in our datatset: Ashley wanted to be a unicorn for Halloween. She looked all over for a unicorn costume. She wasn’t able to find one. • "A Double Barrelled Detective Story" by Mark Twain: You will go and find him. I have known his hiding-place for eleven years; it cost me five years and more of inquiry