SC2: Towards Enhancing Content Preservation and Style Consistency in Long Text Style Transfer (2024)

Abstract: Text style transfer (TST) aims to vary the style polarity of text while preserving the semantic content. Although recent advancements have demonstrated remarkable progress in short TST, it remains a relatively straightforward task with limited practical applications. The more comprehensive long TST task presents two challenges: (1) existing methods encounter difficulties in accurately evaluating content attributes in multiple words, leading to content degradation; (2) the conventional vanilla style classifier loss encounters obstacles in maintaining consistent style across multiple generated sentences. In this paper, we propose a novel method SC2, where a multilayer Joint Style-Content Weighed (JSCW) module and a Style Consistency loss are designed to address the two issues. The JSCW simultaneously assesses the amounts of style and content attributes within a token, aiming to acquire a lossless content representation and thereby enhancing content preservation. The multiple JSCW layers further progressively refine content representations. We design a style consistency loss to ensure the generated multiple sentences consistently reflect the target style polarity. Moreover, we incorporate a denoising non-autoregressive decoder to accelerate the training. We conduct plentiful experiments and the results show significant improvements of SC2 over competitive baselines. Our code: https://github.com/jiezhao6/SC2.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9949–9960 August 11-16, 2024 ©2024 Association for Computational Linguistics
In this paper, we propose a novel method SC2, where a multilayer Joint Style-Content Weighed (JSCW) module and a Style Consistency loss are designed to address the two issues. The JSCW simultaneously assesses the amounts of style and content attributes within a token, aiming to acquire a lossless content representation and thereby enhancing content preservation. The multiple JSCW layers further progressively refine content representations. We design a style consistency loss to ensure the generated multiple sentences consistently reflect the target style polarity. Moreover, we incorporate a denoising non-autoregressive decoder to accelerate the training. We conduct plentiful experiments and the results show significant improvements of SC2 over competitive baselines. Our code: https://github.com/jiezhao6/SC2.

heading: 1 Introduction
text: Text style transfer (TST) aims to generate a text exhibiting a desired style based on the source text (e.g., negative → positive), while endeavoring to faithfully preserve the semantic content. The applications of TST cover a wide range of user-centric natural language generation tasks, such as personalized dialogue systems (Gao et al., 2019), educa-
∗Corresponding author.
tional platforms (Wang et al., 2019), and writing assistants (Syed et al., 2020).
Given the scarcity of parallel data (i.e., text pairs conveying the same content but differing in styles) and the labor-intensive nature of annotating such pairs, existing research has predominantly focused on unsupervised TST. Recent contributions in this domain, including studies by (Zhou et al., 2020; Lee et al., 2021; Huang et al., 2021; Suzgun et al., 2022; Ramesh Kashyap et al., 2022; Han et al., 2023), have demonstrated significant progress. Despite notable success, these works primarily concentrate on the transfer of a single sentence, which we call short TST. This is a relatively simple task and is difficult to apply to complex scenarios, i.e., long TST such as transferring news articles and novels. It is challenging to achieve desirable content preservation and style consistency across multiple sentences for these methods. In a very recent study, Zhu et al. (2023) first concentrated on the long TST task and proposed StoryTrans, which learns content representations and fills stylistic tokens in separate stages. While somewhat effective, challenges persist in preserving content and ensuring style consistency.
Content Preservation: The critical factor for preserving content lies in accurately assessing the amount of content attribute (CA) within a token to improve the content representation. In tradi-
9949
tional approaches, the content learning primarily involves explicitly (Wu et al., 2019; Huang et al., 2021) or implicitly (Zhou et al., 2020; Lee et al., 2021) removing style tokens. In these processes, they evaluate the relevance between text and style, which solely consider the amount of style attribute (SA) within a token and neglect the CA amount (Figure 1 (a)). This results in tokens with both strong style and content attributes, such as “euphonious”1, potentially receiving higher SA scores, making them more drastic to be removed and consequently leading to content degradation. On the other hand, in StoryTrans, the disentanglement of content from style is achieved by encouraging texts with distinct styles to be close together in the content space ( Firure 1 (b)). However, owing to the non-parallel nature of the data, this unavoidably results in a loss of content information.
Style Consistency: To control the style polarity of generated text, existing methods employ a style discriminator that operates on the entire output text. However, in the context of long TST, it becomes challenging for the discriminator to ensure that the style of the generated multiple sentences is consistent. As a result, some generated sentences might exhibit strong target style polarity while others remain weak, which creates a less reader-friendly outcome. Therefore, we argue that maintaining style (polarity) consistency across multiple sentences is crucial.
To tackle the above issues, we propose a novel approach aimed at enhancing content preservation and maintaining style consistency in long TST. Our approach achieves these objectives by carefully designing a multilayer Joint Style-Content Weigher (JSCW) module and a Style Consistency loss, thus we call it SC2. (1) The JSCW utilizes the convolution operation to measure the SA amount within the center token. Simultaneously, it assesses the CA amount by computing and integrating the content relevance of a token across all sentences. Then by normalizing these two amounts and weighting the CA amount to the tokens’ representations, we obtain preliminary content representation. Furthermore, we employ multiple JSCW layers to progressively refine content representations. Finally, we fuse the target style information to content representations and feed them to the decoder to generate target text. (2) For the other challenge, we design
1“Euphonious” is strongly associated with a positive sentiment style and concurrently indicates content related to music.
a contrastive learning-based style consistency loss. It brings each generated sentence closer to the previously generated sentences or target sentences in the corpus, while farther away from the sentences in the source text.
Additionally, within the unsupervised long TST setting, directly employing an autoregressive (AR) decoder substantially slows down the training since the masked self-attention technique (Vaswani et al., 2017) cannot be exploited. Hence, drawing inspiration from the research on non-AR (NAR) text generation (Gu et al., 2017; Huang et al., 2021), we incorporate an auxiliary denoising NAR decoder. It parallelly generates pseudo-target texts, which are then fed into the AR decoder to accelerate the training process. Our main contributions are summarized as follows:
• We propose to explicitly and simultaneously assess the SA and CA amounts within tokens to learn lossless content representations and show that this idea can significantly improve content preservation in long TST. • We first propose the concept of style consistency for long TST, and design a corresponding loss to encourage the generated text to consistently maintain style polarity across multiple sentences. • Extensive experiments are conducted on both Chinese and English datasets to verify the effectiveness of the proposed SC2. The results demonstrate significant improvements over competitive baselines.

heading: 2 Related Work
text: Text Style Transfer. Recently, there has been considerable research attention on TST. The predominant of current efforts lies in unsupervised TST since acquiring parallel data is labor-intensive, even impossible to crowdsource for some styles (Jin et al., 2022). One line explicitly disentangles text into content and style representations, and then incorporates the target style information into the content-based generation process. Li et al. (2018); Wu et al. (2019); Lee et al. (2021) proposed using frequency ratio or attention scores to measure the SAs within tokens, and then entirely or partially removing the stylistic tokens. Tokpo and Calders (2022) introduced the LIME explainer (Ribeiro et al., 2016) to identify biased (style) tokens and then masked them for non-biased text generation. John et al. (2019) and Yi et al. (2020) employed
the adversarial loss to learn the style-agnostic content representation. Krishna et al. (2020) employed a pretrained paraphrase model to strip away the style information and generated target text conditioning on the style-dependent inverse paraphrase model. The other line uses entangled representations to perform TST. Jin et al. (2019) and Xu et al. (2023) created pseudo-parallel data and then trained a Seq2Seq model. Syed et al. (2020) trained a generator for each target style. Riley et al. (2021) extracted a style vector from the adjacent sentence and used it to condition the decoder. Our work falls into the first line. Nevertheless, what distinguishes our work is the explicit and simultaneous measurement of SA and CA amounts within tokens.
The aforementioned works all handle the task of Single-Sentence (SS) transfer, conforming to an SS-input and SS-output paradigm. Differing from these approaches, Cheng et al. (2020) introduced the use of context sentences as additional information to preserve topical coherence between the generated sentence and its surrounding context. Although it employs multiple sentences as input, fundamentally, this work still falls under the setting of SS transfer. Zhu et al. (2023) introduced the task of long TST and proposed a two-stage training and inference framework to learn content representations by aligning source texts and target texts. We follow this task, aiming to enhance content preservation and style consistency.
Non-AutoRegressive (NAR) Text Generation. Gu et al. (2017) first introduced the NAR concept in machine translation which produces output tokens in parallel. Subsequent works introduce the NAR decoder into other domains, such as dialogue generation (Han et al., 2020), speech synthesis (Peng et al., 2020), and short TST (Huang et al., 2021). Inspired by these works, we incorporate an auxiliary denoising NAR decoder to accelerate the training of SC2.

heading: 3 Approach

heading: 3.1 Problem Definition
text: Let x = (x1, x2, · · · , xTx) be a source text with multiple sentences and s ∈ S be its style label, where S is the style label set. Our goal is to develop a model fθ that can generate a text y = (y1, y2, · · · , yTy) exhibiting a desired target style ŝ (ŝ ∈ S and ŝ ̸= s) for a given text x. The generated text y shares the same semantic content
with x. Subsequently, we use bold face lower/upper case letters to denote vectors/matrices respectively.

heading: 3.2 Model Architecture
text: As shown in Figure 2, the proposed SC2 fits within the encoder-decoder framework, and mainly consists of multilayer JSCW module, style fusion module, and denoising NAR decoder. After the encoding of original tokens, the JSCW evaluates the SA and CA amounts simultaneously and learns elementary content representations. And we propose to progressively refine content representations by multiple JSCW layers. Subsequently, the style fusion module combines the style information and style-agnostic content representations to provide signals of transfer direction. And the results are fed into the decoder. Additionally, the denoising NAR decoder generates pseudo-target text to accelerate the training. For simplicity, our discussion in the following section focuses on the transfer direction of s → ŝ. Nevertheless, our approach can be easily extended to handle transfers among multiple styles by incorporating multiple style embeddings.

heading: 3.2.1 Multilayer Joint Style-Content Weigher
text: Assuming that the length of x is n and it consists of m sentences, we insert a special token ⟨SEN⟩ at the end of each sentence, inspired by (Lee et al., 2020; Zhu et al., 2023). Consequently, we feed x into the encoder, obtaining n + m hidden states (i.e., {xi ∈ Rd}n+mi=1 ). For convenience, we use xi (i ∈ [1, n]) and ej (j ∈ [1,m]) to denote the representations of i-th word token and j-th sentence token respectively. Inspired by the work of (Lee et al., 2021), we assume that the representation of a word token is comprised of style information αixi and content information βixi, where αi is the SA score (amount), βi is the CA score (amount), and αi + βi = 1. The essence of the JSCW lies in discerning the values of αi and βi simultaneously to effectively disentangle the content from style for a word token.
Style Weigher. We introduce a learnable style embedding s ∈ Rd to assist the computation of SA score, which will also serve as a style control signal. A straightforward idea is to directly measure the SA score based on the current word token (e.g., xiWss, where Ws is a bilinear term enabling flexible estimation). However, individual words sometimes do not reflect their association with a specific style. Therefore, we employ a con-
volution operation applied to a window of 2h+ 1 tokens, generating a new feature used to assess the SA score of the center token. Formally, the nonnormalizing SA score of a word token is computed as follows:
β̃i = σ(Wconvxi−h:i:i+h + bconv)Wss, (1)
where σ(x) = max(x, 0) is the rectifier activation function, and Wconv and bconv are convolution matrix and bias vector respectively.
Content Weigher. The content of a word token varies across diverse contexts, exemplified by the polysemy. Hence, employing a single representation akin to style embedding is inadvisable for assessing a word’s CA score. On the other hand, in a multi-sentence scenario, a word’s semantics may be interconnected with or influenced by sentences beyond its immediate sentence. To address these challenges, we compute the content-relevance scores between a word and all sentences, subsequently attentionally aggregating them as the nonnormalizing CA score. Specifically, we employ the following equation to achieve this objective:
µi,j = exp(xiWcej)∑m j=1 exp(xiWcej) ,
α̃i = ∑m
j=1 µi,j(xiWcej),
(2)
where Wc is a bilinear term. After obtaining the individual SA score and CA score, we normalize them as: [αi, βi] = softmax([α̃i, β̃i]). Finally, the content representation of a word token is the weighted xi:
xci = βixi. (3)
We define the above disentanglement process (Eqs. (1)∼(3)) as a JSCW layer. However, the assessment of the CA score might introduce bias due to the potential inclusion of style information in the sentence representation ej . To address this concern, we modify the JSCW layer by incorporating a subsequent Transformer layer (Vaswani et al., 2017). Leveraging multi-head self-attention mechanisms, this Transformer layer utilizes elementary content representations obtained from the original JSCW layer to learn new sentence representations. And we employ multiple JSCW layers, which progressively diminishes the style information within sentence representations and enables a more precise assessment of CA score.

heading: 3.2.2 Style Fusion Module
text: After extracting the words’ content representations, we integrate the style information into them to provide transfer signals. We concatenate the target style embedding with content representations and employ multiple Transformer layers (MTL) as the fusion network to learn style-aware representations. Formally, the process is formulated as:
Zŝ = MTL(Q = K = V = Z̃ŝ), (4)
where Z̃ŝ = [ŝ,xc1,x c 1, · · · ,xcn] ∈ Rd×(n+1), and Q, K and V respectively represent the query, key, and value within the Transformer layer. In summary, we denote the encoder, multiple JSCW layers, and the style fusion module as a network F , i.e., Zŝ = F (x, ŝ).

heading: 3.2.3 Decoder
text: We employ an autoregressive (AR) decoder to generate target text. The decoder accesses style-
aware words’ representations Zŝ through the crossattention layer. The AR decoder is defined as:
PDAR(y | Zŝ) = ∏Ty
t=1 PAR(yt | y<t,Zŝ) (5)
However, a practical issue arises in terms of time consumption during training. In the unsupervised setting, the decoder cannot leverage the masked self-attention technique (Vaswani et al., 2017) to efficiently train the network and can only generate target tokens one by one. For long TST, it significantly increases the training time. Hence, inspired by recent research on non-AR (NAR) text generation (Gu et al., 2017; Huang et al., 2021), we propose a simple but effective denoising NAR decoder2, which generates pseudo-target tokens in parallel, to accelerate the training. The generated text yp will be fed into the AR decoder to exploit the masked self-attention to generate target-styled text. The denoising NAR decoder is defined as:
PDNAR(y p | Zŝ,Znŝ ) = ∏Ty t=1 PDNAR(y p t | Zŝ,Znŝ ), (6)
where Znŝ is a corrupted version of Zŝ. Specifically, we randomly swap a token with one of its 2k neighboring tokens, where the swapping probability is p. We also employ a source style embedding in the fusion module and use the NAR to obtain xp. During training, the NAR decoder will learn to acquire the capability of reconstructing source texts of various styles. It serves as providing pseudo-labels to the AR decoder, which is somewhat similar to some pseudo-parallel data construction approaches (Jin et al., 2019). The key difference lies in the endto-end training for both denoising NAR and AR decoders.

heading: 3.3 Training Objectives
text: In this section, we elaborate on the training objectives employed in SC2.
Style-Oriented Objectives. To constrain the style polarity of the generated text, we employ a document-level objective and design a sentencelevel objective. We pre-train two style classifiers Cdocu and Csent using standard cross-entropy loss on the entire long text and segmented sentences, respectively. The weights of these two classifiers are frozen during the training of SC2.
2The NAR decoder will be removed during inference since a simple NAR generator may not perform very well.
We first use the widely employed vanilla style transfer loss based on the document-level classifier:
Lstyd = −Ey∼DAR [logPCdocu(ŝ | y)].
Secondly, we consider maintaining the consistency of style polarity across multiple sentences. It is challenging for the document-level style classifier to achieve this goal. Therefore, we derive the following contrastive style consistency loss:
Lstys = − 1m̂ m̂∑ j=1 j−1∑ j′=1 log( exp((oyj ·o y j′ )/τ)∑m j′′=1 exp((o y j ·oxj′′ )/τ) ),
where τ is a temperature parameter, m̂ (m) is the number of sentences in y (x), j and j′ (j′′) are the sentence indexes in y (x), and o is the output of the last layer of the sentence-level classifier. When j = 1, the oyj′ is obtained by randomly sampling a target sentence from the training corpus. This loss makes the j-th generated sentence maintain stylistic similarity with sentences 1 ∼ j − 1 and target sentences in corpus, while diverging from the source sentences. This ensures both the style consistency and polarity of the generated text.
The overall loss for controlling style polarity is defined as:
Lsty = Lstyd + Lstys . (7)
Content-Oriented Objectives. We employ a reconstruction loss containing self- and cyclereconstruction, which is formulated as:
Lrec = −Ex∼Px [logPDAR(x | F (x, s))]− Ex∼Px [logPDAR(x | F (y, s))]. (8)
NAR Decoder-Oriented Objective. The denoising NAR decoder is trained to reconstruct the source text, and we use the following loss:
LNAR = −Ex∼Px [logPDNAR(xp | Zs,Zns )], (9)
where Zs = F (x, s).
Disentanglement-Oriented Objective. We empirically observe that during the early stage of training, the distribution formed by [αi, βi] tends to align with the Bernoulli distribution. This undermines the disentanglement learning. We design a penalizing term to mitigate this issue.
Ldis = ∑n
i=1 max(0, ϵ−H([αi, βi])), (10)
where H(·) represents the entropy of a distribution, and ϵ is a threshold.
In summary, the overall loss of SC2 is defined to balance the above losses:
L = Lrec + λ1Lsty + λ2LNAR + λ3Ldis. (11)
An training issue arises with Eqs. (7)∼(8), where discrete tokens in y prevent the back-propagation of gradients. We address this issue by employing the soft-sampling strategy (Dai et al., 2019).

heading: 4 Experiments

heading: 4.1 Dataset
text: Following the work of Zhu et al. (2023), we employ the stylized story dataset to evaluate the performance of SC2. The dataset encompasses subdatasets in both Chinese and English. Important statistics are summarized in Table 1.
The Chinese sub-dataset comprises three styles of texts: LuXun (LX)’s novels, JinYong (JY)’s novels3, and Fairy Tale (FT), totaling approximately 7.5k samples for training. The corresponding tasks involve transforming an FT into text with LX’s style (FT → LX) or JY’s style (FT → JY). The English sub-dataset contains everyday stories and fragments from Shakespeare (SP)’s plays. There are approximately 2.3k training samples. The aim is to transfer an everyday story (ER) into the style of Shakespeare’s plays (ER → SP).

heading: 4.2 Baselines
text: We employ StoryTrans (Zhu et al., 2023), a twostage framework aiming to learn content representations and filling stylistic tokens for long TST, as the strongest baseline in our experiments. Additionally, we compare SC2 with the following baselines designed for short TST: Style Transformer (Dai et al., 2019) equips the power of attention mechanism within the Transformer (Vaswani et al., 2017) architecture to learn entangled text representations. StyleLM (Syed et al., 2020) fine-tunes a pretrained language model (PLM) on a corrupted version of the style-specific corpus for each style. Reverse Attention (Lee et al., 2021) employs reverse attention scores, which assess each token’s contribution to the style classification, to implicitly remove the style information in tokens.
3LuXun was a realism novelist and JinYong was known for his martial arts novels.
AugZero-Shot (Reif et al., 2022) is an augmented zero-shot prompting method, which prompts the PLM with diverse sentence-rewriting examples.

heading: 4.3 Evaluation Metrics
text: Following previous studies of (Syed et al., 2020; Lee et al., 2021; Reif et al., 2022; Zhu et al., 2023), we employ both automatic and human evaluations.

heading: 4.3.1 Automatic Evaluation
text: Style Transfer Accuracy: To evaluate the style polarity of the generated text, we separately pretrained document- and sentence-level style classifiers for each language. The style classifier utilizes the same architecture as the encoder of SC2 and incorporates a mean-pooling layer and a classification layer. In the case of Chinese, our style classifier achieves accuracies of 100% and 98% for document- and sentence-level test corpus, respectively. For English, these accuracies stand at 100% and 100%. Consequently, we employ two metrics, denoted as Accd and Accs, to measure documentand sentence-level style transfer accuracy, respectively.
Content Preservation: Firstly, we employ BLEU1 and BLEU2 (Papineni et al., 2002) to evaluate the content preservation through n-gram overlap. Secondly, we employ the BERT Score (Zhang et al., 2020) metric. It leverages PLMs to assess the semantic similarity of tokens and provides a more human-like evaluation. Precision (BSP), recall (BSR), and F1 (BSF1) scores are reported.
Overall: Recognizing that content preservation and style transfer as two major challenges within the TST task, we report two metrics for the overall performance (Lee et al., 2021; Zhu et al., 2023) and use them as the main metrics. Specifically, we employ the geometric means of Accd and BLEU (G-BLd), and Accd and BSF1 (G-BSd)).
Zhu et al. (2023) have noted that the perplexity based on PLMs is unreliable for fluency evaluation on this dataset. Consequently, we employ the man-
ual evaluation to assess the fluency of the generated text.

heading: 4.3.2 Human Evaluation
text: We randomly select 100 samples from the Chinese test set and generate target texts with LX style and JY style for baselines and proposed SC2. Given the source text and the target style, three Chinese native speakers are assigned the task of scoring for style polarity (Sty), content preservation (Con), and fluency (Flu), using a scale of 1 (very bad) to 3 (very good). We report the average scores across the three annotators as our final results.

heading: 4.4 Implementation Details
text: We employ LongLMSMALL (Guan et al., 2022) and T5SMALL (Raffel et al., 2020) as backbone models for Chinese and English data respectively. For the NAR decoder, we use 6 Transformer layers. The number of JSCW layers is set to 3. For Chinese data, we set λ1/λ2/λ3 to 0.05/1/1, p/k for the NAR decoder to 0.3/2, the threshold ϵ to 0.15, the batch size to 2, and the learning rate to 5 × 10−5 (Adam optimizer). For English data, the hyperparameters are the same except that λ1/p/k are set to 0.001/0.1/1. The experiments are conducted on one RTX 4060Ti GPU (16G) and the training of SC2 approximately takes 7~9 hours.

heading: 4.5 Results
text: Table 2 shows the experimental results using automatic metrics. We can obtain the following observations: (1) the methods (StoryTrans, SC2) that are
carefully designed for long TST usually perform better than traditional methods (Style Trnasformer, StyleLM, Reverse Attention, and AugZero-Shot). This demonstrates a specialized model is essential for long TST. (2) The Style Transformer and StyleLM exhibit significant shortcomings in controlling style polarity, while the high BLEU scores and Bert scores suggest substantial copying of the source text. This may be attributed to the difficulty of transferring entangled text representations to target tokens. (3) Compared to StoryTrans, the proposed SC2 demonstrates significant advantages in both style polarity control and content preservation, particularly concerning the control of sentence style. (4) The difficulties of imitating the writing styles of different authors vary. The similar BLEU metrics for SC2 suggest consistent surfacelevel content preservation across three sub-tasks. However, significant differences are observed in the control of document- and sentence-level style polarities for three sub-tasks. (5) For the overall metrics, we use t-test with significance level 0.05 to test the significance of performance difference. Results show that SC2 significantly outperforms all the baselines.
As for the human evaluations, the results are shown in Table 3. Here we neglect the baselines of Style Transformer and StyleLM, which show poor performance in automatic evaluation. For content preservation and style polarity, the results from manual evaluation align with those from automatic evaluation, with our proposed SC2 achieving the highest scores. Additionally, annotators provided
evaluations on fluency, where our proposed method significantly outperformed the StoryTrans. While the AugZero-Shot excels in fluency, its style polarity and content preservation are both too weak. Results indicate the superiority of the proposed SC2 across various metrics.

heading: 4.6 Analysis
text: Case Study: Table 4 illustrates examples of generated texts with the target style of JY by the best baseline StoryTrans and SC2. In the text generated by StoryTrans, a substantial portion exhibits low relevance with the source text, indicating poor content retention. For style polarity, we observe that SC2 rewrites more of the source text with the target style and inserts more target-style phrases. In terms of fluency, StoryTrans performs less satisfactorily, as exemplified by the generation of peculiar phrases such as “在宁静的一头燕时” (“at a tranquil swallow”). It is evident that, in terms
of content preservation, style polarity, and fluency, SC2 outperforms StoryTrans significantly.
Ablation Study: We conduct ablation experiments on the English dataset to demonstrate the effectiveness of two crucial components in SC2. The results are presented in Table 5, where “-” indicates that we remove the corresponding component. Particularly, “-JSCW” refers to the removal of the calculation of CA score, forming a new computational process as βi = sigmoid(β̃i), αi = 1 − βi. we can see that the performance of these variants drops apparently, which confirms the effectiveness of these proposed components in SC2.
Visualization of the Latent Space: In Figure 3, we employ the t-SNE to visualize the original, content and fused latent space of Chinese and English test datasets. For the Chinese dataset, we use “JY” as the target style. The representation of an input text x is the average of all tokens. It is evident
that the entangled features, disentangled content features, and fused stylistic features exhibit three distinct distributions. This demonstrates that our approach can effectively learn content information from the raw data and integrate the target style information into it.

heading: 4.7 Extrinsic Experiment
text: We further conduct an extrinsic experiment to show that the proposed SC2 can function as a data augmenter, improving the performance of data-scarce natural language processing tasks. Specifically, we concentrate on a charge prediction task within the legal domain. Based on the factual details of legal cases, it aims at predicting the ultimate charges convicted by courts for suspects. Traditional methods, which utilize professional legal-linguistic style (PLLS) data, typically benefit only legal experts because the domain discrepancy between PLLS and non-PLLS degrades the models’ performance on non-PLLS texts. However, unprofessional users also show an increasing demand on such a prediction service. The scarcity of non-PLLS data poses significant challenges in developing effective models that benefit ordinary users (Zhao et al., 2023).
We address this problem by training SC2 to transfer the PLLS fact descriptions to non-PLLS texts, and then developing a charge prediction model based on the hybrid data (we mix PLLS and nonPLLS texts). The utilized datasets are CAIL (Xiao et al., 2018) and NCCP (Zhao et al., 2023) for PLLS and non-PLLS respectively. We employ three charge prediction methods (FewShot (Hu et al., 2018), CECP (Zhao et al., 2022) and DLCCP (Zhao et al., 2023)) and three domain adaptation methods (FADA (Motiian et al., 2017), d-SNE (Xu et al., 2019) and DAGE (Morsing et al., 2021)) as baselines, following the work of (Zhao et al., 2023). We also employ the long TST method StoryTrans (Zhu et al., 2023) as a data augmenter. The settings for training SC2 and StoryTrans are consistent with that of the Chinese sub-dataset in this paper. And the settings for training baselines and charge prediction model are consistent with (Zhao et al., 2023). We employ Accuracy (Acc.), Macro-
Precision (MP), Macro-Recall (MR), and MacroF1 (F1) as the evaluation metrics. The results are shown in Table 6. It is evident that the proposed SC2 effectively improves the performance of nonPLLS charge prediction compared to all baselines.

heading: 5 Conclusion
text: In this paper, we propose a novel method SC2 for long TST. We design multiple JSCW layers to progressively refine content representations of tokens, thus enhancing content preservation. And we design a style consistency loss to ensure the generated sentences share similarities in style polarity. Experimental results with automatic and human evaluations confirm the effectiveness of SC2 compared to competitive baselines.
Limitations
We have not yet evaluated the performance of SC2 with larger pre-trained language models since they are not mainstream methods for TST and they exceed the capacity of our available GPU resources. Another limitation is that, for both Chinese and English, the data used in our experiments only involves a single type of long TST, specifically imitating the writing style of a particular author. We also find the styles to be quite distinct because of the high accuracy scores obtained by style classifiers. This limitation arises from the fact that there is only one available dataset for the long TST in the existing research. We leave the construction of diverse data (e.g., formality or humor transfer, and
texts with closer styles) and the comprehensive validation of SC2’s generalizability and adaptability to future work.
Ethics Statement
Long TST involves generating target text based on both the original text and target style signals. The nature of text generation and the diversified stylistic corpus raise potential ethical concerns, including the risk of generating text that may be inappropriate, offensive, or biased. On the other hand, the model carries the risk of being maliciously exploited, such as generating fake news and fabricating political statements. Future research on how to mitigate these issues is in crucial need.
We hired three Chinese native speakers as annotators to manually evaluate the performance of the proposed method and baselines. Considering the wage standards of China, annotators will get 2.0 yuan (RMB) for each sample.

heading: Acknowledgements
text: This research was supported by the National Natural Science Foundation of China (Grant Nos. 62133012, 61936006, 62103314, 62073255, 62303366), the Key Research and Development Program of Shaanxi (Program No. 2020ZDLGY0407), Innovation Capability Support Program of Shaanxi (Program No. 2021TD-05) and Natural Science Basic Research Program of Shaanxi under Grant No.2023-JC-QN-0648.

heading: Future Work
text: , formality or humor transfer, and
texts with closer styles) and the comprehensive validation of SC2’s generalizability and adaptability to future work