Counterfactual Generation with Identifiability Guarantees
Authors: Hanqi Yan
Lingjing Kong
Lin Gui
Yuejie Chi
Eric Xing
Yulan He
Kun Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Causal Representation Learning
Identifiability
Counterfactual Generation
Latent variable models
Disentanglement.
TL;DR: We provide novel identifiability theory for counterfactual generation and design a principled framework grounded by theory.

Abstract: Counterfactual generation lies at the core of various machine learning tasks, including image translation and controllable text generation. This generation process usually requires the identification of the disentangled latent representations, such as content and style, that underlie the observed data. However, it becomes more  challenging when faced with a scarcity of paired data and labelling information. Existing disentangled methods crucially rely on oversimplified assumptions, such as assuming independent content and style variables, to identify the latent variables, even though such assumptions may not hold for complex data distributions. For instance, food reviews tend to involve words like “tasty”, whereas movie reviews commonly contain words such as “thrilling” for the same positive sentiment. This problem is exacerbated when data are sampled from multiple domains since the dependence between content and style may vary significantly over domains. In this work, we tackle the domain-varying dependence between the content and the style variables inherent in the counterfactual generation task. We provide identification guarantees for such latent-variable models by leveraging the relative sparsity of the influences from different latent variables. Our theoretical insights enable the development of a doMain AdapTive counTerfactual gEneration model, called (MATTE). Our theoretically grounded framework achieves state-of-the-art performance in unsupervised style transfer tasks, where neither paired data nor style labels are utilized, across four large-scale datasets.

Counterfactual Generation with Identifiability Guarantees
Hanqi Yan 1 Lingjing Kong"*, Lin Gui Yuejie Chi?, Eric Xinga+4, Yulan He Kun Zhang?*
University of Warwick, 2 Carnegie Mellon University, 'King's ' College London, "Mohamed Bin Zayed University Artificial Intelligence
Abstract
Counterfactual generation lies at the core of various machine learning tasks, in- cluding image translation and controllable text generation. This generation process usually requires the identification the disentangled latent representations, such content and style, that underlie the observed data. However, it becomes challenging when faced with a scarcity of paired data and labeling information. Existing disentangled methods crucially rely oversimplified assumptions, such as assuming independent content and style variables, to identify the latent variables, even though such assumptions may not hold for complex data distributions. For instance, food reviews tend to involve words like "tasty", whereas movie reviews commonly contain words such as "thrilling" for the same positive sentiment. This problem is exacerbated when data are sampled from multiple domains since the dependence between content and style may vary significantly over domains. In this work, we tackle the Nomain-varying dependence between the content and the style variables inherent in the counterfactual generation task. We provide identification guarantees for such ditent-varible models by leveraging the relative sparsity of the influences from different latent variables. Our theoretical insights enable the development of do.ain AdapTTive coun't coun Terfactual gEneration model, called (MATTE). Our theoretically grounded framework achieves state-of-the-art performance in unsupervised style transfer tasks, where neither paired data nor style labels are utilized, across four large-scale datasets.
Introduction
Counterfactual generation serves as crucial component in various machine learning applications, such controllable text generation and image translation. These applications aim producing new
data with desirable style attributes (e.g., sentiment, tense, or hair color) while preserving the other
information (e.g., topic identity) [Li et al., 2019 2022 Xie al., 2023 Isola al. 2017 Zhu et al., (2177). Consequently, the central challenge in counterfactual generation is to learn the underlying disentangled representations.
To achieve this goal, prior work leverages either paired data that only differ in style components Rao and Tetreault 2018 [Shang et al. 2019 Xu et al.
2019b Wang al. 2019b), utilizes style labeling
information John et al., 2019 He et al.) 2020
Dathathri et al. 2020 Yang and Klein 2021 Liu
al. 2022 However, collecting paired data labels be labour- intensive and even infeasible in
many real world applications [Chou et al., 2022 Calderon et al. 2022 Xie et al.] 2023). This has
prompted recent work [Kong et 2022 Xie et al.
2003 delve into unsupervised identification of
latent variables by tapping into multiple domains.
To attain identifiability guarantees, a prevalent
assumption made in these works [Kong al.
2022 Xie et al. 2023 is that the content and the style
Equual Contribution. Work was done when Hanqi Yan visiting student at MBZUAI. Correspondence to: Yulan He (yulan.he @kcl.ak.kkk and Kun Zhang (kunzl @cmu.edu).
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

latent variables are independent of each other. However, this assumption is often violated in practical applications. First, the dependence between content and style variables can be highly pronounced. For example, to express a positive sentiment, words such as "lasty" and "flavor" are typically used in conjunction with food-related content. In contrast, words like "thrilling" are more commonly used with movie-related content [Li et al., 2019. 2022]. Moreover, the dependence between content and style often varies across different distributions. For example, a particular cuisine may be highly favored locally but not well received internationally. This varying dependence between content and style variables poses a significant challenge in obtaining the identifiability guarantee. To the best of our knowledge, this issue has not been addressed in previous studies.
In this paper, we address the identification problem of the the varying dependence between content and style (see Fi To this end, we adopt a natural notion of influence sparsity inherent to range of unstructured data, including natural languages, for which the influences from the content and the style differ in their scopes. Specifically, the influence of the style variable on the text is typically sparser compared to that of the content variable, as is often localized to smaller fraction of the words (Li et al., 2018 and plays secondary role in word selection. For instance, the tense of a sentence is typically reflected in only its verbs which are affected by the sentence content information. Our contributions can be summarised as: I) We show identification guarantees for both the content and the style components, even when their interdependence varies. This approach removes the necessity for a large number of domains with specific variance properties [Kong et al.] 2022 Xie et al. 2023]_ 2) Guided by our theoretical findings, we design doMain AdapTive coun'l T erfactual gEneration model (MATTE). It does not require paired data or style annotations but allows style intervention, even across substantially different domains. 3) We validate our theoretical discoveries by demonstrating state-of-the-art performance on the unsupervised style transfer task, which demands representation disentanglement, an integral aspect of counterfactual generation.
model that takes into account
2 Related work
Label-free Style Transfer on variation autoencoders (VAEs). To perform style transfer, existing methods that use parallel or non-parallel labelled data often rely on style annotations to refine the attribute-bearing representations, although some argue that disentanglement is not necessary for style edit (Sudhakar et al.) 2019 Wang et al. 2019a Dai al. 2019]. In practice, disentangled methods typically employ adversarial objectives to ensure the content encoder remains independent of style information or leverage style discriminators refine the derived style variables (Hu et al.
2017 [Shen et al.) 2017 Li et al., 2018 Keskar et al., 2019 John et al.) 2019 SSudhaaar et al. 2019
Dathathri al. 2020 Hu et al.) 2017 Dathathii al.. 2020 Yang and Klein 2021 Liu et al. 2022
Several studies have tackled this task without style labels, Riley et al. [2020] emphasized the implicit
style connection between adjacent sentences and used T5 [Raffel al. 2020 extract the style vector for conditional generation. CPVAE Xu et al. [2020] split the latent variable into content and style variables and mapped them to a k-dimensional simplex for k-way sentiment modeling. Our work aligns more closely with CPVAE and follows VAE-based label-free disentangled learning from Rata-generation perspective Higgins et al. [2017), Kumar et al. [2018). Mathicu et al. [2019] Xu et al.
2020), Mita et al. [202T] Fan et al. 2023]..
Latent-variable identification. Representation learning serves the cornerstone for generative models, where the goal is to create representations that effectively capture underlying factors in the disentangled data- aett-generattin process. In the realm of linear generative functions, Independent component analysis (ICA) [Comon] 1994] Bell and Sejnowskil 1995) is a classical approach known for its identifiability. However, when dealing with nonlinear functions, ICA is proven unidentifiable without the inclusion of additional assumptions (Hyvärinen and Pajunen] 1999]. To tackle this prob- lem, recent work incorporated supplementary information [Hyvarinen et al. 2019 Sorrenson et al.) 2020 Hälvä and Hyvarinen| 2020 Khemakhem et al./ 2020 Kong et al. 2022]. e.g., class/domain that are twice the number of latent components, which can be unfeasible when dealing with high-dimensional representations. Another line of work [Zimmermann et al.] [022] /von Kügelgen et al., al.]] [020]]]000]]]] Locatello et al.) 2020 Gresele et al. 20199 Kong et al.) [223a b) leverages paired data (e.g., two rotated versions of the
labels. However, these approaches require a number of domains/claseee
same image) to identify the shared latent factor within the pair. The third line of work [Lachapelle]
eet al. 2022 Yang et al. 2022 Zheng et al., [022] makes sparsity assumptions on the nonlinear generating function. Although their sparsity assumption alleviates some of the explicit requirements in the previous two types of work, it may not hold for complex data distributions. For instance, in the

---

case of generating text, each topic-related latent factor may influence a large number of components. Instead, we adopt a relative sparsity assumption, where we only require the influence of one subspace to be sparser than the other. Unlike prior work (Zheng et al. 2022 each latent variable is allowed to influence non-sparse set of components, and the influence can overlap arbitrarily within each subspace. Importantly, we necessitate neither many domains/classes nor paired data as prior work mentioned above.
3 Disentangled Representation for Counterfactual Generation
In this section, we discuss the connection between counterfactual generation and the identification of the data-generating process shown in Fig
11
S
Disentangled latent representation. The data-generating pro- cess in Figure can be expressed in Equation
c~ c~ p(clu), s ~ p(s]c, u), x x= [[ s),
(I)
where the data (e.g., text) x E X' are generated by latent vari- ables z := [c, Z C Rd through smooth and invertible generating function g(.) 2 + X. The latent space comprises two subspaces: the content variable c E C C Rdd and the style variable s E S C Rd. We define с as the description of the main topic, e.g.. "We ordered the steak recommended by the waitress,", ", and comprises supplementary details connected to the primary topic, e.g., the sentiment towards the dish, as exemplified "it was delicious!" Consequently, the counterfactual generation task here is to preserve the content information с while altering the stylistic aspects represented by s.
The data generation pro- cess: The grey shading indicates the variable observed. The observed variable (i.e., text) x generated from content e and style s. Both con- tent e and style s are influenced by the domain variable u and the con- tent also influences the style. 8 is the exogenous variable of s, representing the independent information of s.
Figure I:
Content-style dependence. many real-world problems, content e can significantly impact style s. For instance, when it comes to the positive descriptions of food (content), words like "delicious" are more prevalent than terms like "effective". Intuitively, the content acts to constrain the range of vocabulary choices for style s. As a result, the counterfactually generated data should preserve the inherent relationship between e and s. We directly model this dependence as:
(2) where Is characterizes the influence from to s and the exogenous variable š accounts for the inherent randomness of s. In the running example, š can be interpreted as the randomness involved in choosing a word from the vocabulary defined by content c, encompassing words similar to "delicious"
== (s c, u),
such as "tasty", "auuty""""uumny""""" "yummy"". In contrast, prior work (Kong et al.] 2022 Xie et al. 2023 assumes the independence between e and s thus neglecting this dependence.
Challenges from multiple domains. As we outlined in Section | the ability to handle domain shift crucial for unsupervised counterfactual generation. Domain embedding u represents a specific domain and the domain distribution shift influences both the marginal distribution of content p( u)) and the dependence of content on style p(s/c, u). The change in content distribution p(c/u) across different domains u reflects the variability in the subjects of sentences across these domains. For example, it can manifest as change from discussing food in restaurant reviews to actors in movie reviews. The change in content-style dependence p(s/c, u) signifies that identical sentence subjects (i.e., content) could be associated with disparate stylistic descriptions instance, the same political question could provoke significantly different sentiments among various demographic groups. Such considerations are absent in prior work [Kong et al. [022]. Here, we learn shared model (f(cl-). (S, -)) and domain-specific embeddings u. This approach enables effective knowledge transfer across domains and manages distribution shifts efficiently. For a target domain T, which may have limited available data, we can learn u using small amount of unlabeled data x" while preserving the multi-domain information in the shared model.
different domains. For
In light of the above discussion, the essence of counterfactual generation now revolves around the task of discerning the disentangled representation (c. 5) within the data-generating process (Fig I across various domains with unlabeled data (x, u): if we can successfully identify (c,5), we could perform counterfactual reasoning by manipulating & while preserving both the content information and the content-style dependence.
3

---

4 Identifiability of the Latent Variables
In this section, we introduce the identification theory for the content e and the style s sequentially and then discuss their implications for methodological development.
We introduce notations and definitions that use throughout this work. When working with matrices, we adopt the following indexing notations: for a matrix M, the i-th row is denoted as M the j-th
column is denoted as M.... and the (š.j) entry is denoted as [M] We can also use this notation to
refer to specific sets of indices within a matrix. For an index set Z C {1,.... m) x {I,.... .....,,.. n). we use I, to denote the set of column indices whose row coordinate is i, i.c., I, = {i (i.j) IJ.
and analogously I to denote the set of row indices whose column coordinate is j.
In addition, we define subspace of I" using an index set S: R3 {z E [ & S,2; 2,
0}. i..., it
consists of all vectors in IR" whose entries are zero for all indices not in S. Finally, we can define the support of matrix- function M(x) X - IR mx as the set of indices whose corresponding entries are nonzero for some input value x, i.e., Supp(M)) == [i.,)) Ex E X' s.t., [M(x)]s, # 0}.
4.1 Influence Sparsity for Content Identification
Sparser
We show that the subspace can be identified. That is, mate a generative model (pa. Ps e: 9) following the data-generating process in Equation [ and the estimated variable č can capture all the information of с without interference from S. In the following, we denote the Jacobian matrices Jg (z)'s and J (2)'s supports and & respectively. Further, we denote as set of matrices with the same support that of the matrix-valued function J (c)J Assumption 1 (Content identification).
esti-
X1
X2
X3
X4
Xs
9
C2
partially intersecting
Figure 2: Sparsity in Jg.
i. g is smooth and invertible and its inverse g-1 is also smooth.
it. For all E (1....,dz) dz). there exist ) and T E T, such that
(Iz(z()))
= Rye. and (2( E R
9.." iii. For every pair (cj.: with Je E [dd] and js E {de + 1....,dd},, d.}. the influence of sj. sparser than that of Cjo ie., 119 llo 119.g. llo
Theorem 1. We assume the data data-generating process in Equation 7 with Assumption 1 If for given dimensions (de, de), a generative model (pe Pajas 9) follows the same generating process and achieves the following objective:
arg min
Cniall Pe.Pa.8 J. (L
Σ
subject to: px(x) = Px(x), Vx E
then the estimated variable è is an one-to-one mapping of the true variable c. That is, there exists an invertible function h.).)) such that he(c).
proof can be found in Appendix A.5
Interpretation. Theorem 1 states that by matching the marginal distribution Px (x) under a sparsity constraint of & subspace, warrants that the content information can be fully retained without being entangled with the style information for successful counterfactual generation. We can further identify š from that of s when the dependence function is invertible in its argument 8 [Kong et al. 2022]
successfully eliminate the influence of s from the estimated é. This
ensures that the information of all latent variables (c. s] is preserved in the observed variables x, which is a necessary condition for latent-variable identification [Hyvarinen al. 2019 Kong et al. 2022]. J Assumption I Ti.. ensures that the influence from the latent variable varies sufficiently over its domain. This excludes degenerate cases where the Jacobian matrix is partially constant, and thus, its support fails to faithfully capture the influence between latent variables and the observed variables. Assumption M.. encodes the observation that the s subspace exerts a relatively sparser influence on the observed data than the subspace e (Fig 2).
Discussion on assumptions. Assumption
2 As our theory is not contingent on the availability of multiple domains, we drop the domain index u in our notations in this section for ease of exposition.

---

This is reasonable for language, where the main event largely predominant the sentence and the stylistic variable play complementary and local information for particular attributes. e.g.. tense, sentiment, and formality [Xu et al. 2019a Wang et al. [021 Ross et al. 2021]. For the language data, x corresponds to piece of text (e.g., sentence) with its dimension dx equal to the number of words multiplied by the word embedding dimension, i.e., multiple dimensions of x correspond to single word. Therefore, even if a word is simultancously influenced by both с and s, the influence from the content tends to be denser this word's embedding dimension, content usually takes precedence in word selection over style.
Contrast with prior work. Zheng al. 2022 impose sparse influence constraints on the generating function in an absolute sense - each latent component should have a very sparse influence on the observed data. In contrast, Theorem only calls for relative sparsity between two subspaces where each latent component's influence may not be sparse and unique as in Zheng et al. 2022]
We believe this is reasonable for many real-world applications like languages, Kong al. 2022
assume the independence between the two subspaces and identify the content subspace by resorting to its invariance and sufficient variability of the style subspace over multiple domains. However, as discussed in Section 3] the invariance of the content subspace is often violated, and so is the independence assumption. In contrast, we permit the content subspace to vary over domains and allow for the dependence between the two subspaces.
4.2 Partially Intersecting Influences for Style Identification
In this section, show the identifiability for the style subspace s, under one additional condition: the influences from the two subspaces с and s do not fully overlap.
Assumption (Partially intersecting influence supports). For every pair (c,,, sj.),, the supports of their infiuences x do not fully intersect, i.e., [[G.j. n G.J, llo min 11G. ll
Theorem 2.
We follow the tetta-generating process Equation I Hand Assumption and Assumption We optimize the objective function in Equation 3 logether with
Σ
| n n
min
The estimated style variable à is a one-to-one mapping the true variable That is, there exists an invertible mapping h.(-) between s and 8, i.e., $ = hs(s).
The proof can be found in Appendix A.6
Interpretation. Theorem 2 states that we can recover the style subspace if the influences from the two subspaces do not interfere with each other (Fig(2) This condition endows the subspaces distinguishing footprints and thus forbids the content information in с from contaminating the estimated style variable S. The identification of is crucial to counterfactual generation tasks: if the estimated style variable á does capture all the true style variable s, intervening on á cannot fully alter the original style that is intended to be changed.
Discussion on assumptions. Assumption prrescribes that each content component Cj. and each style component $j, do not fully contain each other's influence support. Together with Theorem I this assumption is essential to the identification of s, without which 8j. may absorb the influence from Cj.- Assumption 2 does not demand the supports of the entire subspaces and to be partially intersecting or even disjoint, and the latter directly implies Assumption 2 This assumption is plausible for many real-world data distributions, especially for unstructured data like languages and images certain dimensions in the pixels and word embeddings may reflect the information of either the content or the style.
Contrast with prior work. Kong al. [0222 obtains the identifiability of the style subspace by exploiting the access to multiple domains over which the marginal distribution of (i.e., p(s) uu) varies substantially over domains This hinges on the independence between the two subspaces and is not applicable when the marginal distribution of s only varies over the content c, i.c., p(s|c, u).
5 A Framework for Unsupervised Counterfactual Generation
In this section, we translate the theoretical insights outlined in Section into an unsupervised counterfactual generation framework. Guided by the theory, we can approximate the underlying data-generating process depicted in Fig and recover the disentangled latent components.
5

---

Figure 3: Our VAE--based framework MATTE. During training, input x is fed to the encoder fex derive latent variable & = [c. s). which is then passed to the decoder Sds for reconstruction. Flow modules, denoted as re and r., implemented to model the causal influences respectively, which yields the creation of exogenous variables and To generate transferred data Xuunserrs we intervene on the style exogenous variable š while keeping the original content variable unchanged (indicated by the green arrows).
x
Viransfer
(0.))
с and
5
S- ((,1))
In the following, we will describe each module in our VAE-based estimation framework (Fig 3). the learning objective, and the procedure for counterfactual generation.
5.1 VAE-based Estimation Framework
Given imput sentences x from various domains, we use the VAE encoder fene to parameterize the
posterior distribution ạ.., (z|x) and sample ff (z|x). The posterior sample is then fed into
the VAE decoder Iöze for reconstruction Xxc0 = Idce Z) as In conventional VAE training.
We split z into two components: с and s. As shown in Fig | both and s encompass information of particular domain u, and is also influenced by We parameterize such influences using flow-based models [Dolatabadi et al. 2020 Durkan et al. 2019] and r,, respectively:
Č= rereCc; u), 5 (s;
(5)
where è and & are exogenous variables that are independent of each other, and u and (u, c) act as contextual information for the flow models r=(:; u) and r=(:; c). This design promotes parameter sharing across domains, as we only need to learn a domain embedding u (c.f., a separate flow model per domain). As part of the
(ELBO) objective in VAE, regularize the distributions of = [ to align with the prior p(z) using Kullback-Leebllrr (KL. divergence. Consequently, the VAE learning objective can be expressed as:
soaddenc-lower-booee
LVAE log P Bioe (Xeee) + KL(ggaa.re..r (2xx)) (Z)))
(6)
where the prior p(ū) is set to standard Gaussian distribution, I), consistent with typical VAE implementations.
5.2 Sparsity Regularization for Identification Guarantees
Guided by the insights from Theorem [ and Theorem 2 the sparsity constraint on the influence of (i.e., Equation 3) and the partially intersecting influence constraint (i.c., Equation 4 are crucial to faithfully recover and disentangle the latent representations and
Sparsity of the style influence. To implement Equation 3 for the decoder function on-the-fly and apply & regularization to the columns corresponding to the style variable [J s.. (2)):dd++1dd to control sparsity. That is, Lipursity [[[J sam. (x)]:d.+1:dd I|
we compute the Jacobian matrix J Jus (z)
Partially intersecting influences. To encourage sparsity in the intersection of influence between and (as defined in Equation 44), we select K output dimensions I, of J au (z) that capture the most substantial influence from s and another set of K output dimensions I. that receive the least influence from c. Subsequently, we apply & regularization to the influence from e on the output dimensions at the intersection I, n. i.c., Cpartill = ] ae:
Il
Content variable masking. In practice, the content di dimensionality de is design choice. When d, is set excessively large, the sparsity regularization term Lapursity may cause the style variable s to lose its influence, squeezing the information of into the content variable To handle this issue, apply trainable soft mask that operates on c to dynamically control its dimensionality.
In sum, the overall training objective is as follows:
L = LVAE + Asparstyy Capanity + Apartial Cpartial + Ae-mask Le-mask.
where A's are weight parameters to balance various loss terms.
For the sake of simplicity, in this section, discuss estimated variables without the 2 notation, in

---

5.3 Style Intervention
As discussed in Section 3 the content-style dependence should be preserved when generating counterfactual text ensure linguistic consistency. This can be achieved by intervening on the exogenous style variable š of the original sample. Specifically, we feed the original sample x to the encoder fene to obtain variables [c,s]. Subsequently, we pass the style variable s through the flow models r, to obtain its exogenous counterpart i.e., = rs (s: c, u1. To carry out style transfer, we set the original variable á to the desired style value Surasttrr which is the average of the exogenous style values of randomly selected samples with the desired style. As the flow model r. is invertible, we can obtain the transferred style variable Stranser = rr (Stransferi C, u), which, together with the original content variable c, generates the new sample Xtransfer Sdder Stranfer]). This process is illustrated in Fig BigBur using green arrows. We demonstrate the importance preserving the content-style dependence and provide evidence that our approach can indeed fulfill this purpose (Fig 4
Experimental Results
We validate our theoretical findings by conducting experiments multiple- domainn sentiment transfer tasks, which require effective disentanglement of factors, a concept at the core of our identifiability theory.
Datasets and Evaluation Schema. The proposed method is trained four-domain datasets (Tab I]. i.e. movie
Table 1: Dataset on four domains.
(Imdb) [Diao et al. a 2014], restaurant (Yelp) [Li et al. 22018] e-
Domains
Train
Dev
Test
commerce (Amazon) [i et al. 2018) and news (Yahoo) (Zhang]
IMDB
344.175
27,530
27,530
Yelp
444,102
63,484
1000
ct al.
2015 Li et al. 2019
From common practice Yang 2018 Lample al. 2019]. we evaluate the generated
Amazon
554,998
2,000
1.000
et al.
Yahoo
4,000
4,000
4,000
sentences in terms of the four automatic metrics: (1) (e)uurccy We train CNN classifier
the original style-labelled dataset, which has over 95.0% accuracy when evaluated on the four separate validation datasets. Subsequently, we employ it to evaluate the transformed sentences, gauging how effectively they convey the intended attributes. (2) BLEU (Pa pineni et al.) 2002]. It compares the n-grams in the generated text with those in the original text, measuring how well the original content is retained [ (3) G-score. It represents the geometric mean of the predicted probability for the ground-truth style category and the BLEU score. Due to its comprehensive nature, it is our primary metric. (4) Fluency. It is the perplexity score of
GPT-2 [Radford et al. 2019] lower perplexity values indicate a higher levels of fluency. For human evaluation, we invited three evaluators proficient in English to rate the sentiment reverse, semantic preservation, fluency and overall transfer quality using 5-point Likert scale, where higher scores signify better performance. Furthermore, they were asked to rank the generated sentences produced from different models, with the option to include tied items in their ranking.
6.1 Sentiment transfer
Baselines, We compare our model with the state-of-the-arg art text transfer models that do not rely on style labels, along with a supervised model, B-GST (Sudhakar et al. 2019]. which is based on GPT2 [Radford et al. 2019] and accomplishes style transfer through a combination of deletion, retrieval, and generation. The other VAE-based baselines can be divided into two groups based on their architecture: those with LSTM backbones and those utilizing pretrained language models (PLM). Within the LSTM group, 3-VAE (Higgins al. [0077] encourages disentanglement by progressively increasing the latent code capacity. JointTrain [i et al. 2022) uses the Glo Ve embedding to initialize and learns through LSTM. CPVAE (Xu al. 2020 is the state-of-the-art unsupervised
style transfer model, which maps the style variable to A-dimensional probability simplex to model different style categories. the PLM group, we use GPT2 [Radford al.
2019] as the backbone and
introduce an additional variational layer after fine-tuning its embedding layer to generate the latent variable z, referred to as GPT2-FT. Also, we consider Optimus (Li et al.
2020 which is one of the
most widely- pretrained VAE models, utilizing BERT (Devlin et al. GPT2 as the decoder.
2019 as the encoder and
'Dataset and detailed experiment configurations can be found in Appendix. A.1
'e also adopt CTC score [Deng et al. 2021). to mitigate potential issues brought by the word-overlay measurements in BLEU as considers the matching embeddings. The evaluation results are shown in Table
7

---

IMDB
Yelp
Medel
Aec(")
BLBUNT
G-score(1)
PPL(())
Aceth'
BLEEU11>
G-scwer()) ()
PPE(()
B-0ST
Sadhakar et al
2019
36,20 10.80
50.45 12.62
32.09 11.81
48.58 -2.08
$2.00
10:20
32,06
11.34
35.43 10.92
50.45 1.338
PLM LSTM
Eiggans
38.27 11.03
11.37 13.03
9.05 11119
43.59 -3.07
40.30 10.92
7.58 1:.773
11.08
59.34
Joinns
2023
24.13 1:052
23.26 1:2.55
12.28::1.01
70.11
1011:22.0 76
14.20
10.62
31,72
1:191
12.74 10.96
84,07 1:237
CPVVE
Xu
2020
20.15 10.40
49.82 15.25
20.01
10.96
70.18
-2.78
14.50 10.30
51,47 11.81
16.81 10:92
72.81 1.226
GF12 FT
Kadlord et
2019
15.20 1..25
25.93
13.16
12.19
2:2.84
71.08
=22
12.00
10.42
39.6241.52
14.69 21.35
78.37 2:214
Optimuu
14:07
1.0.0
1:168
17.47
1:1.2
61.90
22.61
13.60
10.30
21.24
52.56
1:201
NATTE
32.43 10.28
45.10
12.91
25.92
11:82
50,08
22.00
34,30
10:25
50.14
12.51
26.34 11.37
51,51
12.00
Amazon
Yahoo
Model
Aec(T)
BLEUNT)
G-score(1)
PPL())
Acett)
BLEUR))
G-score(")
PPL())
B-03T Sudhakkr
2019
60.45
56.02 12.36
47.67
49.01 13.18
$4.30 10.00
40,39 12.81
38.65
58,20
LSTM
Higgins
2017
50.08.0000
8:04
12.62
9.39
11.24
33,09 22:50
55.47 10.40
3.77 41.32
5.85 41.71
52.17 23.00
Doin:Troin d
A022
32,90 =:1.02
25:21 12.16
18.33
#:1077
84.63#2.70 76
35.33 2:0.25
14.04
#1.72
11,62:0922
67.34 #2.84
CPVA Xu et
2020
32,60 1.020
41.08 11.28
30.06 11.15
77.61
12
43.92 10.30
25,44 11.37
20.28 10.95
76.28 12.67
QP12 FT
Radford et
30,46 1:0.30
40.34112.82
26.721:1.50
79.36.22.63
17.90 10.40
44.19
15.99
70.99 12337
Optiimun ei al.
2000
24,50 1.0.20 20
62:50 1:55
28.53
1:1
74.66
:3 10
27.10 10.15
32.73
1.
19.17
3.1.6 60
73,18 1:270
MATTE
34,50 +0.24
52.25 +1.48
35.73
+1.14
63.37
22
58.45 +0.20
42,40
+1.35
29.01
#2.30
56.12 #2.67
Table 2: Comparison with unsupervised methods across four domain datascts with supervised B-GST as an upper bound reference. The lop and the second- results are in bold.
Quantitative performance. Among LSTM baselines in Table 2 3-VAE shows high sentiment transfer accuracy and fluency but poor content preservation. We observed that many generated sentences follow simple but repetitive patterns, e.g., 2.2% transferred sentences in Yelp containing the phrase "I highly recommend" while only 0.6% original sentences do. They are fluent and correctly sentiment flipped but their semantics are significantly different from the original sentences, indicating a generation degradation problem 6 CPVAE achieves an overall better G-Score than all the other baselines across three domains (except for Yelp). Compared with the other LTSM-basee methods, its superiority in content preservation is pronounced. PLMs models achieve overall better BLEU scores compared with the LSTM group. Optimus outperforms GPT2-FT, which can be partly explained by the fact that the variational layer in Optimus has been pretrained on 1,990K Wikipedia sentences. Our model is built on top of CPVAE with the proposed causal influence modules and sparsity regularisations. It gains consistent improvements in G-score and fluency across all datasets over all the other unsupervised methods. Compared with the supervised method, despite a relatively large gap in accuracy due to the lack of supervision, our approach achieves comparable BLEU scores. The human evaluation results in Table 3 show that human annotators favour Op t imus in terms of content preservation and fluency, but MATTE ranks the best cesttperformin method with 58.5% support set, compared 41.00% for Op t timss.
Style Content Fluency
Best rank(5k)
See1:
This guy an awful actor. X
CPVAE
The guy is very flavorfial. V
CPVAE
1.30
2.78
3,20
21.50
Optinus
The guy is an amazing actor /
Optimua 1.41
3.79
4.12
41.00
MATTE
This guy umazing actor! /
MATTE
1.99
2.91
3.53
58.50
Sre2: had long time now still love /
Table 3: Human evaluation results from three annotators. The Cohen's Kappa co- efficient among every two annotators over the Best-rank is 0.46. Human annotators favour Opt in terms of content preser- vation and fluency, but MATTE ranks the best- best-performing method with more than 58% support set after considering the style reverse success rate.
CPVAE
Tad it long time before and I've never caten it.
Optimus
It is a long time now and I always get this food. /
MATTE
bad it long time now and I never played it X
Sec3 These come in handy with those tender special moments.
CPVAE
These come in hundy with those sexy care employees.
Optinue
These are fateless in their only safe place.
MATTE
These come in hundy with those poorly executed characteristies. X
Table 4: Generated style-transferred Sentences. V. X represent
sentiment polarity.
Qualitative results. We randomly selected three sentences from the test sets and analyzed the results generated by the top- oop-erforming baselines in Table For Src 1, although all the methods successfully transfer the original sentiment of the sentence, CPVAE generates the word 'flavourful' for the content guy, resulting in an unnatural sentence. This issue arises because CPVAE fails to identify the domain-specipp content-style dependency, i.e., the Sre domain is in IMDB, while the transferred sentence incorrectly uses the style word 'flavourful' which is commonly used in Yelp. While Optimus can generate relatively fluent sentences partly due to its powerful decoder, it hardly
'The diversity-n
Li et al. 2016 also indicates repetitious pattern and the evaluation results are in Table 10
8

---

IMDB
Yelp
Model
Aco(t)
BLEU(1)
G-scoro(t)
PPL())
Acc(")
BLEU(")
G-score(f)
PPL())
Backbone
20.15
49.82
20.01
70.18
14.50
51.47
16.84
72.81
Indop Kong al. 2022
45.00*
30.88
19.89
61.85*
61.90*
21.24*
73.78
CausalDap
28.71
39.63
21.85*
53.25*
22.10*
48.98
25.98*
55. 14*
19.
Capuriity
21.55
56.59
20.90
65.26
13.20
56.26
14,59
54,10
30.18
31.95
25.57A
54.66
33.70
49.09
25.81
$2.874
1w. Ceveu (Pall)
32.43
45.10
25.92
50.08
34.30
50.14
26.34
51.51
Table 5: Ablation results on sentiment transfer on two domains. CausalDep incorporates style flow Ta to model dependency of с on while Indep assumes the independence between the two variables. marks the improvements overBackbone, while over the CausalDep.
maintains the original semantics (Src 2, 3), indicating lack of effective disentanglement between and These failure modes demonstrate the importance of a proper disentanglement of content and style and modelling the causal influence between the two across domains. Benefiting from theoretical insights, our approach manages to reflect the content influence across different domains in Src and retain the content information in Src2, 3.
6.2 Ablation Study
Ablation studies in Table 5 are used to ver- ify our theoretical results in $ 4 On top of the backhone, CPVAE, we incrementally add each component of our method: (1)Indep con- siders the domain influence on (i.e., the re module in Fig 3). while neglecting the inde- pendence between e and s. It experiences large accuracy boost in conjunction with a significant degradation in BLEU, suggesting poor retention of the content information. (2) CausalDep takes into account the dependency between content and style by incorporating the module r' in Fig 3 This ameliorates the content retention problem and strikes an over- all better balance, as reflected by the raised BLEU score and G-score, although causal de- pendence in CausalDep is not sufficient for identification without proper regularization. (3) After introducing the style sparsity regu- larization Espasity specified in Theorem we observe a significant increase of BLE over CausalDep, verifying Theorem that
Number of Samples
Original Fipped
30
Original Flpped
40-
10
500
1000
500
1000
1500
of Lacent Variatres
NLL of Latent Variabies
This one does the job I need for
This one does the job bbought it for sameane else on.
This one coes the job ineec for kids ?
Overalli love them, , and would probably buy them again.
Since I love them, and everything went through fine.
Overal cidn't want but would absolutely buy something else
Thank
operation intuitive
are no instructions..
They're te very durable and carries an äfter every minute.
Thank you amezon very fast es they dearly makes opening process
Figure 4: Histograms of negative log-likelihood (NLL) of 1000 Amazon test samples evaluated the original latent variable and intervened ones. left: flips s. right flips s. The table shows the corresponding sentences.
the style influence sparsity facilitates content identification $4.1 1 (4) We further introduce Lpartial inspired by Theorem [ which controls the intersection of content and style influence supports. This improvement in style identification, i.e., the recovery of accuracy over Lsparsity corroborates Theorem 2 (5) The incorporation of Le-maax arrive at our full model, which further improves the style identification, consistent with our motivation in $ 5.3 It also exhibits the best G-score across all the datasets, with the most predominant improvement over CausalDep on the Yahoo dataset, where the G-score increases from 21.39% to 29.01 %.
The importance of content-style dependence. We demonstrate the importance of content-style dependence by visualizing the changes in negative log-likelihood (NLL) induced by different ways of style intervention, namely flipping in our method and flipping which breach the content-style dependence. If the NLL increases after the style transfer, indicates that the new variables are located in lower density region [Zheng al., 2022 Xu et al., 2020). . Thowss the histograms of NLLs for all the Amazon test samples, both before and after style transfer. We can see that the
NLL distribution changes negligibly when flip s, in contrast with the significant change caused by flipping s. This implies that flipping å enables better preservation of the joint distribution of the
The results Amazon and Yahoo in Appendix, Table

---

original sentence. The generated sentences resulting from flipping & exhibit a higher level of semantic fidelity the original sentence, with a clear inverse sentiment.
6.3 Comparison with large language model
As widely recognized, large language models (LLMs) have demonstrated an impressive capability in text generation. However, we consider the principles of counterfactual generation to be complemen- tary the development of LLMs. We aim leverage our theoretical insights further enhance the capabilities of LLMs. We provide examples in Table 6 where LLMs struggle with sentiment transfer, primarily due their tendency to overlook the broader and implicit sentiments while accurately altering invidivual sentiment words. Consequently, is reasonable to anticipate that LLMs could benefit from the principles of representation learning, as developed in our work.
Sre: The buttons extend the arms worked exactly one time before breaking.
ChaGPPTpp:: The buttons extend arms failed work from the beginning, never functioning even once. CharGPT-p2: The buttons to extend the arms never worked, even once, and remained functional until they broke.
Our: The buttons extend the worked exactly described.
Sre: love that it uses natural ingredients but it was ineffective on my skin.
CharGPT-pI: I dislike that it uses natural ingredients, but it was highly effective on skin.
CharGPT.p2: dislike that
natural in ingredients, but it was highly effective my skin.
Our: like that it uses natural ingredients, and it was also good.
Sre: This case is cute however this is the only good thing about it. cute; however, is the only good thing about
CharGPT-pl: This case
CharGPT.p2: This case is not cute at all; however, the only bad thing about
Ours: This case is cute and overall a valuable product.
Table 6: A Sentiment transfer example, on which ChatGPT fails to completely reverse the overall sentiment of the sentence, although successfully negates individual words within text. In contrast, our method achieves the sentiment reversal with minimal changes. CharGPT-pl and CharGPT-p2 represent results obtained from two different prompts, i.c. pl: "Flip the sentiment of the following sentences, but keep the content unchanged as much as possible."; "; p2: "Please invert the sentiment while preserving content as much as possible in the following sentence that originates from the original domain.".
6.4 Visualization of style variable
We further validate our theoretical insights within additional content-style disentanglement scenarios. As tense has rela- tively sparse influence on sentences compared to their content, choose tense (past and present) as another style for illus- tration. Specifically, we collect 1000 sentences in either past or present tense from the Yelp Dev set and derive their style representations, denoted as s. by feeding these sentences into our well-trained model. The projection results of CPVAE and MATTE are shown i5 The distinct separation between the red and blue data points indicates a more discriminative and better disentangled style variabl. However, in the case of CPVAE, some blue data points are mixed within the lower portion of the red region.
Figure 5: The style variables of sen- tences in past-tense (blue) and present- tense (red) following a UMAP projec- tion. Left: CPVAE; Right: MATTE.
7 Conclusion and limitations
Prior work (Kong et al. 2022 Xie et al. 2023 have employed multiple domains achieve unsuper- vised representation disentanglement However, the assumed independence between the content and style variables often does not hold in real-world data distributions, particularly in natural languages. To tackle this challenge, we address the identification problem in latent-variable models by leveraging the sparsity structure in the data-gener process. This approach provides identifiability guarantees for both the content and the style variables. We have implemented controllable text generation method based on these theoretical guarantees. Our method outperforms existing methods on various large-scale style transfer benchmark datasets, thus validating our theory. It is important to note that while our method shows promising empirical results for natural languages, the sparsity assumption (Assumption] I ji..) may not hold for certain data distributions like images, where the style compo- nent could exert dense influences on pixel values. In such cases, we may explore other forms of inherent sparsity in the given distribution, e.g., sparse dependencies between content and style or sparse changes over multiple domains, to achieve identifiability guarantees and develop empirical approaches accordingly.
10

---

Acknowledgements
We thank anonymous reviewers for their constructive feedback. This work was funded by the UK Engineering and Physical Sciences Research Council (grant no. EP/T017112/1, EP/T0171 12/2.. EP/V048597/1, EP/X019063/1). YH is supported by a Turing AI Fellowship funded by the UK Research and Innovation (grant no. EP/V020579/1, EP/V020579/2), The work of LK and YC is supported part by NSF under the grants CCF-1900 199 and DMS-2134080. This project is also partially supported by NSF Grant 2229881, 1, the National Institutes of Health (NIH) under Contract R01HL159805, a grant from Apple Inc., a grant from KDDI Research Inc., and generous gifts from Salesforce Inc., Microsoft Research, and Amazon Research.
References
A. J. Bell and T. J. Sejnowski. An maximtiiiiimaiiiiitttiio approach to blind separation and blind deconvolution. Neural computation, 7(6): 1959, 1995.
N. Calderon, E. Ben-David, A. Feder, and R. Reichart. DoCoGen: Domain counterfactual generation for low resource domain adaptation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 7727-7746, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi:
URL
htttps://aclanthologs www.oorg/.......oo acl aalaacllacc acl-log.. 533
Y.-.. Chou, C. Moreira, P. Bruza, C. Ouyang, and Jorge. Counterfactuals and causability in explainable artificial intelligence: Theory, algorithms, and applications. Information Fusion, 81: 59-83, 2022.
P. Comon. Independent component analysis, a new concept? Signal processing, 36(3):287-314, 1994.
N. Dai, J. Liang, X. Qiu, and X. Huang. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5997-6007, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.11866// 1/199-161 URL htttt::::scaaaathhhoaa
org/
P19-1600 1601.
S. Dathathri, A. Madotto, J. Lan, J. Hung, E. Frank, P. Molino, J. Yosinski, and R. Liu. Plug and play language models: A simple approach controlled text generation. In International Conference on Learning Representations, 2020. URL /ttps:///oenrreviee wwwpeeerrrveieeeeeeoeeeeeeeeeeee wtw.:eeeeeeetttonndddddiei..eeee 1d-H1edEyBKDS
M. Deng, B. Tan, Z. Liu, E. Xing, and Z. Hu. Compression. transduction, and creation: A unified framework for evaluating natural language generation. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pages 7580-7605, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10. 18653/v11 2021 www.eeemnlaaaaa main.599. URL https: http///aaaaatttotoeo org 2021 emnlp-main. 599
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186. Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/N19- 1423. URL htttss://aclnthhlle www.oogyyorrr.ooo N19-1423
Q. Diao, M. Qiu, C.-Y. Wu, J. Smola, J. Jiang, and C. Wang. Jointly modeling aspects, ratings and sentiments for movie recommendation (jmars). In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 193-202, 2014.
H. M. Dolatabadi, S. M. Erfani, and C. Leckie. Invertible generative modeling using linear rational splines. In Chiappa and R. Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings Machine Learning Research, pages 4236-4246. PMLR, 2020. URL http::/proceedingg mlr, http./prree/ree////oaaaaaaaaaad. html
11

---

C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios. Neural spline flows. Advances in neural information processing systems. 32, 2019.
D. Fan, Y. Hou, and C. Gao. Cf-vae: Causal disentangled representation learning with vae and causal flows. arXiv preprint arXiv:2304.09010, 2023.
L. Gresele, P. K. Rubenstein, A. Mehrjou, F. Locatello, and B. Schölkopf. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ica, 2019.
H. Hälvä and A. Hyvarinen. Hidden markov nonlinear ica: Unsupervised learning from nonstationary time series. Conference on Uncertainty Artificial Intelligence, pages 939-948. PMLR, 2020.
J. He, X. Wang, G. Neubig, and T. Berg-Kirkpattrinn A probabilistic formulation of unsupervised text style transfer. CoRR, abs/2002.03912, 2020. URL https://aaxiiv http:s//aarrrv.oorrr/a////.....0 03912
I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerch- ner. VAE: Learning basic visual concepts with constrained variational framework. In International Conference on Learning Representations, 2017. URL https:://pperreriee. net/ foorm?id=SSyff2Ugg]
Z. Hu, Z. Yang. X. Liang, R. Salakhutdinov, and E. P. Xing. Toward controlled generation of text. In International conference on machine learning, pages 1587-1596. PMLR, 2017.
C.WW. Huang, D. Krueger, A. Lacoste, and A. Courville. Neural autoregressive flows. In International Conference on Machine Learning, pages 2078-2087. PMLR, 2018.
A. Hyvärinen and P. Pajunen. Nonlinear independent component analysis: Existence and uniqueness results. Neural networks, 12(3):429-4335 1999.
A. Hyvarinen, H. Sasaki, and R. Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages $59-868. PMLR, 2019.
Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
V. John, I.. Mou, H. Bahuleyan, and O. Vechtomova. Disentangled representation learning for non-parallel text style transfer. In A. Korhonen, D. R. Traum, and L.. Mårquez, edi- tors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 424- 434. Association for Computational Linguistics, 2019. doi: 10.18653/11/119-1441 URL https://doi. org/10. 10. 11653/11//199-1101
N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and Socher. CTRL: A conditional transformer language model for controllable generation. CoRR, abs/1909.05858, 2019. URL http: org/abs 05858
I. Khemakhem, D. Kingma, R. Monti, and A. Hyvarinen. Variational autoencoders and nonlinear ica: unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217 PMLR, 2020.
L. Kong, Xie, W. Yao, Y. Zheng, Chen, P. Stojanov, V. Akinwande, and K. Zhang. Partial disentanglement for domain adaptation. In International Conference on Machine Learning, pages 11455-11472, PMLR, 2022.
L. Kong, B. Huang, F. Xie, E. Xing, Y. Chi, and K. Zhang. Identification of nonlinear latent hierarchical models, 2023a.
L. Kong, M. Q. Ma, G. Chen, E. P. Xing, Y. Chi, L.PP. Morency, and K. Zhang. Understanding masked autoencoders via hierarchical latent variable models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7918-7928, June 2023b.
12

---

A. Kumar, P. Sattigeri, and A. Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. 6th International Conference on Learning Representations. ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.ner, 2018. URL (tRlssttss:////tt htttps://openreviier net forfforrmmmdd-1127722
S. Lachapelle, P. Rodriguez, Y. Sharma, K. E. Everett, R. Le Priol, A. Lacoste, and S. Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. In Conference on Causal Learning and Reasoning, pages 428-484. PMLR, 2022.
G. Lample, S. Subramanian, E. Smith, L. Denoyer, M. Ranzato, and Y.-.. Boureau. Multiple-attriuute text rewriting. In International Conference on Learning Representations, 2019.
C. Li, X. Gao, Y. Li, B. Peng, X. Li, Y. Zhang, and J. Gao. Optimus: Organizing sentences via pre-trained modeling of latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages (678-4699, Online, Nov. 2020. Association for Computational Linguistics. doi: 10. 10.18653/v http:///l22220mmmmm// main.378. URL ccttss:/aclanttollle
0rg/200......000 emnlp poo-main.333 378
D. Li, Y. Zhang, Z. Gan, Y. Cheng, C. Brockett, B. Dolan, and M. Sun. Domain adaptive text style transfer. In K. Inui, J. Jiang, V. Ng. and X. Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNNP--JCNIPP 2019, Hong Kong, China, November 3-7, 2019, pages 1302-3311. Association for Computational Linguistics, 2019. doi: 10. 10.18653/w1//D1--1335 URL htttts:///o/lo org/10..1110 18653/v1/D19 1325.
J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan A for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110-119, San Diego, California, June 2016. Association for Computational Linguistics. doi: 10.18653/v I/N1 1014. URL
objective function
Nrg/N16-0014
J. Li, R. Jia, H. He, and P. Liang. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In M. A. Walker, H. Ji, and A. Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 1865-1874. Association for Computational Linguistics, 2018. doi: 10.18653/ v1/n18-1169, URL http::t/ps:/ooo http:///ooooor///////// 1ee111..1666/// 1at/3//vv/n188111
X. Li, X. Long, Y. Xia, and S. Li. Low resource style transfer via domain adaptive meta learning. In M. Carpuat, M. de Marneffe, and I. V. M. Ruíz, editors, Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 3014-3026. Association for Computational Linguisties, 2022. doi: 10. 122655//110222222aallmm. 10. 10w53181611v//
URL
https://doi. https://doi.orr//0.
nww.aaaaa--aaaaaa maaacclmain..... 220
G. Liu, Z. Feng, Y. Gao, Z. Yang. X. Liang, J. Bao, X. He, S. Cui, Z. Li, and Z. Hu. Composable text controls in latent space with odes, 2022.
F. Locatello, B. Poole, G. Rätsch B. Schölkopf, O. Bachem, and M. Tschannen. Weakly-supervised disentanglement without compromises. In International Conference on Machine Learning, pages 6348 6359. PMLR, 2020.
E. Mathieu, T. Rainforth, N. Siddharth, and Y. W. Teh. Disentangling disentanglement in variational autoencoders. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California. USA, volume 97 of Proceedings of Machine Learning Research, pages 4402-4412. PMLR, 2019. URL http://prroceedinge mlr.
atresss/97/mathheuul.2 htt////tthhhee.... html
G. Mita, M. Filippone, and P. Michiardi. An identifiable double vae for disentangled representations. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 7769-7779. PMLR, 18 (8-24JJ Jul 2021. URL httph::t://// https://proceedinggg/ mlr greess/v139 mi ta21a. html
13

---

K. Papineni, S. Roukos, Т. Ward, and W. Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia. PA, USA, pages 311-318. ACI., 2002. doi: 10.3115/ 0173083.1073133 URL atttps://cclattbllooy.orr/// wwwp:ooyyyrrr/2221/1/00
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555 5551, 2020.
S. Rao and J. R. Tetreault. Dear sir madam, may i introduce the gyafe dataset: Corpus, bench- marks and metrics for formality style transfer. In North American Chapter of the Association for Computational Linguistics, 2018.
P. Riley, N. Constant, M. Guo, G. Kumar, D. C. Uthus, and Z. Parekh. Textsettr: Label-free text style extraction and tunable targeted restyling. CoRR, abs/2010.03802, 2020. URL https //arxiv org/ abs abs/2010. 20100 03802
A. Ross, T. Wu, H. Peng, M. E. Peters, and M. Gardner. Tailor: Generating and perturbing text with semantic controls. arXiv preprint arXiv:2107.07150 2021.
M. Shang, P. Li, Z. Fu, L. Bing, D. Zhao. S. Shi, and R. Yan. Semi-supervine text style transfer: Cross projection in latent space. In Conference on Empirical Methods in Natural Language Processing. 2019.
T. Shen, T. Lei, R. Barzilay, and T. S. Jaakkola. Style transfer from non-parallel text by cross- alignment. CoRR, (bss/1705.0966 2017. URL htttp://axxiv.org//4e 1ttps//1050000555555
P. Sorrenson, C. Rother, and U. Köthe. Disentanglement by nonlinear ica with general incompressible- flow networks (gin). arXiv preprint #xXiv:2001.04872, 2020.
A. Sudhakar, B. Upadhyay, and A. Maheswaran. "transforming" delete, retrieve, generate approach for controlled text style transfer. In K. Inui, Jiang, V. Ng, and X. Wan, editors, Proceedings the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-UCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3267-3277. Association for Computational Linguistics, 2019. doi: 10.18653/ 1322. URL https: ptttps://do... org/10 18653/v1/D19 1322 1322
von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, Schölkopf. M. Besserve, and F. Locatello. Self-supervised learning with data augmentations provably isolates content from style. arXiv preprint arXiv: 100.016691 2021.
H. Wang, C. Zhou, C. Yang, H. Yang, and J. Hc. Controllable gradient item retrieval. In Proceedings of the Web Conference 2021, pages 768-777, 2021.
K. Wang, H. Hua, and X. Wan. Controllable unsupervised text attribute transfer via editing entangled latent representation. Advances in Neural Information Processing Systems, 32, 2019a. 9a.
Y. Wang, Y. Wu, L. Mou, Z. Li, and W. Chao. Harnessing pre-trained neural networks with rules for formality style transfer. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UCNLP), pages 3573-3578, 2019b.
S. Xie, L. Kong, M. Gong, and K. Zhang. Multi-domain image generation and translation with identifiability guarantees. In The Eleventh International Conference Learning Representations, 2023.
XU, J. C. K. Cheung, and Y. Cao. On variational learning of controllable representations for text without supervision. In International Conference on Machine Learning, 2019a.
14

---

P. Xu, J. C. K. Cheung, and Y. Cao. On variational learning of controllable representations for text without supervision. In Proceedings the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 1553-10544 PMLR, 2020.
mlr press/v119/
html sum0
R. Xu, T. Ge, and F. Wei. Formality style transfer with hybrid textual annotations. ArXiv. abs/1903, 10t3//933,,,,, 2019b.
K. Yang and D. Klein. FUDGE: controlled text generation with future discriminators. In K. Toutanova, A. Rumshisky, L. Zettlemoyer D. Hakkani-Tür, Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, and Y. Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6- 6-11, 2021, pages 3511-3535. Association for Computational Linguistics. 2021. doi: 10. 10..1853//1120 1/202 wtw.l//.aaannnnannnnnaaaaaaaa .naacl-main. 276. URL https: http:///aabbooooo111811/////////
naacl - main.276
X. Yang. Y. Wang, J. Sun, X. Zhang, S. Zhang, 7. Li, and J. Yan. Nonlinear ICA using volume- preserving transformations. International Conference on Learning Representations, 2022. URL https://openreviier www. htt///ooummmm/ dimimmmii-AMMi AeAddddAMMkpppp. 9kppC1
Z. Yang, Z. Hu, C. Dyer, E. P. Xing, and T. Berg- Kirkpatrick. Unsupervised text style transfer using language models as discriminators. Advances in Neural Information Processing Systems, 31, 2018.
X. Zhang, J. Zhao, and Y. LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
Y. Zheng, I. Ng. and K. Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. httpp::/pppeerree//
net/forummid Wo 1HF2wWNZb
J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle- consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel. Contrastive learning inverts the data generating process, 2022.
15

---

A
Appendix
A.1
Implementation Details
In this section, we first introduce the dataset. We then provide the network architecture details of MATTE. The hyperparameter selection criteria and the training details are summarized.
A.1.1 Dataset
We use four domain datasets to train our unsupervised model, i.e., Imdb, Yelp, Amazon and Ya- hoo, and follow the data split provided by Li et al. [2019]. The datasets can be downloaded via https://github.com /ttp
The dataset details can be found in Table [I We the sequence length L as 25, which is the 90 percentile of the sentence length of the training dataset. Therefore, shorter sentences are padded and longer sentences are clipped. The vocabulary size is set to 10000. For the sentiment transfer, we collect 100 positive sentences in dev set based on their sentiment labels to derive Strnsser flip the sentiment of the negative sentences in the test set, and vice versa. For the tense transfer, we use stanfordnlp tool R to identify the tense for the main verb, then collect 100 sentences of present tense dev set derive Stransfer to flip the past-tense sentences.
A.1.2 Model Architecture
We summarize our network architecture below and describe it in detail in Table 7
According Xu et al. [2020]. the encoder is fed with text span x(t, t, + m] extracted from the original sentence where is random word position index, is set to 10 if ( + m is smaller than L.. Woord is the word embedding dimension, set to 256. Hamm is the hidden states of LSTM, set to 1024. H₂ is the dimension of the latent variable, set to 80. The output of the encoder is the p, a and z. All of them are in shape [BS, Hz].
Encoder:
Decoder: Decoder is fed with the input sentence span and the generated latent variable. The final reconstructed sentence span is one timestamp delay compared to the input span, i.e., +1 +1) This is generated by applying beam search to the sequence output probability over the vocabulary V. The Lrecon is to calculate the cross-entropy loss between the output probability and target sequence span.
Content Flow re: We apply Deep Dense Sigmoid Flow (DDSF) [Huang et al. 2018 to derive the content noise term. To incorporate the domain information, leverage the domain embedding (after MLP) to parameterize the flow model.
Style Flow r,: We apply spline flow [Durkan et al. 2019 to derive the noise term. Similarly, we use the conditional flow with extra input. The conditional input is the combination of content variable and domain embedding. Specially, they are concatenated firstly and the result are fed into MIP with Tanh activation derive attention score The conditional input is actually the doctProdeut.
A.1.3 Training
Training details. The models were implemented in PyTorch 2.0. and Python 3,9. The VAE network trained for maximum of 25 epochs and a mini-batch size of 64 used. We use early stops if the validation reconstruction loss does not decrease for three epochs. For the encoder, we use the Adam optimizer and the learning rate of 0.001. For the decoder, we use SGD with learning rate of 0.1. For the content and style flow, we use Adam optimizer and the learning rate is 0.001. We set three different random seeds and report the average results.
Training objective. The VAE-hased model is mainly trained with freeon and LVAE- We use a training trick better jointly train the other three objectives. The Esparsity could cram the information of to c, while the Le-mask is used to prevent the ill- ill-posed situation where s have zero influence. Therefore,
https://stanforddupp wit.ub......uuu......gp..g..b..t 10/ wwt.auaooooottaaaon.......
The implementation refers to ConditionedSllnee in https:tpps.. wrpo/d/sstebbbbbooom////tttoo//n
16

---

Module
Description
Output
1. Encoder
Encoder for Input sentence
Input +m
random span of sentence get word embedding Bi-direction, 2layers sentencet-level Rep. 1s and
WordEmb
BS x m x Hword
Bi-LSTM
BS X Hism
Average Pooling
BS X Himm
ML.P
BS (2 H2)
reparameterization
Sampling
BS x H,
2. Domain Embedding
Embedding Layer
Input u
number of domain -> Udim
BS Udim
3. Content Flow Te
Input: c,u
domain as flow conditional input u - conditional context
MLP
BS x |Hre]
DDSF
get content noise term č
BS X
4. Style Flow T.
Input: c, s
content and domain as flow context combine and Tanh activation, get attention score &
Concatenate
BS X (Cdim + Udim)
MLP
BS X Cdim
Element-wise Multiplication
BS X Cdim
SplineFlow
get style noise term S
BS X Sdim
5. Decoder Ts
Input: Z, X
generate the next token
Bi-LSTM
Bi-direction, 2layers
BS x m X Hısm
MLP
output word probability
BS V
Table 7: MATTE overall architecture. DDSF is deep dense sigmoid flow, and SplineFlow is neural spline flow. the length of randomly extracted text span from input sentence x.
we involve both Esparsity and Co-mask at the beginning of the training phrase. For Cpartiat it is used to sparsify the influence intersection but their separate influences change very frequently in the initial training stages. So we involve it after 3 epochs.
Computing hardware and running time. We used machine with the following CPU specifications: AMD EPYC 7282 CPU. We use NVIDIA GeForce RTX 3090 with 24GB GPU memory. It costs approximately 190ms
model on this machine per epoch.
A.2
Additional Results
This section presents additional results on the hyperparameter sensitivity and the ablation studies on more datasets.
A.2.1 Hyperparameter Sensitivity
We discuss the effect of the three loss weights AspursityAAputtee and Ae-mask in the training objective. have performed a grid search of Xquarity [1E-4,1E-3,,1E2 Apanial [3E-5,3E-3,3E-1 and
The best configuration is (Aquristy: Apartial Ac-mask] [1E-4,3E-3, 1E-4]. The model performance relatively sensitive Aspansity so we plot the sentiment accuracy and BLEU function of Aspassiyy in Figure 6
Ac-mack E 11E-4,11E3,,111
17

---

Amazon
Yelp
Yahoo
Yelp
0.00EH00
1.00E-04
1.00E-03
1.005-02
0.00E+00
1.00E:04
1.00E 03
1,005-02
Spanity Weight
Sparsity Weight
Figure 6: The Sentiment Acc (left) and BLEU (right) with different Aspusiity Among the four datasets, sentiment generally decreases the Aspurityy becomes larger, BLEU increases instead. This observation aligns with content identifiability theory. We determine Aspanity with hest G-score, i.e., IE-4.
A.2.2 Ablation Results on Amazon and Yahoo Datasets
show the ablation study of the Amazon and Yahoo datasets Table The full model achieves the best G-score and PPL on the two datasets. CausalDep improves the BLEU and PPL. Aquisity greatly improves the content preservation at the cost of sentiment acc. After incorporating the Lpartial and Le-musk, the sentiment acc is recovered.
Amazon
Yahoo
Model
Acc(t)
BLEU(f)
G-score(t) PPL())
Aco(f)
BLEU(1)
G-score(f)
PPL(.)
Backbone
77.61
20.28
Indep (Kong al. 2022
48.80*
39.50
31.76*
77.95
51.70
23:44
21.12*
56.95*
CausalDep
33.50*
45.25*
32.54*
66.98*
41.50
31.55*
21.39*
64.29*
tw. Ceppassity
27.10
62.734
34.734
63.37
27.32
48.21
25.16
60.034
Eparesat
33.10
58.54
34.04
64.42
38.12
41.74
28.03
$8.04
ww. / Ceran (Full)
3
52.25
35.73
63.37
38.454
42.40
29.01
56.12
Table 8: Ablation results sentiment transfer on two domains. CausalDep incorporates style flow to model dependency of while Indep assumes the independence between the two variables. marks the improvements overBackbone, while over the CausalDep.
A.3 Semantic Preservation Measurement by score score
As BLEU has limitations in capturing semantic relatedness beyond literal word-level overlap, adopt CTC score [Deng et al. 2021 as complementary evaluation for semantics preservation measurement. For the semantics alignment from b, CTC considers the matching embeddings, i.., maximum cosine similarity of all the tokens in with the tokens in b, and vice versa. Then, the final semantic preservation in Fl -style definition with one direction result as precision, and the other one as recall. The evaluation results of all the baselines and MATTE are shown in Table [ The CTC score still favours Optimus and MATTE, with most inferior results on 8-VAE, which similar trends under the BLEU evaluation schema. Admittedly, the CTC score differences are less discriminative than BLEU this phoneme also observed Liu al. 2022).
A.4 Diversity measurements for generated sentences
To further demonstrate the generation degradation issue generate oversimplified and repetitious sentences, we use diversity-2 [Li et al., 2016), the ratio of distinct two-grams in all the two-grams in generated sentences to evaluate the transferred sentences. The diversity-2 for original sentences also included for better comparison. The results in Table show that all the other methods except for 3-VAR generated sentences with similar diversity-2 as original sentences, but the sentences generated by B- V-VE have much lower diversity than the original ones.
18

---

IMDB
Yelp
Amazon
Yahoo
BGST
0,468
0.458
0.472
0.458
3-VAE
0,436
0.433
0.433
0.413
JointTrain
0.456
0.462
0.455
0.437
CPVAE
0.462
0.463
0.461
0.443
GPT2-FT
0,459
0,459
0.458
0.448
Optimus
0.465
0.468
0.465
0.446
Matte
0.465
0.464
0.466
0.452
Table 9: CTC score, a complementary evaluation for semantics preservation. impressive performance, and Optimus and Matte exhibit the overall best results.
8-VAE displays the least
Dataset
IMDB (0.34)
Yclp (0.63)
Amazon (0.64)
Yahoo(0.44)
8-VAE
0.11
0.46
0.37
0.22
JointTrain
0.21
0.59
0.56
0.37
CPVAE
0.32
0.59
0.57
0.45
MATTE
0.32
0.62
0.61
0.45
Table 10: Diversity-2 for the transferred sentences. Diversity for the original sentences is included in the bracket for comparison. VAE has significantly fewer distinct 2-gram than original datasets. This results consistent with evaluation results BLEU.
A.5 Proof for Theorem
1
The original Assumption T and Theorem are copied below for reference.
Assumption 1 (Content identification).
i. g smooth and invertible and its inverse g- is also smooth.
ii For all E fl....,day det. there exist ( and T E T, such that = Rdi and [J, (z(0)) E De."
(z()(
ii.. For every pair (Ojes 84.) with je E [de] and ja E (de + I,...,dz) d.), the influence of 8j, is sparser than that of Cjo ie., 11
1l:
Theorem 1. We assume the data-generating process in Equation D with Assumption 7 If for given dimensions (dey dg), a generative model (pe, Pajes 9) follows the same generating process and achieves the following objective:
arg min PesPesh
Σ
subject to: (x(x) (x(x), Vx E
JsE(deti....dd
then the estimated variable è is one-to-one mapping of the true variable That is, there exists invertible function he(.) such that ha(c).
Proof. We first define the notation
[c, s) and the indeterminacy function:
= 9 og.
which is invertible function h: 2 2 g is invertible by Assumption According the chain rule, we have the following relation among the Jacobian matrices:
(2) = (z)J (z).
(8)
We define the support notations as follows:
supp (opp(J((z))
= supp(J ( (2)),
T = (
19

---

In the following, we will show that (Jc.js) T for any je E fl...., det and js E {de + ,....,dy de}. That is, 1 (z)l 0, for any Je {i...., de) and js E {do + I,..., de + de), which implies that is not influenced by s.
11. for any i E {I,...,do, de, + drg.. there exists {z (1 19 12=1 such that
Because of Assumption
span( (J (z (t.)) [=
for any jo Ii.s we can express canonical basis Gual forms basis of Rd, Sr." Since (Jg(2(0) )::)))
vector 0% R as:
i = Σ ar Jn J (z(0))
(9)
EE
where E R is a coefficient. Also, following Assumption (f1,22 E and
there exists deterministic matrix T where T # 0 iff
T Jo:: 20 T Σ ar oz. Jg(2! Rd. 6...
(10)
6E9,.
where E is because each element in the summation belongs to gd. 9
Therefore,
Vj Gi,, T3. E Rd. 9..
Equivalently, we have:
v(i,j) EG
(1} x Ty: C.
(11)
As both Jg and Jg are invertible, J, (z) is invertible matrix and thus has non-zero determinant. Expressing J (z) with the Leibniz formulae gives:
d. J, [z((z)(()))) 5=1
de)()n(z))
Σ
sign( II
#0,
(12)
oePs
where Pd, is the set of all d, de-perrutatioos.
Equation 12 indicates that there exists a E Pd,. such that 11 J)(2) # 0. Equivalently, we have
Vj [d.), (o(i).))
(13)
Therefore, for specific ja E {de + I....,,d, d= it follows that (o(ja). Ja) E T. Further, Equation II shows that for any ig E [dz], s.t., ((s)) E G, we have (ix) x ToGD). C G. Together, it follows that
(iz, ois))ss)))))))) E G = (in.js) EG.
(14)
Equation 14 suggests that the column (jaa) of the true generating function support G is included in the column js of the estimated generating function support g. Together with Assumption follows that
ii..
Σ
G
Σ
(15)
+.((C++.......)
where the permutation o--) connects the indices and those We note that Equation 15 is lower-bound of the objective Equation 3 which can be attained by minimizer I = g-
In the following, we show by contradiction that the support of J does not contain (jer ja), for any je [d.] and any ja {de + 1,..,, ....,,d ......,d.,...................... i... (a) $ T.
20

---

We suppose that a specific (%) T, where J' E [d.] and any d E {de +1,...... d We note that the argument for Equation 14 also applies to (2).22 E T for any J1. E [d.]. Thus, we would have
(ee E T
(iar E G, Viz E {i E [dx] (Le. g}.
(16)
It would follow that
E
G 11
|
E
19owwll + | l1o
+
E
9 + |5 U
E
9 1|9.5:llo
ROS
E
1lo
()) j.E.doth....d.
(17)
where the inequality (1) is due to Assumption 1 that the influence of x is denser than that of
8.
However, discussed above, there exists an optimizer that attains the lower-bound Equation 15
Equation 17 contradicts the minimization objective Equation 3 Therefore, (121%) & T, for any H' E [de] and any j'g E {de + L,..,,,, de).
As discussed above, this implies that is not influenced by Further, it follows from the invertibility of ft()) that [z)(eeie.., = 0, for any je E {l...., de) ) and j. E {de+1.... de de which implies that è is not influenced by s. These two conditions and the invertibility of l(-)) imply that and form one-to-one mapping.
A.6 Proof for Theorem 2
The original Assumption ii.. and Theorem are copied below for reference.
Assumption 2 (Partially intersecting influence supports). For every pair (c,, 8j ). the supports of their influence on x do not fully intersect, i.e., [19.j. n.g. llo min lll lll
Theorem 2. We follow the data- generating process Equation 7 and Assumption 7 and Assumption 2 We optimize the objective function in Equation together with
Σ (l..,)e (ddcxxx
@ T G
min
The estimated style variable is a one-to-one mapping to the true variable s. That is, there exists an invertible mapping h-(-) between s and 8, i.e., $ = (s(s).
Proof. As shown in Section A.5 there exists permmuttttto af-) such that Vj E [d2], (o(),,) ET. Also, we have shown in Theorem that je) & T for je [de] and ja E {de l,...,, dz). which
implies that (jss) {de + l,.... ). Thus, it follows that for any je [de], (jee) E [de].
In the following, we show by contradiction that (js:Ja) & TT for any j. E {de + 1....,d.) je E [de]. We suppose that (1'..') E T. Analogous to Equation 16 we would have that
dz) and
(1'r22) ETT (1a.1) G Vig {i [d,] (tz, 9).
(18)
It would follow that E.d. 2 (.00%) U Also, to attain the objective Equation in Theorem
we have J' = a- "U') E (d. + +1. I,..., dz). s.t. G..; G..g. It would follow that G..g 2 9. Further, we would have
| 0 9 || = | = | l1
g0o0g no neoussloo:
(19)
(2)
21

---

where (2) is due to Assumption 2
We note that the lower-bound for Equation is
Σ
|ci n n l1o
Σ
( 0 (9 l
(edd) (e x (f..t
(20)
E
11 n nioallo
(eda)E (eundde [dcc+l...dd
(21)
which can be achieved by G = G. Note that the lower-bounds for both Equation and Equation 4 can be attained simultaneously by g = G. Hence, optimizing the sum of the two objectives does not alter the optimal value of either.
Applying a similar argument as that in Equation 15
we would have that
Σ
Gidee no
(I d.)x
Σ
| n n lb. + +1s n siilll
(n.s)e(.....d bx 1100..000
2
Σ
|19 n nooosollo + Gius n nhiillo
(22)
(eds)E(I,...) x (1
|| n noo $.000 n G
(3) (e.n)e
a. MO05
E
1 1lo
(e (de+)
where (3) due to Equation 19 Hence, this was not the minimizer of Equation 4 By contradiction,
we have that (jx,Ja) & T for any j. {de + +...... ......d.) and ja E de]. This implies that is influenced by Further, it follows from the invertibiliity of h(-) that [J, [J) is E ) and E [de]. which implies that & is not influenced by These two conditions and the invertibility of l(-)) imply that and form a one-to-one mapping.
0, for any
22


Reviews:
[{"review_id": "24ynUqNJtH", "reviewer": "Reviewer_pLx5", "summary": "This paper discusses controllable text generation and tackles the dependence between content and style in the counterfactual generation task. Identification guarantees are proven and used to enhance disentangling of the variables.", "questions": "\n+ Presentation Issues\n\nWhat is 'relative sparsity'?\n\nL21: 'state-of-art' -> 'state-of-the-art'\n\n'grey shade' -> 'grey-shaded nodes'\n\nNot sure about the difference between ';' and ',' in Eqn 2.\n\nSome explanation of the assumptions using natural language would assist readers understand the work.\n\n+ Missing reference\n\n[A] A causal lens for controllable text generation (NeurIPS 2021)\n\n[B] Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation (EMNLP 2022)\n", "limitations": "The evaluation is limited to 1) only automatic methods without human inspection and 2) using a singular model architecture in experiments. All these may lead to biased discussion.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "Style and content disentanglement, especially for scenarios with sparse data, is very challenging and important to language generation.\n\nA theoretical discussion on sparsity constraints is interesting.\n\nThe following practice is motivated by the theory and the experimental results are quite positive.", "weaknesses": "\n+ The writing requires improvement:\n\nThe definition of some key concepts of this paper are missing, such as 'identifiability guarantee' and 'relative sparsity'.\n\nThe connection between the generative model (described in Sec 3) to previous works is not discussed. Comparison to an existing counterfactual generation framework is appreciated.\n\nThe paper confounds 'sentiment' and 'style', which gets worse when it is heavily used as the running example throughout the paper. Sentiment is more of the semantic aspect of the text.\n\nMany methods and technical choices are only expressed in math without enough explanation and motivation.\n\n+ Some technical questions:\n\nu is used in Sec 3 but missed in Sec 4. How is it considered?\n\nWhat is the definition of T(z)?\n\nAssumption 1.i requires g to be invertible, but sparse matrices are usually not invertible, as they are not full rank.\n\nThe connection between the proposed theory and practice is loose. One issue is the assumptions are strict, while they are used as losses (which means the assumptions may not hold during training).\n\nConsidering there are many missing definitions and explanations, I cannot judge the correctness of the theoretical part.\n\nThe latest best baseline methods [28,52] are published in 2020, you may consider later work such as [A,B] as baselines.\n\nThere is no human study and style classification only relies on one model. There could be some biases in the evaluation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable time and your detailed feedback! We address each point as follows.\n\n### Writing\n\n**(1) Key Concepts.**\n\n**Identifiability guarantee** refers to the standard notion of identifiability in statistics (https://en.wikipedia.org/wiki/Identifiability), which describes the possibility of learning the true statistical model (up to certain equivalent classes) from its samples. The formal characterization of identifiability is presented in Theorem 1 (line 174) and Theorem 2 (line 216). In revision, we will highlight that identifiability refers to the standard notion in statistics. In case you see any other missing concepts, please let us know.\n\n**Relative sparsity** is formally presented as Assumption 1. iii and Figure 2. Intuitively, it prescribes that the influence from the style latent variable on the text is sparser than (thus “relative”) that from the content latent variable, as discussed in lines 188-198. This assumption encodes the belief that content information often plays a more prominent role in determining the text than style information. \n\n**(2) Comparison with existing methods.**\n\nThanks for your constructive suggestion -- we will highlight this part to improve our paper. Our main contribution is to propose two flexible assumptions to replace existing unrealistic assumptions (e.g., the independence between $c$ and $s$) via regularisations. It can be applied to broader generation problems -- the sentiment transfer is one example that satisfies our assumption. Our framework is fully unsupervised in training -- without paired data and style labels. In contrast, [A] incorporates style classifiers and [B] is pretrained on style-labeled data. Thus, they are not directly comparable with our approach, which is similar to unsupervised disentangled learning (Line 74). Based on CPVAE, the SOTA unsupervised style transfer model, MATTE has the following merits:\n\n(i) It is driven by the identifiability guarantee that enforces style sparsity and intersection minimization. Therefore, it is theory-grounded for the content and style disentanglement.\n\n(ii) Domain variables are introduced in the data generation process, and the dependency between content and style varies from different domains. CPVAE does not consider the multiple-domain situation. \n\n**(3) Sentiment is a style?**\n\nThanks for raising this fundamental question. We agree that the usage of 'style' might cause confusion, as you pointed out. In this paper, we decompose latent variables into two parts, following the assumed generating process to facilitate counterfactual generation. It uses a broader sense of \"style\" to refer to the latent variables with a sparse influence on the text and influenced by specific content variables. We believe that sentiment, in many cases, falls into such a category, e.g., gender, tense. Thanks to your comment, we will consider renaming the two latent variables to avoid confusion or adding a footnote to make this point clear to readers.\n\n### Technical\n\n**(1) $u$ not discussed in Sec4.**\n\nGreat question! Actually, even without multiple domains, the properties (graph structures & sparsities) of the generating process grant identifiability. Therefore, we didn’t discuss $u$ in Section 4. We will add a remark to clarify this point in our revision.\n\n**(2) Definition of the $T(z)$.**\n\n$T(z)$ (defined in line 164) is the transition matrix between the Jacobian of the true generating function $J_{g}$ and that of the estimated generating function $J_{\\hat{g}}$.\n\n**(3) Invertibility and sparsity.**\n\nExcellent question! We note that our conditions do not constrain the absolute sparsity of the entire Jacobian matrix. Concretely. Assumption 1. iii. only requires the Jacobian matrix dimensions corresponding to the style variable to be sparser than those for the content variable, which can be true even if the Jacobian matrix is dense. Therefore, the sparsity and the invertibility assumptions are compatible.\n\n**(4) Assumptions may not hold during training.**\n\nThank you for raising this question! We note that the assumptions are made w.r.t. the true data-generating process that generates the dataset. We implement losses to drive the estimated model to satisfy these conditions at the training optimum. It does not affect our theory whether these conditions are met during training. The decrease in corresponding loss actually indicates better identifiability. \n\n**(5) Human evaluation.**\n\nThank you for the valuable suggestions. Inspired by [1], we established the criteria to assess the four aspects: style, meaning, fluency, and overall transfer quality. The first three metrics use a 5-point Likert scale (higher scores indicate better performance), and the last is rank-based. We randomly selected 100 examples and compared the results from the top-performing baselines in **Table 1** in PDF. \n\n**(6) Multiple style evaluators.**\n\nWe totally agree that using multiple classifiers is better to avoid performance fluctuations, though much existing work neglects this issue [2,3,4].  We adopt the pretrained BertClassifier and fine-tune it for one epoch. The model with the best dev performance (96.23%) is used in our evaluation.  The results evaluated by BertClassifier are in **Table 2** in PDF.\n\n### Presentation\n\nWe use a semicolon “;” to delineate the function inputs and the function parameters, i.e., $f$( input1, …; parameter1, …), which is often adopted in the literature. In Eqn 2, we use this notation to indicate that $g_{s}$ transforms $\\tilde{s}$ to $s$, and this transformation is determined by $c$ and $u$.\n\nPlease let us know if you have further concerns and consider raising scores if we have clarified your concerns -- thank you!\n\n[1] A Review of Human Evaluation for Style Transfer. GEM21\n\n[2] Style Transfer from Non-Parallel Text by Cross-Alignment. Neurips17\n\n[3] Multiple-attribute text rewriting. ICLR19\n\n[4] A Probabilistic Formulation Of Unsupervised Text Style Transfer. ICLR20"}, {"review_id": "XakDylRtyE", "reviewer": "Reviewer_pDz1", "summary": "The work \"Controllable Text Generation with Identifiability Guarantee\" presents a model for unsupervised counterfactual generation based on a model with disentangled style and content. The model is based on a variational autoencoder augmented with two flow-based models operating in the latent space. The flow-based models perform the disentanglement and, since they are invertible, allow to intervene on the style variable for counterfactual generation.", "questions": "Please see previous section", "limitations": "The approach proposed in the paper has some technical limitations leading to a quality decrease. However, the proven identifiability guarantees lead to the mitigation of some of the potential societal risks. E.g. if the algorithm is used to change the style of some text, its semantics should be kept intact, to avoid the occasional creation of misinformation messages.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Overall, I find the basic idea and the model itself interesting and novel. I like how the paper presents theoretical results that are then illustrated by the model rather than just an experimental evaluation. However, there are several reservations that weigh the scales towards rejection for me.", "weaknesses": "First, the experimental evaluation is underwhelming, for several reasons:\n\n(i) the only setting provided is sentiment transfer, which significantly restricts the claims made for counterfactual generation in general; in fact, I would argue that the introduction and abstract are much more general than the actual results, and would advise to rewrite the introduction to mention that the model is only proven to work for sentiment transfer;\n\n(ii) I'm not sure that the BLEU metric and consequently the G-score make a lot of sense here since BLEU simply shows how much of the original wording is preserved; e.g., replacing a word with a synonym reduces BLEU but, all else being equal, arguably makes for better counterfactual generation since it makes generated sentences more varied and hence useful, e.g., as synthetic data;\n\n(iii) moreover, I'm not sure I understood the accuracy metric as presented: e.g., the IMDB dataset only has positive and negative sentiments, the authors claim that their classifier has 95% accuracy on original validation sets (this makes sense), but then Table 1 shows IMDB accuracies for counterfactuals ranging from 14% to 38% -- so that's much worse than chance for all methods including the supervised upper bound?.. it may be a misunderstanding on my part, but the paper does not clarify this at all;\n\n(iv) the qualitative results in Table 2 are also unclear: e.g., the authors claim that Optimus alters the semantics but doesn't MATTE also do it in Src 2? plus, I couldn't understand Src 4 at all, either the original or transferred versions, they don't make any sense.\n\nSecond, I'm afraid that these days a text generation model has to compare itself with modern large language models, while the authors choose a GPT-2-based model from 2019 as their best baseline (and a baseline that serves as an unachievable upper bound since it's supervised). I wonder how well GPT-3.5 would do if you just asked it nicely to \"please invert the sentiment while preserving content as much as possible in the following sentence that originates from the domain of movie reviews\", maybe with a couple of generic examples in the prompt?\n\nThird, related work would benefit from a section on topic modeling that has developed joint sentiment-topic models such as ASUM or USTM that are quite similar in their basic assumptions to the presented model. Topic models obviously cannot serve for counterfactual generation but it looks like they were an influence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the valuable comments and suggestions. We have revised and included some results and we hope the revision could address your concerns.\n\n### Task is limited\n\nThank you for the suggestion! We will make it explicit that our empirical study is limited to sentiment transfer.\nAt the same time, to make our contribution even clearer, we will highlight that our primary contribution is to establish a theoretical foundation for counterfactual generation and reason about theoretical guarantees, which is often missing in the existing literature. The resulting theory and framework are general and can inform the algorithmic design for many other tasks.\n\nFurthermore, we evaluate MATTE on tense transfer task and results are in **Table 3** in the PDF.  Specially, we follow [5] to extract the tense for the main verb using the StanfordNLP Parser,  evaluate the transferred sentences by the parser. It outperforms the best baseline over 6\\% with impressive style representation mapping results.\n\n### BLEU is not good\n\n\nThank you for sharing your insights. Although many existing papers adopt BLEU [1, 2], we completely agree with you that BLEU has limitations in capturing semantic relatedness beyond literal word-level overlap. To make this issue clearer to the audience, we will make a footnote in the paper. We can observe that another metric, sentiment accuracy, also favours our methods in the same way. Furthermore, we adopt one metric, CTC score [3], to avoid the aforementioned issues of BLEU, as it considers the matching embeddings, i.e., cosine similarity of pretrained word embedding rather than the “hard match”. The results in **Table 4** (in the PDF) show that $\\beta$-VAE displays the least impressive performance, and Optimus and Matte exhibit the overall best results. Admittedly, results differences are less discriminative than BLEU -- this phoneme is also observed in [4]. \n\n### Sentiment acc\n\nWe apologize for any confusion. The evaluation has successfully achieved a 95% accuracy for binary sentiment classification on the original sentences in the dev set, showcasing its reliability as a sentiment classifier. Consequently, we employ this classifier to assess whether the intended sentiment is accurately conveyed. Specifically, if the transferred sentence is assessed as the opposite sentiment (evaluated by the sentiment classifier), then the sentiment accuracy numerator is incremented by one. In the worst case, all the generated sentences fail to express the opposite sentiment after the intervention, and the sentiment acc is 0. Therefore, the random sentiment transfer acc is not 50\\% as in classification problems.\n\n### Case study examples are not clear\n\nThanks for your valuable comments on the qualitative results. We will improve this part by providing clearer examples and descriptions in the manuscript. \n\n###  Compared to GPT3.5\n\nWe fully agree with you that evaluating LLMs will provide valuable insights. As well acknowledged, LLMs have demonstrated an impressive ability to generate fluent text. That said, we view the principles for counterfactual generation as complementary to the development of LLMs, and we hope that our theoretical insights can further enhance LLMs. In fact, we supply examples below that LLMs falter on sentiment transfer. Chatgpt can precisely detect the sentiment words and find a good replacement to complete a fluent sentence. This word-level replacement favours sentences with explicit and straight sentiment expressions. Our methods, based on representation learning, can probably better capture mixed and implicit sentiment expression, where ChatGPT neglects the overall sentiment. Thus, one may anticipate that LLMs would benefit from principles for representation learning, as developed in our work.\n\nExamples: Source sentence (S), ChatGPT (C), MATTE (M).\n~~~\nS: I had to leave a message, and they never called me back.\nC: I had to leave a message, and they promptly called me back.\nM: I had to leave a message, and they kept thinking that ended up that finally helped me.\n\nS: This case is cute however this is the only good thing about it. \nC: This case is not cute; however, it is the only good thing about it. \nM: This case is cute and overall a valuable product.\n\nS: The buttons to extend the arms worked exactly one time before breaking. \nC: The buttons to extend the arms failed to work from the beginning, never functioning even once.\nM: The buttons to extend the arms worked exactly as described.\n~~~\n### Connection with topic modelling:\n\nThank you for the great suggestion! Taking the ASUM as an example, it is built on Latent Dirichlet Allocation (LDA) and models the generation of words based on the hierarchy of latent variables that is conditional on sentiment. Words are generated from the sentiment-aspect-word multinomial distribution. It requires a predefined list of polarity words to set the asymmetric sentiment priors. In contrast, in our model, the style (or sentiment) depends on the content (or aspect), and we don’t need to pre-define the sentiment prior.   More concretely, the key differences are (a) we introduce the domain index $u$ to allow for varying dependencies between content and style across different domains; (b) we can apply an arbitrary function to model the dependency between content and style, i.e., Eqn (2); (c) based on the generation process, we propose two flexible constraints to the distributions to achieve the identifiability guarantee.\n\nPlease let us know if you have further concerns, and please consider raising the score if we have clarified your concerns -- thank you!\n\n### References\n\n[1] Semi-Supervised Formality Style Transfer with Consistency Training\n\n[2] Deep Learning for Text Style Transfer: A Survey\n\n[3] Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation\n\n[4] Composable Text Controls in Latent Space with ODEs\n\n[5] Educating Text Autoencoders: Latent Representation Guidance via Denoising\n"}, {"review_id": "ea6FOPKjDK", "reviewer": "Reviewer_zc3n", "summary": "This paper wants to do a controllable text style transfer by tackling the dependence between the content and the style variables. They adopt the concept of influence sparsity requiring the influence of the style variable to be sparser than the content variable. They evaluate their method on several NLP datasets to show the style transferred text generation.\n", "questions": "- The author mentioned their method could leverage the data from multiple domain. How is this reflected in your results?\n- In your experiment, the goal is to do a “style” transfer which means transferring the sentiment. However the accuracy (which means the intended attributes are expressed, Line 299) is very low, how can you guarantee your style transfer is successful? Also what is the label distribution for each data? What is the accuracy of a random guess in each case?\n- From Table 2, the transferred texts seem only change several tokens. So how does your method compare to a naive antonym replacement? \n- There are other unsupervised text-style transfer learning algorithms that are more up-to-date, please check: https://github.com/fuzhenxin/Style-Transfer-in-Text \n", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper aims to do controllable text style transfer which is an interesting application in NLP domain. In this paper the authors relax the independence between the style and context fills the gap in the literature. ", "weaknesses": "The main idea of this paper is to disentangle the content from the style. However throughout the whole paper, I don’t think the authors define clearly what is a “style”. In the literature, people usually define the sentence structure (e.g, dependency parsing tree) as the “style” and the semantics as the “content”. While in this paper, in Introduction (line 37), the author mentioned a positive sentiment is “style”, but later in Sec.3 (Line 113), they refer to the positive descriptions of something as “content”. \n\nThe experiment only shows the style transfer on sentiment perspective and lacks comparison to many advanced baselines.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the valuable comments and suggestions.  We address each point as follows.\n\n### Advanced baselines and evaluation on other styles\n\nThanks for providing the comprehensive list! It should be noted that our proposed method does not rely on attribute/style labels in the training, nor the paired data, while the unsupervised methods in the list are only free from the paired data. They still rely on the style annotation for each training sample. In this sense, our method is similar to unsupervised disentangled learning ($\\beta$-VAE, DAAE[1], DCTC [2]). As far as we know, the most recent work without style labelling should be CPVAE [3], and both CPVAE and our Matte only need a small set of sentences with the desired styles to derive $\\tilde{s}_\\text{transfer}$ when conducting the sentiment transfer (inference phase). See Sec5.3. \n\nOur main contribution is to propose two flexible assumptions to replace existing unrealistic assumptions.  It can be applied to broader generation problems -- the sentiment transfer is one example that satisfies our assumption that sentiment is relatively sparse and is influenced by the content.  Thanks for your comments about experiments on more styles, we add an experiment on tense transfer, which transfers the tense between present and past. Specially, we follow [1] to extract the tense for the main verb using the StanfordNLP Parser, and measure the tense of the transferred sentences by the parser. We compare with DAAE and DCTC and the results are in **Table 3** in the PDF. Our method outperforms the other baselines over 6\\%, with an illustration of a discriminative representation s for different tense.\n\n### Style or Content\n\nThanks for raising this fundamental question. We agree that the usage of 'style' might cause confusion, as you pointed out. In this paper, we decompose latent variables into two parts, which follow the assumed generating process to facilitate counterfactual generation. It uses a broader sense of \"style\", which are latent variables that have a sparse influence on the text (i.e., relative to the \"content\" latent variable) and are influenced by specific content variables. We believe that sentiment, in many cases, falls into such a category, as discussed in the examples, e.g., lines 36-39. Thanks to your comment, we will consider renaming the two latent variables to avoid confusion cause by “content” and “style” or add a footnote to make this point clear to readers. \n\n### Domain in results\n\nThank you for the nice question! We train all baselines and the proposed approach with multiple domain datasets. They differ in how to leverage the domain information.  The effects can be seen from both quantity and qualitative aspects:\nIn Table 1, all the baselines in the LSTM group first derive a domain embedding $u$ and then concatenate $u$ to the sentence-level representation to differentiate their domain source. For our approach built on CPVAE, we adopt $u$ in the content-style dependency modelling described in Eqn(2). We can see clear improvement over CPVAE. Besides, in Table 2, the first example, “The guy is an awful actor → The guy is very flavour” shows that the simple concatenation of domain indexes in CPVAE fails to capture the domain-adaptive content-style dependency: the sentiment change leads to unnatural content-style dependence, although the polarity is correctly flipped. \n\n### Analysis of style acc results\n\nWe apply a sentiment classifier with 95% acc in all the validation datasets to evaluate whether the intended sentiment is expressed in the transferred sentences. That is, if the original sentence is positive/negative (according to the data annotation) and the transferred sentence is negative/positive (evaluated by the classifier), then the sentiment accuracy numerator is increased by one. So, it is not applicable that the sentiment acc has to be over 50% (random guess) if there is only binary sentiment polarity. \n\n### Antonym replacement baseline\n\nThanks for sharing your insights. We totally agree that word substitution is important in many text rewriting tasks, and naive antonym replacement can flip the overall sentiment with minor modifications when the given sentences have simple structures and explicit sentiment indications. We supply the comparison results in the table below with antonym replacement over the four datasets. Specially, we adopt the NLTK tool to do pos tagging to the original sentence and replace the words annotated as “ADJ” (adjective) with its antonym found in WordNet.  The performances vary a lot from different datasets. Our method performs relatively stable across different datasets. It can be partly explained by the variations of the sentences and the representation learning approach can benefit generalizability. Some examples are shown below. We will add a footnote to Table 2 to elaborate on this insight.\n\n**Table**:  Sentiment acc of antonym replacement and Matte. \n\n|                     | Imdb  | Yelp  | Amazon | Yahoo |\n|---------------------|-------|-------|--------|-------|\n| Antonym replacement | 11.40 | 14.40 | 10.30  | 21.75 |\n| Matte               | 32.43 | 34.30 | 34.50  | 38.45 |\n\nWe also supply some cases below: \n~~~\nsrc: Need a cheap car charger and it seems to do the job.\nanto: Need a expensive car charger and it seems to do the job .\nours: Need a cheap car charger and it seems to be unaffordable.\n\nsrc: I will stick to my Xbox for now, thanks.\nanoto: I will stick to my Xbox for now, thanks.\nours: I will buy it to replace my Xbox, thanks.\n~~~\n\nPlease let us know if you have further concerns, and please consider raising the score if we have addressed your concerns -- thank you!\n\n### References: \n\n[1] Educating Text Autoencoders: Latent Representation Guidance via Denoising. ICML20\n\n[2] Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders. EMNLP21\n\n[3] On variational learning of controllable representations for text without supervision. ICML20\n\n"}, {"review_id": "sz8LAixkB6", "reviewer": "Reviewer_1Pqp", "summary": "In the paper, the authors take varying dependence between content and style into account in the counterfactual generation process. The identification problem is addressed when using a VAE for the task. It is proved that the subspaces of latent variables for contents and style are identifiable. Then based on the theorems and assumptions the authors propose to build a VAE based model with sparsity regularization to solve the problem. The proposed framework is tested on four datasets and gets relatively high scores compared with other unsupervised baselines. Ablation study and case study are also conducted to consolidate the conclusion.", "questions": "Thanks for authors' detailed rebuttal, which addressed my concerns. I'll keep my score.", "limitations": "Limitation discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Detailed mathematical proof is done for the conclusion. \n2. The model performs well compared with other unsupervised baselines. \n3. The proposed framework may be applied in various models in the future.", "weaknesses": "1. Human evaluation might be necessary to prove the framework actually reaches the original expectation. The automatic metrics may not be adequate.\n2. The MATTE does not perform well enough in the experiments. It performs poorer in accuracy and perlexity than beta-VAE. The higher scores based on word-overlap can mean little in such situation.\n3. Performance of VAE-based models always varies according to random factors, and it’s better to note the mean and std. of the results in multiple trials.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the insightful comments and suggestions.  We hope that generation tasks, including advanced LLMs would benefit from principles for representation learning, as developed in our work. \n\n### Human Evaluation\nThank you for providing us with valuable suggestions to enhance our work. In line with the settings in [10], we have established the following criteria to assess the style, meaning, fluency, and overall transfer quality. Due to time constraints, we randomly selected 100 examples from the four datasets and gathered the generated sentences from the top-performing baselines in each group, namely CPVAE, Optimus, and our approach. We have invited three English-fluent evaluators to rate the first three metrics using a 5-point Likert scale (higher scores indicating better performance) and ask them to rank the three generated sentences (tied ranks are permitted). The Cohen’s Kappa coefficient among every two annotators over the rank-based metric 0.46. The results are supplied in **Table 1** in the attached PDF.  The results show that human annotators favour Optimus in terms of content preservation and fluency, which can be attributed to its powerful decoder in generating fluent sentences. After considering the style transfer correctness, Matte ranks as the best-performing method with more than 58% support set. \n\n### $\\beta$-VAE performance\n\nThanks for pointing out this result. $\\beta$-VAE indeed demonstrates higher acc and lower perplexity. However, we have observed that many generated sentences follow simple but repetitive patterns. For example, 2.2\\% transferred sentences in the Yelp domain containing the phrase “I highly recommend” while only 0.6\\% original sentences do. While these sentences are fluent and express the desirable sentiment, they significantly differ from the original sentences and indicate a generation degradation problem.\n\n In this context, BLEU can play a role in avoiding pitfalls because the significantly lower BLEU probably indicates the decline in the model reconstruction ability-language modelling ability. Therefore, we use G-score as the primary metric to balance sentiment acc and semantic preservation.\n\nWe also noticed another metric, “diversity-n”[9] can be a good indicator for oversimplified and repetition generation. It measures the ratio of distinct n-grams in all the n-grams in the generated sentences. All the other methods except for $\\beta$-VAE generated sentences with similar diversity-2 as the original sentences, but the sentences generated by $\\beta$-VAE have much lower diversity than the original ones (see Table below).  We will make it clearer to the audience. Thank you for your valuable comments again. \n\n**Table**: Diversity-2 for the transferred sentences. Diversity for the original sentences is added in the bracket for comparison. $\\beta$-VAE has significantly fewer distinct 2-gram than original datasets.\n| Dataset     | IMDB (0.34) | Yelp(0.63) | Amazon(0.64) | Yahoo(0.44) |\n|-------------|-------------|------------|--------------|-------------|\n| $\\beta$-VAE | 0.11        | 0.46       | 0.37         | 0.22        |\n| JointTrain  | 0.21        | 0.59       | 0.56         | 0.37        |\n| CPVAE       | 0.32        | 0.59       | 0.57         | 0.45        |\n| MATTE       | 0.32        | 0.62       | 0.61         | 0.45        |\n\n### BLEU metric\n\nThank you for sharing your insights. Although many existing papers adopt BLEU [1, 2, 3, 4, 5] and it is still an open question of how to measure semantics preservation due to word variation [6], we completely agree with you that BLEU has limitations in capturing semantic relatedness beyond literal word-level overlap. To make this issue clearer to the audience, we will make a footnote in the paper. We can observe that another metric, sentiment accuracy, also favours our methods similarly. Furthermore, we adopt one metric, CTC score [7], to avoid the aforementioned issues of BLEU, as it considers the matching embeddings, i.e., cosine similarity of pretrained word embedding rather than the “hard match”. The results in **Table 4** in the attached PDF show $\\beta$-VAE display the least impressive performance and, Optimus and Matte exhibit the overall best results. Admittedly, results are less discriminative than BLEU -- this phoneme is also observed in [8]. \n\n\n### Mean and std.\n\nThank you for the suggestion! We will add the mean and std to our draft. We also included the results in **Figure 1** in the PDF.\n\nPlease let us know if you have further concerns -- thank you!\n\n### References:\n\n[1] Semi-Supervised Formality Style Transfer with Consistency Training, ACL22 \n\n[2] Deep Learning for Text Style Transfer: A Survey, CL21\n\n[3] A Probabilistic Formulation Of Unsupervised Text Style Transfer. ICLR20\n\n[4] Style transformer: Unpaired text style transfer without disentangled latent representation, ACL19\n\n[5] Unsupervised text style transfer using language models as discriminators. Neurips18\n\n[6] Repairing the cracked foundation: A survey of obstacles in evaluation practices for generated text. JAIR23\n\n[7] Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation. EMNLP21\n\n[8] Composable Text Controls in Latent Space with ODEs. Arxiv 2022\n\n[9] A diversity-promoting objective function for neural conversation models. ACL16\n\n[10] A Review of Human Evaluation for Style Transfer. GEM21\n\n\n"}, {"review_id": "bbpjCpMw8f", "reviewer": "Reviewer_yXyx", "summary": "This paper proposes a framework for unsupervised style transfer by disentangling the content and style representations. Unlike previous research, they do not assume independence between the content and style variables in the generation process but rather only a lower influence of the style variable compared to the content variable on the generation process. Based on this assumption, the paper then introduces two theories with corresponding proofs of the identifiability of both the style and content variables under these assumptions. That is, the content information can be preserved without needing style.\n\nGiven the theoretical discussion, the paper then proposes a framework based on variational autoencoders (VAE) to address the task of style transfer through style and content disentanglement. The framework is then evaluated empirically on the task by comparing it to other state-of-the-art baselines. In the results, the paper demonstrates the gain from their implementation in most measures considered. The ablation study further illustrates the importance of every component added to their model.\n", "questions": "- The submission title differs from the title of the paper. Is that on purpose?\n\n- Line 153-156: If I understood correctly, then the described subspace should consist of all vectors whose entries are *not zero* for all indices in S. but in the paper says *zero for all indices in S*\n\n- In Section 5.3, the paper mentions that the s_(transfer) is computed from the average of randomly sampled style values of the *desired style*. What is the desired style here?\n\n- The paper considers the G-score the most important measure because of its comprehensive assessment. It would be nice if the paper clarified what it means to have a comprehensive assessment.\n\n- In section 6.1, It would be nice if the paper clarified the implemented model and the baselines. I learned later in the results section that the model is built on top of CPAVE\n\n- In Table-2 what are src1,2, and 3", "limitations": "- The central limiting aspect of this paper is the assumption that style has a lower influence on the generated text compared to the content, which doesn't hold for all text generation tasks. However, this limitation is acknowledged by the authors.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper contributes to the style transfer task by eliminating the independence assumption between style and content. \n\n- It supports its claims by providing both theoretical backing and empirical evidence. \n\n- It is clear in most of its parts.\n", "weaknesses": "I do not see any major weaknesses in this paper. However, the authors can address some clarity issues in the questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for the valuable comments and suggestions. We also hope that our method could provide insights for the following research. We respond below to address your concerns. \n\n### Title\nThank you for letting us know this. The title should be “Counterfactual generation with identifiability guarantee”. We will make it consistent.\n\n### Subspace and zero-vector\n\nYou are totally right! Thank you for spotting this typo; we will correct it in our manuscript.\n\n### Desired style\n\nGreat question! For general sentence transfer, the desired style is the target attribute of the sentence, e.g., positive sentiment, formal writing style, female subject, etc. We calculate the $s_\\text{transfer}$ over sentences with the target attribute.\n\n\n### Comprehensive assessment\n\nThank you for the good question! There is an inherent trade-off between sentiment transfer and semantic preservation because a larger perturbation could lead to a higher sentiment flip success rate but may distort the original semantics to a greater extent. See Table 2 in [1] for an illustration “$\\beta$-VAE under 1/2/3 sigma perturbation”. Therefore, G-score was proposed in [2] as a single metric to simultaneously consider both aspects and reflect the overall quality of the transfer, and it is followed by lots of related work [3,4].\n\n### Implementation clarification\nThanks for pointing out this issue. You are right -- our model is built on CPVAE. We will improve the baselines section in the revised manuscript to make it clearer.\n\n### src in Table-2\n\nsrc-* refers to the original sentences that express one particular sentiment and will be transferred to new sentences with the opposite sentiment by the listed approaches. \n\n\nPlease let us know if you have further concerns -- thank you!\n\n### References:\n\n[1] On variational learning of controllable representations for text without supervision. ICML20. \n\n[2] Unpaired sentiment-to-sentiment translation: A cycled reinforcement learning approach. ACL18.\n\n[3] Reformulating Unsupervised Style Transfer as Paraphrase Generation. EMNLP20\n\n[4] Deep Learning for Text Style Transfer: A Survey. CL22\n\n"}]

Author rebuttal:
We all thank all reviewers for their valuable feedback and dedicated time! We address the individual comments in separate response and will incorporate the reviews' suggestions in our revision.

Meta-review: The paper describes a controllable text generation framework which is evaluated on one specific task, namely counterfactual generation. 

The reviews raised a set of issues: This task is not properly defined in the paper and does not become clear from the abstract. That limits the accessibility of the paper. The link to style transfer and other conditional text generation tasks remain unclear. The term style is equated to sentiment in the paper, which is not correct; in fact it may be doubted if sentiment is a style. A proper definition of style is missing.

All these issues are, however, mitigated in the discussion, and I am confident that the paper will be improved sufficiently for the camera ready version.
Decision: Accept (poster)