SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hanqi Yan $^{1,4*}$ , Lingjing Kong $^{2*}$ , Lin Gui $^{3}$ , Yuejie Chi $^{2}$ , Eric Xing $^{2,4}$ , Yulan He $^{1,3}$ , Kun Zhang $^{2,4}$ $^{1}$ University of Warwick, $^{2}$  Carnegie Mellon University,  
 $^{3}$ King's College London,  $^{4}$ Mohamed Bin Zayed University of Artificial Intelligence
Counterfactual generation lies at the core of various machine learning tasks, including image translation and controllable text generation. This generation process usually requires the identification of the disentangled latent representations, such as content and style, that underlie the observed data. However, it becomes more challenging when faced with a scarcity of paired data and labeling information. Existing disentangled methods crucially rely on oversimplified assumptions, such as assuming independent content and style variables, to identify the latent variables, even though such assumptions may not hold for complex data distributions. For instance, food reviews tend to involve words like "tasty", whereas movie reviews commonly contain words such as "thrilling" for the same positive sentiment. This problem is exacerbated when data are sampled from multiple domains since the dependence between content and style may vary significantly over domains. In this work, we tackle the domain-varying dependence between the content and the style variables inherent in the counterfactual generation task. We provide identification guarantees for such latent-variable models by leveraging the relative sparsity of the influences from different latent variables. Our theoretical insights enable the development of a doMain AdapTive countTerfactual gEneration model, called (MATTE). Our theoretically grounded framework achieves state-of-the-art performance in unsupervised style transfer tasks, where neither paired data nor style labels are utilized, across four large-scale datasets.
Counterfactual generation serves as a crucial component in various machine learning applications, such as controllable text generation and image translation. These applications aim at producing new data with desirable style attributes (e.g., sentiment, tense, or hair color) while preserving the other core information (e.g., topic or identity) [Li et al., 2019, 2022, Xie et al., 2023, Isola et al., 2017, Zhu et al., 2017]. Consequently, the central challenge in counterfactual generation is to learn the underlying disentangled representations.
To achieve this goal, prior work leverages either paired data that only differ in style components [Rao and Tetreault, 2018, Shang et al., 2019, Xu et al., 2019b, Wang et al., 2019b], or utilizes style labeling information [John et al., 2019, He et al., 2020, Dathathri et al., 2020, Yang and Klein, 2021, Liu et al., 2022]. However, collecting paired data or labels can be labour-intensive and even infeasible in many real-world applications [Chou et al., 2022, Calderon et al., 2022, Xie et al., 2023]. This has prompted recent work [Kong et al., 2022, Xie et al., 2023] to delve into unsupervised identification of latent variables by tapping into multiple domains. To attain identifiability guarantees, a prevalent assumption made in these works [Kong et al., 2022, Xie et al., 2023] is that the content and the style
*Equal Contribution. Work was done when Hanqi Yan was a visiting student at MBZUAI. Correspondence to: Yulan He (yulan.he@kcl.ac.uk) and Kun Zhang (kunz1@cmu.edu).
latent variables are independent of each other. However, this assumption is often violated in practical applications. First, the dependence between content and style variables can be highly pronounced. For example, to express a positive sentiment, words such as "tasty" and "flavor" are typically used in conjunction with food-related content. In contrast, words like "thrilling" are more commonly used with movie-related content [Li et al., 2019, 2022]. Moreover, the dependence between content and style often varies across different distributions. For example, a particular cuisine may be highly favored locally but not well received internationally. This varying dependence between content and style variables poses a significant challenge in obtaining the identifiability guarantee. To the best of our knowledge, this issue has not been addressed in previous studies.
In this paper, we address the identification problem of the latent-variable model that takes into account the varying dependence between content and style (see Fig 1). To this end, we adopt a natural notion of influence sparsity inherent to a range of unstructured data, including natural languages, for which the influences from the content and the style differ in their scopes. Specifically, the influence of the style variable on the text is typically sparser compared to that of the content variable, as it is often localized to a smaller fraction of the words [Li et al., 2018] and plays a secondary role in word selection. For instance, the tense of a sentence is typically reflected in only its verbs which are affected by the sentence content information. Our contributions can be summarised as: 1) We show identification guarantees for both the content and the style components, even when their interdependence varies. This approach removes the necessity for a large number of domains with specific variance properties [Kong et al., 2022, Xie et al., 2023]. 2) Guided by our theoretical findings, we design a doMain AdapTive counTerfactual gEneration model (MATTE). It does not require paired data or style annotations but allows style intervention, even across substantially different domains. 3) We validate our theoretical discoveries by demonstrating state-of-the-art performance on the unsupervised style transfer task, which demands representation disentanglement, an integral aspect of counterfactual generation.
Label-free Style Transfer on variation autoencoders (VAEs). To perform style transfer, existing methods that use parallel or non-parallel labelled data often rely on style annotations to refine the attribute-bearing representations, although some argue that disentanglement is not necessary for style edit [Sudhakar et al., 2019, Wang et al., 2019a, Dai et al., 2019]. In practice, disentangled methods typically employ adversarial objectives to ensure the content encoder remains independent of style information or leverage style discriminators to refine the derived style variables [Hu et al., 2017, Shen et al., 2017, Li et al., 2018, Keskar et al., 2019, John et al., 2019, Sudhakar et al., 2019, Dathathri et al., 2020, Hu et al., 2017, Dathathri et al., 2020, Yang and Klein, 2021, Liu et al., 2022]. Several studies have tackled this task without style labels. Riley et al. [2020] emphasized the implicit style connection between adjacent sentences and used T5 [Raffel et al., 2020] to extract the style vector for conditional generation. CPVAE Xu et al. [2020] split the latent variable into content and style variables and mapped them to a  $k$ -dimensional simplex for  $k$ -way sentiment modeling. Our work aligns more closely with CPVAE and follows VAE-based label-free disentangled learning from data-generation perspective Higgins et al. [2017], Kumar et al. [2018], Mathieu et al. [2019], Xu et al. [2020], Mita et al. [2021], Fan et al. [2023].
Latent-variable identification. Representation learning serves as the cornerstone for generative models, where the goal is to create representations that effectively capture underlying factors in the disentangled data-generation process. In the realm of linear generative functions, Independent component analysis (ICA) [Comon, 1994, Bell and Sejnowski, 1995] is a classical approach known for its identifiability. However, when dealing with nonlinear functions, ICA is proven unidentifiable without the inclusion of additional assumptions [Hyvarinen and Pajunen, 1999]. To tackle this problem, recent work incorporated supplementary information [Hyvarinen et al., 2019, Sorrenson et al., 2020, Halva and Hyvarinen, 2020, Khemakhem et al., 2020, Kong et al., 2022], e.g., class/domain labels. However, these approaches require a number of domains/classes that are twice the number of latent components, which can be unfeasible when dealing with high-dimensional representations. Another line of work [Zimmermann et al., 2022, von Kugelgen et al., 2021, Locatello et al., 2020, Gresele et al., 2019, Kong et al., 2023a,b] leverages paired data (e.g., two rotated versions of the same image) to identify the shared latent factor within the pair. The third line of work [Lachapelle et al., 2022, Yang et al., 2022, Zheng et al., 2022] makes sparsity assumptions on the nonlinear generating function. Although their sparsity assumption alleviates some of the explicit requirements in the previous two types of work, it may not hold for complex data distributions. For instance, in the
case of generating text, each topic-related latent factor may influence a large number of components. Instead, we adopt a relative sparsity assumption, where we only require the influence of one subspace to be sparser than the other. Unlike prior work [Zheng et al., 2022], each latent variable is allowed to influence a non-sparse set of components, and the influence can overlap arbitrarily within each subspace. Importantly, we necessitate neither many domains/classes nor paired data as prior work mentioned above.
3 Disentangled Representation for Counterfactual Generation
In this section, we discuss the connection between counterfactual generation and the identification of the data-generating process shown in Fig 1.
Disentangled latent representation. The data-generating process in Figure 1 can be expressed in Equation 1:
$$
\mathbf {c} \sim p (\mathbf {c} | \mathbf {u}), \mathbf {s} \sim p (\mathbf {s} | \mathbf {c}, \mathbf {u}), \mathbf {x} = g (\mathbf {c}, \mathbf {s}), \tag {1}
$$
where the data (e.g., text)  $\mathbf{x} \in \mathcal{X}$  are generated by latent variables  $\mathbf{z} := [\mathbf{c}, \mathbf{s}] \in \mathcal{Z} \subseteq \mathbb{R}^{d_z}$  through a smooth and invertible generating function  $g(\cdot): \mathcal{Z} \to \mathcal{X}$ . The latent space comprises two subspaces: the content variable  $\mathbf{c} \in \mathcal{C} \subseteq \mathbb{R}^{d_c}$  and the style variable  $\mathbf{s} \in S \subseteq \mathbb{R}^{d_s}$ . We define  $\mathbf{c}$  as the description of the main topic, e.g., "We ordered the steak recommended by the waitress," and  $\mathbf{s}$  comprises supplementary details connected to the primary topic, e.g., the sentiment towards the dish, as exemplified in "it was delicious!". Consequently, the counterfactual generation task here is to preserve the content information  $\mathbf{c}$  while altering the stylistic aspects represented by  $\mathbf{s}$ .
Figure 1: The data generation process: The grey shading indicates the variable is observed. The observed variable (i.e., text)  $\mathbf{x}$  is generated from content  $\mathbf{c}$  and style  $\mathbf{s}$ . Both content  $\mathbf{c}$  and style  $\mathbf{s}$  are influenced by the domain variable  $\mathbf{u}$  and the content also influences the style.  $\tilde{\mathbf{s}}$  is the exogenous variable of  $\mathbf{s}$ , representing the independent information of  $\mathbf{s}$ .
Content-style dependence. In many real-world problems, content c can significantly impact style s. For instance, when it comes to the positive descriptions of food (content), words like "delicious" are more prevalent than terms like "effective". Intuitively, the content c acts to constrain the range of vocabulary choices for style s. As a result, the counterfactually generated data should preserve the inherent relationship between c and s. We directly model this dependence as:
$$
\mathbf {s} := g _ {s} (\tilde {\mathbf {s}}; \mathbf {c}, \mathbf {u}), \tag {2}
$$
where  $g_{s}$  characterizes the influence from c to s and the exogenous variable  $\tilde{\mathbf{s}}$  accounts for the inherent randomness of s. In the running example,  $\tilde{\mathbf{s}}$  can be interpreted as the randomness involved in choosing a word from the vocabulary defined by content c, encompassing words similar to "delicious" such as "tasty", "yummy". In contrast, prior work [Kong et al., 2022, Xie et al., 2023] assumes the independence between c and s thus neglecting this dependence.
Challenges from multiple domains. As we outlined in Section 1, the ability to handle domain shift is crucial for unsupervised counterfactual generation. Domain embedding  $\mathbf{u}$  represents a specific domain and the domain distribution shift influences both the marginal distribution of content  $p(\mathbf{c}|\mathbf{u})$  and the dependence of content on style  $p(\mathbf{s}|\mathbf{c},\mathbf{u})$ . The change in content distribution  $p(\mathbf{c}|\mathbf{u})$  across different domains  $\mathbf{u}$  reflects the variability in the subjects of sentences across these domains. For example, it can manifest as a change from discussing food in restaurant reviews to actors in movie reviews. The change in content-style dependence  $p(\mathbf{s}|\mathbf{c},\mathbf{u})$  signifies that identical sentence subjects (i.e., content) could be associated with disparate stylistic descriptions in different domains. For instance, the same political question could provoke significantly different sentiments among various demographic groups. Such considerations are absent in prior work [Kong et al., 2022]. Here, we learn a shared model  $(\hat{p} (\mathbf{c}|\cdot),g_s(\tilde{\mathbf{s}},\mathbf{c},\cdot))$  and domain-specific embeddings  $\mathbf{u}$ . This approach enables effective knowledge transfer across domains and manages distribution shifts efficiently. For a target domain  $\tau$ , which may have limited available data, we can learn  $\mathbf{u}^{\tau}$  using a small amount of unlabeled data  $\mathbf{x}^{\tau}$  while preserving the multi-domain information in the shared model.
In light of the above discussion, the essence of counterfactual generation now revolves around the task of discerning the disentangled representation  $(\mathbf{c},\tilde{\mathbf{s}})$  within the data-generating process (Fig 1) across various domains with unlabeled data  $(\mathbf{x},\mathbf{u})$ : if we can successfully identify  $(\mathbf{c},\tilde{\mathbf{s}})$ , we could perform counterfactual reasoning by manipulating  $\tilde{\mathbf{s}}$  while preserving both the content information and the content-style dependence.
4 Identifiability of the Latent Variables
In this section, we introduce the identification theory for the content  $\mathbf{c}$  and the style  $\mathbf{s}$  sequentially and then discuss their implications for methodological development.
We introduce notations and definitions that we use throughout this work. When working with matrices, we adopt the following indexing notations: for a matrix  $\mathbf{M}$ , the  $i$ -th row is denoted as  $\mathbf{M}_{i,:}$ , the  $j$ -th column is denoted as  $\mathbf{M}_{j,j}$ , and the  $(i,j)$  entry is denoted as  $[\mathbf{M}]_{i,j}$ . We can also use this notation to refer to specific sets of indices within a matrix. For an index set  $\mathcal{I} \subseteq \{1,\dots,m\} \times \{1,\dots,n\}$ , we use  $\mathcal{I}_{i,:}$  to denote the set of column indices whose row coordinate is  $i$ , i.e.,  $\mathcal{I}_{i,:} := \{j:(i,j) \in \mathcal{I}\}$ , and analogously  $\mathcal{I}_{j,j}$  to denote the set of row indices whose column coordinate is  $j$ .
In addition, we define a subspace of  $\mathbb{R}^n$  using an index set  $S$ :  $\mathbb{R}_S^n = \{\mathbf{z}\in \mathbb{R}^n|\forall i\notin S,z_i = 0\}$ , i.e., it consists of all vectors in  $\mathbb{R}^n$  whose entries are zero for all indices not in  $S$ . Finally, we can define the support of a matrix-valued function  $\mathbf{M}(\mathbf{x}): \mathcal{X} \to \mathbb{R}^{m\times n}$  as the set of indices whose corresponding entries are nonzero for some input value  $\mathbf{x}$ , i.e.,  $\mathrm{Supp}(\mathbf{M}) := \{(i,j):\exists \mathbf{x}\in \mathcal{X},\mathrm{s.t.},[\mathbf{M}(\mathbf{x})]_{i,j}\neq 0\}$ .
4.1 Influence Sparsity for Content Identification
We show that the subspace  $\mathbf{c}$  can be identified. That is, we can estimate a generative model  $(p_{\hat{c}}, p_{\hat{s}|\hat{c}}, \hat{g})^2$  following the data-generating process in Equation 1 and the estimated variable  $\hat{\mathbf{c}}$  can capture all the information of  $\mathbf{c}$  without interference from  $\mathbf{s}$ . In the following, we denote the Jacobian matrices  $\mathbf{J}_g(\mathbf{z})$ 's and  $\mathbf{J}_{\hat{g}}(\hat{\mathbf{z}})$ 's supports as  $\mathcal{G}$  and  $\hat{\mathcal{G}}$  respectively. Further, we denote as  $\mathcal{T}$  a set of matrices with the same support as that of the matrix-valued function  $\mathbf{J}_g^{-1}(\mathbf{c})\mathbf{J}_{\hat{g}}(\hat{\mathbf{c}})$ .
Assumption 1 (Content identification).
Figure 2: Sparsity in  $\mathbf{J}_g$
i.  $g$  is smooth and invertible and its inverse  $g^{-1}$  is also smooth.
ii. For all  $i \in \{1, \ldots, d_x\}$ , there exist  $\{\mathbf{z}^{(\ell)}\}_{\ell=1}^{|G_{i,:}|}$  and  $\mathbf{T} \in \mathcal{T}$ , such that  $\text{span}(\{\mathbf{J}_g(\mathbf{z}^{(\ell)})_{i,:}\}_{\ell=1}^{|G_{i,:}|}) = \mathbb{R}_{\mathcal{G}_{i,:}}^{d_z}$  and  $[\mathbf{J}_g(\mathbf{z}^{(\ell)})\mathbf{T}]_{i,:} \in \mathbb{R}_{\hat{\mathcal{G}}_{i,:}}^{d_z}$ .
iii. For every pair  $(c_{j_c}, s_{j_s})$  with  $j_c \in [d_c]$  and  $j_s \in \{d_c + 1, \ldots, d_z\}$ , the influence of  $s_{j_s}$  is sparser than that of  $c_{j_c}$ , i.e.,  $\| \mathcal{G}_{\cdot,j_c} \|_0 > \| \mathcal{G}_{\cdot,j_s} \|_0$ .
Theorem 1. We assume the data-generating process in Equation 1 with Assumption 1. If for given dimensions  $(d_c, d_s)$ , a generative model  $(p_{\hat{c}}, p_{\hat{s}|\hat{c}}, \hat{g})$  follows the same generating process and achieves the following objective:
$$
\arg \min  _ {p _ {\hat {c}}, p _ {\hat {s}}, \hat {g}} \sum_ {j _ {\hat {s}} \in \left\{d _ {c} + 1, \dots , d _ {z} \right\}} \left\| \hat {\mathcal {G}} _ {., j _ {\hat {s}}} \right\| _ {0} \quad \text {s u b j e c t t o :} p _ {\hat {\mathbf {x}}} (\mathbf {x}) = p _ {\mathbf {x}} (\mathbf {x}), \forall \mathbf {x} \in \mathcal {X}, \tag {3}
$$
then the estimated variable  $\hat{\mathbf{c}}$  is an one-to-one mapping of the true variable  $\mathbf{c}$ . That is, there exists an invertible function  $h_c(\cdot)$  such that  $\hat{\mathbf{c}} = h_c(\mathbf{c})$ .
A proof can be found in Appendix A.5.
Interpretation. Theorem 1 states that by matching the marginal distribution  $p_{\mathbf{x}}(\mathbf{x})$  under a sparsity constraint of  $\hat{\mathbf{s}}$  subspace, we can successfully eliminate the influence of  $\mathbf{s}$  from the estimated  $\hat{\mathbf{c}}$ . This warrants that the content information can be fully retained without being entangled with the style information for a successful counterfactual generation. We can further identify  $\tilde{\mathbf{s}}$  from that of  $\mathbf{s}$  when the dependence  $g_{s}$  function is invertible in its argument  $\tilde{\mathbf{s}}$  [Kong et al., 2022].
Discussion on assumptions. Assumption 1-i. ensures that the information of all latent variables  $[\mathbf{c},\mathbf{s}]$  is preserved in the observed variables  $\mathbf{x}$ , which is a necessary condition for latent-variable identification [Hyvarinen et al., 2019, Kong et al., 2022]. Assumption 1-ii. ensures that the influence from the latent variable  $\mathbf{z}$  varies sufficiently over its domain. This excludes degenerate cases where the Jacobian matrix is partially constant, and thus, its support fails to faithfully capture the influence between latent variables and the observed variables. Assumption 1-iii. encodes the observation that the s subspace exerts a relatively sparser influence on the observed data than the subspace c (Fig 2.).
As our theory is not contingent on the availability of multiple domains, we drop the domain index  $\mathbf{u}$  in our notations in this section for ease of exposition.
This is reasonable for language, where the main event largely predominant the sentence and the stylistic variable play complementary and local information for particular attributes, e.g., tense, sentiment, and formality [Xu et al., 2019a, Wang et al., 2021, Ross et al., 2021]. For the language data, x corresponds to a piece of text (e.g., a sentence) with its dimension  $d_{\mathbf{x}}$  equal to the number of words multiplied by the word embedding dimension, i.e., multiple dimensions of x correspond to a single word. Therefore, even if a word is simultaneously influenced by both c and s, the influence from the content c tends to be denser on this word's embedding dimension, as content usually takes precedence in word selection over style.
Contrast with prior work. Zheng et al. [2022] impose sparse influence constraints on the generating function  $g$  in an absolute sense – each latent component should have a very sparse influence on the observed data. In contrast, Theorem 1 only calls for relative sparsity between two subspaces where each latent component's influence may not be sparse and unique as in Zheng et al. [2022]. We believe this is reasonable for many real-world applications like languages. Kong et al. [2022] assume the independence between the two subspaces and identify the content subspace by resorting to its invariance and sufficient variability of the style subspace over multiple domains. However, as discussed in Section 3, the invariance of the content subspace is often violated, and so is the independence assumption. In contrast, we permit the content subspace to vary over domains and allow for the dependence between the two subspaces.
4.2 Partially Intersecting Influences for Style Identification
In this section, we show the identifiability for the style subspace  $\mathbf{s}$ , under one additional condition: the influences from the two subspaces  $\mathbf{c}$  and  $\mathbf{s}$  do not fully overlap.
Assumption 2 (Partially intersecting influence supports). For every pair  $(c_{j_c}, s_{j_s})$ , the supports of their influences on  $\mathbf{x}$  do not fully intersect, i.e.,  $\| \mathcal{G}_{:,j_c} \cap \mathcal{G}_{:,j_s} \|_0 < \min \{\| \mathcal{G}_{:,j_c} \|_0, \| \mathcal{G}_{:,j_s} \|_0\}$ .
Theorem 2. We follow the data-generating process Equation 1 and Assumption 1 and Assumption 2. We optimize the objective function in Equation 3 together with
$$
\min  _ {\left(j _ {\hat {c}}, j _ {\hat {s}}\right) \in \{1, \dots , d _ {c} \} \times \left\{d _ {c} + 1, \dots , d _ {z} \right\}} \left\| \hat {\mathcal {G}} _ {:, j _ {\hat {c}}} \cap \hat {\mathcal {G}} _ {:, j _ {\hat {s}}} \right\| _ {0}. \tag {4}
$$
The estimated style variable  $\hat{\mathbf{s}}$  is a one-to-one mapping to the true variable  $\mathbf{s}$ . That is, there exists an invertible mapping  $h_s(\cdot)$  between  $\mathbf{s}$  and  $\hat{\mathbf{s}}$ , i.e.,  $\hat{\mathbf{s}} = h_s(\mathbf{s})$ .
The proof can be found in Appendix A.6.
Interpretation. Theorem 2 states that we can recover the style subspace s if the influences from the two subspaces do not interfere with each other (Fig 2). This condition endows the subspaces distinguishing footprints and thus forbids the content information in c from contaminating the estimated style variable  $\hat{\mathbf{s}}$ . The identification of s is crucial to counterfactual generation tasks: if the estimated style variable  $\hat{\mathbf{s}}$  does capture all the true style variable s, intervening on  $\hat{\mathbf{s}}$  cannot fully alter the original style that is intended to be changed.