SlowGuess/ACL_2020_OCR huggingface dataset:
Personality Captioning: Personality Captioning dataset (Shuster et al., 2019) collects captions of images which are written according to 215 different personality traits. These traits can be divided into three categories: positive, neutral, and negative. We select sentences from positive and negative classes for evaluation.
We build the sentence encoder  $q_{\theta}(s, c | x)$  with a one-layer bi-directional LSTM plus a multi-head attention mechanism. The style classifier  $q_{\psi}(y | s)$  is parameterized by a single fully-connected network with the softmax activation. The content-based decoder  $q_{\phi}(x | c)$  is a one-layer uni-directional LSTM
Figure 3: Latent spaces t-SNE plots of IDEL on Yelp.
Figure 4: t-SNE plots of IDEL  $-$  without  $\hat{\mathbf{I}}(s; c)$ .
appended with a linear layer with vocabulary size output, outputting the predicted probability of the next words. The conditional distribution approximation  $p_{\sigma}(s|c)$  is represented by a two-layer fully-connected network with ReLU activation. The generator  $p_{\gamma}(\boldsymbol{x}|s,\boldsymbol{c})$  is built by a two-layer unidirectional LSTM plus a linear projection with output dimension equal to the vocabulary size, providing the next-word prediction based on previous sentence information and the current word.
We initialize and fix our word embeddings by the 300-dimensional pre-trained GloVe vectors (Pennington et al., 2014). The style embedding dimension is set to 32 and the content embedding dimension is 512. We use a standard multivariate normal distribution as the prior of the latent spaces. We train the model with the Adam optimizer (Kingma and Ba, 2014) with initial learning rate of  $5 \times 10^{-5}$ . The batch size is equal to 128.
5.3 Embedding Disentanglement Quality
We first examine the disentangling quality of learned latent embeddings, primarily studying the latent spaces of IDEL on the Yelp dataset.
Latent Space Visualization: We randomly select 1,000 sentences from the Yelp testing set and visualize their latent embeddings in Figure 3, via t-SNE plots (van der Maaten and Hinton, 2008). The blue and red points respectively represent the positive and negative sentences. The left side of the figure shows the style embedding space, which is
well separated into two parts with different colors. It supports the claim that our model learns a semantically meaningful style embedding space. The right side of the figure is the content embedding space, which cannot be distinguished by the style labels (different colors). The lack of difference in the pattern of content embedding also provides evidence that our content embeddings have little correlation with the style labels.
For an ablation study, we train another IDEL model under the same setup, while removing our MI upper bound  $\hat{\mathrm{I}}(s; c)$ . We call this model  $\mathrm{IDEL}^{-}$  in the following experiments. We encode the same sentences used in Figure 3, and display the corresponding embeddings in Figure 4. Compared with results from the original IDEL, the style embedding space (left in Figure 4) is not separated in a clean manner. On the other hand, the positive and negative embeddings become distinguishable in the content embedding space. The difference between Figures 3 and 4 indicates the disentangling effectiveness of our MI upper bound  $\hat{\mathrm{I}}(s; c)$ .
Label-Embedding Correlation: Besides visualization, we also numerically analyze the correlation between latent embeddings and style labels. Inspired by the statistical two-sample test (Gretton et al., 2012), we use the sample-based divergence between the positive embedding distribution  $p(c|y = 1)$  and the negative embedding distribution  $p(c|y = 0)$  as a measurement of label-embedding correlation. We consider four divergences: Mean Absolute Deviation (MAD) (Geary, 1935), Energy Distance (ED) (Sejdinovic et al., 2013), Maximum Mean Discrepancy (MMD) (Gretton et al., 2012), and Wasserstein distance (WD) (Ramdas et al., 2017). For a fair comparison, we re-implement previous text embedding methods and set their content embedding dimension to 512 and the style embedding dimension to 32 (if applicable). Details about the divergences and embedding processing are shown in the Supplementary Material.
From Table 2, the proposed IDEL achieves the lowest divergences between positive and negative content embeddings compared with CtrlGen (Hu et al., 2017), CAAE (Shen et al., 2017), ARAE (Zhao et al., 2018), BackTranslation (BT) (Lample et al., 2019), and DRLST (John et al., 2019), indicating our model better disentangles the content embeddings from the style labels. For style embeddings, we compare IDEL with DRLST, the
Yelp Dataset	Personality Captioning Dataset
Conditional Generation	Style Transfer	Conditional Generation	Style Transfer
ACC	BLEU	GM	ACC	BLEU	S-BLEU	GM	ACC	BLEU	GM	ACC	BLEU	S-BLEU	GM
CtrlGen	82.5	20.8	41.4	83.4	19.4	31.4	37.0	73.6	18.9	37.0	73.3	18.9	30.0	34.6
CAAE	78.9	19.7	39.4	79.3	18.5	28.2	34.6	72.2	19.5	37.5	72.1	18.3	27.4	33.1
ARAE	78.3	23.1	42.4	78.5	21.3	32.5	37.9	72.8	22.5	40.4	71.5	20.4	31.6	35.8
BT	81.4	20.2	40.5	86.3	24.1	35.6	41.9	74.1	21.0	39.4	75.9	23.1	34.2	39.1
DRLST	83.7	22.8	43.7	85.0	23.9	34.9	41.4	74.9	22.0	40.5	75.7	21.9	33.8	38.3
IDEL-	78.1	20.3	39.8	79.1	20.1	27.5	35.1	72.0	19.7	37.7	72.4	19.7	27.1	33.8
IDEL	83.9	23.0	43.9	85.7	24.3	35.2	41.9	75.1	22.3	40.9	75.6	23.3	34.6	39.4
Table 1: Performance comparison of text DRL models. For conditional generation, the GM scores are calculated over ACC and BLEU. For style transfer, the GMs are calculated over ACC, BLEU, S-BLEU(self-BLEU).
Method	MAD	ED	WD	MMD
CtrlGen	0.261	0.105	0.311	0.063
CAAE	0.285	0.112	0.306	0.078
ARAE	0.194	0.050	0.248	0.042
BT	0.211	0.053	0.269	0.049
DRLST	0.181	0.048	0.215	0.031
IDEL-	0.217	0.077	0.293	0.051
IDEL	0.063	0.015	0.084	0.010
Table 2: Sample divergences between positive and negative content embeddings.
Method	MAD	ED	WD	MMD
DRLST	1.024	0.503	1.375	0.286
IDEL-	0.996	0.489	1.124	0.251
IDEL	1.167	0.583	1.392	0.302
Table 3: Sample divergences between positive and negative style embeddings.
only prior method that infers the text style embeddings. Table 3 shows a larger distribution gap between positive and negative style embeddings with IDEL than with DRLST, which demonstrates the proposed IDEL has better style information expression in the style embedding space. The comparison between IDEL and  $\mathrm{IDEL^{-}}$  supports the effectiveness of our MI upper bound minimization.
5.4 Embedding Representation Quality
To show the representation ability of IDEL, we conduct experiments on two text-generation tasks: style transfer and conditional generation.
For style transfer, we encode two sentences into a disentangled representation, and then combine the style embedding from one sentence and the content embedding from another to generate a new sentence via the generator  $p_{\gamma}(\boldsymbol{x}|\boldsymbol{s}, \boldsymbol{c})$ . For conditional generation, we set one of the style or content embeddings to be fixed and sample the other part from the latent prior distribution, and then use the combination to generate text. Since most previous work only embedded the content information, for
fair comparison, we mainly focus on fixing style and sampling context embeddings under the conditional generation setup.
To measure generation quality for both tasks, we test the following metrics (more specific description is provided in the Supplementary Material).
Style Preservation: Following previous work (Hu et al., 2017; Shen et al., 2017; John et al., 2019), we pre-train a style classifier and use it to test whether a generated sentence can be categorized into the correct target style class.
Content Preservation: For style transfer, we measure whether a generation preserves the content information from the original sentence by the self-BLEU score (Zhang et al., 2019, 2020). The self-BLEU is calculated between one original sentence and its style-transferred sentence.
Generation Quality: To measure the generation quality, we calculate the corpus-level BLEU score (Papineni et al., 2002) between a generated sentence and the testing data corpus.
Geometric Mean: We use the geometric mean (GM) (John et al., 2019) of the above metrics to obtain an overall evaluation metric of representiveness of DRL models.
We compare our IDEL with previous state-of-the-art methods on Yelp and Personality Captioning datasets, as shown in Table 1. The references to the other models are mentioned in Section 5.3. Note that the original BackTranslation (BT) method (Lample et al., 2019) is a AutoEncoder framework, that is not able to do conditional generation. To compare with BT fairly, we add a standard Gaussian prior in its latent space to make it a variational auto-encoder model.
From the results in Table 1, ARAE performs well on the conditional generation. Compared to ARAE, our model performance is slightly lower on content preservation (BLEU). In contrast, the style
7536
Content Source	Style Source	Transferred Result
I enjoy it thoroughly!	never before had a bad experience at the habit until tonight.	I dislike it thoroughly.
quality is just so so.	quality is so bad.
I am so grateful.	I am so disgusted.
never before had a bad experience at the habit until tonight.	I am so grateful.	never had a service that was enjoyable experience tonight.
quality is just so so.	never had a unimpressed experience until tonight.
quality of food is fantastic.	never had awesome routine until tonight.
I am so disappointed with palm today.	we were both so impressed.	I am so impressed with palm again.
quality of food is fantastic.	I am good with palm today.
never before had a bad experience at the habit until tonight.	I am so disgusted with palm today.
Table 4: Examples of text style transfer on Yelp dataset. The style-related words are bold.
SA	CP	SF	GM
CtrlGen	71.2 (3.56)	3.25	3.12	3.30
CAAE	63.1 (3.16)	2.83	3.06	3.01
ARAE	68.0 (3.40)	3.44	3.09	3.31
IDEL	73.7 (3.69)	3.39	3.21	3.42
classification score of IDEL has a large margin above that of ARAE. The BackTranslation (BT) has a better performance on style transfer tasks, especially on the Yelp dataset. Our IDEL has a lower style classification accuracy (ACC) than BT on the style transfer task. However, IDEL achieves high BLEU on style transfer, which leads to a high overall GM score on the Personality-Captioning dataset. On the Yelp dataset, IDEL also has a competitive GM score compared with BT. The experiments show a clear trade-off between style preservation and content preservation, in which our IDEL learns more representative disentangled representation and leads to a better balance.
Besides the automatic evaluation metrics mentioned above, we further test our disentangled representation effectiveness by human evaluation. Due to the limitation of manual effort, we only evaluate the style transfer performance on Yelp datasets. The generated sentences are manually evaluated on style accuracy (SA), content preservation (CP), and sentence fluency (SF). The CP and SF scores are between 0 to 5. Details are provided in the Supplementary Material. Our method achieves better style and content preservation, with a little performance sacrifice on sentence fluency.
Table 4 shows three style transfer examples from IDEL on the Yelp dataset. The first example shows three sentences transferred with the style from a given sentence. The other two examples transfer each given sentence based on the styles of three different sentences. Our IDEL not only transfers sentences into target sentiment classes, but also
Table 5: Manual evaluation for style transfer on Yelp. The style accuracy (SA) scores are scaled in range [0, 5] for compatible calculation of geometric mean (GM).
ACC	BLEU	S-BLEU	GM
LVAE	52.1	24.7	20.8	29.9
LVAE + I(s;y)	86.1	23.3	16.4	32.0
LVAE + I(x;c)	50.2	24.0	36.3	34.7
IDEL-	79.1	20.1	27.5	35.1
IDEL*	85.5	24.0	35.0	41.5
IDEL	85.7	24.3	35.2	41.9
Table 6: Ablation tests for style transfer on Yelp.
renders the sentence with more detailed style information (e.g., the degree of the sentiment).
In addition, we conduct an ablation study to test the influence of different objective terms in our model. We re-train the model with different training loss combinations while keeping all other setups the same. In Table 1, IDEL surpasses  $\mathrm{IDEL}^{-}$  (without MI upper bound minimization) with a large gap, demonstrating the effectiveness of our proposed MI upper bound. The vanilla VAE has the best generation quality. However, its transfer style accuracy is slightly better than a random guess. When adding  $\operatorname {I}(s;y)$ , the ACC score significantly improves, but the content preservation (S-BLEU) becomes worse. When adding  $\operatorname {I}(c;x)$ , the content information is well preserved, while the ACC even decreases. By gradually adding MI terms, the model performance becomes more balanced on all the metrics, with the overall GM monotonically increasing. Additionally, we test the influence of the stochastic calculation of  $R_{j}$  in Algorithm 1 (IDEL) with the closed form from Theorem 3.1  $(\mathrm{IDEL}^{*})$ . The stochastic IDEL not only accelerates the training but also gains a performance improvement relative to  $\mathrm{IDEL}^*$ .
We have proposed a novel information-theoretic disentangled text representation learning framework. Following the theoretical guidance from information theory, our method separates the textual information into independent spaces, constituting style and content representations. A sample-based
7537
mutual information upper bound is derived to help reduce the dependence between embedding spaces. Concurrently, the original text information is well preserved by maximizing the mutual information between input sentences and latent representations. In experiments, we introduce several two-sample test statistics to measure label-embedding correlation. The proposed model achieves competitive performance compared with previous methods on both conditional generation and style transfer. For future work, our model can be extended to disentangled representation learning with non-categorical style labels, and applied to zero-shot style transfer with newly-coming unseen styles.
This work was supported by NEC Labs America, and was conducted while the first author was doing an internship at NEC Labs America.
David Barber and Felix V Agakov. 2003. The im algorithm: a variational approach to information maximization. In Advances in neural information processing systems, page None.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Devon Hjelm, and Aaron Courville. 2018. Mutual information neural estimation. In International Conference on Machine Learning, pages 530-539.
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. 2018. Understanding disentangling in beta-vae. arXiv preprint arXiv:1804.03599.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. 2018. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, pages 2610-2620.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems, pages 2172-2180.
Juchieh Chou, Cheng chieh Yeh, Hung yi Lee, and Lin shan Lee. 2018. Multi-target voice conversion without parallel data by adversarially learning disentangled audio representations. In Proc. Interspeech 2018, pages 501-505.
Thomas M Cover and Joy A Thomas. 2012. Elements of information theory. John Wiley & Sons.
Emily L Denton et al. 2017. Unsupervised learning of disentangled representations from video. In Advances in neural information processing systems, pages 4414-4423.
Roy C Geary. 1935. The ratio of the mean deviation to the standard deviation as a test of normality. Biometrika, 27(3/4):310-332.
Behnam Gholami, Prishit Sahu, Ognjen Rudovic, Konstantinos Bousmalis, and Vladimir Pavlovic. 2020. Unsupervised multi-target domain adaptation: An information theoretic approach. IEEE Transactions on Image Processing, 29:3993-4002.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Scholkopf, and Alexander Smola. 2012. A kernel two-sample test. Journal of Machine Learning Research, 13(Mar):723-773.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. 2018. Learning to decompose and disentangle representations for video prediction. In Advances in Neural Information Processing Systems, pages 517-526.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1587-1596. JMLR.org.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv:1412.6980v9.
Diederik P Kingma and Max Welling. 2013. Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114.
Alexander Kraskov, Harald Stögbauer, Ralph G Andrzejak, and Peter Grassberger. 2005. Hierarchical clustering using mutual information. *EPL (Europhysics Letters)*, 70(2):278.
Vinay Kumar Verma, Gundeep Arora, Ashish Mishra, and Piyush Rai. 2018. Generalized zero-shot learning via synthesized examples. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4281-4289.
Guillaume Lample, Sandeep Subramanian, Eric Smith, Ludovic Denoyer, Marc'Aurelio Ranzato, and Y-Lan Boureau. 2019. Multiple-attribute text rewriting. In International Conference on Learning Representations.
Hsin-Ying Lee, Hung-Yu Tseng, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. 2018. Diverse
7538
image-to-image translation via disentangled representations. In Proceedings of the European Conference on Computer Vision (ECCV), pages 35-51.
Yen-Cheng Liu, Yu-Ying Yeh, Tzu-Chien Fu, Sheng-De Wang, Wei-Chen Chiu, and Yu-Chiang Frank Wang. 2018. Detach and adapt: Learning cross-domain disentangled deep representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8867-8876.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Scholkopf, and Olivier Bachem. 2019. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pages 4114-4124.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing high-dimensional data using t-SNE. JMLR.
Marina Meilá. 2007. Comparing clusterings—an information based distance. Journal of multivariate analysis, 98(5):873-895.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. 2019. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171-5180.
Aaditya Ramdas, Nicolas Trillos, and Marco Cuturi. 2017. On Wasserstein two-sample testing and related families of nonparametric tests. Entropy, 19(2):47.
Dino Sejdinovic, Bharath Striperumbudur, Arthur Gretton, Kenji Fukumizu, et al. 2013. Equivalence of distance-based and rkhs-based statistics in hypothesis testing. The Annals of Statistics, 41(5):2263-2291.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830-6841.
Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. 2019. Engaging image captioning via personality. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12516-12526.
Naftali Tishby, Fernando C Pereira, and William Bialek. 2000. The information bottleneck method. arXiv preprint physics/0004057.
Luan Tran, Xi Yin, and Xiaoming Liu. 2017. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1415-1424.
Li Yingzhen and Stephan Mandt. 2018. Disentangled sequential autoencoder. In International Conference on Machine Learning, pages 5656-5665.
Ruiyi Zhang, Changyou Chen, Zhe Gan, Wenlin Wang, Dinghan Shen, Guoyin Wang, Zheng Wen, and Lawrence Carin. 2020. Improving adversarial text generation by modeling the distant future. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Ruiyi Zhang, Tong Yu, Yilin Shen, Hongxia Jin, and Changyou Chen. 2019. Text-based interactive recommendation via constraint-augmented reinforcement learning. In Advances in neural information processing systems, pages 15214-15224.
Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. 2018. Adversarily regularized autoencoders. In Proceedings of the 35th International Conference on Machine Learning, pages 5902-5911.
Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. 2019. Talking face generation by adversarially disentangled audio-visual representation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9299–9306.
7539
Proof of Theorem 3.1. First, we show that
$$
\mathbb {E} _ {p (\boldsymbol {s}, \boldsymbol {c})} [ \log p (\boldsymbol {s} | \boldsymbol {c}) ] - \mathbb {E} _ {p (\boldsymbol {s}) p (\boldsymbol {c})} [ \log p (\boldsymbol {s} | \boldsymbol {c}) ] \geq \mathrm {I} (\boldsymbol {s}; \boldsymbol {c}). \tag {9}
$$
Calculate the gap  $\Delta$  between the left-hand side and right-hand side of Eq. (9):
$$
\begin{array}{l} \Delta = \mathbb {E} _ {p (\boldsymbol {s}, \boldsymbol {c})} [ \log p (\boldsymbol {s} | \boldsymbol {c}) ] \\ - \mathbb {E} _ {p (\boldsymbol {s}) p (\boldsymbol {c})} [ \log p (\boldsymbol {s} | \boldsymbol {c}) ] - \mathrm {I} (\boldsymbol {s}; \boldsymbol {c}) \\ = \mathbb {E} _ {p (\boldsymbol {s}, \boldsymbol {c})} [ \log p (\boldsymbol {s} | \boldsymbol {c}) ] - \mathbb {E} _ {p (\boldsymbol {s}) p (\boldsymbol {c})} [ \log p (\boldsymbol {s} | \boldsymbol {c}) ] \\ - \mathbb {E} _ {p (\boldsymbol {s}, \boldsymbol {c})} [ \log p (\boldsymbol {s} | \boldsymbol {c}) - \log p (\boldsymbol {s}) ] \\ = \mathbb {E} _ {p (\boldsymbol {s}, \boldsymbol {c})} [ \log p (\boldsymbol {s}) ] - \mathbb {E} _ {p (\boldsymbol {s})} \mathbb {E} _ {p (\boldsymbol {c})} [ \log p (\boldsymbol {s} | \boldsymbol {c}) ] \\ = \mathbb {E} _ {p (\boldsymbol {s})} \left[ \log p (\boldsymbol {s}) - \mathbb {E} _ {p (\boldsymbol {c})} [ \log (p (\boldsymbol {s} | \boldsymbol {c}) ] \right] \\ = \mathbb {E} _ {p (\boldsymbol {s})} \left[ \log \left(\mathbb {E} _ {p (\boldsymbol {c})} [ p (\boldsymbol {s} | \boldsymbol {c}) ]\right) - \mathbb {E} _ {p (\boldsymbol {c})} [ \log p (\boldsymbol {s} | \boldsymbol {c}) ] \right] \\ \end{array}
$$
$\geq 0$  (Jensen's Inequality)
Therefore, the inequality in Eq. (9) holds.
Given sample pairs  $\{(s_j, c_j)\}_{j=1}^M \sim p(s, c)$ , the left-hand side of Eq. (9) has an unbiased estimation: