SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\begin{array}{l} \frac {1}{M} \sum_ {j = 1} ^ {M} \mathbb {E} _ {\left(\boldsymbol {s} _ {j}, \boldsymbol {c} _ {j}\right) \sim p (\boldsymbol {s}, \boldsymbol {c})} \left[ \log p \left(\boldsymbol {s} _ {j} \mid \boldsymbol {c} _ {j}\right) \right] \\ - \frac {1}{M ^ {2}} \sum_ {j = 1} ^ {M} \sum_ {k = 1} ^ {M} \mathbb {E} _ {\boldsymbol {s} _ {j} \sim p (\boldsymbol {s})} \mathbb {E} _ {\boldsymbol {c} _ {k} \sim p (\boldsymbol {c})} [ \log p (\boldsymbol {s} _ {j} | \boldsymbol {c} _ {k}) ] \\ = \mathbb {E} \left[ \frac {1}{M} \sum_ {j = 1} ^ {M} \left[ \log p (\boldsymbol {s} _ {j} | \boldsymbol {c} _ {j}) - \frac {1}{M} \sum_ {k = 1} ^ {M} \log p (\boldsymbol {s} _ {j} | \boldsymbol {c} _ {k}) \right] \right] \\ = \mathbb {E} \left[ \frac {1}{M} \sum_ {j = 1} ^ {M} R _ {j} \right], \\ \end{array}
$$
which is what we claim in Theorem 3.1.
Proof of Lower Bounds in Eq. (6).
$$
\begin{array}{l} \mathbf {I} (\boldsymbol {c}; \boldsymbol {x}) = \mathbb {E} _ {\boldsymbol {p} (\boldsymbol {x}, \boldsymbol {c})} [ \log p (\boldsymbol {x} | \boldsymbol {c}) - \log p (\boldsymbol {x}) ] \\ = \mathbf {H} (\boldsymbol {x}) + \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {c})} [ \log p (\boldsymbol {x} | \boldsymbol {c}) ] \\ = \mathrm {H} (\boldsymbol {x}) + \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {c})} [ \log p (\boldsymbol {x} | \boldsymbol {c}) - \log q _ {\phi} (\boldsymbol {x} | \boldsymbol {c}) ] \\ + \mathbb {E} _ {\boldsymbol {p} (\boldsymbol {x}, \boldsymbol {c})} [ \log q _ {\phi} (\boldsymbol {x} | \boldsymbol {c}) ] \\ = \mathrm {H} (\boldsymbol {x}) + \operatorname {K L} (p (\boldsymbol {x} | \boldsymbol {c}) \| q _ {\phi} (\boldsymbol {x} | \boldsymbol {c})) \\ + \mathbb {E} _ {\boldsymbol {p} (\boldsymbol {x}, \boldsymbol {c})} [ \log q _ {\phi} (\boldsymbol {x} | \boldsymbol {c}) ] \\ \geq H (\boldsymbol {x}) + \mathbb {E} _ {p (\boldsymbol {x}, \boldsymbol {c})} [ \log q (\boldsymbol {x} | \boldsymbol {c}) ]. \\ \end{array}
$$
The inequality is based on the fact that the KL-divergence is always non-negative. The lower bound for  $\operatorname{I}(s; y)$  can be also derived in the similar way.
B Sample-based Embedding Divergences
In this section we introduce the implementation details of the calculation about label-embedding correlation. As mentioned in Section 5.4, the distribution divergence between  $p(\boldsymbol{c}|\boldsymbol{y} = 0)$  and  $p(\boldsymbol{c}|\boldsymbol{y} = 1)$  measures the correlation between content embeddings and style labels. Assume  $c_1^{(0)}, c_2^{(0)}, \ldots, c_{N_0}^{(0)} \sim p(\boldsymbol{c}|\boldsymbol{y} = 0)$ , and  $c_1^{(1)}, c_2^{(1)}, \ldots, c_{N_1}^{(1)} \sim p(\boldsymbol{c}|\boldsymbol{y} = 1)$ , then the four metrics MAD, ED, WD, MMD are calculated based on the two groups of samples. With a ground distance  $d(\cdot, \cdot)$ , the implementation of the above four metrics are demonstrated in following:
$$
D _ {\mathrm {M A D}} = d \left(\frac {1}{N _ {0}} \sum_ {i = 1} ^ {N _ {0}} \boldsymbol {c} _ {i} ^ {(0)}, \frac {1}{N _ {1}} \sum_ {j = 1} ^ {N _ {1}} \boldsymbol {c} _ {j} ^ {(1)}\right). \tag {10}
$$
$$
\begin{array}{l} D _ {\mathrm {E D}} = \frac {2}{N _ {0} N _ {1}} \sum_ {i = 1} ^ {N _ {0}} \sum_ {j = 1} ^ {N _ {1}} d \left(\boldsymbol {c} _ {i} ^ {(0)}, \boldsymbol {c} _ {j} ^ {(1)}\right) \\ - \frac {1}{N _ {0} ^ {2}} \sum_ {i = 1} ^ {N _ {0}} \sum_ {j = 1} ^ {N _ {0}} d (\pmb {c} _ {i} ^ {(0)}, \pmb {c} _ {j} ^ {(0)}) \\ - \frac {1}{N _ {1} ^ {2}} \sum_ {i = 1} ^ {N _ {1}} \sum_ {j = 1} ^ {N _ {1}} d \left(\boldsymbol {c} _ {i} ^ {(1)}, \boldsymbol {c} _ {j} ^ {(1)}\right) \tag {11} \\ \end{array}
$$
$$
\begin{array}{l} D _ {\mathrm {W D}} = \min  _ {p _ {i j}} \sum_ {i = 1} ^ {N _ {0}} \sum_ {j = 1} ^ {N _ {1}} p _ {i j} d \left(\boldsymbol {c} _ {i} ^ {(0)}, \boldsymbol {c} _ {j} ^ {(1)}\right) \tag {12} \\ s. t. \sum_ {i = 1} ^ {N _ {0}} p _ {i j} = \frac {1}{N _ {1}}, \sum_ {j = 1} ^ {N _ {1}} p _ {i j} = \frac {1}{N _ {0}}. \\ \end{array}
$$
$$
\begin{array}{l} D _ {\mathrm {M M D}} = \frac {1}{N _ {0} ^ {2}} \sum_ {i = 1} ^ {N _ {0}} \sum_ {j = 1} ^ {N _ {0}} K (\pmb {c} _ {i} ^ {(0)}, \pmb {c} _ {j} ^ {(0)}) \\ + \frac {1}{N _ {1} ^ {2}} \sum_ {i = 1} ^ {N _ {1}} \sum_ {j = 1} ^ {N _ {1}} K (\boldsymbol {c} _ {i} ^ {(1)}, \boldsymbol {c} _ {j} ^ {(1)}) \\ - \frac {2}{N _ {0} N _ {1}} \sum_ {i = 1} ^ {N _ {0}} \sum_ {j = 1} ^ {N _ {1}} K \left(\boldsymbol {c} _ {i} ^ {(0)}, \boldsymbol {c} _ {j} ^ {(1)}\right), \tag {13} \\ \end{array}
$$
where  $K(\cdot ,\cdot)$  is a kernel function. Here we choose  $K(\cdot ,\cdot)$  from RBF kernel family with bandwidth  $w = 1$
For style embedding, the calculation formats are the same as in above equations. The style embeddings and content embeddings have different
dimensions, which leads to the ground metric  $d(\cdot ,\cdot)$  inconsistent. Therefore, instead of using Euclidean distance, we use the cosine distance as the ground metric.
C Detailed Experimental Setups
We set the dimension of style embedding to be smaller than the content embedding, because the content carries more information than the style of sentences. The hyper-parameter  $\beta$  in our loss function is a formal expression of re-weighting the two objectives of disentanglement and autoencoding. In practice, we vary it from 0 to 1 with step 0.1 during the first 10 training epochs. At the beginning of the training, the output latent embeddings are not representative enough. Therefore, we choose a small weight on the disentanglement term to avoid obstructing the learning of representative embeddings. After the latent embedding is sufficiently trained, which can successfully reconstruct the input sentences, we slowly enlarge  $\beta$  for the disentanglement. After  $\beta$  reaches 1, we fix it until all the training epochs are finished.
D Details in Representation Quality Evaluation
For style preservation, we pretrain a style classifier on each dataset. The style classifier is built by a one-layer LSTM appended with a multi-head attention layer. The number of the attention head is set to 6. The classifiers reach  $95\%$  prediction accuracy on Yelp and  $93\%$  prediction accuracy on PersonalityCaptioning. We input transferred sentences into the classifier and test whether the predicted style label is the same as the target style label.
For human evaluation, we transferred 1000 sentences with randomly selected style labels. After the transferring, we ask 10 human annotators to justify the style label, content preservation and content fluency. The style label is 0 or 1 representing the positive or negative sentiment of the given sentence. The content preservation and the content fluency is scored between 0 to 5. To make the style accuracy compatible with the other two scores, we scale it into range [0,5]. If the scores from the two annotators have a difference larger than 2, the scores will not be recorded. In this way, we ensure the evaluation criteria of annotators are similar.
7541