SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\frac {\partial f}{\partial \mu_ {p i}} = \frac {2 \left(\mu_ {p i} - \mu_ {q i}\right)}{\sigma_ {p i} ^ {2}} + 2 \mu_ {p i} \tag {16}
$$
$$
\frac {\partial f}{\partial \sigma_ {p i}} = \frac {- 2 \left(\sigma_ {q i} ^ {2} + \left(\mu_ {q i} - \mu_ {p i}\right) ^ {2}\right)}{\sigma_ {p i} ^ {3}} + 2 \sigma_ {p i}. \tag {17}
$$
When  $\mu_{pi} = \mu_{qi}$  and  $\sigma_{pi} = \sigma_{qi}$ , both partial derivative is not 0. So it is not the stationary point of  $f$ , then it won't be the minima.
A.3 Language Modeling
We investigate the training procedure for different models. We plot the MI  $I_{q}$ ,  $D_{KL}$  in the ELBO and the distance between the approximated posterior and the prior,  $D_{KL}(q_{\phi}(z)||p(z))$ . As in Eq. 4 in the main paper,  $D_{KL}$  in the ELBO is the sum of the other two. Fig. 3 shows these three values throughout the training. Although  $D_{KL}$  is the upper bound of the mutual information, we notice that the gap is usually large. In the initial training stage,  $D_{KL}$  increases in the basic VAE with annealing, while its MI remains small. With the weight decreases, the method finally suffers from posterior collapse. In contrast, our approach can obtain a high MI with a small  $D_{KL}$  value like aggressive VAE. The full results on language modeling are in Table 8.
A.4 CVAE for dialogue generation
Human evaluation: We evaluate the generated responses from three aspects: relevance, fluency and informativeness. Here we introduce the criteria of the evaluation as shown in Table 7. We sample 200 conversations from the test set. For each conversation, we sample three generated responses from each model, totally 600 responses.
Case study: We report 4 examples generated from these three models, shown in Table 9. CVAE (BOW) and our approach both can generate diverse responses. However, responses from ours are more related to the context compared with the other two.
Figure 1: Visualization of 500 sampled data from the synthetic dataset during the training. The x-axis is  $\mu_{x,\theta}$ , the approximate model posterior mean. The y-axis is  $\mu_{x,\phi}$ , which represents the inference posterior mean. b is batch size and  $\gamma$  is 1 in BN.
Figure 2: Visualization of our BN-VAE on different  $\gamma$  for synthetic data.
Figure 3: Training behavior on Yelp. Left/Middle/Right: VAE/Agg-VAE/BN-VAE (all models are with annealing).
Table 7: Human evaluation criteria.
Fluency	Relevance	Informativeness
1 Point	1. Hard to understand
2. Too many syntax mistakes	Not related to the query at all	1. Generic responses.
2. Repeated query.
2 Points	1. Several syntax mistakes but still understandable
2. short responses, e.g., Generic responses	1. Response and query are in the same domain/topic but are not directly related
2. Generic responses	between 1 and 3.
3 Points	Only few syntax mistakes with a moderate length	closely related to the query	1. Creative responses.
2. Contain new information about the query.
Model	Yahoo	Yelp
NLL	KL	MI	AU	NLL	KL	MI	AU
CNN-VAE	≤332.1	10.0	-	-	≤359.1	7.6	-	-
LSTM-LM	328	-	-	-	351.1	-	-	-
VAE	328.6	0.0	0.0	0.0	357.9	0.0	0.0	0.0
β-VAE (0.2)	332.2	19.1	3.3	20.4	360.7	11.7	3.0	10.0
β-VAE (0.4)	328.7	6.3	2.8	8.0	358.2	4.2	2.0	4.2
β-VAE (0.6)	328.5	0.3	0.0	1.0	357.9	0.2	0.1	3.8
β-VAE (0.8)	328.8	0.0	0.0	0.0	358.1	0.0	0.0	0.0
cyclic*	330.6	2.1	2.0	2.3	359.5	2.0	1.9	4.1
Skip-VAE*	328.5	2.3	1.3	8.1	357.6	1.9	1.0	7.4
SA-VAE	327.2	5.2	2.7	9.8	355.9	2.8	1.7	8.4
Agg-VAE	326.7	5.7	2.9	15.0	355.9	3.8	2.4	11.3
FB (4)	331.0	4.1	3.8	3.0	359.2	4.0	1.9	32.0
FB (5)	330.6	5.7	2.0	3.0	359.8	4.9	1.3	32.0
δ-VAE (0.1)*	330.7	3.2	0.0	0.0	359.8	3.2	0.0	0.0
δ-VAE (0.15)*	331.6	4.8	0.0	0.0	360.4	4.8	0.0	0.0
δ-VAE (0.2)*	332.2	6.4	0.0	0.0	361.5	6.4	0.0	0.0
δ-VAE (0.25)*	333.5	8.0	0.0	0.0	362.5	8.0	0.0	0.0
vMF-VAE (13)*	327.4	2.0	-	32.0	357.5	2.0	-	32.0
vMF-VAE (16)*	328.5	3.0	-	32.0	367.8	3.0	-	32.0
vMF-VAE (20)*	329.4	4.0	-	32.0	358.0	4.0	-	32.0
vMF-VAE (23)*	328.7	5.0	-	32.0	357.3	5.0	-	32.0
vMF-VAE (25)*	330.1	6.0	-	32.0	357.8	6.0	-	32.0
vMF-VAE (30)*	329.5	7.0	-	32.0	357.8	7.0	-	32.0
BN-VAE (0.3)*	328.1	1.6	1.4	32.0	356.7	1.7	1.4	32.0
BN-VAE (0.4)*	327.7	2.7	2.2	32.0	356.2	3.1	2.5	32.0
BN-VAE (0.5)*	327.4	4.2	3.3	32.0	356.4	4.4	3.8	32.0
BN-VAE (0.6)*	326.7	6.2	5.6	32.0	356.5	6.5	5.4	32.0
BN-VAE (0.7)*	327.4	8.8	7.4	32.0	355.9	9.1	7.4	32.0
Pretrained encoder
+cyclic*	333.1	25.8	9.1	32.0	361.5	20.5	9.3	32.0
+FB (2)*	327.2	4.3	3.8	32.0	356.6	4.6	4.2	32.0
+FB (3)*	327.1	4.5	3.9	32.0	356.3	5.8	5.2	32.0
+FB (4)*	326.2	8.1	6.8	32.0	356.0	7.6	6.6	32.0
+FB (5)*	326.6	8.9	7.3	32.0	356.5	9.0	7.4	32.0
+FB (6)*	326.6	10.8	8.1	32.0	356.5	12.0	8.6	32.0
+FB (7)*	326.6	12.1	8.5	32.0	356.8	13.4	8.9	32.0
+FB (8)*	326.7	13.6	8.9	32.0	357.5	15.8	9.2	32.0
+δ-VAE (0.15)*	331.0	5.6	1.1	11.2	359.4	5.2	0.5	5.9
vMF-VAE (13)*	328.4	2.0	-	32.0	357.0	2.0	-	32.0
+BN-VAE (0.6)*	326.7	6.4	5.8	32.0	355.5	6.6	5.9	32.0
+BN-VAE (0.7)*	326.5	9.1	7.6	32.0	355.7	9.1	7.5	32.0
Table 8: Results on Yahoo and Yelp datasets. We report mean values across 5 different random runs. * indicates the results are from our experiments, while others are from previous report.
Table 9: Sampled generated responses. Only the last sentence in the context is shown here.
Topic: ETHICS IN GOVERNMENT
Context: have trouble drawing lines as to what's illegal and what's not
Target (statement): well i mean the other problem is that they'are always up for
CVAE1. yeah2. yeah3. yeah	CVAE (BOW)1. yeah2. oh yeah, they're not3. no it's not too bad	BN-CVAE1. it's not a country2. it is the same thing that'swhat i think is about the stateis a state3. yeah it's
Topic: VACATION SPOTS
Context: well i 'll talk to you later
Target (conventional-closing): okay now do you push the buttons now
CVAE1. okay bye - bye2. bye3. okay bye - bye	CVAE (BOW)1. okay so we'll go ahead and start2. so i guess it depends on how muchyou are you3. it's	BN-CVAE1. okay bye - bye2. nice talking to you too3. all right take care bye - bye
Topic: RECYCLING
Context: are they doing a lot of recycling out in georgia
Target (statement-non-opinion): well at my workplace they are we have places foraluminum cans and we have everybody's been unk a separate trash can for
CVAE1. yeah2. yeah3. yeah	CVAE (BOW)1. well that's good for a while i'll tell youthat you're not doing anything at ti andthen you're in a small town2. oh i know i've got a lot of trees andtrees and stuff and3. yeah it's like you know people thatwant to be unk and they're not goingto bother you to make a mess	BN-CVAE1. well we do recyclenewspapers2. yes i do too3. well we're at a point wherewe're going to be a landfillspace
Topic: UNIVERSAL HEALTH INS
Context: some of the good obviously that nobody has to worry about health care
Target (statement-non-opinion): and i guess i'll have to help with grandchildrenone of these days i hope
CVAE1. um - hum2. uh - huh3. uh - huh	CVAE (BOW)1. okay well see we don't have anychoice of any of those2.um - hum3. yeah	BN-CVAE1. well i hope that we shouldhave a balanced budget2. uh - huh3. well that's a good idea