SlowGuess/NAACL_2022_OCR huggingface dataset:
Dataset	Model	Naturalness	TST Acc.	BLEU-2	METEOR	ROUGE-L	CIDEr	Embedding Average	Vector Extrema	Greedy Matching Score
SNLI (E to C)	β-VAE (β = 0.15)	0.982	0.806	0.438	0.270	0.555	2.455	0.950	0.630	0.823
LAAE (λl = 0.05)	0.947	0.837	0.036	0.050	0.160	0.027	0.818	0.320	0.591
DAAE (p = 0.3)	0.979	0.745	0.453	0.275	0.562	2.410	0.954	0.679	0.854
EPAAE (ζ = 2.5)	0.975	0.743	0.524	0.312	0.641	3.266	0.956	0.682	0.856
EPAAE (ζ = 2.5, p = 0.3)	0.975	0.749	0.497	0.300	0.606	2.845	0.956	0.698	0.866
EPAAE (ζ = 2.5, p = 0.1)	0.982	0.749	0.405	0.256	0.522	1.983	0.951	0.648	0.838
SNLI (C to E)	β-VAE (β = 0.15)	0.985	0.239	0.411	0.244	0.527	2.121	0.947	0.599	0.805
LAAE (λl = 0.05)	0.945	0.206	0.035	0.052	0.157	0.025	0.829	0.316	0.594
DAAE (p = 0.3)	0.982	0.292	0.378	0.233	0.506	1.714	0.946	0.622	0.823
EPAAE (ζ = 2.5),	0.979	0.227	0.488	0.289	0.615	2.860	0.956	0.659	0.843
EPAAE (ζ = 2.5, p = 0.1)	0.982	0.271	0.425	0.262	0.557	2.140	0.951	0.650	0.842
EPAAE (ζ = 2.5, p = 0.3)	0.984	0.277	0.370	0.225	0.486	1.582	0.945	0.603	0.815
Table B.10: Direction-wise TST metrics for the SNLI dataset. "Entailment" and "Contradiction" are denoted by "E" and "C" respectively.
Figure B.2: A TSNE plot of encoded latent vectors for all 1950 sentences in the Toy Dataset for the EPAAE model.
Dataset	Model	Naturalness	TST Acc.	BLEU-2	METEOR	ROUGE-L	CIDEr	Embedding Average	Vector Extrema	Greedy Matching Score
DNLI (E to C)	β-VAE (β = 0.15)	0.922	0.645	0.241	0.142	0.350	0.619	0.902	0.500	0.720
LAAE (λl = 0.05)	0.923	0.667	0.203	0.127	0.314	0.376	0.893	0.473	0.701
DAAE (p = 0.3)	0.937	0.687	0.399	0.223	0.496	2.128	0.925	0.593	0.790
EPAAE (ζ = 2.5)	0.934	0.536	0.441	0.252	0.554	2.694	0.941	0.636	0.818
EPAAE (ζ = 2.5,p = 0.1)	0.927	0.638	0.433	0.243	0.533	2.732	0.934	0.617	0.809
EPAAE (ζ = 2.5,p = 0.3)	0.923	0.665	0.390	0.218	0.492	2.048	0.925	0.590	0.790
DNLI (C to E)	β-VAE (β = 0.15)	0.948	0.516	0.245	0.146	0.363	0.689	0.908	0.507	0.724
LAAE (λl = 0.05)	0.932	0.546	0.208	0.129	0.323	0.391	0.900	0.477	0.707
DAAE (p = 0.3)	0.930	0.514	0.433	0.251	0.547	2.411	0.943	0.639	0.817
EPAAE (ζ = 2.5)	0.934	0.492	0.438	0.253	0.560	2.451	0.945	0.635	0.818
EPAAE (ζ = 2.5,p = 0.1)	0.928	0.519	0.482	0.277	0.585	3.092	0.947	0.654	0.830
EPAAE (ζ = 2.5,p = 0.3)	0.945	0.490	0.403	0.233	0.520	2.081	0.940	0.621	0.806
Table B.11: Direction-wise TST metrics for the DNLI dataset. "Entailment" and "Contradiction" are denoted by "E" and "C" respectively.
Dataset	Model	Naturalness	TST Acc.	BLEU-2	METEOR	ROUGE-L	CIDEr	Embedding Average	Vector Extrema	Greedy Matching Score
SciTail (E to N)	β-VAE (β = 0.15)	0.783	0.672	0.086	0.060	0.163	0.106	0.799	0.376	0.640
LAAE (λ1 = 0.05)	0.838	0.550	0.173	0.111	0.261	0.624	0.870	0.459	0.695
DAAE (p = 0.3)	0.804	0.514	0.300	0.185	0.383	1.375	0.918	0.579	0.792
EPAAE (ζ = 2.5)	0.842	0.479	0.351	0.210	0.445	1.941	0.931	0.618	0.816
EPAAE (ζ = 2.5, p = 0.1)	0.834	0.493	0.328	0.198	0.421	1.775	0.929	0.601	0.803
EPAAE (ζ = 2.5, p = 0.3)	0.829	0.571	0.234	0.149	0.314	0.905	0.899	0.517	0.747
SciTail (N to E)	β-VAE (β = 0.15)	0.748	0.397	0.103	0.069	0.183	0.178	0.792	0.399	0.647
LAAE (λ1 = 0.05)	0.823	0.408	0.219	0.138	0.314	0.881	0.878	0.519	0.729
DAAE (p = 0.3)	0.772	0.352	0.349	0.213	0.444	1.745	0.909	0.644	0.817
EPAAE (ζ = 2.5)	0.833	0.327	0.384	0.233	0.497	2.104	0.929	0.676	0.839
EPAAE (ζ = 2.5, p = 0.1)	0.816	0.351	0.375	0.231	0.488	2.087	0.934	0.669	0.834
EPAAE (ζ = 2.5, p = 0.3)	0.786	0.368	0.317	0.194	0.397	1.577	0.913	0.603	0.792
Table B.12: Direction-wise TST metrics for the SciTail dataset. "Entailment" and "Neutrality" are denoted by "E" and "N" respectively
or Passive) and Syntactic PP tag removal (PPR). In the Tenses dataset, each sentence is labelled with "Present", "Past" and "Future". The Voices dataset contains "Active" and "Passive" voices labels and the PPR dataset contains "PP removed" and "PP not removed" labels. The resultant sizes of the test, dev, test splits were 71k,8.8k,8.8k (tenses), 90k,11k,11k (PPR) and 44k,5.5k,5.5k (voices).
D More details on related work
Bowman et al. (2016) extend Variational AutoEncoders (Kingma and Welling, 2014) for text generation and address the posterior collapse problem wherein the decoder completely ignores the latent channel leading to poor generation. Adversarial auto-encoders (Makhzani et al., 2015) substitute the KL loss with an adversarial approach to enforce the latent Gaussian prior. On the other hand, AAEs are shown to naturally avoid the posterior collapse problem and promote strong coupling between the encoder and decoder. Wasserstein autoencoders (Tolstikhin et al., 2018) introduce a family of regularized autoencoders that learn a flexible prior by solving an optimal transport problem to match  $\mathrm{P(z)}$  and  $\mathrm{Q(z)}$  using an adversarial approach. Adversarially regularised autoencoders (ARAEs) follow the Wasserstein Autoencoder framework to learn a learnt prior unlike AAEs, which assume the prior to be a fixed standard Gaussian distribution.
E Computational Expense and Infrastructure used
The most parameter-heavy EPAAE model was from the SNLI dataset and we therefore report statistics for this model. The model has 13 million parameters and each epoch took approximately 60 seconds to train on an Nvidia V100-SMX2 GPU and an Intel(R) Xeon(R) E5-2698 CPU. For
complete details, please refer to the log.txt files present in each model's directory present in https: //github.com/sharan21/EPAAE.