SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[67] Lucas Theis, Wenzhe Shi, Andrew Cunningham, and Ferenc Huszár. Lossy image compression with compressive autoencoders. In International Conference on Learning Representations, 2017.
[68] Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Transformer VQ-VAE for unsupervised unit discovery and speech synthesis: ZeroSpeech 2020 challenge. In INTERSPEECH, 2020.
[69] Frederik Trauble, Anirudh Goyal, Nasim Rahaman, Michael Curtis Mozer, Kenji Kawaguchi, Yoshua Bengio, and Bernhard Schölkopf. Discrete key-value bottleneck. In International Conference on Machine Learning, 2023.
[70] Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with VQVAE. arXiv preprint arXiv:2103.01950, 2021.
[71] Michael L Waskom. Seaborn: statistical data visualization. Journal of Open Source Software, 6 (60):3021, 2021.
[72] James C R Whittington, Will Dorrell, Surya Ganguli, and Timothy Behrens. Disentanglement with biological constraints: A theory of functional cell types. In International Conference on Learning Representations, 2023.
[73] Hanwei Wu and Markus Flierl. Learning product codebooks using vector-quantized autoencoders for image retrieval. In Global Conference on Signal and Information Processing, 2019.
[74] Quanhan Xi and Benjamin Bloem-Reddy. Indeterminacy in generative models: characterization and strong identifiability. In International Conference on Artificial Intelligence and Statistics, 2023.
[75] Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. In International Conference on Learning Representations, 2020.
[76] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. VideoGPT: video generation using VQ-VAE and transformers. arXiv preprint arXiv:2104.10157, 2021.
[77] Xiaojiang Yang, Yi Wang, Jiacheng Sun, Xing Zhang, Shifeng Zhang, Zhenguo Li, and Junchi Yan. Nonlinear ICA using volume-preserving transformations. In International Conference on Learning Representations, 2022.
[78] Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ICA: sparsity and beyond. In Advances in Neural Information Processing Systems, 2022.
A Quantized-Latent Models
This section contains pseudocode for latent quantization and for training QLAE and QLIinfoWGAN-GP.
Algorithm 2 Pseudocode for optimizing a quantized-latent autoencoder (QLAE). Hyperparameter values are detailed in Appendix C.2.
Require: dataset  $\mathcal{D}$  , batch size  $b$  AdamW hyperparameters  $(\alpha ,\beta_{1},\beta_{2}$  weight decay), loss weights  $\lambda \coloneqq (\lambda_{\mathrm{reconstruct}}$  data,  $\lambda_{\mathrm{quantize}},\lambda_{\mathrm{commit}})$    
1: initialize encoder  $\hat{g}^{-1}:\mathcal{X}\to \mathbb{R}^{n_z}$  , discrete value array  $\mathrm{V}\in \mathbb{R}^{n_z\times n_v}$  , decoder  $\hat{g}:\mathbb{R}^{n_z}\rightarrow \mathcal{X}$    
2: while  $(\hat{g}^{-1},\mathrm{V},\hat{g})$  has not converged do   
3: for  $i = 1,\dots ,b$  do   
4:  $x\sim \mathcal{D}$    
5:  $z,\mathcal{L}_{\mathrm{quantize}},\mathcal{L}_{\mathrm{committ}}\gets \mathrm{LatentQuantization}(x,\hat{g}^{-1},\mathrm{V})$ $\triangleright$  Algorithm 1   
6:  $\mathcal{L}_{\mathrm{reconstruct}}$  data  $\leftarrow$  BinaryCrossEntropy  $(\hat{g} (z),x)$    
7:  $\mathcal{L}_{\mathrm{QLAE}}^{(i)}\gets \lambda \cdot (\mathcal{L}_{\mathrm{reconstruct}}$  data,  $\mathcal{L}_{\mathrm{quantize}},\mathcal{L}_{\mathrm{committ}})$    
8:  $(\hat{g}^{-1},\hat{g})\gets \mathrm{AdamW}(\nabla_{(\hat{g}^{-1},\hat{g})}\frac{1}{b}\sum_{i = 1}^{b}\mathcal{L}_{\mathrm{QLAE}}^{(i)},(\hat{g}^{-1},\hat{g}),\alpha ,\beta_{1},\beta_{2},$  weight decay)   
9: V  $\leftarrow$  Adam(  $\nabla_{\mathrm{V}}\frac{1}{b}\sum_{i = 1}^{b}\mathcal{L}_{\mathrm{QLAE}}^{(i)},\mathrm{V},\alpha ,\beta_{1},\beta_{2})$
B Disentanglement Metrics Vignettes
Figure 5: InfoM and InfoC for three models with continuous latents computed from mutual information estimated via histogram binning. The binning strategy has a drastic effect on the metrics, and it is unclear how it should be chosen.
Figure 6: InfoM and InfoC for three models with continuous latents computed from mutual information estimated via  $k$ -neighbors based KSG [60]. The metrics are very robust to the choice of  $k$ .
Figure 7: Nonlinear DCI for three models with continuous latents computed via random forests of 100 trees. D and C are as defined in Eastwood and Williams [17]. I is validation accuracy. DCI is highly sensitive to the maximum tree depth hyperparameter, and it is easy to overestimate D and C if this is not tuned with respect to I.
maximum tree depth
This section contains details on the experiments conducted in this work.
Table 3: Summary of datasets used for empirical evaluation.
dataset	ns	|D|	description
Shapes3D [9]	6	480,000	geometric shapes in a scene with simplistic textures
MPI3D [20]	7	460,800	real robot arm holding objects in a variety of configurations
Falcor3D [53]	7	233,280	single scene illuminated and viewed in varying conditions
Isaac3D [53]	9	737,280	synthetic robot arm holding objects in a variety of configurations
Table 4: Shapes3D sources.
index	description	values
0	floor color	10
1	object color	10
2	camera orientation	10
3	object scale	8
4	object shape	4
5	wall color	15
Table 5: MPI3D sources.
index	description	values
0	object color	4
1	object shape	4
2	object size	2
3	camera height	3
4	background color	3
5	robot x	40
6	robot y	40
Table 6: Falcor3D sources.
index	description	values
0	lighting intensity	5
1	lighting x	6
2	lighting y	6
3	lighting z	6
4	camera x	6
5	camera y	6
6	camera z	6
Table 7: Isaac3D sources.
index	description	values
0	object shape	3
1	robot x	8
2	robot y	5
3	camera height	4
4	object scale	4
5	lighting intensity	4
6	lighting direction	6
7	object color	4
8	wall color	4
Figure 8: Shapes3D data samples.
Figure 9: MPI3D data samples.
Figure 10: Falcor3D data samples.
Figure 11: Isaac3D data samples.
C.2 Hyperparameters
This section specifies fixed and tuned hyperparameters for all methods considered.
Table 8: Fixed latent quantization hyperparameters for QLAE and QLIinfoWGAN-GP.
hyperparameter	value
nv	10
Vjinitialization	linspace(-0.5, 0.5, nv)
λquantize	1 × 10-2
λcommit	1 × 10-2
Table 9: Fixed hyperparameters for all autoencoder variants.
hyperparameter	value
nz	2ns
AdamW learning rate	1 × 10-3
AdamW β1	0.9
AdamW β2	0.99
AdamW updates	≤ 2 × 105
batch size	128
λreconstruct data	1
Table 10: Fixed hyperparameters for all InfoGAN variants.
hyperparameter	value
nz	2ns
AdamW learning rate	2 × 10-4
AdamW β1	0
AdamW β2	0.9
AdamW updates	≤ 2 × 105
batch size	64
λreconstruct latent	200
λgradient penalty	10
λvalue	1
Table 11: Key regularization hyperparameter tuning done for each autoencoder and InfoGAN variant.
method	hyperparameter	values
AE	weight decay	[0,0.001,0.01,0.1]
BioAE [72]	λactivity	[0.01,0.1,1,10,100]
β-VAE [25]	β = λKL	[0.1,0.3,1,3,10]
β-TCVAE [11]	β = λtotal correlation	[0.1,0.3,1,3,10]
VQ-VAE [55]	weight decay	[0.001,0.01,0.1,1]
QLAE (ours)	weight decay	[0.001,0.01,0.1,1]
InfoWGAN-GP [54]	weight decay	[0.0001,0.001,0.01,0.1]
QLInfoWGAN-GP (ours)	weight decay	[0.0001,0.001,0.01,0.1]
C.3 Network Architectures
Inspired by recent works showing how well-designed decoder architectures can facilitate inductive biases relevant for disentanglement [33, 43], we use an expressive architecture for all results presented in this work.
Encoder. We use a simple feedforward convolutional encoder network. Each convolutional block consists of two resolution-preserving convolutional layers (kernel size 3, stride 1) and one downsampling convolutional layer (kernel size 4, stride 2) at a consistent width (number of channels). Each convolution operation is followed by a leaky ReLU (slope 0.3), then instance normalization. There are four such blocks with widths 32, 64, 128, and 256. The  $256 \times 4 \times 4$  output is then flattened. Two dense layers each of width 256 with ReLU activation (and no normalization) follow. The final operation is an affine projection to the latent layer.
Decoder. We use a decoder architecture based on StyleGAN [33]. The latent code  $z$  of shape  $n_z$  passes through two dense layers each of width 256 with ReLU activation (and no normalization); call the output of this  $w$ . We directly parameterize a starting input feature map of shape  $256 \times 4 \times 4$  with all entries initialized to 0.1. Each style-decoding layer consists of processing an input feature map via a transposed convolution followed by a leaky ReLU (slope 0.3) and then an adaptive instance normalization (AdaIN):  $w$  undergoes an affine projection to a scale and bias scalar for each channel of the output feature map, and the output feature map is instance normalized then affinely transformed by spatially broadcasting the scales and biases. Similar to the encoder, style-decoding layers are grouped into blocks, with each block consisting of two resolution-preserving style-decoding layers (kernel size 3, stride 1) and one upsampling style-decoding layer (kernel size 4, stride 2) at a consistent width. There are four such blocks with widths 256, 128, 64, and 32. The final operation is a  $1 \times 1$  convolution of width 3 to yield an output of shape  $3 \times 64 \times 64$ .
C.4 Negative Results
We tried a number of additional modifications to our basic methods, QLAE and QLIinfoWGAN-GP, beyond the ablations presented in the main paper. Here is a list of those that only marginally helped, didn't help, or made things worse:
- Keeping the original VQ-VAE [55] hyperparameter settings for  $\lambda_{\mathrm{quantize}}$  (1) and  $\lambda_{\mathrm{commit}}$  (0.25).
- A step function schedule for the weight decay in AdamW: 0 for the first half of optimization, and a specified value for the second half.
- Linearly annealing  $\lambda_{\mathrm{quantize}}$  and/or  $\lambda_{\mathrm{commit}}$  from 0 up to a specified value.
- Using  $\ell_1$  norm parameter vector regularization, in addition to or without weight decay.
- (QLInfoWGAN-GP-specific) Updating V with the encoder instead of with the decoder.
C.5 Nonlinear DCI Evaluation
For each  $p(\mathbf{s}_i \mid \mathbf{z})$  learning problem, we train a random forest classifier with 100 trees and the information gain splitting criterion. We use a 0.9/0.1 train/test split of the evaluation sample. We tune the maximum tree depth hyperparameter on held-out accuracy (see Figure 7 for an example of why this is important). We compute D and C using the most predictive model's relative feature importances, following their definitions [17]. We use held-out accuracy for I.
D Qualitative Results
Figure 12: Decoded Shapes3D latent interventions and NMI heatmaps for QLAE and the prior method with highest InfoM. In each image block, a single data sample is encoded into its latent representation. In the  $j$ -th column, the  $j$ -th latent is intervened on with a linear interpolation of its range across the dataset. The resulting representation is then decoded. NMI heatmaps are annotated with source names, and inactive latents (as determined by range) have red font. In active latents, the visual variation in the generations caused by each latent tightly corresponds to sources that have significant NMI values in that latent's column. For example, the  $\beta$ -TCVAE's  $\mathbf{z}_1$  has high NMI with object scale and a lower NMI with object shape, and going down  $\mathbf{z}_1$ 's column in the generations, we see consistent changes in object scale and occasional changes in object shape. Inactive latents, e.g.  $\beta$ -TCVAE's  $\mathbf{z}_5$ , correspond to no discernable change in a column.
$\beta$  -TCVAE
Figure 13: Decoded MPI3D latent interventions and NMI heatmaps for QLAE and the prior method with highest InfoM. In each image block, a single data sample is encoded into its latent representation. In the  $j$ -th column, the  $j$ -th latent is intervened on with a linear interpolation of its range across the dataset. The resulting representation is then decoded. NMI heatmaps are annotated with source names, and inactive latents (as determined by range) have red font. In active latents, the visual variation in the generations caused by each latent tightly corresponds to sources that have significant NMI values in that latent's column. For example, either QLAE's  $\mathbf{z}_1$  (first and second data samples) or  $\mathbf{z}_7$  (third data sample) causes changes in the camera height. The sensitivity of the NMI estimation can be observed in the low but non-negligible NMI values between the robot x and robot y sources and the QLAE's  $\{\mathbf{z}_0, \mathbf{z}_3, \mathbf{z}_4, \mathbf{z}_5, \mathbf{z}_7, \mathbf{z}_9, \mathbf{z}_{11}, \mathbf{z}_{13}\}$  manifesting as rare or small changes in pose when intervening on those latents. This sensitivity, however, means that it is important to remove inactive latents from InfoMEC estimation: the  $\beta$ -TCVAE's  $\mathbf{z}_4$  and  $\mathbf{z}_{11}$  are estimated to have substantial NMI with background color and camera height, but do not actually cause any changes in the generations.
QLAE
Figure 14: Decoded Falcon3D latent interventions and NMI heatmaps for QLAE and the prior method with highest InfoM. In each image block, a single data sample is encoded into its latent representation. In the  $j$ -th column, the  $j$ -th latent is intervened on with a linear interpolation of its range across the dataset. The resulting representation is then decoded. NMI heatmaps are annotated with source names, and inactive latents (as determined by range) have red font. In active latents, the visual variation in the generations caused by each latent tightly corresponds to sources that have significant NMI values in that latent's column. This holds for both obvious visual changes in sources like lighting z, and rather subtle visual changes in sources like camera x. Pruning inactive latents from NMI before computing InfoM and InfoC is particularly important for models that optimize for latent shrinkage, such as the  $\beta$ -VAE: (InfoM, InfoC) is (0.71, 0.70) with pruning vs. (0.47, 0.51) without. Pruning is justified by the decoded interventions showing no dependence on the inactive latents.
$\beta$  -VAE
BioAE
Z0 Z1 Z2 Z3 Z4 Z5 Z6 Z7 Z8 Z9 Z10 Z11 Z12 Z13 Z14 Z15 Z16 Z17
Z0 Z1 Z2 Z3 Z4 Z5 Z6 Z7 Z8 Z9 Z10 Z11 Z12 Z13 Z14 Z15 Z16 Z17
Figure 15: Decoded Isaac3D latent interventions and NMI heatmaps for QLAE and the prior method with highest InfoM. In each image block, a single data sample is encoded into its latent representation. In the  $j$ -th column, the  $j$ -th latent is intervened on with a linear interpolation of its range across the dataset. The resulting representation is then decoded. NMI heatmaps are annotated with source names, and inactive latents (as determined by range) have red font. In active latents, the visual variation in the generations caused by each latent tightly corresponds to sources that have significant NMI values in that latent's column.
E Quantitative Results
This section contains unabridged results for the experiments in the main text. Intervals denote  $95\%$  confidence intervals of the mean estimated assuming a  $t$ -distribution. Bolded intervals overlap with the interval with highest endpoint in the column. AE and InfoGAN variants are presented and bolded separately as all AEs are filtered for near-perfect data reconstruction, whereas InfoGANs are generally more lossy.
Table 12: Full results for the Shapes3D dataset.
model	InfoM ↑	InfoE ↑	InfoC ↑	D ↑	I ↑	C ↑	PSNR (dB) ↑
AE	0.41 ± 0.03	0.98 ± 0.01	0.28 ± 0.01	0.11 ± 0.02	0.82 ± 0.01	0.08 ± 0.02	37.3 ± 0.1
β-VAE	0.59 ± 0.02	0.99 ± 0.02	0.49 ± 0.03	0.61 ± 0.02	0.99 ± 0.01	0.47 ± 0.01	35.1 ± 0.1
β-TCVAE	0.61 ± 0.03	0.82 ± 0.02	0.62 ± 0.02	0.46 ± 0.02	0.99 ± 0.01	0.38 ± 0.01	34.6 ± 0.1
BioAE	0.56 ± 0.02	0.98 ± 0.01	0.44 ± 0.02	0.33 ± 0.01	0.93 ± 0.01	0.25 ± 0.01	31.6 ± 0.1
VQ-VAE	0.72 ± 0.03	0.97 ± 0.02	0.47 ± 0.03	0.40 ± 0.01	0.84 ± 0.02	0.34 ± 0.01	36.0 ± 0.0
QLAE (ours)	0.95 ± 0.02	0.99 ± 0.01	0.55 ± 0.02	0.81 ± 0.01	0.99 ± 0.01	0.61 ± 0.01	32.9 ± 0.2
QLAE w/ global codebook	0.96 ± 0.01	0.99 ± 0.00	0.48 ± 0.01	0.83 ± 0.02	0.99 ± 0.01	0.60 ± 0.01	34.8 ± 0.2
QLAE w/o weight decay	0.69 ± 0.02	0.99 ± 0.01	0.51 ± 0.02	0.63 ± 0.02	0.99 ± 0.01	0.47 ± 0.03	37.5 ± 0.3
VQ-VAE w/ weight decay	0.80 ± 0.01	0.99 ± 0.01	0.46 ± 0.02	0.74 ± 0.01	0.99 ± 0.01	0.57 ± 0.01	37.2 ± 0.1
InfoWGAN-GP	0.61 ± 0.02	0.78 ± 0.03	0.41 ± 0.02	0.23 ± 0.01	0.80 ± 0.01	0.18 ± 0.01	12.0 ± 0.6
QLInfoWGAN-GP (ours)	0.73 ± 0.03	0.75 ± 0.03	0.48 ± 0.02	0.38 ± 0.01	0.85 ± 0.01	0.29 ± 0.01	20.4 ± 0.4
QLInfoWGAN-GP w/o w.d.	0.66 ± 0.03	0.57 ± 0.05	0.40 ± 0.04	0.16 ± 0.01	0.71 ± 0.02	0.13 ± 0.02	16.9 ± 0.7
Table 13: Full results for the MPI3D dataset.
model	InfoM ↑	InfoE ↑	InfoC ↑	D ↑	I ↑	C ↑	PSNR (dB) ↑
AE	0.37 ± 0.04	0.72 ± 0.03	0.36 ± 0.03	0.15 ± 0.02	0.83 ± 0.02	0.14 ± 0.02	36.2 ± 0.2
β-VAE	0.45 ± 0.03	0.71 ± 0.03	0.51 ± 0.03	0.31 ± 0.02	0.83 ± 0.02	0.27 ± 0.02	38.4 ± 0.2
β-TCVAE	0.51 ± 0.04	0.60 ± 0.04	0.57 ± 0.04	0.22 ± 0.02	0.77 ± 0.02	0.21 ± 0.02	37.1 ± 0.2
BioAE	0.45 ± 0.03	0.66 ± 0.04	0.36 ± 0.03	0.24 ± 0.02	0.79 ± 0.02	0.19 ± 0.02	37.9 ± 0.2
VQ-VAE	0.43 ± 0.06	0.57 ± 0.04	0.22 ± 0.04	0.09 ± 0.02	0.63 ± 0.02	0.14 ± 0.03	35.3 ± 0.2
QLAE (ours)	0.61 ± 0.04	0.63 ± 0.05	0.51 ± 0.03	0.36 ± 0.04	0.85 ± 0.04	0.36 ± 0.05	37.2 ± 0.6
QLAE w/ global codebook	0.54 ± 0.04	0.62 ± 0.02	0.45 ± 0.03	0.36 ± 0.04	0.86 ± 0.02	0.34 ± 0.02	37.2 ± 0.6
QLAE w/o weight decay	0.51 ± 0.03	0.61 ± 0.02	0.48 ± 0.04	0.30 ± 0.04	0.84 ± 0.06	0.29 ± 0.03	37.7 ± 0.3
VQ-VAE w/ weight decay	0.50 ± 0.04	0.81 ± 0.04	0.41 ± 0.04	0.22 ± 0.02	0.68 ± 0.02	0.20 ± 0.01	34.7 ± 0.2
InfoWGAN-GP	0.43 ± 0.04	0.40 ± 0.05	0.20 ± 0.05	0.09 ± 0.03	0.63 ± 0.01	0.09 ± 0.02	25.2 ± 1.4
QLInfoWGAN-GP (ours)	0.62 ± 0.03	0.51 ± 0.04	0.37 ± 0.04	0.24 ± 0.02	0.71 ± 0.02	0.25 ± 0.02	26.3 ± 1.5
QLInfoWGAN-GP w/o w.d.	0.63 ± 0.04	0.55 ± 0.04	0.40 ± 0.05	0.28 ± 0.01	0.74 ± 0.03	0.23 ± 0.03	26.3 ± 1.5
Table 14: Full results for the Falcor3D dataset.
model	InfoM ↑	InfoE ↑	InfoC ↑	D ↑	I ↑	C ↑	PSNR (dB) ↑
AE	0.39 ± 0.03	0.74 ± 0.03	0.20 ± 0.03	0.08 ± 0.02	0.76 ± 0.02	0.07 ± 0.01	29.9 ± 0.1
β-VAE	0.71 ± 0.05	0.73 ± 0.04	0.70 ± 0.03	0.32 ± 0.02	0.84 ± 0.02	0.28 ± 0.01	29.0 ± 0.2
β-TCVAE	0.66 ± 0.02	0.74 ± 0.04	0.71 ± 0.04	0.36 ± 0.02	0.90 ± 0.02	0.33 ± 0.01	29.6 ± 0.1
BioAE	0.54 ± 0.05	0.73 ± 0.04	0.31 ± 0.01	0.21 ± 0.02	0.81 ± 0.02	0.17 ± 0.02	29.4 ± 0.1
VQ-VAE	0.61 ± 0.04	0.83 ± 0.05	0.42 ± 0.02	0.30 ± 0.03	0.79 ± 0.01	0.29 ± 0.01	29.0 ± 0.1
QLAE (ours)	0.71 ± 0.03	0.77 ± 0.02	0.44 ± 0.02	0.50 ± 0.03	0.96 ± 0.03	0.38 ± 0.02	29.4 ± 0.4
QLAE w/ global codebook	0.59 ± 0.01	0.74 ± 0.03	0.37 ± 0.01	0.36 ± 0.02	0.90 ± 0.02	0.26 ± 0.04	28.4 ± 0.4
QLAE w/o weight decay	0.65 ± 0.02	0.76 ± 0.02	0.43 ± 0.02	0.46 ± 0.02	0.97 ± 0.02	0.36 ± 0.02	29.6 ± 0.4
VQ-VAE w/ weight decay	0.74 ± 0.02	0.86 ± 0.04	0.40 ± 0.03	0.41 ± 0.02	0.85 ± 0.01	0.32 ± 0.02	29.1 ± 0.1
InfoWGAN-GP	0.44 ± 0.04	0.60 ± 0.03	0.30 ± 0.04	0.11 ± 0.01	0.74 ± 0.01	0.08 ± 0.02	18.9 ± 0.6
QLInfoWGAN-GP (ours)	0.54 ± 0.04	0.53 ± 0.04	0.56 ± 0.03	0.20 ± 0.01	0.73 ± 0.02	0.24 ± 0.02	17.6 ± 1.0
QLInfoWGAN-GP w/o w.d.	0.56 ± 0.03	0.56 ± 0.04	0.52 ± 0.02	0.14 ± 0.02	0.72 ± 0.02	0.17 ± 0.02	17.6 ± 1.1
Table 15: Full results for the Isaac3D dataset.
model	InfoM ↑	InfoE ↑	InfoC ↑	D ↑	I ↑	C ↑	PSNR (dB) ↑
AE	0.42 ± 0.04	0.80 ± 0.02	0.21 ± 0.05	0.13 ± 0.02	0.85 ± 0.02	0.10 ± 0.02	38.4 ± 0.1
β-VAE	0.60 ± 0.03	0.80 ± 0.02	0.51 ± 0.03	0.23 ± 0.02	0.88 ± 0.02	0.19 ± 0.01	39.8 ± 0.1
β-TCVAE	0.54 ± 0.02	0.70 ± 0.02	0.46 ± 0.03	0.19 ± 0.02	0.84 ± 0.02	0.16 ± 0.01	38.2 ± 0.1
BioAE	0.63 ± 0.03	0.65 ± 0.03	0.33 ± 0.04	0.38 ± 0.02	0.91 ± 0.01	0.31 ± 0.02	38.0 ± 0.2
VQ-VAE	0.57 ± 0.04	0.87 ± 0.05	0.45 ± 0.04	0.33 ± 0.02	0.89 ± 0.02	0.31 ± 0.01	39.8 ± 0.1
QLAE (ours)	0.78 ± 0.03	0.97 ± 0.03	0.49 ± 0.03	0.69 ± 0.02	0.99 ± 0.01	0.54 ± 0.02	40.4 ± 0.3
QLAE w/ global codebook	0.65 ± 0.03	0.82 ± 0.02	0.46 ± 0.03	0.53 ± 0.03	0.99 ± 0.02	0.43 ± 0.03	39.1 ± 0.4
QLAE w/o weight decay	0.66 ± 0.02	0.89 ± 0.03	0.51 ± 0.02	0.58 ± 0.04	0.99 ± 0.01	0.48 ± 0.04	40.9 ± 0.3
VQ-VAE w/ weight decay	0.73 ± 0.03	0.81 ± 0.03	0.44 ± 0.04	0.34 ± 0.02	0.85 ± 0.02	0.39 ± 0.01	35.3 ± 0.1
InfoWGAN-GP	0.53 ± 0.03	0.51 ± 0.02	0.24 ± 0.02	0.13 ± 0.02	0.71 ± 0.02	0.11 ± 0.02	21.9 ± 0.9
QLInfoWGAN-GP (ours)	0.63 ± 0.03	0.58 ± 0.03	0.49 ± 0.03	0.24 ± 0.01	0.79 ± 0.01	0.25 ± 0.01	25.3 ± 0.7
QLInfoWGAN-GP w/o w.d.	0.59 ± 0.01	0.53 ± 0.04	0.45 ± 0.04	0.20 ± 0.03	0.77 ± 0.02	0.23 ± 0.01	23.2 ± 1.1