SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that in our experiments, we use the MVHG relaxation of Sutter et al. (2023) and can thus leverage that they return one-hot encodings for  $n_k$ . This allows a different path for computing  $\alpha_k$  which circumvents introducing yet another temperature parameter altogether. We refer to our code in the supplement for more details.
Table 3: Total GPU hours per experiment. We report the cumulative training and testing hours to generate the results shown in the main part of this manuscript. We relied on our internal cluster infrastructure equipped with RTX2080Ti GPUs. Hence, we report the number of compute hours for this GPU-type.
Experiment	Computation Time (h)
Clustering (Section 5.1)	100
Partitioning of Generative Factors (Section 5.2)	480
MTL (Section 5.3)	100
In the following, we describe each of our experiments in more detail and provide additional ablations. All our experiments were run on RTX2080Ti GPUs. Each run took 6h-8h (Variational Clustering), 4h-6h (Generative Factor Partitioning), or  $\sim$  1h (Multitask Learning) respectively. We report the training and test time per model. Please note that we can only report the numbers to generate the final results but not the development time.
Code Release The official code can be found under https://github.com/thomassutter/drpm. Please note that the results reported in the main text slightly differ from the ones being generated from the official code. For the main paper, we based our own code for the experiments in Section 5.2 on the disentanglement_lib (Locatello et al., 2020). However, the library is based on Tensorflow v1 (Abadi et al., 2016), which makes it more and more difficult to maintain and install. Therefore, we decided to re-implement everything in PyTorch (Paszke et al., 2019).
While the metrics of our method and the baselines slightly change, the relative performance between them remains the same.
The code and results for the remaining two experiments in Sections 5.1 and 5.3 are the same as in the main text.
C.1 Approximation quality of Lemma 4.2
To provide intuitive understanding of the bounds introduced in Lemma 4.2, we present an experiment in this subsection to demonstrate the behavior of the upper and lower bounds. It is important to note that RPMs are discrete distributions, as the number of possible samples is finite for a given number of elements  $n$  and subsets  $K$ . Therefore, we can estimate the probability of a fixed partition  $\tilde{Y}$  under given  $\omega$  and  $s$  by sampling  $M$  partitions from  $p(Y; \omega, s)$ , counting the occurrences of  $\tilde{Y}$  in the samples, and dividing the count by  $M$ . As  $M$  approaches infinity, we can obtain the true probability mass function (PMF)  $p(Y; \omega, s)$  for every partition  $Y$ .
In our experiment, we set  $n = 5$  and aim to evaluate the quality of our bounds for all possible subset combinations of 5 elements, we thus set  $K = n = 5$ . To obtain a reliable estimate of the true PMF, we set  $M = 10^8$ . From Lemma 4.2, we know that:
$$
\forall \pi \in \Pi_ {Y}: p (\boldsymbol {n}; \boldsymbol {\omega}) p (\pi ; \boldsymbol {s}) \leq p (Y; \boldsymbol {\omega}, \boldsymbol {s}) \leq | \Pi_ {Y} | p (\boldsymbol {n}; \boldsymbol {\omega}) \max _ {\tilde {\pi}} p (\tilde {\pi}; \boldsymbol {s})
$$
$$
p _ {U} (Y; \boldsymbol {\omega}, \boldsymbol {s}) := | \Pi_ {Y} | p (\boldsymbol {n}; \boldsymbol {\omega}) \max  _ {\tilde {\pi}} p (\tilde {\pi}; \boldsymbol {s})
$$
$$
p _ {L} (Y; \boldsymbol {\omega}, \boldsymbol {s}) := \max  _ {\pi \in \Pi_ {Y}} p (\boldsymbol {n}; \boldsymbol {\omega}) p (\pi ; \boldsymbol {s})
$$
In Figure 5, we present the estimated PMF along with the corresponding upper bounds  $(p_U)$  and lower bounds  $(p_L)$  for four different combinations of RPM parameters  $\omega$  and  $s$ . We observe that when all scores  $s$  are equal, as in the priors of the experiments in Sections 5.1 and 5.2,  $p_U(Y; \omega, s)$  approximates  $p(Y; \omega, s)$  well and serves as a reliable estimate of the PMF. However, when the scores vary, the upper bound becomes looser, particularly for lower probability partitions, as it is dominated by the term  $\max_{\tilde{\pi}} p(\tilde{\pi}; s)$ . Although  $p_L(Y; \omega, s)$  appears looser than  $p_U(Y; \omega, s)$  for certain configurations of  $\omega$  and  $s$ , it provides more consistent results across all hyperparameter combinations.
(a)  $\omega = \mathbb{1}, s = \mathbb{1}$
(b)  $\omega \sim U([0,1])$ $s = 11$
(c)  $\omega = \mathbb{1}, s \sim U([0,1])$
(d)  $\omega \sim U([0,1]), s \sim U([0,1])$
Figure 5: Partitions with  $n = 5$  and  $K = 5$  for different  $\omega$  and  $s$ . Each point in the plots corresponds to one of the  $n^K$  different partitions and their respective estimated probability mass and its upper/ lower bounds according to Lemma 4.2.
C.2 Variational Clustering with Random Partition Models
C.2.1 Loss Function
As mentioned in Section 5.1, for a given dataset  $X$  with  $N$  samples, let  $Z$  and  $Y$  contain the respective latent vectors and cluster assignments for each sample in  $X$ . The generative process can then be summarized as follows: First, we sample the cluster assignments  $Y$  from an RPM, i.e.,  $Y \sim P(Y; \omega, s)$ . Given  $Y$ , we can sample the latent variables  $Z$ , where for each  $y$  we have  $z \sim \mathcal{N}(\mu_y, \sigma_y^T \mathbb{I}_l)$ ,  $z \in \mathbb{R}^l$ . Finally, we sample  $X$  by passing each  $z$  through a decoder like in vanilla VAEs. Using Bayes rule and Jensen's inequality, we can then derive the following evidence lower bound (ELBO):
$$
\begin{array}{l} \log (p (X)) = \log \left(\int \sum_ {Y} p (X, Y, Z) d Z\right) \\ \geq \mathbb {E} _ {q (Z, Y | X)} \left[ \log \left(\frac {p (X | Z) p (Z | Y) p (Y)}{q (Z , Y | X)}\right) \right] \\ := \mathcal {L} _ {E L B O} (X) \\ \end{array}
$$
We then assume that we can factorize the approximate posterior as follows:
$$
q (Z, Y | X) = q (Y | X) \prod_ {\boldsymbol {x} \in X} q (\boldsymbol {z} | \boldsymbol {x})
$$
Note that while we do assume conditional independence between  $z$  given its corresponding  $x$ , we model  $q(Y|X)$  with the DRPM and do not have to assume conditional independence between
Figure 6: Generative model of the DRPM clustering model. Generative paths are marked with thin arrows, whereas inference is in bold.
different cluster assignments. This allows us to leverage dependencies between samples from the dataset. Hence, we can rewrite the ELBO as follows:
$$
\begin{array}{l} \mathcal {L} _ {E L B O} (X) = \mathbb {E} _ {q (Z | X)} [ \log (p (X | Z)) ] \\ - \mathbb {E} _ {q (Y | X)} [ K L [ q (Z | X) | | p (Z | Y) ] ] \\ - K L [ q (Y | X) | | p (Y) ] \\ = \sum_ {\boldsymbol {x} \in X} \mathbb {E} _ {q (\boldsymbol {z} | \boldsymbol {x})} [ \log p (\boldsymbol {x} | \boldsymbol {z}) ] \\ - \sum_ {\boldsymbol {x} \in X} \mathbb {E} _ {q (Y | X)} [ K L [ q (\boldsymbol {z} | \boldsymbol {x}) | | p (\boldsymbol {z} | Y) ] ] \\ - K L [ q (Y | X) | | p (Y) ] \\ \end{array}
$$
See Figure 6 for an illustration of the generative process and the assumed inference model. Since computing  $P(Y)$  and  $q(Y|X)$  is intractable, we further apply Lemma 4.2 to approximate the KL-Divergence term in  $\mathcal{L}_{ELBO}$ , leading to the following lower bound:
$$
\begin{array}{l} \mathcal {L} _ {E L B O} \geq \sum_ {\boldsymbol {x} \in X} \mathbb {E} _ {q (\boldsymbol {z} | \boldsymbol {x})} [ \log p (\boldsymbol {x} | \boldsymbol {z}) ] (43) \\ - \sum_ {\boldsymbol {x} \in X} \mathbb {E} _ {q (Y | X)} [ K L [ q (\boldsymbol {z} | \boldsymbol {x}) | | p (\boldsymbol {z} | Y) ] ] (44) \\ - \mathbb {E} _ {q (Y | X)} \left[ \log \frac {\left| \Pi_ {Y} \right| \cdot q (\boldsymbol {n} ; \boldsymbol {\omega} (X))}{p (\boldsymbol {n} ; \boldsymbol {\omega}) p \left(\pi_ {Y} ; \boldsymbol {s}\right)} \right] (45) \\ - \log \left(\max  _ {\tilde {\pi}} q \left(\tilde {\pi}; s (X)\right)\right), (46) \\ \end{array}
$$
where  $\pi_{Y}$  is the permutation that leads to  $Y$  during the two-stage resampling process. Further, we want to control the regularization strength of the KL divergences similar to the  $\beta$ -VAE (Higgins et al., 2016). Since the different terms have different regularizing effects, we rewrite Equations (45)
Figure 7: Autoencoder architecture of the DRPM-VC model.
and (46) and weight the individual terms as follows, leading to our final loss:
$$
\begin{array}{l} \mathcal {L} := - \sum_ {\boldsymbol {x} \in X} \mathbb {E} _ {q (\boldsymbol {z} | \boldsymbol {x})} [ \log p (\boldsymbol {x} | \boldsymbol {z}) ] (47) \\ + \beta \cdot \sum_ {\boldsymbol {x} \in X} \mathbb {E} _ {q (Y | X)} [ K L [ q (\boldsymbol {z} | \boldsymbol {x}) | | p (\boldsymbol {z} | Y) ] ] (48) \\ + \gamma \cdot \mathbb {E} _ {q (Y | X)} \left[ \log \left(\frac {\left| \Pi_ {Y} \right| \cdot q (\boldsymbol {n} ; \boldsymbol {\omega} (X))}{p (\boldsymbol {n} ; \boldsymbol {\omega})}\right) \right] (49) \\ + \delta \cdot \mathbb {E} _ {q (Y | X)} \left[ \log \left(\frac {\operatorname* {m a x} _ {\tilde {\pi}} q (\tilde {\pi} ; \boldsymbol {s} (X))}{p \left(\pi_ {Y} ; \boldsymbol {s}\right)}\right) \right] (50) \\ \end{array}
$$
C.2.2 Architecture
The model for our clustering experiments is a relatively simple, fully-connected autoencoder with a structure as seen in Figure 7. We have a fully connected encoder  $E$  with three layers mapping the input to 500, 500, and 2000 neurons, respectively. We then compute each parameter by passing the encoder output through a linear layer and mapping to the respective parameter dimension in the last layer. In our experiments, we use a latent dimension size of  $l = 10$  for MNIST and  $l = 20$  for FMNIST, such that  $\mu(x), \sigma(x) \in \mathbb{R}^l$ . To understand the architecture choice for the DRPM parameters, let us first take a closer look at Equation (48). For each sample  $x$ , this term minimizes the expected KL divergence between its approximate posterior  $q(z|x) = \mathcal{N}(\mu(x), \mathrm{diag}(\sigma(x)))$  and the prior at index  $y$  given by the partition  $Y$  sampled from the DRPM  $q(Y|X; s, \omega)$ , i.e.,  $\mathcal{N}(\mu_y, \mathrm{diag}(\sigma_y))$ . Ideally, the most likely partition should assign the approximate posterior to the prior that minimizes this KL divergence. We can compute such  $s(X)$  and  $\omega(X)$  given the parameters of the approximate posterior and priors as follows:
$$
\forall \boldsymbol {x} _ {i} \in X: s _ {i} (\boldsymbol {x} _ {i}) = u \cdot (K - \underset {k} {\arg \min } (K L [ \mathcal {N} (\boldsymbol {\mu} (\boldsymbol {x} _ {i}), \operatorname {d i a g} (\boldsymbol {\sigma} (\boldsymbol {x} _ {i})) | | \mathcal {N} (\boldsymbol {\mu} _ {k}, \operatorname {d i a g} (\boldsymbol {\sigma} _ {k})) ])))
$$
$$
\boldsymbol {\omega} (X) = \frac {1}{| X |} \sum_ {\boldsymbol {x} \in X} ^ {N} \left\{\frac {\mathcal {N} (\boldsymbol {x} | \boldsymbol {\mu} _ {k} , \operatorname {d i a g} (\boldsymbol {\sigma} _ {k}))}{\sum_ {k ^ {\prime} = 1} ^ {K} \mathcal {N} (\boldsymbol {x} | \boldsymbol {\mu} _ {k ^ {\prime}} , \operatorname {d i a g} (\boldsymbol {\sigma} _ {k ^ {\prime}}))} \right\} _ {k = 1} ^ {K},
$$
where  $u$  is a scaling constant that controls the probability of sampling the most likely partition. Note that  $\boldsymbol{\omega}$  and  $s$  minimize Equation (48) if defined this way when given the distribution parameters of the approximate posterior and the priors. The only thing that is left unclear is how much  $u$  should scale the scores  $s$ . Ultimately, we leave  $u$  as a learnable parameter but detach the rest of the computation of  $s$  and  $\boldsymbol{\omega}$  from the computational graph to improve stability during training. Finally, once we resample  $z \sim \mathcal{N}(\mu(\boldsymbol{x}), \sigma(\boldsymbol{x}))$ , we pass it through a fully connected decoder  $D$  with four layers mapping  $z$  to 2000, 500, and 500 neurons in the first three layers and then finally back to the input dimension in the last layer to end up with the reconstructed sample  $\hat{\boldsymbol{x}}$ .
C.2.3 Training
As in vanilla VAEs, we can estimate the reconstruction term in Equation (47) with MCMC by applying the reparametrization trick (Kingma and Welling, 2014) to  $q(z|x)$  to sample  $M$  samples  $z^{(i)} \sim q(z|x)$  and compute their reconstruction error to estimate Equation (47). Similarly, we can sample from  $q(Y|X)L$  times to estimate the terms in Equations (48) to (50), such that we minimize
$$
\begin{array}{l} \tilde {\mathcal {L}} := - \sum_ {\boldsymbol {x} \in X} \frac {1}{M} \sum_ {i = 1} ^ {M} \log p (\boldsymbol {x} | \boldsymbol {z} ^ {(i)}) \\ + \frac {\beta}{L} \cdot \sum_ {\boldsymbol {x} \in X} \sum_ {i = 1} ^ {L} K L [ q (\boldsymbol {z} | \boldsymbol {x}) | | p (\boldsymbol {z} | Y ^ {(i)}) ] \\ + \frac {\gamma}{L} \cdot \sum_ {i = 1} ^ {L} \log \left(\frac {\left| \Pi_ {Y ^ {(i)}} \right| \cdot q \left(\boldsymbol {n} ^ {(i)} ; \boldsymbol {\omega} (X)\right)}{p \left(\boldsymbol {n} ^ {(i)} ; \boldsymbol {\omega}\right)}\right) \\ + \frac {\delta}{L} \cdot \sum_ {i = 1} ^ {L} \log \left(\frac {\max  _ {\tilde {\pi}} q (\tilde {\pi} ; \boldsymbol {s} (X))}{p \left(\pi_ {Y ^ {(i)}} ; \boldsymbol {s}\right)}\right) \\ \end{array}
$$
In our experiments, we set  $M = 1$  and  $L = 100$  since the MVHG and PL distributions are not concentrated around their mean very well, and more Monte Carlo samples thus lead to better approximations of the expectation terms. We further set  $\beta = 1$  for MNIST and  $\beta = 0.1$  for FMNIST, and otherwise  $\gamma = 1$ , and  $\delta = 0.01$  for all experiments.
To resample  $n$  and  $\pi$  we need to apply temperature annealing (Grover et al., 2019; Sutter et al., 2023). To do this, we applied the exponential schedule that was originally proposed together with the Gumbel-Softmax trick (Jang et al., 2016; Maddison et al., 2017), i.e.,  $\tau = \max(\tau_{final}, \exp(-rt))$ , where  $t$  is the current training step and  $r$  is the annealing rate. For our experiments, we choose  $r = \frac{\log(\tau_{final}) - \log(\tau_{init})}{100000}$  in order to annealing over 100000 training steps. Like Jang et al. (2016), we set  $\tau_{init} = 1$  and  $\tau_{final} = 0.5$ .
Similar to Jiang et al. (2016), we quickly realized that proper initialization of the cluster parameters and network weights is crucial for variational clustering. In our experiments, we pretrained the autoencoder structure by adapting the contrastive loss of (Li et al., 2022), as they demonstrated that their representations manage to retain clusters in low-dimensional space. Further, we also added a reconstruction loss to initialize the decoder properly. To initialize the prior parameters, we fit a GMM to the pretrained embeddings of the training set and took the resulting Gaussian parameters to initialize our priors. Note that we used the same initialization across all baselines. See Appendix C.2.4 for an ablation where we pretrain with only a reconstruction loss similar to what was proposed with the VaDE baseline.
To optimize the DRPM-VC in our experiments, we used the AdamW (Loshchilov and Hutter, 2019) optimizer with a learning rate of 0.0001 with a batch size of 256 for 1024 epochs. During initial experiments with the DRPM-VC, we realized that the pretrained weights of the encoder would often lose the learned structure in the first couple of training epochs. We suspect this to be an artifact of instabilities induced by temperature annealing. To deal with these problems, we decided to freeze the first three layers of the encoder when training the DRPM-VC, giving us much better results. See Appendix C.2.5 for an ablation where we applied the same optimization procedure to VaDE.
Finally, when training the VaDE baseline and the DRPM-VC on FMNIST, we often observe a local optimum where the prior distributions collapse and become identical. We can solve this problem by refitting the GMM in the latent space every 10 epochs and by using the resulting parameters to reinitialize the prior distributions.
C.2.4 Reconstruction Pretraining
While the results of our variational clustering method depend a lot on the specific pretraining, we want to demonstrate that improvements over the baselines do not depend on the chosen pretraining method. To that end, we repeat our experiments but initialize the weights of our model with an autoencoder that has been trained to minimize the mean squared error between the input and the reconstruction. This initialization procedure was originally proposed in (Jiang et al., 2016). We
Table 4: We compare the clustering performance of the DRPM-VC on test sets of MNIST and FMNIST between GMM in latent space (Latent GMM) and Variational Deep Embedding (VaDE) initializing weights using an autoencoder trained on a reconstruction objective. We measure performance in terms of the Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and cluster accuracy (ACC) over five seeds and put the best model in bold.
MNIST	FMNIST
NMI	ARI	ACC	NMI	ARI	ACC
LATENT GMM	0.75±0.00	0.66±0.01	0.75±0.01	0.56±0.02	0.41±0.03	0.57±0.02
VADE	0.77±0.02	0.62±0.04	0.69±0.04	0.53±0.07	0.35±0.08	0.47±0.09
DRPM-VC	0.74±0.00	0.67±0.01	0.75±0.02	0.59±0.01	0.47±0.02	0.62±0.01
Table 5: We compare the clustering performance of the DRPM-VC on test sets of MNIST and FMNIST between GMM in latent space (Latent GMM), and Variational Deep Embedding (VaDE) when freezing the encoder. We measure performance in terms of the Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and cluster accuracy (ACC) over five seeds and put the best model in bold.
MNIST	FMNIST
NMI	ARI	ACC	NMI	ARI	ACC
LATENT GMM	0.86±0.02	0.83±0.06	0.88±0.07	0.60±0.00	0.47±0.01	0.62±0.01
VADE	0.90±0.02	0.88±0.06	0.92±0.06	0.64±0.01	0.47±0.01	0.59±0.03
DRPM-VC	0.89±0.01	0.88±0.03	0.94±0.02	0.64±0.00	0.51±0.01	0.65±0.00
present the results of this ablation in Table 4. Simply minimizing the reconstruction error does not necessarily retain cluster structures in the latent space. Thus, it does not come as a surprise that overall results get about  $10\%$  to  $20\%$  worse across most metrics, especially for MNIST, while results on FMNIST only slightly decrease. However, we still beat the baselines across most metrics, suggesting that modeling the implicit dependencies between cluster assignments helps to improve variational clustering performance.
C.2.5 Baselines with fixed Encoder
For the experiments in the main text, we wanted to implement the VaDE baseline similar to the original method proposed in Jiang et al. (2016). This means, in contrast to our method, we used their optimization procedure, i.e., Adam with a learning rate of 0.002 with a decay of 0.95 every 10 steps, and did not freeze the encoder as we do for the DRPM-VC. To ensure our results do not stem from this minor discrepancy, we perform an ablation experiment on VaDE using the same optimizer and learning rate as with the DRPM-VC and freeze the encoder backbone. The results of this additional experiment can be found in Appendix C.2.5. As can be seen, VaDE results do improve when adjusting the optimization procedure in this way. However, we still match or improve upon the results of VaDE in most metrics, especially in ARI and ACC, suggesting purer clusters compared to VaDE. We suspect this is because we assign samples to fixed clusters when sampling from the DRPM, whereas VaDE performs soft assignments by marginalizing over a categorical distribution.
C.2.6 Additional Partition Samples
In Section 5.1, we have seen a sample of a partition of the DRPM-VC trained on FMNIST. We provide additional samples for both MNIST and FMNIST at the end of the appendix in Figures 15 and 16. We can see that for both datasets, the DRPM-VC learns coherent representations of each cluster that easily allow us to generate new samples from each class.
(a) NMI when varying  $K$
(b) ARI when varying  $K$
(c) Accuracy when varying  $K$
Figure 8: Clustering performance of Latent GMM, VaDE, and DRPM-VC when varying the number of clusters  $K$  for 5 seeds on Fashion MNIST. DRPM-VC consistently outperforms the baselines except for  $K = 20$ , where all methods perform similarly.
C.2.7 Samples per cluster
In addition to sampling partitions and then generating samples according to the sampled cluster assignments, we can also directly sample from each of the learned priors. We show some examples of this for both MNIST and FMNIST at the end of the appendix in Figures 17 and 18. We can again see that the DRPM-VC learns accurate cluster representations since each of the samples seems to correspond to one of the classes in the datasets. Further, the clusters also seem to capture the diversity in each cluster, as we see a lot of variety across the generated samples.
C.2.8 Varying the number of Clusters
In previous experiments, we assumed that we had access to the true number of clusters of the dataset, which is, of course, not true in practice. We thus also want to investigate the behavior of the DRPM-VC when varying the number of partitions  $K$  of the DRPM and compare it to our baselines. In Figure 8, we show the performance of Latent GMM, VaDE, and DRPM-VC for  $K \in \{6, 8, 10, 12, 14, 16, 18, 20\}$  across 5 different seeds on FMNIST. DRPM-VC clearly outperforms the two baselines for all  $K$ , except for the extreme case of  $K = 20$ , where all models seem to perform similarly. Expectedly, DRPM-VC performs well when  $K$  is close to the true number of clusters, but performance decreases the farther we are from it. To investigate whether the model still learns meaningful patterns when we are far from the true number of clusters, we additionally generate samples from each prior of the DRPM-VC in Figure 9. Interestingly, we can see that DRPM-VC still detects certain structures in the dataset but starts breaking specific FMNIST categories apart. For instance, it splits the clusters sandals/boots into clusters with (Priors 6/15) and without (Priors 4/9) heels or the cluster T-shirt into clothing of lighter (Prior 0) and darker (Prior 3) color. Thus, DRPM-VC allows us to investigate clusters in datasets hierarchically, where low values of  $K$  detect more coarse and higher values of  $K$  more fine-grained patterns in the data.
C.2.9 Clustering of STL-10
We include an additional variational clustering ablation on the STL-10 dataset (Coates et al., 2011) that follows the experimental setup of Jiang et al. (VaDE, 2016). As noted by Jiang et al. (2016),
Figure 9: We generate 32 images from each prior when training DRPM-VC on Fashion MNIST with  $K = 20$ . DRPM-VC starts detecting more fine-grained patterns in the data and splits some of the original clusters, such as sandals, boots, t-shirts, or handbags, into more specific sub-categories.
Table 6: We compare the clustering performance of the DRPM-VC and VaDE on the test set of STL-10 across five seeds. We measure performance in terms of the Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and cluster accuracy (ACC) over five seeds and put the best model in bold.
STL-10
NMI	ARI	ACC
VADE	0.80±0.03	0.76±0.04	0.88±0.02
DRPM-VC	0.83±0.01	0.80±0.02	0.91±0.00
variational clustering algorithms have difficulties clustering in raw pixel space for natural images, which is why we apply DRPM-VC to representations extracted using an Imagenet (Deng et al., 2009) pretrained ResNet-50 (He et al., 2015) as done in Jiang et al. (VaDE, 2016). Note that these results are hard to interpret, as STL-10 is a subset of Imagenet, meaning that representations are relatively easy to cluster as the pretraining task already encourages separation by label. For this reason, we list this experiment separately in the appendix instead of including it in the main text. We present the results of this ablation in Table 6, where we again confirm that modeling cluster assignments with our DRPM can improve upon previous work that modeled the assignments independently.
C.3 Variational Partitioning of Generative Factors