Tractable Representation Learning with Probabilistic Circuits
Authors: Steven Braun
Sahil Sidheekh
Antonio Vergari
Martin Mundt
Sriraam Natarajan
Kristian Kersting
Venue: Accepted by TMLR ()

Abstract: Probabilistic circuits (PCs) are powerful probabilistic models that enable exact and tractable inference, making them highly suitable for probabilistic reasoning and inference tasks. While dominant in neural networks, representation learning with PCs remains underexplored, with prior approaches relying on external neural embeddings or activation-based encodings. To address this gap, we introduce autoencoding probabilistic circuits (APCs), a novel framework leveraging the tractability of PCs to model probabilistic embeddings explicitly. APCs extend PCs by jointly modeling data and embeddings, obtaining embedding representations through tractable probabilistic inference. The PC encoder allows the framework to natively handle arbitrary missing data and is seamlessly integrated with a neural decoder in a hybrid, end-to-end trainable architecture enabled by differentiable sampling. Our empirical evaluation demonstrates that APCs outperform existing PC-based autoencoding methods in reconstruction quality, generate embeddings competitive with, and exhibit superior robustness in handling missing data compared to neural autoencoders. These results highlight APCs as a powerful and flexible representation learning method that exploits the probabilistic inference capabilities of PCs, showing promising directions for robust inference, out-of-distribution detection, and knowledge distillation.

Published in Transactions on Machine Learning Research (12/2025)
Tractable Representation Learning with Probabilistic Circuits
Steven Braun
steven. satten.brwunnssssuuee. ww-.oeesw.dddddrrrsdd.ddd
Technische Universität Darmstadt, Germany
Sahil Sidheekh
sahilsiddhcchh@uudlllass edu
University of Tezas at Dallas, United States
Antonio Vergari
wergari@@dd.cm ac. uk
University of Edinburgh, United Kingdom
Martin Mundt
mundim@uni--remen..
Universitäl Bremen, Germany
Sriraam Natarajan
orinamm.natarrjjannuuaaalla eduu
University of Tezos at Dallas, United States
Kristian Kersting
bristian.kerstinggnnn waarccssss-uuurroeo tu-darastaddd de
Technische Universität Darmstadt, Germany
Reviewed OpenReview: https: opwpreevieeeeeeeeeeemmmee net/ forum?? id= h8075pppKK
Abstract
Probabilistic circuits (PCs) are powerful probabilistic models that enable exact and tractable inference, making them highly suitable for probabilistic reasoning and inference tasks. While dominant in neural networks, representation learning with PCs remains underexplored, with prior approaches relying on external neural embeddings or activation-based encodings. To address this gap, we introduce autoencoding probabilistic circuits (APCs), novel framework leveraging the tractability of PCs to model probabilistic embeddings explicitly. APCs extend PCs by jointly modeling data and embeddings, obtaining embedding representations through tractable probabilistic inference. The PC encoder in APCs delivers hybrid neural autoencoding framework where missing data with arbitrary missing patterns can be natively handled in a principled way while keeping training end-to-end differentiable. Our empirical evaluation demonstrates that APCs outperform existing PC-based autoencoding methods in reconstruction quality, generate embeddings competitive with, and cxhibit superior robustness in handling missing data compared neural auloencoders. These results highlight APCs powerful and flexible representation learning method that exploits the probabilistic inference capabilities of PCs, showing promising directions for robust inference, out-of-distribution detection, and knowledge distillation.
1 Introduction
The ability to learn compact and expressive data representations is fundamental aspect of modern machine learning. Representation learning involves automatically discovering and encoding informative, low- dimensional features or embeddings from raw data. These effectively capture underlying factors of variation to facilitate downstream tasks such as classification, generation, or retrieval. Autoencoders have played a pivotal role in this domain by enabling the discovery of fow-dimensional embeddings that capture latent data features (Hinton & Salakhutdinov, 2006; Rifai et al., 2011). Their impact extends across diverse applications, such self-supervised learning paradigms like joint embedding predictive architectures (Assran al., 2023), natural language processing, where token embeddings are crucial for large language models (Vaswaní et al.,

---

Published in Transactions on Machine Learning Research (12/2025)
2017; Devlin et al., 2019), and computer vision, where embeddings facilitate efficient image retrieval in large databases (Deng et al., 2009). These advancements predominantly rely on neural network-based approaches, highlighting the critical role of embeddings in contemporary representation learning. Neural networks have been extensively studied in the field of representation learning, leading to their probabilistic extension, variational autoencoders (VAEs; Kingma & Welling, 2014; Rezende et al., 2014), later improved and extended along several dimensions such as the introduction of vector quantization (VQ van den Oord et al., 2017; Razavi et al., 2019), and hierarchical (Vahdat & Kautz, 2020; Liévin et al., 2019) and deterministic (Ghosh et al., 2020) formulations.
In recent years, a special kind of neural network has emerged for tractable probabilistic modeling: prob- abilistic circuits (PCs; Darwiche, 2003b; Choi et al., 2020). The computational graphs of PCs are con- strained in the way they are formod, thus trading off expressiveness for tractable inference (Vergari et al., 2019b). For example, given certain structural prop- ertics that are easy to enforce in practice, PCs can allow the tractable marginalization of every subset of the input features in just a single feedforward pass of the computational graph. These models pro- vide unified framework for tractable probabilistic models (Vergari et al., 2021) and address tractabil- ity challenges in deep generative models (Darwiche, 2003a; Poon & Domingos, 2011; Kisa, et al., 2014). They have been successfully applied in diverse areas, including lossless compression (Liu et al., 2022; Severo et al., 2025), biomedical modeling (Dang et al., 2022b;
0%
50%
80%
0%
50%
80%
0%
50%
80%
2
APC MIWAE VAE Data
2
2 2
17
2 2
2
Figure 1: APCs outperform VAE-based models in reconstruction quality under missing data. For experiment setlings, see Section 5.1.
Mathur et al., 2023; 2024), neuro-symbolic AI (Ahmed et al., 2022; Loconte et al., 2023; Karanam et al., 2025;
Kurscheidt et al., 2025), multi-modal fusion (Sidheekh et al., 2025), graph representation and learning (Errica & Niepert, 2023; Zheng et al., 2018; Loconte al., 2023), and constrained text generation (Zhang et al., 2023).
Recent advancements have enhanced POs in expressivity, learning, and scaling through sparsity- regularization (Dang et al., 2022a), structural improvements through subtractive mixtures and squared cir- cuits (Loconte et al., 2024; 2025b; Wang & Broeck, 2025), as well as hybrid approaches like HyperSPNs (Shih et al., 2021), neural conditioning in conditional sum- num-product networks (Shao et al., 2022) and probabilistic neural circuits (Martires, 2024), and latent variable distillation (Liu et al., 2023a;b). However, in contrast to neural networks, the use of PCs for representation learning has remained largely unexplored.
All the above works focus on learning or querying PCs as black-box probability models, i.c., functions that receive some input configuration and output a scalar probabilistic quantity of interest: a marginal or conditional probability or the value of an expectation. However, since PCs are neural networks, one could use them to extract representations and use these for downstream predictive tasks. To date, research into representation learning with PCs has been limited, with two notable exceptions. The first is the investigation of internal PC representations by Vergari et al. (2019a). The second is sum-product autoencoding (SPAE; Vergari et al., 2018), which defines an autoencoding scheme using these representations. However, these prior approaches face severe limitations: the specific circuit structure, and the auloencoding scheme cannot be trained end-to-end. This inherently prevents seamless integration with modern neural architectures.
often unclcar how to extract general representations decoupled from
We present autoencoding probabilistic circuits (APCs), novel framework that introduces end-to-end lcarning of representations in an autoencoding fashion using PCs. APCs leverage the probabilistic capabilities of PCs by employing them as a tractable encoder, generating explicit probabilistic embeddings Z. Critically, embeddings in APCs are obtained through tractable conditional state inference (i.e. sampling or maximum a posteriori) from the joint distribution PX, Z) over data X and embedding random variables Z explicitly modeled by the PC: z ~ Pc X, x,); where X, C X can be partially observed, natively handling missing data. In stark contrast, this is hard to do with fully neural-based autocncoders, where people often resort to heuristics or computationally intensive imputation methods (Nazabal et al., 2020; Maltei & Frellsen, 2019; Simkus & Gutmann, 2023; 2024) to deal with incomplete inputs (see Section 4 and Fig. 1). APCs can
2

---

Published in Transactions on Machine Learning Research (12/2025)
Pe X, Z.)
Data
Embeddings
Xi
8,
21
Neural
2 2~ P= x)
Decoder g(x)
Circuit Encoder C
x
Figure 2: Autoencoding probabilistic circuits (APCs) architecture. An input data point, possibly with (arbitrary) missing values, probabilistically encoded inlo compact embedding space Z using PC pC (X, Z). The embedding is sampled from the data-conditional distribution Pc(Z X). A neural decoder then reconstructs the complete data point X from the embedding. The tractable nature of the PC encoder allows the intrinsic handling of missing data without the need for imputation.
further bridge circuit and neural architectures by allowing for an arbitrary neural network decoder, extending the frameworks capacity. APCs thus offer an elegant and principled solution to the problem of learning representations in the presence of missing data. Fig. illustrates the APC pipeline.
In summary, our key contributions are: (a) autoencoding probabilistic circuits (APCs), novel framework for tractable representation learning with explicit embeddings modeled with PCs; (b) integrating a hybrid architecture that improves the learning of the encoder circuit, combining the strengths of tractable probabilistic and neural methods; (c) enabling end-to-end training between the encoder circuit and neural decoder by an improved differentiable sampling procedure for PCs, leveraging novel advances in gradient estimation; and (d) extensively demonstrating APCs' ability reconstruct inputs, produce useful emboddings for downstream tasks, and knowledge distillation without access to original training data, all in the presence of missing inputs, scenario that neural autoencoders cannot natively handle.
2 Background
To properly introduce APCs later in Section 3, need to briefly provide the necessary background about the mechanism of autocncoding and the formal definitions of PCs.
Autoencoding. The autoencoding mechanism, fundamental to representation learning and central to APCs, is form of self-supervised learning (Hinton & Salakhutdinov, 2006). An autoencoder consists of X + Z, maps an input data point x E X to a lypically lower-dimensional (lZ) < (X]) embedding representation, also called embedding, z Z. The decoder, g: Z + X, then attempts to reconstruct the original input from this embedding, producing & = gz) = (x)( The model is trained by minimizing possibly regularized reconstruction loss, L(x, X), which measures the dissimilarity between the original input and its reconstruction. This process encourages the embedding space Z to capture salient representations and variations within the data (Vincent et al., 2008; Rifai et al., 2011). Several variants of autoencoders (AEs) have been proposed in the literature, sometimes tailored for specific data modality such as text (Sutskever et al., 2014), images (Xu et al., 2014) and graph data such as molecules (Kusner al., 2017)-
two primary components: an encoder and a decoder. The encoder, f:
Variational autoencoders (VAEs; Rezende et al., 2014; Kingma & Welling, 2014) have been introduced probabilistic AEs whose encoders and decoders realize stochastic maps. Specifically, the encoder in VAE draws embedding samples from conditional distribution z p(Z x), which is commonly realized simple distribution (most of the time, an isotropic Gaussian) parameterized by a neural backbone that reccives x as inpul. Analogously, its decoder, draws reconstruction from a distribution & ~ p(X z) conditioned on the embedding z. Again, the distribution is assumed to have a simple form parameterized by a neural network (Ghosh et al., 2020). Neural parameterizations for both the encoder and decoder increase the expressiveness of these models. However, this increased expressiveness comes at the cost of reduced flexibility, especially when the models are required to perform advanced probabilistic inference tasks, such computing marginals.
3

---

Published in Transactions on Machine Learning Research (12/2025)
Pc(X, Z1, 22)
71
Z2
N
X
Zz
X
Z1
X
Z
Figure 3: A PC over a set of data random variables X and two embedding variables Zi and Zz. The black connections illustrate the flow of computation during a forward pass for inference. In contrast, the orange subgraph highlights sampling-induced troc.
A classical example is that of missing input values (Collier et al., 2020; Williams et al., 2018). That is, when only some variables X. C X are observed, a neural paramcterization of the encoder entails that computing p(Z x,) is intractable, as it would require computing the marginal distribution p(X.) = T(X... X,, )dX, where Xm X \ X. are the missing input features. As such, dealing with missing values for both AEs and VAEs is a challenging task, for which several hcuristics and approximations have been proposod (Nazabal et al., 2020; Mattei & Frellsen, 2019; Simkus & Gutmann, 2023; 2024), see Section 4 for a detailed discussion.
Probabilistic circuits. A proiabilistic circuit C, is a computational graph representing function C(X) over its imputs X (this is just a placeholder which can e.g. include data and embedding variables), where C(x) 2 0 for any imput x, making C(X) an unnormalized distribution. The computational graph, parameterized by 8, consists of three types of computational units: input @. sum D and product ②. An input unit c encapsulates a parameterized function fe(X,), with X, S X denoting its scope. The scopes for product and sum units, Xe; are the union of their input units' scopes. Specifically, product unit computes the product of its imputs Naeii(e) d(Xx), and sum unit computes the weighted sum Edeinte) 0g d(X.). We assume sum unit weights be normalized, i.c. Edemne)) 09 =.
PCs can tractably (in time O(C)) and exactly compute evidence, marginal, conditional, and moment queries, which are essential to our proposed circuit-based encoding procedure, fulfills two structural properties: smoothness and decomposability (Darwiche & Marquis, 2002; Choi al., 2020; Vergari et al., 2021).
Structural properties for tractable marginals and sampling. A sum unit is smooth, if all input units e E in(n) have the same scope, i.c., X. = X,, A product unit n is decomposable, if all input units Ci E in(n) have pairwise disjoint scopes, i.e. X., nx,, 0 for c: &c. A PC is smooth and decomposable, if all of its sum units are smooth and all of its product units are decomposable. Fig. 3 presents an example of a smooth and decomposable PC. Smooth and decomposable PCs can be interpreted as hierarchical latent variable models, where discrete latent variable He associated each sum unit c. Such a latent variable has as many states as the number of input connections of its corresponding sum unit (Peharz et al., 2017). Under this light, the weights 09 of a sum unit represent the probabilities of selecting specific mixture component, i.c., one of its input connections. This latent variable interpretation enables exact and efficient rac sampling from the distribution Pe H), where H denote all the discrete latent variables associated to all sum units in the PC.1
Usually, samples from X are retained and those from discarded (Vergari al., 2018).
4

---

Published in Transactions on Machine Learning Research (12/2025)
Sampling proceeds as follows. From the output unit of PC, one traverses the computational graph backwards, outputs before inputs, until the input distribution units are reached. This process constructs a sampling- induced free in the PC. Fig. 3 illustrates this process. For a product unit, all input connections are followed, while for a sum unit, a single connection is sampled with probability proportional to its corresponding weight. Once an input distribution is reached, one can sample a feature correspouding to its scope according to the paramctric distribution encoded in the unit. A complete sample (x, h) retrieved by concatenating all features sampled from input units, and all latent variables H encountered in the backward traversal. However, this sampling procedure is inherently non-differentiable due to the discrete choices made at sum units. While methods to make PC sampling differentiable have been explored (Lang et al., 2022), their practical limitations, such as the poor sample quality and the gradient instability, motivate the need for more robust ways to sample PCs and integrale them in larger neural pipelines.
3 Autoencoding Probabilistic Circuits
We now introduce autoencoding probabilistic circuits (APCs), framework to learn end-to-end representations in an autoencoding fashion using POs, leveraging their tractable inference capabilities. Prior attempts to extract representations from PCs have explored two main avenues, notably within sum-product autoencoding (SPAE; Vergari et al., 2018). First, as PCs can be viewed as a specialized type of neural network, representations (termed ACT embeddings in SPAE) have been derivod from the activations of selected circuit units. However, unlike typical neural networks, PCs are sparse computational graphs that often lack clear, inherent notion of "bottleneck" layer, making it challenging to identify and select which units provide compact and informative representations. Second, given their interpretation latent variables (as discussed in Section 2), representations (termed CAT embeddings in SPAE) have been formed by inferring the maximum a posteriori assignments to these internal latent variables. Yel, modern PCs can involve an enormous number of such internal latent variables, making it non-trivial to select a concise and globally mcaningful subsct for representation. Crucially, these prior approaches primarily derive embeddings post-hoc, without explicitly integrating the learning of embeddings into the training process end-to-end,
hierarchical latent variable models with internal discrete
In contrast, APCs introduce fundamentally different approach by treating embeddings as first-class random variables within the PC itself. Instead of learning separate conditional encoder as in VAEs (i.e., an inference model p(Z| x)). we design a single PC, C, that explicitly models the joint probability distribution Pe Z) over both data X and embeddings Z. This is achieved by representing the embedding variables with their own dedicated, parametric input units within the circuit structure. The primary advantage of this formulation is that the encoder the joint distribution itself, which remains fully tractable. Consequently, we can obtain embedding representations through exact probabilistic inference, such as by sampling from the true conditional posterior ~ pe (e x) even for partially-observee inputs or performing maximum a posteriori (MAP) inference. This provides a more principled and direct loundation for probabilistic representation learning. Note that these explicit imput embeddings Z are different from the internal discrete latent variables H of the PC (see previous section). While the latter are discrete latent variables, the former can be continuous.
3.1 Encoding with Tractable Joint Data-Embedding Distributions
As the encoder model, we construct a smooth and decomposable PC C that models the joint distribution pe (X, Z) of data and embedding random variables. The encoding process fe: X + Z maps data sample x to its embedding representation We define the encoding fc(x) as performing probabilistic state inference in the PC by conditionally sampling an embedding, given the data x. In contrast to the VAE-based formulation, when C is a smooth and decomposable PC, we can perform arbitrary marginalization and conditioning, allowing us to directly sample embedding variables from the data-condiitinaal distribution pe (Z) x). Conditional sampling in PCs consists of two passes through the network. First, a forward pass of the marginal pe (x()) for a given data sample x caches log-likelihoods at cach input edge. Subsequently, a sampling pass with reweighted sum unit weights according to their cached log- likelihooodd samples from the on X x conditioned marginal PC Pdxx=x (Z) = (Z X x). At test time we replace all sampling operations in the eircuit with approximate
5

---

Published in Transactions on Machine Learning Research (12/2025)
maximum posteriori (MAP) inference. ? Note that this is done with the purpose of enhancing the quality of the drawn samples and, while samples are not i.i.d. anymore, they are still valid for the purpose of generative modeling (or in our case, representation learning) and it is a common practice in the literature of VAEs, as discussed in Peharz et al. (2020a). We outline the full encoding algorithm in Appendix Algorithm 1.
Moreover, and in stark contrast to neural encoders, a PC-based encoding scheme allows us to infer the full embedding state z from partial evidence, where only a subset X. S X of the random variables are observed, and the rest is missing Xm. X \ X,, This is achieved through marginalizing the missing variables [ (X; d'm,; without the need for any data imputation. On the other hand, neural networks assume all input values to be given. Consequently, data with missing observations must be manually preprocessed before being fed to a neural network. When not natively accounted for in & specific method, this preprocessing typically involves employing various imputation techniques such as mean imputation, multiple imputation (Rubin, 1989), or advanced methods like approaches (Yoon et al., 2018; Gondara & Wang, 2018; Luo et al., 2018). As a side effect, for neural networks these imputation met usually introduce biases or assumptions that can affect model performance.
algorithms and deep learning
While not explicitly explored in this work, we emphasize that using circuit encoder enables leveraging data-specific input or even embedding distributions to model random variables accurately. This is particularly beneficial in scenarios where we have prior knowledge about the specific data or embedding distributions or when each variable can be effectively represented as a mixture of multiple distributions of different types.
Encoder Structure. A key design choice within the APCs framework is the architecture of the PC encoder. While our approach is agnostic to the specific topology beyond smoothness and decomposability, the chosen structure dictates how the joint distribution pe (X, Z) is decomposed and influences the learned representations This architectural flexibility is both a strength and a challenge. While a similar large design space exists for neural networks, structural choices in PCs have direct and explicit semantic consequences. Specifically, the decomposition defined by product units encodes conditional independence assumptions directly into the model's architecture. This contrasts with neural networks where such relationships are learned implicitly. For APCs, this means the circuit structure determines the precise probabilistic relationship between data X and embeddings Z, directly influencing the meaning of the learned representations. In our experiments, we employ architectures tailored to the data modality.
For tabular data, we use the architectural recipe of Einsumnetworks (Peharz et al., 2020a) with a RAT (Peharz et al., 2020b) structure. Here, inputs are modeled in random order in multiple random permutations without any quantification to which split between data variables is a "good fit", given the data. Therefore, we simply choose to model embodding variables as input units alongside the data variables at the circuit's input layer. For image data, we construct convolutional ional-style layerwise PC that builds a feature hierarchy through alternating product and sum layers. Each product layer reduces the height and width of the input by a factor of 2, building the product over all scopes in disjoint neighboring windows, similar to non-overlapping convolution kernels, when the stride is equal to the kernel size. After each product layer, a sum layer maps all input units of the same scope to a vector of sum units, similar to convolution in- and out-channels in traditional neural convolutionn layers. In this structure, each embedding input is randomly coupled with a random data variable unit via local product unit Further implementation details for these architectures are provided in Appendix B.1.
While these are simplistic choices for how we incorporate the embedding variables in our experiments, one could also move beyond the designs used here by leveraging structure learning algorithms or more principled heuristics (Loconte et al., 2025a). This could e.g. include the strategic placement of embedding variables at various depths within the circuit, allowing for hicrarchical embeddings capable of distinguishing between low-level features and high-level abstractions.
2This is done by replacing sum units with max units and tracing backwards (from the circuit output to inputs) the input. configuration that (approximately) mastly activates the circuit. (Mei al., 2018; Vergari et al., 2021).
6

---

Published in Transactions on Machine Learning Research (12/2025)
Sunbel-Suittax SIMPLE
Figure 4: SIMPLE outperforms Gunbel-Softmax in gradient es- timation as measured by the KL be- tween ground-trute foound-tuuh and learned sum units for different input dimensions d. SIMPLE converges faster and achieves significantly lower KI.-Divegence in synthetic task to learn sun unit pa- rameters using the mean squared error on samples drawn from a fixed ground- truth categorical distribution.
0.2
KL-Divergence
256
128
0.1
d=33
52
0.0
0
200
400
600
800
1000
Iteration
3.2 Differentiable Tractable Probabilistic Encoding
As discussed in Section 2, while exact (conditional) sampling from a smooth and decomposable PC is possible, it presents a challenge for gradient- optimization: not differentiable. This zon-differeniiabiliy arises from the inherent discreteness of the sampling procedure, specifically drawing from categorical distributions defined by sum unit weights and deterministically choosing successor units. Here, make use of the concept of differentiable sampling for PCs introduced in Lang et al. (2022). This allows us to replace the discrete operations present in the sampling procedure of sum units with continuous, and thus differentiable, reparametrizations. We further improve the method proposed in Lang et al. (2022) and replace the Gumbel- Softmax trick with SIMPLE (Ahmed et al., 2023). SIMPLE is a gradient estimator for k-subset sampling that uses discrete sampling in the forward pass and exact conditional marginals in the backward pass. When we set A 1, SIMPLE performs sampling from mixture equal to what we require in sum units and can be used in-place of the Gumbel-Softmax trick. We outline how SIMPLE is applied to sum unit in Appendix A Algorithm 3. By incorporating SIMPLE into the different sampling procedure for PCs, we can improve gradient estimation and overall performance of the autoencoding pipeline, overcoming the initial difficulties of differentiable sampling in PCs reported in Lang et al. (2022), which we show in Fig. 4. We designed a syuthetic task where the objective is to learn sum unit parameters by differentiably sampling from this sum unit and minimizing the sample mean squared error to samples drawn from a fixed ground-truth categorical distribution. In this setting, SIMPLE consistently converged faster, achieved higher accuracy (as measured by the Kullback-Leible divergence between the sum unit and the true categorical distribution), and exhibited lower variance compared to Gumbel-Softmax. These findings validate SIMPLE for more accurate and stable end-to-end training of APCs. We provide full details on this synthetic task in Appendix G.
3.3 Hybrid Decoding with Neural Networks
In APCs, while the encoding phase leverages the principled and tractable nature of PCs, the decoding process employs neural network to reconstruct data from the learned embeddings. This design choice is motivated by the complementary strengths neural networks, particularly their capacity for modeling complex, non-linear mappings. Therefore, in APCs, the decoder go distinct from the PC encoder C. This neural decoder maps the embedding representation back to the original data space X, effectively complementing the probabilistic encoding with the modeling capacity and computational efficiency of neural networks during decoding. The specific architecture of the neural decoder can be tailored to the data type. For instance, CNNs are well-suited for image data, enabling the incorporation of implicit biases beneficial for capturing spatial relationships not explicitly modeled in the encoding circuit. Similarly, Transformer- baseed architectures can be leveraged for sequential or high-dimensional data due to their ability to model long-range dependencies and complex interactions. Beyond these, for achicving high-fidelity reconstructions and sample quality, advanced generative models such as diffusion models could also be integrated decoders.
X is implemented as separate neural network,
7

---

Published in Transactions on Machine Learning Research (12/2025)
Importantly, this neural decoding strategy remains robust even when dealing with partial evidence. The tractable encoder fe's ability to infer complete embedding from a partially observed input ensures that the neural decoder always receives a full embedding vector. Consequently, the decoding process is unaffected by missing data in the input. We empirically explore the benefits of this hybrid approach by comparing APCs with neural decoders to APCs with self-decoding PCs in Section 5.6.
3.4 End-to-End Training of Tractable Encoder and Neural Decoder
APCs are trained end-to-end using objectives that encourage both accurate reconstruction and meaningful and robust embedding representations. We derive our training objective from a standard regularized autoencoding mechanism. The loss function is a sum of three crucial components: a reconstruction term, an embedding prior regularization term, and joint. data-embedding likelihood regularization term:
L = AREC LREC AKL LKL ANLL LNLL.
(1)
The factors À allow us to assign specific weights to different objectives. Such weighting can, e.g., facilitate disentanglement (Higgins et al., 2017). A detailed investigation of alternative configurations for À is left to future work. For all experiments in Section 5, we set AREC = λKL = ANLL = 1, as this configuration already worked well our scenario. In the following, we will describe each objective in detail.
Reconstruction. We use reconstruction term, CREG, ensure that embeddings represent sufficient. information to reconstruct the original input data accurately. This objective, analogous to the used in conventional autoencoders (Hinton & Salakhutdinov, 2006), encourages the model to learn meaningful and compact embedding space that captures essential features of the tata-generating distribution:
B
LREC = B
log po 2:), where 2, ~ [e x:),
(2)
where B is the batch size, x, the i-th input sample.
Embedding Regularization. To ensure the learned embeddings are meaningful and to avoid overfitting, we regularize the embedding distribution to chosen prior. While simple I.p regularization could be applied to the embedding vectors z, PCs allow for a more principled approach. Sampling in a PC instantiates the hidden sum unit variables E H and induces a product unit tree with independent input units. This is highlighted as the orange subgraph in Fig.
Consequently, sampling embeddings from the conditional distribution (Z x), given some data point x, induces a tree C' which embedding variables are mutually conditionally independent, given and h:
Z, 1 X=x. Vj 1......... Zl.j * k.
(3)
This independence allows us to apply statistical distances between the induced embedding distribution and a chosen prior q. Specifically, we use the Kullback-Leible divergence (KL):
1 z B
LKL
Σ
Σ
kL (Zj x;, ha) (Zz))
(4)
where B is the batch size and x; represents the i-th data sample. When choosing distributions of particular parametric forms, e.g. the exponential family, distributions in C are chosen as Gaussian input units in all cases.
compute Eq. (4) analytically in closed form. Embedding
Joint Data and Embedding Likelihood. A key advantage of the PC encoder is its capacity to model the full joint distribution (X, Z). We leverage this by introducing a third loss component that maximizes the joint log-likelihood of the data and their inferred embeddings. This objective steers the encoder's parameters towards a maximum likelihood estimate (MLE) of the joint distribution, regularizing the reconstruction- focused training. Additionally, this helps in mitigating the well-known posterior collapse in which pe (Z X)
8

---

Published in Transactions on Machine Learning Research (12/2025)
degenerates to Pc(Z), by enforcing high likelihood the joint distribution enforcing direct relationship between X and Z in the encoder. Our ablation study in Section 5.6 empirically validates that this term is crucial for achieving high performance. We formalize this objective as the negative log- og-likelihood (NLL):
B Σ B
LNLL =
log pC (x,, where 2, ~ pc x.).
55)
Training Via Knowledge Distillation. While the APOs framework can be trained from scratch on raw data using the aforementioned objectives, its tractable marginals also enable data-free knowledge distillation from pre-trained generative latent variable models like VAEs. In this sccnario, the APC model acts as a student, iteratively learning lo capture the teacher's generalive distribution without direct access to the original training data. The process involves sampling an embedding z from the APC's prior pe which the teacher VAE's decoder gVAE(z) uses to generate synthetic data &VAE. The corresponding embedding ZVAE is then obtained by encoding &VAE with the VAE's encoder fvAE(&vAE). For training the APC student, these synthetic data εVAE and teacher embeddigns VVAE replace the "ground truth" data points X; and their corresponding embeddings 2, in the loss terms. Specifically, we get
2 KL KL (2, XVAE, h;) (l,))
LKL,KD = B
(6)
i=1 1=1
B
INLL,KD =
log DcgCcXXV ZVAE,))
(7)
B
i=1
B
PREC.KD =
Po 2;), where (2 XvAe,)
(8)
B 1=1
A detailed data- free-free knowledgeddisillation algorithm is outlined in Appendix Algorithm 2. While our main focus in Section 5 is on training APCs from scratch, we additionally explore the alternative perspective of training via data-free knowledge distillation from pretrained VAEs in Section 5.5.
4 Related Work
Having introduced APCs in the previous section, we now contextualize our framework by discussing prior work in representation learning using both PCs and neural networks and prior hybrid approaches between these two classes of models.
Representation Learning with Probabilistic Circuits. PCs have seen significant advancements expressivity and learning algorithms including learning tensorized representations (Mari al., 2023; Liu et al., 2023a;b; Ventola et al., 2023; Gala et al., 2024a;b; Loconte et al., 2025a), sparse structures (Di Mauro et al., 2015; 2017; Trapp et al., 2019; Vergari et al., 2019c; Dang et al., 2022a; Yang et al., 2023), and extending them to have negative parameters (Loconte et al., 2024; 2025b; Wang & Broeck, 2025). Nevertheless, their application to representation learning in an autoencoding fashion has been comparatively limited. The main work in this area is aum-product autoencoding (SPAE; Vergari et al., 2018), which proposed two encoding strategies for Sum-Product Networks: CAT embeddings, derived from MAP inference over categorical latent. variables, and ACT emboddings, using circuit activations as continuous representations. However, SPAE derives these embeddings post-hoc from PCs trained for density estimation via maximum likelihood estimation. Crucially, there is no explicit loss function during SPAE's training that directly optimizes the quality of these embeddings, for example, based on their ability to generate accurate reconstructions. In Appendix D, we additionally evaluate the APC framework with an SPAE encoder and decoder and a neural decoder.
Other research has explored integrating PCs with deep generative models (DGMs). This includes using neural networks to guide PC structure learning or act as conditional components (Shih al., 2021; Shao et al., 2022; Martires, 2024), incorporating deep learning inductive biases into PCs (Ventola et al., 2020; Butz et al., 2019; Yu et al., 2022), or using PCs within hybrid DGMs for tasks like knowledge distillation
9

---

Published in Transactions on Machine Learning Research (12/2025)
in combination with normalizing flows and VAEs (Liu et al., 2023a;b; Pevný et al., 2020; Sidheekh et al., 2023; Correia. et al., 2023; Gala et al., 2024a). Dennis & Ventura (2017) proposed a hybrid model using two separate circuits for data and embeddings alongside a neural autoencoder to refine samples. In contrast to these approaches, APCs introduce a framework where probabilistic embeddings are explicitly modeled as random variables within a single PC encoder and are learned end-Lo-end.
Neural Autoencoders for Representation Learning. Neural autoencoders (AFs) form the foundation of many modern representation learning techniques (Hinton & Salakhutdinov, 2006). Early extensions like denoising autoencoders (DAEs; Vincent et al., 2008) and contractive autoencoders (CAEs; Salah et al., 2011) focused on learning robust and invariant features. The advent of variational autoencoders (VAEs; Kingma & Welling, 2014; Rezende et al., 2014) introduced a principled probabilistic approach, enabling generative modeling by learning a stochastic mapping to latent space, typically regularized towards a simple prior. Despite their success, VA often rely on approximate inference and simplified posterior distributions (e.g., Gaussians), which can limit their expressiveness. Significant research has aimed to overcome these limitations through richer posterior familics like normalizing flows (Rezende & Mohamed, 2015; Kingma et al., 2016; Louizos& & Welling, 2016; Horvat & Pfister, 2021; Draxler el al., 2024), hierarchical latent structures (Sønderby et al., 2016; Vahdat & Kautz, 2020), discrete representations like (Q-VAEs (van den Oord et al., 2017; Razavi et al., 2019), and improved training objectives (Burda et al., 2016; Higgins et al., 2017; Tomczak & Welling, 2018). Deterministic variants like regularized autoencoders (RAEs; Ghosh et al., 2020) have also been explored to structure the latent space without the sampling complexities of VAFs, often relying on post-hoc density estimation for generation.
Furthermore, various neural AE adaptations have been proposed to address the real-world scenario of missing data. These include techniques such as imporannc--eiggheee training (MIWAF, Mattei & Frellsen, 2019; not-MIWAE, Ipsen et al., 2021; SupMIWAE, Ipsen et al., 2022), tailored likelihoods for heterogeneous missing data (HI-VAE; Nazabal el al., 2020), modified losses in DAEs (mDAE; Dupuy et al., 2024), and advanced inference or sampling strategies in hierarchical or iterative models (Peis et al., 2022; Kuang et al., 2024). While powerful, these neural autoencoding paradigms, even those specialized for missing data, often involve approximate inference for their probabilistic embeddings
necessitate imputation strategies for incomplete
data. hnother self-supervised paradigm is the masked autoencoder (MAE; He et al., 2022), which is a vision baase (VIT; Dosovitskiy et al., 2020) approach that learns representations by encoding only a selected subset of visible input patches to reconstruct the masked oncs. MAEs are trained on specifically chosen patch masking patterns and map each unmasked input patch to its own patch-embedkding This is distinct from models like VAËs and APCs, which generate a single embedding for the entire input. A related category is the neural process (NP) family (Garnelo et al., 201 2018b;a; Kim al., 2019; Gordon et al., 2020). NPs learn distributions over functions by conditioning on set of observed context points to make predictions at arbitrary target locations. Their focus on learning a function-level representation from a context set distinguishes them from VAËs and APCs, which aim to learn a single, holistic embedding for an entire data instance. In contrast to these fully neural based models, APCs leverage a PC encoder for exact and tractable probabilistic inference over embeddings and the capability to handle missing observations via marginalization. The empirical comparison in the following scction will further substantiate these distinctions.
5 Empirical Evaluation
To measure the effectiveness of APCs, we conduct a comprehensive series of experiments designed to evaluate their performance and robustness across multiple tests against an array of model choices. Our evaluation aims to answer the following four key research questions. RQ1 (Section 5.1): How effectively can APCs reconstruct data, particularly in the presence of missing values, compared to established PC-based and neural approaches? RQ2 (Sections 5.2 and 5.3): Do the explicit probabilistic embeddings learned by APCs yield high-quality, informative representations that are beneficial for downstream classification lasks, especially when inputs are incomplete? RQ3 (Section 5.4): Do modern autoencoder architectures, training procedures, and higher model capacitics already improve robustness against missing data, or does this remain a challenge? RQ4 (Section 5.5): Can APCs act as effective student models for data-free knowledge distillation from pre-trained generative latent variable models and improve robustness to missing data compared to the teacher?
10

---

Published in Transactions on Machine Learning Research (12/2025)
MNNST
F-MNIST
SVHN
CIFAR
Mean Squared Error Mean Squared Error
100
150
100
100
50
100
50
Flowers
LSUN
CelebA
Tiny ImageNet
400
4000
4000
750
500
200
2000
2000
250
0
25
50
30
75
25
75
25
50
75
Missing Data (%)
missForest
VAE
MIWAE
SPAE
APC
Figure 5: APCs can deliver lower reconstruction errors than PC, VAE, and missForest baselines the asymptotic regime of MCAR-style randomly missing data. Reconstruction mean squared error is reported across all image datasets as the degree of MCAR corruption increases from 0% 95%.
To address these questions, we include the existing auttonncoinngg al. (2018) a PC-based reference, a vanilla variational autoencoder (Kingma & Welling, 2014) a neural autoencoding baseline, MIWAE (Mattei & Frellsen, 2019) as VAE variant specifically designed for handling missing data through multiple imputation rounds, and missForest (Stekhoven & Bühlmann, 2011), an iterative imputation method based on random forests a strong classic machinc learning bascline. All models are evaluated on common image benchmark datasels, as well as the 20 DEBD binary tabular datasets (Lowd Davis, 2010; Haaren Davis, 2012; Bekker et al., 2015; Larochelle & Murray, 2011).
SPAE introduced in Vergari
5.1 Reconstructions: Comparing Circuit and Neural Encoders
We assess reconstruction robustness against missing data by evaluating models under two primary missing data mochanisms: missing completely at random (MCAR) and missing at random (MAR). For MCAR, we introduce increasing levels of input corruption by randomly dropping pixels chosen from a uniform distribution, with the percentage of missing data ranging from 0% 95% in 5% steps. This is the default corruption method unless otherwisc specified. For MAR, evaluate specific structured missingnoss patterns, such missing entire regions of an image (e.g., left-to-right, center-to-bordets one could also argue that rather than MAR, this is a data-type specific structured version of MCAR. All models, with the exception of MIWAE, were trained on the complete datasets without missing values. For MIWAE, training was performed using MCAR-style data which 50% entries were missing. For each metric, we report the missing missinng-input MSE averaged over the different. levels of corruptions. We repeat all experiments five times with different. sceds and report their mean and standard deviation. We refer the reader to Appendix B for additional details. Notably, missForeet is imputation algorithm that operates directly in the input space, unlike the autoencoding methods, which learn compressed representation in embedding space. Consequently, when data is missing (0% corruption), missForest. can achieve a perfect reconstruction error of 0.0. In contrast, autoencoder-besse based approaches are inherently limited by their embedding bottleneck, resulting in non-zero reconstruction error even with complete inputs.
We first evaluate the reconstruction capabilities of APCs and examine performance on image data. In Fig. 5, show MSE reconstruction error trends (SSIM in Appendix Figure 12) as the percentage of MCAR-style missing pixels increases from 0% to 95%. While all models perform comparably at low corruption levels, their
11

---

Published in Transactions on Machine Learning Research (12/2025)
MNIST
CIFAR.
CelebA
LSUN
0%
50%
Vert
0%
50%
Vert
Ctr
50%
80%
Vert
Ctr
50%
80%
Vert
Ctr
Didia
ANA PANIE VAL SPRA 1al
FMNIST
SVHN
Flowers
Tiny ImageNet
0%
50%
80%
Vert
Ctr
0%
50%
80%
Vert
Ctr
0%
50%
80%
Vert
Ctr
0%
50%
80%
Vert Ctr
9
9
p
SPAE
ANC MIWAL VAE
10
91919
0
19 9 9
7
10
Figure 6: While neural encoder models quickly collapse with an increase in missing data ratio and mostly fail on MAR corruptions, APCs are able to maintain a stable reconstruction, even high degrees of MAR and MCAR corruption. The first row (Data) shows inputs with different corruptions, and subsequent represent the respective model reconstructions.
results diverge quickly as corruption increases: ncural autocncoders show rapid performance degradation. At the same time, APCs maintain lower reconstruction error even at high corruption levels. is important to highlight. that APC, VAE, and MIWAF utilize the same neural decoder architecture. Therefore, the observed differences performance can be solely attributed the choice of the encoder model and the corresponding encoding scheme. APCs demonstrate superior performance across all datasets, with lower MSE and higher SSIM compared to both neural and PC-based models. As an aggregate, additionally provide the average reconstruction error over all corruption levels for MSE in Appendix Table 2 and for SSIM in Appendix Table 5.
In addition, qualitative examples reconstructions with missing data in Fig. visually confirm our itantitative findings. We show reconstructions for 0%, 50%, and 80% MCAR-style missing data and vertical-band well as center-square MAR-style missing data. As missing data increases, neural autoencoders produce increasingly blurry and eventually unrecognizable reconstructions. In contrast, APCs maintain recognizable reconstructions even at high levels of missing data or images missing complete patches, preserving key structural elements and details across all datasets. We attribute this robustness to the encoder circuit's 's ability to handle partial observations through tractable marginalization natively.
While image data inherently contains spatial structure and corrclations that neural architectures can exploit through layers with implicit biases such convolutions, tabular data typically lacks such inherent structure. We therefore additionally investigate whether APCs' strong performance image datasets generalizes to
12

---

Published in Transactions on Machine Learning Research (12/2025)
MNIST
F-MNIST
Flowers
CIFAR
Downstreeam Accuracy
80
80
60
10
30
60
20
5
20
20
10
25
50
75
25
30
75
25
50
75
25
50
75
Missing Data (%)
VAE
MIWAE
ACA
SPAR-CAT
APC
Random Baseline
Figure 7: APCs produce linearly seperable embeddings better than VAEs. Downstream task accuracy using Logistic Regression model under different MCAR-style corruption levels (0% to 95%) for MNIST, F-MNIST, Flowers, and CIFAR. We observe that APCs generate embeddings even under high corruption levels that are still linearly separable. Neural encoders (VAE/MIWAE), on the other hand, quickly collapse their embedding representation and thus drop in downstream task performance, when data is missing,
tabular data. Appendix Table 3 presents reconstruction performance the 20 DEBD binary tabular datascts under increasing percentages of missing values. APCs outperform all other models 18 out of 20 datasets: demonstrating that their advantages are not limited to image data but generalize to tabular datasets. This is particularly significant for tabular data applications where missing values are common in real-world scenarios, such as medical records, survey and financial data, or environmental modeling.
5.2 Embedding Quality and Downstream Task Performance
While quantitative and qualitative experiments in Section 5.1 so far have investigated reconstruction per- formance, a key indicator of representation learning quality is how well the embeddings can be leveraged for downstream task applications after unsupervised pre-trainèng of the antoencoding model. Following a common practice in representation learning for evaluating the quality of unsupervisedly learned features (Bengio al., 2012; Alain & Bengio, 2017), we assess the utility of our models' embeddings by treating them as fixed imputs for supervised classifier. For this, logislic regression model was trained iteratively using stochastic gradient descent (SGD) on these extracted embeddings. We specifically use a logistic regression model to evaluate the Icarned embeddings' ability to capture independent and useful data reprosentations. For further details on the training protocol of this downstream lask, we refer the reader to Appendix B.
The downstream task performance measured by the downstream task classification accuracy (DS-Acc. 1), providing insights into the quality and lincar separability of the learned representations. Fig. 7 presents downstream classification accuracy of the logistic regression model trained embeddings extracted from four image datasets under increasing levels of MCAR-style corruption. The results reveal a stark contrast between APCs and neural autoencoder variants: while neural encoders achieve slightly higher accuracy with complete data (0% missing), APC embeddings maintain their classification performance even as corruption increases to severe levels (90%), whereas neural encoder embeddings show immediate degradation in downstream performance, quickly converging towards a bascline random guessing estimator, when data is missing. This pattern is consistent across all datasets. The robustness to increasing proportions of missing data can again be attributed to the circuit's fundamental capability for tractable marginalization, allowing inference of embeddings even with incomplete observations without additional imputation methods.
To provide visual insight into these performance differences, Fig. 8 shows (-SNE projections of embedding spaces for the MNIST test dataset split at different corruption levels. Because SPAE-ACT embeddings represent circuit activations, which are log-likelhhodds and can have large negative values, we normalize them before t-SNE to ensure a comparable input range with other models. With complete data (0% corruption), both neural autoencoders and APCs produce well-structured embeddings with clear class
13

---

Published in Transactions on Machine Learning Research (12/2025)
0%
30%
50%
70%
90%
0
/ 2 3
4
0
/
3
44
Data
9
6564
8
VAE
MIWAE
SPAE-CAT SPAE-ACT
APC
Figure &: APCs keep a stable embedding space even when corruption is high. T-SNE, projections of MNIST model embeddings for different corruption levels of missing data. Each dot an MNIST test data point and colored according to its class. While neural encoders (*AE) initially (0%) can separate different classes in the embedding space, their representation starts to heavily degrade with increasing corruption levels until no separation is possible.
separation. However, as corruption increases, neural encoder, even MIWAE, which is specifically trained on missing data, embeddings progressively collapse into an unstructured mass with no class separation. contrast, APC emboddings maintain their structure with distinct class clusters even at 90% corruption, visually confirming our quantitative results of Fig. and their robustness to missing data.
5.3 Embedding Space Analysis
A key advantage of APCs is the fact that the encoder models the joint data-embedding distribution pe (X, Z) explicitly and can tractably compute marginals, enabling additional probabilistic capabilities compared to standard and variational autoencoders. In this section, we briefly examine APCs from a probabilistic perspective by investigating the embedding space and samples decoded from the embedding space.
Unlike VAEs, which implicitly model the approximate posterior distribution Z X) using neural network and rely approximate inference, APCs directly model the joint distribution Pe (X, Z) and allow for exact and tractable probabilistic queries. VAËs are trained to encourage the encoder's 's approximate posterior gZ|| to approach prior distribution, typically standard normal distribution (00 1). This is achieved by incorporating KL term in the loss function that measures the discrepancy between rZ X) and (/ 1). Samples are drawn from (0, after training, assuming the learned approximate posterior a good approximation of the true postcrior p(Z| X). However, the encoder rarely perfectly matches the true posterior, leading to mismatch between the true posterior and the distribution used for sampling. In contrast, APCs allow to sample embeddings directly from the exact marginal distribution z ~ PZ()), obtained tractably
14

---

Published in Transactions on Machine Learning Research (12/2025)
MNIST
LSUN
CelebA
Flowers
1
3
7
APC
7
4 6
1
0
1246
4 1
PC
6676 4
Figure 9: APCs circumvent known visual sampling artifacts of traditional PCs. APC and vanilla PC samples from models trained on MNIST, LSUN (Church), CelcbA, and Flowers. APCs successfully learn the data-generating distribution and can produce novel samples.
through marginalization of the PC: J Pe Z) dX. These emboddings are then decoded to generate new data samples. Fig. 9 showcases the decoded embedding samples across datasets of varying complexity and contrasts them with PC solely trained with MLFE. The generated samples exhibit diversity and maintain the structural characteristics of their respective datasets, ranging from clear digit forms in MNIST to more complex architectural features in LSUN and facial attributes in CelebA. This is additionally confirmed by our quantitative evaluation of FID and KID scores in Appendix Table 10. Vanilla PC samples exhibit well-known visual artifacts related to the circuit structure. APCs circumvent this by deferring the decoding to a neural network. This demonstrates that APCs successfully capture the underlying data distribution through their tractable encoder, generating higher quality samples by decoding from the embedding distribution and natively avoiding posterior approximation discrepancies inherent in VAEs due to the intractable nature of their posterior distributions.
Furthermore, Tomezak & Welling (2018) have shown that the optimal prior p²(z) after fitting a VAE is the aggregate variational postcrior E (z xx). The discrepancy between this complex multimodal distribution and the simple unimodal prior lypically used for sampling leads to the "prior hole problem" (Hoffman & Johnson, 2016; Aneja et al., 2020), where the prior assigns high probability to regions of the embedding space that are not covered by the aggregate posterior. Decoding samples from these "holes" often results in poor quality generations. To mitigate this, standard approaches often resort to a two-stage process: training the VAE first, and subsequently fitting a separate density estimator (e.g., a GMM or another VAF) to the aggregate posterior samples post-hoc (Ghosh et al., 2020; Dai & Wipf, 2019). APCs circumvent this issue entirely without requiring second training stage. The marginal distribution Pc Pc(Z) of the encoder can be viewed direct approxination of this aggregate posterior. Since APCs model the joint distribution, this marginal naturally covers the regions of the embodding space occupied by the data.
5.4 Model Capacity Is Insufficient for Missing Data Robustness Even in State-Of-The-Ae VAEs
Our preceding analyses have primarily focused on neural encoder and decoder architectures. However, the autoencoding, particularly the VAE landscape, has witnessed significant advancements, leading to more sophisticated and complex architectures and training procedures. Notable examples include Vector Quantized VAEs (VQ-VAE) (van den Oord et al., 2017) and VQ-VAE2 (Razavi et al., 2019), as well as very deep models like VDVAE (Child, 2020) and hierarchical ones like NVAE (Vahdat & Kautz, 2020). This progress raises critical question: docs the sheer scale and complexity of modern VAES inherently grant them robustness against missing data? While specialized methods and training recipes exist for this problem, our goal here is not to make a conclusive claim about the general capabilities of all modern VAEs. Instead, we aim to preliminarily investigate whether scaling up a VAE architecture is in itself sufficient to overcome the challenges posed by incomplete inputs. We therefore evaluate NVAE under MAR and MCAR corruptions as representative for modern, high-capacity VAE architectures.
15

---

Published in Transactions on Machine Learning Research (12/2025)
CIEAR
CelebA 90%
0%
70%
90%
Right
Center
0%
70%
Right
Center
Data
NVAE
Finet.
Figure 10: Even modern VAEs cannot natively handle data corruption robustly. Example reconstructious from NVAE and MCAR-finetuned NVAE on CIFAR and CelebA samples with MCAR/MAR corruption. Standard NVAE retains zero-imputed values, collapsing high corruption (e.g., 90%). MCAR- finetuning offers marginal improvement for MCAR patterns but fails to generalize to other MAR patterns (e.g,, missing right half or square in the center).
We employed the official publicly available pretrained NVAE models for CIFAR and CelebA from Vahdat & Kautz (2020). These models were evaluated on image reconstruction tasks with varying levels of data missingness: 0%, 70,, and 90% using MCAR-style patterns and two MAR-style patterns (missing right-half, missing center-square),
illustrated in Fig. 10. Initial experiments revealed that the base NVAE model tended to either keep the zero-imputed values exhibit failure modes such oversaturation, which not seen in previous VAE experiments. To address this, we finetuned each NVAE model (labeled "Finet." in Fig. 10) on its respective dataset with MCAR patterns of 50% uniformly missing pixels for an additional 25% of its original training epochs. This finetuming process requires careful calibration, as finetuning degrades the model's original reconstruction capabilities or can lead to complete collapse. As depicted in Fig. 10, the base NVAE achieves nearly perfect reconstructions on complete data, whereas the finetuned model exhibits significantly poorer performance. This observation is supported by substantial increase in bits- fits-per-dimension (BPD) on the full test sets: from 2.91 to 16.93 for CIFAR and from 2.27 to 104.26 for CelebA, for the base to finetuned models, respectively. Although the finetuned model demonstrates some ability to reconstruct images with 70% missing pixels, it remains susceptible to similar failure modes as the base model at 90% missingness and cannot reconstruct MAR-style corruptions at all.
5.5 Application: Data-Free Knowledge Distillation
Pro-trained autoencoding models are increasingly shared online and offer powerful generative capabilities. However, a significant limitation, demonstrated in our earlier evaluations (Sections 5.1 to 5.3), is their oflen fragile performance when confronted with MCAR and MAR corrupted input data. This raises a practical question: can we transfer the generative knowledge of a pretrained VAE to our APC framework when the original training data is unavailable? Knowledge distillation (KD) has emerged as a powerful technique for model compression, adaptation, and continual learning by transferring knowledge from a teacher model to a student model (Hinton al., 2015). KD has seen an increasing amount of application in natural language processing (Tang et al., 2019; Jiao et al., 2020; Mon et al., 2016), computer vision (Liu et al., 2017; Zhou et al., 2018; Yim et al., 2017), and speech recognition (Hinton et al., 2015; Lu et al., 2017; Ramsay et al., 2019). Traditionally, this process requires access to the original training data, which may not always be available due to privacy concerns, proprietary restrictions, or storage coistraints. Data-free knowledge distillation seeks to overcome these challenges by extracting transferable knowledge solely from the teacher's 's learned representations without requiring access to original data. Approaches range from generative adversarial network (Goodfellow et al., 2014) based and inspired methods (Chen et al., 2019; Han et al., 2021) to those that first generate synthetic data through specific distillation objectives and then train students this
16

---

Published in Transactions on Machine Learning Research (12/2025)
Teacher (VAB)
MSE (CelebA)
APC VAE Corr. Dats
Student (APC)
30000
2000
0
100
DS- Acc. (MNIST)
APC
VAE
20
60
Missing Data (%)
Figure 11: APCs are robust data-free knowledge distillers. Data-free Knowledge Distillation from VAE to APC. The distilled APC model can learn the VAE's distribution and improves in reconstruction and downstream performance under data corruptions.
synthetic data (Mordvintsev et al., 2015; Yin et al., 2020; Braun et al., 2023), align intermediate features between teacher and student (Wang, 2021a) black-box hard-label basod methods (Wang. 2021b).
In this section, we explore the capabilities of APCs as robust data-free knowledge distillers. Given the explicit probabilistic nature and tractable inference in APCs, investigate their potential student models to distill knowledge from generative latent variable models, such as VAEs, while circumventing the need for the original dataset. This approach allows us to examine whether APCs can learn the data distribution captured by a VAE teacher, while additionally benefiting from APCs robustness shown in the previous sections. We outline the data-free knowledge distillation procedure in Appendix Algorithm 2. In short, we sample an embedding from the APC prior (Z) which is used by the teacher VAE to generate synthetic data samples. We then train the APC according to the same objectives outlined in Section 3 by treating the synthetic samples roconstruction ground truth.
In Fig. 11, we show the results of knowledge distillation from VAE to an APC of similar capacity and the same decoder architecture illustratively on two example datasets. The top graph quantifies reconstruction error on CelebA, measured as the MSE, as a function MCAR-style missing data percentage. The teacher VAE exhibits a sharp increase MSE as data corruption intensifies. In contrast, the student APC maintains lower reconstruction error, demonstrating its effectiveness in knowledge distillation and showing the same characteristics of APCs obscrved in the previous sections. In addition, we highlight the qualitative results of reconstructions with 80% randomly missing data the right, where the APC student model more faithfully reconstructs corrupted images compared to its teacher VAE model.
The bottom graph evaluates downstream task performance on MNIST, measured by logistic regression classification accuracy using embeddings obtained from the models. As missing data increases, as observed earlier, the teacher VAE's performance rapidly deteriorates, whereas the student APC maintains high accuracy, indicating superior robustness of its learned embeddings towards corruptions. We once more highlight these results qualitatively
the right t-SNE visualizations of the embedding spaces at 80% missing data, which further reinforces our findings: While the teacher VAE embeddings collapse into dispersed and unstructured distribution, the student APC embeddings maintain well-scparated clusters, suggesting better downstream utility. To provide further evidence, we repeat these experiments for every image and tabular dataset explored in Section and show results for teacher and student reconstructions and downstream task accuracy under full evidence (Full Evi.) and average MSE over an increasing level of corruption (0% 95% in 5% steps) Appendix Table 8. The results confirm that APCs closely match the VAE's performance under full evidence while surpassing it in robustness to missing data, achieving lower reconstruction errors and higher downstream
17

---

Published in Transactions on Machine Learning Research (12/2025)
Table 1: Every APC component makes a distinct and complementary contribution to the framework. Ablation study examining the contribution of key APC components: differentiable sampling for end-to-end training (DifI.), neural decoder (NN-Dec.), KL. embedding regularization (Z-Reg-); and joint data-embedding likeleelioood regularization (p(x, z)). Performance is evaluated using MSE (4), downstream classification accuracy (DS-Acc. 1) under full evidence conditions (Full Evi.), and average MSE metrics measuring robustness.
MSE (4)
DS-Acc (1)
Diff.
NN-Dec.
Z-Reg.
p(x, z)
Full Evi.
MCAR
Full Evi.
MCAR
W S
x
79.98 210.3
63.84 17336
69.28 69
39.16 15.43
55.21 +1.24
26.57 +0.48
59.69 +3.05
58.01 +3.11
LSISZT
/
/
20.73 20.6
17.12 1:030
50.70 13.30
45.79
13.24
X
V
8.18 20.07
8.43 10.02
88.13 20.22
82.94 40.10
V
7.45 20 19
19.87 11.14
85.98 11.77
65.58
13366
7.60
10.07
+.021
86.62 +0.20
81.97
4.13 20.10
5.06
20.05
88.32
20.14
86.85
#0.11
X
441.32 +17.5
291.15
+1.45
34.76
+0.61
24.32
+0.61
193.77 20.00
92.07
1:000
29.80
1:
29.51
1:000
TEAR
60.55
68.31
1:0.0
33.09
29.24 19.24
#:0.06
25.45
22.52
+0.11
36.39
+0.44
31.11
+0.18
35.70 20.33
67.52
1.056
33.71
10.27
25.79 10.50
17.04 29.19
28.23
1:024
35.47
40.2
34.13
20.09
16.29 20.14
20.64
10.0s
37.61
20.27
36.90
10.14
accuracy across most datascts. This highlights APCs' ability effectively distill knowledge from VAEs without access original training data while enhancing robustness to data corruption.
5.6 Ablation Studies
Following extensive evaluation of APCs across diverse range of datasets and against circuit-based, neural, and classic models in the previous sections, contribution of each component within our framework, thereby strengthening the understanding of APCs. As shown in Table 1, we analyze both the removal of all components simultaneously and the exclusion of individual components while maintaining others. We evaluate four key architectural elements: (1) differentiable sampling (DifT.), which enables end-to-end gradient flow between the encoder circuíl. and decoder network; (2) the neural decoder (NN-Dec.), as opposed using the circuit itself for reconstruction;
conduct ablation study to isolate and quantify the
(3) embedding regularization via KL against standard Gaussian prior (Z-Reg.); and (4) joint data-embedding logllikelihood regularization (p(x,z)) For each configuration, Table 1 reports MSE and downstream classification accuracy IDS-Acc.) under full evidence (Full Evi.) conditions, along with the mean MCAR-style corruptions (MCAR) for both MNIST and CIFAR datasets. We deliberately selected these two datasets to evaluate our components across different complexity levels: MNIST serves as a relatively simple benchmark with clear digit structures, while CIFAR represents a more challenging natural image dataset with complex objects, backgrounds, and color variations. We maintain consistent model capacity throughout the experiments to ensure fair comparison across all configurations. Specifically for configurations where NN-Dec. x (indicating the absence of neural decoder), double the encoder size to match the total parameter count of complete APC model, where capacity is evenly distributed between encoder and decoder.
The ablation results in Table 1 highlight the indispensable role of each component in the APC framework. The baseline configuration (first row) is a circuit that performs both encoding and decoding while being optimized solely through reconstruction error minimization, mimicking a vanilla autoencoding scheme without additional objectives. Differentiable sampling (Diff.) is fundamental, as it ensures end-to-end gradient flow from the neural decoder back to the tractable encoder, optimizing the learned representations for reconstruction.
18

---

Published in Transactions on Machine Learning Research (12/2025)
We evaluate differentiable sampling with SIMPLE "v" and with Gumbel-Softmax "(V)". Without this connectivity, training fails to propagate meaningful gradients, resulting in catastrophic degradation: MSE increases by factors of 13.4x on MNIST and 11.9x on CIFAR. Similarly, the neural decoder (NN-Dec.) plays a pivotal role in reconstruction and downstream performance, as its absence leads to severe drops in reconstruction quality, increasing MSE by 5x and 3.7x on MNIST and CIFAR, respectively. This impact is even more pronounced corruption MCAR metrics, demonstrating the decoder's importance for robustness against missing data. Embedding regularization (Z-Reg.) further refines representation learning, nearly halving the MSE on MNIST (from 8.18 to 4.13) and significantly improving the corruption MCAR MSE from 8.43 to 5.06. These results confirm that each component makes a distinct and complementary contribution to the APC framework, with their combination yielding better performance than any partial implementation.
6 Conclusion
In this work, we introduced autoencoding probabilistic circuits, a novel framework for representation learning in PCs that leverages their tractability to model explicit probabilistic embeddings. Autoencoding approaches such as VAEs typically rely on neural networks, which often yield implicit and intractable probabilistic formulations. In contrast, APCs model the joint distribution of data and embeddings using a PC encoder. This enables tractable conditional sampling for encoding, yielding embeddings that are samples from a well-defined conditional probability distribution. Our hybrid architecture combines a PC encoder with a neural network decoder, leveraging the complementary strengths of PCs for tractable probabilistic encoding and the modeling capacity of neural networks for decoding. Our empirical evaluations across a wide range of image and tabular datasets demonstrate the effectiveness of APCs. Crucially, APCs show superior reconstruction performance compared to existing circuit-based autoencoding schemes, and to neural encoders in the face of increasing data corruption and missing values. Furthermore, embeddings learned by APCs prove to be robust and meaningful, maintaining downstream task performance even under high data corruption.
Future Work. While our study has thoroughly examined the APC framework, numerous avenues remain for further research. Using a circuit-based encoder grants explicit control over both data and embedding distributions, lifting the typical restriction to independent Gaussians, as seen VAEs. Moreover, the circuit. structures employed in this work were selected heuristically. However, the broader literature on PCs offers various approaches for learning circuit structures. Incorporating these methods could enable more strategic placement of embedding input units, potentially distributing them across different circuit levels. This could facilitate the formation of low-level and high-level representations, introducing hicrarchical structures in the learned embedding space. Furthermore, Section 5.4 shows that while modern VAEs achieve high-quality reconstructions on complete data using novel scaling and training tricks, they remain vulnerable to corruption. We could thus apply analogous methodologies scale PC encoders, aiming for NVAE-level reconstruction fidelity on complete data while preserving APCs' inherent robustness to data corruptions.
Acknowledgments
This work has benefited from the DYNAMIC center, which is funded by the LOFWE program of the Hessian Ministry of Science and Arts (Grant Number: LOEWE1 00////3399000 00w.0001100000880 as well as the carly stages of the cluster projects by the Deutsche Forschungsgemeinschatt (DFG, German Research Foundation) under Germany's Excellence Strategy Strateyy-"Resonnade "Reasonable AI" (EXC-3057) and "The Adaptive Mind" (EXC-3066);
funding will begin in 2026. Antonio Vergari was funded by the "UNREAL: Unified Reasoning Layer for Trustworthy ML" project (EP/Y023838/1) 1) selected by the ERC and funded by UKRI EPSRC.
References
Kareem Ahmed, Stefano Teso, Kai-Wei Chang, Guy Van den Brocck, and Antonio Vergari. Semantic probabilistic layers for neuro-symbolic learning. In Advances in Neural Informalion Processing Systems (NeurIPS), 2022.
Kareem Ahmed, Zhe Zeng, Mathias Niepert, and Guy Van den Broeck. Simple: A gradient estimator for k-subset. sampling. In International Conference on Learning Representations (ICI.R, 2023.
19

---

Published in Transactions on Machine Learning Research (12/2025)
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. International Conference on Learning Representations (ICLR), 2017.
Jyoti Aneja, Alexander G. Schwing, Jan Kautz, and Arash Vahdat. A contrastive learning approach for training variational autoencoder priors. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVilo, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjic Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. Py'Torch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation. In 29th ACM International Conference on Architectural Support for Programming Languages and Operaling Systems, Volume 2 (ASPLOS '24). ACM, 2024. URL https://pytoree org/ assets/ pensstss/pytrcch2-c pytoochh-2.pppe
Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael G. Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint-embedding predictive architecture. Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Jessa Bokker, Jesse Davis, Arthur Choi, Adnan Darwiche, and Guy Van den Brocck. Tractable learning for complex probability queries. In Advances in Neural Information Processing Systems (NeurIPS), 2015.
Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Representation learning: A review and new perspectives. Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2012.
Steven Braun, Martin Mundt, and Kristian Kersting. Deep classifier mimicry without data access. In International Conference on Arlificial Intelligence and Slalisties (AISTATS), 2023.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In International Conference on Learning Representations (ICLR), 2016.
Cory J. Butz, Jhonatan de S. Oliveira, André E. dos Santos, and André L. Teixcira. Deep convolutional sum-product networks. In Associalion for the Advencement of Artificial Intelligence (AAAT), 2019.
Hanting Chen, Yunhe Wang, Chang Xu, Zhaohui Yang, Chuanjian Liu, Boxin Shi, Chunjing Xu, Chao Xu, and Qi Tian. DAFT: Data-free learning of student networks. In International Conference on Computer Vision (ICCV), 2019.
Rewon Child, Very deep vaes generalize autoregressive models and can outperform them on images, ar preprint arXiv:201 1. 10650, abss2001..1663 2020.
YooJung Choi, Antonio Vergari, and Guy Van den Broeck. Probabilistic circuits: A unifying framework for tractable probabilistic models. Technical report, University of California, Los Angeles (UCLA), 2020.
Mark Collier, Alfredo Nazabal, and Chris Williams. Vaes in the presence of missing data. In 1CML Workshop on the Art of Learning with Missing Values (Artemiss), 2020.
Alvaro H. C. Correia, Gennaro Gala, Erik Quaeghebeur, Cassio de Campos, and Robert Peharz. Continuous mixtures of tractable probabilistic models. In Association for the Advancement of Artificial Intelligence (AAAT), 2023.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations (ICLR), 2019.
Mcihua Dang, Anji Liu, and Guy Van den Broeck. Sparse probabilistic circuits via pruning and growing. In A Aduances in Newral Information Processing Systems (NewrIPS), 2022a.
20

---

Published in Transactions on Machine Learning Research (12/2025)
Meihua Dang, Anji Liu, Xinzhu Wei, Sriram Sankararaman, and Guy Van den Broeck. Tractable and expressive generative models of genetic variation data. In Research in Computational Molecular Biology. 2022b.
Adnan Darwiche. A differential approach to inference in bayesian networks. Journal of ACM (JACM), 2003a.
Adnan Darwiche. A differential approach to inference in bayesian networks. JACM, 2003b.
Adnan Darwiche and Pierre Marquis. A knowledge compilation map. JAIR, 2002.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
Aaron Dennis and Dan Ventura. Autoencoder--nbaanced sum-product networks. In IEEE International
Conference on Machine Learning and Applications (ICMLA), 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training deep bidirectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
Nicola Di Mauro, Antonio Vergari, and Floriana Esposito. Learning accurate cutset networks by exploiting decomposabllity. In International Conference of the Italian Association for Artificial Intelligence (AIXIA). Springer, 2015.
Nicola Di Mauro, Antonio Vergari, Teresa MA Basile, Floriana Esposito, et al. Fast and accurate density ostimation with extremely randomizod cutset networks. In European Conference Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD), 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2020.
Felix Draxler, Peter Sorrenson, Lea Zimmermann, Armand Rousselot, and Ullrich Köthe. Free-form flows: Make any architecture a normalizing flow. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2024.
Marictte Dupuy, Marie Chavent, and Remi Dubois. mdae : modified denoising autoencoder for missing data imputation. Xiv preprint or 2rx: abs/2411.128477, 2024.
Foderico Errica and Mathias Niepert. Tractable probabilistic graph representation lcarning with graph-inducod sum-product networks. arXiv preprint arXiv:2905.105/ 2023.
William Falcon and The PyTorch Lightning team. PyTorch Lightning, March 2019. URL https://github. Com/LigghtinngAA/ligghhee
Gennaro Gala, Cassio de Campos, Robert Peharz, Antonio Vergari, and Erik Quaeghebeur. Probabilistic integral circuits. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2024a.
Gennaro Gala, Cassio P de Campos, Antonio Vergari, and Erik Quaeghebeur. Scaling continuous latent variable models as probabilistic integral circuits. Advances in Neural Information Processing Systems (NeurIPS), 2024b.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning (ICML), 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes, 2018b.
21

---

Published in Transactions on Machine Learning Research (12/2025)
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational to deterministic autoencoders. In International Conference on Learning Representations (TCT.R), 2020-
Lovedeep Gondara and Ke Wang. Mida: Multiple imputation using denoising autoencoders. In Pacific-Asie Conference on Knowledge Discovery and Data Mining, 2018.
Tan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Newral Information Processing Systems (NewrIPS), 2014.
Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. In International Conference on Learning Represenlations (ICLR), 2020.
Ayesha Gurnani, Viraj Mavani, Vandit, Gajjar, and Yash Khandhediya. Flower categorization using deep convolutional neural networks, 2017.
Jan Van Haaren and Jesse Davis. Markov network structure learning: A randomized feature generation approach. In Association for the Advancement of Artificial Intelligence (AAAT), 2012.
Pengchao Han, Jihong Park, Shiqiang Wang, and Yejun Liu. Robustness and diversity secking data-free knowledge distillation. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2021.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in Newral Information Processing Systems (NeurIPS), 2017.
Irina Higgins, Loie Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations (TCLR), 2017.
GeofTrey Hinton, Oriol Vinyals, and JefTrey Dean. Distilling the knowledge a neural network. In Deep Learning and Representation Learning Workshop (NIPS), 2015.
Geoffrey E. Hinton and Ruslan Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 2006.
Matthew D Hoffman and Matthew J Johnson. Elbo surgery: yet another way to carve up the variational evidence lower bound. In Workshop in Advances in Approzimate Bayesian Inference, NIPS, 2016.
Christian Horvat and Jean-Pascal Pfister. Denoising normalizing flow. Advances Neural Information Processing Systems (NeurIPS), 2021.
Nicls Bruun Ipsen, Plerae-Ae Mattci, and Jes Frollson. not-miwac: Deep generative modclling with missing not at random data. In ICLR 2021-Interrational Conference on Learning Representations, 2021.
Niels Bruun Ipsen, Pierre-Alexande Mattei, and Jes Frellsen. How to deal with missing data in supervised deep learning? In 10th International Conference on Learning Representations, 2022.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Athresh Karanam, Saurabh Mathur, Sahil Sidheckh, and Sriraam Natarajan. A unified framework for human-allied learning of probabilistic circuits. In Association for the Advancementt of Artificial Intelligence (AAAT), 2025.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint 1901.000177, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Diederik P. Kingma and Max Welling. Auto-oncoding variational bayes. In International Conference on Learning Represenlations (ICLR), 2014.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, llya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems (NewrIPS), 2016.
Doga Kisa, Guy Van den Broeck, Arthur Choi, and Adnan Darwiche. Probabilistic sentential decision diagrams. In International Conference on Principles of Knowledge Representation and Reasoning (KR), 2014.
Alex Krizhovsky. Learning multiple layers of features from tiny images. Technical report, U. Toronto, 2009.
Shenfen Kuang, Jie Song, Shangjiu Wang, and Huafeng Zhu. Variational autoencoding with conditional iterative sampling for missing data imputation. Mathematics, 2024.
Leander Kurscheidt, Paolo Morettin, Roberto Sebastiani, Andrea Passerini, and Antonio Vergari. A proba- bilistic nouro-symbolic layer for algebraic constraint satisfaction. In The 41st Conference Uncertainty in Artificial Intelligence, 2025.
Matt J Kusner, Brooks Paige, and José Miguel Hornandez-Lobato. Grammar variational autoencoder. In International conference on machine learning, pp. 1954. PMLR, 2017.
Steven Lang, Martin Mundt, Fabrizio Ventola, Robert Peharz, and Kristian Kersting. Elevating perceptual sample quality in probabilistic circuits through differentiable sampling. In Proceedings of Machine Learning Research (PMLR), 2022.
H. Larochelle and lain Murray. The neural autoregressive distribution estimator. In International Joint Conference Artificial Intelligence (IJCAT), 2011.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 1998.
Valentin Liévin, Andrea Dittadi, Lars Maalee, and Ole Winther. Towards hierarchical discrete variational autoencoders. In Proceedings of the 2nd Symposium on Advances in Approzimate Bayesian Inference, 2019.
Auji Liu, Stephan Mandt, and Guy Van den Broeck. Lossless compression with probabilistic circuits. In International Conference on Learning Representations (ICLR), 2022.
Anji Liu, Honghua Zhang, and Guy Van den Broeck. Scaling up probabilistic circuits by latent variable distillation. In International Conference Learning Representations (ICLR), 2023a.
Xuejie Liu, Anji Liu, Guy Van den Broeck, and Yitao Liang. Understanding the distillation process from deep gencrative models to tractable probabilistic circuits. In International Conference on Machine Learning (ICML), 2023b.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning efficient convolutional networks through network slimming. In International Conference on Computer Vision (ICCV), 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In International Conference on Computer Vision (ICCV), December 2015.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Lorenzo Loconte, Nicola Di Mauro, Robert Peharz, and Antonio Vergari. How to turn your knowledge graph embeddings into generative models via probabilistic circuits. In Advances in Neural Information Processing Systems (NewrIPS), 2023.
Lorenzo Loconte, Aleksanteri Mikulus Sladek, Stefan Mengel, Martin Trapp, Arno Solin, Nicolas Gillis, and Antonio Vergari. Subtractive mixture models via squaring: Representation and learning. In International Conference on Learning Representations (ICLR), 2024.
Lorenzo Loconte, Antonio Mari, Gennaro Gala, Robert Peharz, Cassio de Campos, Erik Quaeghebeur, Gennaro Vessio, and Antonio Vergari. What is the relationship between lensor factorizations and circuits (and how can we exploit it)? Transactions on Machine Learning Research (TMLR), 2025a.
Lorenzo Loconte, Stefan Mengel, and Antonio Vergari. Sum of squares circuits. Association for the Advancement of Artificial Intelligence (AAAI), 2025b.
Iya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. arXiv preprint arXis:1711.05100 2017.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In International Conference on Machine Learning (ICML), 2016.
Daniel Lowd and Jesse Davis. Learning markov network structure with decision trees. 2010 IEEE International Conference on Data Mining, 2010.
Liang Lu, Michelle Guo, and Steve Renals. Knowledge distillation for small-footprint highway networks. In International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2017.
Yonghong Luo, Xiangrui Cai, Y. Zhang, Jun Xu, and Xiaojie Yuan. Multivariate time series imputation with generative adversarial networks. In Advances in Neural Information Processing Systems (NewrIPS), 2018.
Antonio Mari, Gennaro Vessio, and Antonio Vergari. Unifying and understanding overparameterized circuit representations via low-rank tensor decompositions. In The 6th Workshop on Traclable Probabilistic Modeling, 2023.
Pedro Zuidberg Dos Martires. Probabilistic neural circuits. In Association for the Advancement of Artificial Intelligence (AAAI), 2024.
Saurabh Mathur, Vibhav Gogate, and Sriraam Natarajan. Knowledge intensive learning of cutset networks. In Uncertainty in Artificial Intelligence (UAT), 2023.
Saurabh Mathur, Alessandro Antonucci, and Sriraam Natarajan. Knowledge intensive learning of credal networks. In Uncertainty in Artificial Intelligence (UAI), 2024.
Plerre-Alexandre Mattei and Jes Frellsen. Miwae: Deep generative modelling and imputation of incomplete data sets. In International Conference on Machine Leurning (ICML), 2019.
Jun Mei, Yong Jiang, and Kewei Tu. Maximum posteriori inference in sum sum- num-product networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Alexander Mordvintsav, Christopher Olah, and Mike Tyka. Inceptionism: Going deeper into neural networks, 2015. URL https://ai.
googleblog. https:///a..ooglebllggg..o
noonng-eeper--nnoonnnooeeee int---inntonnllllnin html.
Lili Mou, Ran Jia, Yan Xu, Ge Li, Lu Zhang, and Zhi Jin. Distilling word embeddings: An encoding approach. In International Conference on Information and Knowledge Management (CIKM), 2016.
Alfredo Nazabal, Pablo M Olmos, Zoubin Ghahramani, and Isabel Valera. Handling incomplete heterogeneous data using vaes. Pattern Recognition, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsuperoised Feature Learning, 2011.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Robert Peharz, Robert Gens, Franz Pernkopf, and Pedro M. Domingos. On the latent variable interpretation sum-product networks. Transactions on Pattern Analysis and Machine Intelligence (TPAMT), 2017.
Robert Pcharz, Steven Lang, Antonio Vergari, Karl Stelzner, Alejandro Molina, Martin Trapp, Guy Van den Broeck, Kristian Kersting, and Zoubin Ghahramani. Einsum networks: Fast and scalable learning of tractable probabilistic circuits. In International Conference on Machine Learning (ICML), 2020a.
Robert Peharz, Antonio Vergari, Karl Stelzner, Alejandro Molina, Xiaoting Shao, Martin Trapp, Kristian Kersting, and Zoubin Ghahramani. Random sum-product networks: A simple and effective approach probabilistic deep learning. In Uncertainty in Artificial Intelligence (UAI), 2020b.
Ignacio Pcis, Chao Ma, and José Miguel Hernández-Lobao.. Missing data imputation and acquisition with deep hierarchical models and hamiltonian monte carlo. In Advances in Neural Information Processing Systems (Ne,rIPS), 2022.
Tomás Pevný, Václav Smidl, Martin Trapp, Ondřej Poláček, and Tomáš Oberhuber. hum-product--reae networks: Exploiting symmetrics using invertible transformations. In International Conference Proba- bilistic Grophicol Models (PGM), 2020.
Hoifung Poon and Pedro M. Domingos. Sun-product networks: A new deep architecture. In Uncerlaindy in
Artificial Intelligence (UAI), 2011.
David B. Ramsay, Kevin Kilgour, Dominik Roblek, and Matthew Sharifi. Low-dimensional bottleneck features for on-device contimuous speech recognition. In Conference of the International Speech Communication Association (ISCA), 2019.
Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse fideliiy images with vq-vac-2. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International Conference Machine Learning (ICML), 2015.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning (ICML), 2014.
Salah Rifai, Xavier Muller, Xavier Glorot, Grégoire Mesnil, Yoshua Bengio, and Pascal Vincent. Learning invariant features through local space contraction. arXiv preprint arXiv:1104.4158, 2011.
Donald B. Rubin. Multiple imputation for nonresponse in surveys. In International Biometric Society, 1989.
Rifai Salah, P Vincent, X Muller, X Gloro, and Y Bengio. Contractive aulo-encoders: Explicit invariance during feature extraction. In International Conference on Machine Learning (ICML), 2011.
Daniel Severo, Jingtong Su, Anji Liu, Jeff Johnson, Brian Karrer, Guy Van Den Broeck, Matthew J. Muckley, and Karen Ullrich. Enhancing and evaluating probabilistic circuits for high-resolution lossless image compression. In Dala Compression Conference (DCC), 2025.
Xiaoting Shao, Alejandro Molina, Antonio Vergari, Karl Stelzner, Robert Peharz, Thomas Liebig, and Kristian Kersting. Conditional sum-product networks: Modular probabilistic circuits via gate functions. International Journal of Approzimate Reasoning (IJAR), 2022.
Andy Shih, Dorsa Sadigh, and Stefano Ermon. Hyperspns: Compact and expressive probabilistic circuits. In Advances in Newral Information Processing Systems (NewrIPS), 2021.
Sahil Sidheekh, Kristian Kersting, and Sriraam Natarajan. Probabilistic flow circuits: Towards unified deep models for tractable probabilistic inference. In Uncerlainty in Artificial Intelligence (UAT), 2023.
25

---

Published in Transactions on Machine Learning Research (12/2025)
Sahil Sidheekh, Pranuthi Tenali, Saurabh Mathur, Erik Blasch, Kristian Kersting, and Sriraam Natarajan. Credibillty-awware multimodal fusion using probabilistic circuits. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2025.
Vaidotas Simkus and Michael U Gutmann. Conditional sampling of variational autoencoders via iterated approximate ancestral sampling. Transactions on Machine Learning Research (TMLR), 2023.
Vaidotas Simkus and Michael U Gutmann. Improving variational antoencoder estimation from incomplete data with mixture variational families. Trunsactions on Machine Leurving Research (TMLR), 2024.
Casper Kaae Sønderby, Tapani Raiko, Lars Maalee, Soren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. In Advances in Newral Information Processing Systems (NeurIPS), 2016.
Daniel J. Stekhoven and Peter Bühlmann. Missforest non-parametric missing value imputation for mixed-type data. Bioinformatics, 2011.
JD Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. In International Conference on Learning Representations (ICLR), 2018.
Nya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems (NeurIPS), 27, 2014.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from bert into simple neural net works. arXiv preprint arXiv:1903, 12136, 2019.
Jakub M. Tomezak and Max Welling, Vae with a vampprior. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2018.
Martin Trapp, Robert Peharz, Hong Ge, Franz Pernkopf, and Zoubin Ghahramani. Bayesian learning of sum-product networks. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, In Advonces in Neural Information Processing Systems (NewrIPS), 2017.
Fabrizio Ventola, Karl Stelzner, Alejandro Molina, and Kristian Kersting. Residual sum-product networks. In European Workshop on Probubilistic Graphical Models, 2020.
Fabrizio Ventola, Steven Braun, Zhongjie Yu, Martin Mundt, and Kristian Kersting, Probabilistic circuits that know what they don't know. In Uncertainly in Artificial Intelligence (UAI), 2023.
Antonio Vergari, Robert Peharz, Nicola Di Mauro, Alejandro Molina, Kristian Kersting, and Floriana Esposito. Sum-product autoencoding: Encoding and decoding representations using sum-product networks. Tn Association for the Advancement of Artificial Intelligence (AAAT), 2018.
Antonio Vergari, Nicola Di Mauro, and Floriana Esposito. Visualizing and understanding sum-product networks. Machine Learning, 2019a.
Antonio Vergari, Nicola Di Mauro, and G Van den Broeck. Tractable probabilistic models: Representations, algorithms, lcarning, and applications. Tutorial at UAI, 2019b.
Antonio Vergari, Alejandro Molina, Robert Peharz, Zoubin Ghahramani, Kristian Kersting, and Isabel Valera. Automatic bayesian density analysis. In Association for the Advancement of Artificial Intelligence (AAAI), 2019c.
26

---

Published in Transactions on Machine Learning Research (12/2025)
Antonio Vergari, YooJung Choi, Anji Liu, Stefano Teso, and Guy Van den Broeck. A compositional atlas of tractable circuit operations for probabilistic inference. In Aduances in Neural Information Processing Systems (NewrIPS), 2021.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autocncoders. In International Conference on Machine Learning (ICML), 2008.
Benjic Wang and Guy Van den Broeck. On the relationship between monotone and squared probabilistic circuits. Associalion for the Advancement of Artificial Intelligence (AAAT), 2025.
Z. Wang. Data-free knowledge distillation with soft targeted transfer set synthesis. In Association for the Advancement of Artificial Intelligence (AAAI), 2021a.
Zi Wang. Zero-shot knowledge distillation from a decision-based black-box model. In International Conference on Machine Learning (ICML), 2021b.
Christopher KI Williams, Charlie Nash, and Alfredo Nazábal. Autoencoders and probabilistic inference with missing data: An exact solution for the factor analysis case. arXiv preprint
2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1rXiv:1708.0774. 708.07747, 2017.
Li Xu, Jimmy S Ren, Ce Liu, and Jiaya Jia. Deep convolutional neural network for image deconvolution. Advances in Neural Information Processing Systems (NeurIPS), 27, 2014.
Yang Yang, Gennaro Gala, and Robert Peharz. Bayesian structure scores for probabilistic circuits. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2023.
Junho Yim, Donggyu Joo, Ji-Hoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Hongxu Yin, Pavlo Molchanov, Jose M. Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K. Jha, and Jan Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Conference on Computer Vision and Pottern Recognition (CVPR). 2020.
Jinsung Yoon, James Jordon, and Mihacla. van der Schaar. GAIN: Missing data imputation using generative adversarial nets. In International Conference on Machine Learning (ICML), 2018.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. LSUN: construction of large-scale image dataset using deep learning with humans in the loop. arXie preprint ar arXiv: 150::1506.03365, 2015.
Zhongjie Yu, Devendra Singh Dhami, and Kristian Kersting- Sum-pproduct-attenion networks: Leveraging self-attention in energy-based probabilistic circuits. In The 5th Workshop on Tractable Probabilistic Modeling, 2022.
Honghua Zhang, Meihua Dang, Nanyun Peng, and Guy Van den Broeck. Tractable control for autoregressive language generation. In International Conference on Machine Learning (ICML), 2023.
Kaiyu Zheng, Andrzej Pronobis, and Rajesh P. N. Rao. Learning graph-structured sum- num-product networks for probabilistic semantic maps. In Association for the Advancement of Artificial Intelligence (AAAT), 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:166,. 061666 2018.
27

---

Published in Transactions on Machine Learning Research (12/2025)
A Encoding Algorithm
Algorithm APC Encoding Procedure
Require: Probabilistic circuit C over data and embeddings Z, an input data point x E X
procedure ENCODE(C: circuit, x: data)
PORWARD(ROOT(());
Forward pass; each sum unit we store its inputs Sample from conditioned PC Clx=x
[x.2] SAMPLE(ROOT(())
5: return 2
6:
7: procedure FORWARD(n: unit x: data)
> General forward pass for circuit units
8: == A then
if X,, missing then
10:
return 1.0
Marginalize missing inputs
11: else
12:
return Pa (x,)
b Evaluate PDF of input units
13:
Y - FORWARD(C, x.) for c E in(n)]
D. Collect inputs for sum/pme /omppoduut units Cache inputs for conditional sampling pass
14:
n.Y 4-
15:
if = X then
16:
return [oe Vc
Product unit
17:
= + then return Eeein(n) Yo 0%
18:
>
Sum unit
19:
20: procedure SAMPLE(M: unit)
D Sampling pass for circuit units
21: if == A then
22:
return X, ~ (X,(),)
Sample from PDF represented by input unit.
23: if = X then
24:
return CONCAT( CONCAT([SAMPLLNC) for e in(n)))
Product unit simply sample all inputs
25:
if = + then
26:
0' < CONDITION DETTITOO_WIIIIISSOO,,... n.
Condition weights on forward pass likelihoods > Sample onc-hot encoded index
27:
+ (IMPLE(0'))
28:
return DOT(s, [SAMPLE(C) for c in(n))
Dot product indexes input samples
29:
30: procedure CONDITION WEIGHTS(O:: weights, 7: inputs)
31:
for 1... SIZE(() do
32:
8, - 0% %
Reweight weights based on input likelihoods
33:
$ = Σ, 8
b Compute normalization constant
34:
for E 1......1 ....(I))) do
35:
0, - 0./s
Normalize weight distribution
36:
return
Conditional sampling in PCs, which forms the basis of our encoding procedure (Appendix Algorithm 1), consists of two passes through the circuit. First, forward pass computes the marginal likelihood of the evidence, Pe(x), for given data sample x. During this pass, likelihoods y are cached at the inputs of each sum unit. Subsequently, a sampling pass traverses the circuit from the rool to the input units. At each sum unit, it uses the cached likelihoods 7 from the forward pass to reweight the mixture parameters, effectively applying Bayes' rule form a posterior over its inputs. It then samples a path according these conditioned weights using the differentiable SIMPLE estimator. This process yields sample from the conditional distribution Pc X x), which serves as the embedding
28

---

Published in Transactions on Machine Learning Research (12/2025)
B Experiment and Evaluation Protocol: Additional Details
All experiments and models are implemented in PyTTrrh (Ansel et al., 2024) and PyTorch Lightning (Falcon
& The PyTorch Lightning team, 2019). Our implementation is available open-source software at https: //github. ///gtthu..commmllan...e.e
Datasets We evaluate our models on various image and tabular datasets. For image data, we include MNIST (LeCun et al., 1998), Fashion MNIST (F-MNIST) (Xiao et al., 2017), CIFAR-10 (Krizhevsky, 2009), CelebA (Liu et al., 2015), SVHN (Netzer et al., 2011), Flowers (Gurnani et al., 2017), LSUN (Church) (Yu et al., 2015), and Tiny-ImageNet (Deng et al., 2009). Note that Tiny-ImageNet is occasionally abbreviated as ImageNet in our tables for brevity. For tabular data, we utilize the 20 datasets from the binary density istimation benchmark DEBD (Lowd & Davis, 2010; Haaren & Davis, 2012; Bekker et al., 2015; Larochelle & Murray, 2011).
Models In our empirical evaluation, compare the proposed APCs against several models to demonstrate their effectiveness. These include SPAE-ACT and SPAE-CAT (Vergari et al., 2018) as circuit-based comparison, as well as vanilla variational autoencoder (VAE) and, to also include more missing-data specific methods MIWAE (Mattei & Frellsen, 2019) and missForest (Stekhoven & Bühlmann, 2011). For all autoencoding modcls, we use an embedding dimension d of for MNIST and F-MNIST, 256 for all other image-based datasets, and 4,3,16,32, and 64 for tabular datasets; depending on their number of features to ensure d < X All models are trained under the same conditions with common and well-established practices.
Metrics We report MSE (L) to mcasure roconstruction fidelity and SSIM (t) to assess perceptual quality based on structural and visual similarity. To assess encoding robustness, we analyze MSE and SSIM across varying levels of MCAR-style input data corruption, where uniformly random corruption is incrementally increased from 0% to 95% in steps of 5%. We then compute the average reconstruction error over all corruption levels for each metric. To ensure that results are not specific to MCAR-style missing data, Appendix Table 4 presents evaluations using nine additional MAR-style corruption variants.
Missing Data For vanilla VAFs, missing values are imputed using a constant zero value. We also explored alternative imputation strategies, including mean imputation and learned normalization (similar to LayerNorm). In our experiments on MNIST, mean imputation offered improved reconstruction quality at the cost of a reduction in downstream (ask accuracy. For more challenging datasets like SVHN and CIFAR, the choice of imputation method did not yield statistically significant differences in performance. All other methods handle missing data without the need for imputation.
Model Capacity For all experiments, we ensure that the model encoder and decoder networks are approximately the same size and depth across all models. We use the same neural decoder architecture across APCss and VAE models to ensure that performance differences can be solely attributed to the encoder.
Circuit Input Units For circuit-based encoders, model data distributions on images with Binomial input units, and Bernoulli input units for the DEBD tabular datascts. Embedding distributions are chosen as Gaussian input units in all cases. We want to highlight once more that the APC framework, in principle, allows for the choice of arbitrary data and embedding distributions.
Sum-Product Autoencoding For SPAE, retrieve the correctly sized embedding vector from layer in the circuit graph, which consists of exactly d units (embedding size). In addition, when evaluating reconstructions, we report results only for SPAE (instead of SPAE-ACT and SPAE-CAT) because the reconstruction output is identical for both SPAE variants, regardless of whether activation embeddings (ACT) or categorical embeddings (CAT) are used. This oquivalence is formally established in Proposition of Vergari et al. (2018). However, in all other evaluations, we distinguish between SPAE-ACT and SPAE-CAT, since their embeddings are inherently different when used in e.g., ream tasks.
29

---

Published in Transactions on Machine Learning Research (12/2025)
Autoencoder Training Each model is trained for 10,000 iterations using the AdamW optimizer (Kingma & Ba, 2015; Loshchilov & Hutter, 2017), and convergence was confirmed for all models by the end of this training period. We use the MSE as <EDC for all models. Training is carried out with a batch size of 512, except for CelebA where a batch size of 256 is used due to larger model sizes and VRAM constraints. The initial learning rate set to 0.1 for APCs and 0.005 for AEs and VA The learning rate is reduced by a factor of 10 at 66% and 90% of the training progress. Additionally, to enhance stability and avoid numerical issues or exploding gradients during the training phase, we utilize an exponential learning rate warmup over the first 2% of training iterations. Empirically, this approach mitigates random training difficulties without affecting the final performance across different random seeds. While, for APCs, we could potentially pretrain the PC-encoder with MLE on the marginal p(X) to speed up convergence, we refrain from doing so in our experiments, to keep the comparison between APC and VAE-based models as fair as possible. All models, with the exception of MIWAE, were trained on the complete datasets without missing values. For MIWAE, training was performed using MCAR-style data in which 50% of entrics were missing.
Downstream Task Training We train the logistic regression downstream task models with a batch size of 512 for 5,000 iterations on datasct embeddings from the respective encoders. We employed an initial learning rate of 0.05, which was reduced by factor of 0.1 al 66% and 90% of the training progress, along with an exponential learning rate warmup over the first 2% of iterations. For SPAE-ACT models specifically, we found it necessary to use the AdamW optimizer with learning rate of 0.01 for 100,0000 itcrations (20x longer) and normalize the embeddings to achieve comparable downstream task performance better than a random guessing baseline.
Visualization and Sampling Protocol. When generating samples for the baseline PC models (e-g., in Fig. 9), we follow the protocol described in the recent addendum to Peharz et al. (2020a) on ar in their Appendix Section D. Specifically, we utilize the standard top-down sampling procedure to hierarchically select the active embedding leaves in the circuit. However, to produce the final image, we use the means of the decoder. As noted by Peharz et al. (2020a), this procedure is directly analogous to the common practice in VAEs where decoder noise is turned off during visualization, In both models, under the assumption of pixel-wise independence in the output distributions, sampling adds independent noise that degrades visual quality without reflecting the model's structural generative capabilities.
B.1 Model Architectures
Neural Encoder and Decoder. For tabular data, we employ a simple linear feed-forward style network with four hidden layers and leaky ReLU (a 0.1) activations as encoder and decoder. We use convolutional and residual layers with ReLU activations instead for image data. For further details, we refer the reader to aoc.mo www.moeees..enndderrr decoder) .nn_ (encoder, decoder) , py files in our source code repository.
Einsumnettwrrk Encoder for Tabular Data. We use Einsum (Peharz et al., 2020a) as circuit structure for APCs and SPAEs with Bernoulli input units for the tabular DEBD dataset (Lowd & Davis, 2010; Haaren & Davis, 2012; Bekker et al., 2015; Larochelle & Murray, 2011). For APCs, embedding input units are inserted at the lowest layer and randomly shuffled with the data distribution input units. For all experiments, we keep a depth of 4, with single repetition and 32 input unit distributions, and the sum units per scope each layer.
ConvPe Encoder for Image Data. For image data, we construct a simple "convolutional" circuit for APCs and SPAEs, where we first map each pixel to its density represented by an input unit layer with 256 output channels (number of imput unit distributions per scope). We then successively reduce the height and width by a factor of 2 with product layers, which builds the product over all scopes in disjoint neighboring windows (think of non-overlapping convolution windows with stride being the window size). After each product layer, a sum layer maps all channels of each scope to a vector of sum unit outputs (think of convolution in- and out-channels). We repeat this until we end up with a single dimension where the scopes cover the full image input. This allows us to choose certain depth (how often do we want to repeat the product-sum combination), and the choice of number of sum output units per sum layer. Embedding input units are
30

---

Published in Transactions on Machine Learning Research (12/2025)
inserted randomly at the lowest layer with product units attached to Z data input units, constructing [Z] combinations of pe(2;) pe(2z). More advanced strategies for embedding insertions remain an avenue for future investigation; for example, embedding input units could be decoupled and integrated at varying hierarchical levels within the circuit.
31

---

Published in Transactions on Machine Learning Research (12/2025)
C Reconstruction Performance: Additional Results
This section provides a more detailed exposition of the reconstruction capabilities of APCs and the compared models, supplementing the empirical evaluations presented in the main body. We include additional quantitalive results, such as SSIM for image dalasels, further explore the impact of various MAR corruption patterns beyond the MCAR scenarios discussed earlier, and present extended qualitative results through more comprehensive sets of reconstruction visualizations across all datasets and corruption types. Furthermore, offer an initial exploration into the potential of APC embeddings for out-of-distribbn detection by analyzing embedding likelihoods.
C.1 Reconstructions: Comparing Circuit and Neural Encoder (Quantitative Results)
In Section 5.1, we visually demonstrated the robustness APCs against increasing data corruption by plotting reconstruction error against the percentage of missing values (ef. Fig. 5). To supplement this visual analysis with direct numerical comparison, we additionally provide detailed quantitative results. Appondix Table presents the average MSE across all MCAR corruption levels for the image datasets, numerically aggregaling and substantiating the trends observed in the plots. We further extend this quantitative analysis to tabular data to demonstrate the gencralizability of our findings. Appendix Table 3 provides the corresponding reconstruction performance on the 20 DEBD datasels.
Table 2: APCs achieve lower reconstruction errors than PC, VAE, and missForest baselines across varying levels of MCAR data. Average reconstruction performance on eight image datasets under incrassing percentages of MCAR-style randomly missing pixels (0%-95%).
APC
SPAE
VAE
MIWAE
missForest
MNIST (LeCun al., 1998)
5.07 20.04
28.46
20.43
35.38 21.74
9.16 20.92
24.51 20.03
F-MNIST (Xiao et al., 2017)
4.33 20.02
18.96 20.11
39.80 21.10
12.54
70.74
24.53 40.02
CIFAR (Krizhevsky, 2009)
20.65 20.00
78.32
55.81
32.80
+0.27
57.61
CelebA (Liu et al., 2015)
166.89
1318.73
20.37
1095.90
576.47
121.8
1855.67
SVHN (Netzer et al., 2011)
4.87 +0.00
35.38
11.02
41.01 20.15
17.28
+0.20
44.19
40.20
Flowers (Gurnani et al., 2017)
45.22 20.14
119.31
107.75 11.8
58.03
20.37
52.65
40.14
LSUN (Yu et al., 2015)
63.70
#0.20
322.22
+3.48
254.75
+2.39
123.58
+1.40
115.28
+0.15
ImageNet (Deng al., 2009)
118.70
20.16
365.06
22.20
1286.51
110.4
204.10
12.04
295.40
20.34
32

---

Published in Transactions on Machine Learning Research (12/2025)
Table 3: APCs achieve lowest reconstruction error on 18 out of 20 tabular dataset. APCs outperform the PC, VAE and missForest baselines on the 20 DEBD binary tabular dataset and achieve the best reconstruction performance on of the 20 datasets under increasing percentages of MCAR-style randomly missing data (0%-95%)
APC
SPAE
VAE
MIWAE
misssorest
accidents
6.16 20.04
12.55
+0.86
7,56
+0.28
7.34
+0.05
6.75 +0.01
ad
5.57 =0.03
231.84
116.4
6.04
#0.17
3.50
20.00
5.57 #0.03
baudio
6.50
20.03
15.60
11.35
7.44
20.01
7.81
20.04
7.50 20.01
bbe
34.66
20.15
161.59
110.8
37.45
20.55
47.08
10.60
35.02 10.15
bnetflix
10.08
=0.02
20.53
13.49
+0.77
14.81
10.80
book
3.65
67.10
19.40
3.73
4.40
20.08
3.87
c20ng
19.12
20.04
123.50
+5.85
20.69
+0.05
25.50
+0.25
20.34
cr52
11.94
20.10
134.30
17.81
12.83
20.18
15.10
10.18
12.75
20.08
cwebkb
21.32
20.14
109.07
#4.38
22.94
20.15
28.53
#0.32
22.71
=0007
dna
15.89
20.12
28.76
+0.91
16.46
20.17
20.46
#0.30
15.89
20.04
jester
9.04
20.02
20.98
1.
17.44
20.10
15.04
10.08
10.66
20.01
kdd
0.19
4.12
10.57
0.19
20.00
0.22
10.00
0.20
20.00
kosarek
1.35
13.57
1.42
1.67
1.40
moviereview
48.06
20.15
141.19
+4.27
50.08
+0.22
70.07 +0.13
48.36
msnbe
1.05 20.03
2.23 1:0.9
1.03
20.01
1.17 20.00
0.99 20.00
nltes
1.12 20.02
2.48 10.00
1.60
20.01
1.37 20.01
1.48 20.00
plants
2.52
20.04
11.62
10.54
3.93
20.05
2.93 10.01
4.67 20.01
pums star
5.82
20.10
23.30
40.74
12.47
#0.40
6.53 #0.10
11.74 #0.02
tmovic
7.32
20.08
57.94
13.49
8.72
20.08
9.04
10.04
10.11 $0.02
Iretail
1.22 20.01
6.22
1.23
1.58
1.22
voting
27.71
264.53
17.30
122.24
39.04
10.48
119.03
C.2 Missing at Random (MAR) Style Corruptions
Table APCs ranks first on all MAR-style corruption types across all datasets. Model ranking per corruption type, aggregated across datascts. Each row represents a different MAR-style corruption applied to the input. images. The severity of each corruption is linearly increased, and the average reconstruction MSE/SSIM is measured. The models ranked based on their robustness to these corruptions aggregated across all image datasets. The reported values represent the ranking based on the average MSE/SSIM. See Appendix Figures 15 and 16 for reconstruction visualizations.
MSE
SSIM
Corruption
APC
SPAE
VAE
MIWAE
APC
SPAE
VAE
MIWAE
left-to-right
3
3
right-to-left
2
3
2
top-to-bottom
1
2
3
3
4
bottom-to-top
1
4
2
3
1
3
2
4
border-fo-center
1
3
2
4
2
3
4
center-to-border
1
3
4
2
1
2
3
4
horizontal-band
1
3
4
2
4
2
3
vertical-be -and
3
4
3
4
salt-and-pepper
2
4
33

---

Published in Transactions on Machine Learning Research (12/2025)
Table 5: APCs achieve higher SSIM scores compared to PC, VAE, and missForest baselines across varying levels of MCAR data. Average reconstruction SSIM performance on eight image datasets under increasing percentages of MCAR-style randomly missing pixels (0%-95%).
APC
SPAE
VAE
MIWAE
missForest
MNIST
93.33 20.04
69.87
20.36
50.74
74
20.62
86.81
11.14
62.29 40.46
F-MNIST
87.40
20.05
68.89
+0.25
51.75
+0.38
78.24
+0.43
54.00 +0.04
CIFAR
78.20
t0.07
50.61
#0.88
61.64
#0.14
74.25
=0.10
57.98 +0.18
CelebA
79.52 20.04
49.02
#0.17
50.30
=0.50
61.26
20.08
37.16 $0.10
SVHN
90.29
20.00
54.88
10.0
61.79
10.18
76.23
20.47
56.86 10.17
Flowers
60.40 20.10
44.01
+0.82
50.63
+0.43
56.68
#0.11
56.94
LSUN
77.14 $0.07
50.73
20.52
55.63
20:28
59.80
20.18
27.32 10.02
ImageNet
71.55 20.04
50.62 +0.38
47.76
+0.25
61.58
+0.15
50.98 +0.05
To further investigate model robustness beyond randomly missing data, Appendix Table 4 presents model rankings across different MAR-style corruption types aggregated over all datasets. For each corruption type, repeat the experiments presented in Appendix Table 2 for all image datasets, assign each method a rank from best (1.) to worst (4.) and then report their final rank, sorted by the average rank over all datasets. We only compare autocncoding methods, as they reconstruct the full image from a low-dimensional embedding space. In contrast, Forss. operates directly in the input space, making it an inappropriale comparison for a ranking in this context. APCs consistently achieve the top rank for all but one corruption type and metric, indicating that their superior performance extends beyond handling missing data to various other forms of data corruption. Vanilla VAEs appear to outrank its MSE and SSTM measurements. This could be attributed to the fact that MIWAF specifically trains a specific type of missing data, MCAR in our case. This leads to decrcased performance in reconstruction when the pattern of missing data substantially changes. We can visually confirm this is Appendix Figures 14 and 16, where MIWAE better at reconstructing MCAR-style missing data than VAEs, but fails similarly when confronted with MAR-style corruptions (horizontal, vertical, center, border).
variant MIWAE in both
C.3 Structured Similarity Index (SSIM) Evaluations
While MSE provides a quantitative mcasure of pixel-wise differences, SSIM is also employed to offer a complementary perspective on reconstruction quality. SSIM designed to better align with human visual perception by evaluating luminance, contrast, and structure similarities between the original and reconstructed images. Consistent with the MSE results, the findings presented in Appendix Table 5 (complementary to Appendix Table 2) and Appendix Figure 12 (complementary to Fig. 5) demonstrate that APCs significantly outperform all other compared methods in preserving image fidelity across all datasets, even as the proportion of missing data increases.
C.4 Additional Reconstruction Visualizations
To provide further visual insights and extend the quantitative reconstructions presented in Fig, 6, we additionally show MCAR-style corruptions for 0%, 30%, 50%, 70%, and 90% uniformly random missing data in Appendix Figures 13 and 15 and horizontal bands, vertical band, center and border MAR-style corruptions in Appendix Figures 14 and 16 on all image datasets for all methods. These results further provide evidence that APCs outperform the alternative methods in all investigated corruption scenarios, providing better reconstructions across the bench.
34

---

Published in Transactions on Machine Learning Research (12/2025)
MNIST
F-MNIST
SVHN
CIFAR
100
100
100
100
SSIM
75
50
Flowers
LSUN
CelebA
Tiny ImageNet
100
100
100
100
SSIM
50
25
50
75
25
50
75
25
50
75
25
75
Missing Data (%)
missForest.
VAE
MIWAE
SPAE
APC
Figure 12: APCs can deliver higher reconstruction SSIM scores compared to PC, VAE, and missForest baselines in the asymptotic regime of MCAR-style randomly missing data. Recon- struction SSIM is reported across all image datasets as the degree MCAR corruption increases from 0% to 95%.
35

---

Published in Transactions on Machine Learning Research (12/2025)
MNIST
CIFAR
CelebA
LSUN
0%
30%
50%
70%
90%
0%
30%
50%
70%
90%
0%
30%
50%
70%
90%
0%
30%
50%
70%
90%
VAE
2
2
APC
VAE
APC
2
2
2
3
3
3
2
3
3 3
3
2
VAE
3
3 2
3 3 3
8
3
3 3
APC
3
3
3
4
4
SPAB 102
4
4
2
2
3
MIWAB
4
4
4
4
4
4
4
4
Figure 13: Reconstructions of various MCAR corruptions for MNIST, CIFAR, CelebA, and LSUN.
36

---

Published in Transactions on Machine Learning Research (12/2025)
MNIST
CIFAR
CelebA
LSUN
Full
Hor
Ctr
Bor
Full
Hor
Ctr
Bor
Full
Hor
Ver
Ctr
Bor
Full
Hor
Ctr
Bor
APC
APO MIWAB VAB
2
2
2
3
3
3
101
3 2
0
3
VAB
3 8
3
3
3
3
8
3
APC
3
3
3
3
4
i
4
4
102
4
5
4
4
1441
4
4
MIWAB
4
4//
4
4
2 40 4
Figure 14: Reconstructions of various MAR corruptions for MNIST, CIEAR, CelebA, and LSUN.
37

---

Published in Transactions on Machine Learning Research (12/2025)
F-MNIST
SVHN
Flowers
Tiny ImageNet
0%
30%
50%
70%
90%
0%
30%
50%
70%
90%
0%
30%
50%
70%
90%
0%
30%
50%
70%
90%
99
9
9
APC VAE SPAE
12
09
9
M
0
9
9
9
5
151515
VAE
5151536
1515:151535 15
151555553
3
38
38
APC MIVAB
8
38
38
88
28
TB
1022
MIWAB
10
Figure 15: Reconstructions of various MCAR corruptions for F-MNIST, SVHN, Flowers, and Tmyy-mm
38

---

Published in Transactions on Machine Learning Research (12/2025)
F-MNIST
SVHN
Flowers
Tiny ImageNet
Full
Hor
Ver
Ctr
Bor
Full
Hor
Ver
Ctr
Bor
Full
Hor
Ver
Ctr
Bor
Full
Hor
Ver
Ctr
Bor
9
9
9
VAB
9
H
9
0
0
9
0
9
8
9
5
5
15
15251
150 15 2
5
15 15.641 16
5
15 2
50
1595
5
38
38
38
53
3
101
38
6
33
8
MIWAE
Figure 16: Reconstructions of various MAR corruptions for F-MNIST, SVHN, Flowers, and Tiny- Image Net.
39

---

Published in Transactions on Machine Learning Research (12/2025)
D Autoencoding Probabilistic Circuit Variants
Having established the contribution of cach component through our ablation studies in Section 5.6, we now build upon the core APC framework and additionally explore several alternative configurations motivated by previous work of sum-product autoeneoding (Vergari et al., 2018). These variations allow us to examine the impact of different encoding and decoding strategies within the APC framework. Specifically, we consider the following:
APC pe: We replace the neural network decoder with the circuit encoder C itself. Leveraging the flexibility of PCs, reconstructions & are generated by sampling from the conditional distribution Pc(X Z). Differentiable sampling enables end-to-end training of this configuration, using the same objective as standard APCs to optimize the circuit for both encoding and decoding.
This configuration trains APC using the SPAE scheme for encoding and decoding, employing the circuit. C for both. This means the encoding and decoding processes are based on ACT embeddings (CAT embeddings lead to identical reconstructions as per Vergari et al. (2018)). Note, that KL regularization of embeddings is omitted duc to the different nature of ACT and CAT embeddings compared to the default APC formulation, and LNLL is limited to pC (X), as explicit embedding representations Z are not directly modeled in the circuit this variant.
APC-SPAEpe:
We combine the ACT encoding from SPAE with a neural decoder. Similar to APC-SPAEpes KL. regularization is not applied and LLLL is limited to PX().. In our experiments, we found that due to circuit activation statistics, it was necessary to add a batch normalization layer on top of the circuit activations for the training with a neural decoder to be stable.
APC-SPAE-ACT....
APC-SPAE-CAT.... nn: Analogous to APC-SPAE-ACTI encoding approach from SPAF, coupled with a neural decoder for reconstruction. Training follows the same strategy as APC-SPAE-ACT... uns excluding KL regularization and limiting LNLL to (X).
this variant utilizes the CAT embedding
In Appendix Tables 6 and 7, we report the reconstruction performance of all APC variants on image and tabular datasets with progressively increasing proportions of randomly missing pixels, contrasting them with full APC and SPAE models. We evaluate reconstruction quality using the average MCAR-style corruption reconstruction error for both MSE and SSIM, which capture the impact of varying corruption levels. As shown, APCs maintain the overall best performance across all datasets. However, we make three key observations. (1) Utilizing the encoding PC directly as a decoder (APC pe) yields inferior performance across all tasks, with a median relative decrease of 7.3x on tabular data 2.7x in image data reconstruction, compared to employing a neural network decoder, highlighting, that the additional modeling capacity and flexibility of a neural decoder is crucial. (2) Integrating the SPAE scheme within the APC framework (APC-SPAF..) also led to reduction in performance, further suggesting that PC decoder is less effective in this context. (3) Incorporating SPAE-ACT and SPAE-CAT encoding mechanisms into the APC framework, while retaining a neural network decoder, consistently improved upon the performance of the baseline SPAE method in all evaluated scenarios, achieving median improvements of 2.5x for ACT and 1.7x for CAT embeddings on image data, and median 3.9x and 4.8x respectively on tabular datasets. We use median values here to reduce the impact of extreme outliers, which we obtained on some of the datasets.
We want to highlight that CAT embeddings (Vergari et al., 2018) can be interpreted as special case of APCs. They are the result of running MaxProdMPE on the circuil Č over V= H) where H are sum unit indicator random variables obtained from the latent variable interpretation introduced in Peharz et al. (2017) and described in Section 2. In contrast, the APC framework allows embedding random variables to appear at arbitrary positions in the circuit graph represented by arbitrary distributions. This comparison highlights the flexibility of APCs compared to the prior autoencoding scheme introduced in Vergari et al. (2018).
40

---

Published in Transactions on Machine Learning Research (12/2025)
Table 6: APCs achieve lower reconstruction errors than PC and SPAE-based variants across varying levels of MCAR data. Average reconstruction performance on cight image datasets under increasing percentages of MCAR-sttle randomly missing pixels (0%-95%)
APC- SPAEpe
APC- SPAE-ACT...
APC- SPAE-CAT
APC
APCpe
SPAE
MSE +
MNIST
5.07 20.04
17.17
#0.41
28.46
=0.43
49.82
$3.01
14.71 40.25
20.07
=1.00
F-MNIST
4.33
17.09
20.00
18.96
=0:11
49.46
22.75
12.79 40.82
17.97
=3.08
CIFAR
20.65 =0.09
59.22
+0.30
78.32
=3.09
102.74
+0.93
31.88 +0.55
42.44
=0.20
CelebA
166.89 20.05
644.36
114.2
1318.73
20.37
2969.30
1235.
1173.57
1155.
1220.76
=019
SVHN
4.87 20.00
32.27
10.38
35.38
11.92
78.01 14:37
17.59 1.024
18.74 20.35
Flowers
45.22 20.14
58.65
10.72
119.31
20.23
110.05
1.5 19
72.46
10.00
75.05
=9.91
LSUN
63.70
20.20
185.92
+3.62
322.22
=3.48
489.87
+18.7
194.10
+16.6
185.69 =8.11
ImageNet
118.70
20.10
276.92
43.07
365.06
=2.20
566.89
411.2
369.20
#38.7
267.96
=20.
SSIM
MNIST
93.33
79.87
69.87
45. 2
82.63
64.52
F-MNIST
87.40 20.05
69.44 1.000
68.89 =0.23
39.11 +::18
75.81 +..58
60.89 =4.83
CIFAR
78.20
58.23
50.61 =0.88
44.18
71.41 +0.18
61.31 -0.14
CelebA
79.52 20.04
58.79 +0.33
49.02 -0.17
26.71 +1.17
49.03 +0.88
44.56 -3.77
SVHN
90.29 20.03
62.82 10.24
54.88 20.91
39.22 4. 76
75.60 10.25
67.86 20:36
Flowers
60.40 20.16
59.16 10.15
44.01 20.82
40.80
19.87
50.35 10.18
51.86 21.12
LSUN
77.14 20.07
59.09 +0.38
50.73 =0.02
37.45 40.98
58.72 40.89
55.39 =6.39
ImageNet
71.55
00.4
55.68
+0.20
50.62
=0.30
37.90
40.77
55.43
40.45
53.50
=0.07
Table 7: APCs achieve lower reconstruction errors than PC and SPAE-based variants across varying levels of MCAR data. Average reconstruction performance on the 20 DEBD binary tabular datasets under increasing percentages of MCAR-style randomly missing pixels (0%-95%).
APC- SPAE
APC- SPAE-ACT...
APC- SPAE-CAT...
APC
APCpe
SPAE
accidents
6.16 20.04
29.16 10.32
12.55
20.88
22.16 11.85
8.31 10.31
6.75 20.01
ad
5.57
346.14
275.00
374.18
7.88
5.57
baudio
6.50 20.08
23.31 #1.31
15.60
#1.83
16.81 +0.72
7.27 +0.16
7.50 #0.01
bbc
34.66
238.77 77
161.59
236.07
42.77
35.02
bnetflix
10.08 20.02
24.87 10.84
20.53 0.99
21.89 20.34
10.56 10.15
10.80 20.00
book
3.65 20.02
121.44 42
67.10 29.40
95.96 4.4.8
3.89
3.87 $0.03
c20ng
19.12 20.04
215.58
15.08
123.50 25.85
187.31
20.60 10 96
20.34 10.02
cr52
11.94 20.10
201.12
15.08
134.30
27.81
217.42
110.8
14.33 10 34
12.75 10.08
cwebkb
21.32 10.14
193,05
+3.82
109.07
+4.38
189.50 50 +7.08
24.40 +0.47
22.70 +0.07
dna
15.89 10.12
46.77 +1.88
28.76 #0.01
34.41 +1.21
21.19 +0.45
15.89 $0.04
jester
9.04 20.02
21.39
10.47
20.98
21.08
10.20
10.66 20.01
kdd
20.00
13.20
13.73
4.12 20.37
7.55 1:
0.33 20 03
0.20 20.00
kosarek
1.35 20.04
43.28
42.04
13.57
36.47
2.15
1.40 20.00
moviereview
48.06 10.15
226.20 44.90
141.19
4.227
234.37
110.9
60.11 4: 89
48.36 20.06
msnbe
1.05
2.68
2.23
2.12
1.67
0.99
ultes
1.12 20.02
3,26 +0.27
2.48 +0.00
2.86 +0.30
1.66 +0.14
1.48 +0.00
plants
2.52
16.44
11.62 +0.54
12.04
3.22
4.67
pumsb__sta star
5.82 20.16
42.21 11.M
23.30
20.71
32.76 11.53
8.99 10
11.74 20.02
tmovie
7.32 20.06
116.74 11.89
57.94
25.48
90.92 110.4
10.39 10.34
10.11 10.02
tretail
1.22 10.01
30.08 42.20
6.22
#1.63
34.79 14.17
2.12 +0
1.22 10.00
voting
27.71
10.18
311.78
48.81
264.53
27.30
303.02
418.6
91.81 46.00
119.05 20.11
41

---

Published in Transactions on Machine Learning Research (12/2025)
E
Knowledge Distillation: Algorithm & Additional Results
Algorithm Data-Free Knowledge Distillation from VAE to APC
Require: Pre-trained Teacher VAE, Student APC, Iterations T procedure KNOWLEDOEEDSTIILATOOVVVAEA.A for ł = ., do
3:
2 r (Z)
> Sample embedding from APC prior > Generate synthetic data with VAE decoder Encode synthetic data with VAE encoder Encode synthetic data with APC encoder b Reconstruct with APC decoder Distillation loss Gradient descent.
4:
XVAE = 9VAE
ZVAE = [VAE (VVAE)
ZAPC (Z XVAB
APC 9APC ZAPC)
C=
log po (XVAE ZAPC) + (L(pc(Z (vAA)) llq(z)) log PC (&VAE, ZVAB)
Update APC parameters to minimize L
10:
return Distilled APC
Appendix Algorithm outlines the process of data-free knowledge distillation from VAE to an APC of similar capacity and the same decoder architecture. The procedure iteratively refines the APC to learn from the VAE without the need for original training data. At cach iteration, embedding z is first sampled from the APC prior (Z). This embedding is then passed through the VAE decoder 9VAE(z) generate synthetic data VAE that aligns with the APC prior. The synthetic data is subsequently re-encoded using the VAE encoder ZVAE = fvAE((vAE), to obtain an embedding according to the VAE's approximate posterior. In parallel, the APC encodes the same synthetic data using (Z XvAE), obtaining an auxiliary embedding ZAPC. This embedding is then passed through the APC decoder JAPC(ZAPC) to reconstruct the synthetic VAE sample, producing &PCC The distillation loss L equal to the main training recipe of APCs and consists of three components: the mean squared error (MSE) between the VAE and APC reconstructions, a Kullback-Leibler divergence (KL) term that encourages the APC 's posterior (Z ŚVAE) to align with standard normal prior WO0, 1), and negative log-likelihood (NLL) term maximizing the joint likelihood of VAE sample and embedding (R ZvAE). Since all steps are differentiable, we can end-to-end train this procedure from Line to Line 7 and update the APC parameters via gradient. descent to minimize L.
To demonstrate that knowledge distillation from VAËs to APCs generalizes beyond the two exemplary datasets considered in Section 5.5, we extend our evaluation all image and tabular datasets introduced in Section We report results for both teacher and student reconstructions, and downstream task accuracy under full evidence (Full Evi.) and MCAR-style corruptions in Appendix Table 8.
42

---

Published in Transactions on Machine Learning Research (12/2025)
Table APCs successfully distill the knowledge from a pre-trained VAE in a data-free setting and exceed their teachers robustness against missing data. Input reconstruction mean squared error (L) and downstream task accuracy (1) performance comparison between VAE teacher and distilled APC student modcls, ovaluated with full evidence (Full Evi.) and under varying levels of missing data (MCAR).
Full Evidence: APCs are successfully distilling the knowledge from their teacher.
MCAR Corruptions: APCs surpass their teacher in robustness against corruptions.
Teacher
Student
Teacher +
Student
MSE (4)
MSE (1)
MNIST
3.15 10.02
6.84
10.17
MNIST
35.38 11.74
6.15 20.11
F-MNIST
5.84 +0.02
8,16
+0.10
F-MNIST
39.80 +1.10
5.33 +0.05
CIFAR
21.84 20.07
25.34
$0.30
CIFAR
55.81 10.47
19.61 20.10
CelebA
336.00
7
402.53
#21.1
CelebA
1095.90 114.0
201.70 #10.4
Flowers
115.04 +2.12
57.66
Flowers
107.75
31.43
LSUN
104.91
119.07
LSUN
254.75
68.71
SVHN
15.04 20.48
13.14 $0..00
SVHN
41.01 +0.15
7.68
ImageNNtt
1375.88 1:00
230.16
111.3
ImageNet.
1286.51
110.4
139.72 26.38
accidents
10.38
#0.87
19.03
#1.79
accidents
7.56 +0.28
7.76 #0.37
ad
3.97 20.22
11.75
#0.84
ad baudio
6.04 #0.17
5.68 20.23
baudio
11.47 +0.18
20.35
+2.98
7.44 +0.01
7.94 +0.40
bbe
71.49 10.55
80.06
13.2
bbc
37.45 10.55
39.04 11.20
bnetflix
17.61 #0:24
21.59
bnetflix
13.49
10.46
book
7.47 +0.08
13.09
#3313
book
3.73 +0.02
6.12
c20ng
38.61 +0.03
64.73
+10.8
c20ng
20.69 +0.05
29.86 +1.85
cr52
22.66 20.21
31.68
2:2.7
cr52
12.83 10.18
15.98 2:0.1
cwebkb
43.63 20.36
57.91
2:.21
cwebkb
22.94 10.15
28.13 11.24
dna
32.34 +0.07
40.59
+3.39
dna
16.46 +0.17
19.29 +1.09
jester
16.38 10.23
25.35
1:274
jester
17.44 10.10
10.37 20.43
kdd
0.37 +0.00
0.43
+0.04
kdd
0.19
0.22
kosarek
2.36 10-03
2.96
kosarek
1.42
1.58
moviereview
102.20 #0.80
131.58
#18.0
moviereview
50.08 +0.22
62.55 #3.60
msnbe
1.58 20.09
1.96 #2.00
msnbe
1.03 #0.01
1.07 #0.04
nltes
1.05 10.00
1.32 40.02
nltes
1.60 10001
1.12 20.01
plants
2.65 20.20
3.55 10.37
plants
3.93 1:000
2.55 20.05
pums star
8.14 #0.70
20.08 45.77
pumsb star
12.47 40.40
9.22 #2.80
tmovie
13.15 +0.03
19.06
tmovie
8.72
8.81
tretail
2.20 20-00
2.54
10.06
tretail
1.23
1.22
voting
51.00 20.97
57.25 122.11
voting
122.24 132.8
28.94
Downstream Accuracy (t)
Downstream Accuracy (1)
MNIST
96.12 10.12
90.39
MNIST
27.49
87.77
F-MNIST
83.02 +0.18
81.04 +0.21
F-MNIST
30.30 +2.04
79.41 +0.24
SVHN
56.92 13.78
23.98 10.35
SVHN
36.38 11.99
22.93 20.29
CIFAR
47.14 40.37
38.62 40.42
CIFAR
33.53 40.00
37.66 20.33
Flowers
9.87 10.05
12.84 23.33
Flowers
4.53 20.00
12.56 20.27
ImageNet
6.60 +0.11
2.97 +0.14
ImageNet
5.54 +0.37
2.97
+0.15
43

---

Published in Transactions on Machine Learning Research (12/2025)
F Embedding-based Out-of-Distribution Detection Using APCs
APC
VAE
0.020
Frequency
0.002
0.015
0.010
0.001
0.000
0.000
-100
-50
0
50
100
150
200
250
-1200
-1000
-800
-600
400
-200
Embedding Log-Likels p(e)
MNIST
CIFAR
LSUN
CelebA
F-MNIST
Flowers
SVHN
Tiny ImageNet
Figure 17: While VAEs map in-distribution and out-of-distribution data to the same embedding range, APCs cleanly separate MNIST from other datasets. As expected, we see some minor overlap with F-MNIST due to their similar pixel statistics, whereas all other datasets have close to no overlap with MNIST. APC and VAE models under the lens of out-of-distributian data: Models were trained on MNIST and embedding log-likelihoods p(z) were evaluated for range of out-of-distribution datasets. For APCs we tractably obtain p(z) J p(x,22 2) dx with marginalivation. For VAEs, we need to compute aggregate posterior from training dataset embeddings.
In addition to generative tasks, we can also investigate the empirical distribution of embedding likelihoods produced by the model. Such analysis could allow aut-of-distribut (OOD) detection, enabling decoding processes or downstream applications to proactively reject embeddings that deviate significantly from the expected embedding distribution, as determined by a predefined threshold. As an initial exploration, Appendix Figure 17 compares the embedding log- likelihood distributions APCs and VAEs for in-distribution (MNIST) and out-of-distribntion datasets. We selected MNIST for OOD experiments because its hand-drawn digit images represent highly constrained and artificial distribution that differs from naturally occurring images. This distinctive characteristic, which MNIST shares with F-MNIST, establishes clear distributional boundary that theoretically enables models to assign lower likelihood scores to distr--distriittion samples compared to in-distribution data points. As previously established, APCs allow us to evaluate the marginal embedding log- lieliikl exactly and tractably. At the same time, VAEs require approximating this distribution using a post-hoc aggregate posterior from training data. As evident Appendix Figure 17, APC embeddings separate in-distribution from out-of-distribution data the likelihood space, with minimal overlap except for F-MNIST, which shares similar pixel statistics with MNIST. In contrast, VAE embeddings exhibit substantial overlap between in-distribution and out-of-distribation likelihood scores, making OOD detection almost impossible. This demonstrates that APCs' explicit probabilistic formulation provides strong reconstruction and representation learning capabilities and hints at out-of-distributton detection capabilities without requiring additional mechanisms or models. We leave further in-depth analysis on this topic to future work.
44

---

Published in Transactions on Machine Learning Research (12/2025)
G
Differentiable Sampling with SIMPLE: Details
Algorithm Sampling sum units with SIMPLE
Require: Normalized weights E [0, for sum unit n, where |in(
1: procedure SIMPLE(0)
log < log(0)
Convert probabilities to log-probabilities Draw independent Gumbel noise for each component. P Perturb log log-prbaabilitie with Gumbel noise Index of maximum perturbed probability (discrete sample) > Construct a one-hot vector s where sæ- b Return s for forward pass, but only pass gradients A (detach)
Draw g (0, 1)D
< log +g
5: d' t arg maxx-=1... 2j
6: s mne-hot(d",D)
7: return (s detacch) () + 0
We additionally improve the differentiable sampling approach for POs originally proposed in Lang et al. (2022) by replacing the Gumbel-Softmax gradient estimator with SIMPLE (Ahmed al., 2023) for k-subsct sampling (with A 1) when sampling from sum units outlined in Appendix Algorithm 3. In contrast. to Gumbel-Softmax, SIMPLE directly propagates the gradients through the unperturbed paramters 0. To quantitatively evaluate this modification, we conducted controlled experiment comparing both gradient estimators on a synthetic task: learning a sum unit distribution with 32,64,128, and 256 imputs to match a known categorical ground truth distribution over 64 categories. We optimized sum unit weights by minimizing the MSE between one-hot encoded samples from both distributions, using AdamW with a learning rate of 0.01 for 1,000 iterations with batch size 64, repeating across 10 random seeds. As shown in Fig. 4, SIMPLE demonstrates faster convergence and higher accuracy in approximating the true distribution, as measured by KL. Notably, SIMPLE at iteration 100 achieves comparable performance to Gumbel-Softmax at iteration 1,000, and continues to improve beyond that point. Furthermore, SIMPLE demonstrates lower variance across different random initializations, indicating more consistent convergence behavior regardless of starting conditions. Upon convergence, SIMPLE achieves a KL of 0.0033 + 0.00065, approximately 25x lower than Gumbel-Softmax's 0.0817 = 0.00602 for D = 64. These results validate our choice of SIMPLE as the gradient estimator for different iable sampling APCs, enabling more accurate and stable training.
45

---

Published in Transactions on Machine Learning Research (12/2025)
H Analysis of Training and Inference Time
Table 9: Comparison of model computational time the CelebA dataset, measured in milliseconds (ms) per batch on an NVIDIA A100 GPU. Inference time corresponds to a single reconstruction pass, and training time corresponds to combined forward and backward pass. Both were measured with a batch size of 256. Inference
Training
Model
Reconstruction (ms) Forward + Backward (ms)
APC
64.9
236.2
SPAE-ACT
166.2
73.2
SPAE-CAT
165.8
73.2
VAE
24.3
43.5
MIWAE
1765.3
3933.
missForest
1417.5
To provide complete picture of the practical trade-offs between the evaluated models, we report their training (forward and backward pass) and inference (reconstruction) times for batch of size 256 on the CelebA dataset. The experiments were conducted on single NVIDIA A100 GPU. We analyze the computational performance of APCs against the baselines. For MIWAF, we used 100 imputation iterations during training and inference to keep it feasible memory-wise, while the official implementation suggests 1000 or even up to 10000 iterations for the best performance.
Inference Speed. VAËs leverage convolutional encoders, benefiting from highly optimized CUDA kernels that accelerate inference. In contrast, APCs and SPAEs employ circuit-based encoders whose operations are not as efficiently implemented on current hardware, leading to slower inference. SPAEs are particularly affected both encoding and decoding performed within the circuit. Iterative imputation methods, such MIWAE and missForest, are substantially slower due to their multi-step, sequential procedures for estimating missing values.
During training, APCs are slower than VAEs and SPAEs. This is attributed to the differentiable sampling mechanism, where the top-down pass currently presents a computational bottleneck. We note this is a limitation of the current implementation rather than a fundamental constraint. The top-down pass is characterized by extremely sparse tensors, only one input to each sum node active (c.f. Appendix Algorithm Line 6), which presents clear opportunity for future optimization via custom, sparsity-aware implementation. The missForest model does not have training time per batch as it is a non- non-paraictric method that builds its imputation model on-the-fly.
Training Speed.
46

---

Published in Transactions on Machine Learning Research (12/2025)
Quantitative Evaluation of Sample Quality
To complement the qualitative analysis in Fig. 9, which demonstrates that APCs circumvent the visual artifacts common to samples from standard PCs, we provide a quantitative evaluation of sample quality. We assess generative fidelity using two standard metrics: the Fréchet Inception Distance (FID) (Heusel et al., 2017) and the Kernel Inception Distance (KID) (Sutherland et al., 2018). Scores are computed using the clean-fid (Parmar et al., 2022) Python package by comparing 50,000 generated samples against 50,000 samples from the respective Lest. sets.
As reported in Appendix Table 10, APCs consistently achieve substantially lower (better) FID and KID scores across all datasets compared to standard PC trained with maximum likelihood. This quantitative result corroborates our qualitative finding that integrating a neural decoder within the APC framework significantly enhanccs sample fidelity. While achicving statc-of-the-art generative fidelity is beyond the scope of this work, we report these metrics primarily to quantify the improvement our hybrid architecture offers over traditional, purely circuit- based generation.
Table 10: APCs achieve better FID and KID scores than vanilla PCs. We report FID and KID for samples generated by APCs and standard PC trained via maximum likelihood. Lower scores indicate better sample quality. The results confirm that the hybrid APC architecture significantly improves sample fidelity over purely circuit-based generation.
MNIST
LSUN
CelebA
Flowers
FID
APC
43.98 11.06
268.35
20.73
118.80
20.58
170.49
11.04
PC
137.65
#0.00
273.32
#0.23
353.02
#0.26
263.09
+0.60
KID
APC
0.0471
10.0015
0.2712
20.0012
0.1346
10.0010
0.1725
2:
PC
0.1493
0010
0.3058
20.0003
0.4607
10 0003
0.2785
2: 0009
47


Reviews:
[{"review_id": "dlvPjnhRCz", "reviewer": "Reviewer_jKFS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a representation learning framework, autoencoding probabilistic circuits, that models a joint distribution p_C(X, Z) over data X and embeddings Z with a probabilistic circuit. Embeddings are obtained by tractable conditional inference z ~ p_C(Z | X=x) which naturally supports arbitrary missingness at test time without imputation. They use a neural net as the decoder.\n\nAt APC sum nodes they replace the Gumbel-Softmax with SIMPLE from Ahmed et al. 2023 to improve gradient quality. They introduce a composite loss with 1) reconstruction error via the neural decoder, 2) a cheap KL regularizer on the embeddings that leverages their independence in the sampling-induced tree and 3) a joint data-embedding log-likelihood term log p_C(x,z) to mitigate posterior collapse. APCs are evaluated on 8 image datasets and 20 binary tabular datasets from the Density Estimation Benchmark Datasets (https://github.com/UCLA-StarAI/Density-Estimation-Datasets). They show that APCs obtain lower MSE / higher SSIM reconstructions under MCAR and MAR corruptions than SPAE, VAE, MIWAE, and missForest. They also show that APC embeddings remain linearly separable (as measured by logistic regression probes) under far more noise than other methods. They conduct ablations on MNIST and CIFAR for the key APC components and show that each makes an important contribution to reconstruction quality and linear probe accuracy. They also show that APCs can distill from pretrained VAEs and improve the robustness of reconstruction and linear probe accuracy under data corruptions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Claim 1. APCs enable tractable, exact probabilistic embeddings and native handling of missing inputs.\n- Evidence: an explicit procedure to do so in algorithm 1 and a discussion of marginalizing X_m instead of imputing in Section 3.1.\n- Verdict: Sufficient, especially since the mechanism is standard for PCs. \n\nClaim 2: SIMPLE improves gradient estimation for differentiable PC sampling over Gumbel-Softmax.\n- Evidence: synthetic task with KL to a ground-truth categorical distribution. Figure 4 and Appendix G show faster convergence, lower variance, lower final KL. \n- Verdict: partially sufficient. The synthetic result is compelling but there is no comparison for APC training on real datasets. A SIMPLE vs. Gumbel-Softmax ablation in Table 1 would strengthen this claim.\n\nClaim 3: APCs achieve lower recon error than SPAE, VAE, MIWAE and missForest baselines under MCAR and MAR missingness.\n- Evidence: MCAR curves (Fig 5) and SSIM curves (Fig 12). Aggregate results under various MCAR levels in Tables 2, 3, 5 and MAR styles in Table 4. Visual comparisons of model samples in Figures 6 and 13-16. \n- Verdict: Sufficient. Strong breadth (8 image + 20 tabular datasets), same neural decoder across APC/VAE/MIWAE makes for fair comparisons. \n\nClaim 4: APC embeddings are more robust for downstream tasks under missingness than neural AEs\n- Evidence: logistic regression accuracy curves across datasets in Figure 7. t-SNE projections in Figure 8. \n- Verdict: sufficient. Consistent trends across datasets. \n\nClaim 5: Each component of APCs (differentiable sampling, neural decoder, KL on Z, joint log p(x,z)) contributes meaningfully to performance.\n- Evidence: Table 1 ablates the four components on MNIST/CIFAR under full-evidence and MCAR missingness.\n- Verdict: Sufficient. Clear ablation results.\n\nClaim 6: Neural decoding beats circuit-only decoding.\n- Evidence: APC_pc vs APC with neural decoder in Table 6 (images) and Table 7 (tabular).\n- Verdict: sufficient. The difference is large and consistent.\n\nClaim 7: APCs effectively distill from a VAE without access to training data and are more robust to missingness than the teacher.\n- Evidence: Figure 11 and Table 8 show that an APC student performs comparably to the VAE teacher on full evidence, but substantially better under missingness. t-SNE plots in Fig 11 under high missingness add qualitative evidence.\n- Verdict: sufficient. Nicely demonstrated on multiple datasets with quantitative and qualitative evidence.\n\nClaim 8: Modern high-capacity VAEs fail under heavy missingness.\n- Evidence: Figure 10 shows failure modes for zero-imputation with VAEs and limited gains from MCAR finetuning.\n- Verdict: insufficient. The evidence only covers one model family and one training recipe with zero-imputation. The claim should be softened or supplemented with masked-modeling or missingness-aware baselines.\n\nClaim 9: APCs avoid typical PC sampling artifacts for images.\n- Evidence: Qualitative samples in Figure 9 compare APC vs vanilla PC\n- Verdict: qualitatively sufficient. A FID-like metric (where applicable) would strengthen the evidence.\n\nPreliminary claim:\n\nClaim 10: OOD detection can be done using APC embedding likelihoods.\n- Evidence: Figure 17 (MNIST in-distribution vs several OODs) shows separation for APCs vs overlap for VAEs.\n- Verdict: indeed preliminary (but interesting!). For this to be a central claim it needs quantitative metrics (e.g. AUROC) and more diverse datasets.", "audience_yn": "Yes", "audience_explanation": "Researchers in tractable probabilistic modeling / probabilistic circuits, representation learning and hybrid probabilistic-neural architectures will find APCs directly relevant. The work is also relevant to those studying learning with missing data.\n\nOn the applied side, the ability to natively handle missingness and perform exact probabilistic queries makes this interesting for healthcare/EHR, survey/financial tabular modeling, remote sensing, and multimodal settings where partial observations are common.", "requested_changes": "1. A SIMPLE vs. Gumbel-Softmax ablation in Table 1. Consider measuring gradient variance.\n2. Claim 8 (above) should be softened or supplemented with masked-modeling or missingness-aware baselines.\n3. Add a FID metric where applicable to Fig 9.", "broader_impact_concerns": "N/A", "additional_comments": "Great work!", "extra_scores": {}, "extra_text": {}}, {"review_id": "f1fPKQvIcT", "reviewer": "Reviewer_Cgbz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "### Summary of Contributions\n\nThe paper introduces Autoencoding Probabilistic Circuits (APCs), a framework for representation learning that explicitly integrates embeddings into probabilistic circuits (PCs). Unlike prior PC-based approaches such as SPAE or neural autoencoders such as VAEs, APCs\n\n1. Model explicit probabilistic embeddings by extending PCs to jointly represent data and embedding variables.\n2. Handle missing data natively through tractable inference and marginalization, without requiring imputation.\n3. Enable hybrid training by coupling a PC encoder with a neural decoder, combining tractable probabilistic reasoning with the expressiveness of deep networks.\n4. Introduce differentiable sampling (via SIMPLE) for end-to-end training of the encoder–decoder architecture.\n5. Demonstrate empirical advantages over VAEs, MIWAE, SPAE, and imputation baselines across image and tabular datasets, particularly in robustness to missing data, embedding quality for downstream tasks, and data-free knowledge distillation.\n\n### Key Strengths\n\n* Novel formulation: APCs treat embeddings as first-class random variables within PCs, a principled and tractable approach absent in prior work.\n* Robustness: Strong empirical performance under missing data (MCAR and MAR), where neural autoencoders degrade severely.\n* Hybrid architecture: Combines the interpretability and tractability of PCs with the flexibility of neural decoders.\n* Differentiable training: Incorporation of SIMPLE leads to stable and effective end-to-end optimization.\n* Versatility: Supports both direct training and data-free knowledge distillation from pretrained models.\n* Thorough evaluation: Results span reconstruction, downstream classification, embedding quality, ablations, and distillation across multiple datasets.\n\n### Key Weaknesses\n\n* Heuristic encoder structures: Placement of embedding variables in the PC encoder is somewhat ad hoc; more principled structure learning could improve performance.\n* Scaling concerns: Current experiments focus on moderate-size datasets; unclear if APCs scale to the largest modern benchmarks such as ImageNet-scale.\n* Comparisons limited: Main baselines are VAEs, MIWAE, SPAE, and imputation methods; stronger recent contenders such as diffusion-based autoencoders or masked autoencoders are not included.\n* Complexity of implementation: Training APCs requires specialized PC infrastructure, potentially limiting accessibility and adoption.\n* Generative fidelity gap: While robust, APC reconstructions may still lag behind state-of-the-art neural generative models on complete non-corrupted data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission provides extensive empirical evidence that directly supports its main claims. The authors evaluate APCs on a wide range of datasets (both image and tabular) and compare against multiple relevant baselines, including VAEs, MIWAE, SPAE, and imputation methods. They demonstrate consistent improvements in reconstruction quality, robustness under missing data, and downstream classification accuracy. The paper also includes qualitative results such as reconstructions and embedding visualizations, along with ablation studies that isolate the contribution of each design choice. These experiments collectively confirm the advantages claimed for APCs. While broader comparisons to more recent models would further strengthen the case, the evidence presented is clear, convincing, and sufficient to substantiate the core contributions.", "audience_yn": "Yes", "audience_explanation": "The findings of this paper would be of interest to the TMLR audience because they advance representation learning with probabilistic circuits, a relatively underexplored yet important direction in tractable probabilistic modeling. The proposed APC framework combines the strengths of probabilistic circuits and neural networks, addressing key challenges such as robustness to missing data, explicit probabilistic embeddings, and data-free knowledge distillation. These contributions are relevant to researchers in machine learning who care about tractable inference, hybrid neural–probabilistic methods, and robust representation learning. Given that TMLR’s audience includes both theoretical and applied ML researchers, the novelty, empirical validation, and potential for follow-up work make this study appealing and timely.", "requested_changes": "### Critical Changes (needed for stronger acceptance)\n\n1. Expand baseline comparisons: include evaluations against more recent and competitive representation learning methods such as masked autoencoders (MAEs) and diffusion-based autoencoders. This would better situate APCs in the current landscape and strengthen the empirical claims.\n2. Clarify encoder design choices: the placement of embedding variables within the probabilistic circuit encoder is currently heuristic. More justification or principled reasoning for the chosen structures would make the approach clearer and more convincing.\n\n### Recommended (non-critical, but would strengthen the work)\n\n1. Scaling experiments: provide preliminary results or discussion on scaling APCs to larger and more complex datasets (for example ImageNet-scale). This would address concerns about applicability to modern large-scale benchmarks.\n\n2. Analysis of computational cost: compare training and inference efficiency of APCs with neural autoencoders and other baselines, to give readers a clearer understanding of the trade-offs involved.\n\n3. Discussion of limitations: expand on potential weaknesses of APCs, such as reliance on specialized PC infrastructure and challenges of broader adoption, to present a more balanced perspective.\n\n4. Qualitative insights: while reconstructions and t-SNE plots are helpful, adding case studies or examples from real-world tabular domains (for example medical data with missing values) would highlight practical benefits.", "broader_impact_concerns": "The paper focuses on methodological advances in representation learning using probabilistic circuits and does not raise direct ethical risks such as those associated with generative text or image models that can be misused. However, there are some broader considerations worth noting. Since the method demonstrates strong robustness to missing data, one potential application area is in sensitive domains such as healthcare, finance, or social science datasets where incomplete records are common. In these settings, model misuse or overreliance on automatically inferred embeddings could lead to downstream decisions that impact individuals. It would therefore be important to include a broader impact statement acknowledging both the promise of APCs in improving robustness and the risks of deploying such models without careful domain-specific validation, fairness auditing, and transparency. Additionally, the requirement for specialized probabilistic circuit infrastructure might limit accessibility to only well-resourced research groups, raising equity concerns in who can practically adopt and extend this work.", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "CMg8WbLwDC", "reviewer": "Reviewer_fhbS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors combine probabilistic circuits (PCs) with end-to-end autoencoding to learn data embeddings on the fly. This allows autoencoding PCs (APCs) to perform arbitrary marginalization and conditioning of latent variables from the data-conditional distribution, in addition to learning the joint distribution over embeddings and data. As a side effect, one can marginalize out missing variables in a relatively assumption-free manner. Overall the framework is a very handy extension of the PC model family, which can also be viewed as a (more flexible) extension to hierarchical VAEs. The empirical evaluation stands out as comprehensive and well thought out, with impressive results on corrupted data. I also want to commend the authors for the clarity of writing, notation, and presentation, which makes the paper a pleasure to read.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- The experiments feature different data types (tabular and images) and many different data sets.\n- The experiments confirm APCs' robustness to data corruption and missingness.\n- The experiments confirm APCs' ability to learn informative embeddings.\n- The experiments indicate a clear improvement over baselines.\n- The results are clear and easy to follow. The choice of baselines is well justified. The appendix contains sufficient detail. Code will be made public upon publication.", "audience_yn": "Yes", "audience_explanation": "Probabilistic circuits (PCs) are a popular family of methods for tractable structured inference via deep learning. Algorithmic improvements and generalizations should be relevant to the community. See also summary of contributions.", "requested_changes": "- The references regarding “tractability challenges in deep generative models” (p.2)  are outdated (the most recent one is from 2014) and should hint at many of the recent developments in tractable inference involving, for instance, normalizing flows [NFs; 1], VAE-NF combinations [2], and free-form flows [3]. These can also tractably compute densities, but, compared to PCs, lack the flexibility of PCs for joint, marginal, and conditional queries within the same model. Thus, the authors can try to clarify the benefits of PCs against this backdrop. Also, the flexibility of PCs can be a curse or a blessing, since there are infinite circuit designs that can be chosen with non-trivial consequences for performance. This limitation is only barely mentioned in the discussion and is probably deserving of more words. \n- Missing value imputation can be flexibly tackled with the neural process (NP) model family [4, see also newer work on NPs], which is closely related to VAEs and should be discussed as well. In fact, a NP should have probably been included as a baseline in the image inpainting experiments (not required, but will definitely strengthen the paper).\n- There are typos in the caption of Figure 3. \n- VAEs can notoriously ignore the latent representation given sufficiently flexible decoders [see 5 for a convincing account], which is a direct consequence of the ELBO. Please, correct me if I am wrong, but I believe the same problem is inherent in the specification of the APC objective (Eq.1). Even if that's not the case, I think the topic deserves attention.\n- The Kullback-Leibler divergence is conventionally abbreviated as KL throughout the literature. There is no need to invent new notation.\n\n[1] Kobyzev, I., Prince, S. J., & Brubaker, M. A. (2020). Normalizing flows: An introduction and review of current methods. IEEE transactions on pattern analysis and machine intelligence, 43(11), 3964-3979.\n\n[2] Horvat, C., & Pfister, J. P. (2021). Denoising normalizing flow. Advances in neural information processing systems, 34, 9099-9111.\n\n[3] Draxler, F., Sorrenson, P., Zimmermann, L., Rousselot, A., & Köthe, U. (2024, April). Free-form flows: Make any architecture a normalizing flow. In International Conference on Artificial Intelligence and Statistics (pp. 2197-2205). PMLR.\n\n[4] Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S. M., & Teh, Y. W. (2018). Neural processes. arXiv preprint arXiv:1807.01622.\n\n[5] Zhao, S., Song, J., & Ermon, S. (2019). InfoVAE: Balancing learning and inference in variational autoencoders. In Proceedings of the aaai conference on artificial intelligence (Vol. 33, No. 01, pp. 5885-5892).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)