SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\hat {f} _ {\mathsf {S}} (\boldsymbol {x}) = \int_ {\boldsymbol {t} \in \mathbb {R} ^ {p}} (- \mathrm {i} \boldsymbol {t} ^ {\top} \boldsymbol {x}) \varphi_ {\mathsf {S}} (\boldsymbol {t}) \lambda_ {p} (\mathrm {d} \boldsymbol {t}) = \sum_ {\mathsf {N} \in \operatorname {c h} (\mathsf {S})} w _ {\mathsf {S}, \mathsf {N}} \underbrace {\int_ {\boldsymbol {t} \in \mathbb {R} ^ {p _ {\mathsf {S}}}} (- \mathrm {i} \boldsymbol {t} ^ {\top} \boldsymbol {x}) \varphi_ {\mathsf {N}} (\boldsymbol {t}) \lambda_ {p _ {\mathsf {S}}} (\mathrm {d} \boldsymbol {t})} _ {= \hat {f} _ {\mathsf {N}} (\boldsymbol {x})}. \tag {12}
$$
Therefore, computing the inverse for  $S$  reduces to inversion at its children.
Product Nodes. If  $\mathsf{N}$  is a product node  $\mathsf{P}$ , then:
$$
\hat {f} _ {\mathrm {P}} (\boldsymbol {x}) = \int_ {\boldsymbol {t} \in \mathbb {R} ^ {p _ {\mathrm {P}}}} \left(- \mathrm {i} \boldsymbol {t} ^ {\top} \boldsymbol {x}\right) \varphi_ {\mathrm {P}} (\boldsymbol {t}) \lambda_ {p _ {\mathrm {P}}} (\mathrm {d} \boldsymbol {t}) = \prod_ {\mathrm {N} \in \operatorname {c h} (\mathrm {P})} \underbrace {\int_ {\boldsymbol {s} \in \mathbb {R} ^ {p _ {\mathrm {N}}}} \left(- \mathrm {i} \boldsymbol {s} ^ {\top} \boldsymbol {x} _ {[ \psi (\mathrm {N}) ]}\right) \varphi_ {\mathrm {N}} (\boldsymbol {s}) \lambda_ {p _ {\mathrm {N}}} (\mathrm {d} \boldsymbol {s})} _ {= \hat {f} _ {\mathrm {N}} (\boldsymbol {x} _ {[ \psi (\mathrm {N}) ]})}, \tag {13}
$$
where we used that  $\lambda_{p_{\mathbb{P}}} = \otimes_{\mathbb{N}\in \mathrm{ch}(\mathbb{P})}\lambda_{p_{\mathbb{N}}}$  is a product measure on a product space, applied Fubini's theorem [Fubini, 1907], and used the additivity property of exponential functions. Consequently, computing the inverse for  $\mathbb{P}$  reduces to inversion at its children.
Through the recursive application of Eq. (12) and Eq. (13), we obtain that Eq. (10) reduces to integration at the leaves and, therefore, can be solved either analytically or efficiently through one-dimensional numerical integration.
4.1.2 Efficient computation of marginals
Similar to PCs over distribution functions, CCs allow efficient computation of arbitrary marginals. Given a CC on RVs  $Z = X \cup Y$ , we can obtain the marginal CC of  $X$  as follows. Let  $n = |X|$ ,  $m = |\mathbf{Y}|$  and let the characteristic function of the circuit be given by
$$
\varphi_ {\mathcal {C}} \left(t _ {1}, \dots , t _ {n}, t _ {n + 1}, \dots , t _ {n + m}\right) = \int_ {\boldsymbol {z} \in \mathbb {R} ^ {n + m}} \exp (\mathrm {i} \boldsymbol {t} ^ {\top} \boldsymbol {z}) \mu_ {\mathsf {S}} (\mathrm {d} \boldsymbol {z}), \tag {14}
$$
where  $\mu_{\mathbb{S}}$  denotes the distribution of the root. Then the marginal CC of  $\mathbf{X}$  is given by setting  $t_j = 0$ ,  $n < j \leq n + m$ . The proof of marginal computation is provided in Appendix B.1.
4.1.3 Efficiently computing moments via differentiation
Characteristic circuits also allow efficient computation of moments of distributions. Let  $k \in \mathbb{N}^+$  be such that the partial derivative  $\frac{\partial^{dk}\varphi_c(t)}{\partial t_1^k\cdots\partial t_d^k}$  exists, then the moment  $\mathbb{M}_k$  exists and can be computed efficiently through the derivative at the leaves
$$
\mathbb {M} _ {k} = \mathrm {i} ^ {- d k} \frac {\partial^ {d k} \varphi_ {\mathcal {C}} (\boldsymbol {t})}{\partial t _ {1} ^ {k} \cdots \partial t _ {d} ^ {k}} \Big | _ {t _ {1} = 0, \dots , t _ {d} = 0} = \mathrm {i} ^ {- d k} \sum_ {i = 1} ^ {\tau} \prod_ {(\mathsf {S}, \mathsf {N}) \in \mathrm {E} (\mathcal {T} _ {i})} w _ {\mathsf {S}, \mathsf {N}} \prod_ {\mathsf {L} \in \mathrm {V} (\mathcal {T} _ {i})} \left. \frac {\mathrm {d} ^ {k} \varphi_ {\mathsf {L}} (t _ {\psi (\mathsf {L})})}{\mathrm {d} t _ {\psi (\mathsf {L})} ^ {k}} \right| _ {t _ {\psi (\mathsf {L})} = 0}. \tag {15}
$$
A detailed proof can be found in Appendix B.2.
4.2 Learning Characteristic Circuits from Data
To learn a characteristic circuit from data there are several options. The first option is parameter learning using a random circuit structure. The random structure is initialized by recursively creating mixtures with random weights for sum nodes and randomly splitting the scopes for product nodes. A leaf node is created with randomly initialized parameters when there is only one scope in a node. Maximising the likelihood at the root of a CC requires one to apply the inversion theorem to the CC for each training data. When a leaf node does not have a closed-form density function, numerical integration could be used to obtain the density value given data, which makes the maximum likelihood estimation (MLE) at the root not guaranteed to be tractable.
As discussed in prior works, see e.g. Yu [2004], minimising a distance function to the ECF is most related to moment-matching approaches, but can result in more accurate fitting results. Therefore, minimising the CFD to the ECF can be beneficial if no tractable form of the likelihood exists but evaluating the characteristic function is tractable. In this case, instead of maximising the likelihood from CC, which is not guaranteed to be tractable, we take the ECF from data as an anchor and minimise the CFD between the CC and ECF:
$$
\frac {1}{k} \sum_ {j = 1} ^ {k} \left| \frac {1}{n} \sum_ {i = 1} ^ {n} \exp (\mathrm {i} \boldsymbol {t} _ {j} ^ {\top} \boldsymbol {x} _ {i}) - \varphi_ {\mathcal {C}} (\boldsymbol {t} _ {j}) \right| ^ {2}. \tag {16}
$$
Applying Sedrakyan's inequality [Sedrakyan, 1997] to Eq. (16), parameter learning can be operated batch-wise:
$$
\frac {1}{k} \sum_ {j = 1} ^ {k} \left| \frac {1}{b} \sum_ {l = 1} ^ {b} \frac {1}{n _ {b}} \sum_ {i = 1} ^ {n _ {b}} \exp (\mathrm {i} \boldsymbol {t} _ {j} ^ {\top} \boldsymbol {x} _ {i}) - \varphi_ {\mathcal {C}} (\boldsymbol {t} _ {j}) \right| ^ {2} \leq \frac {1}{k} \sum_ {j = 1} ^ {k} \frac {1}{b} \sum_ {l = 1} ^ {b} \left| \frac {1}{n _ {b}} \sum_ {i = 1} ^ {n _ {b}} \exp (\mathrm {i} \boldsymbol {t} _ {j} ^ {\top} \boldsymbol {x} _ {i}) - \varphi_ {\mathcal {C}} (\boldsymbol {t} _ {j}) \right| ^ {2},
$$
where  $b$  is the number of batches and  $n_b$  the batch size. This way, parameter learning of CC is similar to training a neural network. Furthermore, if two CCs are compatible, as similarly defined for PCs [Vergari et al., 2021], the CFD between the two CCs can be calculated analytically, see Appendix D for more details.
Algorithm 1 CC Structure Learning
Input: training data  $\mathcal{D}$ , RVs  $X$ , min\_k,  $k_{S}$ ,  $k_{P}$
Output:  $\mathcal{C}$
Function buildCF(D, X)
return  $\mathsf{L}\gets$  univariate  $CF\varphi_{X}(t)$
Function buildSumNode(D, X)
Figure 2: Illustration of the recursive structure learning algorithm. Sum nodes are the result of clustering, having weighted children that are product nodes. Product nodes are the result of independence test, enforcing independence assumptions of their children. Leaf nodes are univariate characteristic functions modelling local data.
Function buildProdNode(D, X)
Partition  $\mathcal{D}$  into  $k_{\mathbb{P}}$  independent subsets  $\mathcal{D}_j$  return  $\mathsf{P}\gets \prod_{j = 1}^{\mathsf{kp}}$  buildSumNode  $(\mathcal{D}_j,\mathbf{X}_j)$ $\mathcal{C}\gets$  buildSumNode  $(\mathcal{D},\mathbf{X})$
Table 1: Average test log-likelihoods from CC after parameter learning by minimising the CFD on synthetic data sets. The CC structure is either generated using Random Structure or learned using the Structure Learning algorithm.
Data Set	Random Structure	Random Structure & Parameter Learning	Structure Learning	Structure Learning & Parameter Learning	Structure Learning (random w) & Parameter Learning
MM	-4.93	-3.50	-2.87	-2.86	-3.34
BN	-6.30	-4.12	-3.27	-3.27	-3.93
However, relying on randomly initialized structures (e.g., due to the fixed split of scopes) may also limit the performance of parameter learning of CC. To overcome this, we derive now a structure learning algorithm to learn the structure of the CC. Inspired by Gens and Domingos [2013], this structure learning recursively splits the data slice and creates sum and product nodes of the CC as summarized in Algorithm 1 and depicted in Fig. 2. To create a sum node S, clustering algorithms, e.g., K-means clustering, is employed to split data instances in the slice into  $k_{\mathrm{S}}$  subsets. The weights of the sum node are then determined by the portion of data instances in each subset. To create a product node P, some independence tests—e.g., G-test of independence or random dependency coefficient (RDC) based splitting [Molina et al., 2018]—are used to decide on splitting the random variables in the data slice into  $k_{\mathrm{P}}$  sub-groups. The sum and product nodes are created recursively until any of the following conditions fulfils: (1) There is only one scope in the data slice, and then a leaf node with the corresponding scope is created. (2) The number of data instances in the data slice is smaller than a pre-defined threshold min_k. In the latter case, a naive factorization is applied to the scopes in the data slice to create a product node, and then create leaves for each scope as children for this product node. When creating a leaf node, the leaf parameters are estimated by MLE if the closed-form density function is available. In the case of ECF leaves, the leaf nodes are created from local data following the definition of ECF in Eq. (4). When there is no closed-form density at a leaf, the parameters of an  $\alpha$ -stable distribution are estimated using the algorithm in McCulloch [1986].
Our intention here is to evaluate the performance of characteristic circuit on synthetic data sets and UCI data sets, consisting of heterogeneous data. The likelihoods were computed based on the inversion theorem. For discrete and Gaussian leaves, the likelihoods were computed analytically. For  $\alpha$ -stable leaves, the likelihoods were computed via numerical integration using the Gauss-Hermit quadrature of degree 50.
Can characteristic circuits approximate known distributions well? We begin by describing and evaluating the performance of CC
Figure 3: The Bayesian network used for BN.
on two synthetic data sets. The first data set consisted of data generated from a mixture of multivariate distributions (denoted as MM):  $p(\pmb{x}) = \sum_{i=1}^{K} w_i p(\pmb{x}_1 \mid \mu_i, \sigma_i^2) p(\pmb{x}_2 \mid \pmb{p}_i)$ , where  $p(\pmb{x} \mid \mu, \sigma^2)$  is the univariate normal distribution with mean  $\mu$  and variance  $\sigma^2$ , and  $p(\pmb{x} \mid \pmb{p})$  is the univariate categorical distribution with  $\pmb{p}$  the vector of probability of seeing each element. In our experiments we set  $K = 2$  and  $w_1 = 0.3$ ,  $w_2 = 0.7$ . For each univariate distribution we set  $\mu_1 = 0$ ,  $\sigma_1^2 = 1$ ,  $\mu_2 = 5$ ,  $\sigma_2^2 = 1$ ,  $\pmb{p}_1 = [0.6, 0.4, 0.0]$  and  $\pmb{p}_2 = [0.1, 0.2, 0.7]$ . The second data set consisted of data generated from a Bayesian network with 5 nodes (denoted as BN), to test the modelling power of characteristic circuits with more RVs and more complex correlations among each RV. The details of the BN are depicted in Fig. 3. Here,  $X_1, X_2, X_3$  and  $X_5$  are binary random variables parameterized by  $\pmb{p}_i$ , and  $X_4$  is a continuous random variable conditioned on  $X_3$ . For both data sets MM and BN, 800 instances were generated for training and 800 for testing.
We first employed parameter learning and evaluated the log-likelihoods from the random structure and after parameter learning. A detailed setting of parameter learning is illustrated in Appendix C.1. The increase of log-likelihoods after parameter learning (columns 2 and 3 in Table 1) implies that
Figure 4: Characteristic circuits approximate the true distributions better than the ECF by providing a smaller CFD. We visualize the CFD for CC with parametric leaves (CC-P  $\square$ ), ECF as leaves (CC-E  $\square$ ), normal distribution as leaves (CC-N  $\square$ ) and a single empirical characteristic function (ECF  $\square$ ) learned from synthetic heterogeneous data (Left: MM, Right: BN). Best viewed in color.
minimising the CFD pushes CC to better approximate the true distribution of data. We then learnt CCs using structure learning with  $\min_k k = 100$ , and  $k_{\mathrm{S}} = k_{\mathrm{P}} = 2$ . Various leaf types were evaluated: CC with ECF as leaves (CC-E), CC with normal distribution for continuous RVs and categorical distributions for discrete RVs, i.e., parametric leaves (CC-P), and CC with normal distribution for all leaf nodes (CC-N). The trained CCs were evaluated with the CFD between the CC and the ground truth CF. For data set BN, the ground truth CF was derived via the algorithm for generating arithmetic circuits that compute network polynomials [Darwiche, 2003]. Following Chwialkowski et al. [2015] and Ansari et al. [2020], we illustrate both the CFD with varying scale  $\eta$  in  $\omega(t; \eta)$  and also optimising  $\eta$  for the largest CFD, shown in Fig. 4. We report average CFD values and standard deviations obtained from five runs. It can be seen from Fig. 4 that both CC-E and CC-P have almost equally lower CFD values and also lower maximum CFD values compared to the ECF, which indicates the characteristic circuit structure better encodes the data distribution than the ECF. The smaller standard deviation values from characteristic circuits compared with ECF also imply that characteristic circuits offer a more stable estimate of the characteristic function from data. For data set MM, the maximum CFD of CC-N is 0.0270 when  $\log(\sigma) = 0.6735$ , which is far more than 0.0006 of CC-P, and thus not visualized in Fig. 4 (Left). This also happens to data set BN, as can be seen in Fig. 4 (Right) that CC-N gives higher CFD than CC-P and CC-E, which implies that assuming a discrete RV as Normal distributed is not a suitable choice for CC. In addition, parameter learning on CCs from structure learning and structure learning with randomized parameters (last 2 columns in Table 1) provides higher log-likelihoods than random structures, which implies a well-initialized structure improves parameter learning. To conclude, CC estimates the data distribution better than ECF, which is justified by the smaller CFD from CC-E compared with ECF.
Can characteristic circuits be better density estimators on heterogeneous data? Real-world tabular data usually contain both discrete and real-valued elements, and thus are in most cases heterogeneous. Therefore, we also conducted density estimation experiments on real-world heterogeneous data sets and compared to state-of-the-art probabilistic circuit methods, including Mixed SPNs (MSPN) [Molina et al., 2018], Automatic Bayesian Density Analysis (ABDA) [Vergari et al., 2019] and Bayesian SPNs (BSPN) [Trapp et al., 2019]. We employed the heterogeneous data from the UCI data sets, see Molina et al. [2018] and Vergari et al. [2019] for more details on the data sets. Similar to the setup in Trapp et al. [2019], a random variable was treated as categorical if less than 20 unique states of that RV were in the training set. All the rest RVs were modelled with either normal distributions (CC-P) or  $\alpha$ -stable distributions (CC-A). Again, we first employed parameter learning with a (fixed) random structure using  $\alpha$ -stable distribution for continuous RVs and report the log-likelihoods (Parameter Learning in Table 2). Note that  $\alpha$ -stable distributions can not be represented with closed-form densities, thus maximising the likelihood from it can not be solved exactly and efficiently. As a comparison, structure learning was also employed with  $min\_k = 100$ ,  $k_{\mathrm{S}} = k_{\mathrm{P}} = 2$  for G-test based splitting (CC-P & CC-A), and with  $min\_k = 100$ ,  $k_{\mathrm{S}} = 2$  for RDC based splitting (CC- $\mathrm{A}^{\mathrm{RDC}}$ ). A detailed description of the experimental settings can be found in Appendix C.2. The test log-likelihoods are presented in Table 2. As one can see, parameter learning performs worse than CC-A but still outperforms some of the baselines. CC-P does not win on all the data sets but is competitive with MSPN and ABDA on most of the data sets. CC-A outperforms the baselines on 8 out of 12 data sets, and CC- $\mathrm{A}^{\mathrm{RDC}}$  outperforms all the other methods
Table 2: Average test log-likelihoods from CC and SOTA algorithms on heterogeneous data.
Data Set	Parameter Learning	MSPN	ABDA	BSPN	Structure Learning
CC-P	CC-A	\( CC-A^{RDC} \)
Abalone	3.06	9.73	2.22	3.92	4.27	15.10	17.75
Adult	-14.47	-44.07	-5.91	-4.62	-31.37	-7.76	-1.43
Australian	-5.59	-36.14	-16.44	-21.51	-30.29	-3.26	-2.94
Autism	-27.80	-39.20	-27.93	-0.47	-34.71	-17.52	-15.5
Breast	-20.39	-28.01	-25.48	-25.02	-54.75	-13.41	-12.36
Chess	-13.33	-13.01	-12.30	-11.54	-13.04	-13.04	-12.40
Crx	-6.82	-36.26	-12.82	-19.38	-32.63	-4.72	-3.19
Dermatology	-45.54	-27.71	-24.98	-23.95	-30.34	-24.92	-23.58
Diabetes	-1.49	-31.22	-17.48	-21.21	-23.01	0.63	0.27
German	-19.54	-26.05	-25.83	-26.76	-27.29	-15.24	-15.02
Student	-33.13	-30.18	-28.73	-29.51	-31.59	-27.92	-26.99
Wine	0.32	-0.13	-10.12	-8.62	-6.92	13.34	13.36
# best	0	0	0	2	0	1	9
on 9 out of 12 data sets. This implies that characteristic circuit, especially with structure learning, is a competitive density estimator compared with SOTA PCs. Actually,  $\alpha$ -stable leaf distributions are a more suitable choice for characteristic circuits on heterogeneous tabular data.
We introduced characteristic circuits (CCs), a novel circuit-based characteristic function estimator that leverages an arithmetic circuit with univariate characteristic function leaves for modelling the joint of heterogeneous data distributions. Compared to existing PCs, characteristic circuits model the characteristic function of data distribution in the continuous spectral domain, providing a unified view for discrete and continuous random variables, and can further model distributions that do not have closed-form probability density functions. We showed that both joint and marginal probability densities can be computed exactly and efficiently using characteristic circuits. Finally, we empirically showed that characteristic circuits approximate the data distribution better than ECF, measured by the squared characteristic function distance, and that characteristic circuits can also be competitive density estimators as they win on 9 out of 12 heterogeneous data sets compared to SOTA models.
There are several avenues for future work. For instance, sampling from characteristic functions and, in turn, characteristic circuits is not straightforward. One should explore existing literature discussing sampling from CFs [Devroye, 1986, Ridout, 2009, Walker, 2017], and adapt them to sampling from CCs. The circuit structure of characteristic circuits generated by structure learning has a high impact on the performance of the characteristic circuit, and therefore an inappropriate structure can limit the modelling power of characteristic circuits. Therefore, one should explore parameter learning of characteristic circuits on more advanced circuit structures [Peharz et al., 2020] and, in particular, using normalizing flows, resulting in what could be called characteristic flows.
Broader Impact. Our contributions are broadly aimed at improving probabilistic modelling. CCs could be used to develop more scalable and more accurate probabilistic models, in particular over mixed domains as common in economics, social science, or medicine. Scaling to even bigger mixed models can open up even more potential applications, but also may require careful design to handle overconfidence and failures of CC.
This work was supported by the Federal Ministry of Education and Research (BMBF) Competence Center for AI and Labour ("KompAKI", FKZ 02L19C150). It benefited from the Hessian Ministry of Higher Education, Research, Science and the Arts (HMWK; projects "The Third Wave of AI" and "The Adaptive Mind"), and the Hessian research priority program LOEWE within the project "WhiteBox". MT acknowledges funding from the Academy of Finland (grant number 347279).
Antonio Vergari, Alejandro Molina, Robert Peharz, Zoubin Ghahramani, Kristian Kersting, and Isabel Valera. Automatic bayesian density analysis. In AAAI Conference on Artificial Intelligence, volume 33, pages 5207-5215, 2019.
Antonio Vergari, YooJung Choi, Anji Liu, Stefano Teso, and Guy Van den Broeck. A compositional atlas of tractable circuit operations for probabilistic inference. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 13189-13201, 2021.
Stephen G Walker. A laplace transform inversion method for probability distribution functions. Statistics and Computing, 27(2):439-448, 2017.
Benjie Wang, Matthew R Wicker, and Marta Kwiatkowska. Tractable uncertainty for structure learning. In International Conference on Machine Learning (ICML), pages 23131-23150, 2022.
Xia Wang and Yongmiao Hong. Characteristic function based testing for conditional independence: A nonparametric regression approach. *Econometric Theory*, 34(4):815-849, 2018.
Jun Yu. Empirical characteristic function estimation and its applications. Econometric reviews, 2004.
Zhongjie Yu, Fabrizio Ventola, and Kristian Kersting. Whittle networks: A deep likelihood model for time series. In International Conference on Machine Learning (ICML), pages 12177-12186, 2021a.
Zhongjie Yu, Mingye Zhu, Martin Trapp, Arseny Skryagin, and Kristian Kersting. Leveraging probabilistic circuits for nonparametric multi-output regression. In Uncertainty in Artificial Intelligence (UAI), pages 2008-2018, 2021b.
Matej Zečević, Devendra Singh Dhami, Athresh Karanam, Sriraam Natarajan, and Kristian Kersting. Interventional sum-product networks: Causal inference with tractable probabilistic models. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 15019-15031, 2021.
Honghua Zhang, Brendan Juba, and Guy Van den Broeck. Probabilistic generating circuits. In International Conference on Machine Learning (ICML), pages 12447-12457, 2021.
Han Zhao, Tameem Adel, Geoff Gordon, and Brandon Amos. Collapsed variational inference for sum-product networks. In International Conference on Machine Learning (ICML), pages 1310-1318, 2016.