title: Probabilistic Circuits: Going Bayesian and Spectral with Densities and Time Series
Creators: name: Yu, Zhongjie
Publisher: name: UNSPECIFIED
Year: 2025

description: When using a machine learning model for a specific task, users typically want to understand the reliability of the output from the model. Therefore, estimating the uncertainty of the output is a crucial task. A key approach to capturing uncertainty is probabilistic modeling. Traditional probabilistic models have evolved significantly over time: with the current increase in the amount of data, the complexity of data types, and diverse inference demands, new probabilistic models are constantly being proposed to address more complex scenarios. However, despite these advances, probabilistic modeling still faces challenges when dealing with data types such as time series or mixed tabular data. They often struggle to efficiently encode time dependencies or fail to provide a unified view for discrete and continuous random variables. In addition, they do not naturally integrate with deep neural networks limiting their application to more challenging tasks.
In this thesis, we investigate modeling challenging data types, including time series and mixed tabular data with probabilistic circuits, which allow for efficient and flexible probabilistic inference and can also be vectorized to work jointly with deep neural networks. First, we model the time series into the leaf nodes of a probabilistic circuit by utilizing Gaussian processes, and use product nodes to sequentially encode both the output dimensions and the covariate space, resulting in multi-output mixture of Gaussian processes (MOMoGPs). This results in the Bayesian case, enabling efficient computation for multi-input, multi-output regression tasks, and we then show its application in a real-world energy production use case. Secondly, to model the joint distribution of the entire time series, we leverage the Whittle assumption and model the time series in the spectral domain with its Fourier coefficients, resulting in Whittle sum-product networks (WSPNs), one of our spectral cases. This method not only preserves the time series dependencies but also enables efficient and flexible inference for, e.g., anomaly detection via density estimation and forecasting via conditional sampling. It is further extended to work jointly with other deep neural networks to provide useful uncertainty estimates in autoencoding and time series prediction. Lastly, we go one step further in the spectral domain by leveraging the structure of probabilistic circuits to model the characteristic function of probability distributions, resulting in characteristic circuits (CCs). By modeling densities in the spectral domain, characteristic circuits provide a unified view for discrete and continuous random variables, and can represent distributions that do not have closed-form probability density functions. We also show that characteristic circuits can be easily adapted and extended for causal inference in hybrid domains. We validate the proposed MOMoGPs, WSPNs, and CCs with both synthetic and real-world data sets. At the end of the thesis, we highlight interesting directions for future research on probabilistic models for challenging data types.