Bayesian Learning with Deep Q-Exponential Process

Authors: Zhi Chang, Chukwudi Paul Obite, Shuang Zhou, Shiwei Lan
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Deep Models, Inhomogeneous Subjects, Regularization, Latent Representation, Model Expressiveness

Abstract:
Motivated by deep neural networks, the deep Gaussian process (DGP) generalizes the standard GP by stacking multiple layers of GPs. Despite the enhanced expressiveness, GP, as an $L_2$ regularization prior, tends to be over-smooth and sub-optimal for inhomogeneous subjects, such as images with edges. Recently, Q-exponential process (Q-EP) has been proposed as an $L_q$ relaxation to GP and demonstrated with more desirable regularization properties through a parameter $q>0$ with $q=2$ corresponding to GP. Sharing the similar tractability of posterior and predictive distributions with GP, Q-EP can also be stacked to improve its modeling flexibility. In this paper, we generalize Q-EP to deep Q-EP to enjoy both proper regularization and improved expressiveness. The generalization is realized by introducing shallow Q-EP as a latent variable model and then building a hierarchy of the shallow Q-EP layers. Sparse approximation by inducing points and scalable variational strategy are applied to facilitate the inference. We demonstrate the numerical advantages of the proposed deep Q-EP model by comparing with multiple state-of-the-art deep probabilistic models.

Decision: reject
Reviewer scores: 3, 3, 5

Reviews and discussion:
<thread>
creation date:
1730227972525

thread_type:
official_review

<thread_object>
creator: reviewer_qpvv

<content>
summary:
This paper generalizes Q-EP (Q-exponential process)  to deep Q-EP so that one can inherit a flexible regularization through the controlled parameter q, which is advantageous in learning latent representations while modeling data inhomogeneity. This generalization is realized by introducing sallow Q-EP as a latent variable model and then building a hierarchy of the sallaow-Q-EP layers. Finally, an inference mechanism based on a sparse approximation via inducing points and a variational strategy is proposed.

strengths:
Starting from Q-exponential process (Q-EP) that has been proposed as an L_q relaxation to Gaussian process (GP) with more desirable regularization properties through its parameter q (with q = 2 corresponding to GP), this paper proposes a contribution to deep probabilistic models via a new model that
(i) generalizes Q-EP to  deep Gaussian process (GP) with some flexibility of regularization
(ii) uses a scalable variational inference based on sparse approximation via inducing points.

weaknesses:
* The proposed theory presented in section 2 is too heavy and no explanations nor intuitions are supporting it. 
* Line 214 states that "Compared with optimization method (Lawrance, 2003), the Bayesian training procedure (Tistsias 2010) is robust to overfitting ..." without any justification or explanation.
* It seems that the proposed inference method induces a matrix inversion with a complexity leading to O(N^3); would it be possible to use a sequential Monte Carlo as proposed by Wang Yali et al. "Sequential Inference for Deep GP, (2016).
* Experiments are not convincing since most have been done on toy data. Furthermore; It would have been convenient to experiment Deep  GP on non-stationary noise (heteroscedasticity)

questions:
(1) What is the complexity behind your variational strategy? Why it is scalable ? 
(2) your sparse approximation by inducing points is not clear, please elaborate more on that, both on the sparsity approximation and the choice of the inducing points.
(3) is your approach appropriate for dealing with a realistic approach as "coupled tomography imaging" as proposed in (Li, 2003).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730476028162

thread_type:
official_review

<thread_object>
creator: reviewer_sdz5

<content>
summary:
The paper extends the Q-Exponential Processes (QEP), a technique that generalizes Gaussian processes to Q-exponential distributions, to deep learning by stacking them into multiple layers in a similar fashion to deep Gaussian processes. The paper takes over the deep GP inference machinery of performing structured variational inference and adapts it to the QEP case. The paper reports results on a set of regression and classification benchmarks that are in commonplace use in GP research.

strengths:
- The presented material is technically correct, well-structured, and understandable.
- The literature survey is comprehensive.
- The paper reports a comprehensive set of experiments.

weaknesses:
- The idea of generalizing Gaussian processes to the Q-exponential distributions is a solid conceptual step as it enhances the expressive power of the model family. I am having hard time to say the same for making a stack of them. This is a rather incremental step that can be taken by straightforward calculations. I am not able to find a concrete scientific problem behind this enterprise that could be valuable for the machine learning research leadership.
- The applied approximate inference approach is a standard and straightforward application of the well-known solution developed originally for deep GPs. I was not able to detect any technical challenges or original ideas that may be interesting.
- The performance improvement is ignorably small for the classification setup and it is yet to be decided based on R2 scores for the regression setup. Based on bare numbers available in Table 1 it is not straightforward to conclude that Deep QEP brings a clear improvement.

questions:
- Can the authors report R2 score for the regression experiments so that one can quantify the absolute performance improvement by moving from deep GPs to deep QEPs?
- How does the STD column in Table 1 help interpret the results? And is it the standard deviation of MAE across repetitions?
- Likewise, what is the take-home message of Figure 3? In what sense does the Deep(er) QEP plots are better than the alternative deep variants? At first sight it looks to me like all of them perform comparably at the center and all fail at the corners. The rest is undecisive details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730666881592

thread_type:
official_review

<thread_object>
creator: reviewer_bne9

<content>
summary:
The authors introduce deep q-exponential process (deep q-EP) as a modeling technique that combines the recently-proposed q-exponential process of [Li et al, 2023] with the deep Gaussian process approach of [Damianou & Lawrence, 2013].  Deep q-EP is a generalization of the deep Gaussian process for q $\neq$ 2 and allows for more modeling flexibility beyond the Gaussian assumption.  The authors propose and derive a variational inference strategy for the model.  They also perform several experiments involving regression and classification on machine learning datasets and compare the performance of their method with other baselines.

strengths:
- The paper is generally clearly written and easy to follow.  The sections are organized well and the contributions are quite clear.  The authors consistently draw connections with the literature and reference the papers on which their result is built, namely [Li et al, 2023] and [Damianou & Lawrence, 2013]. 
- The idea is an interesting one and has the potential to be influential -- GPs are widely used throughout machine learning and any improvement in a widely-used method can be an important contribution. 
- The variational inference method for q-EP appears to be novel.  As far as I can tell, variational inference has not been applied to this model class until this paper.
- The application of multi-layer deep GP to q-EP also appear to be a novel idea that no one has tried before.

weaknesses:
- Though an interesting idea, the paper is pretty straightforward combination of deep GPs and q-EPs.  In this sense, the novelty may be a bit limited.  
- The main critique I have of the paper is that it provides limited insight into when and why we expect deep q-EPs to be better than deep GPs.  There is one sentence in the abstract -- "Despite the enhanced expressiveness, GP, as an L2 regularization prior, tends to be over-smooth and sub-optimal for inhomogeneous subjects, such as images with edges" -- though this idea is not expanded upon later in the paper.  Are the authors advocating that we should always use deep q-EPs over deep GPs?  Or are there specific types of data in which we should expect deep q-EPs to outperform deep GPs?  Little guidance is given about how a practitioner should think about the relative advantage of each approach and in what situations one is advantageous over the other.    
- I appreciate that the authors conduct several experiments to compare the performance of their method against deep GPs, though the results appear somewhat mixed.  In several cases, the performance of the two methods are quite close and it is unclear to me exactly the benefit of one over the other.  Of course, it's not expected that the new method would outperform deep GPs in every instance.  But for the cases in which deep q-EPs is superior, little explanation/analysis is given on why these results make sense.  I think providing more analysis on which types of data/situations are most suited for deep q-EPs would greatly improve the paper.
- Though deep q-EPs can have any value for q, it appears that the authors mainly use q=1 for the experiments.  There is no explanation given on why this choice was made and how a practitioner should select q in practice.  Is there any theoretical reason why we would expect q=1 to work better than q=2 for the datasets that you experimented with?  It would also be interesting to understand how sensitive are the results to the particular choice of q.

questions:
- Does mean field variational inference approach ever lead to suboptimal inference due to absence of correlation in the posterior?  Can you comment on the choice of using VI in comparison to MCMC (as stated in the origin q-EP paper of Li et al)?  Is there any experiment you can perform to understand how well VI approximates posterior in comparison to MCMC for your q-exponential process? 
- In Figure 1, is there any insight into why q-EP identifies different features that are important in comparison to GP-LVM?
- In Figure 2, it is stated that “Among these models, deep Q-EP yields the most accurate prediction and the tightest uncertainty bound.” But the caption seems to indicate that deep GP performs marginally better.  It is not clear to me that deep Q-EP is more accurate for this example.  Also, since the numbers are very close, do you have the standard deviation of different runs to understand how stable these numbers are depending on initialization?  
- Figure 3 is interesting, though why is it the case that deep q-EP seems to have better decision boundary?  is there anything about the model that makes this result expected?  Would a deeper GP help performance?  Also, could this result be sensitive to the choice of kernel?  If you adjusted the smoothness parameter, do you get different results? 
- In section 5.4, the result between DKL-GP and DKL-QEP appears quite close, at least for MNIST — do you have standard deviations for these runs?  For CIFAR-10, it's a bit strange that all the methods don't perform much better than 70% accuracy.  Though state-of-the-art performance is certainly not expected, this seems too far below what can be achieved using other methods that the experiment may no longer be realistic.  Therefore, I'm not sure this dataset is a practical use case of the method.  

Minor points:
-  Eq. (1) is confusing to read.  r has dependence on u, \mu, and C; this is not obvious in the definition of p.
- In Eq. (2), does it always make sense to assume noise has the same q as the prior?  Is a more realistic assumption that noise is Gaussian with q=2?  How would having a different q in the prior vs. likelihood change the posterior though?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732593255664

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global response

comment:
We thank three anonymous reviewers for their careful reading and constructive comments. Motivated by deep GP improving standard GP for the modeling expressiveness, we generalize Q-EP to deep Q-EP to particularly **enhance its capability of modeling inhomogeneous data with abrupt changes or sharp contrast**. Aiming for more practical applications, we focus on variational Bayes instead of Markov chain Monte Carlo (MCMC). A significant **theoretic barricade** in the setting of Q-EP is that many expectations in the evidence lower bound (ELBO) are no longer analytically tractable with the general power (rather than 2) in the exponent of its density. We solve this challenge by taking advantage of Jensen's inequality.

We value all the critiques by the reviewers to improve the quality of the paper. Therefore, we have made the following major changes:
1. Clarify when deep Q-EP ($q=1$) is more appropriate than deep GP ($q=2$) and why $q=1$ is preferred to $q=2$ for **inhomogeneous data**. This is analogous to $L_1$ regularization vs $L_2$ regularization in function spaces, in the regime of deep probabilistic models.
2. Clarify the theoretic difficulty of developing variational Bayes for deep Q-EP compared with deep GP. This development, rather than straightforward or incremental, not only enables efficient inference for highly flexible deep Q-EP, but also paves ways for developing Stochastic variational inference for more general convex deep probabilistic models.
3. Improve numerical results by adding more experiments, including a simulated classification problem with annulus decision boundaries. This is to contrast when deep GP works fine and when deep Q-EP should be preferred.

Below we provide point-to-point responses to all the reviewers. We have also uploaded a [rebuttal revision](/pdf/4199a4849c7f33c9a002444e452c76534322f7de.pdf), with relevant changes color-coded to address different questions by raised by corresponding reviewers. We hope our efforts have substantially improved the clarity of our paper and could win better appreciation. Thank you!

The rebuttal revision can be downloaded by clicking the original submission link or [$\underline{here}$](/pdf/4199a4849c7f33c9a002444e452c76534322f7de.pdf).

Authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a "deep" variant of the recently proposed q-exponential processes (Deep q-EP), a model which composes two random functions that are q-EP processes a priori. This method is analogous to composing two Gaussian process-distributed functions to form a Deep GP. Overall, in its current state, this paper does not demonstrate why a Deep q-EP should be preferable to a Deep GP. While the authors performed several experiments to compare them against Deep GPs, the results did not show significant benefits in favour of Deep q-EP. Though the authors' revision explained why Deep q-EP may be favoured, this revision did not undergo a complete peer-review process. In addition, the authors limit their experiments to one setting of q-EP (i.e. q=1). As it stands, the paper submitted to ICLR does not make a strong enough case for quality and significance to merit publication at ICLR. I suggest the authors focus on a more thorough evaluation and understanding of Deep q-EP to demonstrate why they are a model of interest more convincingly.