Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce effective inference strategies for deep Wishart processes (DWP). This class of deep models is of interest because it moves away from the featurized representation implicit in most deep models to one that is fully kernelized. Inference relies on a doubly stochastic inducing-point scheme familiar from variational Deep Gaussian Processes and makes use of a custom generalization of the Bartlett decomposition adapted to the DWP setting in order to define a flexible variational family. In experiments on UCI data the method shows small but consistent improvements over Deep Gaussian Processes.
main_review: This is an interesting and technically sound submission. The clarity of exposition is generally high and the work is contextualized with respect to related lines of work in the literature. While the empirical benefits w.r.t. predictive performance of the proposed model and inference scheme are limited (when compared to Deep Gaussian Processes), what makes this approach particularly interesting is the fully kernelized approach that is taken. One might hope that further generalizations could pay larger predictive dividends. That the resulting scheme differs in a non-trivial way from more featurized representation commonly used in various approaches to deep Bayesian models can be read off, e.g., from the computational complexity, which does not depend on layer width (see Sec. 3.5).

More specific questions/comments:
- I'm confused about some of the notation. Should lowercase p be changed to uppercase P everywhere (e.g. Sec 3.1)? Is V_\ell defined correctly on line 163?
- Is Sec. 3.1 the best use of space? It might make sense to move some of these details to the supplementary material and instead focus on one or more of: i) unpack choices made in Eqn 28a; ii) make the comparison to DGP inference more detailed.
- Can you comment on the gradient variance of the sampling scheme in Sec. 3.4? In DGPs we can compute the distribution q(f) = \int du p(f|u)q(u) in closed form and sample from the marginals. This has the nice benefit that the different f_i are iid. Alternatively we could sample u and then sample each f_i conditioned on u, which results in larger gradient variance. If I understand it correctly, you are effectively in this latter regime and thus presumably experience somewhat large gradient variance. Is that correct? I guess this is more or less a direct consequence of maintaining a kernelized approach? 
- You comment (line 230ff) that “to get optimization to work effectively Salimbeni & Deisenroth (2017) were forced to modify the prior to introduce skip connections." How do you know this is a problem with inference and not a problem with model specification? How can you disentangle the two? Shouldn't sufficiently deep DWPs exhibit some of the same pathologies explored in e.g. "Avoiding pathologies in very deep networks" by Duvenaud et. al.?

## After author response

I thank the authors for their response and I maintain my original score. While some of the concerns raised in the reviews have some merit (in particular w.r.t. the breadth of the experiments), I find some of the concerns raised by the more negative reviews less convincing (e.g. "might not attract broad interest"). While the topic explored in this submission is somewhat dry and technical, I believe that this contribution would make a valuable contribution to the conference. Formulating deep models in function space remains a somewhat underexplored (but potentially important) area, and for this reason (among others) I would argue that the solid technical contributions in this submission could be of interest to the NeurIPS community.
limitations_and_societal_impact: I think it would be useful to better understand how the runtime compares e.g. to DGP inference. For example, if the scheme does in fact suffer from large gradient variance, then more optimization steps may be required, etc.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors introduce an approximate inference procedure for the deep Wishart process. In order to do so, they define a generalised singular Wishart process and use this as a variational posterior. The DWP appears to be a model consisting of layers of Wishart process, which are distributions over PSD matrices. This allows one to define a Wishart process whose parameters are the output of another Wishart process. When the a deep GP is constructed with a kernel that is isotropic, the, DGP is a DWP.  However, the usual approximate posterior used in the DGP fails to capture a certain orthogonal invariance property in the features that is present in the true posterior. The DWP on the other hand, deals with Gram matrices instead of features and this requirement from the approximate posterior is automatically satisfied. 
main_review: The motivation for why the DWP is superior to the DGP seems to be that the variational posterior that it has access to can encode the symmetry constraint in the posterior. Excluding this initial motivation and the introduced methods, there is very little theory giving any kind of theoretical guarantees. I am also not convinced that this motivation is sufficient to justify an entirely new inference scheme, given that the current empirical evaluation is a bit limited. Are there any other reasons why we would prefer to do inference with a DWP versus a DGP?


Questions/Suggestions for improvement:
1. As currently written, the paper seems to assume a certain familiarity with the original deep kernel processes in its motivation, which is not (yet) a mainstream model. Since the paper is specifically about the deep Wishart process, apparently a special case of the deep kernel process, in order to make the message more direct to the reader I would suggest only briefly mentioning deep kernel processes and then focussing on the DWP.
2. The paper is not the easiest to read. I enjoyed section 2. In section 3, I would prefer a self-contained theorem characterising the generalised singular Wishart distribution, which for the reader doing their first pass, could be used as a abstract tool to understand the rest of the paper. Do all the equations need to be numbered? It seems like many are never referred to in the main text.
3. Experiments. You only compared the DWP to a DGP with squared exponential kernel. Arguably you have removed the flexibility that the DGP has in that it can use non-isotropic kernels. For a fair comparison, the DGP should have access to non-isotropic kernels.
4. I believe the authors should be more upfront about the requirement that the kernel is isotropic in the abstract, as this is a relatively strong constraint. The authors state that "We mainly consider isotropic kernels", but as far as I can tell, they exclusively consider isotropic kernels.

limitations_and_societal_impact: N/A.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The manuscript proposes a distribution over positive semi-definite matrices based on the singular Bartlett decomposition for a deep Wishart process (DWP), a special case of deep kernel learning [Aitchison et al., 2020] where gram matrices are used as the latent variables and are transformed through nonlinear kernel functions layer by layer. Based on this distribution, the authors developed a doubly stochastic inducing inference method for the DWP (previously inference in a DWP was not available) by adapting the corresponding variational inference method in [Aitchison et al., 2020]. 


main_review: 
(Originality & quality) I think that the manuscript has certain contributions, one of which is developing the variational inference method for the DWP by proposing a distribution over positive semi-definite matrices based on the singular Bartlett decomposition. Although the propose method is not super new, the method does not seem to be trivial. I did not check every derivation but the method looks technically sound.

(Significance)  However, I am concerned that this work might not attract broad interest. The manuscript proposes an inference method (which is not entirely different or new from previous work) to a specific model formulation. In addition, the experimental results section includes only toy datasets and a DGP as a comparison method. It is understandable that the manuscript attempts comparing the proposed method to the corresponding DGP model based on the connection between both formulations, but at the same time this approach would limit the significance of the proposed work. Readers cannot be convinced about the effectiveness of the proposed method to real-world problems.    

Minor comments
It was at the first read a little bit difficult to understand how the proposed distribution (eq. 27) works since the scale parameter \bSigma does not appear in the distribution. It would be helpful if there was a reminder that the Cholesky \bL is calculated from the scale parameter around eq. 27.  
limitations_and_societal_impact: Have the authors adequately addressed the limitations and potential negative societal impact of their work? Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Deep kernel process (DKP, Aitchison et al., 2020) is a new probabilistic model that provides more flexible modelling power than a shallow kernel method. The deep Wishart process is equivalent to deep Gaussian process, but there is no inference algorithm in the original paper. This paper proposes a new variational inference algorithm for it. It proposes a generalised singular Wishart distribution to model the posterior distribution of the random Gram matrix in DWPs and apply the doubly-stochstic inducing-point scheme. This is a nice contribution to the progress of research along the line of DKPs.
main_review: Significance & originality:

DKPs have a good property of preserving the symmetries in latent features that standard variational inference for DGPs does not. Earlier work developed variational inference method for deep inverse Wishart processes which has a different prior distribution from deep GPs. This work provides a variational inference method for deep Wishard processes so that one can compare the performance of DWPs with DGPs side by side. This is a nice contribution to the research in DKPs. The main novelty of this inference method is introducing the generalised Wishart distribution, while the application of doubly-stochstic inducing-point inference is mostly standard.

To better assess the significance of introducing the inference algorithm for DWPs, I think it would be useful to compare also the result of DIWPs. After all, we would like to find a flexible probabilistic model to have a high log-likelihood (or approximately ELBO) on train and test set.


Quality:

The derivation of the generalised Wishart distribution based on Bartlett decomposition and its application in the variational inference algorithm looks good to me although I haven't checked the detailed derivation in the appendix.

I have a question about the use of generalised Wishart distribution as the approximate posterior distribution. Do we know if the true posterior distribution of the Gram matrix has at most \nu ranks or it could actually be higher than that? If the rank in the posterior could go higher, shall we consider the \nu parameter in Q as another hyper-parameter to tune? Besides, could the authors comment on the capacity of the generalised Wishart distribution? How does the flexibility / capacity compare with the Gaussian approximation in DGPs besides its ability to model symmetries?


Clarity:

The authors provide sufficient background to understand deep Wishart processes and the proposed approximate posterior distribution. The methods section also provides an appropriate level of details in the main text to understand its algorithm.

For the experiment section, I would appreciate more analysis of the inference results beyond simply comparing the ELBO and test log-likelihood. It would be very helpful if the authors could show the difference in the inferred posterior using DWP and DGP. After all, the advantage of using DWPs versus DIWPs is that we can have a more direct comparison with DGPs but Table 1 doesn't give much information beside a better ELBO score.

The authors mentioned the computational complexity is lower than the for DGPs because it doesn't have to sample for each latent feature dimension separately. Does it mean the runtime of the proposed method is indeed lower than that for DGPs? If so, it would also be nice to show the improvement of the runtime in the experiments.
limitations_and_societal_impact: The authors answers "Yes" to question 1.b "Did you describe the limitations of your work?" but I don't see where the limitations are discussed.
review_title: null

Meta-review: The authors propose a variational approximation scheme for Deep Kernel Processes (DKP). Building on previous work, the DKP is an alternative representation of the Deep Gaussian Process (DGP), where the representation is over Gramm matrices rather than over function evaluations. 

Reviewers are in broad agreement that this work is a solid contribution in the development of DKPs, and in functional representation of Deep models in general. One reviewer raised a concern about the lack of theoretical guarantees that the new methodology brings over the DGP representation: in the rebuttal, the authors provide convincing arguments that the improvements in the ELBO due to the representation are directly connected to PAC bounds - I'd like to see a little discussion on this in the camera-ready version. 

Decision: Accept (Poster)