Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ##########################################################################
Summary:
 
The manuscript proposes a deep kernel processes model, where gram matrices are transformed by non-linear kernel functions and assumed to follow Wishart distributions. 
    
##########################################################################
Reasons for score: 
 
I think that method provides a new approach to implement deep kernel processes (it is fully kernel-based compared to Deep GPs which is based on the feature based representation). The method has all required features to make kernel based methods practical: it is still an inductive learning method (enable to make predictions for unseen test points) and provides doubly stochastic inducing-point based variational inference (to scale the model to large data set). However, the manuscripts could be improved in some aspects (please read my comments below).
 
##########################################################################
Pros: 
 
1.	I think that the method is carefully designed and equipped with all required features to make kernel based methods practical.
2.	The method includes important models (DGPs, BNNs, and infinite BNNs) as special cases.  
3.	The method shows superior performance over DGPs and infinite BNNs on the benchmark data sets. 
 
##########################################################################
Cons: 
 
1.	Some parts are confusing or unclear. For example, in equation (19b) the dimensions of the matrices are not matched to each other: K(G_{l-1}) \in R^{N_0 X N_0} but V_l V_l^T in R^{PXP}. In addition, in equation (22) please provide the exact form of Q(\Omega, {G_{l}}^L_{l-2}, F_{L+1}|X_i). Equation 19 (a,b,c) do not really show how the method can reduce the computational complexity based on inducing points.    

2.	The manuscript does not include computational (memory and time) complexities. I also think that the method should be compared to the previous approaches, including deep Gaussian processes with sparse approximation, in terms of these computational complexities. 
  
#########################################################################

Minor comments:
I think that it would be better if the authors included more detailed descriptions what “task-dependent representation” in introduction. 
In the sentence above equation (7), “the DKP as an autoregressive process (Fig. 2)”. I think that Fig. 2 is not about “autoregressive process”?

review_title: good paper, but needs some corrections.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a prior distribution over covariance matrices of kernels which is defined as a sequential graphical model where each variable is Wishart distributed and its scale matrix is a non-linear transformation of its predecesor variable on the graph. The paper begins by considering a DGP with isotropic kernels across the layers and realizes that the Gram matrices are Wishart distributed. Based on this, the paper proposes to bypass the inference of the features and sample the Gram matrices directly from Wishart distributions. This insight, in addition to the layered structure of DGPs, gives rise to the proposed prior distribution. Furthermore, given the restrictions of the Wishart distribution for modelling covariance matrices of arbitrary size [1], as well as the conjugacy properties of the inverse Wishart distribution, the paper uses the inverse Wishart distribution instead. Doubly stochastic variational inference is proposed for approximating the posterior distribution which includes the use of inducing points thanks to the marginalization properties of the inverse Wishart distribution. The experimental contribution consists of a comparison against DGP and Neural Network GP on the UCI, MNIST and CIFAR-10 dataset.

In terms of clarity, the readability of the paper is negatively affected by being too broad with the results presented. It seems that the equivalence between GPs with Wishart processes as priors and DGPs, BNN could be a paper on its own. Thus, no topic is presented with the depth needed. In regards to significance, it is not clear from the paper, how extensive is the family of kernels that can be generated from the prior distribution, given that the proposed model is equivalent to DGPs only in the case of isotropic kernels. In that sense, it is not clear from the paper why this approach would be beneficial over DGPs.

[1] Amar Shah, Andrew Wilson, and Zoubin Ghahramani. Student-t processes as alternatives to Gaus-
sian processes. In Artificial intelligence and statistics, pp. 877–885, 2014.
review_title: see review

final_score: rating: 6: Marginally above acceptance threshold
review_content: ---
##### Summary:

This paper proposes deep kernel processes (DKPs), which can be viewed as a specific kind of deep Gaussian processes where the kernel can be written as a function of the Gram matrix. The features in the intermediate layers are integrated out and the Gram matrix are Wishart distributed. A doubly stochastic variational inference method is proposed to learn DKPs. The idea looks novel to me. My major concern is about the writing.

---
##### Pros:

The paper is well-motivated and the idea is neat. For kernels that can be written as functions of the Gram matrix, DKP seems to be a better formulation than DGP. Due to difficulty in posterior inference, the authors propose to use inverse Wishart prior over Gram matrices. The use of inverse Wishart prior is explained and discussed in Appendix, which is satisfying. The authors also explain how the sparse variational inference method for DKP sidesteps the difficulties of capturing permutation and rotational symmetries in standard variational inference of BNNs or DGPs. The experimental results show the effectiveness of the proposed method. 

---
##### Cons: 

The paper is hard to read and understand. There are a lot of undefined or inconsistent notations, for example:
(1) In the beginning of the Background section, the data dimension is sometimes denoted as uppercase $P$, sometimes denoted as lowercase $p$.

(2) Following Eqn. (2), the inverse $\mathbf{\Sigma}^{-1}$ is not defined. What is $\mathbf{\Sigma}$? It seems that by $\mathbf{\Sigma}^{-1}$ the authors mean the inverse of $\mathbf{S}$, but that is a weird notation.

(3) In section 3, the notations $\mathbf{F}_\ell$ and $N_\ell$ are not explained. Although it can be inferred from the equations (4a-4c) or from the DGP literature, it is still obscure. $\mathbf{K}$ is sometimes used as a function that operates on the Gram matrix and sometimes used as the Kernel matrix, which need.

(4) In Eqn. (8), $\bm{G}_{\ell}-1$ should be $\bm{G}_{\ell-1}$.

Following that, the author mentioned “so $\mathbf{G}_{\ell+1}$ can be written as …” but used subscripts $\ell$ and $\ell-1$ in Eqn. (9).

(5) In Eqn. (11), it is not clear what $\mathbf{K}_{\mathrm{features}}$ means.

I just mentioned a few and there are many more. I suggest the authors refine the writing thoroughly.


review_title: Official Blind Review #2

final_score: rating: 7: Good paper, accept
review_content: The paper proposes the deep kernel processes, which can generalize several existing deep kernel methods, including deep Gaussian processes and Bayesian neural networks. I'm not very familiar with the existing literature on the deep kernel models, so I could not properly assess the novelty in this work. However, the work seems significant to me. This new formulation allows the authors to avoid the feature space, and form the deep structure entirely on the kernel space. Combined with the new inference scheme, the authors showed the proposed method has performance gain over DGP and NNGP on many applications.

My main concern about this work is about its practicality. I am not entirely familiar with the current development in this area, but I believe DGP as formulated in this paper is not very popular and scalability might be one of the main reasons. Could the author address some of the limitation on this side? How does it compare to some of the more scalable methods (such as deep kernel learning where only the last layer is GP)? How is the tradeoff between accuracy and runtime for different number of inducing points?
review_title: The paper proposed the deep kernel processes, which is a generalization of several previous deep kernel methods. Based on it, the authors develop a new variational inference method, and achieved superior performance for some popular datasets.

Meta-review: The paper contributes to the literature of deep kernel learning by proposing a deep probabilistic model based on inverse Wishart distributions. This is an interesting addition to the literature, and the authors have provided some experimental evidence of the superiority of their method compared to state of the art on deep GPs and NNGPs. The major concerns with the paper, though, are related to the clarity of writing and its readability. Additionally, the experimental comparison is thin and lacks exploration for alternative configurations for DGPs and NNGPs. 
Decision: Reject