Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: 
In this paper, an extension to deep kernel learning is proposed. A linear kernel is used as the base kernel, which enables exact optimization of the kernel hyperparameters. There is a universal approximator theorem stating that the deep neural network with linear kernel could approximate any kernel function, which is quite obvious from the perspective of random Fourier features as well. Also, multiple neural networks are used to produce features and the features are concatenated.  I am not sure why the word ensemble is used, but it is really a concatenation of features instead of ensembling predictions. Besides the exact inference, standard variational inference is also proposed for the linear base kernel. Experiments are conducted on synthetic data and UCI datasets, with comparison to DKL with linear kernel and deep ensembles. 
In general, I could not find very interesting contributions from the paper. The framework follows closely from DKL with a linear kernel. But I have a question about the proof of the universal approximator theorem, to approximate the different eigenfunctions, I would assume the hidden layer of the neural network to be at least of O(B) where B is the number of eigenfunctions to be approximated. So potentially, the neural network needs to be very wide which might not be practical. Also, I am confused why the “concatenation” of features is referred to as “ensemble” in the paper. It is super confusing to me unless I am not understanding how the features are used jointly. Also, the authors mention that “a learner with M output is simply a concatenation of M single-output learners, suggesting that multi-output learner may help to further reduce the number of H of required learners.” With the same number of nodes in the hidden layers, I don’t think a multi-output learner is equivalent to M different single-output learners, therefore the argument made here might not be valid.
In terms of experiments, it is weird that for DKL linear kernel is used instead of the original spectral mixture kernel with random Fourier features. It makes the most sense to compare the linear kernel with something like RBF or spectral mixture kernel. Also, UCI regression tasks seem to be rather easy and do not necessarily need a deep neural network to do feature engineering. It would be necessary to conduct experiments on more complicated tasks such as images to validate the effectiveness of the proposed linear kernel approach. 

------------After author's response----------------

My major concern is about the connection between the universal approximation theorem and the proposed architecture. In the paper, the authors mentioned that " the following universal approximation theorem for DKL implies that this effect can be compensated by adding parallel learners". However, the fact that a multi-output learner is not equivalent to M different single-output learners makes it hard to justify the proposed architecture theoretically from the universal approximation theorem. 
I think this is something that is crucial to be justified, otherwise, the theory does not really match with the proposed method.
review_title: DKL with linear kernel

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a variant of the Deep Kernel Learning model (DKL) [1] where multiple independent networks are trained for the features instead of a single network. In addition, the paper proposes to use a linear kernel as a base kernel which allows for universal approximation of any arbitrary kernel, as well as allowing for exact inference of the kernel hyperparameters. The use of stochastic variational inference is proposed for inferring the neural networks weights. The model is compared against Deep Ensemble (DE) and DKL on a synthetic dataset and the UCI dataset.

The paper is well written, easy to follow and is technically correct. The main weaknesses of the paper is that by using a linear kernel as the base kernel, it is not clear how the proposed model is different from a Bayesian Neural Network with an extra linear layer. Furthermore, the experiments do not necessarily showcase the strengths of the proposed model, namely, the universal approximation property and the regularization that an ensemble provides. I think the paper could increase its impact by effectively implementing distributed training as well as improving the experiments.

[1] Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning.
In Artificial Intelligence and Statistics, pp. 370–378, 2016b.
review_title: see review

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The authors introduce a deep ensemble kernel learning approach as a linear-based learning combination, from a deep learning scheme, to approximate kernel functions under a Bayesian (GP) framework. Namely, a universal kernel approximation strategy is proposed from eigen-based decomposition and deep learning-based function composition. Then, a variational inference strategy is used to solve the optimization from kernel-based mappings. Two regularization strategies are studied: optimal prior covariance and isotropic covariance. Results demonstrate the benefits of the proposal.

Two comments:
Could you include the GP and sparse GP results to compare the predictions? Your approach can be scalable from the stochastic variational variance; however, the studied databases are small in terms of the number of samples so that the well-known GP algorithms could be compared.
In most of the cases, isotropic and optimal covariance-based methods seem to achieve the same results; why? 

review_title: An interesting contribution well-founded from both mathematical background and computational experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a special case of deep kernel learning (DKL) using a linear base kernel, where the inputs to the kernel are the outputs of neural nets with identical architectures (which could be thought of as learners that get ensembled); the resulting approach is called deep ensemble kernel learning (DEKL). The paper provides a universal kernel approximating result for DEKL (Theorem 1), explains how to train DEKL efficiently for large datasets using variational inference, and demonstrates that DEKL can outperform deep ensembles and DKL on various datasets.

Overall, this paper is well-written and easy to follow, and appears to combine some standard ensembling ideas with DKL. I think the paper would benefit from more extensive literature review and a more thorough experimental results section. See the "weaknesses" listed below for details. In terms of significance, I find this paper to constitute more of an incremental advance.

Strengths:
- very well-presented, intuitive method
- theory and experiments are easy to follow

Weaknesses:
- Theorem 1 seems straightforward and rather unsurprising given the Universal Approximation Theorem; moreover, approximating the kernel to be of the form phi(x)^T phi(x) (so taking V to be the identity matrix) is already known to be able to approximate a variety of kernels with arbitrary accuracy when phi is chosen with appropriate randomness (see Rahimi and Recht’s "Random Features for Large-Scale Kernel Machines" (2007) and also the follow-up paper "Weighted Sums of Random Kitchen Sinks: Replacing minimization with randomization in learning" (2008)) -- perhaps it would be helpful to discuss a bit how your approach relates to these existing methods (there seem to be some similar ideas being used); naturally, a related question I have is how much performance degrades if V is just taken to be identity rather than using the choice given in equation (7)
- for the experiments, providing some basic characteristics of the UCI datasets used would be helpful (e.g., dataset size and dimensionality), to get a sense of how high-dimensional the datasets are
- I didn't really get intuition for when/why DEKL should outperform DKL (especially in terms of dataset characteristics), and am left wondering how much simpler methods would fare on the UCI datasets (e.g., linear regression, random forest regression, or other classical methods); for example, is DKL optimizing over too rich a model class or something?

Typo right after equation (2): $\varphi$ is missing a close parenthesis
review_title: interesting, well-presented method; could improve literature review and make experimental section more thorough

Meta-review: The paper presents a DKL variant with a linear kernel. Representations from several networks is combined through concatenation, making it not quite an ensemble. It's shown that the model is a universal kernel approximator. Experiments are conducted on a large number of UCI datasets.

Following the discussions, the paper still has the following shortcomings:
- some lack of clarity in the presentation (for instance, explaining the equivalence between a multi-output learner and M different single-output learners)
- lack of experiments on data where deep learning is typically used (images); the UCI datasets have structured data and other ensembles like XGBoost may outperform the baselines presented in this paper
- difference in performance between DKL and DEKL, especially since DKL benefits from a larger model space, theoretically. maybe DEKL has better sample complexity, but does this advantage hold in the case of the large datasets that deep learning is used for?
Decision: Reject