Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This work extends the approach of Sun et al. to more than two layers of neural networks by connecting weights to sparse approximations of deep Gaussian processes. 

main_review: Strengths & Weaknesses:

In general this is an interesting paper that brings the idea that connects NN and inducing points further. These contributions include
* extending the approach of Sun et al. to more than two layers of neural networks by considering deep GPs.
* proposing to use the arccosine kernel and softplus activation function.
* careful analysis of the interaction between kernels and activation functions, which provides insights on the importance of matching their spectral properties.
* using the connection to provide pre-training of deep GPs (via fitting a neural network and use it to initialize the mean function), and post-training of NNs (via the ELBO of deep GPs) to improve uncertainty estimation.

However, there are still plenty of details missing to fully understand the model used: 

Although the central idea of inter-domain inducing points is well-explained in section 4, this mostly has appeared in Sun et al. and seems not the main contribution of the paper. Surprisingly, the main contribution (a deep extension) is not discussed at all in the main text. it is unclear how the GP layers are stacked to recover the deep NN, e.g., what are the correspondence between $W_1, B_1, W_2, B_2$ and $W_1, V_1, W_2, V_2$, how the deep GP variational distribution is initialized from the deep NN.

The latter seems mysterious to me since the forward pass of deep GP is not equal to propagating through the mean functions of each layer (note that there are nested expectations on the left side of eq. (4)). I wonder how the authors manage to "develop a DGP that has an identical forward pass to a typical ReLU NN" since identical copy layer-by-layer is not going to make it.
 
Another thing here is that I don't think the claim with ReLU is true, the final model is only tested on softplus and there are problems of ReLU discussed in section 4.3)

These details are important since these are the major differences from the approach of Sun et al., where only neural networks with a single hidden layer are considered. 

Related work:

I'm aware of the work of Sun et al. so I can quickly get a understanding of the improvements made in the paper. However, for others who aren't I feel that this relationship is unclear and not even discussed. The general framework of section 4 has large overlap with their approach but none of these is mentioned. I'd suggest to properly cite the ideas developed in previous work and at the same time point out what new contributions are made in this work. The current form reads like the authors invented activated inducing points, while the (14) is the clearly the main idea of the previous work.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper explores the connection between neural networks and Gaussian processes (GP). Specifically, this paper interprets the neural network activations as the inter-domain inducing variables for a GP. Specifically, this paper adopts the proposed approach in deep gaussian processes, proposing initialize a DGP by a pre-trained neural network based on the connection. They investigated the effectiveness of the approach by multiple experiments.
main_review: ## Method
### The Problem and the Approach
Neural networks are state-of-art models in deep learning, while GPs are standard in Bayesian learning. Despite some connections are drawn for infinite-width neural networks and GPs [1, 2], their developments are in general disjoint. How to combine the merits in both models, which is tackled in this paper, is an important problem. Specifically, this paper proposes the Activated Inducing Variables, which interprets the neural network activations as the inter-domain inducing variables for a GP. In consequence, a one-hidden-layer neural network is equivalent to the predictive mean of a respected variational Gaussian process.

### Comparisons to [3]
The activated inducing variables in this paper is similar to the approach in [3], which limits the novelty of the proposed approach. However, the Deep Gaussian Processes and the ArcCosine kernels are not investigated in [3]. Therefore, it is helpful to present the differences and contributions clearer.

### Deep Gaussian Processes
This paper explores the connections between deep neural networks and deep gaussian processes in the experiments. This connection is further validated by the regression and classification experiments. Therefore, it might be helpful to discuss the connection more clearly. 

Specifically, the background Section 3.2 shows the equivalence between propagating the predictive means in a DGP and a DNN with low-rank matrices $\mathbf{W} \mathbf{V}$. The paper proposes to initialize a DGP with a pre-trained DNN.
1) How is the rank chosen ? 
2) How to decompose a single weight matrix into $\mathbf{W} \mathbf{V}$ to match the DGP ? 
Given that there are infinite options for approaching the low-rank decomposition, are there justifications on how to choose it ? 

In general, I think the connections between deep NNs and deep GPs deserve more discussions in the main text.

### Benefit against Standard Inducing Points
ADGP adopts the "activation inducing variables" for posterior inference in DGPs. However, it is not clear why the "activation inducing variables" could provide a better variational approximation than standard inducing points. In fact, the standard inducing points can be understood as the "activation inducing points" as well by letting the activation to the kernel function: 
$$
\sigma(w^T x) := |w| |x| \hat{k}(t) = k(w, t) 
$$
where $t$ is the inner product between normalized $w$ and $t$, $\hat{k}$ is the function applied to $t$ in Eq(6).
Moreover, using the "standard" "activation inducing points" also prevents the spectral mismatch mentioned in Sec 4.3. Therefore, it is beneficial to discuss the potential differences between them.  

On the other hand, it might be the pretraining procedure that leads to the empirical improvements. Could the authors conduct a similar pretraining phase for the standard inducing points ?

## Clarity
This paper is well-written. The method is clearly presented.

## Experiments
This paper demonstrates promising experimental results, where the proposed ADGP outperforms both standard DGPs and NNs. 

1) Although a vanilla NN usually leads to poor uncertainty estimates, the simple temperature scaling (TS) using the validation set usually improves the performance by a large margin [4]. In Figure 7, could the authors also compare with the NN+TS in terms of the test log likelihoods ? 
2) Similar to the temperature scaling in classification, the single observation variance can be treated as the hyperparameter and tuned using the validation set. Could a similar comparison be conducted in the regression experiment as well ? 


## Other Comments
1) Under the proposed formulation, each NN layer is the predictive mean of one GP layer. However, the whole DNN is not necessarily the predictive mean of the whole DGP. Therefore, it is helpful to make the claim clearer. For example, in Line 32-33.  
2) Line 158: why the Softplus function is not $ \log (1 + \exp(t))$ ? 
3) Are the arguments in Line194-198 supported by a theorem or existing literatures? 
4) Line 222: I think it should be "any isotropic stationary kernels".
5) Line 259: do the "single-layer" and "three-layer" mean "hidden layer" or "any layer" ? 

### References
[1] Radford M. Neal. Bayesian Learning for Neural Networks. Springer, 1995.

[2] Jacot, Arthur, Franck Gabriel, and Clément Hongler. "Neural Tangent Kernel: Convergence and Generalization in Neural Networks."

[3] Shengyang Sun, Jiaxin Shi, and Roger B. Grosse. “Neural Networks as Inter-Domain Inducing Points”. In: 3rd Symposium on Advances in Approximate Bayesian Inference. 2021.

[4] Guo, Chuan, et al. "On calibration of modern neural networks." International Conference on Machine Learning. PMLR, 2017.
limitations_and_societal_impact: To further improve the paper, I think the following are potential directions,
1) discuss the differences with [3]
2) present the connections between DNNs and DGPs clearer, both methodologically and experimentally.
3) discuss the comparisons with standard inducing variables
4) demonstrate the advantage of the proposed approach against NNs more clearly by adding the NN+TS baseline

This paper does not lead to negative societal impacts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to establish an equivalence between the forward passes of neural networks and  deep Gaussian process (DGP). Authors further propose models that can either be seen as neural networks with improved uncertainty or deep Gaussian processes with increased prediction accuracy. This is achieved by developing a DGP that has an identical forward pass to a neural net. The training of DGPs can be improved by taking practices from NNs, e.g., a DGP can be initialised in the same way as neural  net without the specialized initialization in DGP literature. Experiments are performed on UCI benchmarks, MNIST, Fashion-MNIST and CIFAR-10. 
main_review: This paper proposes to establish an equivalence between the forward passes of neural networks and  deep Gaussian process (DGP). Authors further propose models that can either be seen as neural networks with improved uncertainty or deep Gaussian processes with increased prediction accuracy. This is achieved by developing a DGP that has an identical forward pass to a neural net. The training of DGPs can be improved by taking practices from NNs, e.g., a DGP can be initialised in the same way as neural  net without the specialized initialization in DGP literature. Experiments are performed on UCI benchmarks, MNIST, Fashion-MNIST and CIFAR-10. 

Overall, this is an interesting paper that further connects the best practices of Deep GP and neural net literature. I especially like the practical models that authors propose to bring about better training  of deep GP and improve the  accuracy. These are important and interesting  contributions to the machine learning community. Below are some questions and suggestions.

1. Though the code is provided in GPFlow, given the huge popularity of pytorch these days, authors should also release a Pyro based code to maximize the impact.

2. In Figure 6, the proposed model performs worse than DGP on a couple datasets. Is there a general reason? Can you further improve the model to outperform DGP?

3. How correlated are the predictions of the proposed model, DGP, and NN on the tested datasets? 

4. Authors should discuss how the perspective from DGP and this paper differs from neural tangent kernel (NTK). I understand that NTK, DGP are different in terms of how they compare to NN. But how does this paper make a difference in this context?

5. Good to summarize more high level intuition why we can improve training without initialization, because  the mean initialization can simply be viewed as skip connections. How does your model connect to and compare to such skip connections?
limitations_and_societal_impact: None
review_title: null

Meta-review: The paper shows an equivalence between forward passes of a neural network and deep Gaussian processes.  It received a positive review, and low-borderline reviews that were appreciative of the method but concerned about novelty over a workshop paper appearing this year.
This paper was discussed extensively with the AC, Senior AC, and an external expert who provided an additional opinion on the paper.  Quoting from the communication with the external expert:

I would accept this paper. The points the authors make at the very top
are significant by themselves:

- addressing the issue of variance starvation
- the model is proper and not degenerate
- training of the model is possible

The AC and Senior AC are in agreement that there is significant additional novelty and development over Sun et al. [45] to warrant publication of this paper.  The authors should take into account the detailed comments in the reviews pointing out the relationship between the works, and to appropriately include the relationship in the camera ready version.  The paper provides a valuable contribution to unifying DGPs and DNNs, and shows significant novel theoretical and practical results.
Decision: Accept (Poster)