Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work contributes a few interesting ideas for deep sufficient dimension reduction (SDR). Authors developed a new stochastic neural network and proved that the output of last hidden layer satisfies the SDR condition. Authors also discussed carefully the optimization. 
strengths_and_weaknesses: Using stochastic neural network for SDR is new to me, which is also different from existing deep SDR methods. 

My concerns mainly come from the evaluation section. Authors discussed a few issues of exisiting deep SDR methods ([2,20]). However, there is no comparision to any deep methods. Moreover, it is also a little unclear to me how [2] or [20] suffers from these limitations, and how the new method avoids such issue. Could you also empirically valiate this motivation? Additionally, it would be better if authors can compare with kernel dimension reduction. 
questions: please see above about weakness
limitations: It seems authors did not discuss the potential negative social impact.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose a nonlinear sufficient dimension reduction method to extract key information in high-dimensional data. They proposed a stochastic neural network and develop an adaptive stochastic gradient Markov chain Monte Carlo algorithm with a convergence theorem established. Extensive empirical experiments on benchmark and real datasets are run to showcase the good prediction performance of the algorithm. 

strengths_and_weaknesses: Strengths:
1. The authors develop an adaptive stochastic gradient MCMC algorithm for training the StoNet and provide a rigorous study for its convergence under mild conditions. 
2. The authors have provided extensive empirical results to demonstrate the advantage of their method.
Weaknesses:
1. The model for this model seems not novel and I do not find a clear difference between the author's model and the model in the following paper:
[1] Sufficient Dimension Reduction with Deep Neural Networks for Phenotype Prediction by Siqi Liang, Wei-Heng Huang, Faming Liang. Proceedings of the 3rd International Conference on Statistics: Theory and Applications (ICSTA'21)
2. Computational complexity or empirical computational cost analysis and results are not provided so I do not have a clue if the author's algorithm will be much slower than state-of-the-art algorithms in terms of inference.
3. To me, the closest model to the author's work is [1] but no comparison is provided with [1]. 

questions: 1. How is the author's model different from [1] Sufficient Dimension Reduction with Deep Neural Networks for Phenotype Prediction by Siqi Liang, Wei-Heng Huang, Faming Liang. Proceedings of the 3rd International Conference on Statistics: Theory and Applications (ICSTA'21)
I am aware that in [1], a divide-and-conquer DNN is used, except this, is there any difference in the model part? To me, the model in the current paper seems not novel compared with the model in [1]. 

2.Any empirical results to show the computational cost for the author's algorithm and other competing algorithms? It seems to me that the stochastic gradience HMC could be slow for large-scale dnn task even if a mini-batch is used. 

3. How is the author's approach compared with [1] as they are so similar? Is there an apparent advantage for the author's method compared with [1]. Why [1] is not cited or compared as a competing algorithm?
limitations: The authors do not discuss the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper addresses nonlinear sufficient dimension reduction problems using a novel neural network architecture. The key idea is a layer-wise Markovian assumption which ensures the latent representation is identifiable---an issue associated with several related approaches. The authors further provided a stochastic gradient MCMC learning algorithm as well as theoretical analysis of convergence.

strengths_and_weaknesses: [Strengths]
1. The paper is clearly written. The idea of the layer-wise Markovian assumption is interesting and natural in terms of the SDR assumption.
2.  Theoretical analysis is provided which shows the proposed StoNet has asymptotically the same loss function as the desired DNN when the training sample size becomes large.
3. Numerical results demonstrate the efficacy of the proposed algorithm in classification and regression.

[Weaknesses]
1. It is not clear how are the depth h and layer width decided in the decomposition Eq (4). How to avoid learning trivial relations such as identity in the components?
2. The StoNet could be easily overparameterized where noise is also captured. Thus, the learned representation could be more than sufficient. Some form of regularization could be needed.
3. The misclassification rate for GSAVE and GSIR seems unrealistically high compared to the linear methods. Is there intuition why this is the case?
questions: See the main comments.
limitations: N/A
review_title: null

Meta-review: metareview: This paper gives a new method to perform nonlinear sufficient dimension reduction by utilizing a stochastic neural network. The derivation of the proposed method is justified by some theoretical background, and a convergence rate analysis is given for the derived algorithm. The practical performance is evaluated by some numerical experiments on real datasets.  

Although there are some related work, the proposed model is new. The theoretical justification of the proposed method is solid. The paper is overall clearly written.
The numerical experiments properly shows effectiveness of the method (while they are rather small).

In summary, this paper presents a novel method with nice theoretical and numerical justifications. I recommend acceptance. 
confidence: Certain
Decision: Accept