Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network
Authors: Siqi Liang
Yan Sun
Faming Liang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Adaptive Stochastic Gradient MCMC
Big Data
Deep Learning
Stochastic Neural Network
Sufficient Dimension Reduction

Abstract: Sufficient dimension reduction is a powerful tool to extract core information hidden in the high-dimensional data and has potentially many important applications in machine learning tasks. However, the existing nonlinear sufficient dimension reduction  methods often lack the scalability necessary for dealing with large-scale data.  We propose a new type of stochastic neural network under a rigorous probabilistic framework and show that it can be used for sufficient dimension reduction for large-scale data. The proposed stochastic neural network is trained using an adaptive stochastic gradient Markov chain Monte Carlo algorithm, whose convergence is rigorously studied in the paper as well. Through extensive experiments on real-world classification and regression problems, we show that the proposed method compares favorably with the existing  state-of-the-art sufficient dimension reduction methods and is computationally more efficient for large-scale data. 

Nonlinear Sufficient Dimension Reduction with a Stochastic Neural Network
Siqi Liang
Yan Sun Purdue University West Lafayette, IN 47907 sun748@purdue. edu
Purdue University West Lafayette, IN 47906 1 aan2277ppurdur edu
Faming Liang"
Purdue University
West Lafayette, IN 47907 fml imllaanguurdue... edu
Abstract
Sufficient dimension reduction is powerful tool to extract core information hid- den in the high-dimensional data and has potentially many important applications in machine learning tasks. However, the existing nonlinear sufficient dimension reduction methods often lack the scalability necessary for dealing with large-scale data. We propose a new type of stochastic neural network under a rigorous proba- bilistic framework and show that it can be used for sufficient dimension reduction for large-scale data. The proposed stochastic neural network is trained using adaptive stochastic gradient Markov chain Monte Carlo algorithm, whose conver- gence rigorously studied in the paper as well. Through extensive experiments on real-world classification and regression problems, we show that the proposed method compares favorably with the existing tfate-of-the-ar sufficient dimension reduction methods and is computationally more efficient for large-scale scale data.
Introduction
As supervised method, sufficient dimension reduction (SDR) aims to project the data onto lower dimensional space that the output is conditionally independent of the input features given the projected features. Mathematically, the problem of SDR can be described as follows. Let Y E IRd be the response variables, and let X (X₁, Xp) E H be the explanatory variables of dimension The goal of SDR to find lower-dimensional representation E R", as function of X for some q p, such that
P Z), equivalently Y V_
where denotes conditional independence. Intuitively, the definition implies that Z has extracted all the information contained in X for predicting Y. In the literature, SDR has been developed under both linear and nonlinear settings.
Under the linear setting, SDR is to find few linear combinations of that are sufficient to describe the conditional distribution of Y given X, i... finding projection matrix B 12Xx4 such that
YLXB""
(2)
more general definition for linear SDR based on a-field can be found in 7 Towards this goal, a variety of inverse regression methods have been proposed, e.g., sliced inverse regression (SIR)
To whom the correspondence should be addressed: Faming Liang.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

22), sliced average variance estimation (SAVE) 6 81 parametric inverse regression contour regression 25 and directional regression 24 These methods require strict assumptions on the joint distribution of (X, Y) or the conditional distribution of X|Y, which limit their use in practice. To address this issue, some forward regression methods have been developed in the literature, see e.g.. principal Hessian directions [27], minimum average variance estimation [45]. conditional variance estimation [12), among others. These methods require minimal assumptions on the smoothness of the joint distribution (X, Y). but they do not scale well for big data problems. They can become infeasible quickly as both p and n increase, see [2 for more discussions on this issue.
Under the nonlinear setting, SDR is to find a nonlinear function f-) such that
Y ((x).
(3)
A general theory for nonlinear SDR has been developed in A common strategy to achieve nonlinear SDR is to apply the kernel trick to the existing linear SDR methods, where the variable X first mapped to a nigh-dimensional feature space via kernels and then inverse or forward regression methods are performed. This strategy has led to variety of methods such as kernel sliced inverse regression (KSIR) 43 kernel dimension reduction (KDR) [33 14] manifold kernel dimension
reduction (MKDR) 35 generalized sliced inverse regression (GSIR) [22, generalized sliced average
variance estimator (GSAVE) [22], and least square mutual information estimation (LSMIIE) 42 A drawback shared by these methods is that they require to compute the eigenvectors or inverse of an n X n matrix. Therefore, these methods lack the scalability necessary for big data problems. Another strategy to achieve nonlinear SDR is to consider the problem under the multi-index model setting. Under this setting, the methods of forward regression such as those based on the outer product of the gradient 44 [0 have been developed, which often involve decomposccooiiioonnn of p p matrix and are thus unscalable for sigh--imensional problems.
Quite recently, some deep learning- based nonlinear SDR methods have been proposed in the literature, see e.g. 2 29], which are scalable for big data by training the deep neural network (DNN) with a mini-batch strategy. In (21), the authors assume that the response variable Y on the predictors X is fully captured by a regression
Y y= xx x)+e, +
(4)
for an unknown funetion 9()) and low rank parameter matrix B, and they propose a two-stage approach to estimate g(-) and B. They first estimate g(.) by g(-) by fitting the regression Y = g(X) + e with DNN and initialize the estimator of B using the outer product gradient (OPG) approach [5). and then refine the estimators of g(.) and B by optimizing them in a joint manner. However, as pointed out by the authors, this method might not be valid unless the estimate of g(-) is consistent, but the consistency does not generally hold for the fully connected neural networks trained without constraints. Specifically, the universal approximation ability of the DNN can make the latent variable Z B" X unidentifiable from the DNN approximator of g(-); or, said differently, Z can be an arbitrary vector by tuning the size of the DNN to be sufficiently large. A similar issue happened to 21. where the authors propose learn the latent variable Z by optimizing three DNNs to approximate the distributions p( Z [X), x Z) and p Z) Z). respectively, under the framework of variational autoencoder. Again, Z suffers from the identifiability issue due the universal approximation ability of the DNN. In [29]. the authors employ a regular DNN for sufficient dimension reduction. which works only for the case that the distribution of the response variable falls into the exponential family. How to conduct SDR with DNNs for general large-scale data remains an unresolved issue.
We address the above issue by developing a new type of stochastic neural network. The idea can be loosely described as follows. Suppose that we are able to learn a stochastic neural network, which maps X to Y via some stochastic hidden layers and possesses layer-wise Markovian structure. Let h denote the number of hidden layers, and let Y1, y,...., Yn denote the outputs of the respective stochastic hidden layers. By the layer-wise Markovian structure of the stochastic neural network, we can decompose the joint distribution of (Y, Y h. Y ......., Y1) conditioned on X as follows
π YY, Y,, Y ---....., Y, x) = x (Y h (Y (Y 7 (Y,)x),
(5)
where each conditional distribution is modeled by linear or logistic regression (on transformed outputs of the previous layer), while the stochastic neural network still provides good approximation to the underlying DNN under appropriate conditions on the random noise added to each stochastic layer. The laser-wise Markovian structure implies Y Y_LLYY,,, h. and the simple regression structure of [YP2 (Y))) successfully gets around the identifiability issue of the latent variable Z := Yn that has

---

been suffered by some other deep learning-base based methods 22 How to define and learn such stochastic neural network will be detailed in the paper.
Our contribution in this paper is three-fold: (i) We propose a new type of stochastic neural network (abbreviated as "StoNet" hereafter) for sufficient dimension reduction, for which a layer- wise Markovian structure (5) is imposed on the network in training and the size of the noise added to each hidden layer is calibrated for ensuring the StoNet to provide a good approximation to the underlying DNN. (ii) We develop an adaptive stochastic gradient MCMC algorithm for training the StoNet and provides a rigorous study for its convergence under mild conditions. The training algorithm is scalable with respect to big data and it is itself of interest to statistical computing for the problems with latent variables or missing data involved. (iii) We formulate the StoNet as a composition of many simple linear/logistic regressions, making its structure more designable and interpretable. The backward imputation and forward parameter updating mechanism embedded in the proposed training algorithm enables the regression subtasks to communicate globally and update locally. As discussed later, these two features enable the StoNet to solve many important scientific problems, rather than sufficient dimension reduction, in more convenient way than does the conventional DNN. The StoNet bridges us from linear models to deep learning.
Other related works. Stochastic neural networks have long history in machine learning. Famous examples include multilayer generative models [18], restricted Boltzmann machine [19] and deep Boltzmann machine [38]. Recently, some researchers have proposed adding noise to the DNN to improve its fitting and generalization. For example, [39] proposed the dropout method to prevent the DNN from over-fitting by randomly dropping some hidden and visible units during training; proposed adding gradient noise to improve training: [16] 36 47 40) proposed to use stochastic
32
activation functions through adding noise to improve generalization and adversarial robustness, and
proposed to learn the uncertainty parameters of the stochastic activation functions along with the training of the neural network.
48
However, none of the existing stochastic neural networks can be used for sufficient dimension reduction. It is known that the multilayer generative models [18], restricted Boltzmann machine [19] and deep Boltzmann machine (38) can be used for dimension reduction, but under the unsupervised
mode. As explained in [39] the dropout method is essentially a stochastic regularization method, where the likelihood function is penalized in network training and thus the hidden layer output of the resulting neural network does not satisfy 3 In [6), the size of the noise added to the activity function is not well calibrated and it is unclear whether the true log-likelihood function is maximized or not. The same issue happens to (32]; is unclear whether the true log-likelihood function is maximized by the proposed training procedure. In (36), the neural network was trained by maximizing a lower bound of the log-likelihood function instead the true log-likelihooe function; therefore, its hidden layer output does not satisfy (3). In [47], the random noise added to the output of each hidden unit depends on its gradient; the mutual dependence between the gradients destroys the lase Markovian structure of the neural network and thus the hidden layer output does not satisfy (3). Similarly, in [88], independent noise was added to the output of each hidden unit and, therefore, the hidden layer output satisfies neither (5) nor In 40) inclusion of the support vector regression (SVR) layer to the stochastic neural network makes the hidden layer outputs mutually dependent, although the observations are mutually independent.
2 StoNet for Sufficient Dimension Reduction
In this section, we first define the StoNet, then justify its validity universal learner for the map from X to Y by showing that the StoNet has asymptotically the same loss function as a DNN under appropriate conditions, and further justify its use for sufficient dimension reduction.
2.1 The StoNet
Consider a DNN model with & hidden layers. For the sake of simplicity, we assume that the same activation function & is used for each hidden unit. By separating the feeding and activation operators
3

---

of each hidden unit, we can rewrite the DNN in the following form
y = 1 + w X,
Y. b, + w, (Y 1-1, i= ==2,3, 2.3....,
(6)
Y 1 + wh+1 (Y)) ) + ex+l,
where e/+11 ~ ((0 0 1d)_1) ) is Gaussian random error; Y,, b; E R4 for = 1,2, h; Y. b,+1 RRhi (Y (-1)
(Y (1,.,-1)) for i = 2, h (-) is the activation function, and i-i.j is the jth element of Y 111 w, E Rd for i= 1,2,..... ,, and dg denotes the dimension of X. For simplicity, we consider only the regression problems in (6) By replacing the third equation in 6 with logit model, the DNN can be trivially extended to the classification problems.
The StoNet, probabilistic deep learning model, can be constructed by adding auxiliary noise to ys's, i = 1,2,..... h in (6). Mathematically, the StoNet is given by
10%)
12 90%)
Y1 =
b + wi X + E1,
Y. b, + wi V (Y: (-1) + e; = ...,, h,
Y = 1b+1 + wh+1 (Y + Ch+1s
where Y1, Y ...... Yn can be viewed as latent vari- ables. Further, we assume that e, ~ [0, of la,) for i = 1,2,..., h. h 1. For classification networks, the parameter o plays the role of temperature for the binomial multinomial distribution formed the output layer, which works with (of..... o } to- gether control the variation of the latent variables
By t
By th MMp00)) th Logistic/ Linsar Reg esiio
Figure I: An illustrative plot for the structure of a StoNet with two hidden layers.
[Y,...., }- Figure T depicts the architecture of the StoNet. In words, the StoNet has been formulated as a composition of many simple linear/logistic regressions, which makes its structure more designable and interpretable. Refer to Section 5 for more discussions on this issue.
2.2 The StoNet as an Approximator DNN
To show that the StoNet is valid approximator to DNN, i.c., asymptotically they have the same loss function, the following conditions are imposed on the model. To indicate their dependence on the training sample size n, we rewrite as nn for i =
1. Let 0 = (w, b,), ), let (01,0, 22 ... 0 1 denote the parameter vector of StoNet, let do denote the dimension of A, and let θ denote the space of O.
Assumption A1 (i) θ is compact, i.e., θ is contained in a do-ball centered at O with radius r; (ii) E(log . (Y 0))² < 50 for any E O; (iii) the activation function wp(-) is c' -Lipschitz continuous for some constant c'; (iv) the network's 's depth h and widths d, 's are both allowed to increase with
n: (v) nn1 S 0n2 S ... S On,b+1, Nn,h+1 O1), and d (II dd 1 for any E (1,2....,
Condition (i) is more or less a technical condition. As shown in Lemma SI (in supplementary material), the proposed training algorithm for the StoNet ensures the estimates of 0 to be T-puupeer bounded. Condition (ii) is the regularity condition for the distribution of Y. Condition (iii) can be sat- isfied by many activation functions such as tanh, sigmoid and ReLU. Condition (v) constrains the size of the noise added to each hidden layer such that the StoNet has asymptotically the same loss function as the DNN when the training sample size becomes large, where the factor da++1 (TT d)dd dd is derived in the proof of Theorem 2.1 and can be understood as the amplification factor of the noise e₁ at the output layer.
Let L θ E denote the loss function of the DNN as defined (6. which is given by
1 n
L(
log n Y
X 0),
(8)

---

where n denotes the training sample size, and indexes the training samples. Theorem 2.1 shows that the StoNet and the DNN have asymptotically the same training loss function.
Theorem 2.1 Suppose Assumption A7 holds. Then the StoNet 7 and the neural network O have asymptotically the same loss function, i.e.
sup OE9
log IT (Y (0) Y (
mis
0)
Σ log T Y X 0) 0,
00,
(9)
1=1
where Y mis (Y Y2, hn) denotes the collection of all latent variables in the StoNet 7
Let (0) E(log (Y) X 0 )) where the expectation is taken with respect the joint distribution
(X, Y). By Assumption Al (i) (ii) and the law of large numbers,
Σ 71
1
log क (Y
x ()) 0) (0) 4
(10)
holds uniformly over O. Further, we assume the following condition hold for (*):
Assumption A2 (i) (0) is continuous 0 and uniquely maximized at f"; (ūi for any >
0, supece),B( ((e) Q' exists, where B(c) 10 00 00' < c), and 5 = Q"(0")
sup Q* (0) O.
A2 is more or less technical assumption. As shown in 33 (see also (5).. for fully connected D DNN, almost all local energy minima are globally optimal the width of one hidden layer of the DNN is no smaller than the training sample size and the network structure from this layer on is pyramidal. Similarly, M. [T. [50), and [49] proved that the gradient-basee algorithms with random initialization can converge to the global optimum provided that the width of the DNN is polynomial in training sample size. All the existing theory implies that this assumption should not be a practical concern for StoNet long its structure is large enough, possibly over-pare parameterized, such that the data can be well fitted. Further, we assume that each θ for the DNN is unique up to loss-invariant transformations, such reordering some hidden units and simultancously changing the signs of some weights and biases. Such an implicit assumption has often been used theoretical studies for neural networks, see e.g. [28 and 41 for the detail.
Assumption
Theorem 2.2 Suppose Assumptions and A2 hold, and x: (Y, , Y mis] x) is continuous O. més X x(0, 0)). Then ||0,, 0" 2. 0 as cX.
O, arg maxage / Σx= log π (Y
This theorem implies that the DNN 66 can be trained by training the StoNet 7 which are asymptot- ically equivalent as the sample size becomes large. Refer to the supplement for its proof.
2.3 Nonlinear Sufficient Dimension Reduction via StoNet
The joint distribution (Y, Y mis X, ()) for the StoNet can be factored as
T Y. Y mis X, Y X 0
01) 0n41),
(11)
based on the Markovian structure between layers of the StoNet. Therefore,
π YYlY mis: X,0) A) (Y n. 0x+1).
(12)
By Proposition 2.1 of (23). Equation (12) is equivalent to Y Y_IX//,, which coincides with the definition of nonlinear sufficient dimension reduction in (3). In summary, we have the proposition:
Proposition 2.1 For a well trained StoNet for the mapping X + Y, the output of the last hidden layer Yn h satisfies SDR condition in 3
The proof simply follows the above arguments and the properties of the StoNet. Proposition 2.1 implies that the StoNet can be useful and flexible tool for nonlinear SDR. However, the conventional optimization algorithm such as stochastic gradient descent (SGD) is no long applicable for training the StoNet. In the next section, we propose to train the StoNet using adaptive stochastic gradient MCMC algorithm. At the end of the paper, we discuss how to determine the dimension of Y, via regularization at the output layer of the StoNet.
5

---

3 An Adaptive Stochastic Gradient MCMC algorithm
3.1 Algorithm Establishment
Adaptive stochastic gradient MCMC algorithms have been developed in [0] and (9) which work under the framework of stochastic approximation MCMC K] Suppose that we are interested in solving the mean field equation
E ([ H =(220)
(13)
where π (( (Z ()) denotes probability density function parameterized by O. The adaptive stochastic gradient MCMC algorithm works by iterating between the steps: (i) sampling, which is to generate Monte Carlo sample Z (8) from transition kernel that leaves π (2))( as the equilibrium distribution; and (ii) parameter updating, which update (()) based on the current sample (*)
in stochastic approximation scheme. These algorithms are said "adaptive" as the transition kernel used in step (i) changes with iterations through the working estimate ((8)
By Theorem 2.2 the StoNet can be trained by solving the equation
EE (Y mis: 0) / H H(Y mis: (Y mis 0, X. Y JYY mis = 0.
(14)
where HY mis. 0) Ve log ग T(Y, Y mis X 0). Applying the adaptive stochastic gradient MCMC algorithm to (14 leads to Algorithm T where stochastic gradient Hamilton Monte Carlo (SGHMC) [5] is used for simulating the latent variables Y mis- Algorithm iss expected to outperform the basic algorithm by [00. where SGLD is used in the sampling step, due to the accelerated convergence of
SGHMC over SGLD (33]. In Algorithm we let 0
Y x (9.))
(x (+), Y (+)) denote a training
k-() denote the latent variables imputed for the training sample 8, and let (= = ( mis sample at iteration k.
To make the computation for the StoNet scalable with respect to the training sample size, we train the parameter O with mini-batch data and then extract the SDR predictor Y, a. with the full dataset; that is, we can run Algorithm in two stages, namely, O-training and SDR. In the O-training stage, the algorithm is run with mini-batch data until convergence of 0 has been achieved; and in the SDR stage, the algorithm is run with full data for a small number of iterations. set the number of iterations/epochs of the SDR stage to 30. The proposed algorithm has the same order of computational complexity as the standard SGD algorithm, although it can be a little slower than SGD due to multiple iterations being performed each backward sampling step.
this paper, we typically
3.2 Convergence Analysis of Algorithm
1
Notations: We let D (D1, D2.....
... Dr) denote dataset of n observations. For StoNet, D; has in- cluded both the input and output variables of the observation. We let Y mis Z = (Z1,
Zn). log T 3; Di, 0). Let
where 7, is the latent variable corresponding to D;, and let fD. (2, 0)
= (=1. .., =n) be realization of (71, 72..... 7n). and let FD 2 0) = Σ ( O). For simplicity, we assume cx2; = < for 1, 2, .... h, and 7k,s = 7% for 1, 1,....., h+
To facilitate theoretical study, one iteration of Algorithm I is rewritten in the new notations as follows.
(i) (Sampling) Simulate the latent variable Z by setting (k+1) = (1 Ex+ 11)17))0 EK+ (z) ((0)) + / z (++1) = (h) + (k)
(16)
where is the friction coefficient, 3 is the inverse temperature, h indexes the iteration, ck+1 is the learning rate, and V (z o(k)) is estimate of V (ZFD(Z(A) ,(8)).
(ii) (Parameter updating) Update the parameters 0 by setting
g ((k) + 7h41 (2 (k+1) O(()),
(17)
where Mh+1 is the step size, Hz (k+1) ((k)) = [syl Eies, fD, (2 (k+1) gl()) ), and Sk
denotes minibatch of the full dataset.

---

Algorithm 1: An Adaptive SGHMC algorithm for training StoNet
Input: total iteration number K, Monte Carlo step number MCC the learning sequence {<R,s :/ .. = (7h,4 : 1,2, T; .., 1];
1.2....,he h++ 1). and the step size sequence
Initialization: Randomly initialize the network parameters θ 00) = (0)
for k........,K do
STEP 0: Subsampling: Draw mini-batch of data and denote it by S&:
STEP1 1: Backward Sampling
For each observation Sk, sample Y in the order from layer layer 1. More
explicitly, we sample Y! (=(.,)) from the distribution
a (( (0(h-1) 1) 0(h-") 1) ((ak), ++1 y( a( i41 10 y( )T(! 104-1) (m)))
by running SGHMC in tHмC steps:
Initialize v. (+.0) 0, and initialize Y ((kk,))) by the corresponding Y, calculated 6 for = 1, ..... .,. HMC do
for h 1,...,11 do
o( =(1 ex, ( E y((cke-1 log (y(okki- ((t-1)) v(ak(-1))
+ tk, V. y( log (r (r 69411), y(unkttt) + (e
y(ukk.) =)(ukl-)) + E (x.
(15)
where eritit ~ N(0, Id;), Ckis is the learning rate, and is the friction coefficient. end end
Set Y ((a)) Y (shbttm))) for 1,2, ......
STEP 2: Parameter Update
Update the estimates of (k-1)
(0 (0 0, xx ..... 0 by
g(hh = k(k
n
Va, log ग (V (k) (--) y((r))) i=
7he
SA
sESt
where Th, is the step size used for updating 0,. end
Theorem 3.1 Suppose Assumptions Bl B6 (in the supplementary material) hold. If we set en
Ce/(ce + ka) and 7Kk C./(c, + km) for some constants E (0, 1), C. > 0, C., > 0, Ce 2 0 and
Cg 2 0, then there exists an iteration ko and constant Xo > such that for any > ko.
O( (k) - -0 11)) < Ao
(18)
where 0* denotes solution to the equation 14 and the explicit form of Xo is given in Theorem S7
Let JDD& denote the probability law of (Z(k) ((k)) given the dataset D, let an denote the target distribution x22(0', D), let Th E (++1, and let H(p(:.) denote semi-metrie for probability distributions. Theorem 3.2 establishes convergence μD.
Theorem 3.2 Suppose Assumptions B1 B7 (in the supplementary material) hold. Then for any N.
(VOCH)+ + ((K))
1/4
h-1 CoT
Wa(HD.T S C.C.CHHHHe (Hp(Ho JD)E To / [Tg((T))
Σ J=1
which can be made arbitrarily small by choosing large enough value of Ti and small enough values of 6 and Th provided that { and {rk} are set as in Theorem SI Here Css and Cg denote some constants and explicit form of ((K) is given in Theorem S2
7

---

As implied by Theorem 3.2
Z (i) converges weakly to the distribution т TZ D) as k oc, which ensure validity of the decomposition (TT) and thus the followed SDR.
We note that our theory is very different from [0]. First, θ is essentially assumed to be bounded in [10), while our study is under the assumption θ = Rdde Second, for weak convergence of latent variables, only the convergence of the ergodicity average is studied in [10], while we study their convergence in 2-Wasserstein distance such that (11) holds and SDR can be further applied.
4 Numerical Studies
In this section, we empirically evaluate the performance of the StoNet on SDR tasks. We first compare the StoNet with an existing deep SDR method, which validates the StoNet as SDR method. Then we compare the StoNet with some existing linear and nonlinear SDR methods on classification and regression problems. For each problem, we first apply the StoNet and the linear and nonlinear SDR methods to project the training samples onto lower dimensional subspace, and then we train separate classfficatio//eerrsssoo model with the projected training samples. A good SDR method expected to extract the response information contained in the input data as much as possible. The example for multi-label classification is presented the supplement.
4.1 A Validating Example for StoNet
We use the M1 example in 21 to illustrate the non- con-identifiabiliiyy issue suffered by the deep SDR methods developed in 21 and 2]. The dataset consists of 100 independent observations. Each observation is generated from the model Y = ros(X"b)) b) + where X 1220 follows a multivariate Gaussian distribution, b E IR20 is a vector with the first 6 dimensions equal to 1 and the other vB dimensions equal to O, and e follows generalized Gaussian distribution GN V1/2, 0.5). We use the code P provided by [21] to conduct the experiment, which projects the data one-dimensional space by working with a refinement network of structure 20-1-512-1. Let Z; and Z denote two SDR vectors produced by the method in two independent runs with different initializations of network weights. We then test the independence of Z1 and 2 using the R package RCIT The test returns a p-value of 0.4068, which does not reject the null hypothesis that 11 and Ze are independent.
We have also applied the proposed method to the same dataset, where the StoNet has a structure of 10-10-1-1 and tanh is used as the activation function. In this way, the StoNet projects the data to one- dimensional space. The SDR vectors produced in two independent runs (with different initializations of network weights) of the method are collected and tested for their dependence. The test returns a p-value of 0.012, which suggests that the two SDR vectors are not independent.
This example suggests that if a complicated neural network model is used to fit the function g(-) in then the dimension reduced data do not necessarily capture the information of the original data.
4.2 Classification Examples
We first test the StoNet on some binary classification examples taken from 37 and 466. The dimensions of these examples are generally low, ranging from 6 to 21. There two steps for each example: first, we project the training samples onto a low-dimensional subspace with dimension q [p2]] q pp4| and then train a logistic regression model on the projected predictors for the binary classification task. We trained two one-hidden-layer StoNets with [p/2] and [p/4] hidden units, respectively. For comparison, four state-of-the-art non-linear SDR methods, including LSMIF, GSIR, GSAVE and KDRF) were trained to extract nonlinear sufficient predictors. In addition, three popular linear dimension reduction methods were taken baselines for comparison, which include SIR, SAVE and PC The hyperparameters of these methods were determined with 5-fold
"The code is available at http::///...aaaa.....///////////
3 'The package is available at
4 The code for LSMIE is available at http:://ww/ ms k. wttpkkyo..c..///sses/..so hptsso//t/areeeeeo htm1#LSDR the
code for GSIR from Chapter 13 of [3] the code for GSAVE from Chapter 14 of 23 and the code for
KDR is available at hhttpss:: h/ts:://wwww /ww. ism. ac. jp/ fukumizu software.wreoooooo html
'The codes for SIR and SAVE are available in the package sliced downloadable at https://joshloyyal
github. io/sliced/ and the code for PCA is available the package skleam.
8

---

cross-validation in terms of misclassification rates. Random partitioning of the dataset in cross- validation makes their results slightly different in different runs, even when the methods themselves are deterministic. Refer to the supplement for the parameter settings used in the experiments.
The results summarized in Table [ which reports the mean and standard deviation of the misclassification rates averaged over 20 independent trials. Table [ shows that StoNet compares favorably to the existing linear and nonlinear dimension reduction methods.
Table 1: Mean misclassification rates on test sets (with standard deviations given in the parentheses) over 20 independent trials for some binary classification examples. In each row, the best result and those comparable the best one (in a (-test at significance level of 0.05) are highlighted in boldface.
Dutanets
StoNet
LSMIE
GSIR
GSAVE
KDR
SIR
SAVE 0.3000 (11 10)
PCA
Ohyood
(0068)
0.2860 (100))
0.8648.00633
0.2847 10) (00022
0.1373(0117)
0.301 1013L.11 10)
0.06931,0060
0.1733(0013)
0.0667)0071)
0.0947(
0.2713 (.0188)
0,1333(0130)
0,3000, (11 18)
0.1467L00133
2
0.2578,,0772
0.281200110
0,2772 0091)
0,2710,
0.2818(0125) 0.202
0.2870X,00 (075)
0.2857L0122
brensteancer
0.26M21
0,27600.0118
0.2740(.0100
0.2705
0.2749.0105)
0.2831001100)
0.2022(
0.27666,0009
faresollar
2
0.32361.0040)
0.3239 (,0043)
0.3346100033)
0197)
0.3400,0010)
0.3345
0,4161(0138) 0.333
0.3336,0028)
0.3673(0108)
0.3312.0052)
0.3328(0049)
0,4302(0113))
0127)
0.3313/,0046)
0.361210006)
heart
6
3
0.16251.0076)
(.1625(.1062)
0.1729(.0073)
(.1695(,003))
0.1645(00669
0.1650L00063
0.1731.0000)
(.175)
0.1715,0075)
(J870000661)
0. 01070)
(1720K00058)
0.1720X,0073)
0.19101.000 0053)
0.1920 (0123)
0.1830L,0022
5
0.23680, (0000)
0.25L00052)
0,2325| (058))
0.252%,0050
0.2703I 0.2367.8668) (.245)
0.2777 (070))
jemmin
0.2356(.0044)
0.2443(0056)
0.2427(00046)
0.2312(0007)
0.247()0)75)
0.2360.0068)
0.24471
(0022)
0.2350L.051)
0.1091(,0010
0.1336(0011)
0.1095,0016] 0.1150)00055)
0.1269X0003 (031)
0.1422(.0020) 0,1453
0.14866 0013))
waveform
10
0.1079(,0012)
0.1369L (015)
0.1117(0009)
0.1070.0013)
0. 0.1254
0. (.144)00117)
0,14170000200
0,14301 (0220)
We have also tested the methods on a multi-label classification problem with sub-MNIST dataset. Refer to Section [ (of the supplement) for the details of the experiments. The numerical results are summarized in Table 2 which indicates the superiority of SToNet over the existing nonlinear SDR methods in both computational efficiency and prediction accuracy.
Table 2: Misclassification rates on the test set for the MNIST example, where the best misclassification rates achieved by different methods at each dimension q are specified by bold face. The CPU time (in seconds) was recorded on a computer of 2.2 GHz.
q
StoNet
LSMIE
GSIR
GSAVE
Autoencoder
PCA
392
0.0456
0.0596
0.0535
0.1965
0.1002
196
0.0484
0.0686
0.0611
0.2268
0.0782
98
0.0503
0.0756
0.0696
0,2733
0,0843
49
0.0520
0.0816
0.0764
0.3112
0.0889
10
0.0825
0.0872
0.0901
0.4036
0.1644
Average Time(s)
96.18
24hours
16005.59
22154.11
1809.18
5.11
4.3 A Regression Example
The dataset, relative location of CT slices on axial axis 0 contains 53,500 CT images collected from 74 patients. There are 384 features extracted from the CT images, and the response is the relative location of the CT slice on the axial axis of the human body which between 0 and 180 (where denotes the top of the head and 180 the soles of the feet). Our goal is to predict the relative position of CT slices with the high-dimensional features.
Due to the large scale of the dataset and the high computation cost of the nonlinear SDR methods LSMIE, GSIR and GSAVE, we don't 't include them as baselines here. Similar to the previous examples, the experiment was conducted in two stages. First, we applied the dimension reduction methods to project the data onto a -iwer-dimeniionll subspace, and then trained DNN on the projected data for making predictions. Note that for the StoNet, the dimension reduced data can be modeled by linear regression in principle and the DNN is used here merely for fairness of comparison; while for autoencoder and PCA, the use of the DNN for modeling the dimension reduced data seems necessary for such a nonlinear regression problem. The mean squared error (MSE) and Pearson correlation were used as the evaluation metrics to assess the performance of prediction models. For this example,
TThi dataset can be downloaded from UCI Machine Learning repository.

---

As an alternative way to avoid the over sufficiency issue, we can add a post sparsification step to the StoNet, i.e., applying a sparse SDR procedure (e.g., B0 and [31] with Lasso penalty) the output layer regression of a learnt StoNet by noting that the regression formed at each node of the StoNet is we have also trained a DNN with one hidden layer and 100 hidden units as the comparison baseline. Refer to the supplementary material for the hyperparameter settings used in the experiments.
Table 3: Mean MSE and Pearson correlation on the test sets (and their standard deviations in the parentheses) over 10 trails for the Relative location of CT slices on axial axis dataset.
StoNet
Autoencoder
PCA
4
MSE
Corr
MSE
Corr
MSE
Cor
192
0.0002(.000)
0.9986(.000)
0.9267(0017)) 0.0079(.0015)
0.0027(0000)
0.9755(0000)
96
0.0002(.0000
0.9985(0001)
0.0106(.0024)
0.9002(.0237)
0.0026(.0000)
0.9756(.000)
48
0.0002(.0000)
0.9982(.00))
0.0143(0003)
0.8562(.0399)
0.0034(.0000)
0.9682(0001)
24
0.0002(.0000)
0.9980(0000)
0.0185(,0033)
0.8168(0364)
0.0042(.0000)
0.9612(.0001)
12
0,9980(0001)) 0.0002(00000
0.0233(00027)
0.7579(0338)
0.0053(.000)
0.9499(0000))
6
0.0002(00000)
0,9980(0001))
0.0304(.0024)
0.6668(.0300)
0.0102(.0001)
0.9023(0002)
3
0.0004(00000) 0.9965(.0002)
0.0384(.0030)
0.5529(0538)
0.0209(0000)
0.7858(0004)
The results, which are summarized in Figure S2 dimension q decreases, the performance of Autoencoder degrades significantly. In contrast, StoNet can achieve stable and robust performance even when g is reduced to 3. Moreover, for each value of q. StoNet outperforms Autoencoder and PCA significantly in both MSE and Pearson correlation.
(in the supplement) and Table 3 show that as the
5 Conclusion
In this paper, we have proposed the StoNet as a type of stochastic neural network under the rigorous probabilistic framework and used it as a method for nonlinear SDR. The StoNet, as an approximator neural networks, possesses a layer-wise Markovian structure and SDR can be obtained by extracting the output of its last hidden layer. The StoNet overcomes the limitations of the existing nonlinear SDR methods, such inability in dealing with high-dimensional data and computationally inefficiency for large-scale data. We have also proposed an adaptive stochastic gradient MCMC algorithm for training the StoNet and studied its convergence theory. Extensive experimental results show that the StoNet method compares favorably with the existing state-of-the-art nonlinear SDR methods and is computationally more efficient for large-scale data.
In this paper, we study SDR with a given network structure under the assumption that the network structure has been large enough for approximating the underlying true nonlinear function. To determine the optimal depth, layer width, etc., we can combine the proposed method with a sparse deep learning method, e.g. 177 and 41. That is, we can start with wer-parameterized neural network, employ a sparse deep learning technique to learn the network structure, and then employ the proposed method to the learned sparse DNN for the SDR task. We particularly note that the work [1] ensures consistency of the learned sparse DNN structure, which can effectively avoid the over sufficiency issue, e.g., learning a trivial relation such as identity in the components.
multiple index model. In this way, the StoNet, given its universal approximation ability, provides simple method for determining the central space of SDR for general nonlinear regression models.
The StoNet has great potentials in machine learning applications. Like other stochastic neural networks [6, 36 47 40). it can be used to improve generalization and adversarial robustness of the DNN. The StoNet can be easily extended to other neural network architectures such as convolutional neural network (CNN), recurrent neural network (RNN) and Long Short- Term Memory (LSTM) networks. For CNN, the randomization technique can be directly applied the fully connected layers. The same technique can be applied to appropriate layers of the RNN and LSTM as well.
Acknowledgments
Liang's research is support in part by the NSF grants DMS-2015498 and DMS-2210819, and the NIH grant RO1-GM126089.
10

---

References
[1] Zeyuan Alu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-paramellrization. In ICML, 2019.
[2] Ershad Banijamali, Amir-Hossein Karimi, and Ali Ghodsi. Deep variational sufficient dimen- sionality reduction. In Third Workshop on Bayesian Deep Learning (NeurIPS 2018), 2018.
[3] Albert Benveniste, Michael Métivier, and Pierre Priouret. Adaptive Algorithms and Stochastic Approximations. Berlin: Springer, 1990.
[4] Efstathia Bura and Dennis Cook. Estimating the structural dimension of regressions via parametric inverse regression. Journal of The Royal Statistical Society Series B-statistical Methodology, 63:393-410, 2001.
[5] Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In International conference machine learning, pages (683-1691, 2014.
[6] R. Dennis Cook. Save: method for dimension reduction and graphics in regression. Commu- nications in statistics-Theery and methods, 29(9-10):210-22122 2000.
[7] R. Dennis Cook. Fisher lecture: Dimension reduction in regression. 22(1):1-26, 2007.
Statistical Science,
[8] R. Dennis Cook and Weisberg. Discussion of 'sliced inverse regression for dimension reduction," by k.c. li. Journal of the Americun Statistical Association, 86:328-332, 1991.
[9] Wei Deng, Guang Lin, and Faming Liang. A contour stochastic gradient langevin dynamics algorithm for simulations of multi-modal distributions. In H. Larochelle, M. Ranzato, R. Had- sell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 15725 11736. Curran Associates, Inc., 2020.
[10] Wei Deng, Xiao Zhang, Faming Liang, and Guang Lin. An adaptive empirical bayesian method for sparse deep learning. Advances in neural information processing systems, 2019:5563, 2019.
[11] Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In ICML, 2019.
[12] Lukas Fertl and Efstathia Bura. Conditional variance estimator for sufficient dimension reduc- tion. arXiv: Methodology, 2021.
[13] Kenji Fukumizu, Francis Bach, and Michael Jordan. Kernel dimension reduction in regression. The Annals of Statistics, 37(4):1871-1905 2009.
[14] Kenji Fukumizu and Chenlei Leng. Gradient-based kernel dimension reduction for regression. Journal of the American Statistical Association, 109:359 - 370, 2014.
[15] Marco Gori and Alberto Tesi. On the problem of local minima in backpropagation. IEEE Trans. Pattern Anal. Mach. Intell., 14:76-86, 1992.
[16] Çaglar Gülçehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. Noisy activation functions. In ICML., pages 3059-3068, 2016.
[17] Song Han, Jeff Pool, John Tran, and William J. Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems 28, pages 1135-1143, 2015.
[18] Geoffrey Hinton. Learning multiple layers of representation. Trends in Cognitive Sciences, 11(10)442-444,22 2007.
[19] Geoffrey E. Hinton and R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313:504- 507, 2006.
[20] Marian Hristache, Anatoli Juditsky, Jörg Polzehl, and Vladimir Spokoiny. Structure adaptive approach for dimension reduction. The Annals of Statistics, 196))11377-166, 2001.
11

---

[21] Daniel Kapla, Lukas Fertl, and Efstathia Bura. Fusing sufficient dimension reduction with neural networks. Computational Statistics & Data Analysis. 2021.
[22] Kuang- Yao Lee, Bing Li, and Francesca Chiaromonte. A general theory for nonlinear sufficient dimension reduction: Formulation and estimation. The Annals of Statistics, 41(1):221 249, 2013.
[23] Bing Li. Sufficient dimension reduction: Methods and applications with R. CRC Press, 2018.
[24] Bing Li and Shaoli Wang. On directional regression for dimension reduction. Journal of the American Statistical Association, 102:1008
2007.
[25] Bing Dong Li, Hongyuan Zha, and Francesca Chiaromonte. Contour regression: A general approach dimension reduction. Annals of Statistics, 33:1580-1616, 2005.
[26] Ker-Chau Li. Sliced inverse regression for dimension reduction. Journal of the American Statistical Association, 86(414):316-32) 1991.
[27] Ker-Chau Li. On principal hessian directions for data visualization and dimension reduc- tion: Another application of stein's 's lemma. Journal of the American Statistical Association, 87(420):1025-1339 1992.
[28] Faming Liang, Qizhai Li, and Lei Zhou. Bayesian neural networks for selection of drug sensitive genes. Journal of the American Statistical Association, 113(523):955-99) 972, 2018.
[29] Siqi Liang, Wei-Heng Huang, and Faming Liang. Sufficient dimension reduction with deep neural networks for phenotype prediction. Proceedings of the 3rd International Conference on Statistics: Theory and Applications, 2021.
[30] Qian Lin, Zhigen Zhao, and Jun S. Liu. On consistency and sparsity for sliced inverse regression in high dimensions. Annals of Statistics, 46(2):580-610, 2018
[31] Qian Lin, Zhigen Zhao, and Jun S. Liu. Sparse sliced inverse regression via lasso. Journal of the American Statistical Association, 1 114:1726 1739, 2019.
[32] Arvind Neclakantan, Luke Vilnis, Quoe V. Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks. ArXiv, abs/1511.06800 2017.
[33] Christopher Nemeth and Paul Fearnhead. Stochastic gradient markov chain monte carlo. Journal of the American Statistical Association, 116:433 450, 2019.
[34] Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. In International conference on machine learning, pages 2603-2612. PMLR, 2017.
[35] Jens Nilsson, Fei Sha, and Michael I Jordan. Regression on manifolds using kernel dimension reduction. In Proceedings of the 24th international conference on Machine learning, pages 697-704. ACM, 2007.
[36] Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neural networks by noise: Its interpretation and optimization. ArXiv, abs/1710.05179, 2017.
[37] Gunnar Rätsch, Takashi Onoda, and Klaus-Robert Müller. Soft margins for adaboost. Machine Learning, 42:287-320. 2001.
[38] Ruslan Salakhutdinov and Geoffrey Hinton.
Deep boltzmann machines. In Proceedings of the International Conference Artificial Intelligence and Statistics, pages 448-455, 2009.
[39] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-1955 2014.
[40] Yan Sun and Faming Liang. A kernel-expanded stochastic neural network. Journal of the Royal Statistical Society, Series B, 84:547-578, 2022.
12

---

[41] Yan Sun, Qifan Song, and Faming Liang. Consistent sparse deep learning: Theory and computation. Journal of the American Statistical Association, page in press, 2021.
[42] Taiji Suzuki and Masashi Sugiyama. Sufficient dimension reduction via squared-loss mutual information estimation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 804-811. JMLR Workshop and Conference Proceedings. 2010.
[43] Han-Ming Wu. Kernel sliced inverse regression with applications to classification. Journal of Computational and Graphical Statistics, 17(3):590-610, 2008.
[44] Yingeun Xia. A constructive approach to the estimation of dimension reduction directions. The Annals of Statistics, (5(6):2654-2690) 2007.
[45] Yingeun Xia, Howell Tong, WK Li, and Li-Xing Zhu. An adaptive estimation of dimension reduction space. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(3):363-4151 2002.
[46] Makoto Yamada, Gang Niu, Jun Takagi, and Masashi Sugiyama. Computationally efficient sufficient dimension reduction via squared-loss mutual information. In Chun-Nan Hsu and Wee Sun Lee, editors. Proceedings of the Asian Conference on Machine Learning, volume 20 of Proceedings of Machine Learning Research, pages 247-262, South Garden Hotels and Resorts, Taoyuan, Taiwain, 14-15 Nov 2011. PMLR.
[47] Zhonghui You, Jinmian Ye, Kunming Li, and Ping Wang. Adversarial noise layer: Regularize neural network by adding noise. In 2019 IEEE International Conference on Image Processing (ICIP), pages 909-913, 2018.
[48] Tianyuan Yu, Yongxin Yang, Da Li, Timothy M. Hospedales, and T. Xiang. Simple and effective stochastic neural networks. In AAAI, 2021.
[49] Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over- parameterized deep relu networks. Machine Learning, 109:467 492, 2020.
[50] Difan Zou and Quanquan Gu. An improved analysis of training over wee-pparametrrized deep neural networks. NuerlPS, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
13

---

(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "pxuqW08A_sl", "reviewer": "Reviewer_RiM6", "summary": "The authors propose a nonlinear sufficient dimension reduction method to extract key information in high-dimensional data. They proposed a stochastic neural network and develop an adaptive stochastic gradient Markov chain Monte Carlo algorithm with a convergence theorem established. Extensive empirical experiments on benchmark and real datasets are run to showcase the good prediction performance of the algorithm. \n", "questions": "1. How is the author's model different from [1] Sufficient Dimension Reduction with Deep Neural Networks for Phenotype Prediction by Siqi Liang, Wei-Heng Huang, Faming Liang. Proceedings of the 3rd International Conference on Statistics: Theory and Applications (ICSTA'21)\nI am aware that in [1], a divide-and-conquer DNN is used, except this, is there any difference in the model part? To me, the model in the current paper seems not novel compared with the model in [1]. \n\n2.Any empirical results to show the computational cost for the author's algorithm and other competing algorithms? It seems to me that the stochastic gradience HMC could be slow for large-scale dnn task even if a mini-batch is used. \n\n3. How is the author's approach compared with [1] as they are so similar? Is there an apparent advantage for the author's method compared with [1]. Why [1] is not cited or compared as a competing algorithm?", "limitations": "The authors do not discuss the limitations of their work. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors develop an adaptive stochastic gradient MCMC algorithm for training the StoNet and provide a rigorous study for its convergence under mild conditions. \n2. The authors have provided extensive empirical results to demonstrate the advantage of their method.\nWeaknesses:\n1. The model for this model seems not novel and I do not find a clear difference between the author's model and the model in the following paper:\n[1] Sufficient Dimension Reduction with Deep Neural Networks for Phenotype Prediction by Siqi Liang, Wei-Heng Huang, Faming Liang. Proceedings of the 3rd International Conference on Statistics: Theory and Applications (ICSTA'21)\n2. Computational complexity or empirical computational cost analysis and results are not provided so I do not have a clue if the author's algorithm will be much slower than state-of-the-art algorithms in terms of inference.\n3. To me, the closest model to the author's work is [1] but no comparison is provided with [1]. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ws-uGbtLXZm", "reviewer": "Reviewer_yby7", "summary": "This paper addresses nonlinear sufficient dimension reduction problems using a novel neural network architecture. The key idea is a layer-wise Markovian assumption which ensures the latent representation is identifiable---an issue associated with several related approaches. The authors further provided a stochastic gradient MCMC learning algorithm as well as theoretical analysis of convergence.\n", "questions": "See the main comments.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n1. The paper is clearly written. The idea of the layer-wise Markovian assumption is interesting and natural in terms of the SDR assumption.\n2.  Theoretical analysis is provided which shows the proposed StoNet has asymptotically the same loss function as the desired DNN when the training sample size becomes large.\n3. Numerical results demonstrate the efficacy of the proposed algorithm in classification and regression.\n\n[Weaknesses]\n1. It is not clear how are the depth h and layer width decided in the decomposition Eq (4). How to avoid learning trivial relations such as identity in the components?\n2. The StoNet could be easily overparameterized where noise is also captured. Thus, the learned representation could be more than sufficient. Some form of regularization could be needed.\n3. The misclassification rate for GSAVE and GSIR seems unrealistically high compared to the linear methods. Is there intuition why this is the case?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3oLpsum7su", "reviewer": "Reviewer_EGCt", "summary": "This work contributes a few interesting ideas for deep sufficient dimension reduction (SDR). Authors developed a new stochastic neural network and proved that the output of last hidden layer satisfies the SDR condition. Authors also discussed carefully the optimization. ", "questions": "please see above about weakness", "limitations": "It seems authors did not discuss the potential negative social impact.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Using stochastic neural network for SDR is new to me, which is also different from existing deep SDR methods. \n\nMy concerns mainly come from the evaluation section. Authors discussed a few issues of exisiting deep SDR methods ([2,20]). However, there is no comparision to any deep methods. Moreover, it is also a little unclear to me how [2] or [20] suffers from these limitations, and how the new method avoids such issue. Could you also empirically valiate this motivation? Additionally, it would be better if authors can compare with kernel dimension reduction. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept