Factor Normalization for Deep Neural Network Models
Authors: Haobo Qi
Jing Zhou
Hansheng Wang
Venue:  (2021)
Keywords: factor normalization
ultrahigh dimensional features
adaptive learning rate
factor decomposition

Abstract: Deep neural network (DNN) models often involve features of high dimensions. In most cases, the high-dimensional features can be decomposed into two parts. The first part is a low-dimensional factor. The second part is the residual feature, with much-reduced variability and inter-feature correlation. This leads to a number of interesting theoretical findings for deep neural network training. Accordingly, we are inspired to develop a new factor normalization method for better performance. The proposed method leads to a new deep learning model with two important features. First, it allows factor related feature extraction. Second, it allows adaptive learning rates for factors and residuals, respectively. This leads to fast convergence speed on both training and validation datsets. A number of empirical experiments are presented to demonstrate its superior performance. The code is available at https://github.com/HazardNeo4869/FactorNormalization

Under review as a conference paper at ICLR 2021
FACTOR NORMALIZATION FOR DEEP NEURAL NETWORK MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural network (DNN) models often involve features of high dimensions. In most cases, the dish-simensional features can be decomposed into two parts. The first part is low- www-dimensional factor. The second part is the residual feature, with much-reduced variability and inter-feature correlation. This leads to a number of interesting theoretical findings for deep neural network training. Accordingly, we are inspired to develop new factor normalization method for better performance. The proposed method leads new deep learning model with two important fea- tures. First, it allows factor related feature extraction. Second, it allows adaptive learning rates for factors and residuals, respectively. This leads to fast convergence speed on both training and validation datsets. A number of empirical experiments are presented to demonstrate its superior performance. The code is available nttps:/ /ttp::///qttutb com Htr//Haaaaarao http:////ce.aaeeeee////////aoorr
ization
INTRODUCTION
In recent decades, the progress deep learning, together with advances in GPU devices, has led to a growing popularity of deep neural network (DNN) models in both academia and industry. DNN models have been widely used in various fields, such as image classification (Simonyan & Zissermant 2014; He all 2016a), speech recognition (Hinton all 2012; MMas et etlll 2017). and machine translation (Wu et all 2016; Vaswani et all 2017). However, due to their deep structure, most DNN models are extremely difficult train. The practical training of DNN model often highly depends on empirical experience and is extremely time consuming. Therefore, series of effective optimization methods have been developed for fast DNN training.
According to recent survey paper by Sun et all (2019, most of the optimization methods with explicit derivatives can be roughly categorized into two groups: the first-order optimization methods and the high-order optimization methods. The widely used stochastic gradient descent (SGD) algorithm and its variants (Robbins & Monra, [1951; Lain et all 20TT) are typical examples of the first-order optimization methods. The SGD algorithm only computes the first-order derivatives (i.e., the gradient) using randomly sampled batch. By doing so, the SGD algorithm handle large-sized datasets with limited computational resources. Unfortunately, the practical feasibility of SGD comes at the cost of sublinear convergence speed (Johnson Zhangg (01KE). For better convergence speed, various accelerated SGD algorithms have been developed. For instance, the popularly used momentum method (Polyak, 1964; Qian, 1999) and the Nesterov Accelerated Gradient Descent (NAG) (Nesterov, 198% Sutskever et all 2013 method. Both of them took the information from the previous update gradient direction into consideration. Further improvements include AdaGrad (Duchi et all 201I), AdaDelta (Zeiee,, 2012), RMSprop (Tieleman & Hinton, 2012), Adam (Kingma Ba, 2014) and others. For a more stable gradient estimation, the stochastic average gradient (SAG) (Roux et all, (0)) and stochastic variance reduction gradient (SVRG) (Johnson & Zhang, 30 methods are also developed.
Except for the first-order optimization methods, high-order optimization methods also exist. Popular representatives the Newton's method and its variants (Shannd. 1970 atl 2019 Pajarnen et al, 2019). Compared to the first-order methods, high-order methods might lead to faster convergence speed since they take the information of Hessian matrix into consideration.

---

Under review as a conference paper at ICLR 2021
For example, the Newton's method can have quadratic convergence speed under appropriate conditions (Ayricl, 2003). However, calculating and storing the Hessian matrix and its inverse is extremely expensive in terms of both time and storage. This leads to the development of some approximation methods, such as Quasi-Newton method (Avriill, 2000) and stochastic Quasi-Newton method (Luo st 2014). The idea of Quasi stochastic Quasi Newton method is to approximate the inverse Hessian matrix by a positive definite matrix. For example, DFP (Eletcher & Powell, 1963: Davidon, 1991). BFGS (Broyden, 1970; Fletcher & R. 1970; Donald Goldfarti (970)) and L-BFGS (Nocedal & Jorge, 1980: Lm & Nocedal, (989)) methods are popular representatives. Moreover, as a useful technique for fast convergence, various pre-conditioning techniques are also popularly used. (Hnckl,, [1999 Benzil 2002; [Tang et all 2009). The basis idea of pre-conditioning to transform a difficult or ill-conditioned linear system (e-g. AO = b) into an easier system with better condition (Fligham & Mary, 2019). As consequence, the information contained in the feature covariance can be effectively used (Wang et all 20T2). Other interesting methods trying to extract useful information from the feature covariance also exist; see for example Denton et all (20155) Ghiaas & Fowlkes (2016), and Lai at (2017).
However, to our best knowledge, there seems no existing models or methods that are particularly designed for high-dimensional features with a factor structure. In the meanwhile, ample amounts of empirical experience suggest that most high-dimensional features demonstrate a strong factor type of covariance structure. In other words, a significant amount of the feature variability can be explained by a latent factor with very low dimensions. As consequence, we can decompose the original features into two parts. The first is a low dimensional factor part, which accounts for a significant portion of the total volatility. The second is the residual part with factor effects removed. This residual part has the same dimension as the original feature. Consequently, it has much reduced variability. Moreover, the inter-feature correlation is also reduced substantially. To this end, the original learning problem concerning for the high dimensional features can be decomposed into two sub learning problems. The first one is learning problem concerning for the latent factor. This is relatively simple since the dimension of the factor is very low. The second problem is related to the residual feature. Unfortunately, this is still a challenging problem due to the high dimensions. However, compared with the original one, it is much easier because the inter feature dependence has been substantially reduced. For a practical implementation, here a novel method called factor normalization. It starts with a benchmark model (e.g.. VGG or ResNet) and then slightly modifies the benchmark model into a new model structure. Compared with the benchmark model, the new model takes the latent factor and residuals as different inputs. The benchmark model is remained the model in the last layer. This is to compensate for the information loss due to factor extraction. By doing so, the new model allows the factor-related features and residual- processed separately. Furthermore, different (i.e., adaptive) learning rates can be allowed for factor and residuals, respectively. This leads to adaptive learning and thus fast convergence speed.
propose
process the residuals. The latent factor is then put back to
features to be
The rest this article is organized as follows. Section 2 develops our theoretical motivation with statistical insights. Section 3 provides the details of the proposed new model. Section 4 demonstrates the outstanding performance of the propose model via extensive empirical experiments. Section concludes the article with a brief discussion for future research.
2 THEORETICAL MOTIVATION
To motivate our new model, we provide here a number of interesting theoretical motivations from different perspectives. Since the SGD algorithm a stochastic version the GD algorithm, we thus focus on a standard GD algorithm in this section for discussion simplicity.
2.1 THE GD ALGORITHM
Let (X,, Y.) be the observation collected from the i-th instance with < < N, where Y is often the class label and X; = (X.,..., xip) E TRP is the associated p-dimensional feature. The loss function evaluated at i is defined as {(Y,, x. TT,) where 0 E R is the unknown parameter. Then, the global loss is given by (0) N-1 ΣN E(v,, XTO)). 0). The global gradient is given by
2

---

Under review as a conference paper at ICLR 2021
0)x, and E(y,2) = ae(y,))) z)/Oz. Let 6(1) be the estimator obtained in the t-th iteration. Then, the GD algorithm updates the parameter as ((t+1) ((t) -axx (B()).
(N( (N)0)-N-E'
Here, a is scalar and is referred as the learning rate (Rohbmn & Monrd, 1951). Assume that
(x())) reaches its minimum I such that (x(0)= O. We then apply Taylor expansion for (0) at 0. This leads to
((++1) -
[ 0) + +o(10(0) - -
K(Ō 0) + o(1 013),
where IK 1 We refer to K as a contraction operator, and it plays very important role in optimization. Intuitively, all the eigenvalues of K. should lie in (-1,1). Otherwise, the algorithm might not converge numerically.
Proposition 1 Assume (0) to be a positive definite matrix. Let X 2 X2 2 2 Ap > 0 be the eigenvalues of Ex To have the GD algorithum converge, should have 0 1/21-
2.2 CONDITION NUMBER
By Proposition II we know that the learning rate cannot be too large. Otherwise the GD algorithm might not numerically converge. The size of the learning rate is controlled by the largest eigenvalue of the Hessian matrix (0)) The larger A is, the smaller the learning rate must be, and the slower convergence speed should be. This problem is particularly serious if the condition number (i.e., (X)) of the Hessian matrix very large. In that case, the large A₁ value forces the learning rate a to be very small. In the meanwhile, other small eigenvalues (Λ₃ for j & 1) make the convergence speed along the corresponding digen-direction very slow. Consequently, practitioners should wish the condition number of the Hessian matrix to be as small as possible. On the other hand, as other
we mentioned previously, most high-dimensional features have a strong factor structure. words, the size of the top eigenvalues of the covariance matrix Σ is typically much larger than the rest. Consequently, the condition number of its covariance matrix typically very large. It is then of great interest to investigate: how would this factor structure affect the condition number of the Hessian matrix?
To address this important problem, we evaluate the expected Hessian matrix as H = E(Ō(O)) = F(E(Y, x, X)X,X where Ë(y, =) = ae(y, =)/02 stands for the second order derivative of 2) with respect z. For illustration purpose, assume X; is normally distributed with mean Recall that the covariance matrix is E. Define X, =x= E-12 and El/20. We then rewrite H as H /1/2 ((((Y, X, X),,,X,))'' Let A be an arbitrary positive definite matrix. Define Amax and > (A) be the maximum and minimum eigenvalues of A, respectively. We then have Amax
///2Hr1/2 where H E( X,T X,,,)
Amax (max (H) and Amin (IHI) < Amin Amxx (lil)..)). This further suggests that the condition number of Hi (i.e.. con(H)= = Amax (H)/Amin (H)) satisfies the following inequality,
con cHn(H) 2 con(E)/ (H),
(I)
where con(A) stands for the condition number of an arbitrary positive definite matrix. Note that H is the expected Hessian matrix under an extremely ideal situation, where the input feature follows a standard multivariate normal distribution. This arguable the most ideal situation for numerical optimization. We thus can reasonably expect that con(HH (HH) should not be very large in this case. Thus, by (I)) know that con(E) should play important role in determining the con(H). In other words, the large condition number of Σ will affect con (H)) This makes the numerical optimization by standard GD algorithm extremely difficult, and thus calls for novel solution.
2.3 THE FACTOR LINEAR SUBSPACE
To further motivate the proposed model, we provide some theoretical justification from a dif- ferent perspective. Assume a standard factor model as X; BZg + &,, where Z, E IRd is
3

---

Under review as a conference paper at ICLR 2021
vector with low dimensionality (i.e., d < p) and B E Roxd is the corresponding factor loading matrix with < p. Consider the general loss function [(0). Recall that the glob- al gradient is (0) = N - (Y, 0)X, Then, we have (x(() = Q1 + Q2, where Q1 = BNN-N-I E E(YY XT 0)2, S(B). Q2 N -N-EE N ((Y,, X, 0) and S(B) represents the linear subspace spanned by the column vectors of B. Then, the covariance structure of the global gradient can be written as cov(LN(0)
cov(Q1) + cov(Q2). where cov (Q1) BE,B /N, E cov cov [L(Y, X 0) 7 E Rdddd, cov(Q2) Ec/N, and Eg = cov cov(2((, X 0)e. E Raxd. It then of interest to study the relative sizes of cov(Q1) and cov(Q2) under appropriate metrics.
Proposition 2 Assume there exists constant 0 Tmin Tiax oc, such that Timl (mi))
Amax ( < Tmmx and Tmin < Amin (2gg) I Aymax (2s) < Tmax- Furthermore, assume that
Amin B B/p) > Tmin- We then have (cov((11)) tr (ncov(Q22)
Z m
For most DNN models, the parameter dimension is usually very high. This makes the model structure sufficiently flexible. Accordingly, one might expect the associated GD (or SGD) algorithm should search the entire high dimensional parameter space (c.g., 1s.,1K IR") in very flexible way. However, the above proposition indicates that the estimated gradient direction is not flexible we might expect. In contrast, it always has a significant portion (i.e., Q1) trapped in very low-dimensional linear subspace S(B). This brings a positive effect. Since the linear subspace S(B) is of a very low dimension (i.e., d), the overfitting effect due to Q1 becomes negligible. However, this positive effect also comes at a convergence cost. By trapping a significant portion of the gradient direction in SB),, the capability of GD algorithm to explore directions other than S(B) is considerably
sacrificed. This problem can be solved the directions due to the factor (i... Q1) and directions
due to the residual features i.e. Q2) can be treated separately. This is another important theoretical insight that drives us to develop our new model.
THE PROPOSED NEW METHOD
Inspired by the theoretical findings discussed in the previous subsection, we propose here factor normalization method. The new method is particularly designed for deep models with high-dimensional features and factor structures. starts with an important benchmark model (e.g.. VGG or ResNet) and can be implemented by the following three important steps. They are factor decomposition, model reconstruction, and adaptive learning, respectively. The details are discussed in the following subsections.
3.1 FACTOR DECOMPOSITION
In the first step, conduct a standard singular value decomposition for the high-dimensional features. By doing so, a low-dimensional factor can be estimated. Accordingly, we can decom- pose the original features into two parts. The first part is due to the factor and is referred as the factor part. The second part is the original features but has factor effects removed. This part is referred as the residual part (or residual feature). Specifically, the input feature matrix is written as X (X1.., XN) E R Nx with n = min(N, p). Next, we centralize X by its column so that the column mean O. We then conduct singular value decomposition as X == UAV = Ei= Aduro where U = (ui..., un) E NRx and V (vI..., U.)) E RRxn with UU--VT = In and In E gux= stands for an identity matrix. Moreover, A = diag(A)...,,,, a diagonal matrix with A Z X2 2 Z An 2 0. Define Va = (Vi..., We then estimate the latent factor by 2 = XVa, where á E Nxd represents the estimated factor part. Last, we use linear regression to calculate the residual feature matrix as follows & = - - 2.: where ò = (2 2) TT x E dd is the estimated factor loading matrix. By doing so, the original feature matrix X is decomposed into two parts: the factor part Ż and the residual part غ.
3.2 MODEL RECONSTRUCTION
In the second step. we modify a given benchmark model (c.g., a ResNet) into a new one for better performance. More specifically, consider standard DNN model with (1) an input layer with
4

---

Under review as a conference paper at ICLR 2021
Input Feature
Residual Feature
Factor
Input Layer
Input Layer
Model Reconstruction
Danse Layer
Dense Layer
Output
Output
Figure 1: A graphic illustration of MODEL RECONSTRUCTION for factor normalization, The rect- angles represent the feature maps in DNN. The blue dashed rectangle represents the latent layers between the input and the last dense layer. The left panel shows an original DNN structure with the original input feature. The right panel shows modified DNN structure with factor and residual features. As one can see, the factor and residual feature are treated separately. The latent factor is reinserted back into the model before the last dense layer for output.
original feature X: (2) output layer, which is often of dense type structure, and (3) sophisticated latent layers between the imput and output layers. We then replace the original feature of the input layer (i.e., X) with the residual feature (i.e., E). Next, let the residual feature be fully processed by the sophisticated latent layers until the output layer. Before the DNN model constructs the last dense layer for output, we put the estimated latent factor (i.e., 2) back to the output layer to compensate the information loss due to factor extraction.
This leads to new model structure with two interesting characteristics. The high-dimensional resid- ual term still processed by the sophisticated benchmark model. However, the low-dimensional latent factor is not much processed. This interesting structure is particularly desirable for the fol- lowing reasons. First, for most cases, we find one dimensional factor seems already good enough. In this case, the latent factor is univariate. Then, its any nonlinear transformation (e.g., by deep neural network) should make little difference from the original factor, in terms of the information contained. Thus, there little need to further conduct sophisticated nonlinear transformation for the univariate latent factor. In contrast, the residual form remains to be high dimensional. Thus, sophisticated nonlinear transformation still very useful for excellent feature extraction. Second, we want to use different learning rates for factor and residuals for adaptive learning and thus better performance. If the factor put in some intermediated layer, the practical implementation of adap- tive learning by TensorFlow programming could be extremely and unnecessarily complicated. This explains why the latent factor set as the input of the last layer. Figure MI gives more intuitive explanation.
3.3 ADAPTIVE LEARNING
As we mentioned before, with this modified model, the convergence speed is still very slow if we adopt a standard SGD algorithm. For a reliable convergence, a small learning rate should be used for the parameters associated with the factor because factor variability is large. In contrast, much larger learning rates should used for the parameters associated with the residual features, otherwise the convergence speed could be very slow due its small variability. This suggests
5

---

Under review as a conference paper at ICLR 2021
Loss reduction on training dataset
Prediction accuracy on validation dataset
0.375
FN (d=1)
0.0275
(d+2)
0.350
LR FN (d=10)
0.9250
0.325
Accuracy
0.9225
0.300
0.275
0.9175
0.250
0.9150
0.225
0.9125
FN (d=2)
IN (d=10)
0.200
100
Epochs
Epachs
Figure 2: LR stands for logistic regression and FN stands for factor normalization. The left panel shows the training loss on MNIST images of the logistic regression model and FN models optimized by SGD. The right panel shows the prediction accuracy the validation dataset. Different numbers of factors are tested (i.e.,d = 1,,,, 10).
that adaptive (i.c., different) learning rates should be used for the factor and residual. Specifically, assume the learning rate used in standard SGD is a. We then the learning rate for the factor associated learning rate to be a; a / X for j = L...,d. d. Recall that A, is the jth largest singular value of X (column centralized). In contrast, the learning rates used for the residual associated learning rates are set to be the same as
/ / The consequence is that the learning rates should be
used for latent factor become much smaller than those of the residual features. noted that the adaptive learning rates used here are different from those used in AdaGrad or Adam Duchi of all (201II;; Kingma & BE (2 The AdaGrad and Adam methods adjust their learning rates dynamically for each iteration and adaptively for each dimension in the training process.
EXPERIMENTS
To empirically demonstrate the proposed FN model, we conducted various experiments on different models, including logistic regression, multilayer fully connected neural networks and deep convolu- tional neural networks. All the results are based on optimal settings of initial learning rate using grid search and 1/ 1/vl decay rate. The experiments were run on a Tesla K80 GPU with 1 11GB memory.
4.1 LOGISTIC REGRESSION
We start with simple logistic regression model and evaluate the proposed model the MNIST dataset. This is a classification task with 10 classes, 60,000 instances for training, and 10,000 instances for validation. The input feature is 28 x 28 pixel matrix. To implement the logistic regression, we reshape the input feature into a 784 dimension vector. Subsequently, the FN models with various factor dimensions (d 1, 10) are used reconstruct the baseline model. The SGD optimizer is used to optimize both the baseline model (i.e., LR) and FN models. The detailed results are given in Figure For comparison purposes, all the models are trained for a large number of epochs so that its performance on a validation data are fully converged. Specifically, batch size of 200 with 200 epochs are adopted. The left panel of Figure reports the training loss for various models. We find that the FN models consistently outperform their benchmark counterparts, since the loss curves of the FN models are always below the baseline model. The right panel of Figure X displays the results of accuracy. We can see the accuracy curves of FN models reach the plateau earlier than the baseline models. Additionally. it seems that there is little difference in the performance of FN models with different factor dimension d. We are then inspired to fix d 1 for the subsequent experiments.

---

Under review as a conference paper at ICLR 2021
4.2 MULTILAYER NEURAL NETWORKS
We next consider a more complicated multilayer neural network model. The proposed model has two fully connected hidden layers with 1,000 neurons. The Rectified Linear Unit (ReLU) transformation is used in each hidden layer for activation (Kingma & Ba, 2012). The settings of the experiment are almost the same those in subsection 4.1 except for the following differences. First, we only consider the FN model with d = 1. Second, for comprehensive comparison purpose, we consider four different optimizers, they are SGD, NAG, Adagrad and Adam, respectively. The detailed results are given in Figure E The top panel of Figure R reports the training loss in log scale for the four
Adam
MLP
HLP
FNN
MLP
MLP
-2
-2
1.00
150
240
100
15.0
200
3.80
Accuney
6.97
5.75
3.900
0.140
0.077
3.000
1.77%
0.900
MLP
RU*
MUP
0.300
MLP
3.500
150
240
200
Epochs
Ecuuto
Esochs
Figure 3: The top panel is the training loss in log scale for baseline model (blue dash line) and FN model (red solid line) with four different optimizers. The bottom panel is the validation accuracy for baseline model (blue dash line) and FN model (red solid line) with different optimizers.
optimization methods on the baseline model and FN model. We find that for each optimizer, the FN model always achieves smaller training loss than the baseline model. This difference is more obvious the cases of SGD, NAG and Adagrad. The bottom panel of Figure & reports the prediction accuracy on validation dataset. We find that for each optimizer, the accuracy of FN model is higher than that of the baseline model.
4.3 CONVOLUTIONAL NEURAL NETWORKS
Convolutional neural networks (CNNs) are the most popular models used in image classification and object detection tasks. The main difference between CNNs and multilayer neural networks that CNNs contain many convolutional and pooling layers, which provide efficient feature traction and lead to excellent prediction performance. However, CNNs usually have complicated model structures with a large number of parameters. Here, we consider two classical CNN models. They are AlexNet (Krizhevsky et all, 2012) and ResNet50 (He 20166) The dataset used here CIFARIO (Knizhessky & Hinton, (009) and CatDog released by Kaggle in 2013. For illustra- tion purpose, the results of AlexNet on CIFAR10 are summarized in Figure EL while the results of ResNet50 on CatDog are given in Figure & Similar the experiment in subsection 4.2, total of four baseline optimization methods are also evaluated. They are SGD (Robbins & Monro, 1995), NAG (Bengio et all 2013), AdaGrad (Duchi et all, 20LL) and Adam (Kingma & Ba, 2014).
The left panel of Figure ! reports the time consumption (includes the time of conducting SVD) required by different optimization methods (i.e., SGD, NAG, AdaGrad and Adam) to reach the optimal prediction accuracy for baseline model and FN model. The optimal prediction accuracy is set as the accuracy that is 1 1% lower than the optimal accuracy of baseline models. We find that for all the four optimization algorithms (i.e.. SGD, NAG, AdaGrad and Adam), the time is substantially reduced for EN models. The reduction is significant in the cases of SGD and Adam. The right panel of Figure ! compares training loss in log scale. For fair comparison, 250 epochs given
7

---

Under review as a conference paper at ICLR 2021
consumption for optimal baseline acc -1%
Training epachs
6,0
Alexhet
Aexhitt
-0.5
Time (minutco)
-1.0
topiosss
Alediet
Alwodiut
MAG
Adam
SGD
MAG
Adagrad
Figure 4: Results of AlexNet on CIFARIO dataset. benchmark accuracy set as the accuracy that is 1% lower than the optimal accuracy of baseline model. The left panel shows the time cost for each optimization algorithm to obtain the benchmark accuracy between baseline model and FN model. The right panel shows the training loss in log scale after 200 epochs for the four optimization algorithms between baseline model and FN model.
Time consumption for optimal baseline -1%
Training loss after 200 epochs
0.0
1400
Restiet50
1200
1000
Time (mirutco
Lozs
800
600
400
-25
200
-3.0
ResNet50
ResNet50 FN
5GD
NAG
Adagrad
Adam
SGD
NAG
Adagrad
Adam
Figure 5: Results of ResNet50 on the CatDog dataset. A benchmark accuracy is set as the accuracy that is 1% lower than the optimal accuracy of baseline model. The left panel shows the time cost for each optimization algorithm obtain the benchmark accuracy between baseline model and FN model. The right panel shows the training loss in log scale after 200 epochs for the four optimization algorithms between baseline model and FN model.
for different optimizers. We find that the training loss can be relative lower for the baseline model the end of long time training. Overall, we find that the FN model can significantly improve the performance of a bascline model in terms of time cost, training loss, and prediction accuracy. The results of ResNet50 given Figure E qualitatively similar.
CONCLUSION
In this paper, we propose a novel factor normalization method for fast deep neural network training. The idea inspired by the fact that many DNN models involve high dimensional features and these features are often of a strong factor structure. The proposed method has three key components. First, it decomposes high dimensional input feature into two parts. One is the factor part with low
8

---

Under review as a conference paper at ICLR 2021
dimensionalityy, and the other is the residual part. Second, it modifies given DNN model slightly so that the effect of the latent factor and residual feature can processed separately. Last, to train a modified DNN model, new SGD algorithm is developed. It allows adaptive learning rates for the factor part and the residual part.
To conclude this article, present here number of interesting topics for future study. First, current FN model conducts factor decomposition on the input feature. fact, even if different components of the input feature are completely independent of each other, the resulting feature map after a number of convolutional layers might still have a strong factor structure. In this case, whether factor decomposition should be conducted for the feature maps is problem worth studying. Second, the current FN model does not insert the estimated factors back into the DNN models until the last layer. What would happen if the estimated factors were also inserted into earlier layers? This another problem of great interest.
ACKNOWLEDGMENTS
This research is supported by the National Natural Science Foundation of China (No.71702185, No.71873137, N9.11971504, No.71532001, 1555101,,,, No.71332006), Beijing Munici- pal Social Science Foundation (No.19GLC052), National Statistical Science Research Project (No.2020LZ38), Ministry of Education Focus on Humanities and Social Science Research Base (Major Research Plan 17JJD910001) and China's 's National Key Research Special Program (No. 2016YFC0207704), Consulting Research Project Chinese Academy of Engineering(2020-XY 30).
REFERENCES
M. Avriel. Nonlinear Programming: Analysis and Methods. Dover Publications, 2003.
Yoshua Bengio, Nicolas Boullaggr-Lewaaddowski and Razvan Pascanu. Advances in optimizing recurrent networks. In IEEE International Conference on Acoustics, 2013.
Michele Benzi. Preconditioning techniques for large linear systems: A survey. Journal of Compu- tational Physics, 182(2):418 477, 2002.
C. G. Broyden. The convergence of class of double-rank minimization algorithms: 2. the new algorithm. Ima J Appl Math, 1970.
William C. Davidon. Variable metric method for minimization. SIAM Journal on Optimization, 1991.
Emily L Denton, Soumith Chintala, arthur szlam, and Rob Fergus. Deep generative image models using a laplacian pyramid of adversarial networks. In Advances in Neural Information Processing Systems, volume 28, pp. 1486-1494. Curran Associates, Inc., 2015.
Donald and Goldfarb. A family variable-metre methods derived by variational means. Math Comp, 1970.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(7):257-269, 2011.
Fletcher and R. A new approach to variable metric algorithms. Computer J. 13(3):317-322, 1970.
R. Fletcher and M. J. D. Powell. A rapidly convergent descent method for minimization. Computer
J, (2):163-168, 1963.
Golnaz Ghiasi and Charless Fowlkes. Laplacian reconstruction and refinement for semantic seg- mentation. 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Sun Jian. Deep residual learning for image recog- nition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2016b.

---

Under review as a conference paper at ICLR 2021
Nicholas J. Higham and Theo Mary. A preconditioner that exploits low-rank approximations to factorization error. SIAM Journal on entific Compuring, 41(1):A59 2019.
G. Hinton, L. Deng, D. Yu, G. E. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, and T. N. Sainath. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82- 97, 2012.
Jiang Hu, Bo Jiang, Lin Lin, Zaiwenl Wen, and Ya-Xiang Yuan. Structured quasi-newton methods for optimization with orthogonality constraints. SIAM Journal on Scientific Computing, 41:2239- 2269, 2019.
Thomas Huckle. Approximate sparsity patterns for the inverse of a matrix and preconditioning. Applied Numerical Mathematics, 30(2-3):291-303, 1999.
P. Jain, P. Netrapali,, S. M. Kakade, R. Kidambi, and A. Sidford. Parallelizing stochastic gradi- ent descent for least squares regression: Mini-batching, averaging, and model misspecification. Journal of Machine Learning Research, 18:1-42, 2018.
R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduc- tion. News in physiological ences, 1(3):11-332) 2013a.
201.3.
R. Johnson and T. Zhang. Accelerating stochastic gradient descent using predictive variance reduc- tion. News in physiological ences, 13)):315-323, 20133......
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Computer ence, 2014.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Computer Science Department, University of Toronto, Tech. Rep, 1, 01 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. pp. 1097-1105, 2012.
Wei Sheng Lai, Jia Bin Huang, Narendra Ahuja, and Ming Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 5835-5843, 2017.
Dong C. Liu and Jorge Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45(1):503-528, 1989.
Haipeng Luo, Patrick Haffner, and Jean Francois Paiement. Accelerated parallel optimization meth- ods for large scale machine learning. Computer ence, 2014.
Andrew L. Maas, Peng Qi. Ziang Xie, Awni Y. Hannun, Christopher T. Lengerich, Daniel Juraf- sky, and Andrew Y. Ng. Building dnn acoustic models for large vocabulary speech recognition. Computer Speech and Language, 41:195-213, 2017.
Y. E. Nesterov. A method for solving the convex programming problem with convergence rate o(1/k?). Dokl. aokllakad.nauk Sssr, 269, 1983.
Nocedal and Jorge. Updating quasi-newton matrices with limited storage. Mathematics of Compu- fation, (5(15)):773-773, 1980.
Joni Pajarinen, Hong Thai, Riad Akrour, Jan Peters, and Gerhard Neumann. Compatible natural gradient policy search, 2019.
B. T. Polyak. Some methods of speeding up the convergence of iteration methods. Ussr Computa- tional Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Netw, 12(1): 145-151, 1999.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemat- ical Statistics, 22(3):400-40) 1951.
10

---

Under review as a conference paper at ICLR 2021
Nicolas Le Roux, Mark Schmidt, and Francis Bach. A stochastic gradient method with exponen- tial convergence rate for finite training sets. In International Conference on Neural Information Processing Systems, 2012.
D. F. Shanno. 1970.
Conditioning of quasi-newton methods for function minimization. Math. Comp.,
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv 1409. 1556, 09 2014.
Shiliang Sun, Zehui Cao, Han Zhu, and Jing Zhao. A survey of optimization methods from machine learning perspective. IEEE Transactions on Cyhernetics, 2019.
I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. 30th International Conference on Machine Learning, ICML 2013, pp. 1139- 1147, 01 2013.
M. Tang. Nabben, C. Vuik, and Y. A. Erlangga. Comparison of two-level preconditioners de- rived from deflation, domain decomposition and multigrid methods. Journal of entific Computing, 39(3):340-370 2009.
Tijmen Tieleman and Geoffrey Hinton. Lecture 6e rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Yuyang Wang, Alex Smola, Danielle C. Maddix, Jan Gasthaus, Dean Foster, and Tim Januschowski. Deep factors for forecasting, 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arX- iv:1609.08144 08144, 2016.
Matthew D. Zeiler. Adadelta: An adaptive learning rate method. Computer ence, 2012.


Reviews:
[{"review_id": "CYJQSkshC2", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a training scheme based on decomposing input features into two parts which have different training dynamics: a low rank \"factor feature\" computed using PCA on the raw features, and a high rank \"residual\".  The former is processed  by a very shallow network, while the latter passes through the full network, and parameters for each are updated with different learning rates.  Experiments show that the proposed algorithm can speed up wall-time to  a certain accuracy on MNIST and CIFAR10 classification, across several neural network architectures and optimizers.\n\nPros:\n- Simple and straightforward proposal, easy to understand, and seems to lead to improved accuracy/reduced loss early in training.\n- Experiments cover several datasets and architectures spanning very simple to very deep, which is nice to see.\n\nConcerns:\n\n- Very \"mathy\" presentation in Sec. 2 is very dense and difficult to follow, and only seems to motivate the method in the special cases of a very shallow linear regression model, where the gradient is easily related to eigenvalues of the input features $X$ (and assuming a \"strong factor structure\", which has been shown to be reasonable for natural images.)  It's not clear why these results would necessarily generalize to very deep and highly nonlinear networks such as AlexNet.  Perhaps this is the motivation for only processing \"factor features\" with an extremely shallow (albeit still nonlinear) network? If so, this should be more clearly explained in the text.\n\n- The experiments are not very convincing. \n - The baselines trained with vanilla SGD or other optimized (Adam, etc.) are not clearly explained.  Do they use the same feature factorization as SGD+FN or do they consist of only the raw features passed into the full network (i.e. the left side of Figure 1)?  If the latter, the comparisons are not totally fair as the input features and networks differ.\n - Models are not trained until convergence, but only for a fixed amount of time, or until a fixed (but not very high) accuracy is reached.  How does the proposed scheme affect final accuracy/loss at convergence?  (E.g., I'd expect simple logistic regression on MNIST to achieve over 95% accuracy)   Is it strictly a training speedup, while converging to the same performance as baselines?  Or is it only suitable when operating within a limit training time budget?\n  - This issue is illustrated in the training curves in Fig 3 (MLP on MNIST), where the validation accuracy for the proposed SGD+FN appears to be plateauing faster than the baseline SGD curves.\n - Furthermore, the speedup over NAG or Adam on CIFAR10 using  ResNet50 isn't very large. \n - All training comparisons are measured only as a function of wall-time.  But the training hardware is not explained.\n - Missing obvious baselines of training separate models on either the factor features or residual alone.  At least for MNIST I'd expect decent performance to be obtainable from the factor features alone.  Maybe the residual is mostly irrelevant to the task?\n\nOverall I feel that the paper is not ready for publication at this time since the experimental validation is incomplete and does not fully explore the benefits and potential downsides of the proposed method.\n\nOther comments:\n\n- Sec 3.3: \"Adaptive learning\".  If I'm understanding correctly, the learning rates are defined based on the eigenvalues of $X^T X$, but are then kept fixed.  So, unlike e.g. AdaGrad or Adam,  the algorithm is not really adaptive in the sense that the learning rate is kept constant throughout training.\n\n- Sec 3.3: \"a standard SGD algorithm cannot be used to train a DNN model\".   This seems like an unnecessarily strong statement given how commonly SGD is used for DNN training.\n\n- The linked code does not appear to include resnet experiments\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "A_OdbWQgcUu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this paper, a learning method that accelerates the training of DNN is proposed. Given an input X, the proposed method decomposes X as X = BZ + E where BZ is a low-rank approximation of X and E is the residual term. E is used as an input of DNN and Z is used as an additional feature of the input of the last layer. Experiments using MNIST and CIFAR10 show that the proposed method accelerates the speed to reduce the training loss. \n\n\nDetailed comments:\nThe observed phenomena --- the proposed method accelerates the learning speed of SGD --- is quite interesting. I haven't noticed existing studies reporting such findings. However, I feel \"why\" parts are not clearly explained in this paper. For example, I have the following questions.\n- The analysis in Section 2 is based on a shallow network model. How can we apply this analysis to a deep network model?\n- What kind of intuition is behind the architecture design (Fig 1)? Why the residual term should place as the input of the first layer and the main term as the input of the last layer, rather than e.g. some intermediate layer?\n\nAlso, I feel the experiments are not convincing enough. The main motivation of this paper is that \"the ultrahigh dimensional features can be decomposed into two parts\". However, the dimensions of MNIST and CIFAR10 are not quite ultrahigh (28^2=784 and 32^2=1024).  Experiments with more high-dimensional data such as ImageNet would be necessary to convince the research concept. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "5_kAUE_L0ak", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review of \"Factor normalization for DNNs\". \n\nThe paper makes an observation that datasets used for training many deep neural nets exhibit a strong factor structure, i.e. have a small number of dominant principal components explaining most of the variance.  If we were to remove the dominant factors, the residuals would have much weaker correlation structure and allow faster DNN convergence with SGD.   The paper proposes to separate the dominant factors and the residuals,  train the original DNN on residuals with a much faster SGD learning rate,  and then recombine with a shallow small NN learned on the dominant factors trained using its own (slower) learning rate. \n\nFirst the positive aspects:  this is an interesting idea, and I haven't seen it commonly used in practice for DNNs.  While mathematical analysis is conducted for linear models with GD and is fairly straightforward, it nicely illustrates the main issues, and the hope is that the linear intuition continues to apply to deep NNs.  Experimental results suggest that indeed convergence rate can be improved on several datasets.   In terms of criticisms,  there is very limited scholarship of related ideas that have been used both for linear models and for DNNs, in particular (a) various factor-based models that already exist,  (b) preconditioning of linear systems,  (c) neural nets trained on some other sort of residuals -- e.g. laplacian pyramid DNNs .  Another criticism -- is that there is no discussion of how to do large-scale PCA / factor analysis for high-dimensional image data arising in say modern DNN image classification pipelines, and its computational cost,  as simple numpy.linalg.svd won't work.  The paper also claims that strong factor structure (with a small number of dominant components) is prevalent in modern ultra-large scale DNN applications -- I would like to see some references / supporting evidence beyond just computer vision.  \n\nOverall in my opinion the paper needs to consider the context of related works, and has a few other correctable issues, but I would certainly encourage the authors to continue to improve it. \n\nAdditional details: \n\n1)  Prior work -- that should be cited / contrasted with your approach:   \n(a) Since a substantial part of the paper analyzes linear models,  it's important to mention that factor structure has been long exploited in various ML / stats works.  For example factor models, and principal component regression (PCR) attempt to focus the modeling power on the principal components. In other applications (e.g. in financial modeling) one can assume that factors components are less predictable and focus the modeling effort on the residuals. Recent work by Alex Smola et al,  \"Deep factors for forecasting\" has looked at the deep-NN instantiation of this idea.  Basic autoencoders or bottlenecks used in DNN architectures also attempt to capture the low-dimensional structure.  These are all different from what you're doing, but hinge on the same basic concept -- so providing a discussion of your work in context of related work would be important. \nb)  There is a long history of using preconditioners for gradient-based and other iterative solvers of linear systems.  In particular there are some low-rank preconditioners for accelerating convergence of linear systems:  Nicholas Higham, et. al, \"A new preconditioner that exploits low-rank approximations to factorization error\".  There is also work on applying preconditioners specifically for SGD, e.g. see works by Michael Mahoney. \nc)  For some domains a low-pass filter, or some other low-resolution model can serve to replace PCA or factor models. For example Laplacian pyramids have been widely used in image processing to separate the dominant modes from details.  There are existing DNN approaches based on laplacian pyramids: \ne.g. \"Deep laplacian pyramid networks for fast an accurate super-resolution\". \n\n2)  You mention that \"none of these optimization methods has considered the covariance structure\".. related to the Hessian.  There is a substantial effort to develop second-order methods for stochastic gradient descent, in particular at ICML 2020 there was a workshop on this topic.  \"Beyond first order methods in ML\".   http://www.wikicfp.com/cfp/servlet/event.showcfp?eventid=102387&copyownerid=156008\n\n3) In section 2.1. -- it's worth mentioning explicitly that you're specifically analyzing linear regression under gradient descent.  as you look at Loss(y, theta' X).   It's also worth explicitly mentioning that the condition number of the Hessian plays a crucial role in convergence rate of GD.   You talk about the top eigenvalue, where the bottom one in your example is fixed at 1,  but it's worth mentioning the condition number. \n\n4) Sec. 3.1. How do you conduct 'standard principal component analysis' for ultrahigh dimensional features.  This is a computationally tricky problem, \"standard\" methods won't work.   How do you decide on the dimension -- i..e. number of factors to keep? \n\n5) Is the \"time consumption\"  including the time to do PCA? \n\n6) Can you give some references claiming strong factor structure in several DNN applications in ultra-high dimensions?  I do not contest that this is the case -- but it would be useful to have supporting evidence.  What number/fraction of factors is typically required in these applications to capture a nontrivial fraction of variance?\n\n7) Sec. 2.1. Using lower-case kappa for a matrix is strange,  I initially assumed it's a scalar.  Maybe use another capital letter. \n\n8) While the paper is mostly pretty readable, there are various small issues with english language (from stylistic to grammar) use e.g. \"In fact ample amounts of empirical evidence\" --> \"ample empirical evidence\", e.t.c. There are typos in references, e.g.  Zeiler,   \"Computer ence, 2012\". What is that? \n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "-mCRwD6Kdem", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strong points**\n\nThe paper provides a very detailed theoretical analysis of motivation.\n\n**Weak points**\n\nAnalysis in section 2 is based on linear regression, but the proposed method is based on deep models.\n\nThe proposed method is more similar to feature extraction instead of a training method.\n\nMany models in the chart are not fully converged. It is important to compare convergence speed but also the final accuracy. I think they are not converged, because the training and testing curve is perfectly smoothing without any fluctuations.\n\nComparison in chart 3 needs improvement. Since the two models have different input signals and model structure, they inherently need different hyper-parameters (not only learning rate) for best performance. Only comparing them with the same learning rate may not adequate to prove the significance of the proposed method.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposed to pre-process the original input features into a low dimensional term and its corresponding residual term via SVD. The paper empirically demonstrated the neural networks trained on such factorized exhibit faster convergence in training. Several issues of clarity were addressed during the rebuttal period by the authors. 

However, the reviewers still felt that there were some remaining fundamental issues with the paper, 

1)  The motivation is not echoed in the experiments, namely most of the experiments on CIFAR and CatDog dataset using a low dimensional factorization of d=1 which is trivial and often part of the whitening preprocessing. 

2) The proposed factorization via SVD will be difficult to scale up to high dimensional features, large training sets and higher d >> 1.  

3) The empirical experiments show a marginal improvement in the training speed, especially in the image recognition tasks, yet there seems an early plateau in test performance when compared to the baselines.

4) The theoretical analysis in Section 2 studied linear models. Yet, the rest of the paper focuses on non-linear neural networks. It is difficult to see the connection between the analysis and the rest of the paper. 

Thus, I recommend rejection of the paper at this time as the current version of the paper needs further development, and non-trivial modifications, to be broadly applicable.
Decision: Reject