Semi-Implicit Back Propagation
Authors: Ren Liu
Xiaoqun Zhang
Venue:  (2020)
Keywords: Optimization
Neural Network
Proximal mapping
Back propagation
Implicit

Abstract: Neural network has attracted great attention for a long time and many researchers are devoted  to improve the effectiveness of  neural network training algorithms. Though stochastic gradient descent (SGD) and other explicit gradient-based methods are widely adopted, there are still many challenges such as gradient vanishing and small step sizes, which leads to slow convergence and instability of SGD algorithms. Motivated by error back propagation (BP)  and proximal methods, we propose a semi-implicit back propagation method for neural network training. Similar to BP, the difference on the neurons are propagated in a backward fashion and the parameters are updated with proximal mapping. The implicit update for both hidden neurons and parameters allows to choose large step size in the training algorithm. Finally, we also show that any fixed point of  convergent sequences produced by this algorithm is a stationary point of the objective loss function. The experiments on both MNIST and CIFAR-10 demonstrate that the proposed semi-implicit BP algorithm leads to  better performance in terms of both loss decreasing and training/validation accuracy, compared to SGD and a similar algorithm ProxBP.

Under review as a conference paper at ICLR 2020
SEMI-IMPLICIT BACK PROPAGATION
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural network has attracted great attention for long time and many researcher- s are devoted to improve the effectiveness of neural network training algorithms. Though stochastic gradient descent (SGD) and other explicit gradient-based meth- ods are widely adopted, there are still many challenges such as gradient vanishing and small step sizes, which leads to slow convergence and instability of SGD algorithms. Motivated by error back propagation (BP) and proximal methods, we propose a semi-implicit back propagation method for neural network train- ing. Similar to BP, the difference on the neurons are propagated in backward fashion and the parameters are updated with proximal mapping. The implicit up- date for both hidden neurons and parameters allows to choose large step size in the training algorithm. Finally, we also show that any fixed point of convergent sequences produced by this algorithm is stationary point of the objective loss function. The experiments on both MNIST and CIFAR-10 10 demonstrate that the proposed semi-implicit BP algorithm leads better performance in terms of both loss decreasing and algorithm ProxBP.
accuracy, compared to SGD and a similar
INTRODUCTION
Along with the rapid development of computer hardware, neural network methods have achieved enormous success in divers application fields, such as computer vision (Krizhevsky al.) 2012),
speech recognition (Hinton et al. 2012, Sainath et al.) 2013), nature language process (Collobert al. 2011) and so on. The key ingredient of neuron network methods amounts to solve highly non-convex optimization problem. The most basic and popular algorithm is stochastic gradient descent (SGD) (Robbins& Monro, 195T), especially in the form of "error" back propagation (BP) (Rumelhart et al. 1986) that leads to high efficiency for training deep neural networks. Since then many variants of gradient based methods have been proposed, such as Adagrad et al. [011).
Nesterov momentum (Sutskever al. 2013), Adam (Kingma & Ba 2014) and AMSGrad (Reddi ct al. 2019). Recently extensive research are also dedicated to develop second-order algorithms, for example Newton method (Orr& & Müller 2003) and L-BFGS (Le et al. 2011).
is well known that the convergence of explicit gradient descent type approaches require sufficient- ly small step size. For example, for loss function with Lipschitz continuous gradient, the stepsize should be in the range of (0, (0) for L being the Lipschitz constant, which is in general extremely big for real datasets. Another difficulties in gradient descent approaches to propagate the "error" deeply due to nonlinear activation functions, which is commonly known as gradient vanishing. To overcome these problems, implicit updates are more attractive. In (Frerix et al., 2018), proximal back propagation, namely ProxBP, was proposed to utilize the proximal method for the weight up- dating. Alternative approach to reformulate the training problem as a sequence of constrained optimization by introducing the constraints on weights and hidden neurons at each layer. Block co- ordinate descent methods (Carreira-Perpinal & Wang 20144, /Zhan & Brand 2017) were proposed and analyzed to solve this constrained formulation with square loss functions. Along this line, the Alternating direction method of multipliers (ADMM) (Taylor et al. 2016 Zhang et al., (016) were also proposed with extra dual variables updating.
Motivated by proposing implicit weight updates to overcome small step sizes and vanishing gradient problems in SGD, we propose semi-implicit scheme, which has similar form "error" back propagation through neurons, while the parameters are updated through optimization at each layer. can be shown that any fixed point of the sequence generated by the scheme is stationary point of the

---

Under review as a conference paper at ICLR 2020
objective loss function. In contrast explicit gradient descent methods, the proposed method allows to choose large step sizes and leads to better training performance per epoch. The performance is also stable with respect to the choice of stepsizes. Compared to the implicit method ProxBP, the proposed scheme only updates the neurons after the activation and the error is updated in a more implicit way, for which better training and validation performances achieved in the experiments on both MNIST and CIFAR-10.
NOTATIONS
Given onput-output data pairs (X, we consider a N --layer feed-forward fullly connected neural network as shown in Figure Here, the parameters from the i-th layer to the (i 1) layer the
X
Wi.br dy
22
W2.ba
G3
GN-1 _ FN NN- FN
Figure 1: N-layer Neural Network
weight matrix W; and bias b, and a is a non-linear activation function, such as sigmod or ReLU. We denote the neuron vector at i-th layer before activation as Gy, and the neurons after activation as Fr, i.e. Fi X.
Gi+1 W.F. by; = oG
(I)
for 1, .N. - We note that general the last layer, there is no non-linear activation function and FN Gn. For ease of notation, we can use an activation function a as identity. The generic training model aims solve the following minimization problem:
min (0; X, X, = L(Fx, 0
(2)
where 8 denotes the collective parameter set (W, - and is some loss function.
SEMI-IMPLICIT PEC-PPRPPAA METHOD
In the following, first present the classic back propagation (BP) algorithm for an easier introduc- tion the proposed semi-implicit method.
3.1 BACK PROBATION METHOD
The widely used BP method Rumelhart al. (986)) is based gradient descent algorithm:
g++1 = 0² ;J X,Y)
00
where y > 0 is the stepsize. The main idea of BP algorithm is to use an efficient error propagation scheme on the hidden neurons for computing the partial derivatives of the network parameters at each layer. The so-called "error" signal ōN aF* at the last level is propagated to the hidden 0J
neurons 6; = using the chain rule. In fact for square loss function, the gradient at the last layer
8N aL(Fw.g) FN y is indeed error. The propagation from 6,41 to d, for i N 1,....
0F is then calculated as
JJ
0G.+1 0F AJ
(G((1))) 2 8.41).
6 OF, OF, 0G,41 OF.41
W?
(4)
0G++1
2

---

Under review as a conference paper at ICLR 2020
And the partial derivative to W; can be computed as
0J 0G 0F+t ow; =
DJ aw, 0G441 0F++1
(G(+)) aG
c F
(5)
At k-th iteration, after a forward update of the neurons 1.th by ( using the current parameters sets {W.*. b 1, 1....... N -
compute the "error" signal at each neurons sequentially from the + 1 level to level by ( and the parameters W is updated according to the gradient the point W computed by
3.2 SEMI-IMPLICI UPDATES
Compared to the BP method, we propose to update the hidden neurons and the parameters sets at each layer in an implicit way.
At the iteration k given the current estimate 0² W.W. b. we first update the neuron F. and G feedforward fashion as BP method, by using (T) for i 1, N For the backward stage, we start with updating neuron FN at the the last layer using the gradient descent:
N%
OL(FN
N.INNYY Y)
b F 1
OFN
For N - 1..... 1, given F411 the parameters W;, are updated by solving the following optimization problem (once)
Wh+1 arg min (WFF. + b,) F 1|
X 2
W/
6.+1 arg min |o (W F + b,)
k+b p
1b: - 11?
where À > 0 is a parameter that is corresponding to stepsize. This update of parameters is related using an implicit gradient based on so-called proximal mapping. Taking W. an example, the optimality in 7 gives
Wht! W /
where /(W,) ( 11. Compared to a direct gradient descent step, this update unconditionally stable for any stepsize 1/A. We note that proximal mapping was previously proposed for training neural network ProxBP in (Frerix al., 2018). However the update of the parameter sets is different as ProxBP uses Gi+1 for the data fitting at each layer. The two subproblems at each layer can be solved by nonlinear conjugate gradient method.
After the update of W and b/+1,, need to update the hidden neuron F,. As classical BP, first consider the gradient at F.
OJ OF
OF OJ
aG O aJ
(8)
OP
141 OF aG OF% 1+1
oc aFF
can be seen that the partial derivative newly updated W instead of W to compute the error:
= W. Different from BP and ProxBP, we use the
tt IWT+ (Bo )
F k+
F.t 710,
By this formula, the difference can be propagated from the level N to 1. At the last level 1, we only need to update W and 8x+1 F X. The overall semi-implicit back propagation For large scale training problem, the back propagation is
method is summarized used in the form of stochastic gradient descent (SGD) using small set of samples. The proposed semi-implicit method can be easily extended to stochastic version by replacing (X, Y) by a batch set (X miii,, Ymini) at each iteration in Algorithm 1.
Algorithm
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Semi-implicit back propagation
Input: Current parameters 0F {W?, b5} Forward pass
F1 = X for i
N do
G = W pk + b; F o(G
end for
Update on the hidden neurons and the parameters Update on F
fori=
do Implicit update on W by
Error propagation and update on E.t - F
end for
Implicit update on W by
Output: New parameters gh+1 (Wh+1 (
3.3
FIXED POINTS OF SEMI-IMPLICIT METHOD
The follow proposition indicates that any fixed point of the iteration is a stationary point of the objective energy function.
Proposition 1 Assume that L and the activation functions are continuously differentiable. Ok k or for 0" (W7 b then O" is stationary point of the energy function [0; X, Y).
{Wr,b] infers that {P G K {P, Gi} where G W Fr- + bj and G; (PP;)) for 11,.... N. At the last layer, the neuron FN updated with gradient descent:
Proof Due to the forward update, {W*, 8 b/4 k=000
OL(FN, Y)
Fx
F
(10)
OF
Take a limit, we have
lim p k+b
lim F OL Y)
(11)
k0000
k000
OF
OJ
F
(12)
OF
Now we show
lim k0000
1+ P.
F.
11
OFF
0J
aW.
JJ
0
(13)
for i ..... 2 using mathematical induction. The first equation is shown for i N. By the optimality of equation we have
2 (W Wht) = [Wo F, + b,)) o (F k+
OWW +
(14)
4

---

Under review as a conference paper at ICLR 2020
Let l - oo, we obtain
OJ
[0 - b;)
(F 11 0F
F. (F))
0J
10o
}[(F))
0F 1+
OJ
-
0F [(FTT
(15)
aG OF ØJ
-1I OW; 0 01
OJ
=- aww
aw,
It is easy to see that the limit of F." k+ for N 15-1, ,,.,,,15s
lim F
k+
lim
F (W++1)² ( (F - FF+
10000
k000
0J
Fr nWW?) (10 (i++1) 0F
00 01 0J
(16)
F -1 OF 0G;,1 0F
0J
F 1 aF;
With mathematical induction we obtain that
aJ 00-
0.
4 NUMERICAL EXPERIMENTS
In this section, we will compare the performance of BP, ProxBP Frerix et al. (2018) and the pro- posed cemi-implicit BP using MNIST and CIFAT-10 datasets. All the experiments are performed on MATLAB with a 12-core CPU and the same network setting and initializations are used for fair comparison. We use softmax cross-entropy for the loss function 7. and ReLU for the activation func- tion, usually chosen in classification problems. For the linear CG used in ProxBP and nonlinear CG in semi-implicit BP, the iterations number is set as 5. Finally the weights and bias are initialized by normal distribution with average 0 and standard deviation 0.01.
In MNIST experiments, the training set contains 55000 samples and the rest are included in the validation set. For CIFAR-10, 10, set of 45000 samples is used training set and the rest valida- tion set. The training process are performed times, and Table [ shows the average training and validation accuracy achieved by SGD, Semi-implicit BP with different learning rates 1) (for SGD) and 1 1/A for semi-implicit BP on MNIST. It can be seen that after epoch, semi-implicit BP method already achieves high accuracy as high 99% and the performance is very stable with respect to different stepsize choices, while SGD fails for some choices of stepsize 1). For ProxBP, we present the results with 7 as the best performance is achieved with this set of parameters. The highest accuracies are marked in bold in each column, and we can see that semi-implici BP achieves the highest training and test accuracy than BP and ProxBP.
In Figure and 3 we show the performance of the three methods per epoch for MNIST and CIFAR- 10. For the CIFAR-10 experiment, we train a 3072 x 2000 500 x 10 neural network and the rest settings are the same MNIST dataset experiments. For both datasets, we choose the step size 7) = 0.1 for semi-implicit BP and ProxBP, while a smaller one to guarantee SGD achieves best better performance. The evolution of training loss and training accuracy shows that the performance of ProxBP is not as good as SGD for MNIST while it is better than SGD on CIFAR-10. For both experiments, the performance of semi-implicit BP method leads to the fastest convergence. The im- provement on the validation accuracy also demonstrates that the proposed semi-immi semi-implicit BP method also generalize well in comparison the other two methods.
5

---

Under review as a conference paper at ICLR 2020
(784 x 500 x 10)
Traininngvalidationn accuracy
learning rates
100
10
1
0.1
0.01
0.0985
0.1123
0.0980
0.9487
0.8885
SGD, 17
0.1002
0.1126
0.0924
0.9492
0.8988
0.9239
0.9349
0.9390
0,9032
0.8415
ProxBP 1,11
0.9344
0.9374
0.9494
0.9244
0.8832
0.9420
0.9444
0.9383
0.9171
0.8743
ProxBP (1) = 1,1
0.9486
0.9554
0.9494
0.9344
0.9064
0.9780
0,9801
0.9904
0.9737
0.9104
Semi-implicit (A = 1))))
0.9710
0.9724
0.9800
0.9748
0.9338
Semi-implicit
0.9765
0.9752
0.9735
0.9598
0.9206
(1) = 1/A)
0.9736
0.9778
0.9738
0.9672
0.9394
Table I: Training and validation accuracy with different step sizes for 2 epoch.
10
MNST. 78L 500 MLP
MNIST 704 10
MYST 286 500 MI
Ser Amoll!
2.07
Loee
08
0.3
mimppa PresD
10
Epooh
Epoch
Epoch
Figure 2: MNIST. Batch size 100
23
CIFAR-10,
CIFAR-1 2072 2200 500 10MLP
CIARR/OO 3072 2000 500 MLP
Amolle!
0.5
Traking LeKE
82
Prode
87
"rug""
0.5
Epoch
Epocch
Figure 3: CIFAR-10, Batch size 100

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
We proposed a novel optimization scheme in order to overcome the difficulties of small stepsize and vanishing gradient in training neural networks. The computation new scheme in the spirit of error back propagation, with an implicit updates on the parameters sets and semi-implicit updates on the hidden neurons. The experiments on both MNIST and CIFAR-I0 show that the proposed semi-implicit back propagation has better performance per epoch compared to SGD and ProxBP. demonstrated the experiment that larger step sizes can be adopted without losing stability and performance. It can be also seend that the proposed scheme is flexible and some regularization can be casily integrated needed. The fixed points the scheme are shown to be stationary points of the objective loss function and further rigorous theoretical convergence will be explored in an ongoing work.
REFERENCES
Miguel Carreira-Perpinan and Weiran Wang. Distributed optimization of deeply nested systems. In Artificial Intelligence and Statistics, pp. 10-19, 2014.
Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuk- sa. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(1):2493-2533 2537,, 2011.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(7):257-269, 2011.
Frerix, Möllenhoff, M. Moeller, and D. Cremers. Proximal backpropagation. In International Conference on Learning Representations (ICLR), 2018.
Geoffrey Hinton, Li Deng, Dong Yu, George E. Dahl, Abdel Rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, and Tara N. Sainath. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6):82-9) 2012.
Diederík Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6981. 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. In International Conference Neural Information Processing Systems, 2012.
Quoc V Le, Jiquan Ngiam, Adam Coates, Abhik Lahiri, Bobby Prochnow, and Andrew Y Ng. On optimization methods for deep learning. In Proceedings of the 28th International Conference on International Conference Machine Learning, pp. 265-272. Omnipress, 2011.
Genevieve B Orr and Klaus-Robert Müller. Neural networks: tricks of the trade. Springer, 2003.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. arXiv preprint arXiv:1904.002237 2019.
Herbert Robbins and Sutton Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22(3):400-407, 1951.
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323(3):533-536 1986.
Tara N. Sainath, A. R. Mohamed, Brian Kingsbury, and Bhuvana Ramabhadran. Deep convolutional neural networks for Ivesr. In IEEE International Conference on Acoustics, 2013.
I. Sutskever. J. Martens, G. Dahl. and G. Hinton. On the importance of initialization and momentum in deep learning. International Conference on International Conference on Machine Learning, 2013.

---

Under review as a conference paper at ICLR 2020
Gavin Taylor, Ryan Burmeister, Zheng Xu, Bharat Singh, Ankit Patel, and Tom Goldstein. Training neural networks without gradients: scalable admm approach. In International conference on machine learning, pp. 2722-2731,
Ziming Zhang and Matthew Brand. Convergent block coordinate descent for training tikhonov regularized deep neural networks. In Advances in Neural Information Processing Systems, pp. 1721-1730, 2017.
Ziming Zhang, Yuting Chen, and Venkatesh Saligrama. Efficient training of very deep neural net- works for supervised hashing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1487-1495, 2016.
8


Reviews:
[{"review_id": "rkg4bK5jYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work is based on the recent paper 'Proximal Backpropagation', Frerix et al., ICLR 2018, which views error back propagation as block coordinate gradient descent steps on a penalty functional comprised of activations and parameters.\n\nInstead of taking proximal steps on the linear layers as in (Frerix et al., 2018), the authors also pull the non-linearity \\sigma into the proximal steps. Another interesting deviation is the idea to consider the newly updated weights {W_i}^{k+1} when updating the activations F_i^{k+1} in the backward pass.\n\nWhile potentially offering a faster convergence with respect to epochs, the nonlinear updates have two major drawbacks:\n\n1) While there are preliminary theoretical results (fixed points of the method are critical points), it remains unclear whether the computed update is still a descent direction on the original energy.  While not crucial, such a result would be reassuring and might give further insights into the method.\n\n2) Each update requires the solution of a nonconvex, nonlinear least squares problem which is prohibitively expensive to solve. Note that such nonlinear least squares updates are already proposed in (Carreira-Perpinan & Wang, 2014). When using ReLU activation, the non smoothness might be an issue for standard nonlinear least squares solvers such as Levenberg-Marquadt. \n\nFurthermore, the numerical results are unfortunately a bit discouraging. The experiments evaluate toy models on toy datasets and even there only a minor improvement with respect to epochs over SGD and Prox-BP is shown. Furthermore, the plots only consider epochs and not the running time. Due to the non-linear least squares problem, I assume that each epoch for the proposed method is way more costly. Therefore I consider the experimental evaluation too preliminary. A proper evaluation would require an implementation as an optimizer in state-of-the-art deep learning frameworks and a comparison with respect to running time to standard optimizers such as SGD with momentum or Adam on the GPU. \n\nThe reported performances for MNIST are surprisingly poor. Note that vanilla SGD with momentum reaches ~98.6% test set performance on such an architecture, while the overall highest reported accuracy in this paper is 98.0%. This might be due to momentum, and it would be interesting whether the proposed method could be combined with momentum or other optimizers such as Adam as in (Frerix et al. 2018).\n\nOverall, I don't see this a practical algorithm for training deep networks and there are few theoretical results. Therefore, I cannot recommend acceptance at this stage. \n\nTo improve the paper, I would like to see an implementation of the method on the GPU in a recent deep learning framework and an evaluation on larger models / datasets. But I am doubtful this will reach competitive performance to standard optimizers. Also, it would be interesting to see how the precision in the inner nonlinear conjugate gradient solver effects outer convergence. It might be that the subproblem does not have to be solved with very high accuracy. \n\nMinor comments:\n* Missing citation: 'Difference Target Propagation', https://arxiv.org/abs/1412.7525, studies a similar type of algorithm.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SyeIeeDnKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a novel algorithm for computing update directions for neural network's weights.\nThe algorithm consists of the modified backpropagation procedure where a layer's error is computed using implicitly-updated weights.\n\nThe proposed idea is interesting, but its presentation and evaluation could be significantly improved.\nFirst, it is not very clear what motivates the exact form of Semi-implicit BP.\nSecond, I find the notation a bit cumbersome, especially intermediate ^{k+1/2} updates. \nI also suspect that eq. 9 contains an error, probably the l.h.s. of the first part should be \\delta_i^{k+1}?\nAlgorithm 1 is not very helpful because only the forward pass is explained in detail and a reader must refer to the main text to understand the backward pass.\n\nThe experimental evaluation also raises a number of questions.\n1) Why did authors chose only one value of the learning rate and the lambda hyperparameter? A more appropriate comparison would require slightly more extensive hyperparameter search as it may well be that ProxBP would work better with different values.\n2) It is also unclear if 5 CG iterations is enough to solve the intermediate problem. Also all convergence guarantees are only provided for the exact implicit update, so at least one should ensure it is computed well enough.\n3) Isn't it suspucious that ProxBP performed so bad compared to other methods on MNIST? \n4) Should not the set of baselines include more advanced optimizers such as RMSProp, Adam etc? They don't seem to add more computational burden than Semi-implicit BP.\n\nI also think it is worth discussing/investigating if the obtained update directions can be used in other gradient-based optimizers instead of pure gradients and if it can have any advantages.\n\nOverall, it does not feel to me that the paper is ready for publication.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJgOydfpFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an implicit update scheme for the back propagation a anlgorithm. \nThe idea is quite simple and is based on proximal mappings that lead to implicit update.\nSpecifically, every update in the back propagation algorithm is being replaced by an implicit update except for the intermediate parameters that receive a \"semi-implicit\" update.\n\nThe idea is reasonable and seems to lead to good performance. This is more-or-less expected thanks to the superior performance of implicit updates in general. So, it's good that the authors could make this work in the context of deep nets as well. Here are some more critical thoughts about the paper:\n\n\n1) There is not much theoretical justification about the idea in the paper. Proposition 1 is a simple argument about the fixed point of the procedure. The argument could be made more rigorous, right now it is a bit of a sketch. \nApart from Proposition 1, there is no more theory offered. The authors could appeal in the theory of implicit SGD for that, e.g., [1,2,3,4]. This theory suggests a lot of stability properties for the implicit SGD update of Equation (27).\n\n2) Somewhat related to (1), the authors could make a more clear connection to prior work. \nFor example, there is not mention of a very similar idea of \"implicit back propagation\" [5]. \nAlso the literature in implicit SGD procedures is highly relevant.\n\n3) Can we explain the results in Table 1 theoretically?\n\n\n\n[1] Bertsekas, \"Incremental proximal methods for large scale convex optimization. Mathematical\nprogramming\", 2011\n[2] Kulis and Bartlett, \"Implicit online learning\", 2010\n[3] Toulis and Airoldi, \"Asymptotic and finite-sample properties of estimators\nbased on stochastic gradients\", 2017\n[4] Toulis, Airoldi, Rennie, \"Statistical analysis of stochastic gradient methods for generalized linear models\", 2014\n[5] Fagan and Iyengar, \"Robust Implicit Backpropagation\", 2018", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers equivocally reject the paper, which is mostly experimental and the results of which are limited.  The authors do not react to the reviewers' comments.
Decision: Reject