One Forward is Enough for Neural Network Training via Likelihood Ratio Method
Authors: Jinyang Jiang
Zeliang Zhang
Chenliang Xu
Zhaofei Yu
Yijie Peng
Venue: ICLR 2024 poster (2024)
Keywords: stochastic optimization
gradient estimation
Primary area: optimization

Abstract: While backpropagation (BP) is the mainstream approach for gradient computation in neural network training, its heavy reliance on the chain rule of differentiation constrains the designing flexibility of network architecture and training pipelines. We avoid the recursive computation in BP and develop a unified likelihood ratio (ULR) method for gradient estimation with only one forward propagation. Not only can ULR be extended to train a wide variety of neural network architectures, but the computation flow in BP can also be rearranged by ULR for better device adaptation. Moreover, we propose several variance reduction techniques to further accelerate the training process. Our experiments offer numerical results across diverse aspects, including various neural network training scenarios, computation flow rearrangement, and fine-tuning of pre-trained models. All findings demonstrate that ULR effectively enhances the flexibility of neural network training by permitting localized module training without compromising the global objective and significantly boosts the network robustness.

Published as a conference paper at ICLR 2024
ONE FORWARD IS ENOUGH FOR NEURAL NETWORK TRAINING VIA LIKELIHOOD RATIO METHOD
Jinyang Jiang" 2: Zeliang Zhang Chenliang Xu Zhaofei Yu* Yijie Peng 1.2
Wuhan Institute for Artificial Intelligence, Guanghua School of Management, Peking University
2 Xiangjiang Laboratory Department of Computer Science, University of Rochester
4 Institute for Artificial Intelligence, Peking University
j jinyang. j jiangestu, pku. edu.edducc. {zeliang. zhang, chenliang wwwerochester..cooo fyuzf12,
cn
ABSTRACT
While backpropagation (BP) the mainstream approach for gradient computation in neural network training, its heavy reliance on the chain rule of differentiation constrains the designing flexibility of network architecture and training pipelines. We avoid the recursive computation in BP and develop unified likelihood ratio (ULR) method for gradient estimation with only one forward propagation. Not only can ULR be extended to train a wide variety of neural network architectures, but the computation flow in BP can also be rearranged by ULR for better device adaptation. Moreover, we propose several variance reduction techniques to further accelerate the training process. Our experiments offer numerical results across di- aspects, including various neural network training scenarios, computation flow rearrangement, and fine-tuning of pre-trained models. All findings demon- strate that ULR effectively enhances the flexibility of neural network training by permitting localized module training without compromising the global objective and significantly boosts the network robustness.
INTRODUCTION
Since backpropagation (BP) (Rumelhart et al. 1986) has greatly facilitated the success of artificial intelligence (AI) various real-world scenarios (Song et al. 2021a b
Sung al. 2021). researchers are motivated to connect this gradient computation method in neural network training with human
learning behavior (Scellier Bengio, 2017 Lillicrap et al. 2020). However, there is no evidence
that the learning mechanism in biological neurons relies on BP (Hinton[2022), Pursuing alternatives BP holds promise for not only advancing our understanding of learning mechanisms but also developing more robust and interpretable Al systems. Moreover, the significant computational cost associated with BP (Gomez et al. 2011) 7 Zhu et al. (2022) also calls for innovations that simplify and expedite the training process without heavy consumption.
There have been continuous efforts to substitute BP in neural network training. For example, the HSIC bottleneck (Ma et al. 2020), feedback alignment (FA) (Nøkland 2016), and neural tangent kernel (NTK) (Jacot et al. (0118) employ additional mod-
1.00
120
ClassiCeaaionnA. A
Accuracy
0.95
100
Bohning Soniing Time (slepuch)
0.0
-30
0.3
ules with constraints to compute the gradients without relying on the chain rule of differentiation, thus avoiding BP. While model-
0.5
based methods remain biologically implausible and suffer from instability and efficiency, perturbation-baser methods, including forward- -oowward (FF) (Hinton) 2022), evolution strategy (ES) (Salimans al., 2017). and likelihood ratio (LR) (Peng et al. 2022) method, which only involve the forward computation and have little assumption on the model architecture, offer promis- ing path for exploring biologically plausible BP alternatives.
0.75
0.70
Model Scale (# of Siie layets)
Figure 1: Classification accura- cies (curves) and training dura- tions (bars) of three methods till convergence.
These authors contributed equally to this work. Listing order is random.
Corresponding author.

---

Published as a conference paper at ICLR 2024
In the perturbation based category, the preliminary experiment with multilayer perceptrons (MLPs) (Rumelhart et al. 1986) on the MNIST dataset (LeCun 1998) indicates that LR might surpass oth- in terms of efficiency paradigm to adjust each layer separately, and its incompatibility with other existing techniques ham- pers efficiency. The weight randomization in ES is more likely to produce infeasible exploration, resulting in instability, and becomes costly when the parameter dimensionality grows, By contrast, training by LR optimizes the model following unbiased gradient estimates without imposing back- ward recursions and only requires single forward propagation, which can be accelerated by data replication with low consumption and integrated with other training techniques.
accuracy, as shown in Fig. T FF abandons the classical optimization
We argue that LR is better alternative to BP for four reasons. First, breaking the reliance on the chain rule, LR requires only one forward evaluation for efficient learning. Second, L.R has almost no constraints on the model, including differentiability or traceability, enabling explorations of net- work architectures. Third, gradient computation among different modules can be independent LR, which allows for more flexible training paradigm design. Last. LR can enhance the model robust- ness by smoothing the loss landscape. However, the application of LR still faces various challenges. Gradient estimation in different networks calls for unified derivation methodology, implementation- friendly conversion, and practical stabilization tricks. Furthermore, the generalization of LR should not be confined just to the level of neural layers. LR has the potential to be employed technique at various scales, such as in neuron-wise parallelism or computation graph reorganization.
In our paper, we develop unified LR (ULR) method on the most general network possible to cover its essence and application potential. We then generalize pure LR from special case on MLPs to four network architectures representing different gradient computation challenges. Several vari- ance reduction approaches are first proposed by us in the LR context to significantly stabilize and make model training via only one forward propagation much more practical. Meanwhile, LR is also employed to transform the existing training computation flow of BP, adapting to features for modern devices. We conduct experiments of our ULR with the aforementioned network structures on cor- responding tasks and provide two applications, including domain adaption and computation graph rearrangement. Numerical results indicate that ULR effectively enhances the flexibility of neural network training in model and pipeline designing, and significantly improves the model robustness.
2 RELATED WORK
Optimization methods for neural network training can be roughly categorized into two types, with and without BP. BP-based optimization methods have been studied for long time and have been effective in training various neural networks on different tasks. Among these methods, the vanilla BP method is utilized for computing the gradients of the neural network parameters with respect to the global loss function. BP requires the perfect knowledge of computation details between the parameters and loss evaluation to launch a recursive computation, which limits the flexibility of developing neural network architectures and training pipelines. Due to the reliance on the chain rule, this can also be the case for some methods (Baydin et al. 2018) claiming a forward-only training.
Some studies have explored alternative methods without BP for training neural networks by adding extra functional modules to guide model updates. The HSIC bottleneck (Ma et al.) 2020) maxi- mizes the Hilbert-Schmidt independence criterion (Wang et al.. 202I) enhance layer indepen- dence. NTK (Jacot et al. 2018)) introduces intermediary variables to circumvent the limitations of BP. FA (Nokland (016)) introduces intermediary variables to overcome BP limitations but faces instability issues. Since these methods change the classical training paradigm or even give up the gradient information. many algorithmic or hardware technologies developed based on BP by prede- cessors cannot be fully leveraged, making such approaches computationally unfriendly.
Other works still adhere to the original mathematical problem but propose perturbation- based solu- tions from stochastic optimization perspective. [Splll (1992) proposes simultaneous perturbation stochastic approximation (SPSA) method, which approximates the gradient by evaluating the ob- jective function twice with perturbations of opposite signs on the parameters. SPSA is typically sensitive to the choice of hyperparameters. Another approach is ES (Salimans et al. 2017). which injects population of random noises into the model and optimizes the parameters along the opti- mal direction of noises. Both SPSA and ES suffer from heavy performance drops as the number of parameters increases, which hinders the application to sophisticated models in deep learning.
2

---

Published as a conference paper at ICLR 2024
FF (Hinton 2022 modifies the input data and works by distinguishing the positive and negative samples for each module but fails to address the weight-sharing and compatibility issue. Unlike a pathwise gradient computed by the BP, Peng al. (2022) perturb the logit outputs of each neural layer in MLPs and employ the push-out LR technique to estimate the gradient of randomized loss evaluation. However, previous research LR has primarily focused MI.P training as a whole, which can be recognized as specific instance of the architecture extension in this paper.
3 UNIFIED LIKELIHOOD RATIO METHOD FOR FLEXIBLE TRAINING
3.1 GRADIENT ESTIMATION USING PUSH-OUT LR TECHNIQUE
Mainstream neural network training can be roughly divided into loop with three phases: forward evaluation, gradient computation, and parameter update. LR addresses the second issue as BP does, and it can be integrated with any existing method for the other two phases. We generalize the LR method to the training of generic neural networks and only assume the network to have a modular structure, where each module might consist of single multiple neural layers. Given hierarchical neural network, which is sliced into L modules, the input to the I-th module is denoted as x' E RR,,, and the output is given by 1= 0=..... L
x 1++ = (')) +'2, f'm)
where *111 is the output of the 1-th module, se and E O' respectively denote the non- non-maramtt structure and parameter, and a' R441 is newly added zero-mean noise vector with the den- sity J Training the neural network can be regarded as solving such an optimization problem: mingge Exexx E. [[(x")]. where X' denotes the dataset and L is the loss function. A basic ap- proach to perform gradient descent following the stochastic gradient estimation of [[(x")] with respect to the parameters. Denote the Jacobian matrix of y E Rd with respect to x E R as derive the following ULR estimator
Duy xd, Under mild integrability conditions, for the generic network structure:
Theorem 1. Given an imput data x., assume that ge(E) J'(E (xt; is differentiable, and
E
sup Øleel (oag(() < <c.
(1)
R
Then, we have
of [[x"))] ( (e) (2):0') (2
Proof. Notice that conditional on s', xl+1 follows distribution characterized by the density (E)
With the property of conditional expectation, we can push the parameter o out of the loss function [(x²) and into the conditional density below:
Vo Eug [f Ea [f(x o (E))EE
V E.........--- -
[C(x*)]
which is the so-called push-out LR technique. Then, we can obtain
f gd++1
V E [[x+)]]
=
R
E [[(++))S)a)] a In (E)
E
E ((. xt| Du 0') In (o
R
E - Spin E [C( )c + (x): , L ol
E ((xr") TT ( (ar: V In ('))))
The first equality is justified by the LR technique and the dominated convergence theorem under the given integrability condition 1 the third equality holds by the change of variable, and the rest comes from definitions and the iterative property of the conditional expectation.
By selecting appropriate noise distributions and excluding the discontinuous or black-box portions in the network from the modules with parameters, we can ensure the differentiability requirements in
3

---

Published as a conference paper at ICLR 2024
Theorem I Eq. implies that the gradient estimation for the whole model does not require the recursive backward computation and can be further paralleled with only the final evaluation of [(x²) and the local information x' and z'. As shown in Fig. 2 the application of auto- differentiation is confined within modules, where Doupp (ach; (x);0') is com- puted independently from each other with different l. ULR also supports an adaptive perturbation by optimizing the parameterized noise distribu- tion (:; (::0' 0¹) following the gradient:
x=
Unknown
Parameterized Module
z'
Unknown
V [c(=)]] = EE [C(x")V (=))
which could benefit the representation capacity and robustness of the model (Xiao et al. 2022). In the next section, specify f as a multi- variate Gaussian distribution with trainable covariance and zero mean for readability, while other distributions, e.g., the exponential family, are also acceptable.
Figure 2: Computation flow with our ULR in generic networks.
3.2 EXTENSION TO VARIOUS NETWORK ARCHITECTURES
ULR does not impose specific requirements the modules in the neural network, and gradient computation localized. Therefore, we can analyze and extend ULR to almost all computational structures from different networks by treating them single modules. Since these modules can be rewritten as the combination linear transformations and non-linear processes, we can directly perturb the logit output of the former and estimate the gradients with only one forward propagation, completely avoiding the anteediiferentiatioon technique.
Convolutional neural networks (CNNs): Convolution operation is the representative of the spacial parameter sharing, where the outputs are related to the same kernel. Denote the input as x (ar),c E Rea x XWa, where the superscript represents the channel index. The output given by
3A (x 0)3 k. + D 2j,00
where * denotes the convolution operation, 0 E R x he Xwe and = (b)) E Ro are the weight and bias terms, *j.A o is the noise injected to perform ULR, r² denotes the noise magnitude, and x7 is standard Gaussian random variable. The minimum unit go. is involved in the compu- tation of the o-th output channel. Thus, we push 0=.s into the conditional density of v given x² and obtain the gradient estimation for each convolutional kernel channel by Theorem I as below:
V go. E [[())] [[(o") L(o)x² ==]]
where the rest of the network and loss function are abbreviated as L. Eq. (3) quite computationally friendly and can be implemented by the convolution operator in any standard toolkit.
When the kernel tensor is small, the logit perturbation may be costly concerning the parameter size. alternative way to perturb the neuron is to inject noises into the parameters: v = x 0 + b. go.z 00.2 + o where the elements in ea, are independently and normally distributed. Then, we treat 0 as the input and output of an identity mapping module and apply Theorem | resulting in
gr E [[(v)] E (
Since the variances of Eq. (3) and ( both depend on the noise dimension, the selection between the two can be determined by the total dimensions of the output and convolutional kernels.
Recurrent neural networks (RNNs): Gradient estimating in RNNs faces the temporal weight- sharing issue. Denote the t-th input and hidden states as It E Rd2 and hy E Rdh, 1,... T. RNN variants can be summarized as a generic structure and the next hidden state given by
hu ((uu Ut, he-1), 14 ghh
h4-1 1 + Bhh + hhh U: gah + 1z + 2ith
where 0hh and 0zh are weight matrices, bhh and boh denote bias vectors, z/th (E wihh (5 ) ehh eii and og are standard Gaussian random vectors. Taking ghi an example, first
4

---

Published as a conference paper at ICLR 2024
treat it as different in each time step to apply Theorem I and then up the step estimates. Denote the rest of the forward process as L and let h = (h1 hr). The gradient can be unrolled as
Van E...E/e [[C(N]] -E(C)EE
hh 11-1
(5)
Eq. (5) enables the gradients of the RNN variants be computed parallel over input sequences without relying on BP Through Time (BPTT), which is one of the main advantage of Transform- ers (Vaswani et al. 2017)) over RNNs. Eq. (5) is derived under a generic structure, and their specific forms could be simplified for different cells. Please refer to Appendix A.3 for details.
Graph neural networks (GNNs): By leveraging the matrix representation of graph structures, gradient estimation towards unstructured inputs also be derived by ULR. Consider a graph con- volutional network (GCN) (Kipf & Welling, 2016) layer with the input h. E RD]xxd which is the node feature in graph G (V,E). We can perturb the feature extraction to derive an LR estimator, i.e., h' = <(Gh0 + 2). where 0 E Rddixda is the weight matrix, G is fixed graph-based matrix for feature aggregation, yp is the activation function, and z is the noise matrix. The attention mechanism introduced by graph attention networks (GATs) (Velickovic et al. 2017) is also supported by ULR inside the module. Given a GAT layer with single-head attention, the extracted features of the i-th node and the attention coefficient of the (i.j)-th edge with injected noises given by
U: how + Eis Us = (Ui, U2) 0 Gi.s
where E Rd and 0 E R2d are trainable weights, & and Gijj are injected noises, and the output features of each node are weighted by normalized attention coefficients among its neighbor- hood. Since gradient estimations in GNNs is analogous to those in previous networks with Theroem
isolate the parametric linear transformation
module, the derivation of ULR
Spiking neural networks (SNNs): Training SNNs suffers from discontinuity in data and activation. Consider an I.-layer SNN with leaky integrate-and-fig neurons. The time series input of 1-th layer at time step t is denoted xfd E IRd The next membrane potential and spike are given by
2 1-11141 ku - ,11+1) + t + +2 2+1, ++11+1 = T(u +,141 Vin),
where dl is the synaptic weight, is the delay factor decided by the membrane time constant, the Heaviside neuron activation function with threshold Vin zdd = (E)) with gai being standard Gaussian random variable. Potentials are integrated into each neuron and will be released once exceeding the threshold Vin which is a discontinuous process and has to be smoothed when applying BP. However, ULR allows
spike
exclude this process from the module and apply Theorem 1 to the remaining portion. The spike signal of the last layer at the last time step, namely TT.L, is the output of SNN and passed to the loss function L. By handling the sequential weight-sharing in similar manner as in RNNs, we can obtain Ve E[[((TTT E[C(xTT) Et= (E') -eny-beee (x*.)))
3.3 VARIANCE REDUCTION
ULR is a simulation-driven algorithm that relies on the correlation between noise and loss evaluation to estimate the gradient. When the noise dimensionality is huge, this correlation would be too weak to be identified. Hence, we propose several techniques based on the hierarchical structure and Monte Carlo (MC) sampling in neural networks to reduce the estimation variance for efficient training.
Layer-wise noise injection: Since ULR does not require recursive gradient computation, it is al- lowed to independently perturb each layer or even each neuron and estimate the corresponding gradient, which ensures that the correlation is sufficient to produce meaningful estimates. Gradi- ent estimates for modules can be combined to form the gradient for the entire network for a single update, or they can be immediately used for module updates. With the layer-wise perturbation tech- nique, the optimization problem in Section 3.1 transformed into a series of sub-problems:
min [[C("")) 0, v + #0. for l 0, LL 1, eee
which can be interpreted performing stochastic block coordinate descent.
5

---

Published as a conference paper at ICLR 2024
Antithetic variable: We employ antithetic sampling to generate the stochastic components in for- ward propagation, which can effectively reduce the correlation between different evaluations, re- sulting in the decline of gradient estimation variance. Assume the integrand in Eq. (2) is repeatedly evaluated for 2N times. We can generate noises for the first N evaluations, and then flip their signs for the rest. The estimator of Eq. (2) with antithetic variables can be written
VeL [((*))
1
Σ (x's
(o d((=)(n))).
(6)
2N
where g²(=) (1)) (z), and (n) is the noise applied in the n-th pair of evaluations.
Quasi-Monte Carlo (QMC): QMC (Sobol 1990) generates noise and evaluates the integrand ing low-discrepancy sequences, which can fill the integration region more evenly than the pseudo- random sequence used in standard scientific computing toolkits and produce better convergence rate. Morokoff & Caflisch (1995) point out that QMC outperforms MC when the integrand is smooth and the dimensionality is relatively small. Given the smoothness of neural networks and the layer- wise perturbation mentioned earlier, it is desirable to try QMC in ULR estimation.
With these variance reduction approaches, the application of ULR becomes practical, and we present the standard version of neural network training through our ULR method as shown in Appendix B
3.4 COMPUTATION GRAPH REARRANGEMENT VIA ULR
BP on high-performance devices usually suffers from deep recursions and huge computation graph scales. Consider L-module deterministic neural network, where the notations are the same in Section 3.1 The output is given by x/+1 = (xt"; 0'), and the BP gradient is calculated as below:
V Cla = TI 2411 ((*)) Var((zt)
Dioe' ++ V
for 0, L- 11-1. Eq. 7 reflects that BP has to be implemented by the auto-differentiation in a recursive manner, as shown Fig. we add extra noise to the L'-th module, L' [0, L). i.e.
x 11+1 of (x +2;0"') where ~ f(-), then we can treat the network as two clusters connected in series and apply Theorem Ito obtain
D Dj D. DT DTina""
VorE(<(*)) D D E [- In
(8) for .......... L' 1. Path-wise derivatives the rest are the same as Eq. 7 As shown in the middle of Fig. 3 Eq. (8) breaks down the single recursion into two parallelizable ones. Analo- gously, we can introduce noise anywhere in networks to adaptively partition the recursive gradient computation into subsets suitable for parallelization split the computation graph by ULR, such as rearranging BPTT computations in RNNs into several parallel mini-BPTT computations to enhance efficiency as shown the last subfigure of Fig,
various devices. More generally, we can
gradient vanishing
Moo
Meduled
Mor
Module
Moddled
BPTT
(K
(
BP computation flow
Rearranged flow by ULR
fonward signal
~
injected noise
backwerd signal
BPTT rearranged by ULR
Figure 3: The left subfigure presents the BP computation flow: the middle demonstrates the ULR rearranged computation flow with injected noise; the right showcases the application of ULR for parallelizing gradient computation in RNNs.
3.5 RELATIONSHIP BETWEEN ULR AND OTHER METHODS
Consider neural module with noise injected into its parameter, i.c., o f(x; + 2), where x and v are the input and output, respectively, is a non-linear operator, and the noise follows density defined on Q. Other modules in the forward
ULR and other perturbation- methods:

---

Published as a conference paper at ICLR 2024
computation abbreviated loss function L. Let St' ULR estimator by pushing 9 into the density as below:
ty y 8 St. We can obtain the
V VE [C(+)] = V. /
f(c 0)dg E - -(() V fzf(2)
E ([
f(2)
which unifies several previous works (Wierstra et al. 2014 Sehnke et al.) 2010 Nesterov & Spokoiny 2017) Salimans et al. 2017) under the framework of push-out LR method.
ULR and reinforcement learning: We first show that the general framework of reinforcement learning (RI. is coherent with the push-out LR method. The RI agent is modeled as a parameterized probability distribution To, and p represents the transition kernel of the environment. Denote the action and state at the t-th time step as a and 9t, where ae ~> (-)+)s)), S1+1 p(-[st,ae) az). The reward is given by R(s, a), which is function of the state and action sequence. Policy-based RL aims solving the optimization: max, E [R(s, a)]. We can treat different each step and push it into the conditional density of a, given s to perform the LR technique, i.e.,
T-1
T-1
VAE R(s a)] = Σ VOE
E [R(s, a)le; s.] (e(E)s))dE = E R(s,0 a)
Σ 1
Vo In (o(o)l)))
1=0
which results in the well-known policy gradient theorem (Williams 1992). We can derive the RL
gradient using the push-out LR and vice versa. Consider the LR-based RNN training discussed earlier, where the input. 2t and hidden state h,-1 can be viewed as the RI state, and the results of the linear transformation ut and i can he viewed as the actions. The parameterized part and the rest in the RNN cell correspond the RL agent and the simulation environment, respectively. And Eq. (5) can be obtained directly from the policy gradient theorem. Therefore, RL and deep learning can be unified from the perspective of the push-out LR framework. RL essentially leverages the advantages of LR to overcome the issue of black-box simulation environment that affects gradient backward propagation. The detailed derivation in this section is presented in Appendix
EVALUATIONS
4.1 VERIFICATIONS ON THE SCALABILITY ACROSS VARIOUS ARCHITECTURES
Experimental settings: Our study focuses on classification tasks with different modalities using various neural network architectures. For image classification, experiment with CNNs (Le- Cun et al. 1998) (ResNet-5 and VGG-8) on the CIFAR-10 dataset (Krizhevsky et al. 2009) and
SNNs ChooshDastidas & Adeli 2009) the MNIST (LeCun) 1998) and Fashion-Mo MNITT (Xiao ct al. 2017) datasets. We use RNN family (RNN (Medsker & Jain 200T). GRU (Dy & Salem 2017 and LSTM (Hochreiter & Schmidhuber] 1997]) classify the articles on the Ag-News dataset (Zhang & Wallace 2015) and use GNNs (GCN and GAT) to classify the graph nodes on the Cora dataset (Sen et al. 2008 While other non-BP methods, including HSIC, FA, and FF, lack the scalability to all architectures above, we compare our method with two baselines (BP and ES) by using ULR with noise injection only on logits (ULR-L) and weight/logits (ULR-WLL in hy- brid manner. Additionally, results larger datasets, well as comparison in CNN training with omitted non-BP methods, including HSIC, FA, and FF, are presented in Appendices C.6 C.7
Evaluation metrics: We evaluate training methods by the task performance and robustness in differ- ent training contexts. The task performance is characterized as classification accuracies on original datasets. The robustness can be assessed through accuracies corrupted noisy data generated by multiple kinds of adversarial attacks or natural noise injection.
Results: As depicted in Fig. while ES fails to optimize several models (ResNet-5 and GAT), ULR substantially achieves a comparable performance to BP on clean samples, with only a minor gap of 0.22% average, which presents efficient gradient estimation performance and optimization stability. Besides, ULR training surprisingly brings a robustness improvement of 9.53% on average compared with BP. Detailed experiment settings and results are in Appendix C
4.2 EVALUATIONS ON THE FLEXIBILITY OF ULR TRAINING
Benefiting from the independence of the gradient estimation inside each module, we can use ULR to rearrange the computation graph, thus achieving lower training consumption. We present two
7

---

Published as a conference paper at ICLR 2024
Roha-d VOD4
RNN
ESTM ORU
CN
GAT
MNIST
TACK
Checitoatta seneeey
Clasitediion conteedioooouu (ocu030
0.8
G6-
93
Chaviifeaaion
0/1
60
0.
L.R.W1.
Figure 4: Classification accuracies of trained models in different contexts on benign and corrupted
noisy data, represented by blank bars (D) and filled bars (E), respectively.
Table 1: Classification accuracy (%) for the domain adaptation task for AlexNet on the Office-31 dataset. We denote + as adapting the model from the source dataset the target dataset t, A as the Amazon domain, D) as the DSLR domain, and W as the Webcam domain. We take the average training time of different models for efficiency evaluations.
Method BP
A-D
A-W
D-+A
D-#
W-+
W-+D
Average time
51.8
58.4
65.6
94.8
68.8
88.8
202
ULR
53.1
60.2
66.9
95.2
70.3
89.4
187
applications, including the training for domain adaption and rearrangement of BPTT in RNNs, to show the training flexibility introduced by our ULR method.
Demo 1: Domain adaptation. Multiple domain adapta- tion methods are proposed to mitigate the domain shifts between the source and target datasets, among which par- tial domain adaption works on the scene that transfers the partial relevant knowledge from a large-scale source dataset to small-scale unlabeled dataset. Li et al. (2020) propose the Deep Residual Correction Network (DRCN), which plugs the residual correction block into the source network to mitigate the domain discrepancy between dif- ferent domains. As shown in Fig. [ training with BP has to traverse the pre-trained model till the inserted part, while ULR can skip the calculation in late layers and directly compute the gradients of inserted blocks for optimization. As shown in Tab. T we use DRCN with both methods to adapt pre-trained AlexNet on three domains of the Office-3T dataset (Saenko et al.) 2010). We compare the accuracies on different domains and time consump- tion to demonstrate the advances of our method. ULR achieves better performance in terms of both the classification accuracy (1.2% 1) and training efficiency (1.1x speedups) compared to BP.
pro: generic
BP
residiaal correztion block
ULR
Figure 5: Computation flows in do-
main adaption with BP and ULR.
Demo 2: BPTT rearrangement. RNN training is constrained by the recursive gradient computation BPTT. Deep recursive computation implies that the parallelism advantages of advanced hardware cannot be utilized, and it can also lead to numerical issues like gradient vanishing. As mentioned in Section 3.4 we perturb the forward propagation of RNNs
and decouple the computation graph into several subgraphs us- ing ULR. Although the introduced randomness training might require extra evaluations to mitigate uncertainty, we achieve con- siderable efficiency gains at a reasonable level of decoupling due all subgraphs. We test decou-
Classilcation Accency
-19)
Lmining Tinse (sippac))
-18h
to the parallelism of BPTT pling via ULR on the Ag-News dataset. While vanilla BPTT ex- pends 45 steps average compute gradients, ULR partitions the original graph into subgraphs with 5-25 steps. As shown in Fig. 6 ULR boosts BPTT in terms of both the accuracy (up 4.9% T) and efficiency (up to 3.7x speedups), especially when cutting the graph into 15-step subgraphs.
RAN
0201
14
23
BFFT
BPTT recuEsion dopth (stop)
Figure 6: Classification accura- cies (curves) and training dura- tions (bars) of BP and ULR.
4.3 ABLATION STUDY
We study the effect of perturbing different parts neurons and variance reduction techniques. In experiments, we use MC sampling generate the noise for experiments except when claim
8

---

Published as a conference paper at ICLR 2024
Coav-2
Conv-3
Conv-4
FC
1.00
1.00
1.00
Coaine gniiacky
Coaiuo dualartt
Coaiuo dniiartt
1.75
0.7
0.75
0.50
8.50
0.50
0.55
0.55
0.25
10
107
10'
107
10'
107
10
of copies
of copies
of copies
(a) Perturbing the logits.
(b) Perturbing the weights.
(c) Hybrid.
1.00
1.00
1.00
Ootbee amaleaty
Ooeiae amillrit
Coeiov siallaiiy
0.75
0.75
0.75
6.30
0.35
0.00
0.00
0.00
copies
# copoes
(d) w/o layer-wise perturbation.
(e) w/o antithetic variables.
(1) with QMC.
Figure 7: Ablation study on the effect of the perturbation different parts of neuron networks and the variance reduction techniques. Ablation study on more aspects can be found in Appendix C.
otherwise. We conduct experiments using ResNet-5 the CIFAR-10 dataset for the ablation study. We use our ULR method to estimate the gradient and compare with that calculated by BP. We compute the cosine similarity to evaluate the performance. The average cosine similarity serve as measure of estimation variance. A similarity closer to indicates a lower variance.
Impact of the perturbation weights and logits: We present the gradient estimation performance when perturbing the logits only in Fig. 7(a) the weights only in Fig. 7(b) weights and logits observed that perturbing the logits results in a higher cosine similarity between the gradients calculated by
in hybrid manner in Fig. [[c]] In comparison between Fig. [T() and Fig. [7bb
ULR and BP for the last three layers, while perturbing the weights yields better gradient estimation with the same number of copies for the first two layers. Being motivated by this phenomenon, we integrate these two perturbing strategies, where we inject the noise on weights for the first two layers and logits for others. As shown in Fig. [(c) the performance of gradient estimation for all five layers has significant improvement compared purely adding perturbation on weights or logits.
Impact of the variance-reduction techniques: In Fig, 7 we study the effect of three pro- posed variance-rednetion techniques. Without the wise noise injection, it difficult to identify the correlation between the noise in each layer and the loss evaluation. As shown in Fig. 7
requires a much larger number of copies to achieve comparable performance on the first two and last layers, and fails to efficiently estimate the gradients of the rest layers. From Fig. 7 it can be seen that without the antithetic variable, there are low similarities between the estimated gradients by ULR and that by BP for all convolutional layers we apply QMC to generate the noise for gradient estimation. Although the results are close to MC, the neuron-level parallelism allowed by ULR, resulting in a lower dimension of MC integration, which implies that QMC may lead to a more significant improvement.
with large number of copies. In Fig. [7()
5 CONCLUSION
In our work, we generalize LR to the most universal training paradigm, enabling the extension to broader range of neural network architectures and exploration of various training pipeline designs. ULR breaks the necessity of the chain rule and only requires forward evaluation during training, thus eliminating the dependency on recursive gradient computations. With variance reduction techniques, our method achieves comparable performance to BP on clean data and exhibits improved robustness. Moreover, we also discuss the relationship between ULR and other perturbation- based methods and unify RL and deep learning from bidirectional perspective of the general LR framework.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grants 72325007, 72250065, and 72022001, as well as the Special Funds for Guiding Local Scientific and Technological Development by the Central Government under Grant 2023EGA035. Z. Zhang and C. Xu were supported by NSF under Grant 2202124 and the Center of Excellence in Data Science, an Empire State Development-desigee
Center of Excellence. Z. Yu was supported by the Beijing Nova Program under Grant 20230484362. The content of the information does not necessarily reflect the position of the Government, and no official endorsement should be inferred.
REFERENCES
Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, 18:1-43, 2018
Rahul Dey and Fathi M Salem. Gate-variants of gated recurrent unit (gru) neural networks. In 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), pp. 1597- 1600. IEEE, 2017.
Samanwoy Shosh-Dastidar and Hojjat Adeli. Spiking neural networks. International Journal of Neural Systems, 19(04):295-300 2009.
Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual net- work: Backpropagation without storing activations. Advances in neural information processing systems, 30, 2017.
Geoffrey Hinton. The forward-forward algorithm: Some preliminary investigations. arXiv preprint arX1v:2212.13345, 2022.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint
2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmistt/ 1998.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFF, 86(11) (6(11))2278-2322 1998.
Shuang Li, Chi Harold Liu, Qiuxia Lin, Qi Wen, Limin Su, Gao Huang, and Zhengming Ding. Deep residual correction network for partial domain adaptation. IEEE transactions on pattern analysis and machine intelligence, 43(7):2329-235 22444429922244 2020.
Timothy P Lillicrap. Adam Santoro. Luke Marris, Colin Akerman, and Geoffrey Hinton. Back- propagation and the brain. Nature Reviews Neuroscience, 21(6):335-346, 2020.
Wan-Duo Kurt Ma, JP Lewis, and W Bastiaan Kleijn. The hsie bottleneck: Deep learning without back-propagation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5085-5092, 2020.
Larry R Medsker and I.C Jain. Recurrent neural networks. Design and Applications, 5:64-67 2001.
William J Morokoff and Russel E Caflisch. Quasi-monte carlo integration. Journal of Computational Physics, (2(2))218-230. 1995.
10

---

Published as a conference paper at ICLR 2024
Yurii Nesterov and Vladimir Spokoiny. Random gradient-free free minimization of convex functions. Foundations of Computational Mathematics, 17:527-560 2017.
Arild Nøkland. Direct feedback alignment provides learning in deep neural networks. Advances in neural information processing systems, 29, 2016.
Yijie Peng, Li Xiao, Bernd Heidergott, Jeff Hong, and Henry Lam. new likelihood ratio method for training artificial neural networks. INFORMS Journal on Computing, 34(1):638-655, 2022.
David E Rumelhart, Geoffrey E. Hinton, and Ronald J Williams. Learning representations by back- propagating errors. Nature, 323(6088):533-536 1986.
Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In Computer Vision-ECCV 2010: 11th European Conference on Computer Vision, Heraklion, Crete, Greece, September 5-11, 2010, Proceedings, Part IV 11, pp. 213-226. Springer, 2010.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as a scalable alternative reinforcement learning. arXiv preprint arXiv:/703.03880 2017.
Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy- based models and backpropagation. Frontiers in computational neuroscience, 11:24, 2017.
Frank Sehnke, Christian Osendorfer, Thomas RückstieBß Alex Graves, Jan Peters, and Jürgen Schmidhuber. parameter-exploring policy gradients. Neural Networks, 23(4):551-559, 2010.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification network data, Al Magazine, 29(3):93 93, 2008.
IM Sobol. Quasi-monte carlo methods. Progress in Nuclear Energy, 24(1-3):55 61, 1990.
Luchuan Song, Bin Liu, Guojun Yin, Xiaoyi Dong, Yufei Zhang, and Jia-Xuan Bai. TACR-Net: editing on deep video and voice portraits. In Proceedings of the 29th ACM International Confer- on Multimedia, pp. 478-486, 2021a.
Luchuan Song, Bin Liu, and Nenghai Yu. Talking face video generation with editable expression. In Image and Graphics: 11th International Conference, ICIG 2021, Haikou, China, August 6-8, 2021, Proceedings, Part III 11, pp. 753-764. Springer, 2021b.
James Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control, (33))3332-41,, 1992.
Inkyung Sung. Bongjun Choi, and Peter Nielsen. On the training neural network for online path planning with offline path planning algorithms. International Journal of Information Manage- ment, 57:102142, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- tion Processing Systems, 30, 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Ben- gio, al. Graph attention networks. stat, 1050(20):10-48555) 48550, 2017.
Tinghua Wang, Xiaolu Dai, and Yuze Liu. Learning with hilbert-schmidt independence criterion: A review and new perspectives. Knowledge-based systems, 234:107567, 2021.
Daan Wierstra, Tom Schaul, Tobias Glasmachers, Yi Sun, Jan Peters, and Jürgen Schmidhuber. Natural evolution strategies. The Journal of Machine Learning Research, 15(1)99499980 2014.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Reinforcement Learning, pp. 5-32, 1992.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnise a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 1708 2017.

---

Published as a conference paper at ICLR 2024
Li Xiao, Zeliang Zhang, Jinyang Jiang, and Yijie Peng. Noise optimization in artificial neural networks. In 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE). pp. 1595-1600. IEEE, 2022.
Ye Zhang and Byron Wallace. A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification. arXiv preprint arXiv:1510.03820, 2015.
Yaoyu Zhu, Zhaofei Yu, Wei Fang, Xiaodong Xie, Tiejun Huang, and Timothée Masquelier. Train- ing spiking neural networks with event-driven backpropagation. Advances in Neural Information Processing Systems, 35:30528-30541, 2022.
12

---

Published as a conference paper at ICLR 2024
A THEORETICAL DETAILS
A.1 GRADIENT ESTIMATION FOR PARAMETERIZED NOISE DISTRIBUTIONS
Notice that 2 follows parameterized noise distribution (:;: where O' O With the property of conditional expectation, we have
E [[(x")] Ver Se E [[c() ('(::')dds
(()
Under mild integrability conditions, e.g.
Sah.. E [|<(x")(s) sup
V (<; < <0,
d'Ee:
can obtain
{ E [c(xa)]ss] V in (C: (S: g
Vel [[x*1]]
(9)
[C(x")Va In fz (
The first equality is justified by the LR technique and the dominated convergence theorem, the second equality comes from definitions and the iterative property of the conditional expectation.
A.2 GRADIENT ESTIMATION FOR CNNS
(Snt-Bin) o"
The conditional density of r given is fz(() = 11 cialized formulation of integrability condition D.i.e..
I
1
With a spe-
a (E)
E
E [[(())]s,="))
sup OR
<0,
08
can obtain
DE [C(w)
V
EE [[(())SS 20) (E)
a0%2 j.k
[5it -00
E
E [[(())ss,
In
Rhoat west
ōgo,s j.k
go
hon
Was
jtskti Ex,t - 5o (e)
E
E [[C(v)]E,
0o
hea
Whn
E [[(())(s x1|
jttkkt
(o,
Rhout Mest
1=0
go
Po 1
E
C(u)
Jockt
(10)
Catt
0=0
2=0
The second equality is justified using the LR technique and the dominated convergence theorem. The fourth equality comes from the change of variables, and the last one holds because of the iterative property of the conditional expectation. Eq. (10) provides neuron-wise gradient estimation for the convolutional layer. After rearranging, we can obtain a channel-wise representation as below:
((u)s' v o0
V T. [[()) = I.
(11)
Using Eq. [T]. we can estimate the gradient of the convolutional layer at once by utilizing the group convolution functionality available in any machine learning toolkit. The gradient estimation for Do is
13

---

Although 9hh participates in the computation of each u. the elements in (",,".. WT) are autocorrelated so that hard to utilize their joint conditional density. Thus, we treat Ohh in up the corresponding gradient estimates. Published as a conference paper at ICLR 2024
Forward Feeding
Gradient Estimation
A(e)
update
X
gAc
X
(Rauy Mexel
(A..M))
Cig Pay Mha)
17
E(()
Dous Auca Mearl
X
(Go Wa)
Figure 8: Forward feeding and gradient estimation in convolutional modules.
corollary of Eq. 10P, and we can derive the gradient estimation for noise magnitudes in a similar manner as Eq. i.e.,
DIE [C(v]]
han
Was
E
((u)
Ob
0
OIE [Cv]
Bon
Was
((v)
((E2,)) -1)
Og
0
The forward propagation and gradient estimation by ULR in CNNs are shown in Fig. 8
A.3 GRADIENT ESTIMATION FOR RNNS
Since some RNN variants, e.g., LSTMs, have extra data flow going through ip without any pa- rameter, we modify the computation formulation generic RNNs in Section (.2 as below:
(he, ct) = (uur, Ut, 1 G2-1), Ue = -0hh 0h-111111110000 + bh 24
Ur 0=th xe + bbth + zt
where denotes another hidden variable. The training objective becomes [. B c)), where h = (h1, ,,,) and c (cI....,cy)
each calculation step distinct variables, and then Notice that the conditional distribution of u, given h4-1 is N(ū, I^h) with the density fc(E). where We unroll the forward pass to perform the push-out LR method below:
ih h bhh
[/ 1Rth
Van E [Ch,, c)
Σ
V Bha
T
E c)C,h, Vann In (.)
E
E (I
1=1
I
E B140 (
L(h, e) (5
Σ
M
(12)
The second equality can be justified by the dominated convergence theorem under similar integra- bility condition as Eq. (T), i.e., for t 1.... T,
E
E [[C(h, c) 11]
sup
Vann fe(E) de <0.
Ohh e xdg
14

---

Published as a conference paper at ICLR 2024
4
Ap-1
Api
RNN
LSTM
GRU
Common Structure
Figure Different RNN modules and their common structure.
By applying the change of variables and the iterative property of the conditional expectation, we can obtain the last equality. The gradient estimates of other parameters in the generic RNN structure can be derived in similar manner Eq. 11 i.e.
T
V [pchE[((h,c) = E
C(h, ((=*)))
1=1
JA [c c)] E
c(h,c) m/
hh
1=1
EpaxE[((hhc) c)]
((h,
yAn. [C(h,
E
L(h, c)
Σ
(2nh - =s (2 ( -(2%h/
2
1=1
Egan 2
E
c(h,c)
c)
Σ
(1 - 3 ( ((E - (I
1=1
As shown in Fig. the three most widely used RNN cells can be unified as the aforementioned common structure, where hidden states and inputs are delivered to non-linear activation function after parametric linear transformation. The modules in the shadows of the first three subfig- ures of Fig. are specific forms yp in different RNN structures. Since sometimes we can push both the weights of hidden states and inputs into shared conditional density, we can simplify the perturbation and gradient estimation in different RNN cells.
In vanilla RNN cells, the forward process with perturbation is given by
h tanh (0hhh1-11 + 0th + bb + bzh + zt),
where 2, = (E)) e is shared among the linear transformations of the hidden states and inputs, and egch is independent standard Gaussian random vector. The gradient estimates of parameters can be reduced as follows:
ghn [C(h)] E L(NE Σ
Ech
T
grn 1 [C(h)] = E (())==
Σ
E
1=1
T
V MA E [c(h)] = yenn E [[Ch))]
= E
((h)
Σ
1=1
T
1 E
Sec, M --
SAA [[(()]]
L(h)
Σ
2
15

---

Published as a conference paper at ICLR 2024
In LSTM cells, the forward propagation with noises is written
ft sigmoid (8* + th h + bbd + bas + 2f),
& = sigmoid (0* x, + gh h bui + b + 2/).
9t = tanh (0*0 + h4 + 114 + g + =.);
Ot sigmoid (0 + / b + 6 =i),
St 0-1 ④ gt: h, 3 tanh (c+),
where gh,, ghi ghg o E TRd xdx and 6es gzi, ,00,00000000000 000 E Rd xd= are weight matrices in different data flows, bb bii bh, tho o bm, (F8,bro E Rdh are bias terms, of = (2/)beff z (E*) kej.
2 (29) kef. 27 = (e el eg. ej and eq are independent standard Gaussian random vectors. The gradient estimates of the parameters in the first equality are given by
T
Vea pas E (E((h,,)) c)] = E [(h, ((h)c)
Σ (
1=1
T
V p= E [((h, = E ((h,c) Σ
c
24
Vn [A(h, V E (2( =E ((h,c)
Σ of
T
1 E. (g = 2
Σ (1))t(()))))++
[(h, c)
The gradient estimation for the remaining parameters can be derived analogously.
The forward computation with perturbation in GRU cells is given by
T. sigmoid (8 z, + g ghrhh-11111111 + bhr + ba + 2[) yt = sigmoid (0 + 0h0 h,-1 + buy 1 =
n4 = tanh (0""xz x4 + 6an + 2t +re @ (Qhn h + hr + 2\²))
h4 = (1 - yt) O ne + yt & h
where Qhr, "rr glvn E TR xdi and Qer f0y,0"m 00n E RRd Xdz are weight matrices in different linear transformations, bhr, bh,,tm thn 6zr,, (rr, ban E Rd are bias terms, 2 (E")&ee, 2 & =t _/v (I &g 2/th (E*m))ee" ET, ey, e/n and Etth are independent standard Gaussian random vectors. The gradient estimates in the first two equalities are the same as in LSTM cells. And the results in the third equality are written as
T
V ahn E [C(A)] = E
C(h)
T
V E [[(h)] E
)
It
Bhn E [C(A)] EE
((N)
T
pm E [[(A)] = EE
((h)(E
2n
T
V SA E [C(A)] = E
C(h)
(hh a (s!om) ( ( (I
1=1
Vyn B [[(h)) E
[(h) Σ
(I
2m (E ( [ (I
1=1
16

---

Published as a conference paper at ICLR 2024
In simplified schemes, the dimensions of the noise injected into RNN, LSTM and GRU cells d, 4dg, and 4dg, respectively, rather than the original 2d,, Sdh, and 6d,, in generic form.
A.4 GRADIENT ESTIMATION FOR GNNS
Denote the loss evaluation of GNNs as [(h'), where the parameters in other modules are abbrevi- ated. For GCNs, assume zi.j follows a distribution with density (5, and let v = 0 2, where Gh. Notice that given h, the j-th column of follows conditional distribution 1 ul and
with the density g5( I - -1 fi,(5: -s.)). Then, we can derive the gradient estimation by pushing ok, into the conditional density, i.e.,
E [[C(h')
E
E
00k5
RIVI
à
E
E (ECOOSE hj
In Sú, - -Et,)
RIVI
30k.j
T=
VI-1
f1, (2.5))
E
E E(C(NIS M)
Σ
Ui.
fi, Un (s(E)dE
RIVI
M-11
(L
VI-1
E
B(C(HOIS,N)
Σ
II fag(S)dC
fig (S.)
JIVI
==0
I Σ 2
E
L(h')
The second equality is justified using the LR technique and the dominated convergence theorem under similar integrability condition Eq. (.. i...
E
E (CN)E.
Ml sup O EQ 00k
9j (5) dg
< 00.
RV
The fourth equality comes from the change of variables, and the last one holds because of the iterative property of the conditional expectation. For GATs, assume & and Gr follow distributions with the densities f (x) (z) and f f (z), respectively, and let Um = (v,, where m = x [V] + j. The gradient estimates be derived analogously that in GCNs, i.e.,
VI-1
1 (Eis)
a Owkki
E [[Ch''] IE
CC(h)
h
fe (EAs))
-1- 1 Σ
(5,((uu) (G,))
E.. (C('))
E
LL(")
Σ
üx
a0
1=0
1=0
A.5 GRADIENT ESTIMATION FOR SNNS
Similar to the derivation for RNNs, and then sum up the gradient estimates. Let jtd = 0'zt.l, Note that conditional on x' yfd gi! + 2dd follows a distribution ( 20) with the density ft. Then, with specialization of the integrability condition (T. i.e., for 1.....12 .
treat 0¹ in each computation step as different variables,
E
sup M xd;
de <0.
22444
17

---

Published as a conference paper at ICLR 2024
unroll the forward computation and obtain
V
Un4t1 E [k( f (L)(E
E2
[[((
V in (t)
T
Σ
E2
E
[(((">)) xnd (E))-1(6 - )zd ( ( (E)
34111
T
E (1.2 (E')= ((x)1)) T TO(C)As)
E
34141
1=1
E
C(x
L.
(Σ)) (()))
And the gradient estimation for Σ' can be given by
VELEEC((PP1
1 = 2
R
L(IT T.L
-
A.6 GRADIENT ESTIMATION UNDER WEIGHT PERTURBATION
By pushing 9 into the density, obtain
VOl [C(O] = Vo / [( - -OYLE
1 BIC(U)IE Vo 0)S(E O)dE
=- EIC(U)EE v.f(z)
f(E OYdE
f(2)
===
I EE [I(CC)CC] v.f(2)
f(C)dc
f(z)
==
v=I(=) E -c() f(z)
where Ω' {y y ?), the second equality holds by applying the LR technique and the dominated convergence theorem under the assumption that
[
E [E[[()]<]] sup Vof(( - 0) de < co, 0
and the rest comes from the change of variables and the iterative property of the conditional expec- tation.
A.7 RELATIONSHIP BETWEEN ULR AND RI.
We have discussed how derive the key result in policy-based RL by the push-out LR method in our paper. We next provide an inverse perspective on the relationship between LR and RL. Let i4 ghh 1/h and i geh In + bah The generic form of RNNs can be rewritten as
= 0o: + Σ tee
st-1 = (az, st),
(13)
where a, (.._ TT 8t AT T and o, [h,,, J serve the action, state and observation in RL context, respectively. The generic RNN cell can be regarded as a partially observable RI. scenario. The RL agent is modeled single neural layer parameterized by 0 and
18

---

Published as a conference paper at ICLR 2024
E, which is parametric Gaussian distribution widely used in classical RL with continuous action space, while the transition kernel of the RL environment is determined by yp The cost/reward signal given by the loss function, which could be episodic feedback like the classification loss, per- step feedback, such as the mean squared error (MSE) loss for generation problem. If the reward is not decomposable, e.g., the classification loss, we obtain the intermediate result in Section A.3 by the vanilla policy gradient theorem, i.c.,
VolE (C(h, c)] = E C(h, c)
Vo Ss(ae)
Σ E
g2d,
where fe(E) = f(C: 0,) ) is the density of conditional Gaussian distribution N (0o 2) Otherwise, have c(h,c) = Ei= 1(h,, c4) and can utilize the "reward-to-go" technique in the policy gradient theorem to obtain a more refined result, i.e.,
T
T
V VoE [C(h, = E
Uhee co Vo felar)
E
10he Sc
Σ
Eli(,, cj)l,, Ve (o()
R2h,
where the last equality is a direct intermediate result by repeatedly using the push-out LR method.
B
PSEUDOCODE
Algorithm Neural Network Training via Unified Likelihood Ratio Method
Input: Network structure (:: { loss function C(-), input x.", noise density (ー)-))-
1: Initialize network parameter (0') 11-
2: repeat
3: for / 0 to L - parallelly do
4: Generate noise z' ~ f-)) with QMC optional.
5: Compute L.! and z. only with z' and -2 injected to the /-th layer, respectively,
6: Update 8 by estimated gradient following Eq. (6) with loss values L(*t.1) and ((
7: end for
8: until network parameter converges.
Output: Network parameter (-')
C EXPERIMENTAL DETAILS
C.1 GENERIC EXPERIMENT SETTINGS
Platform: We conduct experiments in computational platform with PyTorch 1.14.0 and eight NVIDIA RTX A6000 GPUs. Each A6000 GPU has 48 GB memory.
Datasets: In paper, focus on the classification task. We conduct experiments of ResNet- 5 and VGG-8 on the CIFAR-10 dataset, which consists of 60, 000 32 x 32 color images in 10 classes, with 5, 000 images for training and 000 for testing per class. For experiments the RNN family, we evaluate the proposed method on the Ag-News dataset with classes, which consists of 30, 000 in news articles for training and 1, 900 for testing in each class. For the GNN family,
19

---

Published as a conference paper at ICLR 2024
Table Results ResNet-5 and VGG-8.
Natural Noise
Adversarial Noisc
Distribution shift
Models
Alg.
Ori.
Gaussian
Uniform
Poisson
FGSM I-FGSM MI-FGSM
Grey
RanMask
BP
64.4
59.4
60.7
44.2
12.1
1.1
1.7
53.5
54.1
ES
35.1
35.1
35.0
29.5
20.8
4.1
4.9
25.3
34.3
ResNet-5
ULR-L
62.3
61.4
61.9
50.9
26.0
6.4
9.4
55.9
55.5
ULR- WL
64.8
64.2
64.6
55.3
26.2
6.7
9.4
58.2
59.1
BP 79.8
69.4
76.4
37.8
32.4
1.2
0.9
67.8
61.9
ES
63.7
65.4
35.6
25.7
2.7
2.7
61.9
60.5
VGG-8
ULR-L 58.7
50.6
56.2
30.7
27.9
11.5
10.5
48,9
47.6
ULR-Wi WAR-WW
77.3
72.7
76.7
42.9
34.6
2.8
3,5
71.4
64.9
consider the Cora dataset with 7 classes, which consists of 140 nodes for training and 1,000 nodes for testing. For SNN, we conduct experiments on the MNIST and Fasion-MNIST datasets, which consist of 60, 000 28 x 28 grey images for training and 10, 000 images for testing. We conduct the domain adaption experiments on the Office-31 dataset, which consists of 3 domains with image size 224 x 224, including the Amazon, DSLR, and Webcam.
C.2 EXPERIMENTS FOR CNNS
The ResNet-5 has layers, including convolutional layers and 1 fully connected layer. The residual connection is between the third and fourth convolutional layers. For the convolutional layers, we set the number of kernels as 8, 16, 32, and 32, respectively, all the kernel sizes as 3 x with the stride as 1, and the activation function fully connected layers, we set the number of neurons as 10 for classification with 10 classes. For all the compared training methods,
ResNet-5:
ReLU. We use the batch size as 100. For the
the Adam optimizer with the initialized learning rate as I x 10-3, and train the models with 200 epochs. For the ES and ULR training, we set the for each layer initialized 1 10-3 1 10 1 10-1, 1 10-1 1 10-1, respectively. Due to the high gradient estimation variance, we set the number of copies as 1, 000 for all layers in the ES training method for sufficient optimization. We set the number of copies for each layer as 100, 100, 200, 200, and 50 in ULR training methods.
VGG-8: The VGG-8 has layers, including 6 convolutional layers and 2 fully connected layers. For the convolutional layers, we set the number of kernels as 16, 16, 32, 32, 64, and 64, respectively, all the kernel sizes as 3 with the stride as 1, and the activation function as ReLU. We use the batch size as 100. For the fully connected layers, we set the numbers of neurons as 256 and 10, respectively. For all the compared training methods, we use the Adam optimizer with the initialized learning rate as 1 10 and train the models with 200 epochs. For the ES and ULR training, set the for each layer initialized as x 10 x 10 3 x 10 10-3, x 1,000_0 1 x 10 2. P.1x 10 2 1 10-2 and 1 10-2, respectively. Due the high gradient estimation variance, number of copies as 1, 5,500 for all layers in the ES training method for sufficient optimization. We the number of copies for each layer as 100, 200, 400, 800, 400, 200, 100, and 50 in ULR training methods.
the
Evaluation criteria: We evaluate all the methods for CNNs on the following performance criteria: 1) Task performance: the classification accuracy on the benign samples (Ori.); 2) Natural noise robustness: the classification accuracy on the natural noise corrupted dataset, where we adopt the Gaussian. uniform, and Poisson noises; 3) Adversarial robustness: the classification accuracy on the adversarial examples, which are crafted by the fast gradient sign method (FGSM), iterative fast gradient sign method (I-FGSM), and momentum-base FGSM); 4) distribution shift robustness: the classification accuracy on the corrupted dataset with different distribution corresponding to the original dataset, where we transform the colored dataset into the grey images and apply the random mask to images respectively for the construction of out- of-distribution (OOD) datasets. For the adversarial attacks, set the maximum perturbation for each pixel as 8/255, and the maximum number of iterations as for I-FGSM and MI-FGSM. For all the evaluations, consider the corruption the whole testset
iterative fast gradient sign method (MI-
20

---

Published as a conference paper at ICLR 2024
Table 3: Results of the RNN family.
RNN
GRU
LSTM
Attack
BP
ES
ULR
BP
ES
ULR
BP
ES
ULR
Ori.
64.4
65.9
70.9
88.2
84.6
88.9
89.4
85.7
89.4
RanMask
53.4
53.4
56.7
63.9
59.2
65.5
60.1
56.2
62.1
Shuffle
62.7
63.2
69.0
87.1
80.2
88.3
88.9
82.6
89.3
PWWS
50.0
36.5
54.0
58.5
32.0
63.0
59.0
31.5
62.5
GA
37.0
23.5
40.5
56.5
20.0
60.5
59.5
21.0
62.0
Results: As shown in Tab. 2 ULR-WL can achieve comparable performance to BP with an im- provement of 0.4% for ResNet-5 and a minor drop 2.6% for VGG-8. Meanwhile, ULR-WL has a significant improvement in the adversarial robustness of 8.1% on average. In ResNet-5, the clas- sification accuracy relies more on the training of the later layers, where adding noise logits incurs less cost, thus ULR-L outperforms ES and achieves similar performance ULR-WL. In VGG-8, situation is exactly the opposite. And the success of ULR-Wi. CNNs comes from combining proposed variance reduction methods
the strengths of both noise injection modes. Morcover, are also compatible with ES. The accuracies of ResNet-5 and VGG-8 trained by ES can be boosted 46.9% and 67.2 97.2% respectively, indicating the effectiveness of our tricks in variance reduction. However, even with these tricks, ES struggles to optimize neural networks effectively, demonstrating lower capacity in gradient estimation compared to ULR.
C.3 EXPERIMENTS FOR RNNS
For all the studied models, including the RNN cell, GRU cell, and LSTM cell, we use the Glove vector as the pre-trained embedding with the dimension of 100. We set the number of hidden units as 64 and the number of units for classification as 4. We use the Adadm optimizer with the initialized
learning rate as 1 x 10-3, and train the models with 100 epochs. We set the batch size as 10. For both the ES and ULR training methods, we set the number of copies 200.
Evaluation criteria: We evaluate the methods for RNNs on task performance and robustness. For the robustness evaluation, we adopt two corruptions, 1) RanMask, in which we randomly mask a fixed ratio of 90% of the words in the whole article; 2) Shuffle, which random shuffles words in sentences; 3) adversarial attacks, including the probability-weighee genetic algorithm (GA). We set the maximum ratio of word substitutions as 25%. Due to the low efficiency of adversarial text attacks, consider 200 examples for evaluation adversarial attacks and 1, 000 for other perturbations.
word saliency (PWWS) and
The evaluation results of the RNN family can be shown in Tab. Due the gradient vanishing problem RNN for long sentences, the conventional BP method can not train the neural network well and it only achieves accuracy of 64.7%. Without relying the chain-rule computa- tion, the ULR method can mitigate the gradient vanishing problem and improve the performance of RNN significantly by 5.5%. For training GRU and LSTM, the ULR method can achieve comparable performance on the clean dataset. On the other hand, neural network training using the ULR method can achieve better robustness compared with the BP, namely 2.3% under the RanMask corruption, 2.63% under the Shuffle corruption, and 4.0% under the PWWS attack.
Results:
C.4 EXPERIMENTS FOR GNNs
For all the studied GNN models, we use the Adam optimizer with the learning rate x 10-2, Due to the limited size of dataset, we set the batch size the data size, i.e., the whole 140 nodes. We train the models for 50 epochs.
GCN: The GCN network has three layers, with the number of neurons 1433 (input layer). 32, and 7 (output layer), respectively. For the ULR and ES training methods, we set the a as x 10 for all layers and the number of copies 100.
21

---

Published as a conference paper at ICLR 2024
Table 5: Results of SNN.
Table 4: Results of the GNN family.
MNIST
PaSSi
GCN
GAT
Attack
Attack
BP
ES
ULR
BP
ES
ULR
BP
ES
ULR
BP
ES
ULR
Ori.
91.8
74.6
92.3
72.3
58.6
74.6
Ori.
80.4
57.3
75.6
82.2
34,4
81.8
FGSM
42.6
59.5
81.4
38.0
46.2
73.0
R.A.
62.5
48.2
68.9
70.4
30.1
77.3
I-FGSM
31.5
55.2
82.7
23.1
44.6
71.8
Dice
63.2
45.7
65.4
71.3
28.9
74.6
MI-FGSM
40.8
53.4
83.0
21.9
42.3
72.0
The GAT network has 2 layers. The first layer consists of 4 attention heads computing 32 features each (for total 128 features), followed by exponential linear unit activation. The second layer has a single attention head that computes 4 features for classification. For both ES and ULR training methods, 100. For the ULR training method, we set the number of copies as 1.
GAT:
set 1 10- For the ES training method, we set the number of copics to
Evaluation criteria: We evaluate all the methods for GNNs task performance and robustness. For the robustness evaluation, we adopt two adversarial attacks, including randomly injecting fake edges into the graph (R.A.) and disconnecting internally connecting exterally (DICE). For both the R.A. and DICE attacks, we perturb the edge ratio from 10% to 100% in 10% increments com- pared to the original number of edges and report the average results. For all the evaluations, consider the corruption on the whole test set.
Results: As shown in Tab. compared with ES, ULR achieves comparable performance relative to BP, with a performance gap of 4.8% for GCN and 0.4% for GAT on the clean dataset. ULR average of 4.3% improvement for GCN and an average of 5.1% improvement for GAT on the model robustness, Moreover, the GAT training by ULR does not require any additional copy of data, which has similar computational cost compared BP.
further brings
C.5 EXPERIMENTS FOR SNNS
The SNN has three layers with the number neurons as 784, 50, and 10. For the spiking compu- tation, we set the size of the time window as 5, the decay factor 0.5, the threshold as 0.3, and the length as 0.5.
use the Adadm optimizer with the initialized learning rate as x 10-3 and train the models with 200 epochs. We the batch size 100. For both the ES and ULR training methods, we set the number of copies as 200.
Evaluation criteria:
We evaluate the performance of different optimization methods by task perfor- mance and adversarial robustness. We use FGSM, I-FGSM, and MI-FGSM to craft the adversarial examples MNIST and Fashion-MNSIT datasets, where the maximum perturbation for each pixel is 0.1 for all of the attack methods, the max number of iterations is 15, and the step size is 0.01 for I-FGSM and MI-FGSM. For all the evaluations, we consider the corruption on the whole test set.
Results: As shown Tab. 5) ES fails in training SNN, and the ULR method can achieve a higher classification accuracy on clean and adversarial datasets. Specifically, ULR has an average improve- ment of 1.55% on two clean datasets and 44.3% on adversarial examples. The activation function in SNN is discontinuous, which hinders the application of the chain rule-based BP method, and the BP adopted here is based on approximated computation. The unbiasedness of the ULR gradient estimation leads to superior results.
C.6 VERIHICATION ON LARGER DATASET
To give a further scalability evaluation of the ULR method, we train the ResNet-9 to classify images in the CIFAR-100 dataset. We select BP, ES, ULR-L, and ULR-WL as the baselines. We take the supported maximum number of data copies for gradient estimation in ES, ULR-L, and ULR- WL. We present the results in Tab. 6 From the numerical results, it can be observed that while ES fails to optimize network parameters on the large-scale CIFAR- 100 dataset, ULR achieves a comparable performance to BP, with only minor gap of 0.3%.
22

---

Published as a conference paper at ICLR 2024
Table 6: Classification accuracies of ResNet-9 on the CIFAR-100 dataset.
Methods
BP
ES
ULR-L
ULR-WI.
Acc.
65.2
27.3
59.2
64.9
C.7 COMPARISON WITH OTHER METHODS
To comprehensively compare different optimizations, further train the ResNet-5 on the CIFAR- dataset using BP, FF, FA, HSIC, ES, ULR-L, and ULR-WI As shown in Fig. 10 approaches based on stochastic gradient es-
0.8
Claniitiatton
0.44
timation, especially the ULR methods, can achieve comparable performance with BP. ULR-WL surpasses the runner-up method, HSIC, which belongs to another training strategy category, with significant improvement of 9.0%. should be noticed that FF fails to optimize the model due to the weight-sharing issue, which neither ES nor ULR will suffer from during the training process.
0.8
Figure 10: Classification accuracies of
ResNet-5 on the CIFAR-IO dataset us- ing different optimization methods.
C.8 EXPERIMENTS FOR DOMAIN ADAPTION
We use the pre-trained NeexNet in Pytorch for domain adaption experiments and add residual correction block, consisting of two linear layers with ReLU activation function, after the pooling layer. We use the Adam optimizer with the initialized learning rate x 10-4, 4, For all models, we set the training epochs as 50 and the batch size as 64.
C.9 EXPERIMENTS FOR BPTT REARRANGEMENT
We train the RNN and GRU on the Ag- Ne-News dataset verify the performance of BPTT rearrange- ment via ULR. For all the models, we set the learning rate as x 10 with Adam optimizer and train the models for 100 epochs for sufficient convergence. While the average length of texts in the Ag-News dataset is around 45 words, which corresponds to the average length the gradient com- putation graph as 45 steps, we split the original gradient computation graphs into subgraphs by ULR with depths of 5, 10, 15, 20, and 25, respectively, to give comprehensive study and comparison on the rearrangement performance.
C.10 EXTENDED ABLATION STUDY
Following the same setting in our main text, we provide more extensive ablation study on the VGG-8. The details the network structure have been introduced in Section C.2 Without specific nomination, we perturb the network without QMC in a hybrid manner for gradient estimation, where we inject the noise on weights for the first four layers and on logits for other layers.
Fig. (1 (c) demonstrate the impact of the perturbation on logits, weights, and the hybrid of weights and logits. can noted that compared with adding perturbation on logits and weights, the hybrid manner can achieve much better gradient estimation accuracy both in the first and last several layers.
Fig. TT(d) (f) present the impact of the initialization for noise magnitude Following the design of the affrrmentioned hybrid noise injection mode, we perform ULR with set to x 10-1, 1 x 10-2, 2 and 1 10-3 respectively. The optimal choice of varies for different noise injection modes. We can see that x 10 1 is more suitable for initializing a for the first several convolutional layers to achieve acceptable gradient estimation accuracy. In contrast, the last several layers, especially the fully connected layers, are robust to the initialization of a, which maintains high gradient estimation accuracy for all selected values.
23

---

Published as a conference paper at ICLR 2024
Conv-1
Conv-3
Conv-5
FC-1
Comy.4
Cony-8
FC-2
1.00
1.00
1,00
Col Coe eitalartiy
Coatare stailatiy
0.75
0.77
0.73
0.25
0.20
0.20
0.00
102
102
10
102
10P
102
101
10P
of copire
of copies
of cupies
(a) Perturbing the logits.
(b) Perturbing the weights.
(c) Hybrid.
Creine similerit
Orelten smallarit
1,00
Coaice suuilarier
0.75
0.75
0.75
0.50
0.50
0.00
0.00
0.00
0.00
10a1
10a
102
10'
#
capiee
# of copies
of copies
(d) Initializing
x 10
(e) Initializing as 1 10
(1) Initializing 1 x 10
Coeine simllrity
1.00
Cosire Smilorit
1.00
Smalaciyy
1.00
0.50
0.50
0.00
0.25
0.25
0.20
10'
10
10'
10
10*
102
10*
capies
copies
(g) w/o layer-wise noise injection.
(h) w/o the antithetic variable.
(i) with QMC.
Figure 11:
Ablation study on the effect of the perturbation on different parts of neurons, the initial- ization for noise magnitude a, and the variance reduction techniques.
Fig. [i(g) (i) show the impact of the variance- reduuctinr techniques. From the results in the figures, all the proposed methods, including layer-wise perturbation, antithetic variable, and the use of QMC, play an important role on improving the gradient estimation accuracy.
In word, the conclusion from the ablation study on VGG-8 is consistent with that on ResNet-5.
Furthermore, we conduct another ablation study on perturbation adapting, where the magnitudes of injected noise are optimized following the gradient estimators derived in Appendices A.2 and A.3 Under the same setting in Section 4.1 we respectively use ULR with and without perturbation adapting to train ResNet-5 and VGG-8 on the CIFAR-10 dataset, as well as RNN, GRU, and LSTM on the Ae-News dataset. As reported in Tab. neural networks trained by our ULR method.
[ perturbation adapting boosts the performance of
Table 7: Classification accuracies with different perturbing strategies ULR training.
Perturbation Adaptive
Dataset
Model
Fixed
ResNet-5 VGG-8
64.8
61.3
CIFAR-10
77.3
75.9
RNN
70.9
69.2
Ag-News
GRU
88.9
86.6
LSTM
89.4
87.8
24


Reviews:
[{"review_id": "VRSUMdjhkr", "reviewer": "Reviewer_HFF5", "summary": "This paper presents a unified framework for training neural networks with likelihood ratio estimation methods.\nLikelihood ratio estimation has the benefit of avoiding backpropagation for gradient computation, thus leveraging the backward lock.\nSuch optimization methods are thus amenable to greater parallelization, as gradient computation for different modules can be performed independently of each other given the final loss evaluation.\nThe authors combine two variance reduction techniques to make the technique practical.\nThe proposed method is evaluated on a diverse set of experiments covering various deep learning applications.\nThe ablation studies cover most of the critical components of the proposed method.\nThe paper may lack some discussion about the practical limitations but is overall well-written, with some sparse exceptions, and the research is well-conducted.", "questions": "1) I may have missed it, but it's not clear from the text whether all parameters should be perturbed independently. It seems that it would help to have a better correlation between the noises and the loss, but this does not seem to be the case in the experiments.\nIs there a middle ground between only perturbing the logits of the last layer, all layers, or all neurons independently?\n2) Could the authors provide some hints on the current practical limit of the method with respect to model size and dataset?\nFor example, does this method scale for a resnet18 or resnet50 on CIFAR10 or ImageNet?\n3) [Minor question] \"LR has almost no constraints on the model, including differentiability or traceability, enabling explorations of network architectures.\" --> Could you explain how it enables explorations of network architectures?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) The paper is dense, with all the details needed to implement the proposed method. In particular, gradient estimators for the parameters of ubiquitous neural architecture components are provided.\n2) The proposed framework offers large flexibility with respect to the practical implementation, e.g. for model parallelism.\n3) The proposed set of experiments covers a diverse range of applications in deep learning. The ablation study in the main body covers important aspects of the proposed method, such as the type of perturbation, as well as the practical relevance of the variance reduction techniques employed. Many additional experiments are provided in the appendix.", "weaknesses": "1) I don't understand theorem 1. It seems that the function $g^l$ is implicitly defined in the proof. But I don't understand if this translates into a condition on $f^l$ in the hypothesis of the theorem, and if so, why the condition does not directly involve $f^l$. The function $g$ is not a component of the gradient estimator, so I don't understand why it is present in the hypothesis. Also, the variable $\\xi$ seems to represent $z^l$ (or ($z^{l+1}$) and I'm not sure what this condition implies.\n2) Does the star in equation (3) represent a convolution or a term-by-term product?\n3) The set of experiments chosen are a bit simple. While it is, for example, promising to produce good classification accuracy for MNIST and CIFAR10, the question of scaling these methods to larger models or more complex datasets is not being discussed.\n4) [Minor comments] \"Since these methods change the classical training paradigm or even give up the\ngradient information. Many algorithmic or hardware technologies developed based on BP by predecessors cannot be fully leveraged, making such approaches computationally unfriendly.\" this might have been a single sentence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mmDQ4TMNCS", "reviewer": "Reviewer_hjbZ", "summary": "This paper proposes a unified likelihood ratio (ULR) method to avoid the recursive computation of backpropagation. By imposing noise to input data in forward propagation, ULR can approximate the gradient independently for each layer in backpropagation. The authors show how ULR can be applied to architectures such as DNNs, CNNs, RNNs, GNNs, and SNNs, and also introduce techniques to reduce the variance of ULR. The proposed method not only performs comparably to backpropagation, but is also more efficient than backpropagation in some highly recursive computationally demanding tasks.", "questions": "Why did you plot the experiments in Figures 7 and 11 separately for each layer? What observations can we make there?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper proposes a unified likelihood ratio (ULR) method to avoid the recursive computation of backpropagation. By imposing noise to input data in forward propagation, ULR can approximate the gradient independently for each layer in backpropagation. The authors show how ULR can be applied to architectures such as DNNs, CNNs, RNNs, GNNs, and SNNs, and also introduce techniques to reduce the variance of ULR. The proposed method not only performs comparably to backpropagation, but is also more efficient than backpropagation in some highly recursive computationally demanding tasks.", "weaknesses": "Due to layer-wise noise injection, which is one of the proposed variance reduction techniques, ULR has a forward propagation cost that is proportional to the number of layers. When applied to deep networks with many layers (ResNet-152, BERT-large, etc.), it may have a high computational cost compared to backpropagation. The NNs used in the experiments have a small number of layers (ResNet-5, VGG-8), which does not address this concern.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lat26OLX8I", "reviewer": "Reviewer_s8CG", "summary": "This work proposes the Unified Likelihood Ratio (ULR) a method that allows for more efficient neural network optimization. ULR is based on the old idea of zero-order optimization with likelihood ratio, reinterpreted in way that allows for optimising neural networks while avoiding back propagation, either entirely or some parts of it. By adding noise to either the activations or the weights, with the likelihood ratio method one only needs the loss (which can even be non-differentiable) evaluated at the noisy inputs in order to obtain a gradient for the terms that contributed to the noisy input (instead of the gradient of the loss itself with respect to the noisy input). The authors discuss how this idea generalises to various architectures, while also discussing three specific techniques that reduce the variance of the gradient estimates, which is very important for the practical success of a zero-order method. The authors evaluate ULR on several tasks and architectures while also performing an ablation study on their proposed variance reduction techniques and the impact of the perturbations on various points in the network.", "questions": "I find the overall paper interesting, especially the computation graph rearrangement part. Having said that, ULR seems to also be a combination of existing ideas and techniques, so it is not entirely novel. However, given the relatively good results, I am leaning towards acceptance. My questions and suggestions are the following\n\n- Before section 3.2, the authors discuss about adaptive perturbations by optimising the noise, however, as far as I understand, this is not used anywhere in the paper. How is the performance with adaptive noise?\n- On the RNN side, it seems that the authors add noise on each step, which can compound over time. Why is this strategy preferred compared to, e.g., just adding noise to the weights and evaluating the entire sequence on these noised weights (e.g., as done for convolutional layers). \n- I would suggest the authors to incorporate the variance reduction techniques to ES as well, in order to better highlight what the delta with ULR is. \n- At Figure 6, it is unclear what are the BP and ULR metrics. \n- At Figure 7 what is the reference BP gradient compared against? The true gradient of the noised loss (i.e., the one that includes the noise from ULR) or the gradient of loss without any noise (which is the one we care about)? \n- At the discussion of Figure 5 and Demo 1, the authors argue that the ULR can skip the calculation in late layers, however, this is not entirely true as ULR still needs the forward pass through those layers to compute the loss value. \n- Finally, the accuracy of the base BP model on CIFAR 10 is kind of small; do the authors use data augmentations?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Simple method that works well in practice and tackles the practically relevant problem of efficient neural network optimization, especially given the recent successes of large models. The framework is quite general and the computation graph rearrangement via ULR is especially interesting.\n- The experiments are quite diverse and informative.\n- The paper is generally well-written.\n- The variance reduction techniques highlighted, although quite standard, seem to help quite a bit in practice", "weaknesses": "- The work has a misleading claim; the authors argue that one forward pass is enough however in practice multiple forward passes are used in order to estimate the gradient (to reduce variance / antithetic variables). I would suggest that the authors rephrase the claim appropriately. \n- The extension of LR to various network architectures is straightforward and not particularly novel.\n- As far as I understand, the variance reduction techniques discussed at 3.3 seem to not be applied on the baselines, e.g., ES, so it is unclear where to attribute the gap in performance\n- The architectures and settings are a bit toy; it would be interesting to see how ULR performs on much larger architectures (e.g., ResNet50 and Imagenet or GPT-2 on a reasonably sized text dataset)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XT7K3MQvP1", "reviewer": "Reviewer_3HaG", "summary": "This paper proposes a unified likelihood ratio (ULR) method that extends the existing likelihood ratio (LR) method to deal with neural network models other than MLPs, including CNNs, RNNs, GNNs, and SNNs. ULR injects noise into the output of each layer, which they call a logit, to estimate the gradient of parameters in the layer instead of using the usual back-propagation (BP), which is theoretically justified by the push-out technique proposed in the LR paper.\nSeveral techniques are proposed to reduce the variance of gradient estimation by noise injection, such as layer-wise injection, antithetic variables, and quasi-Monte Carlo.\nExperimental results suggest that ULR allows the training of (relatively shallow) neural networks whose performance is comparable to the model trained with BP.\nIn addition, models trained with ULR are shown to be more robust to inputs corrupted by adversarial or natural noise.", "questions": "- As in the title of the paper, it is claimed that ULR requires only one forward computation to estimate the gradient. However, following Algorithm 1 in the paper, it seems to me that ULR requires additional multiple forward computations with respect to $z^l$, whose size is specified in the appendix as ranging from 50 to 800, to compute the loss values. If this is true, I wonder if ULR really reduces the computational effort compared to BP.\n\n- In section 3.1, it is suggested to optimize the noise distribution parameter. But it seems a bit strange to me to optimize the noise distribution with respect to the loss value. Instead, I think the noise distribution should be optimized to improve the gradient estimation.\n\n- In Figure 4, does ULR-WL for ResNet-5 mean the hybrid model explained in Section 4.3? What about the details of ULR-WL for VGG-8?\n\n- In page 4: $b \\in \\mathbb{R}^{c_\\text{out}}$ -> $b^O \\in \\mathbb{R}^{c_\\text{out}}$", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The existing LR method is extended to deal with various NN architectures, including CNNs, RNNs, GNNs, and SNNs.\n- Variance reduction methods are proposed and empirically shown to be effective.\n- Experiments are performed on CNNs, RNNs, GNNs, and SNNs where ULR is compared to BP.", "weaknesses": "- It is difficult, at least for me, to follow the technical details of the proposed ULR for the following reasons:\n    - In Section 3.1, several expectations ($\\mathbb{E}$) appear to explain the proposed method. But I could not understand in which variables and distributions these expectations are defined.\n    - Conditional expectations such as $\\mathbb{E}[\\mathcal{L}(x^L)|\\xi, x^l]$ are also difficult to understand exactly.\n    - What is $J_{\\theta^l}$ in Eq.(2), which is not defined in the manuscript? Jacobian of $x^{l+1}$ with respect to $\\theta^l$?\n- In experiments, ULR is only applied to relatively shallow NNs, such as ResNet-5 or VGG-8.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)