US 2024/0028879 Al

SYSTEM AND METHOD FOR PARAMETER
MULTIPLEXED GRADIENT DESCENT

CROSS-REFERENCE TO RELATED,
APPLICATION

[0001] This application claims the benefit of priority from
USS. Provisional Patent Application Ser. No. 63/368,800,
filed on Jul. 19, 2022, the disclosure of which is incorporated
herein by reference in its entirety.

STATEMENT REGARDING FEDERAL RIGHTS

[0002] The invention described herein was made with
United States Government support from the National Insti-
tute of Standards and Technology (NIST), an agency of the
United States Department of Commerce. The United States
Government has certain rights in the invention.

FIELD OF THE INVENTION

[0003] The present invention relates generally to neural
networks, and more particularly, to machine learning algo-
rithms for training neural networks.

BACKGROUND OF THE INVENTION

[0004] Artificial neural networks are increasingly being
used as preferred architectures for many computational
applications. Mathematical representations of neural net-
works have been implemented in software with some suc-
cess. Software-implemented neural networks are flexible in
that they can be “trained” to solve many different problems
but often at a significant energy cost associated with both
training and operation. A common method of implementing
a high-performance neural network in a hardware is to train
a specific network for a specific task, and then hard code that
solution directly into the hardware. While this technique can
produce high computing efficiency for a particular imple-
‘mentation, it results in a subsequent inability to reconfigure
the network by changing weights, biases, or interconnec-
tions between neurons, or by adding or removing neurons.
Furthermore, this technique often results in lower accuracy
performance than anticipated due to device variability
‘Accordingly, there are many unsolved technical barriers in
machine learning and neural networks and there is interest in
hardware specifically built to perform machine learning,
e.g., at faster rates or using less energy than other technol-
ogy. These hardware machine learning systems are some-
times called neuromorphic systems.

[0005] Machine learning works in phases that include
training and inference. In the training phase, a model is
provided with a curated dataset so that it can learn to extract
the desired information from the type of data it will analyze.
‘Then, in the inference phase, the model can make predic-
tions based on live data to produce results. However, in
hardware implementations, the training ph: n be difli-
cult to accomplish and is ineflicient on traditional digital
hardware. This has led to significant efforts toward building
custom hardware that can perform machine leaning tasks at
high speeds with lower energy costs. There are hardware
platforms using analog, digital, or mixed-signal processing
that potentially offer increased operational speeds and/or
reduced energy costs. However, such hardware instantia-
tions only perform the inference part of the machine learning
algorithm, and a larger portion of the energy cost is spent
training on datasets, typically via gradient descent. Back-

Jan. 25, 2024

propagation is a commonly used method of computing the
gradient for gradient descent but is challenging to implement
in hardware platforms. Training via gradient descent does
not require backpropagation; backpropagation is only used
to calculate the gradient. Other methods for computing the
gradient in neural networks exist but are less efficient in
software than backpropagation and are seldom used in
machine learning applications.

[0006] Model-free methods that do not require knowledge
of the internal structure of the network (e. topology,
activation function, derivatives, etc.), having the capability
to perturb network parameters and measure network
response, and that can be used to efficiently train modem
neural network architectures are of particular interest. In one
ifference model-free method has been
-the-loop training. However, the require-
ments for extra memory at every synapse and global syn-
chronization in finite-difference model-free method and
other such disadvantages prevent its widespread implemen-
tation in hardware. Other model-free perturbative methods
for neural networks have been investigated but are limited in
scale and comprise small datasets with only a few neurons.
[0007] Accordingly, there is a need for a framework for
implementing model-free perturbative methods in neuro-
morphic hardware platforms. There is a need for model-free
methods for perturbing neural network hardware parameters
and measure the neural network response without requiring
a knowledge of the internal structure of the network.

SUMMARY OF THE INVENTION

[0008] Embodiments of the present invention relate to
systems and model-free methods for perturbing neural net-
work hardware parameters and measuring the neural net-
work response that are implemented natively within the
neural network hardware and without requiring a knowledge
of the internal structure of the network. Embodiments of the
present invention also relate to systems and methods for
configuring neural network hardware such that the network
automatically performs parameter multiplexed gradient
descent, which include adding a time-varying perturbation
to each hardware parameter base value to modulate the cost,
broadcasting the modulated cost signal to all hardware
parameters, and filtering out modulations so as to extract
gradient information.

[0009] Embodiments of the present invention relate to a
multiplexed gradient descent system for training a neural
network implemented in a neuromorphic hardware, said
system including an input layer comprising a first plurality
of neurons configured to receive a plurality of input signals;
a plurality of synaptic circuits for modulating at least one of
a first plurality of neuromorphic hardware signals, wherein
each of the plurality of synaptic circuit comprises a plurality
of neuromorphic hardware elements for generating the at
st one of the first plurality of the neuromorphic hardware
signals, wherein the plurality of the neuromorphic hardware
elements comprises a first plurality of neuromorphic hard-
ware parameters for setting the modulation of the at least one
of the first plurality of the neuromorphic hardware signals to
a predetermined value; a second plurality of neurons for
generating a second plurality of neuromorphic hardware
signals from the modulated first plurality of the neuromor-
phic hardware signals, wherein each of the second plurality
of the neuromorphic hardware signals is a nonlinear function
of the at least one of the first plurality of the neuromorphic



Patent Application Publication

US 2024/0028879 Al

Jan. 25, 2024 Sheet 13 of 18

wm 2-bit parity
a 4-bit parity
om NIST?x7 dataset

10 103 108

10° 103
Time

FIG. 13


US 2024/0028879 Al

cost and C[t] is the cost at timestep t. In embodiments of the
present invention wherein perturbations 6(t) from perturba-
tor 10a have a continuous pattern, perturbator 110a modu-
lates parameter @, at a frequency @, and amplitude A@ to
generate sinusoidal perturbation 6,(t)=A0 sin(«,t). Modulat-
ing each parameter 0; changes output y(t) and, in turn,
changes cost C. Modulating parameter 0, by frequencies ,,
©,, © and so forth will result in cost modulations C(t) added
to the baseline (unperturbed) cost Cy such that cost

CWH=Cy+CI=CHHEAC, sin((Og), oO

[0043] wherein C(t) is the time-varying cost function,
and AC, is the amplitude change in cost C(t) due to
perturbation 6,(t) of parameter 8,.

[0044] Filter 106 extracts the time-varying modulation
€() in cost C(t) due to perturbation 6,(t) of parameter 8,. In
one embodiment of the present invention, filter 106 is a high
pass filter,
[0045] Transmitter 108 transmits the time-varying modu-
lations C(t) extracted by filter 106 to all optimizers 110. In
one embodiment of the present invention, transmitter 108 is
a wireless transmitter. In another embodiment of the present
invention, transmitter 108 is a wired transmitter. The time-
varying modulations C(t) transmitted by transmitter 108 is
received by a receiver 110b included in optimizer 110.
[0046] To perform gradient descent on cost C(t), the
gradient dC/d® must be calculated and parameter 8, adjusted
to minimize cost C(t). The gradient dC/d@ is composed of
partial gradients 8C/38, such that dC/d0=(3C/98,, 3C/30., .
.. ), the estimation of this gradient in neuromorphic hard-
ware is denoted by G.
[0047] The time-varying modulation C(t) transmitted by
transmitter 108 includes contributions from parameters
other than the ith parameter. Each parameter 8, can compute
its own partial derivative 9C/90, and autonomously update
itself. For each parameter @,, the contributions from other
parameters to the time-varying modulation e (t) transmitted
by transmitter 108 are unwanted. When perturbation 0,(t) are
small in amplitude and approximately orthogonal, the
unwanted contributions from other parameters can be fil-
tered out by integrating the product of its local perturbation
6(t) and global modulation C(t) the parameter receives.
[0048] Optimizer 110 further includes a correlator 110c for
extracting AC; from the time-varying modulations C(t)
transmitted by transmitter 108 and received by a receiver
110b. Correlator 110c extracts AC; by integrating the product
of perturbation 6,() and modulation C() extracted by filter
106. The product C(1)6,(t) is referred to as error signal e,(t).
[0049] To ensure the magnitude of the perturbation does
not affect the magnitude of the error, correlator 110c nor-
malizes the product €(1)6,(t) by the square of the amplitude
of the perturbation 6,(t) and eliminates unwanted perturba-
tions (frequencies) from other parameters via the following
integration.

c

wherein G, is an approximation for the partial gradient for
parameter 6,. FIG. 3 shows a schematic representation of
correlator 110c for continuous error computing and integra-
tion in accordance with embodiments of the present inven-
tion. Correlator 110c continuously computes the error signal

@
Con -
40?

Jan. 25, 2024

e\(t) and integrates over time to build up an approximation
of the partial gradient G, for the parameter @,. This integra-
tion produces the partial gradient approximation Gj AC/
A@,. A longer parameter integration time provides better
gradient approximations when the parameters are updated.
[0050] In one embodiment of the present invention, the
integration takes the form of a homodyne detection, where
unwanted perturbations (frequencies) from other parameters
are eliminated via the following integration

[0051] wherein 1/A@,? is a normalization constant. G, is
an approximation for the partial gradient for parameter
6,, and approaches the exact gradient in the double limit
as T-9e0 and the amplitude of perturbation A060.
[0052] Updater 112 uses the accumulated gradient
approximation G, to determine a change in hardware param-
eters 6 and update hardware parameters 8 according to a
gradient descent step 8,6-G,, where 1) is the learning
rate. This includes determining updated perturbations 6,(t)
that perturbator 110a can apply to parameter 0,.
[0053] PMGD systems and methods in accordance with
the present invention can be adapted to apply any collection
of orthogonal and mean zero perturbations, including a
variety of analog and discrete perturbations. In embodiments
of the present invention wherein the perturbations are dis-
crete, correlator 110c determines accumulation of gradient
approximation by a summation of the error signal e,{t] in
each discrete time step t as follows:

Gl=Gl-l}teL4, 4

[0054] wherein e,{t]=C[t]0,{t1/A0,. After time to, updater
112 updates 0,°->6,°-G,{t], where 1 is the learning rate,
using accumulated gradient approximation G,[t] and resets
Git] to zero.

[0055] In embodiments of the present invention wherein
the perturbations are continuous perturbations (analog sys-
tem), correlator 110c determines accumulation of gradient
approximation G,(t) as follows:

G=lp\(eAs)-GAs)/todds, ©)

[0056] wherein e,(t)=C(1)0,(1)/A®,, To is the gradient inte-
gration time, and G,(t) is not reset to zero. After time To,
updater 112 updates 6,°-6,°-G,(0), wherein 1 is the learn-
ing rate, using accumulated gradient approximation G,().

[0057] Perturbator 1102 perturbs and updater 112 updates
all the parameters simultaneously such that the resulting
parameter update corresponds to gradient descent training of
the entire network. Because all the parameters are perturbed
and updated simultaneously, the gradient descent training in
accordance with embodiments of the present invention is
referred to as multiplexed gradient descent.

[0058] Perturbator 110a determines a timescale 1, over
which perturbations occur. In embodiments of the present
invention wherein the perturbations are discrete perturba-
tions (digital perturbations), perturbator 110a updates per-
turbations of each parameter to new values every timescale
1, In embodiments of the present invention wherein the
perturbations are continuous perturbations (analog pertur-
bations), perturbator 110a determines a timescale t,, corre-
sponding to the characteristic timescale of the perturbations.


Patent Application Publication Jan. 25, 2024 Sheet 4 of 18 US 2024/0028879 Al

a

Sb
a]
7
i]

@ 4 .
Time

FIG. 4


US 2024/0028879 Al

hardware signals; a third plurality of neurons for generating
a plurality of output signals from the second plurality of the
neuromorphic hardware signals, wherein the plurality of the
‘output signals represent a prediction of the neural network in
the neuromorphic hardware; a cost element for comparing
the plurality of the output signals with a target output to
generate a plurality of costs, wherein comparing the plural-
ity of the output signals with the target output comprises
applying a plurality of cost functions to the plurality of the
output signals and the target output, wherein each of the
plurality of the cost function is a measure of correspondence
between at least one of the plurality of the output signals and
the target output; a filter for extracting a plurality of modu-
lated cost functions, wherein extracting the plurality of
modulated cost functions comprises determining a plurality
of modulations in the plurality of the costs; a transmitter for
‘transmitting the plurality of the modulated cost functions to
the first plurality of the neuromorphic hardware parameters;
an optimizer in at least one of the plurality of the synaptic
circuits, including a perturbator for applying a perturbation
to at least one of the first plurality of the neuromorphic
hardware parameters, wherein applying the perturbation
modifies the first plurality of the neuromorphic hardware
parameters to a second plurality of neuromorphic hardware
parameters; a receiver for receiving at least one of the
plurality of the transmitted modulated cost functions; and a
correlator for extracting a partial cost gradient from the at
least one of the plurality of the received modulated cost
functions, wherein extracting the partial cost gradient from
the at least one of the plurality of the received modulated
cost functions comprises determining an error signal for at
east one of the second plurality of the neuromorphic hard-
ware parameters, wherein determining the error signal for
the at least one of the second plurality of the neuromorphic
hardware parameters comprises applying a multiplier signal
to each of the plurality of the received modulated cost
functions to correlate the plurality of the received modulated
cost functions with the second plurality of the neuromorphic
hardware parameters; and an updater in at least one of the
plurality of the synaptic circuits for determining a parameter
change for the at least one of the second plurality of the
neuromorphic hardware parameters from the extracted par-
tial cost gradient and updating the at least one of the second
plurality of the neuromorphic hardware parameters with the
parameter change to generate a third plurality of neuromor-
phic hardware parameters. More particularly, the perturba-
tion is a time-varying perturbation.

[0010] In one aspect of the present invention, the pertur-
bation is a discrete perturbation. In one embodiment, the
perturbation is time-multiplexing. In another embodiment,
the perturbation is code-multiplexing.

[0011] In another aspect of the present invention, the
perturbation is an analog perturbation. In one embodiment,
the perturbation is frequency multiplexing.

[0012] Another embodiment of the present invention
relates to a multiplexed gradient descent method for training
a neural network implemented in a neuromorphic hardware,
the method including receiving a first plurality of input
signal from an input layer comprising a first plurality of
neurons; modulating at least one of a first plurality of
neuromorphic hardware signals generated by at least one of
a first plurality of hardware elements in ai least one of a
plurality of synaptic circuits, wherein the at least one of the
first plurality of neuromorphic hardware signals is modu-

Jan. 25, 2024

lated to a predetermined value set by a first plurality of
neuromorphic hardware parameters; applying a first pertur-
bation to each of the first plurality of the neuromorphic
hardware parameters, wherein the applying the perturbation
modifies the first plurality of the neuromorphic hardware
parameters to a second plurality of neuromorphic hardware
parameters; generating at a second plurality of neurons a
second plurality of neuromorphic hardware signals from the
modulated first plurality of the neuromorphic hardware
signals, wherein each of the second plurality of the neuro-
morphic hardware signals is a nonlinear function of the at
least one of the modulated first plurality of the neuromorphic
hardware signals; generating at a third plurality of neurons
a plurality of output signals from the second plurality of the
neuromorphic hardware signals, wherein the plurality of the
output signals represent a prediction of the neural network in
the neuromorphic hardware; comparing at a cost element the
plurality of the output signals with a target output to generate
a plurality of costs, wherein comparing the plurality of the
output signals with the target output comprises applying a
plurality of cost functions to the plurality of the output
signals and the target output, wherein each of the plurality of
the cost function is a measure of correspondence between at
east one of the plurality of the output signals and the target
output; extracting a plurality of modulated cost functions,
wherein extracting the plurality of the modulated cost func-
tions comprises determining a plurality of modulations in
the plurality of the costs; transmitting the plurality of the
modulated cost functions to the second plurality of the
neuromorphic hardware parameters; receiving in at least one
of the plurality of the synaptic circuits at least one of the
plurality of the transmitted modulated cost functions;
extracting in at least one of the plurality of the synaptic
circuits a partial cost gradient from the at least one of the
plurality of the received modulated cost functions; deter-
mining in at least one of the plurality of the synaptic circuits
a parameter change for the at least one of the second
plurality of the neuromorphic hardware parameters from the
extracted partial cost gradient: updating in at least one of the
plurality of the synaptic circuits the at least one of the second
plurality of the neuromorphic hardware parameters with the
parameter change to generate a third plurality of neuromor-
phic hardware parameters; updating the first perturbation to
a second perturbation afier a first predetermined time period;
repeating the extracting the partial cost gradient from the at
least one of the plurality of the received modulated cost
functions for a second predetermined time period; and
receiving a second plurality of input signals and a second
target output to the neuromorphic hardware after a third
predetermined time period. More particularly, the perturba-
tion is a time-varying perturbation. In one embodiment, the
perturbation is time-multiplexing. In another embodiment,
the perturbation is code-multiplexing. In yet another
embodiment, the perturbation is frequency multiplexing.

[0013] In one aspect of the present invention, extracting
the partial cost gradient from the at least one of the plurality
of the received modulated cost functions comprises deter-
mining an error signal for the at least one of the second
plurality of the neuromorphic hardware parameters, wherein
determining the error signal for the at least one of the second
plurality of the neuromorphic hardware parameters com-
prises applying a multiplier signal to each of the plurality of
the received modulated cost functions to correlate the plu-



US 2024/0028879 Al

inputs x(t). A cost C is determined from output y(t) and
target Yrarger(t) at an operational step 1114, and a change in
cost C, or modulation, due to perturbations 6 is computed at
operational step 1116. At operational step 1118, an error
signal e; is determined by integrating a product of perturba-
tion @ and modulation C. PMGD method 1100 updates
gradient approximations G, at operational step 1120, and
updates hardware parameters 8 at operational step 1122.
[0068] Reference to the specific examples which follow
and included herein are intended to provide a clearer under-
standing of systems and methods in accordance with
embodiments of the present invention. The examples should
not be construed as a limitation upon the scope of the present
invention.

Example, Simulation of Parameter Multiplexed
Gradient Descent (PMGD)

[0069] Simulations were performed on modern machine
learning datasets to characterize the utility of a PMGD
method in accordance with embodiments of the present
invention. A goal of the simulation was not to perform
gradient descent as fast as possible on a CPU or GPU, but
rather to emulate hardware implementing PMGD and evalu-
ate its potential performance in a hardware context. In
particular, the simulation estimated the speed, accuracy, and
resilience to noise and fabrication imperfections. The simu-
lator was written in the Julia language and can be run on a
CPU or GPU. Algorithms used in the simulation are pro-
vided in Table 1 and Table 2. The parameters and variables
used in the simulations are provided in Table 3.

TABLE |

Algorithm 1 Discrete algorithm

Initialize parameters @

1

2: for n in num iterations do

3: if (n mod , = 0) then

4: Input new training sample %, Ysargr

5 if (n mod , = 0) oF (n- mod t) = 0) then

6 Set perturbations to zero 6 — 0

7 Update baseline cost Cy = CIF (%; 8), Yearser

8: if (n mod 1, = 0) then

9: Update perturbations 6

10 Compute output y < f (x; 8 + 8)

iL Compute cost C= CUY, Years)

12: Compute change in cost = C= Cy

13: Compute instantaneous error signal ¢ — CO/A?

14 Accumulate gradient approximation G — G+e

Is if (n mod t= 0) then

16: Update parameters @ — @— nG

17. Reset gradient approximation G <0
TABLE 2

Algorithm 2 Analog algorithm

Initialize parameters ®
for t= 0 to T step dt do
if (¢ mod t, = 0) then
Input new training sample X, Yrarer
Update perturbations 6 _
Compute ouput y — £ (x; 8 + 6)
Compute cost Cli) — CY, Yaases)

Compute discretized highpass
ie (Cu~ dt) +C@~Cw~-Ce-do)

SIOeEeRe

Cae
Tip

Jan. 25, 2024

TABLE 2-continued

Algorithm 2 Analog algorithm

9 Compute instantaneous error signal e(t) — COdvA@?
10: Update gradient approximation
Gn tens Boa
Om Le + Fou do
Mm Update parameters @ & 8 — G

TABLE 3

Description Symbol Analog or Digital
Change in the cost due to perturbation both
Perturbation to parameters é both
Parameters 6 both
Input sample x both
Target output g both
Network output y both
Cost c both
Unperturbed baseline cost Co digital
Gradient approximation G both
Instantaneous error signal e both
Leaming rate 7 both
Perturbation amplitude AB both
Input-sample change time constant 7 both
Parameter update time constant t both
Perturbation time constant % digital
Highpass filter time constant Tp analog

[0070] Equivalence to Backpropagation

[0071] A 2-bit parity problem was solved by training a
-1 feedforward network with 9 parameters (6 weights, 3
biases) to verify whether the simulation is capable of mini-
mizing the cost for a sample problem, and that it is equiva-
lent to gradient descent via backpropagation with appropri-
ate parameter choices. The simulation was performed using
alarge value for To and T=, such that a good approximation
of the gradient in G for each training sample is achieved. The
simulation was repeated using ty=1 such that the gradient
approximation G for each sample was relatively poor. FIG.
12 illustrates exemplary measurement data for the number of
epochs and the amount of time (number of iterations of the
simulation) for the two experiments.

[0072] A comparison of the plots in FIG. 12A show that,
at Ty=t,=1000, the system in accordance with an embodi-
ment of the present invention follows a training trajectory
that is nearly identical to the trajectory for backpropagation.
For each sample shown to the network, the gradient approxi
mation G has 1000 timesteps to integrate an accurate esti-
mate that should be very close to the true gradient computed
by backpropagation. When T)=t,=1, however, each sample
only has a single timestep to estimate the gradient before
moving on to the next sample. As a result, the samples must
be shown to the network a greater number of times to
minimize the cost, resulting in a much larger number of
epochs. However, while the t=0,=1 case uses the sample
data less efliciently (requiring more epochs), there is a
tradeoff for data efficiency and run time. A plot of the cost
versus iterations, as shown in FIG. 12B, provides an esti-
mate of how long it will take hardware to train in terms of
real time. As shown in FIG. 12B, shorter ty and t, values
take about half the time to minimize the cost as the longer
values. These examples serve to highlight that while longer



US 2024/0028879 Al

[0032] FIGS. 16A-B illustrate the effect increasing ac on
the training time for different learning rates.

[0033] FIGS. 17A-D illustrate the effect of noisy (stochas-
tic) parameter updates on solving XOR in a 2-21 feedfor-
ward network, measured for various noise amplitudes oo,
[0034] FIGS. 18A-B illustrate the effect of adding random
offsets and scaling to each neuron’s sigmoid activation
function.

DETAILED DESCRIPTION.

[0035] While the making and using of various embodi-
ments of the present invention are discussed in detail below,
it should be appreciated that the present invention provides
many applicable inventive concepts which can be embodied
in a wide variety of specific contexts. The specific embodi-
ments discussed herein are merely illustrative of specific
ways to make and use the invention, and do not delimit the
scope of the present invention. Reference will now be made
to the drawings wherein like numerals refer to like elements
throughout.

[0036] Neurons in a neural network implemented in a
neuromorphic hardware receive input signals x from neu-
rons in a preceding layer and parameters 6 (e.g., weights and
biases) from synapses connecting the neurons to one or more
neurons in the preceding layer. Once the neurons receive the
inputs and parameters, the neurons add up each signal
multiplied by its corresponding parameter and uses them to
compute an output y=f (x; 6) for the neuron. Each neuron in
a neural network does not need to use every neuron in the
preceding layer. The neural network must be trained such
that the output signals of the network correspond to the
desired target outputs Y;.,.¢ The cost function is the mea-
sure of correspondence between network output and the
target and is indicated by C(Y.Y,.;ger) Cost function can be
determined using any technique known in the art. A goal of
the network is to minimize the value of the cost function.
The cost function is minimized when neural network’s
predicted value is substantially close to the target output
Yeager After determining an initial cost function for the
neural network, changes are made to the neural network to
determine whether those changes reduce the value of the
cost function. In some embodiments, the parameters of each
neuron at its synapse that communicate to the next layer of
the network is modified to determine whether the cost
function is reduced. The mechanism through which the
parameters are modified to move the neural network to
parameters with less error is called gradient descent. The
gradient descent mechanism changes the parameters of each
neuron’s input signals and the process is continued until the
decrease in the cost function caused by the change in the
parameters is below a predetermined threshold. Gradient
descent is performed by calculating the gradient dC/d9 and
adjusting the parameters to minimize C.

[0037] Embodiments in accordance with the present
invention provide a parameter-multiplexed gradient descent
(PMGD) system and method for training a neural network in
a neuromorphic hardware by: applying a perturbation 6, to
the base value of every hardware parameter 6,° in the neural
network, which propagate through the neural network to
influence the cost C; continuously determining the output
y(t) and cost function C for the neural network; extracting
time varying component C of the cost function C due to the
perturbations 6,, broadcasting the modulations to the input
parameters x;; extracting a partial cost gradient from the

Jan. 25, 2024

modulations; determining a parameter change for the hard-
ware parameters; and updating the hardware parameters
using the parameter change.

[0038] Referring now to the drawings, and more particu-
larly, to FIGS. 1 and 2, there is shown a PMGD system,
generally designated 100 and schematically showing an
embodiment of the present invention, for training a neural
network in a neuromorphic hardware. PMGD system 100
includes a neural network 102 in a neuromorphic hardware,
a global cost estimator 104, a filter 106, a transmitter 108, an
optimizer 110, an updater 112, and neuromorphic hardware
elements 114.

[0039] Neural network 102 includes an input layer 102a of
neurons for receiving time-varying input signals x(t) and
target output 9(t), synaptic circuits 102b for generating
modulated neuromorphic hardware signals and for transmit-
ting the modulated neuromorphic hardware signals to a
subsequent layer of neurons, a middle layer 102c of neurons
for generating neuromorphic hardware signals that is a
non-linear function of the modulated neuromorphic hard-
ware signals, and an output layer 102d of neurons for
determining output signals y(t), representing a prediction of
neural network 102, from the neuromorphic hardware sig-
nals generated by middle layer 102c of neurons. Fach
synaptic circuit 1026 includes neuromorphic hardware ele-
ments 114 that generate neuromorphic hardware signals, as
shown in FIG. 2. Neuromorphic hardware elements 114
include hardware parameters @ for setting the degree of
modulation of neuromorphic hardware signals. Each synap-
tic circuit 102 further includes an optimizer 110 and
updater 112, as shown in FIG. 2, for updating hardware
parameters @ with a parameter change to generate updated
neuromorphic hardware parameters

[0040] Cost estimator 104 compares output signals y(t)
with target Y,.,ze,(t) to determines a cost C(t)=C(¥(). Yeanger
(0). Cost estimator 104 compares output signals y(t) with
target Yiorgedt) by applying a cost function to each of the
output signals y(t) with target y,,,..,(t) The cost function is
a measure of correspondence between the output signals y(t)
and target y(t). In one embodiment of the present invention,
cost C(y). Yragedt)) is determined by the difference
between output y(t) and target Yyarger(t)

[0041] To obtain target output §(t), hardware parameters 6
must be trained via gradient descent on cost C(y(t), 9(t))
such that combining the trained hardware parameters 0 and
inputs x(t) generate output signal y(t) that corresponds to
Yearged(t) Optimizer 110 includes a perturbator 110a that adds
a time-varying perturbation 6(0) to the base value 0, of each
hardware parameters @ of hardware elements 114 in a
synaptic circuit 102b. Perturbations 6(t) from perturbator
110a of hardware parameters 0 modulates cost C and that
modulation is fed back to parameters 0. Perturbations 6(t)
can have a variety of patterns. In one embodiment of the
present invention, perturbations 6(t) from perturbator 110a
can have discrete pattern (digital). Exemplary discrete pat-
tems of perturbations 6(t) include time-multiplexing, code-
multiplexing, and the like. In another embodiment of the
present invention, perturbations 6(t) can have continuous
pattern (analog). Exemplary continuous patterns of pertur-
bations 6(t) include frequency multiplexing, and the like.
[0042] In embodiments of the present invention wherein
perturbations 6(t) from perturbator 10a have a discrete
pattern, the perturbed cost C[t] due to the perturbation is
determined by C[t]=C[t]-Co, wherein Cy is the unperturbed


Patent Application Publication Jan. 25, 2024 Sheet 17 of 18 | US 2024/0028879 Al

a

Converged fraction

FIG. 17A FIG. 17B

Ps

‘Training time
Training time

FIG. 17C FIG. 17D


Patent Application Publication

Jan. 25, 2024 Sheet 9 of 18

US 2024/0028879 Al

900
RECEIVE INPUTS &
9024 TARGET OUTPUT
MODULATE HARDWARE |_, 994
SIGNALS
906] APPLY PERTURBATION }¢———————})
908 eo
928 ip] INPUT NEW TRAINING | |
y SAMPLES
UPDATE
HARDWARE n 910 932
PARAMETERS
| RESET GRADIENT
¥*| APPROXIMATIONS
926
I 934
- 912
DETERMINE
HARDWARE al UPDATE
PARAMETER ¥*| PERTURBATIONS
CHANGE
n
924 ¥
COMPUTE OUTPUT = L914
EXTRACT PARTIAL
COST GRADIENT
| 0 COMPUTECOST = 4916 ag
—_ S 918
TRANSMIT GRADIENT
MODULATED COST y>| APPROXIMATION
FUNCTIONS OFF
920
n
¥ S

EXTRACT MODULATED
COST FUNCTIONS

FIG. 9


US 2024/0028879 Al
11

difference, coordinate-descent, SPSA, etc.) can be adjusted
via the tuning of the PMGD time-constants, and can even be
adjusted during training if desired. Because it is a model-free
perturbative technique (sometimes called zeroth order opti-
mization), it is applicable to a wide range of systems—it can
be applied to both analog and digital hardware platforms,
and it can be used in the presence of noise and device
imperfections. This overcomes a major barrier to using
hardware platforms based on emerging technologies, which
are often difficult to train. The perturbative techniques of
PMGD systems and methods in accordance with embodi-
ments of the present invention can be used to train recurrent
neural networks, spiking networks and other non-standard
networks at small scale, and other neuromorphic hardware
and other physical neural networks. PMGD systems and
methods in accordance with embodiments of the present
invention can also be implemented directly on-chip with
local, autonomous circuits.

[0092] Parameter multiplexed gradient descent system and
methods in accordance with one or more embodiments of
the present invention can be adapted to a variety of con-
figurations. It is thought that parameter multiplexed gradient
descent system and methods in accordance with various
embodiments of the present invention and many of its
attendant advantages will be understood from the foregoing
description and it will be apparent that various changes may
be made without departing from the spirit and scope of the
invention or sacrificing all of its material advantages, the
form hereinbefore described being merely a preferred or
exemplary embodiment thereof.

[0093] Those familiar with the art will understand that
‘embodiments of the invention may be employed, for various
specific purposes, without departing from the essential sub-
stance thereof. The description of any one embodiment
given above is intended to illustrate an example rather than
to limit the invention. This above description is not intended
to indicate that any one embodiment is necessarily preferred
over any other one for all purposes, or to limit the scope of
the invention by describing any such embodiment, which
invention scope is intended to be determined by the claims,
properly construed, including all subject matter encom-
passed by the doctrine of equivalents as properly applied to
the claims.

What is claimed is:

1. A multiplexed gradient descent system for training a
neural network implemented in a neuromorphic hardware,
said system comprising:

an input layer comprising a first plurality of neurons
configured to receive a plurality of input signa

a plurality of synaptic circuits for modulating at least one
of a first plurality of neuromorphic hardware signals,
wherein each of the plurality of synaptic circuit com-
prises a plurality of neuromorphic hardware elements
for generating the at least one of the first plurality of the
neuromorphic hardware signals, wherein the plurality
of the neuromorphic hardware elements comprises a
first plurality of neuromorphic hardware parameters for
setting the modulation of the at least one of the first
plurality of the neuromorphic hardware signals to a
predetermined value;

a second plurality of neurons for generating a second
plurality of neuromorphic hardware signals from the
modulated first plurality of the neuromorphic hardware
signals, wherein each of the second plurality of the

Jan. 25, 2024

neuromorphic hardware signals is a nonlinear function
of the at least one of the first plurality of the neuro-
morphic hardware signals;

a third plurality of neurons for generating a plurality of
output signals from the second plurality of the neuro-
morphic hardware signals, wherein the plurality of the
output signals represent a prediction of the neural
network in the neuromorphic hardware;

a cost element for comparing the plurality of the output
signals with a target output to generate a plurality of
costs, wherein comparing the plurality of the output
signals with the target output comprises applying a
plurality of cost functions to the plurality of the output
signals and the target output, wherein each of the
plurality of the cost function is a measure of correspon-
dence between at least one of the plurality of the output
signals and the target output;

a filter for extracting a plurality of modulated cost func-
tions, wherein extracting the plurality of modulated
cost functions comprises determining a plurality of
modulations in the plurality of the costs;

a transmitter for transmitting the plurality of the modu-
lated cost functions to the first plurality of the neuro-
morphic hardware parameters;

an optimizer in at least one of the plurality of the synaptic
circuits, comprising:

a perturbator for applying a perturbation to at least one
of the first plurality of the neuromorphic hardware
parameters, wherein applying the perturbation modi
fies the first plurality of the neuromorphic hardware
parameters to a second plurality of neuromorphic
hardware parameters;

a receiver for receiving at least one of the plurality of
the transmitted modulated cost functions; and

a correlator for extracting a partial cost gradient from
the at least one of the plurality of the received
modulated cost functions, wherein extracting the
partial cost gradient from the at least one of the
plurality of the received modulated cost functions
comprises determining an error signal for at least one
of the second plurality of the neuromorphic hardware
parameters, wherein determining the error signal for
the at least one of the second plurality of the neu-
romorphic hardware parameters comprises applying
a multiplier signal to each of the plurality of the
received modulated cost functions to correlate the
plurality of the received modulated cost functions
with the second plurality of the neuromorphic hard-
ware parameters; and

an updater in at least one of the plurality of the synaptic
circuits for determining a parameter change for the at
least one of the second plurality of the neuromorphic
hardware parameters from the extracted partial cost
gradient and updating the at least one of the second
plurality of the neuromorphic hardware parameters
with the parameter change to generate a third plurality
of neuromorphic hardware parameters
2. The multiplexed gradient descent system of claim 1,
wherein the perturbation is a time-varying perturbation.
3. The multiplexed gradient descent system of claim 1,
wherein the perturbation is a discrete perturbation.
4. The multiplexed gradient descent system of claim 3,
wherein the perturbation is time-multiplexing.



Patent Application Publication Jan. 25, 2024 Sheet 6 of 18 US 2024/0028879 Al

fotiintcntinenent

FIG. 6


Patent Application Publication

Jan. 25, 2024 Sheet 5 of 18

Sm:

co

ee ee ere ee ee

CLO LU

i eee a en a |

a
3
Ce
a
g 4 . &
Time

FIG. 5

US 2024/0028879 Al


Patent Application Publication Jan. 25, 2024 Sheet 18 of 18 | US 2024/0028879 Al

a
ny ne
E 8 qe Ol
0 10° eyed
&
&
bo ion
- 3 :
a eo oe ae
ee a
Ca O25
FIG. 18A
i4 eee
| \
1 at
\

Converged fraction

eed

bed
©

i

FIG. 18B


Patent Application Publication Jan. 25,2024 Sheet 8 of 18 US 2024/0028879 Al

ne

g

FIG. 8


US 2024/0028879 Al

Correlator 110c also determines the gradient integration time
To to set how often parameter are updated and determines the
accuracy of the gradient approximation. Correlator 110c
integrates the gradient approximation G for each time period
Tp and updater 112 updates the parameters according to a
gradient descent step 0,>6,-nG,. Updater 112 further deter-
mines a timescale 7, for applying new training samples x,
to neuromorphic hardware. After each t, period, updater 112
discards the old sample and applies new training samples x,
¥ to generate new output y and cost C. te and t, have an
impact on the training of the neural network implemented in
a neuromorphic hardware in accordance with embodiments
of the present invention and can be selected such that
optimizer 110 can optimize using conventional numerical
analysis techniques.

[0059] The perturbation signals 6(t) can take many forms,
but it is preferred that they have a small-amplitude, zero-
mean, and are orthogonal to each other. During a typical
operation of embodiments in accordance with the present
invention, 6(t) is temporarily added to the parameters 0 as a
means of estimating the gradient of the cost function. These
perturbations are distinct from the gradient descent updates
which are applied to the parameters so as to reduce the cost.
[0060] FIG. 4 illustrates an exemplary optimization of a
perturbed neural network implemented in a neuromorphic
hardware in accordance with embodiments of the present
invention. In an exemplary implementation of a forward
finite-difference algorithm within embodiments in accor-
dance with the present invention, as shown in FIG. 4,
perturbator 110a applies discrete perturbation to parameters
such that a single parameter is perturbed by A@ at every t,,
and the parameters are perturbed sequentially. When param-
eter i is perturbed by AO,, the cost changes by AC and the
resulting partial gradient AC/A@,~0C/80, is stored in G,. If
the time period t, for integration is set to Pt,, where P is the
number of parameters in the network, then one element of
gradient is approximated for each t,, every partial gradient
is measured and stored after Pt,,, and the weight is updated
after all the partial gradients are collected.

[0061] FIG. 5 illustrates a second exemplary optimization
of a perturbed neural network implemented in a neuromor-
phic hardware in accordance with embodiments of the
present invention. Optimizer 110 uses the same process as
described for the exemplary optimization shown in FIG. 4
but reduces the integration time ty to a single timestep t,,
i.e., Tot,, Corresponding to a coordinate descent, as shown
in FIG. 5. In this case, rather than storing each G, until all the
partial gradients are fully assembled, updater 112 applies the
weight immediately after each G, is determined. In this
exemplary optimization, G, is used for the weight update and
subsequently discarded.

[0062] FIG. 6 illustrates a third exemplary optimization of
a perturbed neural network implemented in a neuromorphic
hardware in accordance with embodiments of the present
invention. Simultaneous perturbation stochastic approxima-
tion can be implemented by changing the values of the time
constants and the form of the perturbation. Correlator 110c
sets the integration time Tg=t,, and perturbator 110a applies
random, discrete {+A0, —A®} perturbation to every param-
eter at every t,,, as shown in FIG. 6. G, values are not stored,
and additional memories are not needed.

[0063] FIG. 7 illustrates a fourth exemplary optimization
of a perturbed neural network implemented in a neuromor-
phic hardware in accordance with embodiments of the

Jan. 25, 2024

present invention. In this case, t, corresponds to the times-
cale I/Af, where Af is the perturbation bandwidth, the
difference between the maximum and minimum perturba-
tion frequency, Tg is the integration constant, and there is no
discrete update of parameters. 0, is continuously updated
with the output of filter 106 with time constant Ty.

[0064] Neuromorphic hardware may restrict machine
learning datasets composed of large number of input
samples, or training samples, from being presented to the
hardware at a time. These datasets are typically broken into
mini-batches and gradient descent is performed on these
mini-batches. In embodiments in accordance with the pres-
ent invention, updater 110 sets a time constant t, to define
the time period for presenting new training samples x, § to
the hardware. As the sample changes, the integrated gradient
approximation G,(t) will accumulate the error signal e,(t)
from each sample it is presented. After time to, updater 110
updates the parameters 6, with parameter changes deter-
mined using the accumulated gradient approximation G,(t).
Optimizer 110 determines a batch size from a ratio of the
gradient integration time ty and the sample update time 7,.
When t, is shorter than tg, optimizer 110 shows multiple
samples to the network during a single gradient integration
period. As the sample changes, the gradient approximation
G,t) will then include gradient information from each of
those samples. FIG. 8 illustrates an exemplary batching with
three parameters and two input training on a dataset with
four samples using a PMGD system in accordance with
embodiments of the present invention in a neural network in
a neuromorphic hardware. The parameters 0 are updated
every Tg, and during that time, all four training samples are
shown to the network and integrated into the gradient
approximation G (batch size tg/t,=4). G accumulates at each
timestep and is reset during the weight-update process after
each ty period. FIG. 8 shows that updates to 6 occur in the
opposite direction of G.

[0065] FIG. 9 is a flowchart illustrating a parameter mul-
tiplexed gradient descent (PMGD) method 900 in accor-
dance with embodiments of the present invention for train-
ing a neural network in a neuromorphic hardware. An input
layer of neurons receives time-varying inputs x(t) and target
Yrargedt) at operational step 902. Neuromorphic hardware
signals generated by hardware elements in synaptic circuits
1026 are modulated, at operational step 904, to a value
determined by hardware parameters 6. At operational step
906, perturbations are applied to hardware parameters 6 by
perturbator 110a. At decision step 908, PMGD method 900
determines whether the remainder of training time t divided
by time period t, for presenting new training samples x, ¥ to
the hardware (t mod t,) is equal to zero. If the remainder of
training time t divided by time period t, for presenting new
training samples x, ¥ to the hardware (t mod 7,) is equal to
zero, then new training samples are provided as inputs at
step 930 and received at step 902. If the remainder of
training time t divided by time period t, for presenting new
training samples X.Y,_y¢e, 10 the hardware (t mod t,) is not
equal to zero, then, at decision step 910, PMGD method 900
determines whether the remainder of training time divided
by gradient integration time (t mod tg) is equal to zero. If the
remainder of training time divided by gradient integration
time (t mod Tg) is equal to zero, then gradient approxima-
tions G, are reset at step 932. If the remainder of training
time divided by gradient integration time (t mod tq) is not
equal to zero, then, at decision step 912, PMGD method 900



as) United States

US 20240028879A1

a2) Patent Application Publication co) Pub. No.: US 2024/0028879 A1

Buckley et al.

(43) Pub. Date: Jan. 25, 2024

(54) SYSTEM AND METHOD FOR PARAMETER
MULTIPLEXED GRADIENT DESCENT

(71) Applicant: Government of the United States of
America, as represented by the
Secretary of Commerce, Gaithersburg,
MD (US)

(72) Inventors: Sonia Mary Buckley, Buena Vista, CO
(US); Adam Nykoruk MeCaughan,
Denver, CO (US); Andrew Martin
Dienstfrey, Louisville, CO (US
Woo Nam, Boulder, CO (US)

(21) Appl. No.: 18/223,663

(22) Filed: Jul. 19, 2023

Related U.S. Application Data

(60) Provisional application No. 63/368,800, filed on Jul.
19, 2022.

Publication Classification
(51) Int. CL.
GOON 3/063
GOON 3/048
(52) US. Cl.
c

(2006.01)
(2006.01)

GO6N 3/063 (2013.01); GO6N 3/048
(2023.01)

(57) ABSTRACT

Embodiments of the present invention relate to systems and
model-free methods for perturbing neural network hardware
parameters and measure the neural network response that are
implemented natively within the neural network hardware
and without requiring a knowledge of the internal structure
of the network. Embodiments of the present invention also
relate to systems and methods for configuring neural net-
work hardware such that the network automatically per-
forms parameter multiplexed gradient descent, which
include adding a time-varying perturbation to each hardware
parameter base value to modulate the cost, broadcasting the
modulated cost signal to all hardware parameters, and fil-
tering out modulations so as to extract gradient information.

100
\ 08

9

“

a
aS

x

X3

— iON 106

104

WOK OO

C 102b

i
o
Sv
a



Patent Application Publication

100

Jan. 25, 2024 Sheet 11 of 18

US 2024/0028879 Al

RECEIVE INPUTS &
TARGET OUTPUT nti
INTIALIZE PARAMETERS (1104
1124
1106
GRADIENT
y— >| APPROXIMATION
OFF
1168 1126
INPUT NEW
¥— | TRAINING SAMPLES
n
¥
UPDATE Aiuto
PERTURBATIONS
COMPUTE OUTPUT “1112
v
COMPUTE CosT = \“1114
COMPUTE CHANGE | Ay446
IN COST
COMPUTE ERROR | Aii48
SIGNAL
UPDATE
GRADIENT = [1120
APPROXIMATION
UPDATE LAiw2
PARAMETERS
FIG, 11


Patent Application Publication

Jan. 25, 2024 Sheet 10 of 18

US 2024/0028879 Al

1000
N too] RECEIVE INPUTS &
TARGET OUTPUT
10047 \fODULATE HARDWARE | APPLY
SIGNALS PERTURBATION
1028
1006
RESET GRADIENT Comoe 1.09 >—yp| ENPUT NEW TRAINING
APPROXIMATIONS — SAMPLES
1026 n 1030 1032
1008
UPDATE yo] SET PERTURBATIONS
PARAMETERS , TO ZERO
) v
y | 34] UPDATE BASELINE
COST
1024 1010
imod 2 vl UPDATE ly
=0? bs PERTURBATIONS
h 1036
¥
COMPUTE OUTPUT K/*4012
COMPUTE COST h/~*1014
1022
V plots
ACCUMULATE COMPUTE CHANGE IN
GRADIENT COsT
APPROXIMATION T
COMPUTE ERROR SIGNAL 1018
1038
1020
GRADIENT
n y——>| APPROXIMATION
OFF

FIG. 10


US 2024/0028879 Al

extracting a plurality of modulated cost functions,
wherein extracting the plurality of modulated cost
functions comprises determining a plurality of modu-
lations in the plurality of the costs;

transmitting the plurality of the modulated cost functions

to the first plurality of the neuromorphic hardware
parameters;

optimizing in at least one of the plurality of the synaptic

circuits at least one of the plurality of the transmitted
modulated cost functions to determine a parameter
change for the at least one of the first plurality of the
neuromorphic hardware parameters; and

updating the at least one of the first plurality of the

neuromorphic hardware parameters with the parameter
change to generate a second plurality of neuromorphic
hardware parameters.
14. The multiplexed gradient descent method of claim 13,
wherein the perturbation is a time-varying perturbation.
15. The multiplexed gradient descent method of claim 13,
wherein the perturbation is a discrete perturbation.
16. The multiplexed gradient descent method of claim 13,
wherein the perturbation is an analog perturbation.
17. The multiplexed gradient descent method of claim 13,
wherein optimizing the transmitted modulated cost function
comprises
receiving at each of the plurality of the synaptic circuits
the at least one of the plurality of the transmitted
modulated cost functio:

applying a first perturbation to each of the first plurality of
the neuromorphic hardware parameters;

Jan. 25, 2024

extracting a partial cost gradient from the at least one of
the plurality of the received modulated cost functions,
wherein the extracting the partial cost gradient from the
at least one of the plurality of the received modulated
cost functions comprises determining an error signal
for the at least one of the perturbed first plurality of the
neuromorphic hardware parameters, wherein determin-
ing the error signal for the at least one of the perturbed
first plurality of the neuromorphic hardware parameters
comprises applying a multiplier signal to each of the
plurality of the received modulated cost functions to
correlate the plurality of the received modulated cost
functions with the perturbed first plurality of the neu-
romorphic hardware parameters; and

determining the parameter change for the at least one of

the first plurality of the neuromorphic hardware param-
eters from the extracted partial cost gradient.

18. The multiplexed gradient descent method of claim 17,
further comprising updating the first perturbation to a second
perturbation after a first predetermined time period.

19. The multiplexed gradient descent method of claim 18,
further comprising repeating the extracting the partial cost
gradient from the at least one of the plurality of the received
modulated cost functions for a second predetermined time
period.

20. The multiplexed gradient descent method of claim 19,
further comprising receiving a second plurality of input
signals and a second target output to the neuromorphic
hardware after a third predetermined time period.

+e eRe


US 2024/0028879 Al

5. The multiplexed gradient descent system of claim 3,
wherein the perturbation is code-multiplexing.
6. The multiplexed gradient descent system of claim 1,
wherein the perturbation is an analog perturbation.
7. The multiplexed gradient descent system of claim 6,
wherein the perturbation is frequency multiplexing.
8. A multiplexed gradient descent method for training a
neural network implemented in a neuromorphic hardware,
the method comprising:
receiving a first plurality of input signal from an input
layer comprising a first plurality of neuron:

modulating at least one of a first plurality of neuromor-
phic hardware signals generated by at least one of a first
plurality of hardware elements in at least one of a
plurality of synaptic circuits, wherein the at least one of
the first plurality of neuromorphic hardware signals is
modulated to a predetermined value set by a first
plurality of neuromorphic hardware parameters;

applying a first perturbation to each of the first plurality of
the neuromorphic hardware parameters, wherein the
applying the perturbation modifies the first plurality of
the neuromorphic hardware parameters to a second
plurality of neuromorphic hardware parameters;

generating at a second plurality of neurons a second
plurality of neuromorphic hardware signals from the
modulated first plurality of the neuromorphic hardware
signals, wherein each of the second plurality of the
neuromorphic hardware signals is a nonlinear function
of the at least one of the modulated first plurality of the
neuromorphic hardware signals;

generating at a third plurality of neurons a plurality of

output signals from the second plurality of the neuro-
morphic hardware signals, wherein the plurality of the
output signals represent a prediction of the neural
network in the neuromorphic hardware;

comparing at a cost element the plurality of the output

signals with a target output to generate a plurality of
costs, Wherein comparing the plurality of the output
signals with the target output comprises applying a
plurality of cost functions to the plurality of the output
signals and the target output, wherein each of the
plurality of the cost function is a measure of correspon-
dence between at least one of the plurality of the output
signals and the target output;

extracting a plurality of modulated cost functions,

wherein extracting the plurality of the modulated cost
functions comprises determining a plurality of modu-
lations in the plurality of the costs;

transmitting the plurality of the modulated cost functions

to the second plurality of the neuromorphic hardware
parameters;

receiving in at least one of the plurality of the synaptic

circuits at least one of the plurality of the transmitted
modulated cost functions;

extracting in at least one of the plurality of the synaptic

circuits a partial cost gradient from the at least one of
the plurality of the received modulated cost functions;
determining in at least one of the plurality of the synaptic
circuits a parameter change for the at least one of the
second plurality of the neuromorphic hardware param-
eters from the extracted partial cost gradient;
updating in at least one of the plurality of the synaptic
circuits the at least one of the second plurality of the

12

Jan. 25, 2024

neuromorphic hardware parameters with the parameter
change to generate a third plurality of neuromorphic
hardware parameters;

updating the first perturbation to a second perturbation

after a first predetermined time period;

repeating the extracting the partial cost gradient from the

at least one of the plurality of the received modulated
cost functions for a second predetermined time period;
and

receiving a second plurality of input signals and a second

target output to the neuromorphic hardware after a third
predetermined time period.
9. The multiplexed gradient descent method of claim 8,
wherein extracting the partial cost gradient from the at least
one of the plurality of the received modulated cost functions
comprises determining an error signal for the at least one of
the second plurality of the neuromorphic hardware param-
eters, wherein determining the error signal for the at least
one of the second plurality of the neuromorphic hardware
parameters comprises applying a multiplier signal to each of
the plurality of the received modulated cost functions to
correlate the plurality of the received modulated cost func-
tions with the second plurality of the neuromorphic hard-
ware parameters.
10. The multiplexed gradient descent method of claim 8,
wherein the perturbation is time-multiplexing.
11. The multiplexed gradient descent method of claim 8,
wherein the perturbation is code-multiplexing.
12. The multiplexed gradient descent method of claim 8,
wherein the perturbation is frequency multiplexing.
13. A multiplexed gradient descent method for training a
neural network implemented in a neuromorphic hardware,
the method comprising:
receiving a first plurality of input signal from an input
layer comprising a first plurality of neurons;

modulating at least one of a first plurality of neuromor-
phic hardware signals generated by at least one of a first
plurality of hardware elements in at least one of a
plurality of synaptic circuits, wherein the at least one of
the first plurality of the neuromorphic hardware signals
is modulated to a predetermined value set by a first
plurality of neuromorphic hardware parameters;

generating at a second plurality of neurons a second
plurality of neuromorphic hardware signals from the
modulated first plurality of the neuromorphic hardware
signals, wherein each of the second plurality of the
neuromorphic hardware signals is a nonlinear function
of the at least one of the modulated first plurality of the
neuromorphic hardware signals;

generating at a third plurality of neuro plurality of

output signals from the second plurality of the neuro-
morphic hardware signals, wherein the plurality of the
output signals represent a prediction of the neural
network in the neuromorphic hardware;

comparing at a cost element the plurality of the output

signals with a target output to generate a plurality of
costs, wherein comparing the plurality of the output
signals with the target output comprises applying a
plurality of cost functions to the plurality of the output
signals and the target output, wherein each of the
plurality of the cost functions is a measure of corre-
spondence between at least one of plurality of the
output signals and the target output;



Patent Application Publication

pe

Kear

Jan. 25, 2024 Sheet 7 of 18

wot

we

i nnn Sta

. AP a
Sg LF OF Oh Oe

BMPDPD PPD SASS

ee a

Time

FIG. 7

US 2024/0028879 Al


US 2024/0028879 Al

rality of the received modulated cost functions with the
second plurality of the neuromorphic hardware parameters.

[0014] Embodiments of the present invention also relate to
a multiplexed gradient descent method for training a neural
network implemented in a neuromorphic hardware, the
method including receiving a first plurality of input signal
from an input layer comprising a first plurality of neurons;
modulating at least one of a first plurality of neuromorphic
hardware signals generated by at least one of a first plurality
of hardware elements in at least one of a plurality of synaptic
circuits, wherein the at least one of the first plurality of the
neuromorphic hardware signals is modulated to a predeter-
mined value set by a first plurality of neuromorphic hard-
ware parameters; generating at a second plurality of neurons
a second plurality of neuromorphic hardware signals from
the modulated first plurality of the neuromorphic hardware
signals, wherein each of the second plurality of the neuro-
morphic hardware signals is a nonlinear function of the at
east one of the modulated first plurality of the neuromorphic
hardware signals; generating at a third plurality of neurons
a plurality of output signals from the second plurality of the
neuromorphic hardware signals, wherein the plurality of the
output signals represent a prediction of the neural network in
the neuromorphic hardware; comparing at a cost element the
plurality of the output signals with a target output to generate
a plurality of costs, wherein comparing the plurality of the
output signals with the target output comprises applying a
plurality of cost functions to the plurality of the output
signals and the target output, wherein each of the plurality of
the cost functions is a measure of correspondence between
at least one of plurality of the output signals and the target
output; extracting a plurality of modulated cost functions,
wherein extracting the plurality of modulated cost functions
comprises determining a plurality of modulations in the
plurality of the costs; transmitting the plurality of the
modulated cost functions to the first plurality of the neuro-
morphic hardware parameters; optimizing in at least one of
the plurality of the synaptic circuits at least one of the
plurality of the transmitted modulated cost functions to
determine a parameter change for the at least one of the first
plurality of the neuromorphic hardware parameters; and
updating the at least one of the first plurality of the neuro-
morphic hardware parameters with the parameter change to
generate a second plurality of neuromorphic hardware
parameters.

[0015] In one aspect of the present invention, optimizing
the transmitted modulated cost function includes receiving
at each of the plurality of the synaptic circuits the at least one
of the plurality of the transmitted modulated cost functions;
applying a first perturbation to each of the first plurality of
the neuromorphic hardware parameters; extracting a partial
cost gradient from the at least one of the plurality of the
received modulated cost functions, wherein the extracting
the partial cost gradient from the at least one of the plurality
of the received modulated cost functions comprises deter-
mining an error signal for the at least one of the perturbed
first plurality of the neuromorphic hardware parameters,
wherein determining the error signal for the at least one of
the perturbed first plurality of the neuromorphic hardware
parameters comprises applying a multiplier signal to each of
the plurality of the received modulated cost functions to
correlate the plurality of the received modulated cost func-
tions with the perturbed first plurality of the neuromorphic
hardware paramet nd determining the parameter change

Jan. 25, 2024

for the at least one of the first plurality of the neuromorphic
hardware parameters from the extracted partial cost gradi-
ent.

[0016] In another aspect of the present invention, the
multiplexed gradient descent method further includes updat-
ing the first perturbation to a second perturbation after a first
predetermined time period; repeating the extracting the
partial cost gradient from the at least one of the plurality of
the received modulated cost functions for a second prede-
termined time period; and receiving a second plurality of
input signals and a second target output to the neuromorphic
hardware after a third predetermined time period.

BRIEF DESCRIPTION OF DRAWINGS

[0017] FIG. 1 illustrates a system for parameter multi-
plexed gradient descent in accordance with embodiments of
the present invention.

[0018] FIG. 2 illustrates a schematic representation of a
synaptic circuit including an optimizer in accordance with
embodiments of the present invention

[0019] FIG. 3 illustrates a schematic representation of
continuous error computing integration in accordance with
embodiments of the present invention.

[0020] FIG. 4 illustrates a first exemplary optimization of
a perturbed neural network in accordance with embodiments
of the present invention.

[0021] FIG. 5 illustrates a second exemplary optimization
of a perturbed neural network in accordance with embodi-
ments of the present invention.

[0022] FIG. 6 illustrates a third exemplary optimization of
a perturbed neural network in accordance with embodiments
of the present invention.

[0023] FIG. 7 illustrates a fourth exemplary optimization
of a perturbed neural network in accordance with embodi-
ments of the present invention.

[0024] FIG. 8 illustrates an exemplary batching in a neural
network in accordance with embodiments of the present
invention.

[0025] FIG. 9 is a flowchart illustrating a method for
parameter multiplexed gradient descent in accordance with
embodiments of the present invention for training a neural
network in a neuromorphic hardware.

[0026] FIG. 10 is a flowchart illustrating an alternate
method for parameter multiplexed gradient descent in accor-
dance with embodiments of the present invention for train-
ing a neural network in a neuromorphic hardware.

[0027] FIG. 11 is a flowchart illustrating an alternate
method for parameter multiplexed gradient descent in accor-
dance with embodiments of the present invention for train-
ing a neural network in a neuromorphic hardware.

[0028] FIGS. 12A-B illustrate plots of data obtained using
simulations of an exemplary 2-bit problem by training a
2-2-1 feedforward network with 9 parameters using a
method in accordance with embodiments of the present
invention.

[0029] FIG. 13 illustrates plots of angle between gradient
approximation G and the true gradient versus time.

[0030] FIGS. 14A-B illustrate plots showing the effects
for t, on the training of an exemplary 2-bit parity (COR)
problem.

[0031] FIG. 15 illustrates training time distributions for
the 2-bit parity problem using the different perturbation

types.


US 2024/0028879 Al

integration times produce a more accurate gradient approxi-
mation, integration times as short as t,, may also be used to
train a network.

[0073] To quantify the effect of longer integration times on
the accuracy of the gradient approximation, convergence of
the gradient approximation G to the true gradient 8C/36 (as
computed by backpropagation) was measured as a function
of time by simulating with to and t,—t,=1, such that G
is continuously integrated without resetting or updating the
parameters. The angle between the true gradient 3C/3@ and
the approximation G were also computed during the simu-
lation. FIG. 13 shows plots of angle between gradient
approximation G and the true gradient versus time obtained
from simulations of 2-bit parity, 4-bit parity, and NIST7x7
problems. The NIST77 dataset is a small image recognition
problem based on identifying the letters N, I, S, and T on a
7x1-pixel plane. The dataset has the property that it cannot
be solved to greater than 93% with a linear solve. The
solution accuracy for a 49-4-4 feedforward network with
sigmoidal activation functions often exceeds 95% (see Table
5). FIG. 13 confirms that the angle decreases with time as G
aligns with the true gradient. The time axis is in units of t,,.
which is the minimum discrete timestep in this system. For
a real hardware platform, this timestep is approximately the
inference time of the system. In general, the more param-
eters the network has, the longer it takes to converge to the
true gradient

[0074] Mini-Batching

[0075] Investigations of the effects of tg and t, on training
time show that longer ty values result in a more accurate
gradient approximation but reduce the frequency of param-
eter updates. Using a fixed, low 1 value, a 2-2-1 network
was trained to solve 2-bit parity (XOR) for 100 different
random parameter initializations. varying ty but keeping the
batch size t,/t, constant at either 4 or 1. Since the 2-bit
parity dataset is composed of four (X, Yzanger) PAKS, T.-4Ty 8
analogous to gradient descent—all four samples are inte-
grated into the gradient approximation G before performing
a weight update. When t,=tp, the network performs sto-
chastic gradient descent (SGD) with a batch size of 1. FIG.
14A shows the training time as a function of tg and batch
size. Here, training time corresponds to the time at which the
total cost C drops below 0.04, indicating the problem was
solved successfully. In the case where the batch size was 1,
increasing ty increased the training time. However, when the
batch size was 4, increasing t, had little effect on the training
time.

[0076] As with any training process, the training can
become unstable at higher 1 values and fail to solve the task.
The results shown in FIG. 14A are only for a fixed learning
rate, and so, the effect of ty on the maximum achievable
were also examined. As shown in FIG, 14B, as ty is
increased the max 1 decreases, resulting in longer minimum
training times. Here, “max 1)” is the maximum learning rate
where the network successfully solved the 2-bit parity
problem for at least 50 out of 100 random initializations.
[0077] From these results, it can be inferred that a poor
gradient approximation taken with respect to all training
‘examples is more useful than collecting an accurate gradient
with respect to a single example. Waiting a long time for an
extremely accurate gradient and then taking a large step is
less productive than taking a series of short (but less
accurate) steps. Accordingly, implementing an effective gra-
dient descent process in PMGD does not necessarily require

Jan. 25, 2024

additional memory to store accurate, high-bit-depth gradient
values. In the exemplary implementation described herein,
G accumulates with time and so the size of the parameter
update 1G from 0,>6,-nG, grows proportionally to the
integration time. Accordingly, when Ty is larger the effective
step in the direction of the gradient is also larger, and so for
fixed 7 the rate of training therefore remains approximately
constant. If this was not the case, whenever tg is doubled,
would also need to reduce by half to maintain the same
approximate rate of training.

[0078] Analog and Digital Perturbations

[0079] The parameter perturbations can take many differ-
ent forms, provided they are low-amplitude, and their time
averages are pairwise orthogonal or, in a statistical setting,
are uncorrelated. Four types of perturbations were imple-
mented in systems and methods in accordance with embodi-
ments of the present invention: sinusoidal perturbations,
sequential discrete perturbations, discrete code perturba-
tions, and random code perturbations. In sinusoidal pertur-
bations, each parameter is assigned a unique frequency. In
sequential discrete perturbations, parameters are sequen-
tially perturbed, one at a time, by +40. “Code” perturbations
are simultaneous discrete perturbations of {-A®, +A0} for
every parameter every T, timesteps. There are two types of
code-perturbations: the first type consists of a predefined set
of pairwise-orthogonal square wave functions that take the
values of {-A0, +A0}. Each of these perturbation patterns is
a deterministic sequence, and no two parameters have the
same sequence. The second type includes randomly gener-
ated sequences of {-A0, +A0} that are pairwise uncorrelated
and are referred to as “statistically orthogonal.” The statis-
tically orthogonal code-perturbations are less efficient than
the deterministic orthogonal codes because perturbations
from multiple parameters interfere with each other more in
C—any finite sample of the perturbations will have a
non-zero correlation that decreases to zero as the sample size
increases. However, the use of the statistically orthogonal
version allows the perturbations to be generated locally and
randomly. These perturbations may be useful in hardware
implementations, as they are spread-spectrum and single-
frequency noise from external sources is unlikely to corrupt
the feedback signals. To compare the training performance
between different perturbation types, four different pertur-
bation types were applied to the 2-2-1 network to solve the
2-bit parity problem or to show that training can happen in
both a purely analog and purely digital way.

[0080] FIG. 15 shows the training time distributions for
the 2-bit parity problem using the different perturbation
types by measuring their time to train a 2-2-1 network on the
2-bit parity (XOR) problem. The bandwidth for sinusoidal
perturbations was set to be Yt,,. The different perturbation
types were found to be approximately equivalent in terms of
speed of training. This equivalence makes sense when one
considers that the feedback from C has a finite bandwidth
that must be shared between all the parameters—no matter
the encoding (perturbation) scheme, the information carried
in that feedback to the parameters will be limited by that
finite bandwidth.

[0081] Operation on Noisy or Imperfect Hardware
[0082] The fabrication defects and signal noise present in
emerging hardware platforms can pose challenges for cur-
rent training techniques in hardware. The eflects of the
following three different types of imperfections and noise
that could affect hardware systems were investigated: (1)



Patent Application Publication Jan. 25, 2024 Sheet 3 of 18 US 2024/0028879 Al

|___» update 6;

FIG. 3


Patent Application Publication

330

‘Training time

150

iy

Max 1

10°

Jan. 25,2024 Sheet 14 of 18 US 2024/0028879 Al
«lf
* Batch size 79/7, = 1 *
» Batch size 7a /Ty = 4 a
ae
at
on
* il
ot
we
Dood
-
AE ri 08 SN na Oat AT A
o 200 400 600 800
Tg
FIG, 14A
* Batch size T9/Tr = 1
» Batch size 79/T, = 4
a woot
. .; ee call 2
* \, Poll ; ie 2
ka ee 2
i Nic x &
1 some bgt
« en E
ae Ps
i a
0 200 400 600
Vg

FIG. 14B


Patent Application Publication Jan. 25, 2024 Sheet 16 of 18 | US 2024/0028879 Al
iv? e
/ /
/ i
2 Bent
3 settee
= 10% .
E /
# we Be
ee ae ae
at
ae
6 oc/A@ 0.08
FIG. 16A
_*
19 *
_2
1 =
J
10° ~
= z
a a
s =
= 101! ra 10° g
x E
z &
1rd ts
# “» Lio
0 62
oo/A8 2

FIG. 16B


Patent Application Publication Jan. 25, 2024 Sheet 2 of 18

US 2024/0028879 Al

C re
LK4110b
1
noth LP Hy eee
ig
110¢
114114
DY nY
7 Z
114 114

FIG. 2


US 2024/0028879 Al

determines whether the remainder of training time divided
by perturbations timescale (t mod t,) is equal to zero. If the
remainder of training time divided by perturbations times-
cale (t_mod t,) is equal to zero, then, at step 934, perturba-
tions 6 are updated by perturbator 110a and the updated
perturbations are applied to hardware parameters 6 at step
906. If the remainder of training time divided by perturba-
tions timescale (t mod t,) is not equal to zero, then, at step
914, output layer 102d of neurons determines output signals
y(t) from the parameters @ and inputs x(t). A cost C is
determined by cost estimator 104 from output signals y(t)
and target Y,.y2e,(t) at an operational step 916. At decisional
step 918, PMGD method 900 determines whether the train-
ing time t is equal to a predetermined set time T. If PMGD
method 900 determines that the training time t is equal to a
predetermined set time T, then PMGD method 900 resets
gradient approximations G, at step 936. If PMGD method
900 determines that the training time t is not equal to a
predetermined set time T, then a change in cost C, or
modulation, due to perturbations 6 is extracted by filter 106
as modulated cost functions at operational step 920. The
modulated cost functions extracted at step 920 are broad-
casted or transmitted to all hardware parameters 6 in syn-
aptic circuits 1026 at step 922. At operational step 924,
correlator 110c extracts partial cost gradients and accumu-
lates gradient approximations G,. At operational step 926, a
parameter change is determined by updater 110d for hard-
ware parameters 0 from the extracted partial cost gradient.
The parameter change is further used by updater 112 to
update the hardware parameters 6 at step 928.

[0066] FIG. 10 is an alternate flowchart illustrating a
method for parameter multiplexed gradient descent (PMGD)
1000 in accordance with embodiments of the present inven-
tion for training a neural network in a neuromorphic hard-
ware with discrete perturbations. An input layer of neurons
receives time-varying inputs x(t) and target Ygprgq(t) at
operational step 1002. Hardware parameters 6 applied by
synapses to each of the inputs x(t) and transmitted to a
subsequent layer of neurons are initialized at operational
step 1004. In one embodiment, initializing of hardware
parameters 6 include modulation of neuromorphic hardware
signals generated by hardware elements in synaptic circuits
102b to a predetermined value set by hardware parameters
6. At decision step 1006, PMGD method 1000 determines
whether the remainder of training time t divided by time
period t, for presenting new training samples x, y to the
hardware (t mod ,) is equal to zero. If the remainder of
training time t divided by time period t, for presenting new
training samples x, 9 to the hardware (t mod t,) is equal to
zero, then, at step 1030, new training samples are provided
as inputs and received at step 1002. If the remainder of
training time t divided by time period t, for presenting new
training samples X, Y,_,,¢, to the hardware (t mod t,) is not
equal to zero, then, at decision step 1008, PMGD method
1000 determines whether the remainder of training time
divided by gradient integration time (t mod tg) is equal to
zero. If the remainder of training time divided by gradient
integration time (t mod tg) is equal to zero, then, at step
1032, perturbations @ are set to zero and baseline cost Cy is
updated at step 1034. If the remainder of training time
divided by gradient integration time (t mod to) is not equal
to zero, then, at decision step 1010, PMGD method 1000
determines whether the remainder of training time divided
by perturbations timescale (t mod t,) is equal to zero. If the

Jan. 25, 2024

remainder of training time divided by perturbations times-
cale (t mod t,) is equal to zero, then perturbations 6 are
updated at step 1036 and the updated perturbations are
applied to hardware parameters 6 at step 1038. If the
remainder of training time divided by perturbations times-
cale(t mod t,) is not equal to zero, then, at step 1012, output
layer 102d of neurons determines output signals y(t) from
the parameters 0 and inputs x(t). A cost C is determined from
output y(t) and target y,,,e,(t) at an operational step 1014,
and a change in cost C, or modulation, due to perturbations
6 is computed at operational step 1016. At operational step
1018, an error signal e, is determined by integrating a
product of perturbation @ and modulation C. At decision step
1020, PMGD method 1000 determines whether the training
time is equal to a predetermined time T. If the training time
is not equal to the predetermined time T, then, at step 1022,
PMGD method 1000 determines and accumulates gradient
approximations G,. If the training time is equal to the
predetermined time T, then, at step 1038, PMGD method
1000 stops the accumulation of gradient approximations G,.
At decision step 1024, PMGD method 1000 again deter-
mines whether the remainder of training time divided by
gradient integration time (t mod tg) is equal to zero. If the
remainder of training time divided by gradient integration
time (t mod t9) is equal to zero, then PMGD method 1000
updates hardware parameters 6 at step 1026 and resets
gradient approximations G, at step 1028. If the remainder of
training time divided by gradient integration time (t mod te)
gradient integration time ty is not equal to zero, then, at
decision step 1006, PMGD method 1000 determines
whether the remainder of training time t divided by time
period +, for presenting new training samples X.Y;4)2¢, to the
hardware (t mod t,) is equal to zero.

[0067] FIG. 11 is a flowchart illustrating an alternate
method for parameter multiplexed gradient descent (PMGD)
1100 in accordance with embodiments of the present inven-
tion for training a neural network in a neuromorphic hard-
ware with analog perturbations. An input layer of neurons
receives time-varying inputs x(t) and target Y,aee(t) at
operational step 1102. Hardware parameters @ applied by
synapses to each of the inputs x(t) and transmitted to a
subsequent layer of neurons are initialized at operational
step 1104. In one embodiment, initializing of hardware
parameters 6 include modulation of neuromorphic hardware
signals generated by hardware elements in synaptic circuits
1026 to a predetermined value set by hardware parameters
6. At decision step 1106, PMGD method 1100 determines
whether the training time t is equal to a predetermined time
T. If the training time t is equal to the predetermined time T,
then accumulation of gradient approximations G, is turned
off at step 1124. If the training time t is not equal to a
predetermined time T, then, at decision step 1108, PMGD
method 1100 determines whether the remainder of training
time t divided by time period +, for presenting new training
samples X,Y,.;ger t the hardware (t mod t,) is equal to zero.
If the remainder of training time t divided by time period t,
for presenting new training samples X,Y,,,g0, 10 the hardware
(t mod 7,) is equal to zero, then, at step 1126, new training
samples are provided as inputs. If the remainder of training
time t divided by time period t, for presenting new training
samples X,Y;,;zer 0 the hardware (t mod 7,) is not equal to
zero, then PMGD method 1100 updates perturbations 6 at
step 1110, and, at operational step 1112, output layer 102d of
neurons determines an output y(t) from the parameters @ and



Patent Application Publication Jan. 25,2024 Sheet 1 of 18 US 2024/0028879 Al

oN

% 106
x2 104
x3

FIG. 1


US 2024/0028879 Al

stochastic noise on the output cost C,,,,,., (2) stochastic
noise on the parameter update 0,,,,.2. (3) per-neuron defects
in the activation function, where each neuron has a randomly
scaled and offset sigmoidal activation function that is static
in time. These tests were performed on the NIST7x7 dataset
using the 49-4-4 network with 220 parameters and with
Tater.

[0083] _ In the first test, Gaussian noise with mean zero and
standard deviation o,. were added to the cost, applied every
timestep such that noise C(t)CygeqAttCroise(ti Oe). FIG.
16A shows the effect of increasing 0, on the training time
for different learning rates. For a given learning rate, there
is a threshold amount of noise below which the training time
is minimally changed. However, as cost noise increases, the
training time eventually increases and ultimately stops con-
verging. To determine how this noise would affect the
minimum training time for optimized learning rates, the
maximum achievable y value for a range of G¢ was also
measured. FIG. 16B shows this maximum 7 value versus
cost noise, and corresponding minimum training time. The
trend indicates that, 1, can be higher at lower cost noise o¢
and a faster training is possible by reducing the learning rate.

Jan. 25, 2024
10

device variations that may be found in hardware, for
instance in analog VLSI neurons. The sigmoid activation
function for each neuron k was modified to a general logistic
function f,(a)=a,(1-e"F*-)-!4b,. The variations were all
Gaussian, and the scaling factors ot, and f, had a standard
deviation o, and a mean of 1, while the offset factors a, and
B, also had a standard deviation of o,, but were mean-zero.
[0088] Adding defects to the network’s activation func-
tions had a relatively small effect on the training time (FIG.
18A). Even with relatively large variations in the activation
functions (9,-0.25), the network only took about twice as
much time to fully train the NIST7x7 dataset. FIG. 18B
illustrates converged fraction versus the standard deviation
(,) of the logistic function parameters.

[0089] Dataset Results

[0090] PMGD system and method in accordance with
embodiments of the present invention was compared with
backpropagation on a variety of tasks for different network
architectures and hyperparameters. Tables 4 and 5 provide a
comparison of the accuracies obtained with PMGD and
backpropagation for different datasets and various hyperpa-

[0084] In the next test, the effect of noisy parameter rameter choices (to, t,, 1), batch size), with t, fixed at 1
updates on the training process was analyzed. For this
experiment, any updates to parameter included a randomly- TABLE 4
applied deviation such that 6<—-6-nG+6,,,.,,.. Where O,,o:5¢ 18 Sau, Parameters
Gaussian with mean zero and standard deviation og, nor- 2 s
malized by A®, such that ®,,.i4~N(O, G9/A8). Task Network 0% tM batch size
[0085] It was discovered that larger values of a 0, can = a So
prevent convergence entirely (FIG. 17). In the presence of Sanity jaa mL |
this noise, increasing » can improve the convergence of the Ws: 44 30011 Os i
problem, as highlighted by the a ,=0.1 and Gg=0.3 lines in_Fashion-MNIST ayer CNN 14378 1st 91000
FIG. 17A. At very small 9 values, its likely that O,i¢ Will Fashion MNIST layer CNN M4378 10, 1-9 1090
overwhelm the very small mG in 8=-9-NG48 cise. Making Fashion-MNIST — 2-layer CNN 14378-1000 19 1000
1 larger could prevent G from being drowned out by 0, CHARIO Sinyer CNN 26154 1191000,
At very large 1) values, the usual gradient-descent instability
starts to dominate and the convergence approaches zero. For
TABLE 5
Setup Accuracy
Task Network ) 10" steps 10° steps 10% steps 107 steps backprop
2bit parity 2 9 100% 100% 100% 100% 100%
NaS-T 49-4-4 220 81% 94% «= «97.7% 99.8%
ST 49-44 220 45% 93% 98.7% 99.8%
Fashion-MNIST 2-layer CNN. 14378 66.3% 793% 835% 88.6%
Fashion-MNIST 2-layer CNN 14378 66.3% 9.2% 834% —-BB.6%
Fashion-MNIST 2-layer CNN 14378 66.3% 77.7% 84.7% 88.6%
Fashion-MNIST —2-layer CNN 14378, 59.6% 791% 86.1% 88.6%
CIFAR-10 S-layer CNN 26154 23% 43.8% «60.7% «68%

agiven y, small values of og marginally increase the training
time, but the effect is less significant than changing the
learning rate 77 (FIG. 17C).

[0086] Another method to reduce the impact of 0,,,:.¢ is to
increase the integration time of the gradient. When To is
increased, G is accumulated for a longer time and becomes
proportionally larger. FIGS. 17B and 17D show that even the
largest og value has little effect on the result.

[0087] In the fourth test, the effect of including “defects”
in the neuronal activation functions was analyzed. Here,
neuronal activation functions were no longer identical sig-
moid functions, but had fixed random offsets and scaling that
were static in time. These variations emulate device-to-

[0091] Parameter multiplexed gradient descent system and
methods in accordance with embodiments of the present
invention has several advantages over previous gradient
descent systems and methods. With realistic timescales for
emerging hardware, training using PMGD systems and
methods in accordance with embodiments of the present
invention is capable of training emerging hardware in orders
of magnitude faster than backpropagation in terms of wall-
clock time to solution on a standard GPU/CPU. The PMGD.
systems and methods in accordance with embodiments of
the present invention allows the implementation of multiple
optimization algorithms using a single, global, cost signal
and local parameter updates. The algorithm used (e.g. finite-


Patent Application Publication Jan. 25, 2024 Sheet 12 of 18 — US 2024/0028879 Al

: i:
= ba Te Tal
(Po = %~100
' . see backprop
6 —s

if? 6108 61 Ie Ot t®

Epochs
FIG. 12A
6.34
Z
g
2
0
G i 2
‘Time (iterations) xig®

FIG. 12B


Patent Application Publication Jan. 25,2024 Sheet 15 of 18 US 2024/0028879 Al

xif

g ioe seal —
bs
= -
=
&
&
m= 4

2

sinusoidal code sequential

FIG. 15