What is wrong with backpropagation1.

the perceptual system needs toperform inference and learningin real time without stopping
to perform backpropagation2.

it requires perfect knowledgeof the computation performedin the forward pass in order tocompute the correct
derivatives. (It is possible toresort to reinforcementlearning)

The Forward-Forward Algorithm• Replace the forward and backward passes of backpropagation by two forward passes.
• Positive pass on real data adjusts the weight to increase the goodness in every hidden layer• Negative pass on negative data adjusts the weight to decrease the goodness in every hidden layer• The aim of the learning is to make the goodness be well above θ for real data and well below θ for negativedata.
Goodness=

θ: Hyperparameter
σ: logistic function

Where does the real & negative data from?
784

0 0 00001000

MNIST Dataset

Real label

Real data

Negative data
1000000000

1x28x28

Random label
The second way to get negative data:
Negative data is generated by doing a single forward passthrough the net to get probabilities for all the classes and then choosing between the incorrect classesin proportion to their probabilities. This makes the training much more efficient.
0.3

0.01

0.01

0.01

0.01

0.01

0.6

0.01

0.01

0.03

The Forward-Forward Algorithm
FCRELU

Goodness=Layer normalization

Goodness ＞ θrelu

784 x 1000……
LN

1000

784

negative data

• Negative pass on negative data
adjusts the weight to decrease thegoodness in every hidden layer

LN

1000

784

real data

Goodness > θ• Positive pass on real data adjusts
the weight to increase thegoodness in every hidden layer

Goodness < θGoodness < θ

Hidden layer1

Hidden layer2

The aim of the learning is tomake the goodness bewell above some threshold value
for real data and well below thatvalue for negative data.

How to predict

FC
RELU
Layer normalization

Goodness=
1000000000
0100000000

SumGoodness00010000000

SumGoodness10001000000

SumGoodness2
SumGoodness3
SumGoodness4

LN

1000

relu

0000010000

784 x 1000

0000100000

Goodness

Goodness

SumGoodness50000001000

SumGoodness60000000100

SumGoodness7
SumGoodness80000000010

SumGoodness90000000010
784

layer1layer2

Another way to predict(quick but sub-optimal)
FC

Goodness=softmax

RELU

N x 10Layer normalization

ten entries of 0.1

layer1

LN

1000

relu

784 x 1000

784……

layer2

The advantage of FF• It can be used when the precise details of the forward computation areunknow.

• It can learn while pipelining sequential data through a neural network
ever without storing the neural activities or stopping to propagate
error derivatives.

What is wrong with FF• FF is somewhat slower than backpropagation• FF does not generalize quite as well as backpropagation• What is learned in later layers cannot affect what is learned in earlierlayers.

How to resolve the third disadvantage?
What is learned in later layers cannot affect what is learned in earlier layers.

acbActivity vector b =0.3 * Activity vector a+ 0.7 * LN(Activity vector c)Experiments with CIFAR-10

Conclusion: Although the test performance of FF isworse than backpropagation it is only slightly worse,
even when there are complicated confounding
backgrounds.

Max(min) ssq: maximize(minimize) the sum of the squared activities

Other interesting points• An energy efficient way to multiply an activity vector by a weightmatrix is to implement activities as voltages and weights asconductances.
• There are many other possible activation function to explore in thecontext of FF.
• Mortal Computation makes trillion parameter neural net to consume a
few watts.

Code