A Dualistic View of Activationsin Deep Neural Networks
Ehsan Amid
Joint work with: Rohan Anil, Manfred Warmuth, Abel L. Peirson, Tomer Koren, Yatong Chen,Christopher Fifty, Vlad Fineberg

Overview● Convexity and Duality● A Dualistic View of Activations in DNNs
● Application○LocoProp: Local Loss Optimization [AISTATS22]● Conclusions and Future Directions

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Convexity and Duality
Convex Sets, Convex Functions, Legendre Dual, Bregman DivergenceConvex Sets and Convex Functions
A set is convex if all the points along the line connecting two points inside theset are also belong to the set

Source code: https://tex.stackexchange.com/questions/639115/how-to-draw-convex-strictly-convex-and-nonconvex-sets

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Convex Sets and Convex Functions
A function is convex if it has a convex domain andA Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Legendre Dual
Given a (convex) function, the Legendre dual convex function is given byA Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Legendre DualWhenis strictly convex, we have the dually coupled variables:

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Bregman Divergence: Definition

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Bregman Divergence: Properties
Convexity: always in the left argument (not necessarily the right)
Non-negativity:

andiffGradient:
Many well-known cases:
Squared Euclidean:
Relative Entropy:(with)(with)A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Matching Loss: Definition
Definition: integral under the curve… is a Bregman divergence!

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Matching Loss Example: Softmax → KL Divergence
Link function:

Integral function:

Matching loss (in terms of post-activations):

A Dualistic View of Activations in Deep Neural Networks — Ehsan AmidA Dualistic View of Activations inDNNs
The Role of Activation Functions

Which link functions are valid?
Forto be a “valid” link function (i.e., gradient of a strictly convex function):● Needs to be strictly increasing in 1-D● And cyclically strictly monotone in higher dimensionMany activation functions
in modern neural networks(Linear, Leaky ReLU, tanh,sigmoid, softmax*, etc.)
satisfy this property!

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Matching Losses of Common Activation Functions

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Matching Losses of Common Activation Functions

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Application
LocoProp: Local Loss Optimization

Local Loss Optimization
LocoProp conceives neural networks as a modular composition of layers

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Local Loss Optimization
Preand post

activations in layer m

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

The Case of Squared Loss
Local regularized squared loss:
whereloss to the target

keep the weight close

GD w.r.t. the final loss

Solution: fixed point equation

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

The Case of Squared Loss
How about a single iteration?

This is just a single step of BackProp!
A Dualistic View of Activations in Deep Neural Networks — Ehsan AmidThe Case of Squared Loss

There is a closed-form solution:

gradient descentpreconditioner matrix

A preconditioned gradient descent!

A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Using Matching Losses
Replace the squared loss with the matching loss:
where

where
target post-activation

MD w.r.t. the finalloss

Still convex in the weights, and yields a preconditioned update
A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Experiments
Results on the deep autoencoder benchmark

LocoProp performs competitive to second-order methodsA Dualistic View of Activations in Deep Neural Networks — Ehsan Amid

Conclusions and Future Directions

Conclusions and Future Directions
Further applications:● Layerwise Fisher approximation via local sampling [NerIPS22-HITY]● Bregman knowledge distillation [TMLR23]● Robust bi-tempered loss [NeurIPS19]
Future directions:● No-sample layerwise Fisher approximation● Extension to non-monotonic activations● Low-rank compression of the weights● Learning losses and activation functions● …
A Dualistic View of Activations in Deep Neural Networks — Ehsan Amid