Approximate Empirical Bayes for Deep
Neural Networks∗∗

Han Zhao , Yao-Hung Hubert Tsai , Ruslan Salakhutdinov and Geoff Gordon
Machine Learning Department, Carnegie Mellon University• Maximum likelihood estimation: rich neural networks overfit on
small datasets.
• Regularizations: Weight decay, early stopping, dropout, DeCov,
spectral regularization, etc.
Contributions:• We propose an approximate empirical Bayes (AEB) framework forlearning neural networks.
• We give a block coordinate ascent algorithm to optimize theweight matrix.

Experiments

Approximate Empirical Bayes:Introduce a matrix-variate normal distribution over the weight matrix:

Multi-class classification: MNIST dataset.
Network structure: CONV5×5×1×10-CONV5×5×10×20-FC320×50-FC50×10.
Only impose prior distribution on the weight matrix of the last layer.

W ∼ MN (0p×d, Σr , Σc),p• Σr ∈ S++: Covariance matrix over row vectors.
• Σc ∈ Sd++: Covariance matrix over column vectors.
• MN (A, Σr , Σc) is a matrix-variate normal with mean vec(A) andcovariance Σr ⊗ Σc.
The true empirical Bayes approach is intractable for neural networks,
hence we approximate it with the following optimization formulation:
max max
W

Motivating Example: A simple two-layer feed-forward neural network:
Inputlayer

Hiddenlayer

Outputlayer

Input #1
Input #2
OutputInput #3
Input #4

TForward propagation: ŷ = a h,h = σ(W x).

• Input x, hidden layer h, a single output y ∈ R.
• Weights: W and a.
• ReLU activation σ(·), `2 loss as objective function.
Backward propagation:• W ← W − α(ŷ − y)(a h0)xT .
• The gradient matrix is rank one.
• Rows/Columns of the updates are correlated with each other.
It is beneficial to learn from the experience of others.
– Bradley Efron

UAI-2018

Σr ,Σclog p(D | W ) + log p(W | Σr , Σc)Plugging in the prior distribution leads to a constrained optimization
problem:
12
1/2
1/2 2min min|ŷ(W, a) − y| + λ||Ωr W Ωc ||FW,a Ωr ,Ωc2− λ (d log det(Ωr ) + p log det(Ωc))
subject touIp  Ωr  vIp, uId  Ωc  vId−1:• Ωr := Σ−1c and Ωc = Σc are the corresponding precision matrices.

Block Coordinate Ascent Algorithm:
0 :
(0)
(0)
(0)
(0)
Input: Initial value w = {a , W }, Ωr and Ωc , first-order optimization algorithm A, constants 0 < u ≤ v.
1: for t = 1, . . . , ∞ until convergence do
(t−1)
(t)
2:
Fix Ω(t−1),Ω,optimizew
by backpropagation and algorcrithm A(t) (t−1)
(t)T
3:
Ω(t)
←InvThresholding(WΩW, d, u, v)
rc(t)(t)T (t)(t)4:Ωc ← InvThresholding(W Ωr W , p, u, v)
5: end for6:

procedure I NV T HRESHOLDING(∆, m, u, v)
8:Compute SVD: Qdiag(r)QT = SVD(∆)
9:
Threshold r0 ← T[u,v](m/r)
10:
return Qdiag(r0)QT
11: end procedure
7:

Aug. 10th, 2018

100
95

Accuracy

Learning Parameters in Neural Networks:

The Model and Algorithm

90
85
80
CNN
CNN­WeightDecay
CNN­Dropout
CNN­AEB

75
102

104

103

Train Size

Multi-task regression: SARCOS dataset.
Goal: Map from a 21-dimensional input space (7 joint positions, 7joint velocities, 7 joint accelerations) to the corresponding 7 jointtorques. The training set and test set contain 44,484 and 4,449 examples. Network structure: FC21×256-FC256×100-FC100×7.
0.6

MTLMTL-AEB

0.5
Explained Variance

Summary

0.4
0.3
0.2
0.1
0.0

1234
Task index

567

Uncertainty in Deep Learning Workshop