Learning Neural Networks with Adaptive
Regularization
Han Zhaohan.zhao@cs.cmu.edu
Machine Learning Department, Carnegie Mellon University

Sep. 27th, 2019

Joint work with Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, and Geoﬀrey J.
Gordon1

Recent Success of Deep Learning
ImageNet (Top 5) Classification Error:

2

Recent Success of Deep Learning
Go: AlphaGo vs Seudo Lee and Jie Ke

3

Recent Success of Deep Learning
Dota 2: OpenAI Five-

Beats professional player on One-on-One game, 2017-

Beats amateur and semi-professional teams on Five-on-Five games,
20184Recent Success of Deep Learning
Other application domains:-

Natural Language Processing: Neural Machine Translation(2014)
Artificial Game Playing: Atari (2015), AlphaGo (2016), StarCraft
II (2018)
Automatic Speech Recognition (2010)
Computer Vision: Image generation (2014)
Healthcare: Drug discovery…

5

Recent Success of Deep Learning
However, successful training of neural networks requires:-

Large-scale datasets
Powerful computational resources

ImageNet: ~1M images, ~1K classes
6

Recent Success of Deep Learning
However, successful training of neural networks requires:-

Large-scale datasets
Powerful computational resources

Go: training data ~ almost infinity: simulated self-playing game7

Recent Success of Deep Learning
However, successful training of neural networks requires:-

Large-scale datasets
Powerful computational resources

How to eﬀectively regularize the training of neural networks
when there is only few data available?

8Outline•

Deep Learning Preliminary•

Learning with Adaptive Regularization•

Experiments•

Conclusion

9

Deep Learning Preliminary
A simple fully-connected neural network:
n
D
=
{x
,
y
}
i i i=1 ,- Data set:<latexit sha1_base64="3LOznu55k6vwUT1HfPK/1hPK/8g=">AAAEp3icfZNLbxMxEMfdNkAJrxSOXCyiSiBFVbYglQORKtFK4UQJfUTKppHX6yRWvPbG9jbZWOaj8T24c4XPgDcJSuIIRlppdn5/z/gxE6WMKl2v/9jZ3Svdu/9g/2H50eMnT59VDp5fK5FJTK6wYEK2I6QIo5xcaaoZaaeSoCRi5CYafSz4zR2Rigp+qfOUdBM04LRPMdIu1Ku0wwTpIUbMnFnYgKEJp9MercG8R0PbM7QR2FteC8cZiuckpDxstW7jhcL9mHoNBqGtfQv7QiLGYBHt8G6vUq0f1ecGt51g6VTB0i56B3snYSxwlhCuMUNKdYJ6qrsGSU0xI7YcZoqkCI/QgHScy1FCVNfMr8DCQxeJoduC+7iG8+j6CoMSpfIkcsriwMpnRfBfTA8T68eKKsrCuW0wJ9ZCMLW5osi7He5kuv++ayhPM004XpyinzGoBSweC8ZUEqxZ7hyEJXUXAfEQSYS1e9LyIfRLpGgmbHmjRIJGRAqRuJtCHBPmNsDJBIskQTw27q1ia+Y9EEXmk/XohNDBUP8V9M3EF0ynKzj1YZ6vYO7D2WwFZ1t1J/+pqVK+pEXbft3GWhKyprj0Fc3mGm36lEjpGsqEd0iSVFEmuCeI3VSuJTjzE8SkT8ZFAlx0smtE3vBLjJ3GmsaaxFOcT9PVs5z7FVqtFWxZ60Yt8Adr27k+PgreHh1/eVc9/bAcun3wErwCr0EATsApaIILcAUw+A5+gl/gd+lN6XPputReSHd3lmtegA0roT9lpKOn</latexit>-

Model:

xi 2 Rd , yi 2 {0, 1}, 8i 2 [n]f (x; ✓) = (WL g(WL 1 · · · W2 g(W1 x)))<latexit sha1_base64="Z7TStNJx6HWv+I0fUH2MA3AQ+SI=">AAAEinicfZNbaxNBFMenbdQaq6b66MtgKTQPlmwqVImFgi1E6EOtbVPohjA7ezYZMpftzKS5LPth/DS+6qPfxtk0kmSCHtjlcH7/c87cTpRyZmyt9nttfaP06PGTzaflZ1vPX7ysbL+6NmqgKVxRxZW+iYgBziRcWWY53KQaiIg4tKL+54K37kEbpuSlHafQFqQrWcIosS7UqTSSvXA0aoS2B5ZU8REODesKstfqnOGu+2dn74I8pLGyBrc69WkswC6lWq12Kju1/drU8KoTzJwdNLPzzvbGYRgrOhAgLeXEmNugltp2RrRllENeDgcGUkL7pAu3zpVEgGln013meNdFYpwo7T5p8TS6mJERYcxYRE4piO0ZnxXBfzHbE7kfK7qYHE9tiTmxVYqb5Yyi7mr4dmCTD+2MyXRgQdKHXSQDjq3CxX3gmGmglo+dQ6hm7iAw7RFNqHW3Vt7FfouUTFReXmohSB+0UsKdFJEUuFuAhCFVQhAZZyGTcZ6FRW4UZV9yjw6BdXv2ryDJhr5gNJrDkQ/H4zkc+3AymcPJSt/hf3qaVM4oJTz7toqtBlhQXPqKZnOBNn0KWrsHlYX3RENqGFfSE8Ru8BYKnPgFYkjgrihAi5fsHqI88lvcOU2eHS1IPMXpKJ1fy6nf4eJiDi/y3I1a4A/WqnNd3w8O9utf3+8cf5oN3SZ6g96iPRSgQ3SMmugcXSGKvqMf6Cf6Vdoq1UsfS40H6fraLOc1WrLSyR/0/5VT</latexit>

10

Deep Learning Preliminary
A simple fully-connected neural network:
n
D
=
{x
,
y
}
i i i=1 ,- Data set:<latexit sha1_base64="3LOznu55k6vwUT1HfPK/1hPK/8g=">AAAEp3icfZNLbxMxEMfdNkAJrxSOXCyiSiBFVbYglQORKtFK4UQJfUTKppHX6yRWvPbG9jbZWOaj8T24c4XPgDcJSuIIRlppdn5/z/gxE6WMKl2v/9jZ3Svdu/9g/2H50eMnT59VDp5fK5FJTK6wYEK2I6QIo5xcaaoZaaeSoCRi5CYafSz4zR2Rigp+qfOUdBM04LRPMdIu1Ku0wwTpIUbMnFnYgKEJp9MercG8R0PbM7QR2FteC8cZiuckpDxstW7jhcL9mHoNBqGtfQv7QiLGYBHt8G6vUq0f1ecGt51g6VTB0i56B3snYSxwlhCuMUNKdYJ6qrsGSU0xI7YcZoqkCI/QgHScy1FCVNfMr8DCQxeJoduC+7iG8+j6CoMSpfIkcsriwMpnRfBfTA8T68eKKsrCuW0wJ9ZCMLW5osi7He5kuv++ayhPM004XpyinzGoBSweC8ZUEqxZ7hyEJXUXAfEQSYS1e9LyIfRLpGgmbHmjRIJGRAqRuJtCHBPmNsDJBIskQTw27q1ia+Y9EEXmk/XohNDBUP8V9M3EF0ynKzj1YZ6vYO7D2WwFZ1t1J/+pqVK+pEXbft3GWhKyprj0Fc3mGm36lEjpGsqEd0iSVFEmuCeI3VSuJTjzE8SkT8ZFAlx0smtE3vBLjJ3GmsaaxFOcT9PVs5z7FVqtFWxZ60Yt8Adr27k+PgreHh1/eVc9/bAcun3wErwCr0EATsApaIILcAUw+A5+gl/gd+lN6XPputReSHd3lmtegA0roT9lpKOn</latexit>-

Model:

xi 2 Rd , yi 2 {0, 1}, 8i 2 [n]f (x; ✓) = (WL g(WL 1 · · · W2 g(W1 x)))<latexit sha1_base64="Z7TStNJx6HWv+I0fUH2MA3AQ+SI=">AAAEinicfZNbaxNBFMenbdQaq6b66MtgKTQPlmwqVImFgi1E6EOtbVPohjA7ezYZMpftzKS5LPth/DS+6qPfxtk0kmSCHtjlcH7/c87cTpRyZmyt9nttfaP06PGTzaflZ1vPX7ysbL+6NmqgKVxRxZW+iYgBziRcWWY53KQaiIg4tKL+54K37kEbpuSlHafQFqQrWcIosS7UqTSSvXA0aoS2B5ZU8REODesKstfqnOGu+2dn74I8pLGyBrc69WkswC6lWq12Kju1/drU8KoTzJwdNLPzzvbGYRgrOhAgLeXEmNugltp2RrRllENeDgcGUkL7pAu3zpVEgGln013meNdFYpwo7T5p8TS6mJERYcxYRE4piO0ZnxXBfzHbE7kfK7qYHE9tiTmxVYqb5Yyi7mr4dmCTD+2MyXRgQdKHXSQDjq3CxX3gmGmglo+dQ6hm7iAw7RFNqHW3Vt7FfouUTFReXmohSB+0UsKdFJEUuFuAhCFVQhAZZyGTcZ6FRW4UZV9yjw6BdXv2ryDJhr5gNJrDkQ/H4zkc+3AymcPJSt/hf3qaVM4oJTz7toqtBlhQXPqKZnOBNn0KWrsHlYX3RENqGFfSE8Ru8BYKnPgFYkjgrihAi5fsHqI88lvcOU2eHS1IPMXpKJ1fy6nf4eJiDi/y3I1a4A/WqnNd3w8O9utf3+8cf5oN3SZ6g96iPRSgQ3SMmugcXSGKvqMf6Cf6Vdoq1UsfS40H6fraLOc1WrLSyR/0/5VT</latexit>-

Output layer activation function: (·), the sigmoid/softmax
function for classification, linear for regression<latexit sha1_base64="8eBDVMQ2WxeHpO0KZJwT4MF7hww=">AAAEWnicfZPNahsxEMeV2G0Tpx9J21svS00gvRg7KaSHBgJtwL2lbpwEvCZotbO2sD7Wkjb2Wuxz9No+VqEPU63jYlumHVgY5vefGe1IE6WMatNs/trarlQfPX6ys1vbe/rs+Yv9g5fXWmaKQJdIJtVthDUwKqBrqGFwmyrAPGJwE40+lfzmHpSmUlyZPIU+xwNBE0qwcaF+qOmA46OQxNK8u9uvNxvNuQWbTmvh1NHCLu8OKqdhLEnGQRjCsNa9VjM1fYuVoYRBUQszDSkmIzyAnnMF5qD7dn7qIjh0kThIpHKfMME8upphMdc655FTcmyG2mdl8F/MDHnhx8ouugjmtsac2EjJ9HpGWXcz3MtM8qFvqUgzA4I8/EWSscDIoJxvEFMFxLDcOZgo6gYRkCFWmBh3C7XDwG+R4pksamstOB6BkpK7SWFBgLkDCJgQyTkWsQ2piAsblrlRZL8UHp0AHQzNX0FiJ75gOl3CqQ/zfAlzH85mSzjb6Dv5T0+digUlmNlvm9gogBXFla9ot1do26eglHtQNrzHClJNmRSeIHaLtFLgs18ghgTGZQFSvmT3EMWZ32LsNIU9W5F4iotpuryWC79Dp7OEnaJwq9byF2vTuT5utE4ax1/f188/LpZuB71Bb9ERaqFTdI7a6BJ1EUFj9B39QD8rv6vb1d3q3oN0e2uR8wqtWfX1HzwFhlA=</latexit>

11

Deep Learning Preliminary
A simple fully-connected neural network:
n
D
=
{x
,
y
}
i i i=1 ,- Data set:<latexit sha1_base64="3LOznu55k6vwUT1HfPK/1hPK/8g=">AAAEp3icfZNLbxMxEMfdNkAJrxSOXCyiSiBFVbYglQORKtFK4UQJfUTKppHX6yRWvPbG9jbZWOaj8T24c4XPgDcJSuIIRlppdn5/z/gxE6WMKl2v/9jZ3Svdu/9g/2H50eMnT59VDp5fK5FJTK6wYEK2I6QIo5xcaaoZaaeSoCRi5CYafSz4zR2Rigp+qfOUdBM04LRPMdIu1Ku0wwTpIUbMnFnYgKEJp9MercG8R0PbM7QR2FteC8cZiuckpDxstW7jhcL9mHoNBqGtfQv7QiLGYBHt8G6vUq0f1ecGt51g6VTB0i56B3snYSxwlhCuMUNKdYJ6qrsGSU0xI7YcZoqkCI/QgHScy1FCVNfMr8DCQxeJoduC+7iG8+j6CoMSpfIkcsriwMpnRfBfTA8T68eKKsrCuW0wJ9ZCMLW5osi7He5kuv++ayhPM004XpyinzGoBSweC8ZUEqxZ7hyEJXUXAfEQSYS1e9LyIfRLpGgmbHmjRIJGRAqRuJtCHBPmNsDJBIskQTw27q1ia+Y9EEXmk/XohNDBUP8V9M3EF0ynKzj1YZ6vYO7D2WwFZ1t1J/+pqVK+pEXbft3GWhKyprj0Fc3mGm36lEjpGsqEd0iSVFEmuCeI3VSuJTjzE8SkT8ZFAlx0smtE3vBLjJ3GmsaaxFOcT9PVs5z7FVqtFWxZ60Yt8Adr27k+PgreHh1/eVc9/bAcun3wErwCr0EATsApaIILcAUw+A5+gl/gd+lN6XPputReSHd3lmtegA0roT9lpKOn</latexit>-

Model:

xi 2 Rd , yi 2 {0, 1}, 8i 2 [n]f (x; ✓) = (WL g(WL 1 · · · W2 g(W1 x)))<latexit sha1_base64="Z7TStNJx6HWv+I0fUH2MA3AQ+SI=">AAAEinicfZNbaxNBFMenbdQaq6b66MtgKTQPlmwqVImFgi1E6EOtbVPohjA7ezYZMpftzKS5LPth/DS+6qPfxtk0kmSCHtjlcH7/c87cTpRyZmyt9nttfaP06PGTzaflZ1vPX7ysbL+6NmqgKVxRxZW+iYgBziRcWWY53KQaiIg4tKL+54K37kEbpuSlHafQFqQrWcIosS7UqTSSvXA0aoS2B5ZU8REODesKstfqnOGu+2dn74I8pLGyBrc69WkswC6lWq12Kju1/drU8KoTzJwdNLPzzvbGYRgrOhAgLeXEmNugltp2RrRllENeDgcGUkL7pAu3zpVEgGln013meNdFYpwo7T5p8TS6mJERYcxYRE4piO0ZnxXBfzHbE7kfK7qYHE9tiTmxVYqb5Yyi7mr4dmCTD+2MyXRgQdKHXSQDjq3CxX3gmGmglo+dQ6hm7iAw7RFNqHW3Vt7FfouUTFReXmohSB+0UsKdFJEUuFuAhCFVQhAZZyGTcZ6FRW4UZV9yjw6BdXv2ryDJhr5gNJrDkQ/H4zkc+3AymcPJSt/hf3qaVM4oJTz7toqtBlhQXPqKZnOBNn0KWrsHlYX3RENqGFfSE8Ru8BYKnPgFYkjgrihAi5fsHqI88lvcOU2eHS1IPMXpKJ1fy6nf4eJiDi/y3I1a4A/WqnNd3w8O9utf3+8cf5oN3SZ6g96iPRSgQ3SMmugcXSGKvqMf6Cf6Vdoq1UsfS40H6fraLOc1WrLSyR/0/5VT</latexit>-

Output layer activation function: (·), the sigmoid/softmax
function for classification, linear for regression
Hidden layer activation function: g(t) = max{0, t}, the ReLU
function<latexit sha1_base64="8eBDVMQ2WxeHpO0KZJwT4MF7hww=">AAAEWnicfZPNahsxEMeV2G0Tpx9J21svS00gvRg7KaSHBgJtwL2lbpwEvCZotbO2sD7Wkjb2Wuxz9No+VqEPU63jYlumHVgY5vefGe1IE6WMatNs/trarlQfPX6ys1vbe/rs+Yv9g5fXWmaKQJdIJtVthDUwKqBrqGFwmyrAPGJwE40+lfzmHpSmUlyZPIU+xwNBE0qwcaF+qOmA46OQxNK8u9uvNxvNuQWbTmvh1NHCLu8OKqdhLEnGQRjCsNa9VjM1fYuVoYRBUQszDSkmIzyAnnMF5qD7dn7qIjh0kThIpHKfMME8upphMdc655FTcmyG2mdl8F/MDHnhx8ouugjmtsac2EjJ9HpGWXcz3MtM8qFvqUgzA4I8/EWSscDIoJxvEFMFxLDcOZgo6gYRkCFWmBh3C7XDwG+R4pksamstOB6BkpK7SWFBgLkDCJgQyTkWsQ2piAsblrlRZL8UHp0AHQzNX0FiJ75gOl3CqQ/zfAlzH85mSzjb6Dv5T0+digUlmNlvm9gogBXFla9ot1do26eglHtQNrzHClJNmRSeIHaLtFLgs18ghgTGZQFSvmT3EMWZ32LsNIU9W5F4iotpuryWC79Dp7OEnaJwq9byF2vTuT5utE4ax1/f188/LpZuB71Bb9ERaqFTdI7a6BJ1EUFj9B39QD8rv6vb1d3q3oN0e2uR8wqtWfX1HzwFhlA=</latexit><latexit sha1_base64="B2RCqwW7dNmSwy5aZAN2WN0qPJ0=">AAAEYnicfZPfaxpBEMc30bap/RGtj+3DUgmkUETTQvpQIdAG7FtqYxLwRPb2Rl3cH5fdNXoe97f0tf2T+t4/pHvGoq60AwfDfL4zsze7E8acGdto/NrbLxQfPHx08Lj05Omz54flyosro6aaQpcqrvRNSAxwJqFrmeVwE2sgIuRwHU4+5fz6DrRhSl7aJIa+ICPJhowS60KDcnV0bN/gFg4EmQdp4y22QTYo1xr1xtLwrtNcOTW0sotBpXAaRIpOBUhLOTGm12zEtp8SbRnlkJWCqYGY0AkZQc+5kggw/XR5+gwfuUiEh0q7T1q8jG5mpEQYk4jQKQWxY+OzPPgvZsci82N5F5PhpW0xJ7ZKcbOdkdfdDfemdvihnzIZTy1Iev8XwynHVuF8zjhiGqjliXMI1cwNAtMx0YRadxulI+y3iMlCZaWtFoJMQCsl3KSIpMDdASTMqBKCyCgNmIyyNMhzwzD9knl0Bmw0tn8Fw3TmC+bzNZz7MEnWMPHhYrGGi52+s//0NLFcUUp4+m0XWw2wobj0Fe32Bm37FLR2DyoN7oiG2DCupCeI3EJtFPjsF4hgCLd5AZq/ZPcQZctvces0WdrakHiK83m8vpZzv0Ons4adLF+1pr9Yu87VSb35rn7y9X3t7ONq6Q7QS/QaHaMmOkVnqI0uUBdRlKDv6Af6WfhdLBUrxeq9dH9vlVNFW1Z89QeCGofk</latexit>

12

Deep Learning Preliminary
A simple fully-connected neural network:
n
D
=
{x
,
y
}
i i i=1 ,- Data set:<latexit sha1_base64="3LOznu55k6vwUT1HfPK/1hPK/8g=">AAAEp3icfZNLbxMxEMfdNkAJrxSOXCyiSiBFVbYglQORKtFK4UQJfUTKppHX6yRWvPbG9jbZWOaj8T24c4XPgDcJSuIIRlppdn5/z/gxE6WMKl2v/9jZ3Svdu/9g/2H50eMnT59VDp5fK5FJTK6wYEK2I6QIo5xcaaoZaaeSoCRi5CYafSz4zR2Rigp+qfOUdBM04LRPMdIu1Ku0wwTpIUbMnFnYgKEJp9MercG8R0PbM7QR2FteC8cZiuckpDxstW7jhcL9mHoNBqGtfQv7QiLGYBHt8G6vUq0f1ecGt51g6VTB0i56B3snYSxwlhCuMUNKdYJ6qrsGSU0xI7YcZoqkCI/QgHScy1FCVNfMr8DCQxeJoduC+7iG8+j6CoMSpfIkcsriwMpnRfBfTA8T68eKKsrCuW0wJ9ZCMLW5osi7He5kuv++ayhPM004XpyinzGoBSweC8ZUEqxZ7hyEJXUXAfEQSYS1e9LyIfRLpGgmbHmjRIJGRAqRuJtCHBPmNsDJBIskQTw27q1ia+Y9EEXmk/XohNDBUP8V9M3EF0ynKzj1YZ6vYO7D2WwFZ1t1J/+pqVK+pEXbft3GWhKyprj0Fc3mGm36lEjpGsqEd0iSVFEmuCeI3VSuJTjzE8SkT8ZFAlx0smtE3vBLjJ3GmsaaxFOcT9PVs5z7FVqtFWxZ60Yt8Adr27k+PgreHh1/eVc9/bAcun3wErwCr0EATsApaIILcAUw+A5+gl/gd+lN6XPputReSHd3lmtegA0roT9lpKOn</latexit>-

Model:

xi 2 Rd , yi 2 {0, 1}, 8i 2 [n]f (x; ✓) = (WL g(WL 1 · · · W2 g(W1 x)))<latexit sha1_base64="Z7TStNJx6HWv+I0fUH2MA3AQ+SI=">AAAEinicfZNbaxNBFMenbdQaq6b66MtgKTQPlmwqVImFgi1E6EOtbVPohjA7ezYZMpftzKS5LPth/DS+6qPfxtk0kmSCHtjlcH7/c87cTpRyZmyt9nttfaP06PGTzaflZ1vPX7ysbL+6NmqgKVxRxZW+iYgBziRcWWY53KQaiIg4tKL+54K37kEbpuSlHafQFqQrWcIosS7UqTSSvXA0aoS2B5ZU8REODesKstfqnOGu+2dn74I8pLGyBrc69WkswC6lWq12Kju1/drU8KoTzJwdNLPzzvbGYRgrOhAgLeXEmNugltp2RrRllENeDgcGUkL7pAu3zpVEgGln013meNdFYpwo7T5p8TS6mJERYcxYRE4piO0ZnxXBfzHbE7kfK7qYHE9tiTmxVYqb5Yyi7mr4dmCTD+2MyXRgQdKHXSQDjq3CxX3gmGmglo+dQ6hm7iAw7RFNqHW3Vt7FfouUTFReXmohSB+0UsKdFJEUuFuAhCFVQhAZZyGTcZ6FRW4UZV9yjw6BdXv2ryDJhr5gNJrDkQ/H4zkc+3AymcPJSt/hf3qaVM4oJTz7toqtBlhQXPqKZnOBNn0KWrsHlYX3RENqGFfSE8Ru8BYKnPgFYkjgrihAi5fsHqI88lvcOU2eHS1IPMXpKJ1fy6nf4eJiDi/y3I1a4A/WqnNd3w8O9utf3+8cf5oN3SZ6g96iPRSgQ3SMmugcXSGKvqMf6Cf6Vdoq1UsfS40H6fraLOc1WrLSyR/0/5VT</latexit>

Output layer activation function: (·), the sigmoid/softmax
function for classification, linear for regression- Hidden layer activation function: g(t) = max{0, t}, the ReLU
function- Essentially: a sequence of aﬃnetransformation + component-wise nonlinear
activation-<latexit sha1_base64="8eBDVMQ2WxeHpO0KZJwT4MF7hww=">AAAEWnicfZPNahsxEMeV2G0Tpx9J21svS00gvRg7KaSHBgJtwL2lbpwEvCZotbO2sD7Wkjb2Wuxz9No+VqEPU63jYlumHVgY5vefGe1IE6WMatNs/trarlQfPX6ys1vbe/rs+Yv9g5fXWmaKQJdIJtVthDUwKqBrqGFwmyrAPGJwE40+lfzmHpSmUlyZPIU+xwNBE0qwcaF+qOmA46OQxNK8u9uvNxvNuQWbTmvh1NHCLu8OKqdhLEnGQRjCsNa9VjM1fYuVoYRBUQszDSkmIzyAnnMF5qD7dn7qIjh0kThIpHKfMME8upphMdc655FTcmyG2mdl8F/MDHnhx8ouugjmtsac2EjJ9HpGWXcz3MtM8qFvqUgzA4I8/EWSscDIoJxvEFMFxLDcOZgo6gYRkCFWmBh3C7XDwG+R4pksamstOB6BkpK7SWFBgLkDCJgQyTkWsQ2piAsblrlRZL8UHp0AHQzNX0FiJ75gOl3CqQ/zfAlzH85mSzjb6Dv5T0+digUlmNlvm9gogBXFla9ot1do26eglHtQNrzHClJNmRSeIHaLtFLgs18ghgTGZQFSvmT3EMWZ32LsNIU9W5F4iotpuryWC79Dp7OEnaJwq9byF2vTuT5utE4ax1/f188/LpZuB71Bb9ERaqFTdI7a6BJ1EUFj9B39QD8rv6vb1d3q3oN0e2uR8wqtWfX1HzwFhlA=</latexit><latexit sha1_base64="B2RCqwW7dNmSwy5aZAN2WN0qPJ0=">AAAEYnicfZPfaxpBEMc30bap/RGtj+3DUgmkUETTQvpQIdAG7FtqYxLwRPb2Rl3cH5fdNXoe97f0tf2T+t4/pHvGoq60AwfDfL4zsze7E8acGdto/NrbLxQfPHx08Lj05Omz54flyosro6aaQpcqrvRNSAxwJqFrmeVwE2sgIuRwHU4+5fz6DrRhSl7aJIa+ICPJhowS60KDcnV0bN/gFg4EmQdp4y22QTYo1xr1xtLwrtNcOTW0sotBpXAaRIpOBUhLOTGm12zEtp8SbRnlkJWCqYGY0AkZQc+5kggw/XR5+gwfuUiEh0q7T1q8jG5mpEQYk4jQKQWxY+OzPPgvZsci82N5F5PhpW0xJ7ZKcbOdkdfdDfemdvihnzIZTy1Iev8XwynHVuF8zjhiGqjliXMI1cwNAtMx0YRadxulI+y3iMlCZaWtFoJMQCsl3KSIpMDdASTMqBKCyCgNmIyyNMhzwzD9knl0Bmw0tn8Fw3TmC+bzNZz7MEnWMPHhYrGGi52+s//0NLFcUUp4+m0XWw2wobj0Fe32Bm37FLR2DyoN7oiG2DCupCeI3EJtFPjsF4hgCLd5AZq/ZPcQZctvces0WdrakHiK83m8vpZzv0Ons4adLF+1pr9Yu87VSb35rn7y9X3t7ONq6Q7QS/QaHaMmOkVnqI0uUBdRlKDv6Af6WfhdLBUrxeq9dH9vlVNFW1Z89QeCGofk</latexit>

13

Deep Learning Preliminary
A simple fully-connected neural network:-

Loss function: Maximum Log-likelihood Estimation (MLE)ŷ := f (x; ✓) =<latexit sha1_base64="2b7bTe3J2wZRbXygSMf1q49Kux4=">AAAE2nicfZPNbhMxEMedNkAJXy0cOHAxVJVSCapsQSoSRKoElYLEoZR+Sd0o8npnE6tee2s7TTbWXrihXnk5HoM3wJsEkjgCS6sdz+/vmV3PTJRxpk2j8bOyslq9dfvO2t3avfsPHj5a33h8qmVfUTihkkt1HhENnAk4McxwOM8UkDTicBZdfij52TUozaQ4NnkG7ZR0BUsYJca5Ous3YY8Ymxc4jCGBK5zUw5SYXpTYYYHf4dD0wJDtZqhZNyUhh8TUzzr2c4G7s82rwB2nsTQau+3uPCvJ33ihYt2e2V584ZAJXG+8xMF2Z32zsdMYL7xsBFNjE03XYWdjdS+MJe2nIAzlROuLoJGZtiXKMMqhqIV9DRmhl6QLF84UJAXdtuNbK/CW88Q4kco9wuCxd/6EJanWeRo5ZfkL2mel81/M9NLC95VZdIHHa4E5sZGS68UTZdxl90XfJG/blomsb0DQyV8kfY6NxGV9ccwUUMNzZxCqmLsITHtEEWpcF9S2sJ8iIyNZ1BZSpOQSlJSpuykiKHD3AQIGVKYpEbF19YoLO6lqZD8VHh1AWdY/gsQOfMFwOINDH+b5DOY+HI1mcLSUd/CfnDoTU0oJt1+XsVEAc4pjX9FqzdGWT0Ep11A2vCYKMs24FJ4gdoM8F+CjH2A8e2UAWnaya0TR9FNcOU1hm3MST3EwzGZlOfAzHB3N4FFRuFEL/MFaNk53d4LXO7tf3mzuv58O3Rp6hl6gOgrQHtpHLXSIThBFvypPK7jyvBpWv1W/V28m0pXK9MwTtLCqP34DB9y0AQ==</latexit>(WL g (WL 1 · · · W2 g (W1 x))) 2 (0, 1)

14Deep Learning Preliminary
A simple fully-connected neural network:-

Loss function: Maximum Log-likelihood Estimation (MLE)ŷ := f (x; ✓) =<latexit sha1_base64="2b7bTe3J2wZRbXygSMf1q49Kux4=">AAAE2nicfZPNbhMxEMedNkAJXy0cOHAxVJVSCapsQSoSRKoElYLEoZR+Sd0o8npnE6tee2s7TTbWXrihXnk5HoM3wJsEkjgCS6sdz+/vmV3PTJRxpk2j8bOyslq9dfvO2t3avfsPHj5a33h8qmVfUTihkkt1HhENnAk4McxwOM8UkDTicBZdfij52TUozaQ4NnkG7ZR0BUsYJca5Ous3YY8Ymxc4jCGBK5zUw5SYXpTYYYHf4dD0wJDtZqhZNyUhh8TUzzr2c4G7s82rwB2nsTQau+3uPCvJ33ihYt2e2V584ZAJXG+8xMF2Z32zsdMYL7xsBFNjE03XYWdjdS+MJe2nIAzlROuLoJGZtiXKMMqhqIV9DRmhl6QLF84UJAXdtuNbK/CW88Q4kco9wuCxd/6EJanWeRo5ZfkL2mel81/M9NLC95VZdIHHa4E5sZGS68UTZdxl90XfJG/blomsb0DQyV8kfY6NxGV9ccwUUMNzZxCqmLsITHtEEWpcF9S2sJ8iIyNZ1BZSpOQSlJSpuykiKHD3AQIGVKYpEbF19YoLO6lqZD8VHh1AWdY/gsQOfMFwOINDH+b5DOY+HI1mcLSUd/CfnDoTU0oJt1+XsVEAc4pjX9FqzdGWT0Ep11A2vCYKMs24FJ4gdoM8F+CjH2A8e2UAWnaya0TR9FNcOU1hm3MST3EwzGZlOfAzHB3N4FFRuFEL/MFaNk53d4LXO7tf3mzuv58O3Rp6hl6gOgrQHtpHLXSIThBFvypPK7jyvBpWv1W/V28m0pXK9MwTtLCqP34DB9y0AQ==</latexit>-(WL g (WL 1 · · · W2 g (W1 x))) 2 (0, 1)

Optimization: stochastic gradient descent✓<latexit sha1_base64="dZYI4k6w64YxmKxvEQ1i/aS+e40=">AAAEnXicfVNdaxNBFJ22UWv8SvXRBwdDoT4YkirUl0JAAxGk1Ng0hWwIs7N3kyGzM9uZ2eZj2X/ln9FX/SHOJpFsJuiFhcs5596zM3euH3OmTb3+Y2//oHTv/oPDh+VHj588fVY5en6tZaIodKnkUt34RANnArqGGQ43sQIS+Rx6/uRjzvfuQGkmxZWZxzCIyEiwkFFiLDSsXHhmDIZ4HEJDlJJTvALwW+yNSBQR7IWK0NSLiTKMcOwB5ycrzZusAC+RbFip1mv1ZeDdpLFOqmgdl8OjgzMvkDSJQBjKidb9Rj02gzRvSzlkZS/REBM6ISPo21SQCPQgXR48w8cWCXAolf2EwUu0WJGSSOt55FtlRMxYu1wO/osz4yhzsdxFZ3gZW5wVGym53q7I++7C/cSEHwYpE3FiQNDVKcKEYyNxPiIcMAXU8LlNCFXMXgSmY2KnYOwgy8fYtYjJQmblLYuITEBJGdmbIoICtz8gYEqlHagIUo+JwM4ur/X99HPmsFNgo7H5KwjTqSuYzTbkzCXn8w05d8nFYkMudnyn//HUsVizlPD02y5tFEBBceUq2u0C23ZZsG9fWcEdURBrxqVwBIHdxUKDT26DAEK4zRvQ/CXbhyjOXYtbq8nS84LEUbRm8WYsLdeh09mQnSxftYa7WLvJ9Wmt8a52+vV9tdlcL90heoleoxPUQGeoidroEnURRd/RT/QL/S69KrVKX0oXK+n+3rrmBdqKUu8PvXGg9g==</latexit>

✓

@`(✓)
@✓

15

Deep Learning Preliminary
Let’s consider a two-layer NN as illustration:-

Hidden layer h 2 R with output ŷ 2 R for regression problem:p<latexit sha1_base64="xOI6kVeGz7z3vz9Q9bOXoVOnyYw=">AAAEYXicfZNLbxoxEMedQNuEvgg95rIqitQTgjRSeomE1Eait5SGJBJLkdc7Gyz82Nje8LD2q/TafqWe+0XqBSrAqB1ppdH8/p6Z9XiilFFtms1fe/ul8pOnzw4OK89fvHz1unpUu9EyUwR6RDKp7iKsgVEBPUMNg7tUAeYRg9to/LHgt4+gNJXi2sxSGHB8L2hCCTYuNKzWQo7NKErsKA+pCLvdb+mwWm82mgsLdp3WyqmjlV0Nj0rnYSxJxkEYwrDW/VYzNQOLlaGEQV4JMw0pJmN8D33nCsxBD+yi+Tw4cZE4SKRynzDBIrp5wmKu9YxHTlm0qn1WBP/FzIjnfqyoovNgYVvMiY2UTG+fKPLuhvuZST4MLBVpZkCQ5V8kGQuMDIprDmKqgBg2cw4mirqLCMgIK0yMG0blJPBLpHgu88pWCY7HoKTk7qawIMBcAwImRHKORWzdtOLcLqcX2c+5RydA70fmryCxE18wna7h1Iez2RrOfDifr+F8p+7kPzV1KlaUYGa/7mKjADYU176i09mgHZ+CUu5B2fARK0g1ZVJ4gtjt00aCT36CGBJ4KBKQ4iW7hygu/BIPTpPbiw2Jp7icpuuxXPoVut017Oa5W7WWv1i7zs1po/W+cfrlrN5ur5buAB2jt+gdaqFz1EYddIV6iKAp+o5+oJ+l3+XDcrVcW0r391Zn3qAtKx//AVxiiSI=</latexit><latexit sha1_base64="BDyH9eGdqbFIiulEd7nETwB9B9s=">AAAEWnicfZPNahsxEMcV220Tpx9J21svS02gJ2OnhfQSMLQB95a6cRLwmqDVztrC+lhL2thrsc/Ra/tYhT5MtY6LbZl2YGGY339mtCNNlDKqTav1a69SrT16/GT/oH749NnzF0fHL6+1zBSBPpFMqtsIa2BUQN9Qw+A2VYB5xOAmmnwq+c09KE2luDJ5CkOOR4ImlGDjQsNwjI3Ni5CKsNe7O2q0mq2lBbtOe+U00Mou746rZ2EsScZBGMKw1oN2KzVDi5WhhEFRDzMNKSYTPIKBcwXmoId2eeoiOHGROEikcp8wwTK6mWEx1zrnkVNybMbaZ2XwX8yMeeHHyi66CJa2xZzYSMn0dkZZdzc8yEzycWipSDMDgjz8RZKxwMignG8QUwXEsNw5mCjqBhGQMVaYGHcL9ZPAb5HihSzqWy04noCSkrtJYUGAuQMImBHJORaxdXcVFzYsc6PIfik8OgM6Gpu/gsTOfMF8voZzH+b5GuY+XCzWcLHTd/afnjoVK0ows992sVEAG4orX9HtbtCuT0Ep96BseI8VpJoyKTxB7BZpo8Bnv0AMCUzLAqR8ye4hinO/xdRpCnu+IfEUF/N0fS0Xfodebw17ReFWre0v1q5zfdpsv2+efv3Q6HRWS7eP3qC36B1qozPUQV10ifqIoCn6jn6gn9XftUrtoHb4IK3srXJeoS2rvf4DGaGGzQ==</latexit>ŷ = aT h,h = g(W x),<latexit sha1_base64="Bbdq1NbW4I7rTAm6AjFf208GSH8=">AAAEpnicfZPfbtMwFMa9rcAo/za45MZimlSkqWoH0ripVAkmlRs0yrpOarriOE5r1bEz21mbWn40HoRrbuEdcNpCW1dgKdLJ+X0+X+LjE6aMKl2rfd/Z3Svdu/9g/2H50eMnT58dHD6/UiKTmHSwYEJeh0gRRjnpaKoZuU4lQUnISDccvy94945IRQW/1HlK+gkachpTjLRLDQ66wQhpk1vYgEGC9CiMDbI3l39fRvYEBrcZitYyTjusdIPp9PXJAnUDyoN2+8akgaYJUTCyg4OjWrU2X3A7qC+DI7BcF4PDvbMgEjhLCNeYIaV69Vqq+wZJTTEjthxkiqQIj9GQ9FzIkTPqm/kJWHjsMhGMhXQP13CeXd9hUKJUnoROWfyG8lmR/BfTo8T6ucJFWThfG8yJtRBMbe4o6m6ne5mO3/UN5WmmCceLv4gzBrWARa9gRCXBmuUuQFhSdxAQj5BEWLuOlo+hb5GimbDlDYsEjYkUInEnhTgmzH0AJxMskgTxyLi+RdYsOhuaj9ajE0KHI/1HEJuJL5hOV3DqwzxfwdyHs9kKzrZ8J//xVClfUoyY+bKNtSRkTXHpK1qtNdryKZHSXSgT3CFJUkWZ4J4gckO5VuCDXyAiMbktCuDiJruLyBu+xa3TWNNYk3iK82m6asu579Bur2DbFqNW9wdrO7g6rdbfVE8/vz1qNpdDtw9eglegAurgDDRBC1yADsDgG/gBfoJfpUrpU6lT6i6kuzvLPS/Axip9/Q1tcaNy</latexit>-

Loss function:<latexit sha1_base64="5068Ut2xndESTYPdnCH2BK1pbqw=">AAAEf3icfZPNbhMxEMfdNkAJH03hyMWiqmgliDYBqXCIVAkqhVsp/ZK6ofJ6ZxurXntrO002ll+Dp+EK78Db4E2DkjiCkVYaze/vmR2PJyk40yaKfq+srtXu3X+w/rD+6PGTpxuNzWenWg4UhRMquVTnCdHAmYATwwyH80IByRMOZ8n1x4qf3YLSTIpjUxbQy8mVYBmjxPjQZSOKgfOds9c4zonpJ5klbhd3cJwpQm3L2bbbifvE2NLhN7jc/da+bGxFzWhieNlpTZ0tNLXDy821vTiVdJCDMJQTrS9aUWF6lijDKAdXjwcaCkKvyRVceFeQHHTPTlpzeNtHUpxJ5T9h8CQ6f8KSXOsyT7yyakCHrAr+i5l+7sJYVUU7PLEF5sVGSq4XT1R5l8MXA5O971kmioEBQe+6yAYcG4mrIeCUKaCGl94hVDF/EZj2ib9y40dV38ZhiYKMpasvlMjJNSgpc39TRFDg/gcEDKnMcyJSGzOROns308R+dgEdArvqm7+CzA5DwWg0g6MQluUMliEcj2dwvFR3+J+auhBTSgm3X5exUQBziuNQ0e3O0W5IQSn/oGx8SxQUmnEpAkHqt20uwacwQQoZ3FQJaPWS/UMUnbDEjdc425mTBIqDUTEby0FY4ehoBo+c86vWChdr2TltN1tvm+0v77b296dLt45eoJdoB7XQHtpHXXSIThBF39EP9BP9qq3UXtWatehOuroyPfMcLVjtwx/TK5Jt</latexit>

1`(W, a) = (ŷ2

W 2 Rp⇥d
y)

2

Input
layer

Hiddenlayer

Outputlayer

Input #1
Input #2
OutputInput #3
Input #416

Deep Learning Preliminary
Let’s consider a two-layer NN as illustration:-

Hidden layer h 2 R with output ŷ 2 R for regression problem:p<latexit sha1_base64="xOI6kVeGz7z3vz9Q9bOXoVOnyYw=">AAAEYXicfZNLbxoxEMedQNuEvgg95rIqitQTgjRSeomE1Eait5SGJBJLkdc7Gyz82Nje8LD2q/TafqWe+0XqBSrAqB1ppdH8/p6Z9XiilFFtms1fe/ul8pOnzw4OK89fvHz1unpUu9EyUwR6RDKp7iKsgVEBPUMNg7tUAeYRg9to/LHgt4+gNJXi2sxSGHB8L2hCCTYuNKzWQo7NKErsKA+pCLvdb+mwWm82mgsLdp3WyqmjlV0Nj0rnYSxJxkEYwrDW/VYzNQOLlaGEQV4JMw0pJmN8D33nCsxBD+yi+Tw4cZE4SKRynzDBIrp5wmKu9YxHTlm0qn1WBP/FzIjnfqyoovNgYVvMiY2UTG+fKPLuhvuZST4MLBVpZkCQ5V8kGQuMDIprDmKqgBg2cw4mirqLCMgIK0yMG0blJPBLpHgu88pWCY7HoKTk7qawIMBcAwImRHKORWzdtOLcLqcX2c+5RydA70fmryCxE18wna7h1Iez2RrOfDifr+F8p+7kPzV1KlaUYGa/7mKjADYU176i09mgHZ+CUu5B2fARK0g1ZVJ4gtjt00aCT36CGBJ4KBKQ4iW7hygu/BIPTpPbiw2Jp7icpuuxXPoVut017Oa5W7WWv1i7zs1po/W+cfrlrN5ur5buAB2jt+gdaqFz1EYddIV6iKAp+o5+oJ+l3+XDcrVcW0r391Zn3qAtKx//AVxiiSI=</latexit><latexit sha1_base64="BDyH9eGdqbFIiulEd7nETwB9B9s=">AAAEWnicfZPNahsxEMcV220Tpx9J21svS02gJ2OnhfQSMLQB95a6cRLwmqDVztrC+lhL2thrsc/Ra/tYhT5MtY6LbZl2YGGY339mtCNNlDKqTav1a69SrT16/GT/oH749NnzF0fHL6+1zBSBPpFMqtsIa2BUQN9Qw+A2VYB5xOAmmnwq+c09KE2luDJ5CkOOR4ImlGDjQsNwjI3Ni5CKsNe7O2q0mq2lBbtOe+U00Mou746rZ2EsScZBGMKw1oN2KzVDi5WhhEFRDzMNKSYTPIKBcwXmoId2eeoiOHGROEikcp8wwTK6mWEx1zrnkVNybMbaZ2XwX8yMeeHHyi66CJa2xZzYSMn0dkZZdzc8yEzycWipSDMDgjz8RZKxwMignG8QUwXEsNw5mCjqBhGQMVaYGHcL9ZPAb5HihSzqWy04noCSkrtJYUGAuQMImBHJORaxdXcVFzYsc6PIfik8OgM6Gpu/gsTOfMF8voZzH+b5GuY+XCzWcLHTd/afnjoVK0ows992sVEAG4orX9HtbtCuT0Ep96BseI8VpJoyKTxB7BZpo8Bnv0AMCUzLAqR8ye4hinO/xdRpCnu+IfEUF/N0fS0Xfodebw17ReFWre0v1q5zfdpsv2+efv3Q6HRWS7eP3qC36B1qozPUQV10ifqIoCn6jn6gn9XftUrtoHb4IK3srXJeoS2rvf4DGaGGzQ==</latexit>ŷ = aT h,h = g(W x),<latexit sha1_base64="Bbdq1NbW4I7rTAm6AjFf208GSH8=">AAAEpnicfZPfbtMwFMa9rcAo/za45MZimlSkqWoH0ripVAkmlRs0yrpOarriOE5r1bEz21mbWn40HoRrbuEdcNpCW1dgKdLJ+X0+X+LjE6aMKl2rfd/Z3Svdu/9g/2H50eMnT58dHD6/UiKTmHSwYEJeh0gRRjnpaKoZuU4lQUnISDccvy94945IRQW/1HlK+gkachpTjLRLDQ66wQhpk1vYgEGC9CiMDbI3l39fRvYEBrcZitYyTjusdIPp9PXJAnUDyoN2+8akgaYJUTCyg4OjWrU2X3A7qC+DI7BcF4PDvbMgEjhLCNeYIaV69Vqq+wZJTTEjthxkiqQIj9GQ9FzIkTPqm/kJWHjsMhGMhXQP13CeXd9hUKJUnoROWfyG8lmR/BfTo8T6ucJFWThfG8yJtRBMbe4o6m6ne5mO3/UN5WmmCceLv4gzBrWARa9gRCXBmuUuQFhSdxAQj5BEWLuOlo+hb5GimbDlDYsEjYkUInEnhTgmzH0AJxMskgTxyLi+RdYsOhuaj9ajE0KHI/1HEJuJL5hOV3DqwzxfwdyHs9kKzrZ8J//xVClfUoyY+bKNtSRkTXHpK1qtNdryKZHSXSgT3CFJUkWZ4J4gckO5VuCDXyAiMbktCuDiJruLyBu+xa3TWNNYk3iK82m6asu579Bur2DbFqNW9wdrO7g6rdbfVE8/vz1qNpdDtw9eglegAurgDDRBC1yADsDgG/gBfoJfpUrpU6lT6i6kuzvLPS/Axip9/Q1tcaNy</latexit>-

Loss function:<latexit sha1_base64="5068Ut2xndESTYPdnCH2BK1pbqw=">AAAEf3icfZPNbhMxEMfdNkAJH03hyMWiqmgliDYBqXCIVAkqhVsp/ZK6ofJ6ZxurXntrO002ll+Dp+EK78Db4E2DkjiCkVYaze/vmR2PJyk40yaKfq+srtXu3X+w/rD+6PGTpxuNzWenWg4UhRMquVTnCdHAmYATwwyH80IByRMOZ8n1x4qf3YLSTIpjUxbQy8mVYBmjxPjQZSOKgfOds9c4zonpJ5klbhd3cJwpQm3L2bbbifvE2NLhN7jc/da+bGxFzWhieNlpTZ0tNLXDy821vTiVdJCDMJQTrS9aUWF6lijDKAdXjwcaCkKvyRVceFeQHHTPTlpzeNtHUpxJ5T9h8CQ6f8KSXOsyT7yyakCHrAr+i5l+7sJYVUU7PLEF5sVGSq4XT1R5l8MXA5O971kmioEBQe+6yAYcG4mrIeCUKaCGl94hVDF/EZj2ib9y40dV38ZhiYKMpasvlMjJNSgpc39TRFDg/gcEDKnMcyJSGzOROns308R+dgEdArvqm7+CzA5DwWg0g6MQluUMliEcj2dwvFR3+J+auhBTSgm3X5exUQBziuNQ0e3O0W5IQSn/oGx8SxQUmnEpAkHqt20uwacwQQoZ3FQJaPWS/UMUnbDEjdc425mTBIqDUTEby0FY4ehoBo+c86vWChdr2TltN1tvm+0v77b296dLt45eoJdoB7XQHtpHXXSIThBF39EP9BP9qq3UXtWatehOuroyPfMcLVjtwx/TK5Jt</latexit>-

1`(W, a) = (ŷ2

W 2 Rp⇥dy)2Consider the update formula for weight matrix W :<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

W<latexit sha1_base64="rEvTqU47rdSWclaA+M06aAAtk5A=">AAAEl3icfZPNbhMxEMfdNkAJXymcEBeLqKI9UCUFqVwqIkFRuLWhbSplQ+X1ziZWvPbWdppsVn4lnoYLB3gWvEmqJI5gpJVm5/f3jO3xhCln2tRqvzY2t0r37j/Yflh+9PjJ02eVneeXWg4VhQsquVRXIdHAmYALwwyHq1QBSUIO7XDwqeDtW1CaSXFushS6CekJFjNKjAtdV5ptHHCIDVFKjnAbv8VBjyQJwXtBn5g8sy6S7e8FCTH9MM6JDShTFN/99+2bfRyMx9/PryvV2kFtanjdqc+dKprb6fXO1lEQSTpMQBjKidadei013ZwowygHWw6GGlJCB6QHHecKkoDu5tMjW7zrIhGOpXKfMHgaXV6Rk0TrLAmdstiq9lkR/Bcz/cT6saKKtnhqK8yJjZRcr64o8q6HO0MTf+jmTKRDA4LOThEPOTYSF83BEVNADc+cQ6hi7iIw7RNFqHEtLO9iv0RKJtKWV0okZABKysTdFBEUuNuAgBGVrqciygMmIpvPuhfmX61HR8B6fXMniPORLxiPF3DswyxbwMyHk8kCTtbqjv5TU6diTinh+bd1bBTAkuLcVzSbS7TpU3AvXznBLVGQasal8ASRm8KlBJ/9BBHEcFMkoMVLdg9RHPslbpzG5sdLEk9xMk4XbTnxK7RaC9iy1o1a3R+sdefy8KD+7uDw7H210ZgP3TZ6hV6jPVRHR6iBmugUXSCKfqCf6Df6U3pZ+lj6UmrOpJsb8zUv0IqVzv4C2sGcZQ==</latexit>

W

(ŷ

0

y)(a h )x

T

Input
layer

Hiddenlayer

Outputlayer

Input #1
Input #2
OutputInput #3
Input #417

Deep Learning Preliminary
Let’s consider a two-layer NN as illustration:-

Gradient update formula for weight matrix W :<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

W<latexit sha1_base64="rEvTqU47rdSWclaA+M06aAAtk5A=">AAAEl3icfZPNbhMxEMfdNkAJXymcEBeLqKI9UCUFqVwqIkFRuLWhbSplQ+X1ziZWvPbWdppsVn4lnoYLB3gWvEmqJI5gpJVm5/f3jO3xhCln2tRqvzY2t0r37j/Yflh+9PjJ02eVneeXWg4VhQsquVRXIdHAmYALwwyHq1QBSUIO7XDwqeDtW1CaSXFushS6CekJFjNKjAtdV5ptHHCIDVFKjnAbv8VBjyQJwXtBn5g8sy6S7e8FCTH9MM6JDShTFN/99+2bfRyMx9/PryvV2kFtanjdqc+dKprb6fXO1lEQSTpMQBjKidadei013ZwowygHWw6GGlJCB6QHHecKkoDu5tMjW7zrIhGOpXKfMHgaXV6Rk0TrLAmdstiq9lkR/Bcz/cT6saKKtnhqK8yJjZRcr64o8q6HO0MTf+jmTKRDA4LOThEPOTYSF83BEVNADc+cQ6hi7iIw7RNFqHEtLO9iv0RKJtKWV0okZABKysTdFBEUuNuAgBGVrqciygMmIpvPuhfmX61HR8B6fXMniPORLxiPF3DswyxbwMyHk8kCTtbqjv5TU6diTinh+bd1bBTAkuLcVzSbS7TpU3AvXznBLVGQasal8ASRm8KlBJ/9BBHEcFMkoMVLdg9RHPslbpzG5sdLEk9xMk4XbTnxK7RaC9iy1o1a3R+sdefy8KD+7uDw7H210ZgP3TZ6hV6jPVRHR6iBmugUXSCKfqCf6Df6U3pZ+lj6UmrOpJsb8zUv0IqVzv4C2sGcZQ==</latexit>

-

W

(ŷ

y)(a h0 )xT

h0: component-wise derivative of h w.r.t. its input argument<latexit sha1_base64="Qnh0LKhisAHt2yCMr8Z9LhybTPE=">AAAEWHicfZNNbxoxEIadsG0S+pWkx15WRVF7QpBUSi+RkNpI9JbSkEQCFHm9s2Dhj43tDSzW/oxe29/V/pp6gQowakdaaTTP63m9tidKGdWm0fi1s1sJnjzd2z+oPnv+4uWrw6PjGy0zRaBLJJPqLsIaGBXQNdQwuEsVYB4xuI3Gn0p++whKUymuTZ7CgOOhoAkl2LhSr8+xGUWJHRXv7g9rjXpjHuF20lwmNbSMq/ujynk/liTjIAxhWOtes5GagcXKUMKgqPYzDSkmYzyEnksF5qAHdr7nIjxxlThMpHKfMOG8ur7CYq51ziOnLPeofVYW/8XMiBd+rXTRRTiPDebERkqmN1eUfbfLvcwkHweWijQzIMjiL5KMhUaG5emGMVVADMtdgomi7iBCMsIKE+PuoHoS+hYpnsmiumHB8RiUlNydFBYEmNuAgAmRnGMR2z4VcWEX1xbZL4VHJ0CHI/NXkNiJL5hOV3DqwzxfwdyHs9kKzrZ8J//x1KlYUoKZ/baNjQJYU1z7inZ7jbZ9Ckq5B2X7j1hBqimTwhPEbozWGnz2G8SQwEPZgJQv2T1EceFbPDhNYS/WJJ7icpquruXSd+h0VrBTFG7Umv5gbSc3p/XmWf3064daq7Ucun30Br1F71ETnaMWaqMr1EUESfQd/UA/K78DFOwFBwvp7s5yzWu0EcHxHzRphdE=</latexit><latexit sha1_base64="uI9tuE+QtZWpxTrhiEoAirL+hV8=">AAAEV3icfZNNbxoxEIadQFNKv0J77GVVFKknBGml9BIJKYlEbykNSVRAkdc7Cxb+2NjewGLtv+i1/V/5Na0XqACjdqSVRvO8ntdre8KEUW2azce9/VL5ycHTyrPq8xcvX70+rL251jJVBHpEMqluQ6yBUQE9Qw2D20QB5iGDm3ByVvCbB1CaSnFlsgSGHI8EjSnBxpW+Dzg24zC24/zusN5sNBcR7CatVVJHq7i8q5VOBpEkKQdhCMNa91vNxAwtVoYSBnl1kGpIMJngEfRdKjAHPbSLLefBkatEQSyV+4QJFtXNFRZzrTMeOmWxRe2zovgvZsY892uFi86DRWwxJzZSMr29oui7W+6nJv48tFQkqQFBln8RpywwMigON4ioAmJY5hJMFHUHEZAxVpgYdwXVo8C3SPBc5tUtC44noKTk7qSwIMDcBgRMieQci8gOqIhyu7y10H7JPToFOhqbv4LYTn3BbLaGMx9m2RpmPpzP13C+4zv9j6dOxIoSzOy3XWwUwIbiyld0Ohu041NQyj0oO3jAChJNmRSeIHJTtNHg3G8QQQz3RQNSvGT3EMWpb3HvNLk93ZB4iotZsr6WC9+h213Dbl6MWssfrN3k+rjR+tg4/vqp3m6vhq6C3qH36ANqoRPURh10iXqIIIF+oJ/oV+mx9Lt8UK4spft7qzVv0VaUa38AlQSGnw==</latexit>

Input
layer

Hiddenlayer

Outputlayer

Input #1
Input #2
OutputInput #3
Input #418

Deep Learning Preliminary
Let’s consider a two-layer NN as illustration:-

Gradient update formula for weight matrix W :<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

WW<latexit sha1_base64="rEvTqU47rdSWclaA+M06aAAtk5A=">AAAEl3icfZPNbhMxEMfdNkAJXymcEBeLqKI9UCUFqVwqIkFRuLWhbSplQ+X1ziZWvPbWdppsVn4lnoYLB3gWvEmqJI5gpJVm5/f3jO3xhCln2tRqvzY2t0r37j/Yflh+9PjJ02eVneeXWg4VhQsquVRXIdHAmYALwwyHq1QBSUIO7XDwqeDtW1CaSXFushS6CekJFjNKjAtdV5ptHHCIDVFKjnAbv8VBjyQJwXtBn5g8sy6S7e8FCTH9MM6JDShTFN/99+2bfRyMx9/PryvV2kFtanjdqc+dKprb6fXO1lEQSTpMQBjKidadei013ZwowygHWw6GGlJCB6QHHecKkoDu5tMjW7zrIhGOpXKfMHgaXV6Rk0TrLAmdstiq9lkR/Bcz/cT6saKKtnhqK8yJjZRcr64o8q6HO0MTf+jmTKRDA4LOThEPOTYSF83BEVNADc+cQ6hi7iIw7RNFqHEtLO9iv0RKJtKWV0okZABKysTdFBEUuNuAgBGVrqciygMmIpvPuhfmX61HR8B6fXMniPORLxiPF3DswyxbwMyHk8kCTtbqjv5TU6diTinh+bd1bBTAkuLcVzSbS7TpU3AvXznBLVGQasal8ASRm8KlBJ/9BBHEcFMkoMVLdg9RHPslbpzG5sdLEk9xMk4XbTnxK7RaC9iy1o1a3R+sdefy8KD+7uDw7H210ZgP3TZ6hV6jPVRHR6iBmugUXSCKfqCf6Df6U3pZ+lj6UmrOpJsb8zUv0IqVzv4C2sGcZQ==</latexit>-(ŷy)(a h0 )xT

h0: component-wise derivative of h w.r.t. its input argument
The gradient w.r.t. W is always rank-1: rows/columns of W
<latexit sha1_base64="Qnh0LKhisAHt2yCMr8Z9LhybTPE=">AAAEWHicfZNNbxoxEIadsG0S+pWkx15WRVF7QpBUSi+RkNpI9JbSkEQCFHm9s2Dhj43tDSzW/oxe29/V/pp6gQowakdaaTTP63m9tidKGdWm0fi1s1sJnjzd2z+oPnv+4uWrw6PjGy0zRaBLJJPqLsIaGBXQNdQwuEsVYB4xuI3Gn0p++whKUymuTZ7CgOOhoAkl2LhSr8+xGUWJHRXv7g9rjXpjHuF20lwmNbSMq/ujynk/liTjIAxhWOtes5GagcXKUMKgqPYzDSkmYzyEnksF5qAHdr7nIjxxlThMpHKfMOG8ur7CYq51ziOnLPeofVYW/8XMiBd+rXTRRTiPDebERkqmN1eUfbfLvcwkHweWijQzIMjiL5KMhUaG5emGMVVADMtdgomi7iBCMsIKE+PuoHoS+hYpnsmiumHB8RiUlNydFBYEmNuAgAmRnGMR2z4VcWEX1xbZL4VHJ0CHI/NXkNiJL5hOV3DqwzxfwdyHs9kKzrZ8J//x1KlYUoKZ/baNjQJYU1z7inZ7jbZ9Ckq5B2X7j1hBqimTwhPEbozWGnz2G8SQwEPZgJQv2T1EceFbPDhNYS/WJJ7icpquruXSd+h0VrBTFG7Umv5gbSc3p/XmWf3064daq7Ucun30Br1F71ETnaMWaqMr1EUESfQd/UA/K78DFOwFBwvp7s5yzWu0EcHxHzRphdE=</latexit><latexit sha1_base64="uI9tuE+QtZWpxTrhiEoAirL+hV8=">AAAEV3icfZNNbxoxEIadQFNKv0J77GVVFKknBGml9BIJKYlEbykNSVRAkdc7Cxb+2NjewGLtv+i1/V/5Na0XqACjdqSVRvO8ntdre8KEUW2azce9/VL5ycHTyrPq8xcvX70+rL251jJVBHpEMqluQ6yBUQE9Qw2D20QB5iGDm3ByVvCbB1CaSnFlsgSGHI8EjSnBxpW+Dzg24zC24/zusN5sNBcR7CatVVJHq7i8q5VOBpEkKQdhCMNa91vNxAwtVoYSBnl1kGpIMJngEfRdKjAHPbSLLefBkatEQSyV+4QJFtXNFRZzrTMeOmWxRe2zovgvZsY892uFi86DRWwxJzZSMr29oui7W+6nJv48tFQkqQFBln8RpywwMigON4ioAmJY5hJMFHUHEZAxVpgYdwXVo8C3SPBc5tUtC44noKTk7qSwIMDcBgRMieQci8gOqIhyu7y10H7JPToFOhqbv4LYTn3BbLaGMx9m2RpmPpzP13C+4zv9j6dOxIoSzOy3XWwUwIbiyld0Ohu041NQyj0oO3jAChJNmRSeIHJTtNHg3G8QQQz3RQNSvGT3EMWpb3HvNLk93ZB4iotZsr6WC9+h213Dbl6MWssfrN3k+rjR+tg4/vqp3m6vhq6C3qH36ANqoRPURh10iXqIIIF+oJ/oV+mx9Lt8UK4spft7qzVv0VaUa38AlQSGnw==</latexit><latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit><latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

are correlated
Input
layer

Hiddenlayer

Outputlayer

Input #1
Input #2
OutputInput #3
Input #419

Deep Learning Preliminary
Let’s consider a two-layer NN as illustration:-

Gradient update formula for weight matrix W :<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

WW<latexit sha1_base64="rEvTqU47rdSWclaA+M06aAAtk5A=">AAAEl3icfZPNbhMxEMfdNkAJXymcEBeLqKI9UCUFqVwqIkFRuLWhbSplQ+X1ziZWvPbWdppsVn4lnoYLB3gWvEmqJI5gpJVm5/f3jO3xhCln2tRqvzY2t0r37j/Yflh+9PjJ02eVneeXWg4VhQsquVRXIdHAmYALwwyHq1QBSUIO7XDwqeDtW1CaSXFushS6CekJFjNKjAtdV5ptHHCIDVFKjnAbv8VBjyQJwXtBn5g8sy6S7e8FCTH9MM6JDShTFN/99+2bfRyMx9/PryvV2kFtanjdqc+dKprb6fXO1lEQSTpMQBjKidadei013ZwowygHWw6GGlJCB6QHHecKkoDu5tMjW7zrIhGOpXKfMHgaXV6Rk0TrLAmdstiq9lkR/Bcz/cT6saKKtnhqK8yJjZRcr64o8q6HO0MTf+jmTKRDA4LOThEPOTYSF83BEVNADc+cQ6hi7iIw7RNFqHEtLO9iv0RKJtKWV0okZABKysTdFBEUuNuAgBGVrqciygMmIpvPuhfmX61HR8B6fXMniPORLxiPF3DswyxbwMyHk8kCTtbqjv5TU6diTinh+bd1bBTAkuLcVzSbS7TpU3AvXznBLVGQasal8ASRm8KlBJ/9BBHEcFMkoMVLdg9RHPslbpzG5sdLEk9xMk4XbTnxK7RaC9iy1o1a3R+sdefy8KD+7uDw7H210ZgP3TZ6hV6jPVRHR6iBmugUXSCKfqCf6Df6U3pZ+lj6UmrOpJsb8zUv0IqVzv4C2sGcZQ==</latexit>-(ŷy)(a h0 )xT

h0: component-wise derivative of h w.r.t. its input argument
The gradient w.r.t. W is always rank-1: rows/columns of W
<latexit sha1_base64="Qnh0LKhisAHt2yCMr8Z9LhybTPE=">AAAEWHicfZNNbxoxEIadsG0S+pWkx15WRVF7QpBUSi+RkNpI9JbSkEQCFHm9s2Dhj43tDSzW/oxe29/V/pp6gQowakdaaTTP63m9tidKGdWm0fi1s1sJnjzd2z+oPnv+4uWrw6PjGy0zRaBLJJPqLsIaGBXQNdQwuEsVYB4xuI3Gn0p++whKUymuTZ7CgOOhoAkl2LhSr8+xGUWJHRXv7g9rjXpjHuF20lwmNbSMq/ujynk/liTjIAxhWOtes5GagcXKUMKgqPYzDSkmYzyEnksF5qAHdr7nIjxxlThMpHKfMOG8ur7CYq51ziOnLPeofVYW/8XMiBd+rXTRRTiPDebERkqmN1eUfbfLvcwkHweWijQzIMjiL5KMhUaG5emGMVVADMtdgomi7iBCMsIKE+PuoHoS+hYpnsmiumHB8RiUlNydFBYEmNuAgAmRnGMR2z4VcWEX1xbZL4VHJ0CHI/NXkNiJL5hOV3DqwzxfwdyHs9kKzrZ8J//x1KlYUoKZ/baNjQJYU1z7inZ7jbZ9Ckq5B2X7j1hBqimTwhPEbozWGnz2G8SQwEPZgJQv2T1EceFbPDhNYS/WJJ7icpquruXSd+h0VrBTFG7Umv5gbSc3p/XmWf3064daq7Ucun30Br1F71ETnaMWaqMr1EUESfQd/UA/K78DFOwFBwvp7s5yzWu0EcHxHzRphdE=</latexit><latexit sha1_base64="uI9tuE+QtZWpxTrhiEoAirL+hV8=">AAAEV3icfZNNbxoxEIadQFNKv0J77GVVFKknBGml9BIJKYlEbykNSVRAkdc7Cxb+2NjewGLtv+i1/V/5Na0XqACjdqSVRvO8ntdre8KEUW2azce9/VL5ycHTyrPq8xcvX70+rL251jJVBHpEMqluQ6yBUQE9Qw2D20QB5iGDm3ByVvCbB1CaSnFlsgSGHI8EjSnBxpW+Dzg24zC24/zusN5sNBcR7CatVVJHq7i8q5VOBpEkKQdhCMNa91vNxAwtVoYSBnl1kGpIMJngEfRdKjAHPbSLLefBkatEQSyV+4QJFtXNFRZzrTMeOmWxRe2zovgvZsY892uFi86DRWwxJzZSMr29oui7W+6nJv48tFQkqQFBln8RpywwMigON4ioAmJY5hJMFHUHEZAxVpgYdwXVo8C3SPBc5tUtC44noKTk7qSwIMDcBgRMieQci8gOqIhyu7y10H7JPToFOhqbv4LYTn3BbLaGMx9m2RpmPpzP13C+4zv9j6dOxIoSzOy3XWwUwIbiyld0Ohu041NQyj0oO3jAChJNmRSeIHJTtNHg3G8QQQz3RQNSvGT3EMWpb3HvNLk93ZB4iotZsr6WC9+h213Dbl6MWssfrN3k+rjR+tg4/vqp3m6vhq6C3qH36ANqoRPURh10iXqIIIF+oJ/oV+mx9Lt8UK4spft7qzVv0VaUa38AlQSGnw==</latexit><latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit><latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

are correlated
Input
layer

Hiddenlayer

Update each row/column by taking informationfrom other rows/columns into consideration?

Outputlayer

Input #1
Input #2

OutputInput #3
Input #420

Outline•

Deep Learning Preliminary•

Learning with Adaptive Regularization•

Experiments•

Conclusion

21

Learning with Adaptive Regularization
Recall the gradient update formula for W :<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

W<latexit sha1_base64="rEvTqU47rdSWclaA+M06aAAtk5A=">AAAEl3icfZPNbhMxEMfdNkAJXymcEBeLqKI9UCUFqVwqIkFRuLWhbSplQ+X1ziZWvPbWdppsVn4lnoYLB3gWvEmqJI5gpJVm5/f3jO3xhCln2tRqvzY2t0r37j/Yflh+9PjJ02eVneeXWg4VhQsquVRXIdHAmYALwwyHq1QBSUIO7XDwqeDtW1CaSXFushS6CekJFjNKjAtdV5ptHHCIDVFKjnAbv8VBjyQJwXtBn5g8sy6S7e8FCTH9MM6JDShTFN/99+2bfRyMx9/PryvV2kFtanjdqc+dKprb6fXO1lEQSTpMQBjKidadei013ZwowygHWw6GGlJCB6QHHecKkoDu5tMjW7zrIhGOpXKfMHgaXV6Rk0TrLAmdstiq9lkR/Bcz/cT6saKKtnhqK8yJjZRcr64o8q6HO0MTf+jmTKRDA4LOThEPOTYSF83BEVNADc+cQ6hi7iIw7RNFqHEtLO9iv0RKJtKWV0okZABKysTdFBEUuNuAgBGVrqciygMmIpvPuhfmX61HR8B6fXMniPORLxiPF3DswyxbwMyHk8kCTtbqjv5TU6diTinh+bd1bBTAkuLcVzSbS7TpU3AvXznBLVGQasal8ASRm8KlBJ/9BBHEcFMkoMVLdg9RHPslbpzG5sdLEk9xMk4XbTnxK7RaC9iy1o1a3R+sdefy8KD+7uDw7H210ZgP3TZ6hV6jPVRHR6iBmugUXSCKfqCf6Df6U3pZ+lj6UmrOpJsb8zUv0IqVzv4C2sGcZQ==</latexit>W(ŷy)(a h0 )xT

How can we make use of this structure?

22

Learning with Adaptive Regularization
Recall the gradient update formula for W :<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

W<latexit sha1_base64="rEvTqU47rdSWclaA+M06aAAtk5A=">AAAEl3icfZPNbhMxEMfdNkAJXymcEBeLqKI9UCUFqVwqIkFRuLWhbSplQ+X1ziZWvPbWdppsVn4lnoYLB3gWvEmqJI5gpJVm5/f3jO3xhCln2tRqvzY2t0r37j/Yflh+9PjJ02eVneeXWg4VhQsquVRXIdHAmYALwwyHq1QBSUIO7XDwqeDtW1CaSXFushS6CekJFjNKjAtdV5ptHHCIDVFKjnAbv8VBjyQJwXtBn5g8sy6S7e8FCTH9MM6JDShTFN/99+2bfRyMx9/PryvV2kFtanjdqc+dKprb6fXO1lEQSTpMQBjKidadei013ZwowygHWw6GGlJCB6QHHecKkoDu5tMjW7zrIhGOpXKfMHgaXV6Rk0TrLAmdstiq9lkR/Bcz/cT6saKKtnhqK8yJjZRcr64o8q6HO0MTf+jmTKRDA4LOThEPOTYSF83BEVNADc+cQ6hi7iIw7RNFqHEtLO9iv0RKJtKWV0okZABKysTdFBEUuNuAgBGVrqciygMmIpvPuhfmX61HR8B6fXMniPORLxiPF3DswyxbwMyHk8kCTtbqjv5TU6diTinh+bd1bBTAkuLcVzSbS7TpU3AvXznBLVGQasal8ASRm8KlBJ/9BBHEcFMkoMVLdg9RHPslbpzG5sdLEk9xMk4XbTnxK7RaC9iy1o1a3R+sdefy8KD+7uDw7H210ZgP3TZ6hV6jPVRHR6iBmugUXSCKfqCf6Df6U3pZ+lj6UmrOpJsb8zUv0IqVzv4C2sGcZQ==</latexit>W(ŷy)(a h0 )xT

How can we make use of this structure?

“Learning from the experience of others”— Prof. Bradley Efron
23

Learning with Adaptive Regularization
Recall the gradient update formula for W :<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

WW<latexit sha1_base64="rEvTqU47rdSWclaA+M06aAAtk5A=">AAAEl3icfZPNbhMxEMfdNkAJXymcEBeLqKI9UCUFqVwqIkFRuLWhbSplQ+X1ziZWvPbWdppsVn4lnoYLB3gWvEmqJI5gpJVm5/f3jO3xhCln2tRqvzY2t0r37j/Yflh+9PjJ02eVneeXWg4VhQsquVRXIdHAmYALwwyHq1QBSUIO7XDwqeDtW1CaSXFushS6CekJFjNKjAtdV5ptHHCIDVFKjnAbv8VBjyQJwXtBn5g8sy6S7e8FCTH9MM6JDShTFN/99+2bfRyMx9/PryvV2kFtanjdqc+dKprb6fXO1lEQSTpMQBjKidadei013ZwowygHWw6GGlJCB6QHHecKkoDu5tMjW7zrIhGOpXKfMHgaXV6Rk0TrLAmdstiq9lkR/Bcz/cT6saKKtnhqK8yJjZRcr64o8q6HO0MTf+jmTKRDA4LOThEPOTYSF83BEVNADc+cQ6hi7iIw7RNFqHEtLO9iv0RKJtKWV0okZABKysTdFBEUuNuAgBGVrqciygMmIpvPuhfmX61HR8B6fXMniPORLxiPF3DswyxbwMyHk8kCTtbqjv5TU6diTinh+bd1bBTAkuLcVzSbS7TpU3AvXznBLVGQasal8ASRm8KlBJ/9BBHEcFMkoMVLdg9RHPslbpzG5sdLEk9xMk4XbTnxK7RaC9iy1o1a3R+sdefy8KD+7uDw7H210ZgP3TZ6hV6jPVRHR6iBmugUXSCKfqCf6Df6U3pZ+lj6UmrOpJsb8zUv0IqVzv4C2sGcZQ==</latexit>(ŷy)(a h0 )xT

Now consider the following multivariate normal distribution:
W ⇠ MN (0p⇥d , ⌃r , ⌃c )<latexit sha1_base64="2jPnLCaGp3V4EB9VFeAvBrFw9kA=">AAAFGHicfZRLbxMxEMfdNkAJrxaOXCyqSkVC1aZFgkulSlApHIDSt9SNIq93klj1q7bzXO3X4MBn4cAFEDfEjW+DNw1N4hVYWu1ofn/P7HjGm2jOrIui3wuLS5UbN28t367euXvv/oOV1YcnVnUNhWOquDJnCbHAmYRjxxyHM22AiITDaXLxquCnPTCWKXnkhhoagrQlazFKnHc1V6JTHFsmcCyI61DCs7fv8o2omenYMQEWp/kzHFPVM1cv+rS5shZtRuOFy0ZtYqyhydpvri79jFNFuwKko5xYe16LtGtkxDhGOeTVuGtBE3pB2nDuTUl83kY2Li3H696T4pYy/pEOj72zOzIirB2KxCuLCmzICue/mOuIPPQVWWyOx2uOebFTitv5HUXcsvu861ovGxmTuutA0qsqWl2OncJFE3DKDFDHh94g1DB/EJh2iCHU+VZV13GYQpORyqtzKQS5AKOU8CdFJAXuP0BCnyohiEyzmMk0z8ZNTZLsTR7QPrB2x/0VtLJ+KBgMpnAQwuFwCochHI2mcFTK2/9PTqvlhBZzeFjGzgDMKI5CRb0+Q+shBWP8QGVxjxjQlnElA0Hqb9tMgNdhgBRacFkEoMUk+0GUO2GKS6/Js50ZSaDYG+hpW/bCDAcHU3iQFw2fpcUt9PyQtQVpmrwE6TWkYU/8pPT8wI3rnxx/FCbXpqDvBbTL0bWh18wH9/+AWnjjy8bJ1mZte3Prw/O13d3J32AZPUZP0AaqoRdoF9XRPjpGFH1En9FX9K3yqfKl8r3y40q6uDDZ8wjNrcqvP2Pr06Y=</latexit>

p
d
⌃
2
S
,
⌃
2
S
r
c
++ are row and column covariance matrices++<latexit sha1_base64="sBjluQ2xat1TU7PVAXr2Z4CQNYM=">AAAFLXicfZTNbhMxEMfdNkAJXy0cuayIKlUqipKCBAcqVaKVwonSNm2lbIi83klixV+1nc/V8gi8Cheu8BYckBBXTrwD3jQ0iVdgaaXR/P6esT2zEylGja1Uvq+srhVu3Ly1frt45+69+w82Nh+eGdnXBOpEMqkvImyAUQF1Sy2DC6UB84jBedR7nfHzAWhDpTi1YwVNjjuCtinB1rlaG9shkQMdUhGeHB20kp2d9L16+iFzkiVn3NooVcqV6QryRnVmlNBsHbU2136HsSR9DsISho1pVCvKNhOsLSUM0mLYN6Aw6eEONJwpMAfTTKZXSoMt54mDttTuEzaYehd3JJgbM+aRU3Jsu8ZnmfNfzHZ56vuyLCYNpmuJObGVkpnlHVncvLvRt+2XzYQK1bcgyNUt2n0WWBlkjx/EVAOxbOwMTDR1DxGQLtaYWFei4lbgp1B4ItPiUgqOe6Cl5O6lsCDA3AEEDInkHIs4cTWL0yTM9kZR8ib16BBop2v/CtrJ0BeMRnM48uF4PIdjH04mczjJ5R3+J6dRYkYJZslJHlsNsKA49RW12gKt+RS0dg2VhAOsQRnKpPAEsfvLFgIc+AFiaMNlFoBknewaUez5KS6dJk32FiSe4nCk5mU59DMcH8/hcZoVfJFmP6jjJ7TDcUunOUiuIfFr4jpl4Bpuev/Z81f85Epn9C2HTj660uSa+cHdaJif2lXNDYiqPw7yxtluufqsvPvueWn/1WxUrKPH6AnaRlX0Au2jGjpCdUTQR/QZfUFfC58K3wo/Cj+vpKsrsz2P0NIq/PoDe/jdZg==</latexit>

24

Learning with Adaptive Regularization
Recall the gradient update formula for W :<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

WW<latexit sha1_base64="rEvTqU47rdSWclaA+M06aAAtk5A=">AAAEl3icfZPNbhMxEMfdNkAJXymcEBeLqKI9UCUFqVwqIkFRuLWhbSplQ+X1ziZWvPbWdppsVn4lnoYLB3gWvEmqJI5gpJVm5/f3jO3xhCln2tRqvzY2t0r37j/Yflh+9PjJ02eVneeXWg4VhQsquVRXIdHAmYALwwyHq1QBSUIO7XDwqeDtW1CaSXFushS6CekJFjNKjAtdV5ptHHCIDVFKjnAbv8VBjyQJwXtBn5g8sy6S7e8FCTH9MM6JDShTFN/99+2bfRyMx9/PryvV2kFtanjdqc+dKprb6fXO1lEQSTpMQBjKidadei013ZwowygHWw6GGlJCB6QHHecKkoDu5tMjW7zrIhGOpXKfMHgaXV6Rk0TrLAmdstiq9lkR/Bcz/cT6saKKtnhqK8yJjZRcr64o8q6HO0MTf+jmTKRDA4LOThEPOTYSF83BEVNADc+cQ6hi7iIw7RNFqHEtLO9iv0RKJtKWV0okZABKysTdFBEUuNuAgBGVrqciygMmIpvPuhfmX61HR8B6fXMniPORLxiPF3DswyxbwMyHk8kCTtbqjv5TU6diTinh+bd1bBTAkuLcVzSbS7TpU3AvXznBLVGQasal8ASRm8KlBJ/9BBHEcFMkoMVLdg9RHPslbpzG5sdLEk9xMk4XbTnxK7RaC9iy1o1a3R+sdefy8KD+7uDw7H210ZgP3TZ6hV6jPVRHR6iBmugUXSCKfqCf6Df6U3pZ+lj6UmrOpJsb8zUv0IqVzv4C2sGcZQ==</latexit>(ŷy)(a h0 )xT

Now consider the following multivariate normal distribution:
W ⇠ MN (0p⇥d , ⌃r , ⌃c )<latexit sha1_base64="2jPnLCaGp3V4EB9VFeAvBrFw9kA=">AAAFGHicfZRLbxMxEMfdNkAJrxaOXCyqSkVC1aZFgkulSlApHIDSt9SNIq93klj1q7bzXO3X4MBn4cAFEDfEjW+DNw1N4hVYWu1ofn/P7HjGm2jOrIui3wuLS5UbN28t367euXvv/oOV1YcnVnUNhWOquDJnCbHAmYRjxxyHM22AiITDaXLxquCnPTCWKXnkhhoagrQlazFKnHc1V6JTHFsmcCyI61DCs7fv8o2omenYMQEWp/kzHFPVM1cv+rS5shZtRuOFy0ZtYqyhydpvri79jFNFuwKko5xYe16LtGtkxDhGOeTVuGtBE3pB2nDuTUl83kY2Li3H696T4pYy/pEOj72zOzIirB2KxCuLCmzICue/mOuIPPQVWWyOx2uOebFTitv5HUXcsvu861ovGxmTuutA0qsqWl2OncJFE3DKDFDHh94g1DB/EJh2iCHU+VZV13GYQpORyqtzKQS5AKOU8CdFJAXuP0BCnyohiEyzmMk0z8ZNTZLsTR7QPrB2x/0VtLJ+KBgMpnAQwuFwCochHI2mcFTK2/9PTqvlhBZzeFjGzgDMKI5CRb0+Q+shBWP8QGVxjxjQlnElA0Hqb9tMgNdhgBRacFkEoMUk+0GUO2GKS6/Js50ZSaDYG+hpW/bCDAcHU3iQFw2fpcUt9PyQtQVpmrwE6TWkYU/8pPT8wI3rnxx/FCbXpqDvBbTL0bWh18wH9/+AWnjjy8bJ1mZte3Prw/O13d3J32AZPUZP0AaqoRdoF9XRPjpGFH1En9FX9K3yqfKl8r3y40q6uDDZ8wjNrcqvP2Pr06Y=</latexit>

p
d
⌃
2
S
,
⌃
2
S
r
c
++ are row and column covariance matrices++<latexit sha1_base64="sBjluQ2xat1TU7PVAXr2Z4CQNYM=">AAAFLXicfZTNbhMxEMfdNkAJXy0cuayIKlUqipKCBAcqVaKVwonSNm2lbIi83klixV+1nc/V8gi8Cheu8BYckBBXTrwD3jQ0iVdgaaXR/P6esT2zEylGja1Uvq+srhVu3Ly1frt45+69+w82Nh+eGdnXBOpEMqkvImyAUQF1Sy2DC6UB84jBedR7nfHzAWhDpTi1YwVNjjuCtinB1rlaG9shkQMdUhGeHB20kp2d9L16+iFzkiVn3NooVcqV6QryRnVmlNBsHbU2136HsSR9DsISho1pVCvKNhOsLSUM0mLYN6Aw6eEONJwpMAfTTKZXSoMt54mDttTuEzaYehd3JJgbM+aRU3Jsu8ZnmfNfzHZ56vuyLCYNpmuJObGVkpnlHVncvLvRt+2XzYQK1bcgyNUt2n0WWBlkjx/EVAOxbOwMTDR1DxGQLtaYWFei4lbgp1B4ItPiUgqOe6Cl5O6lsCDA3AEEDInkHIs4cTWL0yTM9kZR8ib16BBop2v/CtrJ0BeMRnM48uF4PIdjH04mczjJ5R3+J6dRYkYJZslJHlsNsKA49RW12gKt+RS0dg2VhAOsQRnKpPAEsfvLFgIc+AFiaMNlFoBknewaUez5KS6dJk32FiSe4nCk5mU59DMcH8/hcZoVfJFmP6jjJ7TDcUunOUiuIfFr4jpl4Bpuev/Z81f85Epn9C2HTj660uSa+cHdaJif2lXNDYiqPw7yxtluufqsvPvueWn/1WxUrKPH6AnaRlX0Au2jGjpCdUTQR/QZfUFfC58K3wo/Cj+vpKsrsz2P0NIq/PoDe/jdZg==</latexit>-

Equivalently, this means<latexit sha1_base64="Qqx+H+BEVfsza7hlzH/GIfxjOcM=">AAAFRnicfZRLbxMxEMfdNoUSXikcuVhUlVoJVZuCBAcqFdFK4QKhb6kbRV7vJLGyftR2nqv9PHwVLlxBfAi4Ia5409AkXoGllWbn9/eM7Rk7UgkzNgi+Ly2vlFZv3V67U7577/6Dh5X1R2dG9jSFUyoTqS8iYiBhAk4tswlcKA2ERwmcR923OT/vgzZMihM7UtDgpC1Yi1FinatZeRNaGNq0DzTbOt/GoWEch5zYDiVJ+j7bCpqpCi3jYHCcPcMhlX0aymtH/qO3m5WNYCeYDFw0qlNjA01Hvbm+8iOMJe1xEJYmxJjLaqBsIyXaMppAVg57BhShXdKGS2cK4pI10sleM7zpPDFuSe0+YfHEOz8jJdyYEY+cMt+G8Vnu/BezHZ75vjyLyfBkLDAntlImZnFGHrfovuzZ1qtGyoTqWRD0ehetXoKtxHlVcMw0UJuMnEGoZu4gMO0QTah1tStvYj+FImOZlRdScNIFLSV3J0UEhcQtQMCASs6JiNOQiThLJ5WNovRd5tEBsHbH/hW00oEvGA5ncOjD0WgGRz4cj2dwXMg7+E9Oo8SU5s14XMRWA8wpTnxFrTZHaz4FrV1DpWGfaFCGJVJ4gthdv7kAB36AGFpwlQegeSe7RhR7foorp8nSvTmJpzgcqllZDv0MR0czeJTlBZ+n+fVz/Ji1OWnqrADpDaR+TVynuCtvJ/ufHn/gJ1c6px84tIvRlaY3zA9+XD+YrdpVzT0QVf85KBpnuzvV5zu7H19s7L+ePhVr6Al6irZQFb1E+6iG6ugUUfQJfUFf0bfS59LP0q/S72vp8tJ0zmO0MFbRH1a85uQ=</latexit><latexit sha1_base64="qJXK3lT3fLsGs4lMEblYG5l3CnM=">AAAFzHicfZTNbhMxEMc3pYESCrQguHBZUVVKEG2TUAkOVKpEK5UDtLRpU6lOI8c7m1jd9bpe59OyxIl35AV4DuxNaBIHsLTa0fz+nrE9Y7d4RFNZLv/MLd1bzt9/sPKw8Gj18ZOna+vPLtKkKwickyRKxGULpxBRBueSygguuQActyKot24+WV7vgUhpwmpyyKER4zajISVYGldz7Tsv1lFMA0SSnnjr2x8p+Xs+CgUmCsGAowhCWdxCUhQz0bXaquh6Jhyb17XSThUJ2u7IklbFKuK0dK14sFPVKAA5nmU8cw5iJcahm2sb5e1yNvxFozIxNrzJOGmuL++iICHdGJgkEU7Tq0qZy4bCQlISgS6gbgockxvchitjMhxD2lDZUWl/03gCP0yE+Zj0M+/sDIXjNB3GLaOMseykLrPOfzHZibXrs1lS7WdjjhmxTJIonZ9h4y66r7oy/NBQlPGuBEbGuwi7kS8T3xbVD6gAIqOhMTAR1ByETzrYVFCa0hc2fTcFx6NEF+ZSxPgGRJLE5qQwIxCZBTDokySOMQsUoizQCtm5rZb6rB3aB1v9P4JQ9V3BYDCFAxcOh1M4dOFoNIWjhbz9/+RMOZtQgiN1toilAJhR1FzF0dEMPXIpCGEaSqEeFsBTGiXMEQTm9s4EOHADBBDCrQ1AbCebRmR7bopbo9Fqb0biKA4HfFqWQzfD6ekUnmpb8Flqr6XhZ7Qd46bQC5DcQeLWxHRKzzRctv/J8Zfd5FxYehxDezE6F+SOucHPTg6mq86qdgDmngv4YlzHHAQ2ed8ohIVZmmkr80dvrfVXoTLvllY1Yd+ZivuqLBoX1e3Ku+3qt92N/Y+TF2fFe+W99opexXvv7XtH3ol37hHvV2419yL3Mv81L/Mqr8fSpdxkznNvbuR//AY/yRQ8</latexit>

vec(W ) ⇠ N (0p⇥d , ⌃c ⌦ ⌃r )exp
Tr(⌃r 1 W ⌃c 1 W T )/2p(W | ⌃r , ⌃c ) =(2⇡)pd/2 det(⌃r )d/2 det(⌃c )p/2

25

Learning with Adaptive Regularization
Recall the gradient update formula for W :<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

W

W

(ŷ

<latexit sha1_base64="rEvTqU47rdSWclaA+M06aAAtk5A=">AAAEl3icfZPNbhMxEMfdNkAJXymcEBeLqKI9UCUFqVwqIkFRuLWhbSplQ+X1ziZWvPbWdppsVn4lnoYLB3gWvEmqJI5gpJVm5/f3jO3xhCln2tRqvzY2t0r37j/Yflh+9PjJ02eVneeXWg4VhQsquVRXIdHAmYALwwyHq1QBSUIO7XDwqeDtW1CaSXFushS6CekJFjNKjAtdV5ptHHCIDVFKjnAbv8VBjyQJwXtBn5g8sy6S7e8FCTH9MM6JDShTFN/99+2bfRyMx9/PryvV2kFtanjdqc+dKprb6fXO1lEQSTpMQBjKidadei013ZwowygHWw6GGlJCB6QHHecKkoDu5tMjW7zrIhGOpXKfMHgaXV6Rk0TrLAmdstiq9lkR/Bcz/cT6saKKtnhqK8yJjZRcr64o8q6HO0MTf+jmTKRDA4LOThEPOTYSF83BEVNADc+cQ6hi7iIw7RNFqHEtLO9iv0RKJtKWV0okZABKysTdFBEUuNuAgBGVrqciygMmIpvPuhfmX61HR8B6fXMniPORLxiPF3DswyxbwMyHk8kCTtbqjv5TU6diTinh+bd1bBTAkuLcVzSbS7TpU3AvXznBLVGQasal8ASRm8KlBJ/9BBHEcFMkoMVLdg9RHPslbpzG5sdLEk9xMk4XbTnxK7RaC9iy1o1a3R+sdefy8KD+7uDw7H210ZgP3TZ6hV6jPVRHR6iBmugUXSCKfqCf6Df6U3pZ+lj6UmrOpJsb8zUv0IqVzv4C2sGcZQ==</latexit>y)(a h0 )xT

Now consider the following multivariate normal distribution:
W ⇠ MN (0p⇥d , ⌃r , ⌃c )<latexit sha1_base64="2jPnLCaGp3V4EB9VFeAvBrFw9kA=">AAAFGHicfZRLbxMxEMfdNkAJrxaOXCyqSkVC1aZFgkulSlApHIDSt9SNIq93klj1q7bzXO3X4MBn4cAFEDfEjW+DNw1N4hVYWu1ofn/P7HjGm2jOrIui3wuLS5UbN28t367euXvv/oOV1YcnVnUNhWOquDJnCbHAmYRjxxyHM22AiITDaXLxquCnPTCWKXnkhhoagrQlazFKnHc1V6JTHFsmcCyI61DCs7fv8o2omenYMQEWp/kzHFPVM1cv+rS5shZtRuOFy0ZtYqyhydpvri79jFNFuwKko5xYe16LtGtkxDhGOeTVuGtBE3pB2nDuTUl83kY2Li3H696T4pYy/pEOj72zOzIirB2KxCuLCmzICue/mOuIPPQVWWyOx2uOebFTitv5HUXcsvu861ovGxmTuutA0qsqWl2OncJFE3DKDFDHh94g1DB/EJh2iCHU+VZV13GYQpORyqtzKQS5AKOU8CdFJAXuP0BCnyohiEyzmMk0z8ZNTZLsTR7QPrB2x/0VtLJ+KBgMpnAQwuFwCochHI2mcFTK2/9PTqvlhBZzeFjGzgDMKI5CRb0+Q+shBWP8QGVxjxjQlnElA0Hqb9tMgNdhgBRacFkEoMUk+0GUO2GKS6/Js50ZSaDYG+hpW/bCDAcHU3iQFw2fpcUt9PyQtQVpmrwE6TWkYU/8pPT8wI3rnxx/FCbXpqDvBbTL0bWh18wH9/+AWnjjy8bJ1mZte3Prw/O13d3J32AZPUZP0AaqoRdoF9XRPjpGFH1En9FX9K3yqfKl8r3y40q6uDDZ8wjNrcqvP2Pr06Y=</latexit>

p
d
⌃
2
S
,
⌃
2
S
r
c
++ are row and column covariance matrices++<latexit sha1_base64="sBjluQ2xat1TU7PVAXr2Z4CQNYM=">AAAFLXicfZTNbhMxEMfdNkAJXy0cuayIKlUqipKCBAcqVaKVwonSNm2lbIi83klixV+1nc/V8gi8Cheu8BYckBBXTrwD3jQ0iVdgaaXR/P6esT2zEylGja1Uvq+srhVu3Ly1frt45+69+w82Nh+eGdnXBOpEMqkvImyAUQF1Sy2DC6UB84jBedR7nfHzAWhDpTi1YwVNjjuCtinB1rlaG9shkQMdUhGeHB20kp2d9L16+iFzkiVn3NooVcqV6QryRnVmlNBsHbU2136HsSR9DsISho1pVCvKNhOsLSUM0mLYN6Aw6eEONJwpMAfTTKZXSoMt54mDttTuEzaYehd3JJgbM+aRU3Jsu8ZnmfNfzHZ56vuyLCYNpmuJObGVkpnlHVncvLvRt+2XzYQK1bcgyNUt2n0WWBlkjx/EVAOxbOwMTDR1DxGQLtaYWFei4lbgp1B4ItPiUgqOe6Cl5O6lsCDA3AEEDInkHIs4cTWL0yTM9kZR8ib16BBop2v/CtrJ0BeMRnM48uF4PIdjH04mczjJ5R3+J6dRYkYJZslJHlsNsKA49RW12gKt+RS0dg2VhAOsQRnKpPAEsfvLFgIc+AFiaMNlFoBknewaUez5KS6dJk32FiSe4nCk5mU59DMcH8/hcZoVfJFmP6jjJ7TDcUunOUiuIfFr4jpl4Bpuev/Z81f85Epn9C2HTj660uSa+cHdaJif2lXNDYiqPw7yxtluufqsvPvueWn/1WxUrKPH6AnaRlX0Au2jGjpCdUTQR/QZfUFfC58K3wo/Cj+vpKsrsz2P0NIq/PoDe/jdZg==</latexit>-

Equivalently, this means<latexit sha1_base64="Qqx+H+BEVfsza7hlzH/GIfxjOcM=">AAAFRnicfZRLbxMxEMfdNoUSXikcuVhUlVoJVZuCBAcqFdFK4QKhb6kbRV7vJLGyftR2nqv9PHwVLlxBfAi4Ia5409AkXoGllWbn9/eM7Rk7UgkzNgi+Ly2vlFZv3V67U7577/6Dh5X1R2dG9jSFUyoTqS8iYiBhAk4tswlcKA2ERwmcR923OT/vgzZMihM7UtDgpC1Yi1FinatZeRNaGNq0DzTbOt/GoWEch5zYDiVJ+j7bCpqpCi3jYHCcPcMhlX0aymtH/qO3m5WNYCeYDFw0qlNjA01Hvbm+8iOMJe1xEJYmxJjLaqBsIyXaMppAVg57BhShXdKGS2cK4pI10sleM7zpPDFuSe0+YfHEOz8jJdyYEY+cMt+G8Vnu/BezHZ75vjyLyfBkLDAntlImZnFGHrfovuzZ1qtGyoTqWRD0ehetXoKtxHlVcMw0UJuMnEGoZu4gMO0QTah1tStvYj+FImOZlRdScNIFLSV3J0UEhcQtQMCASs6JiNOQiThLJ5WNovRd5tEBsHbH/hW00oEvGA5ncOjD0WgGRz4cj2dwXMg7+E9Oo8SU5s14XMRWA8wpTnxFrTZHaz4FrV1DpWGfaFCGJVJ4gthdv7kAB36AGFpwlQegeSe7RhR7foorp8nSvTmJpzgcqllZDv0MR0czeJTlBZ+n+fVz/Ji1OWnqrADpDaR+TVynuCtvJ/ufHn/gJ1c6px84tIvRlaY3zA9+XD+YrdpVzT0QVf85KBpnuzvV5zu7H19s7L+ePhVr6Al6irZQFb1E+6iG6ugUUfQJfUFf0bfS59LP0q/S72vp8tJ0zmO0MFbRH1a85uQ=</latexit><latexit sha1_base64="qJXK3lT3fLsGs4lMEblYG5l3CnM=">AAAFzHicfZTNbhMxEMc3pYESCrQguHBZUVVKEG2TUAkOVKpEK5UDtLRpU6lOI8c7m1jd9bpe59OyxIl35AV4DuxNaBIHsLTa0fz+nrE9Y7d4RFNZLv/MLd1bzt9/sPKw8Gj18ZOna+vPLtKkKwickyRKxGULpxBRBueSygguuQActyKot24+WV7vgUhpwmpyyKER4zajISVYGldz7Tsv1lFMA0SSnnjr2x8p+Xs+CgUmCsGAowhCWdxCUhQz0bXaquh6Jhyb17XSThUJ2u7IklbFKuK0dK14sFPVKAA5nmU8cw5iJcahm2sb5e1yNvxFozIxNrzJOGmuL++iICHdGJgkEU7Tq0qZy4bCQlISgS6gbgockxvchitjMhxD2lDZUWl/03gCP0yE+Zj0M+/sDIXjNB3GLaOMseykLrPOfzHZibXrs1lS7WdjjhmxTJIonZ9h4y66r7oy/NBQlPGuBEbGuwi7kS8T3xbVD6gAIqOhMTAR1ByETzrYVFCa0hc2fTcFx6NEF+ZSxPgGRJLE5qQwIxCZBTDokySOMQsUoizQCtm5rZb6rB3aB1v9P4JQ9V3BYDCFAxcOh1M4dOFoNIWjhbz9/+RMOZtQgiN1toilAJhR1FzF0dEMPXIpCGEaSqEeFsBTGiXMEQTm9s4EOHADBBDCrQ1AbCebRmR7bopbo9Fqb0biKA4HfFqWQzfD6ekUnmpb8Flqr6XhZ7Qd46bQC5DcQeLWxHRKzzRctv/J8Zfd5FxYehxDezE6F+SOucHPTg6mq86qdgDmngv4YlzHHAQ2ed8ohIVZmmkr80dvrfVXoTLvllY1Yd+ZivuqLBoX1e3Ku+3qt92N/Y+TF2fFe+W99opexXvv7XtH3ol37hHvV2419yL3Mv81L/Mqr8fSpdxkznNvbuR//AY/yRQ8</latexit>-

Or,

vec(W ) ⇠ N (0p⇥d , ⌃c ⌦ ⌃r )exp
Tr(⌃r 1 W ⌃c 1 W T )/2p(W | ⌃r , ⌃c ) =(2⇡)pd/2 det(⌃r )d/2 det(⌃c )p/2

Wi: ⇠ N (0d , [⌃r ]ii · ⌃c ),<latexit sha1_base64="//yzx4s2pucSx95ecYFZVCxaNAA=">AAAFhHicfZRbT9swFMddWBkr24BdnvYSDSGBhFALmzahMSENpO5lY9ylpqocx20NvmE7vUXWPuc+wB73HeaUjrSOhqVIzvn9/T/JObYjSYk21eqv0tz8o/LC48UnlaWnz54vr6y+uNAiUQifI0GFuoqgxpRwfG6IofhKKgxZRPFldPMl45c9rDQR/MwMJW4y2OGkTRA0LtRa+XnZSsmeDTVhIYOmiyBNv9mNcISV6GFkhGrFW0EjRKKnmk5KbIhiYbJ3tLkV3iYwDpzF3vWDFnJigZzF9XVuoTZbK2vV7ep4BMVJbTJZA5Nx3Fqd/xPGAiUMc4Mo1LpRq0rTTKEyBFFsK2GisYToBnZww005ZFg303GlbLDuInHQFso93ATj6PSKFDKthyxyyuxXtM+y4P+Y6TLrx7Is2gbjMcOc2AhB9eyKzLcYbiSm/bGZEi4Tgzm6+4t2QgMjgqynQUyUqzMduglEirhCBKgLFUTGdb6yHvgpJBwJW5lJweCNa5dgrlKQI0zdB3DcR4IxyOM0JDy26bi7UZR+tR7tY9Lpmn+Cdtr3BYNBDgc+HA5zOPThaJTDUSFv/4GcWvIJzTbkaREbhfGU4sxX1OtTtO5TrJTbUGnYgwpLTajgniB2h3fK4NA3iHEb32YGKNvJbiPyfT/FrdPYdH9K4imOBjJvy5Gf4eQkhyc2a/g0zY6f46ekw2BL2QJE9xD5Pbk/2Hn5q35yqTL6neFO0V0qdM9889Pjw/yrXdfcBVHzr4Pi5GJnu7a7vfPj3drBp8lVsQjegLdgA9TAB3AA6uAYnAMEfpeWSq9Kr8sL5a3ybvn9nXSuNFnzEsyM8ue/WAT7ow==</latexit>

W:j ⇠ N (0p , [⌃c ]jj · ⌃r )26

Learning with Adaptive RegularizationOK, but how to determine the parameters in the prior?

27

Learning with Adaptive RegularizationOK, but how to determine the parameters in the prior?
The empirical Bayes method: set the parameters of the priorfrom data by maximizing the marginal log-likelihood function(Efron and Morris, 1973; Efron and Hastie, 2016)
cr , ⌃
cc = arg max p(D | ⌃r , ⌃c ) = arg max⌃<latexit sha1_base64="+LQ49OOhlwkISZmauxdJVN+Z0Fc=">AAAF1XicfZTNahsxEMfXbdymbtMmLfTSi2gIJCUEOym0hxgCdcA9lKT5ciBrjKwd2yIraSPJn4tyKr32MfpMfZtqN9t4rSUVLAzz+2tGqxlNNwqp0tXqn9Kjx0vlJ0+Xn1Wev1h5+Wp17fWFEkNJ4JyIUMjLLlYQUg7nmuoQLiMJmHVDaHWvvyS8NQKpqOBnehpBm+E+pz1KsLauzupvf0wDGGAd+0SMpNlGCw5iUB35WPYZnnTuJFaRgtto02dYDwgO44bxGQ3yeOvBfT7lGhX2otaWTwKRkFYh1m3Q6qyuV3eq6UJFo5YZ6162jjtrS2/9QJAhA65JiJW6qlUj3Y6x1JSEYCr+UEGEyTXuw5U1OWag2nF6owZtWE+AekLazx439eZ3xJgpNWVdq0z+Q7kscT7E9IAZ15dkUQala4FZsRYiVIs7krhF99VQ9z63Y8qjoQZO7v6iNwyRFiipPQqoBKLDqTUwkdReBCIDLDHRtkMqG8hNEeGZMJWFFAxfgxSC2ZvCnEBoD8BhTARjmAexrW1g4rS03W781Th0DLQ/0P8EvXjsCiaTOZy4cDqdw6kLZ7M5nBXyjv+TU0U8o0k3nhaxlgA5xZmraDZztOlSkNI2VOyPsIRI0VBwRxDYR54L0HADBNCDmyQASTrZNiKvuylurMbE9ZzEURxOonlZDt0MJydzeGKSgudpOhZi/5Tat9yRpgDJPSRuTWynjGzDpf+fXX/VTR7JhB4x6BejR5LcMzf46XFjfuq0ag2w71zCN+s6ikBim/dDnM0gY999399OLDtJau7cKBoXuzu1vZ3d7x/XD/azmbLsvfPee5tezfvkHXhN79g790hppbRX2i/Vy62yKf8o/7yTPiple954C6v86y+3WBj0</latexit>⌃r ,⌃c⌃r ,⌃cZp(D | W ) · p(W | ⌃r , ⌃c ) dW

28

Learning with Adaptive RegularizationOK, but how to determine the parameters in the prior?
The empirical Bayes method: set the parameters of the priorfrom data by maximizing the marginal log-likelihood function(Efron and Morris, 1973; Efron and Hastie, 2016)
cr , ⌃
cc = arg max p(D | ⌃r , ⌃c ) = arg max⌃<latexit sha1_base64="+LQ49OOhlwkISZmauxdJVN+Z0Fc=">AAAF1XicfZTNahsxEMfXbdymbtMmLfTSi2gIJCUEOym0hxgCdcA9lKT5ciBrjKwd2yIraSPJn4tyKr32MfpMfZtqN9t4rSUVLAzz+2tGqxlNNwqp0tXqn9Kjx0vlJ0+Xn1Wev1h5+Wp17fWFEkNJ4JyIUMjLLlYQUg7nmuoQLiMJmHVDaHWvvyS8NQKpqOBnehpBm+E+pz1KsLauzupvf0wDGGAd+0SMpNlGCw5iUB35WPYZnnTuJFaRgtto02dYDwgO44bxGQ3yeOvBfT7lGhX2otaWTwKRkFYh1m3Q6qyuV3eq6UJFo5YZ6162jjtrS2/9QJAhA65JiJW6qlUj3Y6x1JSEYCr+UEGEyTXuw5U1OWag2nF6owZtWE+AekLazx439eZ3xJgpNWVdq0z+Q7kscT7E9IAZ15dkUQala4FZsRYiVIs7krhF99VQ9z63Y8qjoQZO7v6iNwyRFiipPQqoBKLDqTUwkdReBCIDLDHRtkMqG8hNEeGZMJWFFAxfgxSC2ZvCnEBoD8BhTARjmAexrW1g4rS03W781Th0DLQ/0P8EvXjsCiaTOZy4cDqdw6kLZ7M5nBXyjv+TU0U8o0k3nhaxlgA5xZmraDZztOlSkNI2VOyPsIRI0VBwRxDYR54L0HADBNCDmyQASTrZNiKvuylurMbE9ZzEURxOonlZDt0MJydzeGKSgudpOhZi/5Tat9yRpgDJPSRuTWynjGzDpf+fXX/VTR7JhB4x6BejR5LcMzf46XFjfuq0ag2w71zCN+s6ikBim/dDnM0gY999399OLDtJau7cKBoXuzu1vZ3d7x/XD/azmbLsvfPee5tezfvkHXhN79g790hppbRX2i/Vy62yKf8o/7yTPiple954C6v86y+3WBj0</latexit>-⌃r ,⌃c⌃r ,⌃cZp(D | W ) · p(W | ⌃r , ⌃c ) dW

High dimensional integration, intractable to compute, no closed form
solution in general28

Learning with Adaptive RegularizationOK, but how to determine the parameters in the prior?
The empirical Bayes method: set the parameters of the priorfrom data by maximizing the marginal log-likelihood function(Efron and Morris, 1973; Efron and Hastie, 2016)
cr , ⌃
cc = arg max p(D | ⌃r , ⌃c ) = arg max⌃<latexit sha1_base64="+LQ49OOhlwkISZmauxdJVN+Z0Fc=">AAAF1XicfZTNahsxEMfXbdymbtMmLfTSi2gIJCUEOym0hxgCdcA9lKT5ciBrjKwd2yIraSPJn4tyKr32MfpMfZtqN9t4rSUVLAzz+2tGqxlNNwqp0tXqn9Kjx0vlJ0+Xn1Wev1h5+Wp17fWFEkNJ4JyIUMjLLlYQUg7nmuoQLiMJmHVDaHWvvyS8NQKpqOBnehpBm+E+pz1KsLauzupvf0wDGGAd+0SMpNlGCw5iUB35WPYZnnTuJFaRgtto02dYDwgO44bxGQ3yeOvBfT7lGhX2otaWTwKRkFYh1m3Q6qyuV3eq6UJFo5YZ6162jjtrS2/9QJAhA65JiJW6qlUj3Y6x1JSEYCr+UEGEyTXuw5U1OWag2nF6owZtWE+AekLazx439eZ3xJgpNWVdq0z+Q7kscT7E9IAZ15dkUQala4FZsRYiVIs7krhF99VQ9z63Y8qjoQZO7v6iNwyRFiipPQqoBKLDqTUwkdReBCIDLDHRtkMqG8hNEeGZMJWFFAxfgxSC2ZvCnEBoD8BhTARjmAexrW1g4rS03W781Th0DLQ/0P8EvXjsCiaTOZy4cDqdw6kLZ7M5nBXyjv+TU0U8o0k3nhaxlgA5xZmraDZztOlSkNI2VOyPsIRI0VBwRxDYR54L0HADBNCDmyQASTrZNiKvuylurMbE9ZzEURxOonlZDt0MJydzeGKSgudpOhZi/5Tat9yRpgDJPSRuTWynjGzDpf+fXX/VTR7JhB4x6BejR5LcMzf46XFjfuq0ag2w71zCN+s6ikBim/dDnM0gY999399OLDtJau7cKBoXuzu1vZ3d7x/XD/azmbLsvfPee5tezfvkHXhN79g790hppbRX2i/Vy62yKf8o/7yTPiple954C6v86y+3WBj0</latexit>--⌃r ,⌃c⌃r ,⌃cZp(D | W ) · p(W | ⌃r , ⌃c ) dW

High dimensional integration, intractable to compute, no closed form
solution in general
Even with closed form solution, maximization is not necessarily
tractable

28

Learning with Adaptive RegularizationOK, but how to determine the parameters in the prior?
The empirical Bayes method: set the parameters of the priorfrom data by maximizing the marginal log-likelihood function(Efron and Morris, 1973; Efron and Hastie, 2016)
cr , ⌃
cc = arg max p(D | ⌃r , ⌃c ) = arg max⌃<latexit sha1_base64="+LQ49OOhlwkISZmauxdJVN+Z0Fc=">AAAF1XicfZTNahsxEMfXbdymbtMmLfTSi2gIJCUEOym0hxgCdcA9lKT5ciBrjKwd2yIraSPJn4tyKr32MfpMfZtqN9t4rSUVLAzz+2tGqxlNNwqp0tXqn9Kjx0vlJ0+Xn1Wev1h5+Wp17fWFEkNJ4JyIUMjLLlYQUg7nmuoQLiMJmHVDaHWvvyS8NQKpqOBnehpBm+E+pz1KsLauzupvf0wDGGAd+0SMpNlGCw5iUB35WPYZnnTuJFaRgtto02dYDwgO44bxGQ3yeOvBfT7lGhX2otaWTwKRkFYh1m3Q6qyuV3eq6UJFo5YZ6162jjtrS2/9QJAhA65JiJW6qlUj3Y6x1JSEYCr+UEGEyTXuw5U1OWag2nF6owZtWE+AekLazx439eZ3xJgpNWVdq0z+Q7kscT7E9IAZ15dkUQala4FZsRYiVIs7krhF99VQ9z63Y8qjoQZO7v6iNwyRFiipPQqoBKLDqTUwkdReBCIDLDHRtkMqG8hNEeGZMJWFFAxfgxSC2ZvCnEBoD8BhTARjmAexrW1g4rS03W781Th0DLQ/0P8EvXjsCiaTOZy4cDqdw6kLZ7M5nBXyjv+TU0U8o0k3nhaxlgA5xZmraDZztOlSkNI2VOyPsIRI0VBwRxDYR54L0HADBNCDmyQASTrZNiKvuylurMbE9ZzEURxOonlZDt0MJydzeGKSgudpOhZi/5Tat9yRpgDJPSRuTWynjGzDpf+fXX/VTR7JhB4x6BejR5LcMzf46XFjfuq0ag2w71zCN+s6ikBim/dDnM0gY999399OLDtJau7cKBoXuzu1vZ3d7x/XD/azmbLsvfPee5tezfvkHXhN79g790hppbRX2i/Vy62yKf8o/7yTPiple954C6v86y+3WBj0</latexit>-⌃r ,⌃c⌃r ,⌃cZp(D | W ) · p(W | ⌃r , ⌃c ) dW

High dimensional integration, intractable to compute, no closed form
solution in general-

Even with closed form solution, maximization is not necessarily
tractable-

Example with closed-form solution: the James-Stein estimator28

Learning with Adaptive Regularization
Iterative maximization of the joint distribution:-

Given weight matrix W , compute the most likely parameters ⌃r , ⌃cGiven parameters ⌃r , ⌃c, plug in and update W<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit><latexit sha1_base64="UFtjtWYwTLjkMXHSxSVwLO2SSNY=">AAAFOHicfZRbaxQxFMfTOmpdL20VfPFlsBREStmtgj5YKNjC+iCtvUNnKZnM2d2wuTXJXof5Gr7q5/Cb+Oab+OonMNmu3d0MGhjmcH7/nJPknCRVjBpbrX5fWLwV3b5zd+le5f6Dh4+WV1YfnxrZ1QROiGRSn6fYAKMCTiy1DM6VBsxTBmdp573nZz3QhkpxbIcKGhy3BG1Sgq1zJQmRPb0R+x+5XFmrblbHIy4btYmxhibj4HI1eppkknQ5CEsYNuaiVlW2kWNtKWFQVJKuAYVJB7fgwpkCczCNfLzoIl53nixuSu0+YeOxd3ZGjrkxQ546Jce2bULmnf9its2L0OezmCIejznmxFZKZuZn+Lhl90XXNt82cipU14Ig17todllsZeyPN86oBmLZ0BmYaOoOIiZtrDGxrgiV9ThMofBIFpW5FBx3QEvJ3UlhQYC5BQjoE8k5FlmeUJEVeeLnpmn+oQhoH2irbf8Kmnk/FAwGUzgI4XA4hcMQjkZTOCrl7f8np1FiQglm+VEZWw0wozgOFfX6DK2HFLR2DZUnPaxBGcqkCASZu0czAXbDABk04coHIL6TXSOK7TDFldMU+faMJFDsDdS0LHthhsPDKTwsfMFnqb+Ejh/RFseXuihBcgNJWBPXKT3XcOP9T46/GiZX2tN9Dq1ydKXJDQuDHx3sTlc9rtouuHuu4aNz7SvQ2OV9mSdYu6W5tnL/ZMNb7iWphe9G2Tjd2qy92tz69Hpt593kTVlCz9Bz9ALV0Bu0g+roAJ0gghT6jL6gr9G36Ef0M/p1LV1cmMx5guZG9PsPF/vhyg==</latexit><latexit sha1_base64="UFtjtWYwTLjkMXHSxSVwLO2SSNY=">AAAFOHicfZRbaxQxFMfTOmpdL20VfPFlsBREStmtgj5YKNjC+iCtvUNnKZnM2d2wuTXJXof5Gr7q5/Cb+Oab+OonMNmu3d0MGhjmcH7/nJPknCRVjBpbrX5fWLwV3b5zd+le5f6Dh4+WV1YfnxrZ1QROiGRSn6fYAKMCTiy1DM6VBsxTBmdp573nZz3QhkpxbIcKGhy3BG1Sgq1zJQmRPb0R+x+5XFmrblbHIy4btYmxhibj4HI1eppkknQ5CEsYNuaiVlW2kWNtKWFQVJKuAYVJB7fgwpkCczCNfLzoIl53nixuSu0+YeOxd3ZGjrkxQ546Jce2bULmnf9its2L0OezmCIejznmxFZKZuZn+Lhl90XXNt82cipU14Ig17todllsZeyPN86oBmLZ0BmYaOoOIiZtrDGxrgiV9ThMofBIFpW5FBx3QEvJ3UlhQYC5BQjoE8k5FlmeUJEVeeLnpmn+oQhoH2irbf8Kmnk/FAwGUzgI4XA4hcMQjkZTOCrl7f8np1FiQglm+VEZWw0wozgOFfX6DK2HFLR2DZUnPaxBGcqkCASZu0czAXbDABk04coHIL6TXSOK7TDFldMU+faMJFDsDdS0LHthhsPDKTwsfMFnqb+Ejh/RFseXuihBcgNJWBPXKT3XcOP9T46/GiZX2tN9Dq1ydKXJDQuDHx3sTlc9rtouuHuu4aNz7SvQ2OV9mSdYu6W5tnL/ZMNb7iWphe9G2Tjd2qy92tz69Hpt593kTVlCz9Bz9ALV0Bu0g+roAJ0gghT6jL6gr9G36Ef0M/p1LV1cmMx5guZG9PsPF/vhyg==</latexit><latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

max max log p(D | W ) + log p(W | ⌃c , ⌃r )<latexit sha1_base64="bPoqFZWsxyun4M9dx6mC59s0+zM=">AAAFf3icfZTbbhMxEIbd0kAJh7YgccONRVXRQhVtChIgUakSqRQuUM9NpW4Ueb2TxOquvbWd42p5LZ6FC27hNbC3aZN4BZaSHc33e8b2jB0kEVPa834uLN5bKt1/sPyw/Ojxk6crq2vPzpXoSQpnVERCXgREQcQ4nGmmI7hIJJA4iKARXH2xvNEHqZjgp3qUQDMmHc7ajBJtXK3VIz8mw1bayG6+PhV9uW3/afYd+5Ho4GTTIN2lJEprRsVC3NjCb29Zw3py/TbOJ2+1Vte9ipcPXDSqE2MdTcZha23phR8K2ouBaxoRpS6rXqKbKZGa0Qiyst9TkBB6RTpwaUxOYlDNNN97hjeMJ8RtIc2Pa5x7Z2ekJFZqFAdGabehXGad/2K6G2euz2ZRGc7HHDNiLUSk5mfYuEX3ZU+3PzZTxpOeBk5vdtHuRVgLbKuEQyaB6mhkDEIlMweBaZdIQrWpZXkDuykSMhZZeS5FTK5AChGbkyKcQmQWwGFARRwTHqY+42GW5pUNgvRr5tABsE5X3wra6cAVDIdTOHThaDSFIxeOx1M4LuQd/CenSviE2mY8KWItAWYUp66iXp+hdZeClKahUr9PJCSKRYI7gtBcx5kANTdACG24tgGo7WTTiHzXTXFtNFm6OyNxFPvDZFqWfTfD8fEUHme24LPUXj/DT1gnJi2ZFSC9g9StiemUvmm4fP+T4/fc5Im09CCGTjF6Iukdc4OfHNamq86rVgNzzyV8M66DBCQxed+kPpFmaaatzNfftpZ5Saruu1E0zncq1XeVnaP363ufJ2/KMnqJXqFNVEUf0B6qo0N0hij6gX6h3+hPaaH0ulQpeTfSxYXJnOdobpQ+/QXLkPi3</latexit>W⌃r ,⌃c29

Learning with Adaptive Regularization
Iterative maximization of the joint distribution:
Given weight matrix W , compute the most likely parameters ⌃r , ⌃cGiven parameters ⌃r , ⌃c, plug in and update W-<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>-<latexit sha1_base64="UFtjtWYwTLjkMXHSxSVwLO2SSNY=">AAAFOHicfZRbaxQxFMfTOmpdL20VfPFlsBREStmtgj5YKNjC+iCtvUNnKZnM2d2wuTXJXof5Gr7q5/Cb+Oab+OonMNmu3d0MGhjmcH7/nJPknCRVjBpbrX5fWLwV3b5zd+le5f6Dh4+WV1YfnxrZ1QROiGRSn6fYAKMCTiy1DM6VBsxTBmdp573nZz3QhkpxbIcKGhy3BG1Sgq1zJQmRPb0R+x+5XFmrblbHIy4btYmxhibj4HI1eppkknQ5CEsYNuaiVlW2kWNtKWFQVJKuAYVJB7fgwpkCczCNfLzoIl53nixuSu0+YeOxd3ZGjrkxQ546Jce2bULmnf9its2L0OezmCIejznmxFZKZuZn+Lhl90XXNt82cipU14Ig17todllsZeyPN86oBmLZ0BmYaOoOIiZtrDGxrgiV9ThMofBIFpW5FBx3QEvJ3UlhQYC5BQjoE8k5FlmeUJEVeeLnpmn+oQhoH2irbf8Kmnk/FAwGUzgI4XA4hcMQjkZTOCrl7f8np1FiQglm+VEZWw0wozgOFfX6DK2HFLR2DZUnPaxBGcqkCASZu0czAXbDABk04coHIL6TXSOK7TDFldMU+faMJFDsDdS0LHthhsPDKTwsfMFnqb+Ejh/RFseXuihBcgNJWBPXKT3XcOP9T46/GiZX2tN9Dq1ydKXJDQuDHx3sTlc9rtouuHuu4aNz7SvQ2OV9mSdYu6W5tnL/ZMNb7iWphe9G2Tjd2qy92tz69Hpt593kTVlCz9Bz9ALV0Bu0g+roAJ0gghT6jL6gr9G36Ef0M/p1LV1cmMx5guZG9PsPF/vhyg==</latexit><latexit sha1_base64="UFtjtWYwTLjkMXHSxSVwLO2SSNY=">AAAFOHicfZRbaxQxFMfTOmpdL20VfPFlsBREStmtgj5YKNjC+iCtvUNnKZnM2d2wuTXJXof5Gr7q5/Cb+Oab+OonMNmu3d0MGhjmcH7/nJPknCRVjBpbrX5fWLwV3b5zd+le5f6Dh4+WV1YfnxrZ1QROiGRSn6fYAKMCTiy1DM6VBsxTBmdp573nZz3QhkpxbIcKGhy3BG1Sgq1zJQmRPb0R+x+5XFmrblbHIy4btYmxhibj4HI1eppkknQ5CEsYNuaiVlW2kWNtKWFQVJKuAYVJB7fgwpkCczCNfLzoIl53nixuSu0+YeOxd3ZGjrkxQ546Jce2bULmnf9its2L0OezmCIejznmxFZKZuZn+Lhl90XXNt82cipU14Ig17todllsZeyPN86oBmLZ0BmYaOoOIiZtrDGxrgiV9ThMofBIFpW5FBx3QEvJ3UlhQYC5BQjoE8k5FlmeUJEVeeLnpmn+oQhoH2irbf8Kmnk/FAwGUzgI4XA4hcMQjkZTOCrl7f8np1FiQglm+VEZWw0wozgOFfX6DK2HFLR2DZUnPaxBGcqkCASZu0czAXbDABk04coHIL6TXSOK7TDFldMU+faMJFDsDdS0LHthhsPDKTwsfMFnqb+Ejh/RFseXuihBcgNJWBPXKT3XcOP9T46/GiZX2tN9Dq1ydKXJDQuDHx3sTlc9rtouuHuu4aNz7SvQ2OV9mSdYu6W5tnL/ZMNb7iWphe9G2Tjd2qy92tz69Hpt593kTVlCz9Bz9ALV0Bu0g+roAJ0gghT6jL6gr9G36Ef0M/p1LV1cmMx5guZG9PsPF/vhyg==</latexit><latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

max max log p(D | W ) + log p(W | ⌃c , ⌃r )<latexit sha1_base64="bPoqFZWsxyun4M9dx6mC59s0+zM=">AAAFf3icfZTbbhMxEIbd0kAJh7YgccONRVXRQhVtChIgUakSqRQuUM9NpW4Ueb2TxOquvbWd42p5LZ6FC27hNbC3aZN4BZaSHc33e8b2jB0kEVPa834uLN5bKt1/sPyw/Ojxk6crq2vPzpXoSQpnVERCXgREQcQ4nGmmI7hIJJA4iKARXH2xvNEHqZjgp3qUQDMmHc7ajBJtXK3VIz8mw1bayG6+PhV9uW3/afYd+5Ho4GTTIN2lJEprRsVC3NjCb29Zw3py/TbOJ2+1Vte9ipcPXDSqE2MdTcZha23phR8K2ouBaxoRpS6rXqKbKZGa0Qiyst9TkBB6RTpwaUxOYlDNNN97hjeMJ8RtIc2Pa5x7Z2ekJFZqFAdGabehXGad/2K6G2euz2ZRGc7HHDNiLUSk5mfYuEX3ZU+3PzZTxpOeBk5vdtHuRVgLbKuEQyaB6mhkDEIlMweBaZdIQrWpZXkDuykSMhZZeS5FTK5AChGbkyKcQmQWwGFARRwTHqY+42GW5pUNgvRr5tABsE5X3wra6cAVDIdTOHThaDSFIxeOx1M4LuQd/CenSviE2mY8KWItAWYUp66iXp+hdZeClKahUr9PJCSKRYI7gtBcx5kANTdACG24tgGo7WTTiHzXTXFtNFm6OyNxFPvDZFqWfTfD8fEUHme24LPUXj/DT1gnJi2ZFSC9g9StiemUvmm4fP+T4/fc5Im09CCGTjF6Iukdc4OfHNamq86rVgNzzyV8M66DBCQxed+kPpFmaaatzNfftpZ5Saruu1E0zncq1XeVnaP363ufJ2/KMnqJXqFNVEUf0B6qo0N0hij6gX6h3+hPaaH0ulQpeTfSxYXJnOdobpQ+/QXLkPi3</latexit>W⌃r ,⌃cInstantiate the above principle using the matrix-variate normal
distribution, we arrive at the following optimization problem:
min minW,a ⌦r ,⌦c

1 X(ŷ(xi ; W, a)2nsubject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>

i2[n]

⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )

vId

30

Learning with Adaptive Regularization
Instantiate the above principle using the matrix-variate normal
distribution, we arrive at the following optimization problem:
min minW,a ⌦r ,⌦c

1 X(ŷ(xi ; W, a)2nsubject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>

i2[n]

⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )

vId

1
1
:
:
⌦
=
⌃
,
⌦
=
⌃
r
c
r
c are the precision matrices<latexit sha1_base64="SrDXgMngN/LB2bDkpBAVkT6mTJI=">AAAF0HicfZRLbxMxEMddaKCEVwtHLiuqSgiVKilIcKBSJVopHFBL+pS6IXK8k2TVtb21nedqQVwRJz4NV/gWfBvG29DdeAFLUUbz+3tmvDN2J45CbWq1XwvXri9WbtxculW9fefuvfvLKw+OtRwoBkdMRlKddqiGKBRwZEITwWmsgPJOBCed8zeWnwxB6VCKQzOJocVpT4TdkFGDrvbyuh8r5QfQhQvPZ3KoPiTP6un6R3Szgptl7vbyam2jli2vbNRnxiqZrf32yuJXP5BswEEYFlGtz+q12LQSqkzIIkir/kBDTNk57cEZmoJy0K0kO1bqraEn8LpS4U8YL/MWdySUaz3hHVRyavraZdb5L2b6PHV9NotOvWzNMRQbKSM9v8PGLbvPBqb7qpWEIh4YEOzyFN1B5Bnp2QZ4QaiAmWiCBmUqxA/hsT5VlBlsU3XNc1PEdCrT6lwKTs9BScnxS1HBIMICBIyY5JyKIPFDEaSJb/d2Osnb1KEjCHt980fQTUauYDzO4diFk0kOJy6cTnM4LeUd/SenjsWMMholB2VsFEBBcegqGo0CbbgUlMKBSvwhVRDrMJLCEQR40woBdtwA2U2wAZidZBxEseWmuEBNmmwVJI5idxznbdl1MzSbOWymtuFFam8m8oOwx2lbpSXIriBze4KTMsSBy84/+/w1Nzm+AUj3OPTK0fEhuGJu8IP9nbzqrGs7gPdcwTt07cWgKOZ9mvhUYWk4Vvjvr1vrr8LEN1jGoVsAVk/z2qlbeyR7AdjmoeFbyynfDmyfGjut+H7V3deqbBxvbtSfb2y+f7G6/Xr2ki2RR+QxeULq5CXZJg2yT44II9/Id/KD/Kw0K+PKp8rnS+m1hdmeh2RuVb78BuM1HRA=</latexit>

31

Learning with Adaptive Regularization
Instantiate the above principle using the matrix-variate normal
distribution, we arrive at the following optimization problem:
min minW,a ⌦r ,⌦c

1 X(ŷ(xi ; W, a)2nsubject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>

i2[n]

⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )

vId

1
1
:
:
⌦
=
⌃
,
⌦
=
⌃
r
c
r
c are the precision matrices<latexit sha1_base64="SrDXgMngN/LB2bDkpBAVkT6mTJI=">AAAF0HicfZRLbxMxEMddaKCEVwtHLiuqSgiVKilIcKBSJVopHFBL+pS6IXK8k2TVtb21nedqQVwRJz4NV/gWfBvG29DdeAFLUUbz+3tmvDN2J45CbWq1XwvXri9WbtxculW9fefuvfvLKw+OtRwoBkdMRlKddqiGKBRwZEITwWmsgPJOBCed8zeWnwxB6VCKQzOJocVpT4TdkFGDrvbyuh8r5QfQhQvPZ3KoPiTP6un6R3Szgptl7vbyam2jli2vbNRnxiqZrf32yuJXP5BswEEYFlGtz+q12LQSqkzIIkir/kBDTNk57cEZmoJy0K0kO1bqraEn8LpS4U8YL/MWdySUaz3hHVRyavraZdb5L2b6PHV9NotOvWzNMRQbKSM9v8PGLbvPBqb7qpWEIh4YEOzyFN1B5Bnp2QZ4QaiAmWiCBmUqxA/hsT5VlBlsU3XNc1PEdCrT6lwKTs9BScnxS1HBIMICBIyY5JyKIPFDEaSJb/d2Osnb1KEjCHt980fQTUauYDzO4diFk0kOJy6cTnM4LeUd/SenjsWMMholB2VsFEBBcegqGo0CbbgUlMKBSvwhVRDrMJLCEQR40woBdtwA2U2wAZidZBxEseWmuEBNmmwVJI5idxznbdl1MzSbOWymtuFFam8m8oOwx2lbpSXIriBze4KTMsSBy84/+/w1Nzm+AUj3OPTK0fEhuGJu8IP9nbzqrGs7gPdcwTt07cWgKOZ9mvhUYWk4Vvjvr1vrr8LEN1jGoVsAVk/z2qlbeyR7AdjmoeFbyynfDmyfGjut+H7V3deqbBxvbtSfb2y+f7G6/Xr2ki2RR+QxeULq5CXZJg2yT44II9/Id/KD/Kw0K+PKp8rnS+m1hdmeh2RuVb78BuM1HRA=</latexit>-

The first term corresponds to data fitting term

32

Learning with Adaptive Regularization
Instantiate the above principle using the matrix-variate normal
distribution, we arrive at the following optimization problem:
min minW,a ⌦r ,⌦c

1 X(ŷ(xi ; W, a)2nsubject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>

i2[n]

⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )

vId

1
1
:
:
⌦
=
⌃
,
⌦
=
⌃
r
c
r
c are the precision matrices<latexit sha1_base64="SrDXgMngN/LB2bDkpBAVkT6mTJI=">AAAF0HicfZRLbxMxEMddaKCEVwtHLiuqSgiVKilIcKBSJVopHFBL+pS6IXK8k2TVtb21nedqQVwRJz4NV/gWfBvG29DdeAFLUUbz+3tmvDN2J45CbWq1XwvXri9WbtxculW9fefuvfvLKw+OtRwoBkdMRlKddqiGKBRwZEITwWmsgPJOBCed8zeWnwxB6VCKQzOJocVpT4TdkFGDrvbyuh8r5QfQhQvPZ3KoPiTP6un6R3Szgptl7vbyam2jli2vbNRnxiqZrf32yuJXP5BswEEYFlGtz+q12LQSqkzIIkir/kBDTNk57cEZmoJy0K0kO1bqraEn8LpS4U8YL/MWdySUaz3hHVRyavraZdb5L2b6PHV9NotOvWzNMRQbKSM9v8PGLbvPBqb7qpWEIh4YEOzyFN1B5Bnp2QZ4QaiAmWiCBmUqxA/hsT5VlBlsU3XNc1PEdCrT6lwKTs9BScnxS1HBIMICBIyY5JyKIPFDEaSJb/d2Osnb1KEjCHt980fQTUauYDzO4diFk0kOJy6cTnM4LeUd/SenjsWMMholB2VsFEBBcegqGo0CbbgUlMKBSvwhVRDrMJLCEQR40woBdtwA2U2wAZidZBxEseWmuEBNmmwVJI5idxznbdl1MzSbOWymtuFFam8m8oOwx2lbpSXIriBze4KTMsSBy84/+/w1Nzm+AUj3OPTK0fEhuGJu8IP9nbzqrGs7gPdcwTt07cWgKOZ9mvhUYWk4Vvjvr1vrr8LEN1jGoVsAVk/z2qlbeyR7AdjmoeFbyynfDmyfGjut+H7V3deqbBxvbtSfb2y+f7G6/Xr2ki2RR+QxeULq5CXZJg2yT44II9/Id/KD/Kw0K+PKp8rnS+m1hdmeh2RuVb78BuM1HRA=</latexit>-

The first term corresponds to data fitting term
The second term corresponds to regularization term

33

Learning with Adaptive Regularization
Instantiate the above principle using the matrix-variate normal
distribution, we arrive at the following optimization problem:
min minW,a ⌦r ,⌦c

1 X(ŷ(xi ; W, a)2nsubject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>

i2[n]

⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )

vId

1
1
:
:
⌦
=
⌃
,
⌦
=
⌃
r
c
r
c are the precision matrices<latexit sha1_base64="SrDXgMngN/LB2bDkpBAVkT6mTJI=">AAAF0HicfZRLbxMxEMddaKCEVwtHLiuqSgiVKilIcKBSJVopHFBL+pS6IXK8k2TVtb21nedqQVwRJz4NV/gWfBvG29DdeAFLUUbz+3tmvDN2J45CbWq1XwvXri9WbtxculW9fefuvfvLKw+OtRwoBkdMRlKddqiGKBRwZEITwWmsgPJOBCed8zeWnwxB6VCKQzOJocVpT4TdkFGDrvbyuh8r5QfQhQvPZ3KoPiTP6un6R3Szgptl7vbyam2jli2vbNRnxiqZrf32yuJXP5BswEEYFlGtz+q12LQSqkzIIkir/kBDTNk57cEZmoJy0K0kO1bqraEn8LpS4U8YL/MWdySUaz3hHVRyavraZdb5L2b6PHV9NotOvWzNMRQbKSM9v8PGLbvPBqb7qpWEIh4YEOzyFN1B5Bnp2QZ4QaiAmWiCBmUqxA/hsT5VlBlsU3XNc1PEdCrT6lwKTs9BScnxS1HBIMICBIyY5JyKIPFDEaSJb/d2Osnb1KEjCHt980fQTUauYDzO4diFk0kOJy6cTnM4LeUd/SenjsWMMholB2VsFEBBcegqGo0CbbgUlMKBSvwhVRDrMJLCEQR40woBdtwA2U2wAZidZBxEseWmuEBNmmwVJI5idxznbdl1MzSbOWymtuFFam8m8oOwx2lbpSXIriBze4KTMsSBy84/+/w1Nzm+AUj3OPTK0fEhuGJu8IP9nbzqrGs7gPdcwTt07cWgKOZ9mvhUYWk4Vvjvr1vrr8LEN1jGoVsAVk/z2qlbeyR7AdjmoeFbyynfDmyfGjut+H7V3deqbBxvbtSfb2y+f7G6/Xr2ki2RR+QxeULq5CXZJg2yT44II9/Id/KD/Kw0K+PKp8rnS+m1hdmeh2RuVb78BuM1HRA=</latexit>-

The first term corresponds to data fitting term
The second term corresponds to regularization term
is a trade-oﬀ hyper-parameter that only depends on p, d<latexit sha1_base64="UUM5FwLzjzLnPKjPBfIIQkCSvtI=">AAAFqXicfZTbbhMxEIbd0kAJpxYuuVkRVUKAqqQgwQWVKtFK4QL1fBDdqPJ6J8kqPmxt57jaN+CGW3gy3oZxGrqJF7AUZTTf75l/7dmNUp4YW6//Wlq+s1K5e2/1fvXBw0ePn6ytPz0zqq8ZnDLFlb6IqAGeSDi1ieVwkWqgIuJwHvU+OX4+AG0SJU/sOIWWoB2ZtBNGLabOQ47SmF6t1eqb9ekKykFjFtTIbB1cra98C2PF+gKkZZwac9mop7aVUW0TxiGvhn0DKWU92oFLDCUVYFrZ1G8ebGAmDtpK40/aYJqd35FRYcxYRKgU1HaNz1zyX8x2Re7nXBeTB9O1wFBsleJmcYerW05f9m37QytLZNq3INnNU7T7PLAqcCcbxIkGZvkYA8p0ggcRsC7VlFk8/+pG4LdI6UTl1YUWgvZAKyXwpKhkwNGAhCFTQlAZZ2Ei4zwL3d4oyj7nHh1C0unaP4J2NvQFo1EBRz4cjws49uFkUsBJqe/wPz1NKmeUUZ4dl7HVAHOKE1/RbM7Rpk9BaxyoLBxQDalJuJKeIMZXaK7Arl8ghjZcuwLMTTIOotz2W1yjJs+25ySeYm+UFtey53c4OirgUe4ufJ4yNXD+j5OOoFc6L0F2C5l/JzgpAxy46fPPjr/uN0+1o/sCOuXqqWa3zC9+fLBbuJ7e2i7ge67hC6b2U9AU+77KQqrRGo4V/odvXPRXYRZatHHiG0D3tPBOfe9cdWJwl4dB6CLPvhvYLrVuWvH71fC/VuXgbGuz8XZz6/Bdbefj7Eu2Sp6TF+QlaZD3ZIc0yQE5JYz0yHfyg/ysvK4cVi4qX2+ky0uzPc/Iwqqw320ADag=</latexit><latexit sha1_base64="C1cxZ8VIC35Zf3o0j2AdFC++ZfY=">AAAFpnicfZRdTxNBFIYHpIr1C/TSm42EhBhCWjDRC0lIhKReKAgUSNgGZ2dP2wnzxcz0c7P/wHirf81/40ypbDurTtL05DzvnPPuzNlNFKPG1mq/FhbvLVXuP1h+WH30+MnTZyurz8+M7GkCTSKZ1BcJNsCogKallsGF0oB5wuA8uf7g+XkftKFSnNqRghbHHUHblGDrU2ozSq9W1mpbtcmKykF9Gqyh6Tq6Wl36FqeS9DgISxg25rJeU7aVYW0pYZBX454Bhck17sClCwXmYFrZxGwerbtMGrWldj9ho0l2dkeGuTEjnjglx7ZrQuaT/2K2y/Mw57uYPJqsOebEVkpm5nf4uuX0Zc+237UyKlTPgiC3T9HuscjKyB9rlFINxLKRCzDR1B1ERLpYY2Ld4VfXo7CFwmOZV+dacHwNWkruTgoLAswZEDAgknMs0iymIs2z2O9NkuxjHtAB0E7X/hG0s0EoGA4LOAzhaFTAUQjH4wKOS30H/+lplJhSgll2UsZWA8woTkNFozFDGyEFrd1AZXEfa1CGMikCQeren5kC+2GBFNpw4wsQP8luEMVu2OLGafJsd0YSKA6GqriWg7DD8XEBj3N/4bOUyL73f0I7HF/pvATJHSThnbhJ6buBmzz/9PhrYXOlPT3k0ClXV5rcsbD4ydF+4Xpya/vg3nMNn1zqUIHGru/rLMbaWXNj5f7jTR/9VZjF1tk4DQ0497jwjkPvTHZS8JfngthHgX0/sF1s/bS671c9/FqVg7PtrfrO1vaXN2t776dfsmX0Er1CG6iO3qI91EBHqIkI6qLv6Af6WdmofK40K+e30sWF6Z4XaG5Vvv4GdXIL7Q==</latexit>

34

Learning with Adaptive Regularization
Maximum-A-Posteriori estimation
min minW,a ⌦r ,⌦c

1 X(ŷ(xi ; W, a)2nsubject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>

i2[n]

⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )vId

The optimization formulation defines a sequence of MAP problems:

35

Learning with Adaptive Regularization
Maximum-A-Posteriori estimation
min minW,a ⌦r ,⌦c

1 X(ŷ(xi ; W, a)
2n
i2[n]

subject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )vId

The optimization formulation defines a sequence of MAP problems:
-

(t)
(t)
(⌦
,
⌦
c ), the second term becomes
At iteration t, with r<latexit sha1_base64="uP6tlPRTnfYKOCCX/PtmEfTov7o=">AAAFo3icfZRbaxNBFMentdEab60++rJYCiJSklbQBwsFW4iItE3TC3RDmZ09SYbOZTszuQ77CfRRP5zfxpk0dpNZdSDkcH7/Oee/M2c3yRjVplb7tbR8b6Vy/8Hqw+qjx0+ePltbf36mZV8ROCWSSXWRYA2MCjg11DC4yBRgnjA4T64/eX4+AKWpFC0zzqDNcVfQDiXYuNSxuVrbqG3VpisqB/VZsIFm6+hqfeV7nErS5yAMYVjry3otM22LlaGEQV6N+xoyTK5xFy5dKDAH3bZTp3m06TJp1JHK/YSJptn5HRZzrcc8cUqOTU+HzCf/xUyP52HOd9F5NF0LzImNlEwv7vB1y+nLvul8aFsqsr4BQW6fotNnkZGRP9MopQqIYWMXYKKoO4iI9LDCxLiTr25GYYsMT2ReXWjB8TUoKbk7KSwIMGdAwJBIzrFIbUxFmtvY700S+zkP6BBot2f+CDp2GApGowKOQjgeF3AcwsmkgJNS3+F/eupMzCjBzJ6UsVEAc4pWqGg05mgjpKCUGygbD7CCTFMmRSBI3cszV2A/LJBCB258AeIn2Q2i2A1b3DhNbnfnJIHiYJQV13IQdmg2C9jM/YXPUyIH3v8J7XJ8pfISJHeQhHfiJmXgBm76/LPjr4XNM+XpIYduuXqmyB0Li58c7Reup7e2D+49V/DVpQ4zUNj1fWNjrJw1N1buP37ro78KbWycjVZowLnHhXccemeym4K/PBfEPgrs+4HtYeOn1X2/6uHXqhycbW/Vd7a2j99t7H2cfclW0Uv0Cr1GdfQe7aEGOkKniCBA39AP9LOyWflSaVZat9LlpdmeF2hhVdq/AXM3CyM=</latexit><latexit sha1_base64="8RdSHMjImX5+hWSUeYXBcBefoR8=">AAAFvHicfVTbbhMxEHWhgRJuLbwg8bKiqtSiqkoKEjxQVIlWCg+opXepGyKvd5Kssra3tnO1li/gM3iF/+FvGKehSbyApShHc45nznrGjrI00aZS+bVw6/Zi6c7dpXvl+w8ePnq8vPLkTMuuYnDKZCrVRUQ1pImAU5OYFC4yBZRHKZxHnQ+OP++B0okUJ2aYQZ3TlkiaCaMGQ43lZ+thptQXu2428s0AMbvGG43l1cpWZbyCIqhOwCqZrMPGyuK3MJasy0EYllKtL6uVzNQtVSZhKeTlsKsho6xDW3CJUFAOum7Hn5AHaxiJg6ZU+BMmGEdnd1jKtR7yCJWcmrb2ORf8F2faPPdjrorOg/Ga41BspEz1/A6Xtxi+7Jrm27pNRNY1INj1VzS7aWBk4A47iBMFzKRDBJSpBA8iYG2qKDPYkvJa4JfI6Ejm5bkSnHZAScnxpKhgkKIBAX0mOacitmEi4tyGbm8U2Y+5x/YhabXNH0HT9n3BYDAlBz45HE7JoU+ORlNyVKjb/09NnYkJy2hqj4u0UQAzihNfUavNsDWfBaVwoGzYowoynaRSeIIYb9VMgj0/QQxNuHIJmJtkHESx45e4Qk1ud2YknmJ/kE3bsu9XODqakke5a/gsy2TP+T9OWpw2VF4g2Q3J/J7gpPRw4MbfPzn+il8crzuyBxxaxex4/W84P/nx4d7U9bhre4D3XMEnDB1koCjWfWlDqtAajhX+h5sO/VVoQ4M2TnwD6J5OvVPfeypbMbjmIQgd8uy7gW1T46YV36+q/1oVwdn2VvXV1vbn16u77yYv2RJ5Tl6QdVIlb8guqZFDckoY+Uq+kx/kZ+l9KS51SvxaemthsucpmVul3m++ORPq</latexit>

1/2 2
1/2
1/2 2
1/2
1/2
2
||⌦1/2W⌦||=||vec(⌦
W
⌦
)||
=
||(⌦
⌦
⌦
)
vec(W
)||
r
c
F
r
c
2
c
r
2
<latexit sha1_base64="2iEghX5/pBsnm9qO0+LfK2G4TY4=">AAAGQHicfVTNbtNAEHZKAyUt0MKRi0VUqUWoxAEJDlSqRIvCAbX0L5XiEK3Xk8Sq7XXXm9+NeSgegbdA4gG4Ia6cmE1MnawLK0WZzPfNzJeZ2XUi34tFpfKtsHRruXj7zsrd0uravfsP1jcensesxymcUeYzfuGQGHwvhDPhCR8uIg4kcHyoO5dvFV7vA489Fp6KUQTNgHRCr+1RItDVWu9PJnbE+SdpPa8mdTTpzJxMWu8+Vc1dE/E+UCFvZClaNaVtZW6bCS+A2Mxitj/PstTTiNZ6ubJTmR4zb1ipUTbSc9TaWP5iu4z2AggF9UkcN6xKJJqScOFRH5KS3YshIvSSdKCBZkhQQFNOG5SYm+hxzTbj+AmFOfXOR0gSxPEocJAZENGNdUw5/4WJbpDoPlUlTszpWcCQLBjz48UIlTfvbvRE+3VTemHUExDS2b9o93xTMFON0nQ9jj31R2gQyj1shEm7hBMqcOClTVMvEZExS0oLJQJyCZyxADtFQgo+CghhQFkQkNCVthe6ibRVrOPI94mGDsDrdMVfQlsOdMJwmIFDHRyNMnCkg+NxBo5zdQf/qRlHYYpS4suTPCw4wBzjVGfUanNoTUeBc1woafcJhyj2fBZqBBfv7FyCfT2BC224Ugmo2mRcxHBXL3GFnETuzlE0xsEwysZyoFc4Ps7A40QNfB6lrK/0n3idgLR4kgPpNUj1meCmqEs8/f9p+yt6cbzxiB4G0Mlnx/fhGtOTnxztZ6qnU9sHvOccPqDrMAJOsO5TaROO0nCt8Nt+pqwbidIWKONUF4DqSaad6Np91nFBDQ8NW1mafLWwXSLy26q6kjSsJpaFoZD4M9kqW9sJvnKW/qbljfPqjvVip/rxZXnvTfrerRiPjSfGlmEZr4w9o2YcGWcGNb4XlgqrhbXi1+KP4s/irxl1qZDGPDIWTvH3HyxVRlw=</latexit>

36

Learning with Adaptive Regularization
Maximum-A-Posteriori estimation
min minW,a ⌦r ,⌦c

1 X(ŷ(xi ; W, a)
2n
i2[n]

subject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )vId

The optimization formulation defines a sequence of MAP problems:
-

(t)
(t)
(⌦
,
⌦
c ), the second term becomes
At iteration t, with r<latexit sha1_base64="uP6tlPRTnfYKOCCX/PtmEfTov7o=">AAAFo3icfZRbaxNBFMentdEab60++rJYCiJSklbQBwsFW4iItE3TC3RDmZ09SYbOZTszuQ77CfRRP5zfxpk0dpNZdSDkcH7/Oee/M2c3yRjVplb7tbR8b6Vy/8Hqw+qjx0+ePltbf36mZV8ROCWSSXWRYA2MCjg11DC4yBRgnjA4T64/eX4+AKWpFC0zzqDNcVfQDiXYuNSxuVrbqG3VpisqB/VZsIFm6+hqfeV7nErS5yAMYVjry3otM22LlaGEQV6N+xoyTK5xFy5dKDAH3bZTp3m06TJp1JHK/YSJptn5HRZzrcc8cUqOTU+HzCf/xUyP52HOd9F5NF0LzImNlEwv7vB1y+nLvul8aFsqsr4BQW6fotNnkZGRP9MopQqIYWMXYKKoO4iI9LDCxLiTr25GYYsMT2ReXWjB8TUoKbk7KSwIMGdAwJBIzrFIbUxFmtvY700S+zkP6BBot2f+CDp2GApGowKOQjgeF3AcwsmkgJNS3+F/eupMzCjBzJ6UsVEAc4pWqGg05mgjpKCUGygbD7CCTFMmRSBI3cszV2A/LJBCB258AeIn2Q2i2A1b3DhNbnfnJIHiYJQV13IQdmg2C9jM/YXPUyIH3v8J7XJ8pfISJHeQhHfiJmXgBm76/LPjr4XNM+XpIYduuXqmyB0Li58c7Reup7e2D+49V/DVpQ4zUNj1fWNjrJw1N1buP37ro78KbWycjVZowLnHhXccemeym4K/PBfEPgrs+4HtYeOn1X2/6uHXqhycbW/Vd7a2j99t7H2cfclW0Uv0Cr1GdfQe7aEGOkKniCBA39AP9LOyWflSaVZat9LlpdmeF2hhVdq/AXM3CyM=</latexit><latexit sha1_base64="8RdSHMjImX5+hWSUeYXBcBefoR8=">AAAFvHicfVTbbhMxEHWhgRJuLbwg8bKiqtSiqkoKEjxQVIlWCg+opXepGyKvd5Kssra3tnO1li/gM3iF/+FvGKehSbyApShHc45nznrGjrI00aZS+bVw6/Zi6c7dpXvl+w8ePnq8vPLkTMuuYnDKZCrVRUQ1pImAU5OYFC4yBZRHKZxHnQ+OP++B0okUJ2aYQZ3TlkiaCaMGQ43lZ+thptQXu2428s0AMbvGG43l1cpWZbyCIqhOwCqZrMPGyuK3MJasy0EYllKtL6uVzNQtVSZhKeTlsKsho6xDW3CJUFAOum7Hn5AHaxiJg6ZU+BMmGEdnd1jKtR7yCJWcmrb2ORf8F2faPPdjrorOg/Ga41BspEz1/A6Xtxi+7Jrm27pNRNY1INj1VzS7aWBk4A47iBMFzKRDBJSpBA8iYG2qKDPYkvJa4JfI6Ejm5bkSnHZAScnxpKhgkKIBAX0mOacitmEi4tyGbm8U2Y+5x/YhabXNH0HT9n3BYDAlBz45HE7JoU+ORlNyVKjb/09NnYkJy2hqj4u0UQAzihNfUavNsDWfBaVwoGzYowoynaRSeIIYb9VMgj0/QQxNuHIJmJtkHESx45e4Qk1ud2YknmJ/kE3bsu9XODqakke5a/gsy2TP+T9OWpw2VF4g2Q3J/J7gpPRw4MbfPzn+il8crzuyBxxaxex4/W84P/nx4d7U9bhre4D3XMEnDB1koCjWfWlDqtAajhX+h5sO/VVoQ4M2TnwD6J5OvVPfeypbMbjmIQgd8uy7gW1T46YV36+q/1oVwdn2VvXV1vbn16u77yYv2RJ5Tl6QdVIlb8guqZFDckoY+Uq+kx/kZ+l9KS51SvxaemthsucpmVul3m++ORPq</latexit>

1/2 2
1/2
1/2 2
1/2
1/2
2
||⌦1/2W⌦||=||vec(⌦
W
⌦
)||
=
||(⌦
⌦
⌦
)
vec(W
)||
r
c
F
r
c
2
c
r
2
<latexit sha1_base64="2iEghX5/pBsnm9qO0+LfK2G4TY4=">AAAGQHicfVTNbtNAEHZKAyUt0MKRi0VUqUWoxAEJDlSqRIvCAbX0L5XiEK3Xk8Sq7XXXm9+NeSgegbdA4gG4Ia6cmE1MnawLK0WZzPfNzJeZ2XUi34tFpfKtsHRruXj7zsrd0uravfsP1jcensesxymcUeYzfuGQGHwvhDPhCR8uIg4kcHyoO5dvFV7vA489Fp6KUQTNgHRCr+1RItDVWu9PJnbE+SdpPa8mdTTpzJxMWu8+Vc1dE/E+UCFvZClaNaVtZW6bCS+A2Mxitj/PstTTiNZ6ubJTmR4zb1ipUTbSc9TaWP5iu4z2AggF9UkcN6xKJJqScOFRH5KS3YshIvSSdKCBZkhQQFNOG5SYm+hxzTbj+AmFOfXOR0gSxPEocJAZENGNdUw5/4WJbpDoPlUlTszpWcCQLBjz48UIlTfvbvRE+3VTemHUExDS2b9o93xTMFON0nQ9jj31R2gQyj1shEm7hBMqcOClTVMvEZExS0oLJQJyCZyxADtFQgo+CghhQFkQkNCVthe6ibRVrOPI94mGDsDrdMVfQlsOdMJwmIFDHRyNMnCkg+NxBo5zdQf/qRlHYYpS4suTPCw4wBzjVGfUanNoTUeBc1woafcJhyj2fBZqBBfv7FyCfT2BC224Ugmo2mRcxHBXL3GFnETuzlE0xsEwysZyoFc4Ps7A40QNfB6lrK/0n3idgLR4kgPpNUj1meCmqEs8/f9p+yt6cbzxiB4G0Mlnx/fhGtOTnxztZ6qnU9sHvOccPqDrMAJOsO5TaROO0nCt8Nt+pqwbidIWKONUF4DqSaad6Np91nFBDQ8NW1mafLWwXSLy26q6kjSsJpaFoZD4M9kqW9sJvnKW/qbljfPqjvVip/rxZXnvTfrerRiPjSfGlmEZr4w9o2YcGWcGNb4XlgqrhbXi1+KP4s/irxl1qZDGPDIWTvH3HyxVRlw=</latexit>-

Tikhonov regularization imposed on W with the Tikhonov matrix as
1/2:= ⌦1/2⌦⌦cr<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit><latexit sha1_base64="E26863x7NYoorNZXdh7YyhuXtXo=">AAAF9HicfVTNbhMxEHZLAyX8tXDksiKqVCpUsgEJDlSqRCvCAbX0X8qGyPFOEqu79tbr/Fr7DjwAN8SVAweu8Bq8DeM0NIkXsBRldr7P841nxm4mEU91ufxrYfHaUuH6jeWbxVu379y9t7J6/ySVXcXgmMlIqrMmTSHiAo411xGcJQpo3IzgtHn+2uKnPVApl+JIDxOox7QteIszqtHVWNkI3tA4pl4QQgsuvCBR7IPxn1ayQGoeQ2o96tLTWCmVN8vj5eUNf2KUyGTtN1aXvgWhZN0YhGYRTdOaX0503VClOYsgKwbdFBLKzmkbamgKioJ1Mz5U5q2hJ/RaUuFPaG/snd1haJymw7iJzJjqTupi1vkvTHfizPVZlTTzxmsOQ7KWMkrnd9i4eXetq1sv64aLpKtBsMtTtLqRp6Vny++FXAHT0RANyhTHQnisQxVlGptUXPNciYSOZFack4jpOSgpY6wUFQwiTEBAn0lsowhNwEWYmcDubTbN28xB+8DbHf2H0DJ9lzAYTMGBCw6HU3DogqPRFBzldPv/0UwTMUEZjcxhHtYKYIZx5DKq1Rm06qKgFA6UCXpUQZLySAqHEOI9mwmw4wYY3w0bgNlJxkEUW67EBXIyszVDcRi7g2Tall1X4eBgCh5ktuGzKJM9m/8hb8e0obIcyK5A5vYEJ6WHAzc+/6T8ZVccbziiezG089HxPbjC3OCH+zvTrMdd2wG85wreoWsvAUVRd8MEVGFqOFb4Hzyx1l+JJtCYxpGbAGZPp7lTN/dItkOwzUMjsJaTvh3YDtX5abVVyWp+HWVhoA1+Zusl/7F95Xz3TcsbJ5VN/9lm5f3z0varyXu3TB6SR2Sd+OQF2SZVsk+OCSMfyXfyg/ws9AqfCp8LXy6piwuTPQ/I3Cp8/Q1duSre</latexit>

37

Learning with Adaptive Regularization
Maximum-A-Posteriori estimation
min minW,a ⌦r ,⌦c

1 X(ŷ(xi ; W, a)
2n
i2[n]

subject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )vId

The optimization formulation defines a sequence of MAP problems:
-

(t)
(t)
(⌦
,
⌦
c ), the second term becomes
At iteration t, with r<latexit sha1_base64="uP6tlPRTnfYKOCCX/PtmEfTov7o=">AAAFo3icfZRbaxNBFMentdEab60++rJYCiJSklbQBwsFW4iItE3TC3RDmZ09SYbOZTszuQ77CfRRP5zfxpk0dpNZdSDkcH7/Oee/M2c3yRjVplb7tbR8b6Vy/8Hqw+qjx0+ePltbf36mZV8ROCWSSXWRYA2MCjg11DC4yBRgnjA4T64/eX4+AKWpFC0zzqDNcVfQDiXYuNSxuVrbqG3VpisqB/VZsIFm6+hqfeV7nErS5yAMYVjry3otM22LlaGEQV6N+xoyTK5xFy5dKDAH3bZTp3m06TJp1JHK/YSJptn5HRZzrcc8cUqOTU+HzCf/xUyP52HOd9F5NF0LzImNlEwv7vB1y+nLvul8aFsqsr4BQW6fotNnkZGRP9MopQqIYWMXYKKoO4iI9LDCxLiTr25GYYsMT2ReXWjB8TUoKbk7KSwIMGdAwJBIzrFIbUxFmtvY700S+zkP6BBot2f+CDp2GApGowKOQjgeF3AcwsmkgJNS3+F/eupMzCjBzJ6UsVEAc4pWqGg05mgjpKCUGygbD7CCTFMmRSBI3cszV2A/LJBCB258AeIn2Q2i2A1b3DhNbnfnJIHiYJQV13IQdmg2C9jM/YXPUyIH3v8J7XJ8pfISJHeQhHfiJmXgBm76/LPjr4XNM+XpIYduuXqmyB0Li58c7Reup7e2D+49V/DVpQ4zUNj1fWNjrJw1N1buP37ro78KbWycjVZowLnHhXccemeym4K/PBfEPgrs+4HtYeOn1X2/6uHXqhycbW/Vd7a2j99t7H2cfclW0Uv0Cr1GdfQe7aEGOkKniCBA39AP9LOyWflSaVZat9LlpdmeF2hhVdq/AXM3CyM=</latexit><latexit sha1_base64="8RdSHMjImX5+hWSUeYXBcBefoR8=">AAAFvHicfVTbbhMxEHWhgRJuLbwg8bKiqtSiqkoKEjxQVIlWCg+opXepGyKvd5Kssra3tnO1li/gM3iF/+FvGKehSbyApShHc45nznrGjrI00aZS+bVw6/Zi6c7dpXvl+w8ePnq8vPLkTMuuYnDKZCrVRUQ1pImAU5OYFC4yBZRHKZxHnQ+OP++B0okUJ2aYQZ3TlkiaCaMGQ43lZ+thptQXu2428s0AMbvGG43l1cpWZbyCIqhOwCqZrMPGyuK3MJasy0EYllKtL6uVzNQtVSZhKeTlsKsho6xDW3CJUFAOum7Hn5AHaxiJg6ZU+BMmGEdnd1jKtR7yCJWcmrb2ORf8F2faPPdjrorOg/Ga41BspEz1/A6Xtxi+7Jrm27pNRNY1INj1VzS7aWBk4A47iBMFzKRDBJSpBA8iYG2qKDPYkvJa4JfI6Ejm5bkSnHZAScnxpKhgkKIBAX0mOacitmEi4tyGbm8U2Y+5x/YhabXNH0HT9n3BYDAlBz45HE7JoU+ORlNyVKjb/09NnYkJy2hqj4u0UQAzihNfUavNsDWfBaVwoGzYowoynaRSeIIYb9VMgj0/QQxNuHIJmJtkHESx45e4Qk1ud2YknmJ/kE3bsu9XODqakke5a/gsy2TP+T9OWpw2VF4g2Q3J/J7gpPRw4MbfPzn+il8crzuyBxxaxex4/W84P/nx4d7U9bhre4D3XMEnDB1koCjWfWlDqtAajhX+h5sO/VVoQ4M2TnwD6J5OvVPfeypbMbjmIQgd8uy7gW1T46YV36+q/1oVwdn2VvXV1vbn16u77yYv2RJ5Tl6QdVIlb8guqZFDckoY+Uq+kx/kZ+l9KS51SvxaemthsucpmVul3m++ORPq</latexit>

1/2 2
1/2
1/2 2
1/2
1/2
2
||⌦1/2W⌦||=||vec(⌦
W
⌦
)||
=
||(⌦
⌦
⌦
)
vec(W
)||
r
c
F
r
c
2
c
r
2
<latexit sha1_base64="2iEghX5/pBsnm9qO0+LfK2G4TY4=">AAAGQHicfVTNbtNAEHZKAyUt0MKRi0VUqUWoxAEJDlSqRIvCAbX0L5XiEK3Xk8Sq7XXXm9+NeSgegbdA4gG4Ia6cmE1MnawLK0WZzPfNzJeZ2XUi34tFpfKtsHRruXj7zsrd0uravfsP1jcensesxymcUeYzfuGQGHwvhDPhCR8uIg4kcHyoO5dvFV7vA489Fp6KUQTNgHRCr+1RItDVWu9PJnbE+SdpPa8mdTTpzJxMWu8+Vc1dE/E+UCFvZClaNaVtZW6bCS+A2Mxitj/PstTTiNZ6ubJTmR4zb1ipUTbSc9TaWP5iu4z2AggF9UkcN6xKJJqScOFRH5KS3YshIvSSdKCBZkhQQFNOG5SYm+hxzTbj+AmFOfXOR0gSxPEocJAZENGNdUw5/4WJbpDoPlUlTszpWcCQLBjz48UIlTfvbvRE+3VTemHUExDS2b9o93xTMFON0nQ9jj31R2gQyj1shEm7hBMqcOClTVMvEZExS0oLJQJyCZyxADtFQgo+CghhQFkQkNCVthe6ibRVrOPI94mGDsDrdMVfQlsOdMJwmIFDHRyNMnCkg+NxBo5zdQf/qRlHYYpS4suTPCw4wBzjVGfUanNoTUeBc1woafcJhyj2fBZqBBfv7FyCfT2BC224Ugmo2mRcxHBXL3GFnETuzlE0xsEwysZyoFc4Ps7A40QNfB6lrK/0n3idgLR4kgPpNUj1meCmqEs8/f9p+yt6cbzxiB4G0Mlnx/fhGtOTnxztZ6qnU9sHvOccPqDrMAJOsO5TaROO0nCt8Nt+pqwbidIWKONUF4DqSaad6Np91nFBDQ8NW1mafLWwXSLy26q6kjSsJpaFoZD4M9kqW9sJvnKW/qbljfPqjvVip/rxZXnvTfrerRiPjSfGlmEZr4w9o2YcGWcGNb4XlgqrhbXi1+KP4s/irxl1qZDGPDIWTvH3HyxVRlw=</latexit>-

Tikhonov regularization imposed on W with the Tikhonov matrix as
1/2:= ⌦1/2⌦⌦cr<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit><latexit sha1_base64="E26863x7NYoorNZXdh7YyhuXtXo=">AAAF9HicfVTNbhMxEHZLAyX8tXDksiKqVCpUsgEJDlSqRCvCAbX0X8qGyPFOEqu79tbr/Fr7DjwAN8SVAweu8Bq8DeM0NIkXsBRldr7P841nxm4mEU91ufxrYfHaUuH6jeWbxVu379y9t7J6/ySVXcXgmMlIqrMmTSHiAo411xGcJQpo3IzgtHn+2uKnPVApl+JIDxOox7QteIszqtHVWNkI3tA4pl4QQgsuvCBR7IPxn1ayQGoeQ2o96tLTWCmVN8vj5eUNf2KUyGTtN1aXvgWhZN0YhGYRTdOaX0503VClOYsgKwbdFBLKzmkbamgKioJ1Mz5U5q2hJ/RaUuFPaG/snd1haJymw7iJzJjqTupi1vkvTHfizPVZlTTzxmsOQ7KWMkrnd9i4eXetq1sv64aLpKtBsMtTtLqRp6Vny++FXAHT0RANyhTHQnisQxVlGptUXPNciYSOZFack4jpOSgpY6wUFQwiTEBAn0lsowhNwEWYmcDubTbN28xB+8DbHf2H0DJ9lzAYTMGBCw6HU3DogqPRFBzldPv/0UwTMUEZjcxhHtYKYIZx5DKq1Rm06qKgFA6UCXpUQZLySAqHEOI9mwmw4wYY3w0bgNlJxkEUW67EBXIyszVDcRi7g2Tall1X4eBgCh5ktuGzKJM9m/8hb8e0obIcyK5A5vYEJ6WHAzc+/6T8ZVccbziiezG089HxPbjC3OCH+zvTrMdd2wG85wreoWsvAUVRd8MEVGFqOFb4Hzyx1l+JJtCYxpGbAGZPp7lTN/dItkOwzUMjsJaTvh3YDtX5abVVyWp+HWVhoA1+Zusl/7F95Xz3TcsbJ5VN/9lm5f3z0varyXu3TB6SR2Sd+OQF2SZVsk+OCSMfyXfyg/ws9AqfCp8LXy6piwuTPQ/I3Cp8/Q1duSre</latexit>-

The Tikhonov matrix is not predefined, but learned from data adaptively
38

Learning with Adaptive Regularization
Volume minimizationlog det(·) is a concave function over the positive-definite cone. For any
m
A
,
A
2
S
++, the following inequality holds:
pair of matrices 1 2<latexit sha1_base64="sL0mOAK0Nc+dhxgO9IzOmfi/tlI=">AAAF1nicfVTLbhMxFHWhgRJeLSzZjIgqtQhVmYIECypVopXCArX0LWVC5fHcJFbH9tR2niOzQ2xZsoVP4G/4G+w0dBIPYCnKnXuO7zlzfcdxllKl6/VfCzduLlZu3V66U7177/6Dh8srj06U6EkCx0SkQp7FWEFKORxrqlM4yyRgFqdwGl+8dfhpH6Sigh/pUQYthjuctinB2qY+RqnoJKDXIpIIvX6+XKtv1CcrKAfhNKih6do/X1n8GSWC9BhwTVKsVDOsZ7qVY6kpScFUo56CDJML3IGmDTlmoFr5xLYJVm0mCdpC2h/XwSQ7uyPHTKkRiy2TYd1VPuaS/8J0lxk/51SUCSZrDrNkLUSq5ne4uuV0s6fbr1s55VlPAydXb9HupYEWgWtwkFAJRKcjG2AiqW1EQLpYYqLtMVRXA18iw2NhqnMSDF+AFILZTmFOILUGOAyIYAzzJI8oT0weub1xnL8zHjoA2unqP4R2PvAJw2EBDn1wNCrAkQ+OxwU4LukO/qOpMj5FCU7zwzKsJcAM48hnNBozaMNHQUo7UHnUxxIyRVPBPUJiv6SZAjt+gQTacOkKEDfJdhD5li9xaTkm35qheIzdYVYcy66vcHBQgAfGHfgsSkTf+T+kHYbPpSmB5Bok/pnYSenbgZu8/7T9dV88kw7dY9ApV88kucb84of7O4XryantgP3OJby3qb0MJLa6z/IIS2vNjpX9j5676K/EPNLWxpFvwLrHhXfse7+6ocwkiFzk2XcD28W6PK2uK6YZtqwsDHVuH81aLVw39pYL/TutHJxsboQvNjY/vKxtv5ned0voCXqK1lCIXqFt1ED76BgRJNE39B39qJxVPlU+V75cUW8sTPc8RnOr8vU3qgwfuQ==</latexit><latexit sha1_base64="I5o3C+kVizOrVwLaHVTp+YmXgXw=">AAAF4HicfVRLb9NAEN5CAyW8WpC4cLGIKhVaVXFBggOVWtFK4YBa+pbiYK3Xk8Sq7XV3N8/Fd26IK0eucOLf8G+YTUOTrIGVLI/n+3a+2ZnxBlkcSVWt/pq7dn2+dOPmwq3y7Tt3791fXHpwInlHMDhmPObiLKAS4iiFYxWpGM4yATQJYjgNzt8Y/LQLQkY8PVKDDBoJbaVRM2JUoctffLTtu2vOtr/hRal3uL/j69XV/EPiL1aq69XRcoqGOzYqZLz2/aX5n17IWSeBVLGYSll3q5lqaCpUxGLIy15HQkbZOW1BHc2UJiAbenSA3FlGT+g0ucAnVc7IO71D00TKQRIgM6GqLW3MOP+FqXaS2z6jInNntGYwJCvOYzm7w8Qtuusd1XzV0FGadRSk7PIUzU7sKO6YUjthJICpeIAGZSLCQjisTQVlChtSXnZsiYwOeV6ekUjoOQjOE6wUTRnEmEAKPcaThKahxp6FufbM3iDQb3ML7UHUaqs/hKbu2YR+fwL2bXAwmIADGxwOJ+CwoNv7j6bM0jHKaKwPi7ASAFOMI5tRq02hNRsFIXCgtNelAjIZxTy1CCH+U1MBduwAITThwgRgZpJxENNNW+ICObnenKJYjN1+NmnLrq1wcDABD3LT8GmU8a7J/zBqJdQXeQFkVyCze4KT0sWBG51/XP6qLZ4Jg+4l0CpGzwS7wuzgeDVMsh51bQfwPxfwDl17GQiKus+0RwWmhmOFb2/NWH8lak9hGkd2Apg9neRO7dxj3grBNA8Nz1hW+mZg21QVp9VUJa+7DZSFvtL4ma9U3Kc53nKufacVjZONdff5+sb7F5Wt1+P7boE8Jk/ICnHJS7JFamSfHBNGPpJv5Dv5UQpKn0qfS18uqdfmxnsekplV+vobAV8h5Q==</latexit>log det(A1 )  log det(A2 ) + hr log det(A2 ), A1<latexit sha1_base64="lUU9ELiy6Tt7E7rMM2tC2y3bcuU=">AAAGSHicfVTNbtNAEHZKAyX8tZQblxVRqxbaKg5IcKBSEa0UDqil/1IcovVmklhdr9315ne1PBSPwRtw5wG4IW6M07RJ1sBKtsfzfTvfeGc8fsyDRJVK33Nzt+bzt+8s3C3cu//g4aPFpcenSdSRDE5YxCN57tMEeCDgRAWKw3ksgYY+hzP/4n2Kn3VBJkEkjtUghlpIWyJoBowqdNUXv3g8ankNUGvv6u46WSUeh0sy5Syvkxf4TkWLA/EE9TmdhTcI7iSbeC97ckzzCqvb2SBKpuZnvemakdgmCeuLxdJWabRI1nDHRtEZr4P60vxXrxGxTghCMU6TpOqWYlXTVKqAcTAFr5NATNkFbUEVTUFDSGp6dFCGrKCnQZqRxEsoMvJO79A0TJJB6CMzpKqd2Fjq/Bem2qGxfalKYshozWBIVlHEk9kdadysu9pRzTc1HYi4o0Cwq69odjhREUlLShqBBKb4AA3KZIAHQVibSsoUFr6wQmyJmA4jU5iRCOkFyCgK8aSoYMAxAQE9FoUhFQ3tBaJhtJfu9X39wVhoD4JWW10TmrpnE/r9Cdi3wcFgAg5scDicgMOMbu8/mkksxiijXB9lYSUBphjHNqNSmUIrNgpSYkNpr0slxEnAI2ERGvjvTgXYtQM0oAmXaQCWdjI2oti2JS6RY/T2FMVi7PXjSVn2bIXDwwl4aNKCT6Ms6qb5HwWtkNalyYDsBmR2TbBTuthwo+8fH3/JFo9liu6H0MpGjyW7wezgRwe7k6xHVdsF/M8lfETXfgySou5z7VGJqWFb4dPbSK2/EjXOHKOP7QQwezrJndq549zCsWX09QCz008btk1VtlvTUzFVt4ay0FcaX81a0V03OOVce6ZljdPylvtyq/zpVXHn7XjeLThPnWfOmuM6r50dp+IcOCcOc37kCrnl3JP8t/zP/K/87yvqXG68Z9mZWbfn/gB9jkFA</latexit>= log det(A2 ) + Tr(A2 1 A1 )

m

A2 i

39

Learning with Adaptive Regularization
Volume minimizationlog det(·) is a concave function over the positive-definite cone. For any
m
A
,
A
2
S
++, the following inequality holds:
pair of matrices 1 2<latexit sha1_base64="sL0mOAK0Nc+dhxgO9IzOmfi/tlI=">AAAF1nicfVTLbhMxFHWhgRJeLSzZjIgqtQhVmYIECypVopXCArX0LWVC5fHcJFbH9tR2niOzQ2xZsoVP4G/4G+w0dBIPYCnKnXuO7zlzfcdxllKl6/VfCzduLlZu3V66U7177/6Dh8srj06U6EkCx0SkQp7FWEFKORxrqlM4yyRgFqdwGl+8dfhpH6Sigh/pUQYthjuctinB2qY+RqnoJKDXIpIIvX6+XKtv1CcrKAfhNKih6do/X1n8GSWC9BhwTVKsVDOsZ7qVY6kpScFUo56CDJML3IGmDTlmoFr5xLYJVm0mCdpC2h/XwSQ7uyPHTKkRiy2TYd1VPuaS/8J0lxk/51SUCSZrDrNkLUSq5ne4uuV0s6fbr1s55VlPAydXb9HupYEWgWtwkFAJRKcjG2AiqW1EQLpYYqLtMVRXA18iw2NhqnMSDF+AFILZTmFOILUGOAyIYAzzJI8oT0weub1xnL8zHjoA2unqP4R2PvAJw2EBDn1wNCrAkQ+OxwU4LukO/qOpMj5FCU7zwzKsJcAM48hnNBozaMNHQUo7UHnUxxIyRVPBPUJiv6SZAjt+gQTacOkKEDfJdhD5li9xaTkm35qheIzdYVYcy66vcHBQgAfGHfgsSkTf+T+kHYbPpSmB5Bok/pnYSenbgZu8/7T9dV88kw7dY9ApV88kucb84of7O4XryantgP3OJby3qb0MJLa6z/IIS2vNjpX9j5676K/EPNLWxpFvwLrHhXfse7+6ocwkiFzk2XcD28W6PK2uK6YZtqwsDHVuH81aLVw39pYL/TutHJxsboQvNjY/vKxtv5ned0voCXqK1lCIXqFt1ED76BgRJNE39B39qJxVPlU+V75cUW8sTPc8RnOr8vU3qgwfuQ==</latexit><latexit sha1_base64="I5o3C+kVizOrVwLaHVTp+YmXgXw=">AAAF4HicfVRLb9NAEN5CAyW8WpC4cLGIKhVaVXFBggOVWtFK4YBa+pbiYK3Xk8Sq7XV3N8/Fd26IK0eucOLf8G+YTUOTrIGVLI/n+3a+2ZnxBlkcSVWt/pq7dn2+dOPmwq3y7Tt3791fXHpwInlHMDhmPObiLKAS4iiFYxWpGM4yATQJYjgNzt8Y/LQLQkY8PVKDDBoJbaVRM2JUoctffLTtu2vOtr/hRal3uL/j69XV/EPiL1aq69XRcoqGOzYqZLz2/aX5n17IWSeBVLGYSll3q5lqaCpUxGLIy15HQkbZOW1BHc2UJiAbenSA3FlGT+g0ucAnVc7IO71D00TKQRIgM6GqLW3MOP+FqXaS2z6jInNntGYwJCvOYzm7w8Qtuusd1XzV0FGadRSk7PIUzU7sKO6YUjthJICpeIAGZSLCQjisTQVlChtSXnZsiYwOeV6ekUjoOQjOE6wUTRnEmEAKPcaThKahxp6FufbM3iDQb3ML7UHUaqs/hKbu2YR+fwL2bXAwmIADGxwOJ+CwoNv7j6bM0jHKaKwPi7ASAFOMI5tRq02hNRsFIXCgtNelAjIZxTy1CCH+U1MBduwAITThwgRgZpJxENNNW+ICObnenKJYjN1+NmnLrq1wcDABD3LT8GmU8a7J/zBqJdQXeQFkVyCze4KT0sWBG51/XP6qLZ4Jg+4l0CpGzwS7wuzgeDVMsh51bQfwPxfwDl17GQiKus+0RwWmhmOFb2/NWH8lak9hGkd2Apg9neRO7dxj3grBNA8Nz1hW+mZg21QVp9VUJa+7DZSFvtL4ma9U3Kc53nKufacVjZONdff5+sb7F5Wt1+P7boE8Jk/ICnHJS7JFamSfHBNGPpJv5Dv5UQpKn0qfS18uqdfmxnsekplV+vobAV8h5Q==</latexit>log det(A1 )  log det(A2 ) + hr log det(A2 ), A1= log det(A2 ) + Tr(A2 1 A1 )<latexit sha1_base64="lUU9ELiy6Tt7E7rMM2tC2y3bcuU=">AAAGSHicfVTNbtNAEHZKAyX8tZQblxVRqxbaKg5IcKBSEa0UDqil/1IcovVmklhdr9315ne1PBSPwRtw5wG4IW6M07RJ1sBKtsfzfTvfeGc8fsyDRJVK33Nzt+bzt+8s3C3cu//g4aPFpcenSdSRDE5YxCN57tMEeCDgRAWKw3ksgYY+hzP/4n2Kn3VBJkEkjtUghlpIWyJoBowqdNUXv3g8ankNUGvv6u46WSUeh0sy5Syvkxf4TkWLA/EE9TmdhTcI7iSbeC97ckzzCqvb2SBKpuZnvemakdgmCeuLxdJWabRI1nDHRtEZr4P60vxXrxGxTghCMU6TpOqWYlXTVKqAcTAFr5NATNkFbUEVTUFDSGp6dFCGrKCnQZqRxEsoMvJO79A0TJJB6CMzpKqd2Fjq/Bem2qGxfalKYshozWBIVlHEk9kdadysu9pRzTc1HYi4o0Cwq69odjhREUlLShqBBKb4AA3KZIAHQVibSsoUFr6wQmyJmA4jU5iRCOkFyCgK8aSoYMAxAQE9FoUhFQ3tBaJhtJfu9X39wVhoD4JWW10TmrpnE/r9Cdi3wcFgAg5scDicgMOMbu8/mkksxiijXB9lYSUBphjHNqNSmUIrNgpSYkNpr0slxEnAI2ERGvjvTgXYtQM0oAmXaQCWdjI2oti2JS6RY/T2FMVi7PXjSVn2bIXDwwl4aNKCT6Ms6qb5HwWtkNalyYDsBmR2TbBTuthwo+8fH3/JFo9liu6H0MpGjyW7wezgRwe7k6xHVdsF/M8lfETXfgySou5z7VGJqWFb4dPbSK2/EjXOHKOP7QQwezrJndq549zCsWX09QCz008btk1VtlvTUzFVt4ay0FcaX81a0V03OOVce6ZljdPylvtyq/zpVXHn7XjeLThPnWfOmuM6r50dp+IcOCcOc37kCrnl3JP8t/zP/K/87yvqXG68Z9mZWbfn/gB9jkFA</latexit>

m

A2 i

Apply the above inequality twice, we obtain:Td log det(W ⌦c W /2d) <latexit sha1_base64="b0i+in/ljENAkxN/dENH62iT0J0=">AAAGg3icjVTbTttAEHUopDS9QfvYh64aUZECIUmR2ociIRWk9KGFQiBIcYg260liYXvNenNd7U/1rZ/Sv+k4BJys26orWR7POXtmPDs77dBzI1kq/cosPVheyT5cfZR7/OTps+dr6y8uIt4XDM4Z97i4bNMIPDeAc+lKDy5DAdRve1BvX3+O8foAROTyoCbHITR92g3cjsuoRFdr7SfB5dge79oOyM26HQpG6le13YpTIG+J7cEN2UlwhEWBbBG7IyhTZa0q2pZi6ib3ewtkhzjhNrHtXKweJupXtVvmbiVM1MN5dfaf6sVcay1fKpami6SN8szIW7N10lpf/mE7nPV9CCTzaBQ1yqVQNhUV0mUe6JzdjyCk7Jp2oYFmQH2ImmpaY0020OOQDhf4BJJMvfM7FPWjaOy3kelT2YtMLHb+DZM9X5u+OEqkyXQtYEiWnHvR4o5YN+1u9GXnY1O5QdiXELDbv+j0PSI5ibuBOK4AJr0xGpQJFwtBWI9i9SX2TG6DmCFCOuE6txDCp9cgOPexUjRg4GECAQwZ930aOMp2A0crO97bbqsv2kCH4HZ78o7QUUOTMBol4MgEx+MEHJvgZJKAk1Tc4T9iRmEwQxn11FkalgJgjlEzGdXqHFo1URACG0rZAyogjFyPBwbBwWs/J3BoCjjQgZtYgMWdjI0Y7JshbpCj1f4cxWAcjcLkWI7MCKenCXiq4wOfRxkfxPmfuV2ftoROgeweZOaZYKcMsOGm/z8rf8kMjrcd0WMfuml1nAH3mCl+dnKYZD09tUPAey7gK7qOQxAU475TNhWYGrYVvu3t2PojUeHk0apmJoDZ0yR3auaOkwwHmVZ3I81MP27YHpXpbo2rohvlJoaFkVT4qTfz5YLGKVc2Z1rauKgUy++Lle97+YNPs3m3ar2y3libVtn6YB1YVevEOrdY5nXmKPMtc5xdyW5lK9m9W+pSZrbnpbWwsvu/AckPU00=</latexit>

p log det(W T ⌦r W/2p) 1d log det(⌦r ) + Tr(⌦r W ⌦c W T )21
p log det(⌦c ) + Tr(⌦r W ⌦c W T )
2

dp,
dp.

40

Learning with Adaptive Regularization
Volume minimization
Apply the above inequality twice, we obtain:Td log det(W ⌦c W /2d) <latexit sha1_base64="b0i+in/ljENAkxN/dENH62iT0J0=">AAAGg3icjVTbTttAEHUopDS9QfvYh64aUZECIUmR2ociIRWk9KGFQiBIcYg260liYXvNenNd7U/1rZ/Sv+k4BJys26orWR7POXtmPDs77dBzI1kq/cosPVheyT5cfZR7/OTps+dr6y8uIt4XDM4Z97i4bNMIPDeAc+lKDy5DAdRve1BvX3+O8foAROTyoCbHITR92g3cjsuoRFdr7SfB5dge79oOyM26HQpG6le13YpTIG+J7cEN2UlwhEWBbBG7IyhTZa0q2pZi6ib3ewtkhzjhNrHtXKweJupXtVvmbiVM1MN5dfaf6sVcay1fKpami6SN8szIW7N10lpf/mE7nPV9CCTzaBQ1yqVQNhUV0mUe6JzdjyCk7Jp2oYFmQH2ImmpaY0020OOQDhf4BJJMvfM7FPWjaOy3kelT2YtMLHb+DZM9X5u+OEqkyXQtYEiWnHvR4o5YN+1u9GXnY1O5QdiXELDbv+j0PSI5ibuBOK4AJr0xGpQJFwtBWI9i9SX2TG6DmCFCOuE6txDCp9cgOPexUjRg4GECAQwZ930aOMp2A0crO97bbqsv2kCH4HZ78o7QUUOTMBol4MgEx+MEHJvgZJKAk1Tc4T9iRmEwQxn11FkalgJgjlEzGdXqHFo1URACG0rZAyogjFyPBwbBwWs/J3BoCjjQgZtYgMWdjI0Y7JshbpCj1f4cxWAcjcLkWI7MCKenCXiq4wOfRxkfxPmfuV2ftoROgeweZOaZYKcMsOGm/z8rf8kMjrcd0WMfuml1nAH3mCl+dnKYZD09tUPAey7gK7qOQxAU475TNhWYGrYVvu3t2PojUeHk0apmJoDZ0yR3auaOkwwHmVZ3I81MP27YHpXpbo2rohvlJoaFkVT4qTfz5YLGKVc2Z1rauKgUy++Lle97+YNPs3m3ar2y3libVtn6YB1YVevEOrdY5nXmKPMtc5xdyW5lK9m9W+pSZrbnpbWwsvu/AckPU00=</latexit>

p log det(W T ⌦r W/2p) 1d log det(⌦r ) + Tr(⌦r W ⌦c W T )21
p log det(⌦c ) + Tr(⌦r W ⌦c W T )
2

dp,
dp.

which implies:

1/2 2
d log det(W ⌦c W T ) + p log det(W T ⌦r W )  ||⌦1/2
W
⌦
r
c ||F

d log det(⌦r ) + p log det(⌦c ) + c<latexit sha1_base64="0PHbjs09wJ0TY26SBSC22q40yHg=">AAAGSnicfVTNbtNAEHZKU0oo0ILgwmVFVCnhpyQBCQ5UqkSLwgG1tGlTqU6i9XqSWLW97nrz61jimXgOXoAX4AG4IS7MpmmdrIGVLI/n++bHM7NjBa4TylLpe2bpxnJ25ebqrdzttTt3761v3D8JeU8wOGbc5eLUoiG4jg/H0pEunAYCqGe5ULfO3yu83gcROtyvyVEADY92fKftMCpR1Vr/Ypsu75g2yELdDAQj9WatSJ6RIFE3awgIUi8S04ULMpmoz2ZUflmJpyaX4mTS+tCskBfEtJxOIfGqyIsOlU1RsZSatdbzpa3S9JC0UJ4JeWN2Dloby19Nm7OeB75kLg3Ds3IpkI2ICukwF+Kc2QshoOycduAMRZ96EDaiaalisokam7S5wMeXZKqdt4ioF4Yjz0KmR2U31DGl/Bcmu16s61SUMCbTs4AhWXLuhosWym9afdaT7beNyPGDngSfXf5Fu+cSyYlqKrEdAUy6IxQoEw4WgrAuFZRJbH1uk+ghAjrmcW4hhEfPQXDuYaWoz8DFBHwYMO551Lcj0/HtODKVrWVFH2MNHYDT6corQjsa6IThMAGHOjgaJeBIB8fjBByn4g7+EzMM/BnKqBsdpWEpAOYYNZ1Rrc6hVR0FIXCgIrNPBQSh43JfI9h4e+cc7OoObGjDhXLA1CTjIPrbeogL5MTR9hxFY+wNg6Qte3qEw8MEPIxVw+dRxvsq/yOn49GWiFMguwaZ3hOclD4O3PT/Z+Uv6cHx3iO670En7R03wDWmOz862E2ynnZtF/CeC/iEqv0ABMW4TyOTCkwNxwrf5nMl/ZUYmRLTqOkJYPY0yZ3queOqwk0VR1c7S09fDWyXyvS0qqrEZ+UGhoWhjPAzLuTLxRi3XFnfaWnhpLJVfrVV+fw6v/Nutu9WjcfGE6NglI03xo5RNQ6MY4MZPzJrmYeZR9lv2Z/ZX9nfl9SlzMzmgbFwVpb/AGsRRpM=</latexit>

41

Learning with Adaptive Regularization
Volume minimization
Apply the above inequality twice, we obtain:Td log det(W ⌦c W /2d) <latexit sha1_base64="b0i+in/ljENAkxN/dENH62iT0J0=">AAAGg3icjVTbTttAEHUopDS9QfvYh64aUZECIUmR2ociIRWk9KGFQiBIcYg260liYXvNenNd7U/1rZ/Sv+k4BJys26orWR7POXtmPDs77dBzI1kq/cosPVheyT5cfZR7/OTps+dr6y8uIt4XDM4Z97i4bNMIPDeAc+lKDy5DAdRve1BvX3+O8foAROTyoCbHITR92g3cjsuoRFdr7SfB5dge79oOyM26HQpG6le13YpTIG+J7cEN2UlwhEWBbBG7IyhTZa0q2pZi6ib3ewtkhzjhNrHtXKweJupXtVvmbiVM1MN5dfaf6sVcay1fKpami6SN8szIW7N10lpf/mE7nPV9CCTzaBQ1yqVQNhUV0mUe6JzdjyCk7Jp2oYFmQH2ImmpaY0020OOQDhf4BJJMvfM7FPWjaOy3kelT2YtMLHb+DZM9X5u+OEqkyXQtYEiWnHvR4o5YN+1u9GXnY1O5QdiXELDbv+j0PSI5ibuBOK4AJr0xGpQJFwtBWI9i9SX2TG6DmCFCOuE6txDCp9cgOPexUjRg4GECAQwZ930aOMp2A0crO97bbqsv2kCH4HZ78o7QUUOTMBol4MgEx+MEHJvgZJKAk1Tc4T9iRmEwQxn11FkalgJgjlEzGdXqHFo1URACG0rZAyogjFyPBwbBwWs/J3BoCjjQgZtYgMWdjI0Y7JshbpCj1f4cxWAcjcLkWI7MCKenCXiq4wOfRxkfxPmfuV2ftoROgeweZOaZYKcMsOGm/z8rf8kMjrcd0WMfuml1nAH3mCl+dnKYZD09tUPAey7gK7qOQxAU475TNhWYGrYVvu3t2PojUeHk0apmJoDZ0yR3auaOkwwHmVZ3I81MP27YHpXpbo2rohvlJoaFkVT4qTfz5YLGKVc2Z1rauKgUy++Lle97+YNPs3m3ar2y3libVtn6YB1YVevEOrdY5nXmKPMtc5xdyW5lK9m9W+pSZrbnpbWwsvu/AckPU00=</latexit>

p log det(W T ⌦r W/2p) 1d log det(⌦r ) + Tr(⌦r W ⌦c W T )21
p log det(⌦c ) + Tr(⌦r W ⌦c W T )
2

dp,
dp.

which implies:

1/2 2
d log det(W ⌦c W T ) + p log det(W T ⌦r W )  ||⌦1/2
W
⌦
r
c ||F

d log det(⌦r ) + p log det(⌦c ) + c<latexit sha1_base64="0PHbjs09wJ0TY26SBSC22q40yHg=">AAAGSnicfVTNbtNAEHZKU0oo0ILgwmVFVCnhpyQBCQ5UqkSLwgG1tGlTqU6i9XqSWLW97nrz61jimXgOXoAX4AG4IS7MpmmdrIGVLI/n++bHM7NjBa4TylLpe2bpxnJ25ebqrdzttTt3761v3D8JeU8wOGbc5eLUoiG4jg/H0pEunAYCqGe5ULfO3yu83gcROtyvyVEADY92fKftMCpR1Vr/Ypsu75g2yELdDAQj9WatSJ6RIFE3awgIUi8S04ULMpmoz2ZUflmJpyaX4mTS+tCskBfEtJxOIfGqyIsOlU1RsZSatdbzpa3S9JC0UJ4JeWN2Dloby19Nm7OeB75kLg3Ds3IpkI2ICukwF+Kc2QshoOycduAMRZ96EDaiaalisokam7S5wMeXZKqdt4ioF4Yjz0KmR2U31DGl/Bcmu16s61SUMCbTs4AhWXLuhosWym9afdaT7beNyPGDngSfXf5Fu+cSyYlqKrEdAUy6IxQoEw4WgrAuFZRJbH1uk+ghAjrmcW4hhEfPQXDuYaWoz8DFBHwYMO551Lcj0/HtODKVrWVFH2MNHYDT6corQjsa6IThMAGHOjgaJeBIB8fjBByn4g7+EzMM/BnKqBsdpWEpAOYYNZ1Rrc6hVR0FIXCgIrNPBQSh43JfI9h4e+cc7OoObGjDhXLA1CTjIPrbeogL5MTR9hxFY+wNg6Qte3qEw8MEPIxVw+dRxvsq/yOn49GWiFMguwaZ3hOclD4O3PT/Z+Uv6cHx3iO670En7R03wDWmOz862E2ynnZtF/CeC/iEqv0ABMW4TyOTCkwNxwrf5nMl/ZUYmRLTqOkJYPY0yZ3queOqwk0VR1c7S09fDWyXyvS0qqrEZ+UGhoWhjPAzLuTLxRi3XFnfaWnhpLJVfrVV+fw6v/Nutu9WjcfGE6NglI03xo5RNQ6MY4MZPzJrmYeZR9lv2Z/ZX9nfl9SlzMzmgbFwVpb/AGsRRpM=</latexit>

Squared volume ofthe parallelepiped spanned byrow/column vectorsof preconditioned W

Our regularizer<latexit sha1_base64="afdHXKm1PEcIFWEq8199Onu/chk=">AAAETnicfZPNbhMxEMfdboEQvlo4clkRVeIUJaVSuVSKBJXCrQ1NUymJKq93NrHij63tbbKx9gm4wntx5UW4IfCmQUkcwUgrjeb3nxl71hOljGrTaPzY2Q32Hjx8VHlcffL02fMX+wcvr7TMFIEukUyq6whrYFRA11DD4DpVgHnEoBdNPpS8dwdKUykuTZ7CkOORoAkl2LjQRe9mv9aoNxYWbjvNpVNDSzu/OQhOBrEkGQdhCMNa95uN1AwtVoYSBkV1kGlIMZngEfSdKzAHPbSLkxbhoYvEYSKV+4QJF9H1DIu51jmPnJJjM9Y+K4P/YmbMCz9WdtFFuLAN5sRGSqY3M8q62+F+ZpL3Q0tFmhkQ5P4WScZCI8NypmFMFRDDcudgoqgbREjGWGFi3OSrh6HfIsVzWVQ3WnA8ASUld5PCggBzBxAwJZJzLGI7oCIu7KDMjSL7qfDoFOhobP4KEjv1BbPZCs58mOcrmPtwPl/B+Vbf6X966lQsKcHMft7GRgGsKS59Rbu9Rts+BaXcg7KDO6wg1ZRJ4QlitzxrBT76BWJI4LYsQMqX7B6iOPVb3DpNYU/XJJ7ibJaufsuZ36HTWcFOUbhVa/qLte1cHdWb7+pHF8e1Vmu5dBX0Gr1Bb1ETnaAWaqNz1EUEAfqCvqJvwffgZ/Ar+H0v3d1Z5rxCG7ZX+QNcQ4Rs</latexit>

42

Learning with Adaptive Regularization
How to optimize (⌦r , ⌦c ) ?
<latexit sha1_base64="dqOSiMRaptH+HQV6xkr+TT5NMDg=">AAAF1HicfVRLb9NAEN5CAyW8WjhysYgqtaiq4oIEBypVopXCAbX0LcVRtV5PEqvrXXd38zQ+Ia4cucJv4N/wb5hNQ52sgZXijOf7dubzzOyGKY+1qdd/Ldy6vVi5c3fpXvX+g4ePHi+vPDnVsqcYnDDJpToPqQYeCzgxseFwniqgScjhLLx8Z/GzPigdS3FsRim0EtoRcTtm1KArWAtSpTY8fLL1i+VafbM+WV7Z8KdGjUzXwcXK4s8gkqyXgDCMU62bfj01rYwqEzMOeTXoaUgpu6QdaKIpaAK6lU1E594qeiKvLRX+hPEm3tkdGU20HiUhMhNqutrFrPNfmOkmueuzWXTuTdYchmQjJdfzO2zcsrvZM+03rSwWac+AYNdf0e5xz0jPlteLYgXM8BEalKkYC+GxLlWUGWxCddVzU6R0LPPqXIqEXoKSMsFKUcGAowABAyaThIooC2IR5Vlg94Zh9j530AHEna75Q2hnA5cwHBbg0AVHowIcueB4XIDjUt7Bf3LqVExRRnl2VIaNAphhHLuMRmMGbbgoKIUDlQV9qiDVMZfCIUR4jmYC7LoBImjDlQ3A7CTjIIptN8UVcvJse4biMPaGadGWPTfD4WEBHua24bMok32r/yjuJPRC5SWQ3YDM7QlOSh8HbvL90/LX3eR4xBHdT6BTjo4H/wZzgx8d7BaqJ13bBTznCj6gaz8FRTHviyygCqXhWOF/sGGtvxKzwKCMY1cAqqeFdupq57ITgW0eGoG1HPl2YLvUlKfVViVv+i1MC0OT4Wu+VvPXc7zlfPdOKxunW5v+y82tj69qO2+n990SeUaekzXik9dkhzTIATkhjKTkG/lOflROK58qnytfrqm3FqZ7npK5Vfn6G3aPHmE=</latexit>

min minW,a ⌦r ,⌦c

1 X(ŷ(xi ; W, a)2nsubject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>

i2[n]

⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )

vId

43

Learning with Adaptive Regularization
How to optimize (⌦r , ⌦c ) ?
<latexit sha1_base64="dqOSiMRaptH+HQV6xkr+TT5NMDg=">AAAF1HicfVRLb9NAEN5CAyW8WjhysYgqtaiq4oIEBypVopXCAbX0LcVRtV5PEqvrXXd38zQ+Ia4cucJv4N/wb5hNQ52sgZXijOf7dubzzOyGKY+1qdd/Ldy6vVi5c3fpXvX+g4ePHi+vPDnVsqcYnDDJpToPqQYeCzgxseFwniqgScjhLLx8Z/GzPigdS3FsRim0EtoRcTtm1KArWAtSpTY8fLL1i+VafbM+WV7Z8KdGjUzXwcXK4s8gkqyXgDCMU62bfj01rYwqEzMOeTXoaUgpu6QdaKIpaAK6lU1E594qeiKvLRX+hPEm3tkdGU20HiUhMhNqutrFrPNfmOkmueuzWXTuTdYchmQjJdfzO2zcsrvZM+03rSwWac+AYNdf0e5xz0jPlteLYgXM8BEalKkYC+GxLlWUGWxCddVzU6R0LPPqXIqEXoKSMsFKUcGAowABAyaThIooC2IR5Vlg94Zh9j530AHEna75Q2hnA5cwHBbg0AVHowIcueB4XIDjUt7Bf3LqVExRRnl2VIaNAphhHLuMRmMGbbgoKIUDlQV9qiDVMZfCIUR4jmYC7LoBImjDlQ3A7CTjIIptN8UVcvJse4biMPaGadGWPTfD4WEBHua24bMok32r/yjuJPRC5SWQ3YDM7QlOSh8HbvL90/LX3eR4xBHdT6BTjo4H/wZzgx8d7BaqJ13bBTznCj6gaz8FRTHviyygCqXhWOF/sGGtvxKzwKCMY1cAqqeFdupq57ITgW0eGoG1HPl2YLvUlKfVViVv+i1MC0OT4Wu+VvPXc7zlfPdOKxunW5v+y82tj69qO2+n990SeUaekzXik9dkhzTIATkhjKTkG/lOflROK58qnytfrqm3FqZ7npK5Vfn6G3aPHmE=</latexit>

1 X(ŷ(xi ; W, a)2nmin minW,a ⌦r ,⌦ci2[n]subject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )vId- For fixed W and ⌦c, the objective function is convex in ⌦r<latexit sha1_base64="6T3asusjEPCEK52gk0msQW2zayM=">AAAFyXicfVRbb9MwFPbGCqPcNnjkJaKaNBCamoEED0yaxCYVIbT7RWqqyXFO26iOndnuNcoTj7zCH+Hf8G847srSOoClqifn+3y+L8cnDlMea1Ov/1pavrNSuXtv9X71wcNHj5+srT8917KvGJwxyaW6DKkGHgs4M7HhcJkqoEnI4SLsfbT4xQCUjqU4NeMUWgntiLgdM2owdXRxtVarb9WnyysH/iyokdk6vFpf+RlEkvUTEIZxqnXTr6emlVFlYsYhrwZ9DSllPdqBJoaCJqBb2dRp7m1gJvLaUuFPGG+and+R0UTrcRIiM6Gmq13MJv+FmW6SuzmronNvuhYwJBspuV7cYeuW082+ab9vZbFI+wYEu3mLdp97Rnq2p14UK2CGjzGgTMXYCI91qaLMYOerG54rkdKJzKsLEgntgZIywU5RwYCjAQFDJpOEiigLYhHlWWD3hmH2KXfQIcSdrvlDaGdDlzAaFeDIBcfjAhy74GRSgJOS7vA/mjoVM5RRnp2UYaMA5hinLqPRmEMbLgpK4UBlwYAqSHXMpXAIEX48cwX23AIRtOHaFmB2knEQxY4rcY2cPNuZoziM/VFaHMu+q3B8XIDHuT3weZTJgfV/EncSeqXyEshuQeaeCU7KAAdu+v6z9tdd8VRZ9CCBTrl6qtgt5hY/OdwrXE9PbQ/wO1fwBVMHKSiKuq+ygCq0hmOF/8FrG/2VmAUGbZy6BtA9LbxT1zuXnQjs4WEQ2Mixbwe2S015Wm1X8qbfQlkYmQwf882a/zLHW85377RycL695b/Z2j56W9v9MLvvVslz8oJsEp+8I7ukQQ7JGWEEyDfynfyofK5cV0aVyQ11eWm25xlZWJWvvwFBPxpc</latexit><latexit sha1_base64="O3mXawymR6ta5/qBZpvtZhyYxqo=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3axWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pn0qH7GXTL0fHc32B+8KOD3UL1tGu7gOdcwid07ecgKeZ9aSIqURqOFb6jV876K9FEGmUc+wJQPS20U197KroxuOahETnLk+8Gtkd1eVpdVWwzbGFaGGmDn3ajFr6weMuF/p1WNk63NsPXm1tf3tR23s/uuxXyjDwnGyQkb8kOaZADckIY6ZFv5Dv5Uflc0RVTsdfUW0uzPU/Jwqp8/Q2ylRvE</latexit><latexit sha1_base64="sxM8U0tIxERXSq4P3fTrZOgvV70=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3mxWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pj2UZ0P4NuOXou2Q3mBz862C1UT7u2C3jOJXxC134OkmLelyaiEqXhWOE7euWsvxJNpFHGsS8A1dNCO/W1p6Ibg2seGpGzPPluYHtUl6fVVcU2wxamhZE2+Gk3auELi7dc6N9pZeN0azN8vbn15U1t5/3svlshz8hzskFC8pbskAY5ICeEkR75Rr6TH5XPFV0xFXtNvbU02/OULKzK198DjxvT</latexit>

44

Learning with Adaptive Regularization
How to optimize (⌦r , ⌦c ) ?
<latexit sha1_base64="dqOSiMRaptH+HQV6xkr+TT5NMDg=">AAAF1HicfVRLb9NAEN5CAyW8WjhysYgqtaiq4oIEBypVopXCAbX0LcVRtV5PEqvrXXd38zQ+Ia4cucJv4N/wb5hNQ52sgZXijOf7dubzzOyGKY+1qdd/Ldy6vVi5c3fpXvX+g4ePHi+vPDnVsqcYnDDJpToPqQYeCzgxseFwniqgScjhLLx8Z/GzPigdS3FsRim0EtoRcTtm1KArWAtSpTY8fLL1i+VafbM+WV7Z8KdGjUzXwcXK4s8gkqyXgDCMU62bfj01rYwqEzMOeTXoaUgpu6QdaKIpaAK6lU1E594qeiKvLRX+hPEm3tkdGU20HiUhMhNqutrFrPNfmOkmueuzWXTuTdYchmQjJdfzO2zcsrvZM+03rSwWac+AYNdf0e5xz0jPlteLYgXM8BEalKkYC+GxLlWUGWxCddVzU6R0LPPqXIqEXoKSMsFKUcGAowABAyaThIooC2IR5Vlg94Zh9j530AHEna75Q2hnA5cwHBbg0AVHowIcueB4XIDjUt7Bf3LqVExRRnl2VIaNAphhHLuMRmMGbbgoKIUDlQV9qiDVMZfCIUR4jmYC7LoBImjDlQ3A7CTjIIptN8UVcvJse4biMPaGadGWPTfD4WEBHua24bMok32r/yjuJPRC5SWQ3YDM7QlOSh8HbvL90/LX3eR4xBHdT6BTjo4H/wZzgx8d7BaqJ13bBTznCj6gaz8FRTHviyygCqXhWOF/sGGtvxKzwKCMY1cAqqeFdupq57ITgW0eGoG1HPl2YLvUlKfVViVv+i1MC0OT4Wu+VvPXc7zlfPdOKxunW5v+y82tj69qO2+n990SeUaekzXik9dkhzTIATkhjKTkG/lOflROK58qnytfrqm3FqZ7npK5Vfn6G3aPHmE=</latexit>

1 X(ŷ(xi ; W, a)2nmin minW,a ⌦r ,⌦ci2[n]subject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )vId- For fixed W and ⌦c, the objective function is convex in ⌦r- For fixed W and ⌦r, the objective function is convex in ⌦c<latexit sha1_base64="6T3asusjEPCEK52gk0msQW2zayM=">AAAFyXicfVRbb9MwFPbGCqPcNnjkJaKaNBCamoEED0yaxCYVIbT7RWqqyXFO26iOndnuNcoTj7zCH+Hf8G847srSOoClqifn+3y+L8cnDlMea1Ov/1pavrNSuXtv9X71wcNHj5+srT8917KvGJwxyaW6DKkGHgs4M7HhcJkqoEnI4SLsfbT4xQCUjqU4NeMUWgntiLgdM2owdXRxtVarb9WnyysH/iyokdk6vFpf+RlEkvUTEIZxqnXTr6emlVFlYsYhrwZ9DSllPdqBJoaCJqBb2dRp7m1gJvLaUuFPGG+and+R0UTrcRIiM6Gmq13MJv+FmW6SuzmronNvuhYwJBspuV7cYeuW082+ab9vZbFI+wYEu3mLdp97Rnq2p14UK2CGjzGgTMXYCI91qaLMYOerG54rkdKJzKsLEgntgZIywU5RwYCjAQFDJpOEiigLYhHlWWD3hmH2KXfQIcSdrvlDaGdDlzAaFeDIBcfjAhy74GRSgJOS7vA/mjoVM5RRnp2UYaMA5hinLqPRmEMbLgpK4UBlwYAqSHXMpXAIEX48cwX23AIRtOHaFmB2knEQxY4rcY2cPNuZoziM/VFaHMu+q3B8XIDHuT3weZTJgfV/EncSeqXyEshuQeaeCU7KAAdu+v6z9tdd8VRZ9CCBTrl6qtgt5hY/OdwrXE9PbQ/wO1fwBVMHKSiKuq+ygCq0hmOF/8FrG/2VmAUGbZy6BtA9LbxT1zuXnQjs4WEQ2Mixbwe2S015Wm1X8qbfQlkYmQwf882a/zLHW85377RycL695b/Z2j56W9v9MLvvVslz8oJsEp+8I7ukQQ7JGWEEyDfynfyofK5cV0aVyQ11eWm25xlZWJWvvwFBPxpc</latexit><latexit sha1_base64="O3mXawymR6ta5/qBZpvtZhyYxqo=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3axWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pn0qH7GXTL0fHc32B+8KOD3UL1tGu7gOdcwid07ecgKeZ9aSIqURqOFb6jV876K9FEGmUc+wJQPS20U197KroxuOahETnLk+8Gtkd1eVpdVWwzbGFaGGmDn3ajFr6weMuF/p1WNk63NsPXm1tf3tR23s/uuxXyjDwnGyQkb8kOaZADckIY6ZFv5Dv5Uflc0RVTsdfUW0uzPU/Jwqp8/Q2ylRvE</latexit><latexit sha1_base64="6T3asusjEPCEK52gk0msQW2zayM=">AAAFyXicfVRbb9MwFPbGCqPcNnjkJaKaNBCamoEED0yaxCYVIbT7RWqqyXFO26iOndnuNcoTj7zCH+Hf8G847srSOoClqifn+3y+L8cnDlMea1Ov/1pavrNSuXtv9X71wcNHj5+srT8917KvGJwxyaW6DKkGHgs4M7HhcJkqoEnI4SLsfbT4xQCUjqU4NeMUWgntiLgdM2owdXRxtVarb9WnyysH/iyokdk6vFpf+RlEkvUTEIZxqnXTr6emlVFlYsYhrwZ9DSllPdqBJoaCJqBb2dRp7m1gJvLaUuFPGG+and+R0UTrcRIiM6Gmq13MJv+FmW6SuzmronNvuhYwJBspuV7cYeuW082+ab9vZbFI+wYEu3mLdp97Rnq2p14UK2CGjzGgTMXYCI91qaLMYOerG54rkdKJzKsLEgntgZIywU5RwYCjAQFDJpOEiigLYhHlWWD3hmH2KXfQIcSdrvlDaGdDlzAaFeDIBcfjAhy74GRSgJOS7vA/mjoVM5RRnp2UYaMA5hinLqPRmEMbLgpK4UBlwYAqSHXMpXAIEX48cwX23AIRtOHaFmB2knEQxY4rcY2cPNuZoziM/VFaHMu+q3B8XIDHuT3weZTJgfV/EncSeqXyEshuQeaeCU7KAAdu+v6z9tdd8VRZ9CCBTrl6qtgt5hY/OdwrXE9PbQ/wO1fwBVMHKSiKuq+ygCq0hmOF/8FrG/2VmAUGbZy6BtA9LbxT1zuXnQjs4WEQ2Mixbwe2S015Wm1X8qbfQlkYmQwf882a/zLHW85377RycL695b/Z2j56W9v9MLvvVslz8oJsEp+8I7ukQQ7JGWEEyDfynfyofK5cV0aVyQ11eWm25xlZWJWvvwFBPxpc</latexit><latexit sha1_base64="sxM8U0tIxERXSq4P3fTrZOgvV70=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3mxWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pj2UZ0P4NuOXou2Q3mBz862C1UT7u2C3jOJXxC134OkmLelyaiEqXhWOE7euWsvxJNpFHGsS8A1dNCO/W1p6Ibg2seGpGzPPluYHtUl6fVVcU2wxamhZE2+Gk3auELi7dc6N9pZeN0azN8vbn15U1t5/3svlshz8hzskFC8pbskAY5ICeEkR75Rr6TH5XPFV0xFXtNvbU02/OULKzK198DjxvT</latexit><latexit sha1_base64="sxM8U0tIxERXSq4P3fTrZOgvV70=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3mxWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pj2UZ0P4NuOXou2Q3mBz862C1UT7u2C3jOJXxC134OkmLelyaiEqXhWOE7euWsvxJNpFHGsS8A1dNCO/W1p6Ibg2seGpGzPPluYHtUl6fVVcU2wxamhZE2+Gk3auELi7dc6N9pZeN0azN8vbn15U1t5/3svlshz8hzskFC8pbskAY5ICeEkR75Rr6TH5XPFV0xFXtNvbU02/OULKzK198DjxvT</latexit><latexit sha1_base64="O3mXawymR6ta5/qBZpvtZhyYxqo=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3axWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pn0qH7GXTL0fHc32B+8KOD3UL1tGu7gOdcwid07ecgKeZ9aSIqURqOFb6jV876K9FEGmUc+wJQPS20U197KroxuOahETnLk+8Gtkd1eVpdVWwzbGFaGGmDn3ajFr6weMuF/p1WNk63NsPXm1tf3tR23s/uuxXyjDwnGyQkb8kOaZADckIY6ZFv5Dv5Uflc0RVTsdfUW0uzPU/Jwqp8/Q2ylRvE</latexit>

45

Learning with Adaptive Regularization
How to optimize (⌦r , ⌦c ) ?
<latexit sha1_base64="dqOSiMRaptH+HQV6xkr+TT5NMDg=">AAAF1HicfVRLb9NAEN5CAyW8WjhysYgqtaiq4oIEBypVopXCAbX0LcVRtV5PEqvrXXd38zQ+Ia4cucJv4N/wb5hNQ52sgZXijOf7dubzzOyGKY+1qdd/Ldy6vVi5c3fpXvX+g4ePHi+vPDnVsqcYnDDJpToPqQYeCzgxseFwniqgScjhLLx8Z/GzPigdS3FsRim0EtoRcTtm1KArWAtSpTY8fLL1i+VafbM+WV7Z8KdGjUzXwcXK4s8gkqyXgDCMU62bfj01rYwqEzMOeTXoaUgpu6QdaKIpaAK6lU1E594qeiKvLRX+hPEm3tkdGU20HiUhMhNqutrFrPNfmOkmueuzWXTuTdYchmQjJdfzO2zcsrvZM+03rSwWac+AYNdf0e5xz0jPlteLYgXM8BEalKkYC+GxLlWUGWxCddVzU6R0LPPqXIqEXoKSMsFKUcGAowABAyaThIooC2IR5Vlg94Zh9j530AHEna75Q2hnA5cwHBbg0AVHowIcueB4XIDjUt7Bf3LqVExRRnl2VIaNAphhHLuMRmMGbbgoKIUDlQV9qiDVMZfCIUR4jmYC7LoBImjDlQ3A7CTjIIptN8UVcvJse4biMPaGadGWPTfD4WEBHua24bMok32r/yjuJPRC5SWQ3YDM7QlOSh8HbvL90/LX3eR4xBHdT6BTjo4H/wZzgx8d7BaqJ13bBTznCj6gaz8FRTHviyygCqXhWOF/sGGtvxKzwKCMY1cAqqeFdupq57ITgW0eGoG1HPl2YLvUlKfVViVv+i1MC0OT4Wu+VvPXc7zlfPdOKxunW5v+y82tj69qO2+n990SeUaekzXik9dkhzTIATkhjKTkG/lOflROK58qnytfrqm3FqZ7npK5Vfn6G3aPHmE=</latexit>

1 X(ŷ(xi ; W, a)2nmin minW,a ⌦r ,⌦ci2[n]subject to uIp<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>⌦r

vIp , uId

1/2 2yi )2 + ||⌦1/2
W
⌦
r
c ||F

⌦c

d log det(⌦r ) + p log det(⌦c )vId- For fixed W and ⌦c, the objective function is convex in ⌦r- For fixed W and ⌦r, the objective function is convex in ⌦c<latexit sha1_base64="6T3asusjEPCEK52gk0msQW2zayM=">AAAFyXicfVRbb9MwFPbGCqPcNnjkJaKaNBCamoEED0yaxCYVIbT7RWqqyXFO26iOndnuNcoTj7zCH+Hf8G847srSOoClqifn+3y+L8cnDlMea1Ov/1pavrNSuXtv9X71wcNHj5+srT8917KvGJwxyaW6DKkGHgs4M7HhcJkqoEnI4SLsfbT4xQCUjqU4NeMUWgntiLgdM2owdXRxtVarb9WnyysH/iyokdk6vFpf+RlEkvUTEIZxqnXTr6emlVFlYsYhrwZ9DSllPdqBJoaCJqBb2dRp7m1gJvLaUuFPGG+and+R0UTrcRIiM6Gmq13MJv+FmW6SuzmronNvuhYwJBspuV7cYeuW082+ab9vZbFI+wYEu3mLdp97Rnq2p14UK2CGjzGgTMXYCI91qaLMYOerG54rkdKJzKsLEgntgZIywU5RwYCjAQFDJpOEiigLYhHlWWD3hmH2KXfQIcSdrvlDaGdDlzAaFeDIBcfjAhy74GRSgJOS7vA/mjoVM5RRnp2UYaMA5hinLqPRmEMbLgpK4UBlwYAqSHXMpXAIEX48cwX23AIRtOHaFmB2knEQxY4rcY2cPNuZoziM/VFaHMu+q3B8XIDHuT3weZTJgfV/EncSeqXyEshuQeaeCU7KAAdu+v6z9tdd8VRZ9CCBTrl6qtgt5hY/OdwrXE9PbQ/wO1fwBVMHKSiKuq+ygCq0hmOF/8FrG/2VmAUGbZy6BtA9LbxT1zuXnQjs4WEQ2Mixbwe2S015Wm1X8qbfQlkYmQwf882a/zLHW85377RycL695b/Z2j56W9v9MLvvVslz8oJsEp+8I7ukQQ7JGWEEyDfynfyofK5cV0aVyQ11eWm25xlZWJWvvwFBPxpc</latexit><latexit sha1_base64="O3mXawymR6ta5/qBZpvtZhyYxqo=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3axWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pn0qH7GXTL0fHc32B+8KOD3UL1tGu7gOdcwid07ecgKeZ9aSIqURqOFb6jV876K9FEGmUc+wJQPS20U197KroxuOahETnLk+8Gtkd1eVpdVWwzbGFaGGmDn3ajFr6weMuF/p1WNk63NsPXm1tf3tR23s/uuxXyjDwnGyQkb8kOaZADckIY6ZFv5Dv5Uflc0RVTsdfUW0uzPU/Jwqp8/Q2ylRvE</latexit><latexit sha1_base64="6T3asusjEPCEK52gk0msQW2zayM=">AAAFyXicfVRbb9MwFPbGCqPcNnjkJaKaNBCamoEED0yaxCYVIbT7RWqqyXFO26iOndnuNcoTj7zCH+Hf8G847srSOoClqifn+3y+L8cnDlMea1Ov/1pavrNSuXtv9X71wcNHj5+srT8917KvGJwxyaW6DKkGHgs4M7HhcJkqoEnI4SLsfbT4xQCUjqU4NeMUWgntiLgdM2owdXRxtVarb9WnyysH/iyokdk6vFpf+RlEkvUTEIZxqnXTr6emlVFlYsYhrwZ9DSllPdqBJoaCJqBb2dRp7m1gJvLaUuFPGG+and+R0UTrcRIiM6Gmq13MJv+FmW6SuzmronNvuhYwJBspuV7cYeuW082+ab9vZbFI+wYEu3mLdp97Rnq2p14UK2CGjzGgTMXYCI91qaLMYOerG54rkdKJzKsLEgntgZIywU5RwYCjAQFDJpOEiigLYhHlWWD3hmH2KXfQIcSdrvlDaGdDlzAaFeDIBcfjAhy74GRSgJOS7vA/mjoVM5RRnp2UYaMA5hinLqPRmEMbLgpK4UBlwYAqSHXMpXAIEX48cwX23AIRtOHaFmB2knEQxY4rcY2cPNuZoziM/VFaHMu+q3B8XIDHuT3weZTJgfV/EncSeqXyEshuQeaeCU7KAAdu+v6z9tdd8VRZ9CCBTrl6qtgt5hY/OdwrXE9PbQ/wO1fwBVMHKSiKuq+ygCq0hmOF/8FrG/2VmAUGbZy6BtA9LbxT1zuXnQjs4WEQ2Mixbwe2S015Wm1X8qbfQlkYmQwf882a/zLHW85377RycL695b/Z2j56W9v9MLvvVslz8oJsEp+8I7ukQQ7JGWEEyDfynfyofK5cV0aVyQ11eWm25xlZWJWvvwFBPxpc</latexit><latexit sha1_base64="sxM8U0tIxERXSq4P3fTrZOgvV70=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3mxWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pj2UZ0P4NuOXou2Q3mBz862C1UT7u2C3jOJXxC134OkmLelyaiEqXhWOE7euWsvxJNpFHGsS8A1dNCO/W1p6Ibg2seGpGzPPluYHtUl6fVVcU2wxamhZE2+Gk3auELi7dc6N9pZeN0azN8vbn15U1t5/3svlshz8hzskFC8pbskAY5ICeEkR75Rr6TH5XPFV0xFXtNvbU02/OULKzK198DjxvT</latexit><latexit sha1_base64="sxM8U0tIxERXSq4P3fTrZOgvV70=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3mxWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pj2UZ0P4NuOXou2Q3mBz862C1UT7u2C3jOJXxC134OkmLelyaiEqXhWOE7euWsvxJNpFHGsS8A1dNCO/W1p6Ibg2seGpGzPPluYHtUl6fVVcU2wxamhZE2+Gk3auELi7dc6N9pZeN0azN8vbn15U1t5/3svlshz8hzskFC8pbskAY5ICeEkR75Rr6TH5XPFV0xFXtNvbU02/OULKzK198DjxvT</latexit><latexit sha1_base64="O3mXawymR6ta5/qBZpvtZhyYxqo=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3axWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pn0qH7GXTL0fHc32B+8KOD3UL1tGu7gOdcwid07ecgKeZ9aSIqURqOFb6jV876K9FEGmUc+wJQPS20U197KroxuOahETnLk+8Gtkd1eVpdVWwzbGFaGGmDn3ajFr6weMuF/p1WNk63NsPXm1tf3tR23s/uuxXyjDwnGyQkb8kOaZADckIY6ZFv5Dv5Uflc0RVTsdfUW0uzPU/Jwqp8/Q2ylRvE</latexit>- The objective function is symmetric w.r.t. ⌦r and ⌦c<latexit sha1_base64="ih/Ug2voogaiyPf4f/hF9ucRCPk=">AAAFzXicfVRNb9NAEN2WBkr4auHIxSKqVBCq4oIEBypVopXCAbW0TVspjqrNepKsau+6u+t8GXPlyBX+Bv+Gf8NsGupkDaxkeTzv7czzzOx2kohrU6//Wlq+tVK5fWf1bvXe/QcPH62tPz7VMlUMmkxGUp13qIaIC2gabiI4TxTQuBPBWefyvcXPBqA0l+LEjBNox7QneJczatDVDBKlvIu1Wn2rPl1e2fBnRo3M1uHF+srPIJQsjUEYFlGtW349Me2MKsNZBHk1SDUklF3SHrTQFDQG3c6manNvAz2h15UKH2G8qXd+R0ZjrcdxB5kxNX3tYtb5L8z049z12Sw696ZrAUOykTLSizts3LK7lZru23bGRZIaEOz6L7pp5Bnp2bp6IVfATDRGgzLFsRAe61NFmcHqVzc8N0VCJzKvLqSI6SUoKWOsFBUMIhQgYMhkHFMRZgEXYZ4Fdm+nk33IHXQIvNc3fwjdbOgSRqMCHLngeFyAYxecTApwUso7/E9OnYgZymiUHZdhowDmGCcuo9GYQxsuCkrhQGXBgCpINI+kcAghHqC5AHtugBC6cGUDMDvJOIhix01xhZw825mjOIz9UVK0Zd/NcHRUgEe5bfg8yuTA6j/mvZheqLwEshuQuT3BSRngwE3/f1b+upscDzeiBzH0ytETxW4wN/jx4V6hetq1PcBzruAjug4SUBTzvsgCqlAajhW+g5fW+isxCwzKOHEFoHpaaKeu9kj2QrDNQyOwliPfDmyfmvK02qrkLb+NaWFkMvzMN2v+8xxvOd+908rG6faW/2pr+9Pr2u672X23Sp6SZ2ST+OQN2SUNckiahBFOvpHv5EfloJJWPle+XFOXl2Z7npCFVfn6G9qqG/0=</latexit><latexit sha1_base64="O3mXawymR6ta5/qBZpvtZhyYxqo=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3axWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pn0qH7GXTL0fHc32B+8KOD3UL1tGu7gOdcwid07ecgKeZ9aSIqURqOFb6jV876K9FEGmUc+wJQPS20U197KroxuOahETnLk+8Gtkd1eVpdVWwzbGFaGGmDn3ajFr6weMuF/p1WNk63NsPXm1tf3tR23s/uuxXyjDwnGyQkb8kOaZADckIY6ZFv5Dv5Uflc0RVTsdfUW0uzPU/Jwqp8/Q2ylRvE</latexit>

It suﬃces if we focus on and solve one of them46

Learning with Adaptive Regularization
For fixed W and ⌦c, consider the optimization over ⌦r:<latexit sha1_base64="6T3asusjEPCEK52gk0msQW2zayM=">AAAFyXicfVRbb9MwFPbGCqPcNnjkJaKaNBCamoEED0yaxCYVIbT7RWqqyXFO26iOndnuNcoTj7zCH+Hf8G847srSOoClqifn+3y+L8cnDlMea1Ov/1pavrNSuXtv9X71wcNHj5+srT8917KvGJwxyaW6DKkGHgs4M7HhcJkqoEnI4SLsfbT4xQCUjqU4NeMUWgntiLgdM2owdXRxtVarb9WnyysH/iyokdk6vFpf+RlEkvUTEIZxqnXTr6emlVFlYsYhrwZ9DSllPdqBJoaCJqBb2dRp7m1gJvLaUuFPGG+and+R0UTrcRIiM6Gmq13MJv+FmW6SuzmronNvuhYwJBspuV7cYeuW082+ab9vZbFI+wYEu3mLdp97Rnq2p14UK2CGjzGgTMXYCI91qaLMYOerG54rkdKJzKsLEgntgZIywU5RwYCjAQFDJpOEiigLYhHlWWD3hmH2KXfQIcSdrvlDaGdDlzAaFeDIBcfjAhy74GRSgJOS7vA/mjoVM5RRnp2UYaMA5hinLqPRmEMbLgpK4UBlwYAqSHXMpXAIEX48cwX23AIRtOHaFmB2knEQxY4rcY2cPNuZoziM/VFaHMu+q3B8XIDHuT3weZTJgfV/EncSeqXyEshuQeaeCU7KAAdu+v6z9tdd8VRZ9CCBTrl6qtgt5hY/OdwrXE9PbQ/wO1fwBVMHKSiKuq+ygCq0hmOF/8FrG/2VmAUGbZy6BtA9LbxT1zuXnQjs4WEQ2Mixbwe2S015Wm1X8qbfQlkYmQwf882a/zLHW85377RycL695b/Z2j56W9v9MLvvVslz8oJsEp+8I7ukQQ7JGWEEyDfynfyofK5cV0aVyQ11eWm25xlZWJWvvwFBPxpc</latexit>

min<latexit sha1_base64="26QU4fsGr7YU80QGsmAK457/ct8=">AAAGP3icfVRLb9NAEHaggZLyaOHIZUVU1KJSJQUJDlSqRCulB9TSpg8pDtF6PUlM/cp6nddq/xT/gJ+B+AHcEFduzDppk6yBlWzPzvftzOzMeJzY9xJRqXwr3Lq9VLxzd/leaeX+g4ePVtcenydRyhmcsciP+KVDE/C9EM6EJ3y4jDnQwPHhwrl6r/GLPvDEi8K6GMXQDGgn9NoeowJVrdX0ObEDL2xJO+ZcEbuXUpfYgm/oPbnANyMXn+qbhLwkru1HHRdEhm1u2T1NtgUMhUxS5zMwQUSkJiZSctiKCRKBQU9/+bXcR6C1Wq5sV7JF8kJ1KpSt6TpurS19sd2IpQGEgvk0SRrVSiyaknLhMR9UyU4TiCm7oh1ooBjSAJKmzPKjyDpqXNKOOD6hIJl2/oSkQZKMAgeZARXdxMS08l+Y6AbK1GkviSLZWsCQLKLITxZPaLt5dSMV7bdN6YVxKiBkk1u0Ux9zTHQliethQoU/QoEy7mEiCOtSTpnAepfWiekipuNIlRZcBPQKeBQFmCkaMvAxgBAGLAoCGrrS9kJXSVufdRx5qAx0AF6nK64JbTkwCcPhDBya4Gg0A0cmOB7PwHHO7+A/PpM4nKKM+vI0DwsOMMeom4xabQ6tmShwjg0l7T7lECeeH4UGwcVfds7AvmnAhTb0tAGmOxkbMdw1XfSQo+TuHMVgHAzjWVkOTA8nJzPwROmCz6Ms6uv4T71OQFtc5UB2AzKzJtgpfWy47P7T9FdM53qISPsogE7eOo6SG8w0fnq8P4s6q9o+4H/O4QOqjmLgFP2+kDblGBq2FX7tLS39lShxgClZNwPA6OksdmrGPhluKhNsLRnh64btUpHvVp0V1ag25WQW4lZtlKubCqdc1ZxpeeF8Z7v6anvn4+vy3rvpvFu2nlrPrA2rar2x9qyadWydWcz6XigUSoWV4tfij+LP4q8J9VZheuaJtbCKv/8AopJGIA==</latexit>⌦r<latexit sha1_base64="O3mXawymR6ta5/qBZpvtZhyYxqo=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3axWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pn0qH7GXTL0fHc32B+8KOD3UL1tGu7gOdcwid07ecgKeZ9aSIqURqOFb6jV876K9FEGmUc+wJQPS20U197KroxuOahETnLk+8Gtkd1eVpdVWwzbGFaGGmDn3ajFr6weMuF/p1WNk63NsPXm1tf3tR23s/uuxXyjDwnGyQkb8kOaZADckIY6ZFv5Dv5Uflc0RVTsdfUW0uzPU/Jwqp8/Q2ylRvE</latexit><latexit sha1_base64="sxM8U0tIxERXSq4P3fTrZOgvV70=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3mxWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pj2UZ0P4NuOXou2Q3mBz862C1UT7u2C3jOJXxC134OkmLelyaiEqXhWOE7euWsvxJNpFHGsS8A1dNCO/W1p6Ibg2seGpGzPPluYHtUl6fVVcU2wxamhZE2+Gk3auELi7dc6N9pZeN0azN8vbn15U1t5/3svlshz8hzskFC8pbskAY5ICeEkR75Rr6TH5XPFV0xFXtNvbU02/OULKzK198DjxvT</latexit>Tr(⌦r W ⌦c W T )d log det(⌦r ),subject to uIp

Define the constraint set: C := {A 2 Sp++ | uIp A
function(0
If A 2 C
IC (A) =1 Otherwise<latexit sha1_base64="2kXY2aOtT325DixfBQ243M1PZ3Q=">AAAGDnicfVRLbxMxEHZLAyW8WjhyWRFVKhRVSUGCA5VaNZXSA2rpEykbIq/XSazu2q7XeVr7H/gdHDlwQ1w5ci2/hvE2bRIvYCnK7HyfZz7PjB3IiCW6XL6cm7+1ULh9Z/Fu8d79Bw8fLS0/Pk1EVxF6QkQk1McAJzRinJ5opiP6USqK4yCiZ8H5jsXPelQlTPBjPZS0EeM2Zy1GsAZXc6nqx1h3CI7MTuqHtEUvPN9s+4x7/tFBtWnW1tJP0o9Z6HX3mtKH2AQo29dGzzrT5lKpvF7Olpc3KmOjhMbroLm88NUPBenGlGsS4SSpV8pSNwxWmpGIpkW/m1CJyTlu0zqYHMc0aZjsuKm3Ap7QawkFP669zDu9w+A4SYZxAEx7uMTFrPNfmO7EqeuzWZLUy9YMBmQtRJTM7rBx8+56V7feNgzjsqspJ1enaHUjTwvPNsYLGZRUR0MwMFEMCuGRDlaYaGhfccVzU0g8EmlxJkWMz6kSIoZKYU5oBAI47RMRx5iHBpoapibrdxCYvdRB+5S1O/qa0DJ9lzAYTMCBCw6HE3DogqPRBBzl8vb/kzORfIzaET3Kw1pROsU4dhm12hRac1GqFAyU8XtYUZmwSHCHEMINnApQdQNkN8YGIHaSYRD5ppviAjip2ZyiOIzdgZy0ZdfNcHg4AQ9T2/BplIie1X/E2jFuqjQHkhuQuD2BSenBwGXnH5e/7CaXyqL7MW3no0tFbjA3ODwdE9VZ16oU7rmi78G1L6nCkPeF8bECaTBW8O+/tNZficbXIOPYFQDq8UQ7drVHoh1S2zwwfGs58u3AdrDOT6utSlqvNCAtHWgDn+lqqfLcvnIV903LG6cb65VX6xsfXpe23o3fu0X0FD1Dq6iC3qAtVEMH6AQR9AX9Qpfod+Fz4Vvhe+HHFXV+brznCZpZhZ9/AELrNdI=</latexit>

⌦r

vIp

vIp } and indicator<latexit sha1_base64="8Jyrg7SNPq/yVM4bLaX3HJEOC6w=">AAAGPnicfVTNbhMxEN5AA234a+HIxSIqahGqkoIEByq1aiulB9TSfykbRV7vbGJ1d721nV/LL8UT8BrwANwQV47YaegmXsDSamfn+zwznvnWQRZTIWu1r6U7dxfK9+4vLlUePHz0+MnyytNzwXqcwBlhMeOXARYQ0xTOJJUxXGYccBLEcBFc7Vr8og9cUJaeylEGrQR3UhpRgqVxtZeln2DZDQJ1oNtqYhMcq12t13bW0RbyA+jQVBGTQehKDb1EvoShVOggQhrt+DSd2YN8v2I8kRzd8g5lF/iACtAVH9JwGqi9XK1t1CYLFY361Kh603XUXln47IeM9BJIJYmxEM16LZMthbmkJLbBewIyTK5wB5rGTHECoqUm7dFo1XhCFDFunlSiiXd2h8KJEKMkMEx7GOFi1vkvTHYT7fpsFqHRZM1hhiwZi8X8Dhu36G72ZPS+pWia9SSk5OYUUS9GkiE7SBRSDkTGI2NgwqlpBCJdzDGRZtyVVeSmyPCY6cpcigRfAWcsMZ3CKYHYFJDCgLAkwWZWZpShVrk+HHQAtNOVfwiRGriE4TAHhy44GuXgyAXH4xwcF/IO/pNTZKnOZXxShCUHmGGcuoxGYwZtuChwbgSl/D7mkAkas9QhhOaPnQmw5wYIIYJrG4BYJRshpltuimvD0WprhuIw9odZPpZ9N8PxcQ4eazvwWZSwvq3/hHYS3Oa6AJJbkLgzMUrpG8FNzj9tf81NnnGLHibQKUbPOLnF3OAnR3t51ZOp7YH5zzl8NK7DDDg2eV8pH3NTmpGVefuvrfVXovKlKePULcBUj/PasVt7zDoh2OEZw7eWU74VbBfLolptV3Sz3lI3V5751GvV+rq95erunVY0zjc36m82Nj+9rW5/mN53i95z74W35tW9d9621/COvDOPeN9KXmmpVCl/KX8v/yj/vKHeKU33PPPmVvnXb7N3RfM=</latexit>

47

Learning with Adaptive Regularization
For fixed W and ⌦c, consider the optimization over ⌦r:<latexit sha1_base64="6T3asusjEPCEK52gk0msQW2zayM=">AAAFyXicfVRbb9MwFPbGCqPcNnjkJaKaNBCamoEED0yaxCYVIbT7RWqqyXFO26iOndnuNcoTj7zCH+Hf8G847srSOoClqifn+3y+L8cnDlMea1Ov/1pavrNSuXtv9X71wcNHj5+srT8917KvGJwxyaW6DKkGHgs4M7HhcJkqoEnI4SLsfbT4xQCUjqU4NeMUWgntiLgdM2owdXRxtVarb9WnyysH/iyokdk6vFpf+RlEkvUTEIZxqnXTr6emlVFlYsYhrwZ9DSllPdqBJoaCJqBb2dRp7m1gJvLaUuFPGG+and+R0UTrcRIiM6Gmq13MJv+FmW6SuzmronNvuhYwJBspuV7cYeuW082+ab9vZbFI+wYEu3mLdp97Rnq2p14UK2CGjzGgTMXYCI91qaLMYOerG54rkdKJzKsLEgntgZIywU5RwYCjAQFDJpOEiigLYhHlWWD3hmH2KXfQIcSdrvlDaGdDlzAaFeDIBcfjAhy74GRSgJOS7vA/mjoVM5RRnp2UYaMA5hinLqPRmEMbLgpK4UBlwYAqSHXMpXAIEX48cwX23AIRtOHaFmB2knEQxY4rcY2cPNuZoziM/VFaHMu+q3B8XIDHuT3weZTJgfV/EncSeqXyEshuQeaeCU7KAAdu+v6z9tdd8VRZ9CCBTrl6qtgt5hY/OdwrXE9PbQ/wO1fwBVMHKSiKuq+ygCq0hmOF/8FrG/2VmAUGbZy6BtA9LbxT1zuXnQjs4WEQ2Mixbwe2S015Wm1X8qbfQlkYmQwf882a/zLHW85377RycL695b/Z2j56W9v9MLvvVslz8oJsEp+8I7ukQQ7JGWEEyDfynfyofK5cV0aVyQ11eWm25xlZWJWvvwFBPxpc</latexit>

min<latexit sha1_base64="26QU4fsGr7YU80QGsmAK457/ct8=">AAAGP3icfVRLb9NAEHaggZLyaOHIZUVU1KJSJQUJDlSqRCulB9TSpg8pDtF6PUlM/cp6nddq/xT/gJ+B+AHcEFduzDppk6yBlWzPzvftzOzMeJzY9xJRqXwr3Lq9VLxzd/leaeX+g4ePVtcenydRyhmcsciP+KVDE/C9EM6EJ3y4jDnQwPHhwrl6r/GLPvDEi8K6GMXQDGgn9NoeowJVrdX0ObEDL2xJO+ZcEbuXUpfYgm/oPbnANyMXn+qbhLwkru1HHRdEhm1u2T1NtgUMhUxS5zMwQUSkJiZSctiKCRKBQU9/+bXcR6C1Wq5sV7JF8kJ1KpSt6TpurS19sd2IpQGEgvk0SRrVSiyaknLhMR9UyU4TiCm7oh1ooBjSAJKmzPKjyDpqXNKOOD6hIJl2/oSkQZKMAgeZARXdxMS08l+Y6AbK1GkviSLZWsCQLKLITxZPaLt5dSMV7bdN6YVxKiBkk1u0Ux9zTHQliethQoU/QoEy7mEiCOtSTpnAepfWiekipuNIlRZcBPQKeBQFmCkaMvAxgBAGLAoCGrrS9kJXSVufdRx5qAx0AF6nK64JbTkwCcPhDBya4Gg0A0cmOB7PwHHO7+A/PpM4nKKM+vI0DwsOMMeom4xabQ6tmShwjg0l7T7lECeeH4UGwcVfds7AvmnAhTb0tAGmOxkbMdw1XfSQo+TuHMVgHAzjWVkOTA8nJzPwROmCz6Ms6uv4T71OQFtc5UB2AzKzJtgpfWy47P7T9FdM53qISPsogE7eOo6SG8w0fnq8P4s6q9o+4H/O4QOqjmLgFP2+kDblGBq2FX7tLS39lShxgClZNwPA6OksdmrGPhluKhNsLRnh64btUpHvVp0V1ag25WQW4lZtlKubCqdc1ZxpeeF8Z7v6anvn4+vy3rvpvFu2nlrPrA2rar2x9qyadWydWcz6XigUSoWV4tfij+LP4q8J9VZheuaJtbCKv/8AopJGIA==</latexit>⌦r<latexit sha1_base64="O3mXawymR6ta5/qBZpvtZhyYxqo=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3axWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pn0qH7GXTL0fHc32B+8KOD3UL1tGu7gOdcwid07ecgKeZ9aSIqURqOFb6jV876K9FEGmUc+wJQPS20U197KroxuOahETnLk+8Gtkd1eVpdVWwzbGFaGGmDn3ajFr6weMuF/p1WNk63NsPXm1tf3tR23s/uuxXyjDwnGyQkb8kOaZADckIY6ZFv5Dv5Uflc0RVTsdfUW0uzPU/Jwqp8/Q2ylRvE</latexit><latexit sha1_base64="sxM8U0tIxERXSq4P3fTrZOgvV70=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3mxWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pj2UZ0P4NuOXou2Q3mBz862C1UT7u2C3jOJXxC134OkmLelyaiEqXhWOE7euWsvxJNpFHGsS8A1dNCO/W1p6Ibg2seGpGzPPluYHtUl6fVVcU2wxamhZE2+Gk3auELi7dc6N9pZeN0azN8vbn15U1t5/3svlshz8hzskFC8pbskAY5ICeEkR75Rr6TH5XPFV0xFXtNvbU02/OULKzK198DjxvT</latexit>Tr(⌦r W ⌦c W T )d log det(⌦r ),subject to uIp

Define the constraint set: C := {A 2 Sp++ | uIp A
function(0
If A 2 C
IC (A) =1 Otherwise<latexit sha1_base64="2kXY2aOtT325DixfBQ243M1PZ3Q=">AAAGDnicfVRLbxMxEHZLAyW8WjhyWRFVKhRVSUGCA5VaNZXSA2rpEykbIq/XSazu2q7XeVr7H/gdHDlwQ1w5ci2/hvE2bRIvYCnK7HyfZz7PjB3IiCW6XL6cm7+1ULh9Z/Fu8d79Bw8fLS0/Pk1EVxF6QkQk1McAJzRinJ5opiP6USqK4yCiZ8H5jsXPelQlTPBjPZS0EeM2Zy1GsAZXc6nqx1h3CI7MTuqHtEUvPN9s+4x7/tFBtWnW1tJP0o9Z6HX3mtKH2AQo29dGzzrT5lKpvF7Olpc3KmOjhMbroLm88NUPBenGlGsS4SSpV8pSNwxWmpGIpkW/m1CJyTlu0zqYHMc0aZjsuKm3Ap7QawkFP669zDu9w+A4SYZxAEx7uMTFrPNfmO7EqeuzWZLUy9YMBmQtRJTM7rBx8+56V7feNgzjsqspJ1enaHUjTwvPNsYLGZRUR0MwMFEMCuGRDlaYaGhfccVzU0g8EmlxJkWMz6kSIoZKYU5oBAI47RMRx5iHBpoapibrdxCYvdRB+5S1O/qa0DJ9lzAYTMCBCw6HE3DogqPRBBzl8vb/kzORfIzaET3Kw1pROsU4dhm12hRac1GqFAyU8XtYUZmwSHCHEMINnApQdQNkN8YGIHaSYRD5ppviAjip2ZyiOIzdgZy0ZdfNcHg4AQ9T2/BplIie1X/E2jFuqjQHkhuQuD2BSenBwGXnH5e/7CaXyqL7MW3no0tFbjA3ODwdE9VZ16oU7rmi78G1L6nCkPeF8bECaTBW8O+/tNZficbXIOPYFQDq8UQ7drVHoh1S2zwwfGs58u3AdrDOT6utSlqvNCAtHWgDn+lqqfLcvnIV903LG6cb65VX6xsfXpe23o3fu0X0FD1Dq6iC3qAtVEMH6AQR9AX9Qpfod+Fz4Vvhe+HHFXV+brznCZpZhZ9/AELrNdI=</latexit>

⌦r

vIp

vIp } and indicator<latexit sha1_base64="8Jyrg7SNPq/yVM4bLaX3HJEOC6w=">AAAGPnicfVTNbhMxEN5AA234a+HIxSIqahGqkoIEByq1aiulB9TSfykbRV7vbGJ1d721nV/LL8UT8BrwANwQV47YaegmXsDSamfn+zwznvnWQRZTIWu1r6U7dxfK9+4vLlUePHz0+MnyytNzwXqcwBlhMeOXARYQ0xTOJJUxXGYccBLEcBFc7Vr8og9cUJaeylEGrQR3UhpRgqVxtZeln2DZDQJ1oNtqYhMcq12t13bW0RbyA+jQVBGTQehKDb1EvoShVOggQhrt+DSd2YN8v2I8kRzd8g5lF/iACtAVH9JwGqi9XK1t1CYLFY361Kh603XUXln47IeM9BJIJYmxEM16LZMthbmkJLbBewIyTK5wB5rGTHECoqUm7dFo1XhCFDFunlSiiXd2h8KJEKMkMEx7GOFi1vkvTHYT7fpsFqHRZM1hhiwZi8X8Dhu36G72ZPS+pWia9SSk5OYUUS9GkiE7SBRSDkTGI2NgwqlpBCJdzDGRZtyVVeSmyPCY6cpcigRfAWcsMZ3CKYHYFJDCgLAkwWZWZpShVrk+HHQAtNOVfwiRGriE4TAHhy44GuXgyAXH4xwcF/IO/pNTZKnOZXxShCUHmGGcuoxGYwZtuChwbgSl/D7mkAkas9QhhOaPnQmw5wYIIYJrG4BYJRshpltuimvD0WprhuIw9odZPpZ9N8PxcQ4eazvwWZSwvq3/hHYS3Oa6AJJbkLgzMUrpG8FNzj9tf81NnnGLHibQKUbPOLnF3OAnR3t51ZOp7YH5zzl8NK7DDDg2eV8pH3NTmpGVefuvrfVXovKlKePULcBUj/PasVt7zDoh2OEZw7eWU74VbBfLolptV3Sz3lI3V5751GvV+rq95erunVY0zjc36m82Nj+9rW5/mN53i95z74W35tW9d9621/COvDOPeN9KXmmpVCl/KX8v/yj/vKHeKU33PPPmVvnXb7N3RfM=</latexit>

Now we have an equivalent unconstrained convex optimization problem:
min<latexit sha1_base64="lRKJ7aSdOiD/NO/jNNpesr1LkOE=">AAAGJHicfVTNbtNAEN6WBkr4a+HIZUVU1PJTJQUJDlSq1FYKB9TSpk1FHKLNepNYtXfd9eZ3ta/DA/AEPAA3xIELR67wCMwmoU7WwEq2x/N98+OZ8TTjMEhUsfhtYfHKUu7qteXr+Rs3b92+s7J69zQRXUnZCRWhkGdNkrAw4OxEBSpkZ7FkJGqGrNo837V4tcdkEgheUcOY1SPS5kEroESBqrHy7iH2ooA3tBdLabB30SU+9pRct++4CneKq+8rGxg/xb4XirbP1BjbwI/BkqhOs6lfG7C3MiWh3jVmQmisFIqbxfHBWaE0FQpoeg4bq0sfPV/QbsS4oiFJklqpGKu6JlIFNGQm73UTFhN6TtqsBiInEUvqelwEg9dA4+OWkHBxhcfaWQtNoiQZRk1g2lQTF7PKf2GqExlXZ6MkBo/PHAZkJUSYzFtYv1l1rataL+s64HFXMU4nX9HqhlgJbNuF/UAyqsIhCITKAAqBaYdIQhU0Nb+G3RAxGQmTnwsRkXMmhYigUoRTFkICnPWpiCLCfe0F3Dc67aSD9lnQ7qg/hJbuu4TBIAUHLjgcpuDQBUejFBxl4vb/EzOJuUkH7jgLK8nYDKPiMsrlGbTsokxKGCjt9YhkcRKEgjsEH/7LGQd7rgOftdiFdUDtJMMg8m03xAVwjN6eoTiM/UGctmXfjXB0lIJHxjZ8FqWiZ/M/DtoRaUiTAeklSN2ewKT0YODG3z8tf9ENbjeF9g4i1s56h31xibnOjw/30qzHXdtj8J9L9gZUBzGTBOI+0h6RkBqMFTy9J1b6K1HDljK64iYA2ZM0d+LmPtlgZix4VnLStwPbISo7rbYqplaqQ1g2UBpezXqhtGFgy5XcnZYVTrc2S882t94+L+y8mu67ZXQfPUDrqIReoB1URofoBFH0Cf1AP9Gv3Ifc59yX3NcJdXFhanMPzZ3c99+EuT10</latexit>⌦r

Tr(⌦r W ⌦c W T )d log det(⌦r ) + IC (⌦r )48

Learning with Adaptive Regularization
For fixed W and ⌦c, consider the optimization over ⌦r:<latexit sha1_base64="6T3asusjEPCEK52gk0msQW2zayM=">AAAFyXicfVRbb9MwFPbGCqPcNnjkJaKaNBCamoEED0yaxCYVIbT7RWqqyXFO26iOndnuNcoTj7zCH+Hf8G847srSOoClqifn+3y+L8cnDlMea1Ov/1pavrNSuXtv9X71wcNHj5+srT8917KvGJwxyaW6DKkGHgs4M7HhcJkqoEnI4SLsfbT4xQCUjqU4NeMUWgntiLgdM2owdXRxtVarb9WnyysH/iyokdk6vFpf+RlEkvUTEIZxqnXTr6emlVFlYsYhrwZ9DSllPdqBJoaCJqBb2dRp7m1gJvLaUuFPGG+and+R0UTrcRIiM6Gmq13MJv+FmW6SuzmronNvuhYwJBspuV7cYeuW082+ab9vZbFI+wYEu3mLdp97Rnq2p14UK2CGjzGgTMXYCI91qaLMYOerG54rkdKJzKsLEgntgZIywU5RwYCjAQFDJpOEiigLYhHlWWD3hmH2KXfQIcSdrvlDaGdDlzAaFeDIBcfjAhy74GRSgJOS7vA/mjoVM5RRnp2UYaMA5hinLqPRmEMbLgpK4UBlwYAqSHXMpXAIEX48cwX23AIRtOHaFmB2knEQxY4rcY2cPNuZoziM/VFaHMu+q3B8XIDHuT3weZTJgfV/EncSeqXyEshuQeaeCU7KAAdu+v6z9tdd8VRZ9CCBTrl6qtgt5hY/OdwrXE9PbQ/wO1fwBVMHKSiKuq+ygCq0hmOF/8FrG/2VmAUGbZy6BtA9LbxT1zuXnQjs4WEQ2Mixbwe2S015Wm1X8qbfQlkYmQwf882a/zLHW85377RycL695b/Z2j56W9v9MLvvVslz8oJsEp+8I7ukQQ7JGWEEyDfynfyofK5cV0aVyQ11eWm25xlZWJWvvwFBPxpc</latexit>

min<latexit sha1_base64="26QU4fsGr7YU80QGsmAK457/ct8=">AAAGP3icfVRLb9NAEHaggZLyaOHIZUVU1KJSJQUJDlSqRCulB9TSpg8pDtF6PUlM/cp6nddq/xT/gJ+B+AHcEFduzDppk6yBlWzPzvftzOzMeJzY9xJRqXwr3Lq9VLxzd/leaeX+g4ePVtcenydRyhmcsciP+KVDE/C9EM6EJ3y4jDnQwPHhwrl6r/GLPvDEi8K6GMXQDGgn9NoeowJVrdX0ObEDL2xJO+ZcEbuXUpfYgm/oPbnANyMXn+qbhLwkru1HHRdEhm1u2T1NtgUMhUxS5zMwQUSkJiZSctiKCRKBQU9/+bXcR6C1Wq5sV7JF8kJ1KpSt6TpurS19sd2IpQGEgvk0SRrVSiyaknLhMR9UyU4TiCm7oh1ooBjSAJKmzPKjyDpqXNKOOD6hIJl2/oSkQZKMAgeZARXdxMS08l+Y6AbK1GkviSLZWsCQLKLITxZPaLt5dSMV7bdN6YVxKiBkk1u0Ux9zTHQliethQoU/QoEy7mEiCOtSTpnAepfWiekipuNIlRZcBPQKeBQFmCkaMvAxgBAGLAoCGrrS9kJXSVufdRx5qAx0AF6nK64JbTkwCcPhDBya4Gg0A0cmOB7PwHHO7+A/PpM4nKKM+vI0DwsOMMeom4xabQ6tmShwjg0l7T7lECeeH4UGwcVfds7AvmnAhTb0tAGmOxkbMdw1XfSQo+TuHMVgHAzjWVkOTA8nJzPwROmCz6Ms6uv4T71OQFtc5UB2AzKzJtgpfWy47P7T9FdM53qISPsogE7eOo6SG8w0fnq8P4s6q9o+4H/O4QOqjmLgFP2+kDblGBq2FX7tLS39lShxgClZNwPA6OksdmrGPhluKhNsLRnh64btUpHvVp0V1ag25WQW4lZtlKubCqdc1ZxpeeF8Z7v6anvn4+vy3rvpvFu2nlrPrA2rar2x9qyadWydWcz6XigUSoWV4tfij+LP4q8J9VZheuaJtbCKv/8AopJGIA==</latexit>⌦r<latexit sha1_base64="O3mXawymR6ta5/qBZpvtZhyYxqo=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3axWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pn0qH7GXTL0fHc32B+8KOD3UL1tGu7gOdcwid07ecgKeZ9aSIqURqOFb6jV876K9FEGmUc+wJQPS20U197KroxuOahETnLk+8Gtkd1eVpdVWwzbGFaGGmDn3ajFr6weMuF/p1WNk63NsPXm1tf3tR23s/uuxXyjDwnGyQkb8kOaZADckIY6ZFv5Dv5Uflc0RVTsdfUW0uzPU/Jwqp8/Q2ylRvE</latexit><latexit sha1_base64="sxM8U0tIxERXSq4P3fTrZOgvV70=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3mxWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pj2UZ0P4NuOXou2Q3mBz862C1UT7u2C3jOJXxC134OkmLelyaiEqXhWOE7euWsvxJNpFHGsS8A1dNCO/W1p6Ibg2seGpGzPPluYHtUl6fVVcU2wxamhZE2+Gk3auELi7dc6N9pZeN0azN8vbn15U1t5/3svlshz8hzskFC8pbskAY5ICeEkR75Rr6TH5XPFV0xFXtNvbU02/OULKzK198DjxvT</latexit>Tr(⌦r W ⌦c W T )d log det(⌦r ),subject to uIp

Define the constraint set: C := {A 2 Sp++ | uIp A
function(0
If A 2 C
IC (A) =1 Otherwise<latexit sha1_base64="2kXY2aOtT325DixfBQ243M1PZ3Q=">AAAGDnicfVRLbxMxEHZLAyW8WjhyWRFVKhRVSUGCA5VaNZXSA2rpEykbIq/XSazu2q7XeVr7H/gdHDlwQ1w5ci2/hvE2bRIvYCnK7HyfZz7PjB3IiCW6XL6cm7+1ULh9Z/Fu8d79Bw8fLS0/Pk1EVxF6QkQk1McAJzRinJ5opiP6USqK4yCiZ8H5jsXPelQlTPBjPZS0EeM2Zy1GsAZXc6nqx1h3CI7MTuqHtEUvPN9s+4x7/tFBtWnW1tJP0o9Z6HX3mtKH2AQo29dGzzrT5lKpvF7Olpc3KmOjhMbroLm88NUPBenGlGsS4SSpV8pSNwxWmpGIpkW/m1CJyTlu0zqYHMc0aZjsuKm3Ap7QawkFP669zDu9w+A4SYZxAEx7uMTFrPNfmO7EqeuzWZLUy9YMBmQtRJTM7rBx8+56V7feNgzjsqspJ1enaHUjTwvPNsYLGZRUR0MwMFEMCuGRDlaYaGhfccVzU0g8EmlxJkWMz6kSIoZKYU5oBAI47RMRx5iHBpoapibrdxCYvdRB+5S1O/qa0DJ9lzAYTMCBCw6HE3DogqPRBBzl8vb/kzORfIzaET3Kw1pROsU4dhm12hRac1GqFAyU8XtYUZmwSHCHEMINnApQdQNkN8YGIHaSYRD5ppviAjip2ZyiOIzdgZy0ZdfNcHg4AQ9T2/BplIie1X/E2jFuqjQHkhuQuD2BSenBwGXnH5e/7CaXyqL7MW3no0tFbjA3ODwdE9VZ16oU7rmi78G1L6nCkPeF8bECaTBW8O+/tNZficbXIOPYFQDq8UQ7drVHoh1S2zwwfGs58u3AdrDOT6utSlqvNCAtHWgDn+lqqfLcvnIV903LG6cb65VX6xsfXpe23o3fu0X0FD1Dq6iC3qAtVEMH6AQR9AX9Qpfod+Fz4Vvhe+HHFXV+brznCZpZhZ9/AELrNdI=</latexit>

⌦r

vIp

vIp } and indicator<latexit sha1_base64="8Jyrg7SNPq/yVM4bLaX3HJEOC6w=">AAAGPnicfVTNbhMxEN5AA234a+HIxSIqahGqkoIEByq1aiulB9TSfykbRV7vbGJ1d721nV/LL8UT8BrwANwQV47YaegmXsDSamfn+zwznvnWQRZTIWu1r6U7dxfK9+4vLlUePHz0+MnyytNzwXqcwBlhMeOXARYQ0xTOJJUxXGYccBLEcBFc7Vr8og9cUJaeylEGrQR3UhpRgqVxtZeln2DZDQJ1oNtqYhMcq12t13bW0RbyA+jQVBGTQehKDb1EvoShVOggQhrt+DSd2YN8v2I8kRzd8g5lF/iACtAVH9JwGqi9XK1t1CYLFY361Kh603XUXln47IeM9BJIJYmxEM16LZMthbmkJLbBewIyTK5wB5rGTHECoqUm7dFo1XhCFDFunlSiiXd2h8KJEKMkMEx7GOFi1vkvTHYT7fpsFqHRZM1hhiwZi8X8Dhu36G72ZPS+pWia9SSk5OYUUS9GkiE7SBRSDkTGI2NgwqlpBCJdzDGRZtyVVeSmyPCY6cpcigRfAWcsMZ3CKYHYFJDCgLAkwWZWZpShVrk+HHQAtNOVfwiRGriE4TAHhy44GuXgyAXH4xwcF/IO/pNTZKnOZXxShCUHmGGcuoxGYwZtuChwbgSl/D7mkAkas9QhhOaPnQmw5wYIIYJrG4BYJRshpltuimvD0WprhuIw9odZPpZ9N8PxcQ4eazvwWZSwvq3/hHYS3Oa6AJJbkLgzMUrpG8FNzj9tf81NnnGLHibQKUbPOLnF3OAnR3t51ZOp7YH5zzl8NK7DDDg2eV8pH3NTmpGVefuvrfVXovKlKePULcBUj/PasVt7zDoh2OEZw7eWU74VbBfLolptV3Sz3lI3V5751GvV+rq95erunVY0zjc36m82Nj+9rW5/mN53i95z74W35tW9d9621/COvDOPeN9KXmmpVCl/KX8v/yj/vKHeKU33PPPmVvnXb7N3RfM=</latexit>

Now we have an equivalent unconstrained convex optimization problem:
min<latexit sha1_base64="lRKJ7aSdOiD/NO/jNNpesr1LkOE=">AAAGJHicfVTNbtNAEN6WBkr4a+HIZUVU1PJTJQUJDlSq1FYKB9TSpk1FHKLNepNYtXfd9eZ3ta/DA/AEPAA3xIELR67wCMwmoU7WwEq2x/N98+OZ8TTjMEhUsfhtYfHKUu7qteXr+Rs3b92+s7J69zQRXUnZCRWhkGdNkrAw4OxEBSpkZ7FkJGqGrNo837V4tcdkEgheUcOY1SPS5kEroESBqrHy7iH2ooA3tBdLabB30SU+9pRct++4CneKq+8rGxg/xb4XirbP1BjbwI/BkqhOs6lfG7C3MiWh3jVmQmisFIqbxfHBWaE0FQpoeg4bq0sfPV/QbsS4oiFJklqpGKu6JlIFNGQm73UTFhN6TtqsBiInEUvqelwEg9dA4+OWkHBxhcfaWQtNoiQZRk1g2lQTF7PKf2GqExlXZ6MkBo/PHAZkJUSYzFtYv1l1rataL+s64HFXMU4nX9HqhlgJbNuF/UAyqsIhCITKAAqBaYdIQhU0Nb+G3RAxGQmTnwsRkXMmhYigUoRTFkICnPWpiCLCfe0F3Dc67aSD9lnQ7qg/hJbuu4TBIAUHLjgcpuDQBUejFBxl4vb/EzOJuUkH7jgLK8nYDKPiMsrlGbTsokxKGCjt9YhkcRKEgjsEH/7LGQd7rgOftdiFdUDtJMMg8m03xAVwjN6eoTiM/UGctmXfjXB0lIJHxjZ8FqWiZ/M/DtoRaUiTAeklSN2ewKT0YODG3z8tf9ENbjeF9g4i1s56h31xibnOjw/30qzHXdtj8J9L9gZUBzGTBOI+0h6RkBqMFTy9J1b6K1HDljK64iYA2ZM0d+LmPtlgZix4VnLStwPbISo7rbYqplaqQ1g2UBpezXqhtGFgy5XcnZYVTrc2S882t94+L+y8mu67ZXQfPUDrqIReoB1URofoBFH0Cf1AP9Gv3Ifc59yX3NcJdXFhanMPzZ3c99+EuT10</latexit>⌦r

Tr(⌦r W ⌦c W T )d log det(⌦r ) + IC (⌦r )The first-order optimality condition tells us:
02@<latexit sha1_base64="Ue9g0E5iTRMt4sX51DJZfrgVKc0=">AAAGcXicfVTbThsxEN1QSGlaWmifqr5YREhJKTQLldoHIiEBUvrQQiEQpGyIHK83WbE3vM7V8vf0vX/T7+gPdLwJbOKltZRkMud4Zuw5nk7kuTGvVH7nlp4sr+Sfrj4rPH+x9vLV+sbrqzjsM0IvSeiF7LqDY+q5Ab3kLvfodcQo9jsebXRujxTeGFAWu2FQ5+OItnzcDVzHJZiDq73+s4IsN0BWhBl3sWd51OEly2GYCFMKW1qclayIMdSAb4IaN/UyQjvI8sKuTXkCldE2snzMe52O+CrbIrEJ9sSRlFOCxdxuj5dRNY3y0VZRALwRO6a8j6B2fX80RHu9WNmtJAtlDXNmFI3ZOmtvLP+y7JD0fRpw4uE4bpqViLeEOifxqCxY/ZhGmNziLm2CGWCfxi2R3KhEW+CxkRMy+AQcJd75HQL7cTz2O8BUpcY6ppz/wnjPl7pPZYklStYCBmQehl68uEPFzbqbfe58aQk3iPqcBmR6CqfvIR4i1Xtku4wS7o3BwIS5cBGI9DD0moNCCltITxHhSSgLCyl8fEtZGPpwUzgg1IMCAjokoe/jwBYgJVuKVAwaOqRKB/cERwx1wmiUgiMdHI9TcKyDk0kKTjJ5h//JGUeBTAV3kYU5o3SOUdcZtdocWtNRyhgISlgDzGgUu14YaAQbHvlcgGM9gE0deqcCEKVkEGJQ1VPcAUeK6hxFY5yMorQtJ3qG8/MUPJeq4fMoCQeq/gu36+M2kxmQPIBE7wkoZQCCS84/u/6KnhweN6CnPu1mo8OseMD04Bdnx2nVSdeOKbxzRr+B6zSiDEPe98LCDEoDWcGv9UFZjxIFzDkp6noBUD1Oa8d67dMhKBPDUpZWvhJsD/OsWtWtyKbZgrR0xAX8laWiWZYw5Ux9pmWNq71dc39378en4uHBbN6tGu+MTaNkmMZn49CoGWfGpUFya7n93EGuuvIn/zaP8ptT6lJutueNsbDy238BKwZT0w==</latexit>✓

1
Tr(⌦r W ⌦c W T )d

log det(⌦r ) + IC (⌦r )◆= W ⌦c W T /d⌦r 1 + NC (⌦r )49

Learning with Adaptive Regularization
For fixed W and ⌦c, consider the optimization over ⌦r:<latexit sha1_base64="6T3asusjEPCEK52gk0msQW2zayM=">AAAFyXicfVRbb9MwFPbGCqPcNnjkJaKaNBCamoEED0yaxCYVIbT7RWqqyXFO26iOndnuNcoTj7zCH+Hf8G847srSOoClqifn+3y+L8cnDlMea1Ov/1pavrNSuXtv9X71wcNHj5+srT8917KvGJwxyaW6DKkGHgs4M7HhcJkqoEnI4SLsfbT4xQCUjqU4NeMUWgntiLgdM2owdXRxtVarb9WnyysH/iyokdk6vFpf+RlEkvUTEIZxqnXTr6emlVFlYsYhrwZ9DSllPdqBJoaCJqBb2dRp7m1gJvLaUuFPGG+and+R0UTrcRIiM6Gmq13MJv+FmW6SuzmronNvuhYwJBspuV7cYeuW082+ab9vZbFI+wYEu3mLdp97Rnq2p14UK2CGjzGgTMXYCI91qaLMYOerG54rkdKJzKsLEgntgZIywU5RwYCjAQFDJpOEiigLYhHlWWD3hmH2KXfQIcSdrvlDaGdDlzAaFeDIBcfjAhy74GRSgJOS7vA/mjoVM5RRnp2UYaMA5hinLqPRmEMbLgpK4UBlwYAqSHXMpXAIEX48cwX23AIRtOHaFmB2knEQxY4rcY2cPNuZoziM/VFaHMu+q3B8XIDHuT3weZTJgfV/EncSeqXyEshuQeaeCU7KAAdu+v6z9tdd8VRZ9CCBTrl6qtgt5hY/OdwrXE9PbQ/wO1fwBVMHKSiKuq+ygCq0hmOF/8FrG/2VmAUGbZy6BtA9LbxT1zuXnQjs4WEQ2Mixbwe2S015Wm1X8qbfQlkYmQwf882a/zLHW85377RycL695b/Z2j56W9v9MLvvVslz8oJsEp+8I7ukQQ7JGWEEyDfynfyofK5cV0aVyQ11eWm25xlZWJWvvwFBPxpc</latexit>

min<latexit sha1_base64="26QU4fsGr7YU80QGsmAK457/ct8=">AAAGP3icfVRLb9NAEHaggZLyaOHIZUVU1KJSJQUJDlSqRCulB9TSpg8pDtF6PUlM/cp6nddq/xT/gJ+B+AHcEFduzDppk6yBlWzPzvftzOzMeJzY9xJRqXwr3Lq9VLxzd/leaeX+g4ePVtcenydRyhmcsciP+KVDE/C9EM6EJ3y4jDnQwPHhwrl6r/GLPvDEi8K6GMXQDGgn9NoeowJVrdX0ObEDL2xJO+ZcEbuXUpfYgm/oPbnANyMXn+qbhLwkru1HHRdEhm1u2T1NtgUMhUxS5zMwQUSkJiZSctiKCRKBQU9/+bXcR6C1Wq5sV7JF8kJ1KpSt6TpurS19sd2IpQGEgvk0SRrVSiyaknLhMR9UyU4TiCm7oh1ooBjSAJKmzPKjyDpqXNKOOD6hIJl2/oSkQZKMAgeZARXdxMS08l+Y6AbK1GkviSLZWsCQLKLITxZPaLt5dSMV7bdN6YVxKiBkk1u0Ux9zTHQliethQoU/QoEy7mEiCOtSTpnAepfWiekipuNIlRZcBPQKeBQFmCkaMvAxgBAGLAoCGrrS9kJXSVufdRx5qAx0AF6nK64JbTkwCcPhDBya4Gg0A0cmOB7PwHHO7+A/PpM4nKKM+vI0DwsOMMeom4xabQ6tmShwjg0l7T7lECeeH4UGwcVfds7AvmnAhTb0tAGmOxkbMdw1XfSQo+TuHMVgHAzjWVkOTA8nJzPwROmCz6Ms6uv4T71OQFtc5UB2AzKzJtgpfWy47P7T9FdM53qISPsogE7eOo6SG8w0fnq8P4s6q9o+4H/O4QOqjmLgFP2+kDblGBq2FX7tLS39lShxgClZNwPA6OksdmrGPhluKhNsLRnh64btUpHvVp0V1ag25WQW4lZtlKubCqdc1ZxpeeF8Z7v6anvn4+vy3rvpvFu2nlrPrA2rar2x9qyadWydWcz6XigUSoWV4tfij+LP4q8J9VZheuaJtbCKv/8AopJGIA==</latexit>⌦r<latexit sha1_base64="O3mXawymR6ta5/qBZpvtZhyYxqo=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3axWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pn0qH7GXTL0fHc32B+8KOD3UL1tGu7gOdcwid07ecgKeZ9aSIqURqOFb6jV876K9FEGmUc+wJQPS20U197KroxuOahETnLk+8Gtkd1eVpdVWwzbGFaGGmDn3ajFr6weMuF/p1WNk63NsPXm1tf3tR23s/uuxXyjDwnGyQkb8kOaZADckIY6ZFv5Dv5Uflc0RVTsdfUW0uzPU/Jwqp8/Q2ylRvE</latexit>Tr(⌦r W ⌦c W T )<latexit sha1_base64="sxM8U0tIxERXSq4P3fTrZOgvV70=">AAAFzHicfVRLbxMxEHahgRJeLRy5rIgqFYSqbEGCA5Uq0UrhAC19S9mocryTZNVde2s7T8tXjlzhd/Bv+DeM09BNvICl1c7O93nm25mx23maKF2v/1q6dXu5cufuyr3q/QcPHz1eXXtyqkRfMjhhIhXyvE0VpAmHE53oFM5zCTRrp3DWvvzg8LMBSJUIfqzHObQy2uVJJ2FUO1eUS3mxWqtv1qcrKBvhzKiR2Tq4WFv+GcWC9TPgmqVUqWZYz3XLUKkTloKtRn0FOWWXtAtNNDnNQLXMVKwN1tETBx0h8eE6mHrndxiaKTXO2sjMqO4pH3POf2G6l1nf57IoG0zXAoZkLUSqFne4uGV3s68771om4XlfA2fXf9Hpp4EWgStrECcSmE7HaFAmEyxEwHpUUqax+NX1wE+R04mw1YUUGb0EKUSGlaKcQYoCOAyZyDLKYxMlPLYmcnvbbfPReugQkm5P/yF0zNAnjEYFOPLB8bgAxz44mRTgpJR3+J+cKuczlNHUHJVhLQHmGMc+o9GYQxs+ClLiQJloQCXkKkkF9wgxnp+5ALt+gBg6cOUCMDfJOIh8209xhRxrtucoHmNvlBdt2fMzHB4W4KF1DZ9HmRg4/UdJN6MX0pZAdgMyvyc4KQMcuOn/z8pf95Pj2UZ0P4NuOXou2Q3mBz862C1UT7u2C3jOJXxC134OkmLelyaiEqXhWOE7euWsvxJNpFHGsS8A1dNCO/W1p6Ibg2seGpGzPPluYHtUl6fVVcU2wxamhZE2+Gk3auELi7dc6N9pZeN0azN8vbn15U1t5/3svlshz8hzskFC8pbskAY5ICeEkR75Rr6TH5XPFV0xFXtNvbU02/OULKzK198DjxvT</latexit>

d log det(⌦r ),subject to uIp

⌦r

vIp

The first-order optimality condition tells us:
02@<latexit sha1_base64="Ue9g0E5iTRMt4sX51DJZfrgVKc0=">AAAGcXicfVTbThsxEN1QSGlaWmifqr5YREhJKTQLldoHIiEBUvrQQiEQpGyIHK83WbE3vM7V8vf0vX/T7+gPdLwJbOKltZRkMud4Zuw5nk7kuTGvVH7nlp4sr+Sfrj4rPH+x9vLV+sbrqzjsM0IvSeiF7LqDY+q5Ab3kLvfodcQo9jsebXRujxTeGFAWu2FQ5+OItnzcDVzHJZiDq73+s4IsN0BWhBl3sWd51OEly2GYCFMKW1qclayIMdSAb4IaN/UyQjvI8sKuTXkCldE2snzMe52O+CrbIrEJ9sSRlFOCxdxuj5dRNY3y0VZRALwRO6a8j6B2fX80RHu9WNmtJAtlDXNmFI3ZOmtvLP+y7JD0fRpw4uE4bpqViLeEOifxqCxY/ZhGmNziLm2CGWCfxi2R3KhEW+CxkRMy+AQcJd75HQL7cTz2O8BUpcY6ppz/wnjPl7pPZYklStYCBmQehl68uEPFzbqbfe58aQk3iPqcBmR6CqfvIR4i1Xtku4wS7o3BwIS5cBGI9DD0moNCCltITxHhSSgLCyl8fEtZGPpwUzgg1IMCAjokoe/jwBYgJVuKVAwaOqRKB/cERwx1wmiUgiMdHI9TcKyDk0kKTjJ5h//JGUeBTAV3kYU5o3SOUdcZtdocWtNRyhgISlgDzGgUu14YaAQbHvlcgGM9gE0deqcCEKVkEGJQ1VPcAUeK6hxFY5yMorQtJ3qG8/MUPJeq4fMoCQeq/gu36+M2kxmQPIBE7wkoZQCCS84/u/6KnhweN6CnPu1mo8OseMD04Bdnx2nVSdeOKbxzRr+B6zSiDEPe98LCDEoDWcGv9UFZjxIFzDkp6noBUD1Oa8d67dMhKBPDUpZWvhJsD/OsWtWtyKbZgrR0xAX8laWiWZYw5Ux9pmWNq71dc39378en4uHBbN6tGu+MTaNkmMZn49CoGWfGpUFya7n93EGuuvIn/zaP8ptT6lJutueNsbDy238BKwZT0w==</latexit>✓

1
Tr(⌦r W ⌦c W T )d

log det(⌦r ) + IC (⌦r )where<latexit sha1_base64="YHvUliikuesZnzOEZk0HdVrSAaU=">AAAGLXicfVRLbxMxEHZLAyW8WjhysYgqJahUSUGCA5X6lMIBWtr0odZp5HgnyareR73O0/JvQuJf8AM4ICGuHLnCDTsN3cQBLK12dr7PM996xlOPuZ/IYvHLzOyNuczNW/O3s3fu3rv/YGHx4VEStQWDQxbxSJzUaQLcD+FQ+pLDSSyABnUOx/WLLYsfd0AkfhRWZD+GakCbod/wGZXGVVtgJKCyxShX73RNXX9saZ3fKBAPGnBJ1CbxQ3Kwt30ek8D3MJEiv3leyZ/iZ3ijUCAcLnFxGZNGJCjn+NSy00BE1xZyxZXicOFpozQycmi09mqLcx+JF7F2AKFknCbJWakYy6qiQvqMg86SdgIxZRe0CWfGDGkASVUNT0PjJePxsBFjnlDioXd8h6JBkvSDumFalYmLWee/MNkKtOuzWRKNh2sCM2QZRTyZ3GHjTrvP2rLxqqr8MG5LCNnVXzTaHMsI27phzxfAJO8bgzLhm4PArEUFZdJUN7uE3RQxHUQ6O5EioBcgoigwJ0VDBtwICKHLoiCgoadM1Tx91QH1unqjHbQLfrMl/xAaqusSer0U7Llgv5+CfRccDFJwMJW3+5+cSRzqtGkPpmEpAMYYFZdRLo+hZRcFIUxDKdKhAuLE51HoEDxzQccCbLsBhvfHBmC2k00jhmtuikvD0WptjOIwdnpxWpYdN8P+fgrua1vwcZRFHav/wG8GtCb0FMiuQebWxHRKxzTc8P9Hx190k8fCorsBNKejx4JdY25wM0pS1cOqbYO55wLeGtduDIKavE8VocJIM21l3mTZWn8lKjOPtKq4Aox6mmqnrnYeNT2wxTMGsZYj3zZsi8rpbrWnos9KVZMWelKZT53PlQp2ypXcmTZtHK2ulJ6vrL5/kVt/PZp38+gxeoLyqIReonVURnvoEDH0Cf1AP9GvzIfM58zXzLcr6uzMaM8jNLEy338DEDBBQQ==</latexit>

NC (A) := {B 2 Sp | Tr(B T (Z◆= W ⌦c W T /d⌦r 1 + NC (⌦r )A))  0, 8Z 2 C}

is the normal cone w.r.t. the constraint set C at A.
<latexit sha1_base64="j73KwWKUhWZknInarU83VTyoloE=">AAAF03icfVTLbhMxFHWhgRJeLSzZjIgqFYSqTEGCBZUqtZXCArW0TVspE1Uez00yqh9T23las0FsWbKFf+Bv+BvsaWgSD2Apyp17ju8543vHcUZTpev1X0u3bi9X7txduVe9/+Dho8era09OlehLAk0iqJDnMVZAUw5NnWoK55kEzGIKZ/HlrsPPBiBVKviJHmfQZrjL005KsLapVsSw7hFMzW5+sVqrb9aLFZSDcBrU0HQdXqwt/4wSQfoMuCYUK9UK65luGyx1Sijk1aivIMPkEnehZUOOGai2KTznwbrNJEFHSPvjOiiy8zsMZkqNWWyZzqPyMZf8F6Z7LPdzTkXlQbEWMEvWQlC1uMPVLadbfd151zYpz/oaOLl+i06fBloE7nSDJJVANB3bABOZ2oMISA9LTLTtQXU98CUyPBF5dUGC4UuQQjB7UpgToNYAhyERjGGemCjlSW6KtsWx+ZB76BDSbk//IXTM0CeMRjNw5IPj8Qwc++BkMgMnJd3hfzRVxqeom7TjMqwlwBzjxGc0GnNow0dBSjtQJhpgCZlKqeAeIbGf0VyBPb9AAh24cgWIm2Q7iHzbl7iynNxsz1E8xv4om7Vl31c4OpqBR7lr+DxKxMD5P067DF/IvASSG5D4PbGTMrADV7z/9PjrvngmHXrAoFuunklyg/nFjw/3Zq6Lru2B/c4lfLSpgwwktrovTYSltWbHyv5Hr1z0V6KJtLVx4huw7vHMO/a9U9FNwDXPBpGLPPtuYHtYl6fVnUreCttWFkba2Md8oxa+cLdc6N9p5eB0azN8vbn16U1t5/30vltBz9BztIFC9BbtoAY6RE1EkEDf0Hf0o9KsmMrnypdr6q2l6Z6naGFVvv4GEPYe2g==</latexit><latexit sha1_base64="o0J29swJaNReTs3+/A/mzzrRBmY=">AAAFyXicfVRbb9MwFPbGCqNctsEjLxHVpIHQ1AwkeGDSEJtUhNDuF6mpJsc5baPFdma71yhPPPIKf4R/w7/huCtL6wCWqp6c7/P5vhyfOEyTWJt6/dfC4p2lyt17y/erDx4+eryyuvbkTMueYnDKZCLVRUg1JLGAUxObBC5SBZSHCZyHVx8tft4HpWMpTswohRanHRG3Y0YNpg4/XK7W6pv1yfLKgT8NamS6Di7Xln4GkWQ9DsKwhGrd9OupaWVUmZglkFeDnoaUsivagSaGgnLQrWziNPfWMRN5banwJ4w3yc7uyCjXesRDZHJqutrFbPJfmOny3M1ZFZ17kzWHIdlImej5HbZuOd3smfa7VhaLtGdAsJu3aPcSz0jP9tSLYgXMJCMMKFMxNsJjXaooM9j56rrnSqR0LPPqnASnV6Ck5NgpKhgkaEDAgEnOqYiyIBZRngV2bxhmn3IHHUDc6Zo/hHY2cAnDYQEOXXA0KsCRC47HBTgu6Q7+o6lTMUUZTbLjMmwUwAzjxGU0GjNow0VBKRyoLOhTBamOEykcQoQfz0yBXbdABG24tgWYnWQcRLHtSlwjJ8+2ZygOY2+YFsey5yocHRXgUW4PfBZlsm/9H8cdTi9VXgLZLcjcM8FJ6ePATd5/2v66K54qi+5z6JSrp4rdYm7x44PdwvXk1HYBv3MFXzC1n4KiqPsyC6hCazhW+B+8stFfiVlg0MaJawDd08I7db0nshOBPTwMAhs59u3AdqkpT6vtSt70WygLQ5PhY75R81/keMv57p1WDs62Nv3Xm1uHb2o776f33TJ5Rp6TDeKTt2SHNMgBOSWMAPlGvpMflc+V68qwMr6hLi5M9zwlc6vy9TfKghpG</latexit>

50

Learning with Adaptive Regularization
The first-order optimality condition tells us:
02@<latexit sha1_base64="Ue9g0E5iTRMt4sX51DJZfrgVKc0=">AAAGcXicfVTbThsxEN1QSGlaWmifqr5YREhJKTQLldoHIiEBUvrQQiEQpGyIHK83WbE3vM7V8vf0vX/T7+gPdLwJbOKltZRkMud4Zuw5nk7kuTGvVH7nlp4sr+Sfrj4rPH+x9vLV+sbrqzjsM0IvSeiF7LqDY+q5Ab3kLvfodcQo9jsebXRujxTeGFAWu2FQ5+OItnzcDVzHJZiDq73+s4IsN0BWhBl3sWd51OEly2GYCFMKW1qclayIMdSAb4IaN/UyQjvI8sKuTXkCldE2snzMe52O+CrbIrEJ9sSRlFOCxdxuj5dRNY3y0VZRALwRO6a8j6B2fX80RHu9WNmtJAtlDXNmFI3ZOmtvLP+y7JD0fRpw4uE4bpqViLeEOifxqCxY/ZhGmNziLm2CGWCfxi2R3KhEW+CxkRMy+AQcJd75HQL7cTz2O8BUpcY6ppz/wnjPl7pPZYklStYCBmQehl68uEPFzbqbfe58aQk3iPqcBmR6CqfvIR4i1Xtku4wS7o3BwIS5cBGI9DD0moNCCltITxHhSSgLCyl8fEtZGPpwUzgg1IMCAjokoe/jwBYgJVuKVAwaOqRKB/cERwx1wmiUgiMdHI9TcKyDk0kKTjJ5h//JGUeBTAV3kYU5o3SOUdcZtdocWtNRyhgISlgDzGgUu14YaAQbHvlcgGM9gE0deqcCEKVkEGJQ1VPcAUeK6hxFY5yMorQtJ3qG8/MUPJeq4fMoCQeq/gu36+M2kxmQPIBE7wkoZQCCS84/u/6KnhweN6CnPu1mo8OseMD04Bdnx2nVSdeOKbxzRr+B6zSiDEPe98LCDEoDWcGv9UFZjxIFzDkp6noBUD1Oa8d67dMhKBPDUpZWvhJsD/OsWtWtyKbZgrR0xAX8laWiWZYw5Ux9pmWNq71dc39378en4uHBbN6tGu+MTaNkmMZn49CoGWfGpUFya7n93EGuuvIn/zaP8ptT6lJutueNsbDy238BKwZT0w==</latexit>✓

1
Tr(⌦r W ⌦c W T )d

log det(⌦r ) + IC (⌦r )Lemma: Let ⌦r 2 C, then NC (⌦r ) =<latexit sha1_base64="4IJktkuUY2Oz1QVkgz4HbCpEV7A=">AAAF3XicfVRLb9NAEN6WBkp4tKVHLhZRpYJQFRckOFCpUlspHFBL31IcRev1JLFq77q7m6flIzfElSNXuPJv+DfMuqFO1sBKlsfzfTvzeWZ2/SQKla7Xfy0s3lmq3L23fL/64OGjxyura0/OlehLBmdMREJe+lRBFHI406GO4DKRQGM/ggv/as/gFwOQKhT8VI8TaMW0y8NOyKhGV3t13Uuk9ELueDHVPUajdC9rr9bqW/V8OWXDnRo1Ml1H7bWln14gWD8GrllElWq69US3Uip1yCLIql5fQULZFe1CE01OY1CtNFefORvoCZyOkPhw7eTe2R0pjZUaxz4yjUZlY8b5L0z34sz2mSwqc/I1hyFZCxGp+R0mbtnd7OvO21Ya8qSvgbObv+j0I0cLx9TZCUIJTEdjNCiTIRbCYT0qKdPYjeqGY6dI6ERk1bkUMb0CKUSMlaKcQYQCOAyZiGPKgxSbFmRp3jbfT99nFjqEsNvTfwiddGgTRqMCHNngeFyAYxucTApwUso7/E9OlfApaibtpAxrCTDDOLUZjcYM2rBRkBIHKvUGVEKiwkhwixDggZoJsG8HCKAD1yYAM5OMg8h37BTXyMnSnRmKxTgYJUVbDuwMx8cFeJyZhs+iTAyM/pOwG9O2zEoguwWZ3ROclAEOXP7/0/LX7eR41BE9jKFbjp5IdovZwU+O9gvVedf2Ac+5hA/oOkxAUsz7IvWoRGk4Vvj2Xhrrr8TU0yjj1BaA6mmhndraI9ENwDQPDc9YlnwzsD2qy9NqqpI13RamhZFO8TPbrLnPzS3n2nda2Tjf3nJfbW1/fF3bfTe975bJU/KMbBKXvCG7pEGOyBlhZEy+ke/kR6Vd+VT5XPlyQ11cmO5ZJ3Or8vU3tT4iXg==</latexit><latexit sha1_base64="/u6ZN8PDI+xiMHXOMa/g0tFiLeE=">AAAGEXicfVRLbxMxEHZLAyU82sKRy4qoUotolS1IcKBSpbYiHKClbykbIsc7SVbdXW9t57nyr+B3cOHGDXHlyAkJ/gvjNHQTL9TSasfzfZ757Bm7kYSBVOXyz5nZG3OFm7fmbxfv3L13f2Fx6cGJ5B3B4JjxkIuzBpUQBjEcq0CFcJYIoFEjhNPG+bbBT7sgZMDjIzVIoBbRVhw0A0YVuuqLr72IqjajYfpO19OrybbWK14ixKqz6axdT/mQrrl6tb5YKq+XR8PJG+7YKJHx2K8vzX32fM46EcSKhVTKqltOVC2lQgUsBF30OhISys5pC6poxjQCWUtHO9bOMnp8p8kFfrFyRt7JFSmNpBxEDWQavdLGjPN/mGpH2vaZLFI7ozGFIVlxHsrpFSZu3l3tqObLWhrESUdBzC530eyEjuKOqY3jBwKYCgdoUCYCPAiHtamgTGEFi8uOnSKhQ66LUykieg6C8whPisYMQhQQQ4/xKKKxn3pB7OvLEjYa6RttoT0IWm31l9BMezah38/Avg0OBhk4sMHhMAOHuby9a3LKJNZZ1x3mYSUAJhhHNqNSmUArNgpCYEOlXpcKSGQQ8tgi+HgJJwLs2AF8aMKFCcBMJ2Mjxpt2igvk6HRzgmIxdvtJVpZdO8PBQQYeaFPwSZTxrtF/GLQiWhc6B7IrkNk1wU7pYsON9j8+/rKdHG84onsRtPLRE8GuMDv44f5OpnpUtR3Aey7gLbr2EhAU8z5JPSpQGrYV/r2nxvonMfUUyjiyBaB6mmmntvaQt3wwxUPDM5Yl3zRsm6p8t5pT0VW3hmmhr1Kc6pWSu6rxlXPtNy1vnGysu8/WN94/L229Gr938+QReUxWiEtekC1SIfvkmDDyifwgv8jvwsfCl8LXwrdL6uzMeM1DMjUK3/8AiEE32g==</latexit>◆= W ⌦c W T /d⌦r 1 + NC (⌦r )NC (⌦r 1 ).

51

Learning with Adaptive Regularization
The first-order optimality condition tells us:
02@<latexit sha1_base64="Ue9g0E5iTRMt4sX51DJZfrgVKc0=">AAAGcXicfVTbThsxEN1QSGlaWmifqr5YREhJKTQLldoHIiEBUvrQQiEQpGyIHK83WbE3vM7V8vf0vX/T7+gPdLwJbOKltZRkMud4Zuw5nk7kuTGvVH7nlp4sr+Sfrj4rPH+x9vLV+sbrqzjsM0IvSeiF7LqDY+q5Ab3kLvfodcQo9jsebXRujxTeGFAWu2FQ5+OItnzcDVzHJZiDq73+s4IsN0BWhBl3sWd51OEly2GYCFMKW1qclayIMdSAb4IaN/UyQjvI8sKuTXkCldE2snzMe52O+CrbIrEJ9sSRlFOCxdxuj5dRNY3y0VZRALwRO6a8j6B2fX80RHu9WNmtJAtlDXNmFI3ZOmtvLP+y7JD0fRpw4uE4bpqViLeEOifxqCxY/ZhGmNziLm2CGWCfxi2R3KhEW+CxkRMy+AQcJd75HQL7cTz2O8BUpcY6ppz/wnjPl7pPZYklStYCBmQehl68uEPFzbqbfe58aQk3iPqcBmR6CqfvIR4i1Xtku4wS7o3BwIS5cBGI9DD0moNCCltITxHhSSgLCyl8fEtZGPpwUzgg1IMCAjokoe/jwBYgJVuKVAwaOqRKB/cERwx1wmiUgiMdHI9TcKyDk0kKTjJ5h//JGUeBTAV3kYU5o3SOUdcZtdocWtNRyhgISlgDzGgUu14YaAQbHvlcgGM9gE0deqcCEKVkEGJQ1VPcAUeK6hxFY5yMorQtJ3qG8/MUPJeq4fMoCQeq/gu36+M2kxmQPIBE7wkoZQCCS84/u/6KnhweN6CnPu1mo8OseMD04Bdnx2nVSdeOKbxzRr+B6zSiDEPe98LCDEoDWcGv9UFZjxIFzDkp6noBUD1Oa8d67dMhKBPDUpZWvhJsD/OsWtWtyKbZgrR0xAX8laWiWZYw5Ux9pmWNq71dc39378en4uHBbN6tGu+MTaNkmMZn49CoGWfGpUFya7n93EGuuvIn/zaP8ptT6lJutueNsbDy238BKwZT0w==</latexit>✓

1
Tr(⌦r W ⌦c W T )d

log det(⌦r ) + IC (⌦r )Lemma: Let ⌦r 2 C, then NC (⌦r ) =<latexit sha1_base64="4IJktkuUY2Oz1QVkgz4HbCpEV7A=">AAAF3XicfVRLb9NAEN6WBkp4tKVHLhZRpYJQFRckOFCpUlspHFBL31IcRev1JLFq77q7m6flIzfElSNXuPJv+DfMuqFO1sBKlsfzfTvzeWZ2/SQKla7Xfy0s3lmq3L23fL/64OGjxyura0/OlehLBmdMREJe+lRBFHI406GO4DKRQGM/ggv/as/gFwOQKhT8VI8TaMW0y8NOyKhGV3t13Uuk9ELueDHVPUajdC9rr9bqW/V8OWXDnRo1Ml1H7bWln14gWD8GrllElWq69US3Uip1yCLIql5fQULZFe1CE01OY1CtNFefORvoCZyOkPhw7eTe2R0pjZUaxz4yjUZlY8b5L0z34sz2mSwqc/I1hyFZCxGp+R0mbtnd7OvO21Ya8qSvgbObv+j0I0cLx9TZCUIJTEdjNCiTIRbCYT0qKdPYjeqGY6dI6ERk1bkUMb0CKUSMlaKcQYQCOAyZiGPKgxSbFmRp3jbfT99nFjqEsNvTfwiddGgTRqMCHNngeFyAYxucTApwUso7/E9OlfApaibtpAxrCTDDOLUZjcYM2rBRkBIHKvUGVEKiwkhwixDggZoJsG8HCKAD1yYAM5OMg8h37BTXyMnSnRmKxTgYJUVbDuwMx8cFeJyZhs+iTAyM/pOwG9O2zEoguwWZ3ROclAEOXP7/0/LX7eR41BE9jKFbjp5IdovZwU+O9gvVedf2Ac+5hA/oOkxAUsz7IvWoRGk4Vvj2Xhrrr8TU0yjj1BaA6mmhndraI9ENwDQPDc9YlnwzsD2qy9NqqpI13RamhZFO8TPbrLnPzS3n2nda2Tjf3nJfbW1/fF3bfTe975bJU/KMbBKXvCG7pEGOyBlhZEy+ke/kR6Vd+VT5XPlyQ11cmO5ZJ3Or8vU3tT4iXg==</latexit><latexit sha1_base64="/u6ZN8PDI+xiMHXOMa/g0tFiLeE=">AAAGEXicfVRLbxMxEHZLAyU82sKRy4qoUotolS1IcKBSpbYiHKClbykbIsc7SVbdXW9t57nyr+B3cOHGDXHlyAkJ/gvjNHQTL9TSasfzfZ757Bm7kYSBVOXyz5nZG3OFm7fmbxfv3L13f2Fx6cGJ5B3B4JjxkIuzBpUQBjEcq0CFcJYIoFEjhNPG+bbBT7sgZMDjIzVIoBbRVhw0A0YVuuqLr72IqjajYfpO19OrybbWK14ixKqz6axdT/mQrrl6tb5YKq+XR8PJG+7YKJHx2K8vzX32fM46EcSKhVTKqltOVC2lQgUsBF30OhISys5pC6poxjQCWUtHO9bOMnp8p8kFfrFyRt7JFSmNpBxEDWQavdLGjPN/mGpH2vaZLFI7ozGFIVlxHsrpFSZu3l3tqObLWhrESUdBzC530eyEjuKOqY3jBwKYCgdoUCYCPAiHtamgTGEFi8uOnSKhQ66LUykieg6C8whPisYMQhQQQ4/xKKKxn3pB7OvLEjYa6RttoT0IWm31l9BMezah38/Avg0OBhk4sMHhMAOHuby9a3LKJNZZ1x3mYSUAJhhHNqNSmUArNgpCYEOlXpcKSGQQ8tgi+HgJJwLs2AF8aMKFCcBMJ2Mjxpt2igvk6HRzgmIxdvtJVpZdO8PBQQYeaFPwSZTxrtF/GLQiWhc6B7IrkNk1wU7pYsON9j8+/rKdHG84onsRtPLRE8GuMDv44f5OpnpUtR3Aey7gLbr2EhAU8z5JPSpQGrYV/r2nxvonMfUUyjiyBaB6mmmntvaQt3wwxUPDM5Yl3zRsm6p8t5pT0VW3hmmhr1Kc6pWSu6rxlXPtNy1vnGysu8/WN94/L229Gr938+QReUxWiEtekC1SIfvkmDDyifwgv8jvwsfCl8LXwrdL6uzMeM1DMjUK3/8AiEE32g==</latexit>

With this lemma, we have:W ⌦c W T /d<latexit sha1_base64="TlrqV5Uk4eWYFaTqYCYGYleIAwM=">AAAGCnicfVTNbhMxEHahgRL+WjhyWRFVahEt2YIEBypVaiuFA7S0aVMpm0Ze7yRZdXe99Tq/K78Bz8ED9Ia4cuQKEm/DOA3ZxAtYWu14vs8znz1ju3HgJ7Jc/rVw4+Zi4dbtpTvFu/fuP3i4vPLoNOFdweCE8YCLM5cmEPgRnEhfBnAWC6ChG0DNvdjVeK0HIvF5VJXDGBohbUd+y2dUoqu5vFNzYsGs2nn1hWdtWDgR5+mGrSzHjywnpLLDaJB+UM10OtlVam3KW28ul8qb5fGw8oY9MUpkMg6bK4tXjsdZN4RIsoAmSd0ux7KRUiF9FoAqOt0EYsouaBvqaEY0hKSRjreqrFX0eFaLC/wiaY29sytSGibJMHSRqfUmJqad/8JkJ1SmT2dJlDUecxiSJedBMr9Cx827613ZetNI/SjuSojY9S5a3cCS3NJFsTxfAJPBEA3KhI8HYbEOFZRJLF1x1TJTxHTEVXEuRUgvQHAe4knRiEGAAiLoMx6GNPJSrKanrkvouuk7ZaB98Nsd+YfQSvsmYTDIwIEJDocZODTB0SgDR7m8/f/kTOJIZV13nIelAJhhVE1GpTKDVkwUhMCGSp0eFRAnfsAjg+Dh7ZsJsGcG8KAFlzoA052MjRhtmykukaPS7RmKwdgfxFlZ9s0MR0cZeKR0wWdRxnta/7HfDmlTqBzIpiAza4Kd0sOGG+9/cvxlMznecEQPQmjno+OTMcXM4MeHe5nqcdX2AO+5gPfoOohBUMz7LHWoQGnYVvh3nmvrr8TUkSijagpA9TTTTk3tAW97oIuHhqMtQ75u2A6V+W7Vp6LqdgPTwkCmOFVrJXtd4Stnm29a3jjd2rRfbm59fFXaeTt575bIE/KUrBGbvCY7pEIOyQlh5DP5Tn6Qn4VPhavCl8LXa+qNhcmax2RuFL79Bv9ZM50=</latexit>◆= W ⌦c W T /d⌦r 1 + NC (⌦r )NC (⌦r 1 ).
⌦r 1 2 NC (⌦r 1 )

1
T
⌦
W
⌦
W
/d
c
which means the optimal r is the Euclidean projection of
onto C.
<latexit sha1_base64="MdwYTlMqOaiJg6PgK4nHeQswj0Q=">AAAF0XicfVRLbxMxEHZLAyW8WjhyWRFVKgiqbEGCA5Uq0UrhgFr6BGVD5Xgnyapre2s7T2slxJUjV/gT/Bv+DeM0NIkXsLTa2fk+z3w7M3YzSxNtqtVfC4vXlkrXbyzfLN+6fefuvZXV+ydadhWDYyZTqT40qYY0EXBsEpPCh0wB5c0UTpvnbxx+2gOlEymOzDCDBqdtkbQSRg26PkaZUp/sszA/W6lUN6rjFRSNcGJUyGTtn60u/YxiybochGEp1boeVjPTsFSZhKWQl6Ouhoyyc9qGOpqCctANO1acB2voiYOWVPgIE4y9szss5VoPeROZnJqO9jHn/BdmOjz3fS6LzoPxmsOQbKRM9fwOF7forndN61XDJiLrGhDs8i9a3TQwMnC1DeJEATPpEA3KVIKFCFiHKsoMdqC8FvgpMjqSeXkuBafnoKTkWCkqGKQoQECfSc6piG2UiDi3kdvbbNq3uYf2IWl3zB9Cy/Z9wmAwBQc+OBxOwaEPjkZTcFTI2/9PTp2JCcpoag+LsFEAM4wjn1GrzaA1HwWlcKBs1KMKMp2kUniEGA/RTIAdP0AMLbhwAZibZBxEseWnuEBObrdmKB5jd5BN27LrZzg4mIIHuWv4LMpkz+k/TNqcnqm8ALIrkPk9wUnp4cCN/39S/qqfHA84onsc2sXomWJXmB/8cH9nqnrctR3Ac67gHbr2MlAU8z6xEVUoDccK39FTZ/2VaCODMo58AaieTrVTX3sq2zG45qEROcuT7wa2Q01xWl1V8nrYwLQwMBY/8/VK+NjdcqF/pxWNk82N8PnG5vsXle3Xk/tumTwkj8g6CclLsk1qZJ8cE0Y4+Ua+kx+lw9Kw9Ln05ZK6uDDZ84DMrdLX3wwYHbk=</latexit><latexit sha1_base64="XnfAO4hGjo+zgSU7pkfu3WiUymU=">AAAF03icfVRNbxMxEHVLAyV8tXDksiKqVBAq2YIEBypVopXCAbW0SVMpGyrHO0lW2bW3tvNp7QVx5cgV/gP/hn/DOA1N4gUsrXZ23vPM25mxW2kcKV0u/1pZvbFWuHlr/Xbxzt179x9sbD48U6IvGdSYiIU8b1EFccShpiMdw3kqgSatGOqt3juL1wcgVSR4VY9TaCa0w6N2xKhGV6MepJJ59U/VF+HFRqm8U54uL2/4M6NEZuv4YnPtZxAK1k+AaxZTpRp+OdVNQ6WOWAxZMegrSCnr0Q400OQ0AdU0U82Zt4We0GsLiQ/X3tS7uMPQRKlx0kJmQnVXuZh1/gvT3SRzfTaLyrzpWsKQrIWI1fIOGzfvbvR1+03TRDzta+Ds6i/a/djTwrPV9cJIAtPxGA3KZISF8FiXSso09qC45bkpUjoRWXEpRUJ7IIVIsFKUM4hRAIchE0lCeWiCiIeZCezeVsu8zxx0CFGnq/8Q2mboEkajOThywfF4Do5dcDKZg5Nc3uF/cqqUz1BGY3Oah7UEWGBUXUalsoBWXBSkxIEywYBKSFUUC+4QQjxGCwEO3AAhtOHSBmB2knEQ+Z6b4hI5mdlboDiMw1E6b8uhm+HkZA6eZLbhiygTA6v/NOok9EJmOZBdg8ztCU7KAAdu+v+z8pfd5Km06FECnXx0PP3XmBv89PhgrnratQPAcy7hA7qOUpAU8z4zAZUoDccK38Fza/2VaAKNMqquAFRP59qpqz0WnRBs89AIrOXItwPbpTo/rbYqWcNvYloYaYOf2XbJf5rhLee7d1reONvd8V/u7H58Vdp/O7vv1slj8oRsE5+8JvukQo5JjTAiyDfynfwo1Aqm8Lnw5Yq6ujLb84gsrcLX3xKnHh0=</latexit><latexit sha1_base64="j73KwWKUhWZknInarU83VTyoloE=">AAAF03icfVTLbhMxFHWhgRJeLSzZjIgqFYSqTEGCBZUqtZXCArW0TVspE1Uez00yqh9T23las0FsWbKFf+Bv+BvsaWgSD2Apyp17ju8543vHcUZTpev1X0u3bi9X7txduVe9/+Dho8era09OlehLAk0iqJDnMVZAUw5NnWoK55kEzGIKZ/HlrsPPBiBVKviJHmfQZrjL005KsLapVsSw7hFMzW5+sVqrb9aLFZSDcBrU0HQdXqwt/4wSQfoMuCYUK9UK65luGyx1Sijk1aivIMPkEnehZUOOGai2KTznwbrNJEFHSPvjOiiy8zsMZkqNWWyZzqPyMZf8F6Z7LPdzTkXlQbEWMEvWQlC1uMPVLadbfd151zYpz/oaOLl+i06fBloE7nSDJJVANB3bABOZ2oMISA9LTLTtQXU98CUyPBF5dUGC4UuQQjB7UpgToNYAhyERjGGemCjlSW6KtsWx+ZB76BDSbk//IXTM0CeMRjNw5IPj8Qwc++BkMgMnJd3hfzRVxqeom7TjMqwlwBzjxGc0GnNow0dBSjtQJhpgCZlKqeAeIbGf0VyBPb9AAh24cgWIm2Q7iHzbl7iynNxsz1E8xv4om7Vl31c4OpqBR7lr+DxKxMD5P067DF/IvASSG5D4PbGTMrADV7z/9PjrvngmHXrAoFuunklyg/nFjw/3Zq6Lru2B/c4lfLSpgwwktrovTYSltWbHyv5Hr1z0V6KJtLVx4huw7vHMO/a9U9FNwDXPBpGLPPtuYHtYl6fVnUreCttWFkba2Md8oxa+cLdc6N9p5eB0azN8vbn16U1t5/30vltBz9BztIFC9BbtoAY6RE1EkEDf0Hf0o9KsmMrnypdr6q2l6Z6naGFVvv4GEPYe2g==</latexit>

52

Learning with Adaptive Regularization
The first-order optimality condition tells us:
02@<latexit sha1_base64="Ue9g0E5iTRMt4sX51DJZfrgVKc0=">AAAGcXicfVTbThsxEN1QSGlaWmifqr5YREhJKTQLldoHIiEBUvrQQiEQpGyIHK83WbE3vM7V8vf0vX/T7+gPdLwJbOKltZRkMud4Zuw5nk7kuTGvVH7nlp4sr+Sfrj4rPH+x9vLV+sbrqzjsM0IvSeiF7LqDY+q5Ab3kLvfodcQo9jsebXRujxTeGFAWu2FQ5+OItnzcDVzHJZiDq73+s4IsN0BWhBl3sWd51OEly2GYCFMKW1qclayIMdSAb4IaN/UyQjvI8sKuTXkCldE2snzMe52O+CrbIrEJ9sSRlFOCxdxuj5dRNY3y0VZRALwRO6a8j6B2fX80RHu9WNmtJAtlDXNmFI3ZOmtvLP+y7JD0fRpw4uE4bpqViLeEOifxqCxY/ZhGmNziLm2CGWCfxi2R3KhEW+CxkRMy+AQcJd75HQL7cTz2O8BUpcY6ppz/wnjPl7pPZYklStYCBmQehl68uEPFzbqbfe58aQk3iPqcBmR6CqfvIR4i1Xtku4wS7o3BwIS5cBGI9DD0moNCCltITxHhSSgLCyl8fEtZGPpwUzgg1IMCAjokoe/jwBYgJVuKVAwaOqRKB/cERwx1wmiUgiMdHI9TcKyDk0kKTjJ5h//JGUeBTAV3kYU5o3SOUdcZtdocWtNRyhgISlgDzGgUu14YaAQbHvlcgGM9gE0deqcCEKVkEGJQ1VPcAUeK6hxFY5yMorQtJ3qG8/MUPJeq4fMoCQeq/gu36+M2kxmQPIBE7wkoZQCCS84/u/6KnhweN6CnPu1mo8OseMD04Bdnx2nVSdeOKbxzRr+B6zSiDEPe98LCDEoDWcGv9UFZjxIFzDkp6noBUD1Oa8d67dMhKBPDUpZWvhJsD/OsWtWtyKbZgrR0xAX8laWiWZYw5Ux9pmWNq71dc39378en4uHBbN6tGu+MTaNkmMZn49CoGWfGpUFya7n93EGuuvIn/zaP8ptT6lJutueNsbDy238BKwZT0w==</latexit>✓

1
Tr(⌦r W ⌦c W T )d

log det(⌦r ) + IC (⌦r )Lemma: Let ⌦r 2 C, then NC (⌦r ) =<latexit sha1_base64="4IJktkuUY2Oz1QVkgz4HbCpEV7A=">AAAF3XicfVRLb9NAEN6WBkp4tKVHLhZRpYJQFRckOFCpUlspHFBL31IcRev1JLFq77q7m6flIzfElSNXuPJv+DfMuqFO1sBKlsfzfTvzeWZ2/SQKla7Xfy0s3lmq3L23fL/64OGjxyura0/OlehLBmdMREJe+lRBFHI406GO4DKRQGM/ggv/as/gFwOQKhT8VI8TaMW0y8NOyKhGV3t13Uuk9ELueDHVPUajdC9rr9bqW/V8OWXDnRo1Ml1H7bWln14gWD8GrllElWq69US3Uip1yCLIql5fQULZFe1CE01OY1CtNFefORvoCZyOkPhw7eTe2R0pjZUaxz4yjUZlY8b5L0z34sz2mSwqc/I1hyFZCxGp+R0mbtnd7OvO21Ya8qSvgbObv+j0I0cLx9TZCUIJTEdjNCiTIRbCYT0qKdPYjeqGY6dI6ERk1bkUMb0CKUSMlaKcQYQCOAyZiGPKgxSbFmRp3jbfT99nFjqEsNvTfwiddGgTRqMCHNngeFyAYxucTApwUso7/E9OlfApaibtpAxrCTDDOLUZjcYM2rBRkBIHKvUGVEKiwkhwixDggZoJsG8HCKAD1yYAM5OMg8h37BTXyMnSnRmKxTgYJUVbDuwMx8cFeJyZhs+iTAyM/pOwG9O2zEoguwWZ3ROclAEOXP7/0/LX7eR41BE9jKFbjp5IdovZwU+O9gvVedf2Ac+5hA/oOkxAUsz7IvWoRGk4Vvj2Xhrrr8TU0yjj1BaA6mmhndraI9ENwDQPDc9YlnwzsD2qy9NqqpI13RamhZFO8TPbrLnPzS3n2nda2Tjf3nJfbW1/fF3bfTe975bJU/KMbBKXvCG7pEGOyBlhZEy+ke/kR6Vd+VT5XPlyQ11cmO5ZJ3Or8vU3tT4iXg==</latexit><latexit sha1_base64="/u6ZN8PDI+xiMHXOMa/g0tFiLeE=">AAAGEXicfVRLbxMxEHZLAyU82sKRy4qoUotolS1IcKBSpbYiHKClbykbIsc7SVbdXW9t57nyr+B3cOHGDXHlyAkJ/gvjNHQTL9TSasfzfZ757Bm7kYSBVOXyz5nZG3OFm7fmbxfv3L13f2Fx6cGJ5B3B4JjxkIuzBpUQBjEcq0CFcJYIoFEjhNPG+bbBT7sgZMDjIzVIoBbRVhw0A0YVuuqLr72IqjajYfpO19OrybbWK14ixKqz6axdT/mQrrl6tb5YKq+XR8PJG+7YKJHx2K8vzX32fM46EcSKhVTKqltOVC2lQgUsBF30OhISys5pC6poxjQCWUtHO9bOMnp8p8kFfrFyRt7JFSmNpBxEDWQavdLGjPN/mGpH2vaZLFI7ozGFIVlxHsrpFSZu3l3tqObLWhrESUdBzC530eyEjuKOqY3jBwKYCgdoUCYCPAiHtamgTGEFi8uOnSKhQ66LUykieg6C8whPisYMQhQQQ4/xKKKxn3pB7OvLEjYa6RttoT0IWm31l9BMezah38/Avg0OBhk4sMHhMAOHuby9a3LKJNZZ1x3mYSUAJhhHNqNSmUArNgpCYEOlXpcKSGQQ8tgi+HgJJwLs2AF8aMKFCcBMJ2Mjxpt2igvk6HRzgmIxdvtJVpZdO8PBQQYeaFPwSZTxrtF/GLQiWhc6B7IrkNk1wU7pYsON9j8+/rKdHG84onsRtPLRE8GuMDv44f5OpnpUtR3Aey7gLbr2EhAU8z5JPSpQGrYV/r2nxvonMfUUyjiyBaB6mmmntvaQt3wwxUPDM5Yl3zRsm6p8t5pT0VW3hmmhr1Kc6pWSu6rxlXPtNy1vnGysu8/WN94/L229Gr938+QReUxWiEtekC1SIfvkmDDyifwgv8jvwsfCl8LXwrdL6uzMeM1DMjUK3/8AiEE32g==</latexit>

With this lemma, we have:W ⌦c W T /d<latexit sha1_base64="TlrqV5Uk4eWYFaTqYCYGYleIAwM=">AAAGCnicfVTNbhMxEHahgRL+WjhyWRFVahEt2YIEBypVaiuFA7S0aVMpm0Ze7yRZdXe99Tq/K78Bz8ED9Ia4cuQKEm/DOA3ZxAtYWu14vs8znz1ju3HgJ7Jc/rVw4+Zi4dbtpTvFu/fuP3i4vPLoNOFdweCE8YCLM5cmEPgRnEhfBnAWC6ChG0DNvdjVeK0HIvF5VJXDGBohbUd+y2dUoqu5vFNzYsGs2nn1hWdtWDgR5+mGrSzHjywnpLLDaJB+UM10OtlVam3KW28ul8qb5fGw8oY9MUpkMg6bK4tXjsdZN4RIsoAmSd0ux7KRUiF9FoAqOt0EYsouaBvqaEY0hKSRjreqrFX0eFaLC/wiaY29sytSGibJMHSRqfUmJqad/8JkJ1SmT2dJlDUecxiSJedBMr9Cx827613ZetNI/SjuSojY9S5a3cCS3NJFsTxfAJPBEA3KhI8HYbEOFZRJLF1x1TJTxHTEVXEuRUgvQHAe4knRiEGAAiLoMx6GNPJSrKanrkvouuk7ZaB98Nsd+YfQSvsmYTDIwIEJDocZODTB0SgDR7m8/f/kTOJIZV13nIelAJhhVE1GpTKDVkwUhMCGSp0eFRAnfsAjg+Dh7ZsJsGcG8KAFlzoA052MjRhtmykukaPS7RmKwdgfxFlZ9s0MR0cZeKR0wWdRxnta/7HfDmlTqBzIpiAza4Kd0sOGG+9/cvxlMznecEQPQmjno+OTMcXM4MeHe5nqcdX2AO+5gPfoOohBUMz7LHWoQGnYVvh3nmvrr8TUkSijagpA9TTTTk3tAW97oIuHhqMtQ75u2A6V+W7Vp6LqdgPTwkCmOFVrJXtd4Stnm29a3jjd2rRfbm59fFXaeTt575bIE/KUrBGbvCY7pEIOyQlh5DP5Tn6Qn4VPhavCl8LXa+qNhcmax2RuFL79Bv9ZM50=</latexit>◆= W ⌦c W T /d⌦r 1 + NC (⌦r )NC (⌦r 1 ).
⌦r 1 2 NC (⌦r 1 )

1
T
⌦
W
⌦
W
/d
c
which means the optimal r is the Euclidean projection of
onto C. Hence it suﬃces if we can solve the following problem:<latexit sha1_base64="MdwYTlMqOaiJg6PgK4nHeQswj0Q=">AAAF0XicfVRLbxMxEHZLAyW8WjhyWRFVKgiqbEGCA5Uq0UrhgFr6BGVD5Xgnyapre2s7T2slxJUjV/gT/Bv+DeM0NIkXsLTa2fk+z3w7M3YzSxNtqtVfC4vXlkrXbyzfLN+6fefuvZXV+ydadhWDYyZTqT40qYY0EXBsEpPCh0wB5c0UTpvnbxx+2gOlEymOzDCDBqdtkbQSRg26PkaZUp/sszA/W6lUN6rjFRSNcGJUyGTtn60u/YxiybochGEp1boeVjPTsFSZhKWQl6Ouhoyyc9qGOpqCctANO1acB2voiYOWVPgIE4y9szss5VoPeROZnJqO9jHn/BdmOjz3fS6LzoPxmsOQbKRM9fwOF7forndN61XDJiLrGhDs8i9a3TQwMnC1DeJEATPpEA3KVIKFCFiHKsoMdqC8FvgpMjqSeXkuBafnoKTkWCkqGKQoQECfSc6piG2UiDi3kdvbbNq3uYf2IWl3zB9Cy/Z9wmAwBQc+OBxOwaEPjkZTcFTI2/9PTp2JCcpoag+LsFEAM4wjn1GrzaA1HwWlcKBs1KMKMp2kUniEGA/RTIAdP0AMLbhwAZibZBxEseWnuEBObrdmKB5jd5BN27LrZzg4mIIHuWv4LMpkz+k/TNqcnqm8ALIrkPk9wUnp4cCN/39S/qqfHA84onsc2sXomWJXmB/8cH9nqnrctR3Ac67gHbr2MlAU8z6xEVUoDccK39FTZ/2VaCODMo58AaieTrVTX3sq2zG45qEROcuT7wa2Q01xWl1V8nrYwLQwMBY/8/VK+NjdcqF/pxWNk82N8PnG5vsXle3Xk/tumTwkj8g6CclLsk1qZJ8cE0Y4+Ua+kx+lw9Kw9Ln05ZK6uDDZ84DMrdLX3wwYHbk=</latexit><latexit sha1_base64="XnfAO4hGjo+zgSU7pkfu3WiUymU=">AAAF03icfVRNbxMxEHVLAyV8tXDksiKqVBAq2YIEBypVopXCAbW0SVMpGyrHO0lW2bW3tvNp7QVx5cgV/gP/hn/DOA1N4gUsrXZ23vPM25mxW2kcKV0u/1pZvbFWuHlr/Xbxzt179x9sbD48U6IvGdSYiIU8b1EFccShpiMdw3kqgSatGOqt3juL1wcgVSR4VY9TaCa0w6N2xKhGV6MepJJ59U/VF+HFRqm8U54uL2/4M6NEZuv4YnPtZxAK1k+AaxZTpRp+OdVNQ6WOWAxZMegrSCnr0Q400OQ0AdU0U82Zt4We0GsLiQ/X3tS7uMPQRKlx0kJmQnVXuZh1/gvT3SRzfTaLyrzpWsKQrIWI1fIOGzfvbvR1+03TRDzta+Ds6i/a/djTwrPV9cJIAtPxGA3KZISF8FiXSso09qC45bkpUjoRWXEpRUJ7IIVIsFKUM4hRAIchE0lCeWiCiIeZCezeVsu8zxx0CFGnq/8Q2mboEkajOThywfF4Do5dcDKZg5Nc3uF/cqqUz1BGY3Oah7UEWGBUXUalsoBWXBSkxIEywYBKSFUUC+4QQjxGCwEO3AAhtOHSBmB2knEQ+Z6b4hI5mdlboDiMw1E6b8uhm+HkZA6eZLbhiygTA6v/NOok9EJmOZBdg8ztCU7KAAdu+v+z8pfd5Km06FECnXx0PP3XmBv89PhgrnratQPAcy7hA7qOUpAU8z4zAZUoDccK38Fza/2VaAKNMqquAFRP59qpqz0WnRBs89AIrOXItwPbpTo/rbYqWcNvYloYaYOf2XbJf5rhLee7d1reONvd8V/u7H58Vdp/O7vv1slj8oRsE5+8JvukQo5JjTAiyDfynfwo1Aqm8Lnw5Yq6ujLb84gsrcLX3xKnHh0=</latexit><latexit sha1_base64="j73KwWKUhWZknInarU83VTyoloE=">AAAF03icfVTLbhMxFHWhgRJeLSzZjIgqFYSqTEGCBZUqtZXCArW0TVspE1Uez00yqh9T23las0FsWbKFf+Bv+BvsaWgSD2Apyp17ju8543vHcUZTpev1X0u3bi9X7txduVe9/+Dho8era09OlehLAk0iqJDnMVZAUw5NnWoK55kEzGIKZ/HlrsPPBiBVKviJHmfQZrjL005KsLapVsSw7hFMzW5+sVqrb9aLFZSDcBrU0HQdXqwt/4wSQfoMuCYUK9UK65luGyx1Sijk1aivIMPkEnehZUOOGai2KTznwbrNJEFHSPvjOiiy8zsMZkqNWWyZzqPyMZf8F6Z7LPdzTkXlQbEWMEvWQlC1uMPVLadbfd151zYpz/oaOLl+i06fBloE7nSDJJVANB3bABOZ2oMISA9LTLTtQXU98CUyPBF5dUGC4UuQQjB7UpgToNYAhyERjGGemCjlSW6KtsWx+ZB76BDSbk//IXTM0CeMRjNw5IPj8Qwc++BkMgMnJd3hfzRVxqeom7TjMqwlwBzjxGc0GnNow0dBSjtQJhpgCZlKqeAeIbGf0VyBPb9AAh24cgWIm2Q7iHzbl7iynNxsz1E8xv4om7Vl31c4OpqBR7lr+DxKxMD5P067DF/IvASSG5D4PbGTMrADV7z/9PjrvngmHXrAoFuunklyg/nFjw/3Zq6Lru2B/c4lfLSpgwwktrovTYSltWbHyv5Hr1z0V6KJtLVx4huw7vHMO/a9U9FNwDXPBpGLPPtuYHtYl6fVnUreCttWFkba2Md8oxa+cLdc6N9p5eB0azN8vbn16U1t5/30vltBz9BztIFC9BbtoAY6RE1EkEDf0Hf0o9KsmMrnypdr6q2l6Z6naGFVvv4GEPYe2g==</latexit>

min<latexit sha1_base64="qjHvbqvHBCK+YDlKOWYfmhw2dkI=">AAAGOHicfVRLbxMxEN60DZTwauHIxSKqVFCpkoIEBypVokXpAbX0iVSHyOudJKa73o3tzcvZP8WR38GFGzfElSMn7CRtEgewtNrZ+b557Mx4/CRkUpVK33ILi0v5GzeXbxVu37l77/7K6oMzGaeCwimNw1h88ImEkHE4VUyF8CERQCI/hHP/8o3Fz9sgJIv5ieolUI1Ig7M6o0QZVW2F44jxmsaJEBnCrZQEaDCwX+gZwh0WgGJhACN8MKi9/bi1gVuWhhV0lZap/wmoQiq+sk7Rfi1Bhg8UWvYtruS2AWorxdJmaXjQvFAeC0VvfA5rq0ufcRDTNAKuaEikvCiXElXVRChGQ8gKOJWQEHpJGnBhRE4ikFU9LEyG1owmQPVYmIcrNNROW2gSSdmLfMOMiGpKF7PKf2GqGWWuzkaRGRqeGcyQVRyHctbC+p1XX6Sq/qqqGU9SBZyO/qKehqbIyLYQBcwUVIU9IxAqmCkEok0iCFWm0YU15IZISD/OCjMhInIJIo4jUynCKYQmAQ4dGkcR4YHGjAeZxtbW9/V+5qAdYI2muiLUdccldLsTsOuCvd4E7Llgvz8B+3NxO/+JKRM+RikJ9fE8rATAFOPEZVQqU2jFRUEIM1Aat4mARLIw5g4hMHd1ysGu6yCAOrSsA2on2Qwi33ZDtAwn09tTFIex100mbdlzIxwdTcCjzDZ8GqVx2+Z/zBoRqYlsDqTXIHV7YialbQZu+P/j8pfc4HY/aHwQQWPeeyLoNeY6Pz7cnWQ97NoumHsu4J1RHSQgiIn7VGMiTGpmrMwbb1jpr0SNlUnjxE3AZE8muRM39zBumD2XDQVsJSd9O7BNouan1VYluyhX9WgZms9svVh+kpktV3Z32rxwtrVZfr659f5Fcef1eN8te4+8x966V/ZeejtexTv0Tj3qffV+5xZzS/kv+e/5H/mfI+pCbmzz0Js5+V9/AMdrRmM=</latexit>

⌦r

||⌦r

fr ||2F ,⌦subject to uIp

⌦r

vIp

53

Learning with Adaptive Regularization
Given a pair of constants 0 < u  v, define the following thresholding
operator:T[u,v] (x) := max{u, min{v, x}}<latexit sha1_base64="s4CtNfsyVJLHUN2HqW1F6GgIaoQ=">AAAF03icfVRNbxMxEHWhgRK+WjhyWRFVKghV2YIEh1aqRCuFA2ppm7ZSNqoc7yRZ1WtvbefT2gviypEr/Af+Df+GcRqaxAtYijI773neW8+sWxlPtKlWfy3dur1cunN35V75/oOHjx6vrj051bKnGNSZ5FKdt6gGngiom8RwOM8U0LTF4ax1+d7hZ31QOpHixIwyaKa0I5J2wqjBVKMabAe9iMNV0L9YrVQ3q5MVFINwGlTIdB1erC3/jGLJeikIwzjVuhFWM9O0VJmEccjLUU9DRtkl7UADQ0FT0E078ZwH65iJg7ZU+BMmmGTnd1iaaj1KW8hMqelqH3PJf2Gmm+Z+zqnoPJisBQzJRkquF3e4usV0o2fa75o2EVnPgGDXb9Hu8cDIwJ1uECcKmOEjDChTCR5EwLpUUWawB+X1wJfI6Fjm5QWJlF6CkjLFk6KCAUcDAgZMpikVsY0SEec2cntbLfsh99ABJJ2u+UNo24FPGA5n4NAHR6MZOPLB8XgGjgu6g/9o6kxMUUa5PS7CRgHMMU58Rq02h9Z8FJTCgbJRnyrIdMKl8AgxfkZzBfb8AjG04coVYG6ScRDFji9xhZzc7sxRPMb+MJu1Zd9XODqagUe5a/g8ymTf+T9OOim9UHkBZDcg83uCk9LHgZu8//T4q754phx6kEKnWD1T7Abzix8f7s1cT7q2B/idK/iIqYMMFEXdlzaiCq3hWOF/9MpFfyXayKCNE98Auqcz79T3zmUnBtc8DCIXefbdwHapKU6rO5W8ETZRFobG4mO+UQlf5HjLhf6dVgxOtzbD15tbn95Udren990KeUaekw0Skrdkl9TIIakTRiT5Rr6TH6V6yZY+l75cU28tTfc8JQur9PU3DLodvg==</latexit><latexit sha1_base64="ijiExb4GF+9dH7z5X6TEaHB/t44=">AAAF/nicfZTNbhMxEMfd0kAJH23hyGVFVKlFUZWUSnCgUiVaKRxQS5t+SNkocryTxOquvfV6k00sSzwHD8ANIXHiyBXegLfBTtMm8QKWoszO72/P2DN2Ow5pIiuV3wuLd5YKd+8t3y8+ePjo8crq2pOzhKeCwCnhIRcXbZxASBmcSipDuIgF4Kgdwnn78q3l530QCeWsLocxNCPcZbRDCZbG1Vrd8SMse+22quuWaqTlflNvZJueH0AHrjwDM1+lZWNQ5qt+2ct87evWaqmyVRkPL29UJ0YJTcZRa23pmx9wkkbAJAlxkjSqlVg2FRaSkhB00U8TiDG5xF1oGJPhCJKmGm9Pe+vGE3gdLsyPSW/snZ2hcJQkw6htlHYzicus819M9iLt+myURHvjMceMWHIeJvMz7Lp5dyOVnddNRVmcSmDkehedNPQk92whvIAKIDIcGgMTQc1BeKSHBSbSlKu47rkhYjziujgXIsKXIDiPzElhRiA0CTAYEB5FmAXKpyzQ6qa+77RDB0C7PXkj6KiBK8iyKcxcOBxO4dCFo9EUjnJxB/+JmcRsQgkO1UkeSwEwo6i7ilpthtZcCkKYhlJ+HwuIExpy5ggCc+NmFth3FxhfC7sAsZ1sGpHtuiGujEar3RmJozjI4mlZDtwIx8dTeKxtwWcp4X2b/wntRrgldA6SW0jcmphO6ZuGG+9/cvwVN3gsLD2MoJtfPRbklrmLnxztT7MeV20fzD0X8N64DmMQ2MR9oXwsTGqmrcy/X7bWX4XKlyaNupuAyR5Pc8du7iHvBmCLZwzfWk76tmF7WOa71Z6KblSbJixkUplPvVGqbtpXruq+aXnjbHur+nJr+8NOae/N5L1bRs/Qc7SBqugV2kM1dIROEUGf0A/0E/0qfCx8LnwpfL2WLi5M5jxFc6Pw/Q9pMi93</latexit>T[2,4] (z)
6
5
4
3
2
1
0

0

1

2

3

4

5

6 z

54

Learning with Adaptive Regularization
Given a pair of constants 0 < u  v, define the following thresholding
operator:T[u,v] (x) := max{u, min{v, x}}<latexit sha1_base64="s4CtNfsyVJLHUN2HqW1F6GgIaoQ=">AAAF03icfVRNbxMxEHWhgRK+WjhyWRFVKghV2YIEh1aqRCuFA2ppm7ZSNqoc7yRZ1WtvbefT2gviypEr/Af+Df+GcRqaxAtYijI773neW8+sWxlPtKlWfy3dur1cunN35V75/oOHjx6vrj051bKnGNSZ5FKdt6gGngiom8RwOM8U0LTF4ax1+d7hZ31QOpHixIwyaKa0I5J2wqjBVKMabAe9iMNV0L9YrVQ3q5MVFINwGlTIdB1erC3/jGLJeikIwzjVuhFWM9O0VJmEccjLUU9DRtkl7UADQ0FT0E078ZwH65iJg7ZU+BMmmGTnd1iaaj1KW8hMqelqH3PJf2Gmm+Z+zqnoPJisBQzJRkquF3e4usV0o2fa75o2EVnPgGDXb9Hu8cDIwJ1uECcKmOEjDChTCR5EwLpUUWawB+X1wJfI6Fjm5QWJlF6CkjLFk6KCAUcDAgZMpikVsY0SEec2cntbLfsh99ABJJ2u+UNo24FPGA5n4NAHR6MZOPLB8XgGjgu6g/9o6kxMUUa5PS7CRgHMMU58Rq02h9Z8FJTCgbJRnyrIdMKl8AgxfkZzBfb8AjG04coVYG6ScRDFji9xhZzc7sxRPMb+MJu1Zd9XODqagUe5a/g8ymTf+T9OOim9UHkBZDcg83uCk9LHgZu8//T4q754phx6kEKnWD1T7Abzix8f7s1cT7q2B/idK/iIqYMMFEXdlzaiCq3hWOF/9MpFfyXayKCNE98Auqcz79T3zmUnBtc8DCIXefbdwHapKU6rO5W8ETZRFobG4mO+UQlf5HjLhf6dVgxOtzbD15tbn95Udren990KeUaekw0Skrdkl9TIIakTRiT5Rr6TH6V6yZY+l75cU28tTfc8JQur9PU3DLodvg==</latexit><latexit sha1_base64="ijiExb4GF+9dH7z5X6TEaHB/t44=">AAAF/nicfZTNbhMxEMfd0kAJH23hyGVFVKlFUZWUSnCgUiVaKRxQS5t+SNkocryTxOquvfV6k00sSzwHD8ANIXHiyBXegLfBTtMm8QKWoszO72/P2DN2Ow5pIiuV3wuLd5YKd+8t3y8+ePjo8crq2pOzhKeCwCnhIRcXbZxASBmcSipDuIgF4Kgdwnn78q3l530QCeWsLocxNCPcZbRDCZbG1Vrd8SMse+22quuWaqTlflNvZJueH0AHrjwDM1+lZWNQ5qt+2ct87evWaqmyVRkPL29UJ0YJTcZRa23pmx9wkkbAJAlxkjSqlVg2FRaSkhB00U8TiDG5xF1oGJPhCJKmGm9Pe+vGE3gdLsyPSW/snZ2hcJQkw6htlHYzicus819M9iLt+myURHvjMceMWHIeJvMz7Lp5dyOVnddNRVmcSmDkehedNPQk92whvIAKIDIcGgMTQc1BeKSHBSbSlKu47rkhYjziujgXIsKXIDiPzElhRiA0CTAYEB5FmAXKpyzQ6qa+77RDB0C7PXkj6KiBK8iyKcxcOBxO4dCFo9EUjnJxB/+JmcRsQgkO1UkeSwEwo6i7ilpthtZcCkKYhlJ+HwuIExpy5ggCc+NmFth3FxhfC7sAsZ1sGpHtuiGujEar3RmJozjI4mlZDtwIx8dTeKxtwWcp4X2b/wntRrgldA6SW0jcmphO6ZuGG+9/cvwVN3gsLD2MoJtfPRbklrmLnxztT7MeV20fzD0X8N64DmMQ2MR9oXwsTGqmrcy/X7bWX4XKlyaNupuAyR5Pc8du7iHvBmCLZwzfWk76tmF7WOa71Z6KblSbJixkUplPvVGqbtpXruq+aXnjbHur+nJr+8NOae/N5L1bRs/Qc7SBqugV2kM1dIROEUGf0A/0E/0qfCx8LnwpfL2WLi5M5jxFc6Pw/Q9pMi93</latexit>

The Euclidean projection onto C has the following closed form solution:
T
p
f
f
⌦
=
Q⇤Q
⌦
2
S
Theorem: Let r
with eigendecomposition as r
. Thenfr ) = QT[u,v] (⇤)QT
ProjC (⌦.
<latexit sha1_base64="j73KwWKUhWZknInarU83VTyoloE=">AAAF03icfVTLbhMxFHWhgRJeLSzZjIgqFYSqTEGCBZUqtZXCArW0TVspE1Uez00yqh9T23las0FsWbKFf+Bv+BvsaWgSD2Apyp17ju8543vHcUZTpev1X0u3bi9X7txduVe9/+Dho8era09OlehLAk0iqJDnMVZAUw5NnWoK55kEzGIKZ/HlrsPPBiBVKviJHmfQZrjL005KsLapVsSw7hFMzW5+sVqrb9aLFZSDcBrU0HQdXqwt/4wSQfoMuCYUK9UK65luGyx1Sijk1aivIMPkEnehZUOOGai2KTznwbrNJEFHSPvjOiiy8zsMZkqNWWyZzqPyMZf8F6Z7LPdzTkXlQbEWMEvWQlC1uMPVLadbfd151zYpz/oaOLl+i06fBloE7nSDJJVANB3bABOZ2oMISA9LTLTtQXU98CUyPBF5dUGC4UuQQjB7UpgToNYAhyERjGGemCjlSW6KtsWx+ZB76BDSbk//IXTM0CeMRjNw5IPj8Qwc++BkMgMnJd3hfzRVxqeom7TjMqwlwBzjxGc0GnNow0dBSjtQJhpgCZlKqeAeIbGf0VyBPb9AAh24cgWIm2Q7iHzbl7iynNxsz1E8xv4om7Vl31c4OpqBR7lr+DxKxMD5P067DF/IvASSG5D4PbGTMrADV7z/9PjrvngmHXrAoFuunklyg/nFjw/3Zq6Lru2B/c4lfLSpgwwktrovTYSltWbHyv5Hr1z0V6KJtLVx4huw7vHMO/a9U9FNwDXPBpGLPPtuYHtYl6fVnUreCttWFkba2Md8oxa+cLdc6N9p5eB0azN8vbn16U1t5/30vltBz9BztIFC9BbtoAY6RE1EkEDf0Hf0o9KsmMrnypdr6q2l6Z6naGFVvv4GEPYe2g==</latexit><latexit sha1_base64="0kaDhV/BL36VjSCFuiPQMj3iOMc=">AAAF5HicfVRLbxMxEHZLAyW8WrjBZUVUqSBUJQUJDlSqRCuFA2rpW8qGyPFOEqtee2s7T2slfgA3xJUjVxD/hn/DbBqaxAtYijI73+f5xp7xNBPBjS2Xfy0sXlsqXL+xfLN46/adu/dWVu+fGNXVDI6ZEkqfNakBwSUcW24FnCUaaNwUcNo8f5Phpz3Qhit5ZIcJ1GPalrzFGbXoaqw8DPs8AstFBC5MtE5DLoPwcH/nQ9JYKZU3yuMV5I3KxCiRydpvrC79DCPFujFIywQ1plYpJ7buqLacCUiLYddAQtk5bUMNTUljMHU3PkQarKEnClpK40/aYOyd3eFobMwwbiIzprZjfCxz/guznTj1fZmKSYPxmsOQbJUSZn5HFjfvrnVt61XdcZl0LUh2eYpWVwRWBdl1BxHXwKwYokGZ5ngRAetQTZnFohTXAl8ioSOVFuckYnoOWqkYb4pKBgITkNBnKo6pjBzWK0pdmO1tNt3b1EP7wNsd+4fQcn2fMBhMwYEPDodTcOiDo9EUHOV0+//RNImcoIwKd5iHrQaYYRz5jGp1Bq36KGiNDeXCHtWQGC6U9AgRvquZADt+gAhacJEFYFknYyPKLV/iAjmp25qheIzdQTIty66vcHAwBQ/SrOCzKFO9LP9D3o5pQ6c5kF2BzK8JdkoPG258/sn1l33x7Jm7cC+Gdj56otkV5gfHqTDNely1HcB3ruEduvYS0BR1n7qQakwN2wr/w2eZ9VeiCy2mceQngNnTae7Uz12oNo6rdGyEmeWlnzVsh9p8t2a3ktYqdZSFgXX4ma6XKk9SnHIVf6bljZPNjcrzjc33L0rbryfzbpk8Io/JOqmQl2SbVMk+OSaMfCTfyHfyo9AqfCp8Lny5pC4uTPY8IHOr8PU3kRslUw==</latexit><latexit sha1_base64="hq9/1KurJVgeqvAqNpsRwsQ+Sr0=">AAAF6XicfVRLbxMxEHZLAyW8UjhyYEVUqSBUJQUJDlSqRCsFCdRX+pCyIXK8k8SK1956naflIz+AG+LKkRuCH8O/wU5Dk3gBS1Fm5/s833hm7GbCaKpKpV9Ly9dWctdvrN7M37p95+69wtr901T0JIETIpiQ502cAqMcThRVDM4TCThuMjhrdt84/KwPMqWCV9UogXqM25y2KMHKuhqFR+GARqAoi0CHiZQm2A4Ow3c2QISDww/VRqFY2ixNVpA1ylOjiKbroLG28j2MBOnFwBVhOE1r5VKi6hpLRQkDkw97KSSYdHEbatbkOIa0ricnMcG69URBS0j74yqYeOd3aByn6ShuWmaMVSf1Mef8F6Y6sfF9TiU1wWQtYJashGDp4g4XN+uu9VTrVV1TnvQUcHJ5ilaPBUoEruZBRCUQxUbWwERSW4iAdLDERNnO5NcDXyLBY2HyCxIx7oIUIraVwpwAswlwGBARx5hHOqQ8Mjp0e5tN/dZ46ABou6P+EFp64BOGwxk49MHRaAaOfHA8noHjjO7gP5ppwqcowUwfZ2ElAeYYVZ9RqcyhFR8FKe1A6bCPJSQpZYJ7hMherrkAu36ACFpw4QIQN8l2EPm2L3FhOUZvz1E8xt4wmbVlz1c4OpqBR8Y1fB4lou/yP6btGDekyYDkCiR+T+yk9O3ATc4/LX/JF3d3XYf7MbSz0RNJrjA/+PHB7izrSdd2wd5zCe+taz8Bia3uUx1iaVOzY2X/w2fO+itRh8qmUfUTsNnjWe7Yz52Jtn2zzMQIneWl7wa2g1V2Wl1VTK1ct7IwVNp+mo1i+Ymxr1zZf9OyxunWZvn55tbhi+LO6+l7t4oeosdoA5XRS7SDKugAnSCCPqJv6Af6mevmPuU+575cUpeXpnseoIWV+/ob1l4mrw==</latexit><latexit sha1_base64="R1wIqvkWUkJJ76/nIgmZfyCzr+w=">AAAGF3icfVTNbhMxEHZLAyX8tXDksiKq1KKqSgoSHKhUqa0UJKqmbfojZUPkeCeJ6e5663V+LT8IrwEPwA1x5dgrvAiz27RJvIClKLPzfZ757BlPM/J5rIrFq7n5Owu5u/cW7+cfPHz0+MnS8tPTWHQlgxMmfCHPmzQGn4dworjy4TySQIOmD2fNi50EP+uBjLkIq2oYQT2g7ZC3OKMKXY2lfVfBQOmKFJ9MQ7sBVR1Gfb1jzKrb5x4o7nug3UhKs+ZsOYcpo9nUVWTXuuu9OvI+YDqPrh1+rDaWCsWNYrqcrFEaGwUyXpXG8sIX1xOsG0ComE/juFYqRqquqVSc+WDybjeGiLIL2oYamiENIK7r9NzGWUGP57SExF+onNQ7vUPTII6HQROZiezYxhLnvzDVCYztS7LExknXDIZkJYQfz+5I4mbdta5qva1rHkZdBSG7PkWr6ztKOEmFHI9LYMofokGZ5HgRDutQSZnCOuZXHDtFREfC5GdSBPQCpBAB3hQNGfgoIIQ+E0FAQ0+7PPSMvqnke2OhfeDtjrohtHTfJgwGE3Bgg8PhBBza4Gg0AUeZvP3/5Iyj0Eza8zgLKwkwxajajHJ5Ci3bKEiJDaXdHpUQxdwXoUXw8ClOBdi1A3jQgsskAEs6GRsx3LJTXCLH6K0pisXYG0STsuzZGY6OJuCRSQo+jTLRS/Qf83ZAG9JkQHYLMrsm2Ck9bLj0/OPrL9rJk/ev3YMA2tnokWS3mB38uLI7UZ1WbRfwnUvYR9dBBJJi3pfapRKlYVvhv7ueWH8lalehjKotANXTiXZqa/dFG+eYSQ03sSz5ScN2qMp2a3Irplaq6+sZiZ9mtVBaMzjlSvZMyxqnmxulVxubh68L2+/G826RPCcvyCopkTdkm5RJhZwQRr6SK/KL/M59zn3Lfc/9uKbOz433PCMzK/fzD3SLOrU=</latexit>

Simple: Apply eigendecomposition and then threshold the spectrum

55

Learning with Adaptive Regularization
Given a pair of constants 0 < u  v, define the following thresholding
operator:T[u,v] (x) := max{u, min{v, x}}<latexit sha1_base64="s4CtNfsyVJLHUN2HqW1F6GgIaoQ=">AAAF03icfVRNbxMxEHWhgRK+WjhyWRFVKghV2YIEh1aqRCuFA2ppm7ZSNqoc7yRZ1WtvbefT2gviypEr/Af+Df+GcRqaxAtYijI773neW8+sWxlPtKlWfy3dur1cunN35V75/oOHjx6vrj051bKnGNSZ5FKdt6gGngiom8RwOM8U0LTF4ax1+d7hZ31QOpHixIwyaKa0I5J2wqjBVKMabAe9iMNV0L9YrVQ3q5MVFINwGlTIdB1erC3/jGLJeikIwzjVuhFWM9O0VJmEccjLUU9DRtkl7UADQ0FT0E078ZwH65iJg7ZU+BMmmGTnd1iaaj1KW8hMqelqH3PJf2Gmm+Z+zqnoPJisBQzJRkquF3e4usV0o2fa75o2EVnPgGDXb9Hu8cDIwJ1uECcKmOEjDChTCR5EwLpUUWawB+X1wJfI6Fjm5QWJlF6CkjLFk6KCAUcDAgZMpikVsY0SEec2cntbLfsh99ABJJ2u+UNo24FPGA5n4NAHR6MZOPLB8XgGjgu6g/9o6kxMUUa5PS7CRgHMMU58Rq02h9Z8FJTCgbJRnyrIdMKl8AgxfkZzBfb8AjG04coVYG6ScRDFji9xhZzc7sxRPMb+MJu1Zd9XODqagUe5a/g8ymTf+T9OOim9UHkBZDcg83uCk9LHgZu8//T4q754phx6kEKnWD1T7Abzix8f7s1cT7q2B/idK/iIqYMMFEXdlzaiCq3hWOF/9MpFfyXayKCNE98Auqcz79T3zmUnBtc8DCIXefbdwHapKU6rO5W8ETZRFobG4mO+UQlf5HjLhf6dVgxOtzbD15tbn95Udren990KeUaekw0Skrdkl9TIIakTRiT5Rr6TH6V6yZY+l75cU28tTfc8JQur9PU3DLodvg==</latexit><latexit sha1_base64="ijiExb4GF+9dH7z5X6TEaHB/t44=">AAAF/nicfZTNbhMxEMfd0kAJH23hyGVFVKlFUZWUSnCgUiVaKRxQS5t+SNkocryTxOquvfV6k00sSzwHD8ANIXHiyBXegLfBTtMm8QKWoszO72/P2DN2Ow5pIiuV3wuLd5YKd+8t3y8+ePjo8crq2pOzhKeCwCnhIRcXbZxASBmcSipDuIgF4Kgdwnn78q3l530QCeWsLocxNCPcZbRDCZbG1Vrd8SMse+22quuWaqTlflNvZJueH0AHrjwDM1+lZWNQ5qt+2ct87evWaqmyVRkPL29UJ0YJTcZRa23pmx9wkkbAJAlxkjSqlVg2FRaSkhB00U8TiDG5xF1oGJPhCJKmGm9Pe+vGE3gdLsyPSW/snZ2hcJQkw6htlHYzicus819M9iLt+myURHvjMceMWHIeJvMz7Lp5dyOVnddNRVmcSmDkehedNPQk92whvIAKIDIcGgMTQc1BeKSHBSbSlKu47rkhYjziujgXIsKXIDiPzElhRiA0CTAYEB5FmAXKpyzQ6qa+77RDB0C7PXkj6KiBK8iyKcxcOBxO4dCFo9EUjnJxB/+JmcRsQgkO1UkeSwEwo6i7ilpthtZcCkKYhlJ+HwuIExpy5ggCc+NmFth3FxhfC7sAsZ1sGpHtuiGujEar3RmJozjI4mlZDtwIx8dTeKxtwWcp4X2b/wntRrgldA6SW0jcmphO6ZuGG+9/cvwVN3gsLD2MoJtfPRbklrmLnxztT7MeV20fzD0X8N64DmMQ2MR9oXwsTGqmrcy/X7bWX4XKlyaNupuAyR5Pc8du7iHvBmCLZwzfWk76tmF7WOa71Z6KblSbJixkUplPvVGqbtpXruq+aXnjbHur+nJr+8NOae/N5L1bRs/Qc7SBqugV2kM1dIROEUGf0A/0E/0qfCx8LnwpfL2WLi5M5jxFc6Pw/Q9pMi93</latexit>

The Euclidean projection onto C has the following closed form solution:
T
p
f
f
⌦
=
Q⇤Q
⌦
2
S
Theorem: Let r
with eigendecomposition as r
. Thenfr ) = QT[u,v] (⇤)QT
ProjC (⌦.
<latexit sha1_base64="j73KwWKUhWZknInarU83VTyoloE=">AAAF03icfVTLbhMxFHWhgRJeLSzZjIgqFYSqTEGCBZUqtZXCArW0TVspE1Uez00yqh9T23las0FsWbKFf+Bv+BvsaWgSD2Apyp17ju8543vHcUZTpev1X0u3bi9X7txduVe9/+Dho8era09OlehLAk0iqJDnMVZAUw5NnWoK55kEzGIKZ/HlrsPPBiBVKviJHmfQZrjL005KsLapVsSw7hFMzW5+sVqrb9aLFZSDcBrU0HQdXqwt/4wSQfoMuCYUK9UK65luGyx1Sijk1aivIMPkEnehZUOOGai2KTznwbrNJEFHSPvjOiiy8zsMZkqNWWyZzqPyMZf8F6Z7LPdzTkXlQbEWMEvWQlC1uMPVLadbfd151zYpz/oaOLl+i06fBloE7nSDJJVANB3bABOZ2oMISA9LTLTtQXU98CUyPBF5dUGC4UuQQjB7UpgToNYAhyERjGGemCjlSW6KtsWx+ZB76BDSbk//IXTM0CeMRjNw5IPj8Qwc++BkMgMnJd3hfzRVxqeom7TjMqwlwBzjxGc0GnNow0dBSjtQJhpgCZlKqeAeIbGf0VyBPb9AAh24cgWIm2Q7iHzbl7iynNxsz1E8xv4om7Vl31c4OpqBR7lr+DxKxMD5P067DF/IvASSG5D4PbGTMrADV7z/9PjrvngmHXrAoFuunklyg/nFjw/3Zq6Lru2B/c4lfLSpgwwktrovTYSltWbHyv5Hr1z0V6KJtLVx4huw7vHMO/a9U9FNwDXPBpGLPPtuYHtYl6fVnUreCttWFkba2Md8oxa+cLdc6N9p5eB0azN8vbn16U1t5/30vltBz9BztIFC9BbtoAY6RE1EkEDf0Hf0o9KsmMrnypdr6q2l6Z6naGFVvv4GEPYe2g==</latexit><latexit sha1_base64="0kaDhV/BL36VjSCFuiPQMj3iOMc=">AAAF5HicfVRLbxMxEHZLAyW8WrjBZUVUqSBUJQUJDlSqRCuFA2rpW8qGyPFOEqtee2s7T2slfgA3xJUjVxD/hn/DbBqaxAtYijI73+f5xp7xNBPBjS2Xfy0sXlsqXL+xfLN46/adu/dWVu+fGNXVDI6ZEkqfNakBwSUcW24FnCUaaNwUcNo8f5Phpz3Qhit5ZIcJ1GPalrzFGbXoaqw8DPs8AstFBC5MtE5DLoPwcH/nQ9JYKZU3yuMV5I3KxCiRydpvrC79DCPFujFIywQ1plYpJ7buqLacCUiLYddAQtk5bUMNTUljMHU3PkQarKEnClpK40/aYOyd3eFobMwwbiIzprZjfCxz/guznTj1fZmKSYPxmsOQbJUSZn5HFjfvrnVt61XdcZl0LUh2eYpWVwRWBdl1BxHXwKwYokGZ5ngRAetQTZnFohTXAl8ioSOVFuckYnoOWqkYb4pKBgITkNBnKo6pjBzWK0pdmO1tNt3b1EP7wNsd+4fQcn2fMBhMwYEPDodTcOiDo9EUHOV0+//RNImcoIwKd5iHrQaYYRz5jGp1Bq36KGiNDeXCHtWQGC6U9AgRvquZADt+gAhacJEFYFknYyPKLV/iAjmp25qheIzdQTIty66vcHAwBQ/SrOCzKFO9LP9D3o5pQ6c5kF2BzK8JdkoPG258/sn1l33x7Jm7cC+Gdj56otkV5gfHqTDNely1HcB3ruEduvYS0BR1n7qQakwN2wr/w2eZ9VeiCy2mceQngNnTae7Uz12oNo6rdGyEmeWlnzVsh9p8t2a3ktYqdZSFgXX4ma6XKk9SnHIVf6bljZPNjcrzjc33L0rbryfzbpk8Io/JOqmQl2SbVMk+OSaMfCTfyHfyo9AqfCp8Lny5pC4uTPY8IHOr8PU3kRslUw==</latexit><latexit sha1_base64="hq9/1KurJVgeqvAqNpsRwsQ+Sr0=">AAAF6XicfVRLbxMxEHZLAyW8UjhyYEVUqSBUJQUJDlSqRCsFCdRX+pCyIXK8k8SK1956naflIz+AG+LKkRuCH8O/wU5Dk3gBS1Fm5/s833hm7GbCaKpKpV9Ly9dWctdvrN7M37p95+69wtr901T0JIETIpiQ502cAqMcThRVDM4TCThuMjhrdt84/KwPMqWCV9UogXqM25y2KMHKuhqFR+GARqAoi0CHiZQm2A4Ow3c2QISDww/VRqFY2ixNVpA1ylOjiKbroLG28j2MBOnFwBVhOE1r5VKi6hpLRQkDkw97KSSYdHEbatbkOIa0ricnMcG69URBS0j74yqYeOd3aByn6ShuWmaMVSf1Mef8F6Y6sfF9TiU1wWQtYJashGDp4g4XN+uu9VTrVV1TnvQUcHJ5ilaPBUoEruZBRCUQxUbWwERSW4iAdLDERNnO5NcDXyLBY2HyCxIx7oIUIraVwpwAswlwGBARx5hHOqQ8Mjp0e5tN/dZ46ABou6P+EFp64BOGwxk49MHRaAaOfHA8noHjjO7gP5ppwqcowUwfZ2ElAeYYVZ9RqcyhFR8FKe1A6bCPJSQpZYJ7hMherrkAu36ACFpw4QIQN8l2EPm2L3FhOUZvz1E8xt4wmbVlz1c4OpqBR8Y1fB4lou/yP6btGDekyYDkCiR+T+yk9O3ATc4/LX/JF3d3XYf7MbSz0RNJrjA/+PHB7izrSdd2wd5zCe+taz8Bia3uUx1iaVOzY2X/w2fO+itRh8qmUfUTsNnjWe7Yz52Jtn2zzMQIneWl7wa2g1V2Wl1VTK1ct7IwVNp+mo1i+Ymxr1zZf9OyxunWZvn55tbhi+LO6+l7t4oeosdoA5XRS7SDKugAnSCCPqJv6Af6mevmPuU+575cUpeXpnseoIWV+/ob1l4mrw==</latexit><latexit sha1_base64="R1wIqvkWUkJJ76/nIgmZfyCzr+w=">AAAGF3icfVTNbhMxEHZLAyX8tXDksiKq1KKqSgoSHKhUqa0UJKqmbfojZUPkeCeJ6e5663V+LT8IrwEPwA1x5dgrvAiz27RJvIClKLPzfZ757BlPM/J5rIrFq7n5Owu5u/cW7+cfPHz0+MnS8tPTWHQlgxMmfCHPmzQGn4dworjy4TySQIOmD2fNi50EP+uBjLkIq2oYQT2g7ZC3OKMKXY2lfVfBQOmKFJ9MQ7sBVR1Gfb1jzKrb5x4o7nug3UhKs+ZsOYcpo9nUVWTXuuu9OvI+YDqPrh1+rDaWCsWNYrqcrFEaGwUyXpXG8sIX1xOsG0ComE/juFYqRqquqVSc+WDybjeGiLIL2oYamiENIK7r9NzGWUGP57SExF+onNQ7vUPTII6HQROZiezYxhLnvzDVCYztS7LExknXDIZkJYQfz+5I4mbdta5qva1rHkZdBSG7PkWr6ztKOEmFHI9LYMofokGZ5HgRDutQSZnCOuZXHDtFREfC5GdSBPQCpBAB3hQNGfgoIIQ+E0FAQ0+7PPSMvqnke2OhfeDtjrohtHTfJgwGE3Bgg8PhBBza4Gg0AUeZvP3/5Iyj0Eza8zgLKwkwxajajHJ5Ci3bKEiJDaXdHpUQxdwXoUXw8ClOBdi1A3jQgsskAEs6GRsx3LJTXCLH6K0pisXYG0STsuzZGY6OJuCRSQo+jTLRS/Qf83ZAG9JkQHYLMrsm2Ck9bLj0/OPrL9rJk/ev3YMA2tnokWS3mB38uLI7UZ1WbRfwnUvYR9dBBJJi3pfapRKlYVvhv7ueWH8lalehjKotANXTiXZqa/dFG+eYSQ03sSz5ScN2qMp2a3Irplaq6+sZiZ9mtVBaMzjlSvZMyxqnmxulVxubh68L2+/G826RPCcvyCopkTdkm5RJhZwQRr6SK/KL/M59zn3Lfc/9uKbOz433PCMzK/fzD3SLOrU=</latexit>

Simple: Apply eigendecomposition and then threshold the spectrumTr(⌦r W ⌦c W T )min<latexit sha1_base64="26QU4fsGr7YU80QGsmAK457/ct8=">AAAGP3icfVRLb9NAEHaggZLyaOHIZUVU1KJSJQUJDlSqRCulB9TSpg8pDtF6PUlM/cp6nddq/xT/gJ+B+AHcEFduzDppk6yBlWzPzvftzOzMeJzY9xJRqXwr3Lq9VLxzd/leaeX+g4ePVtcenydRyhmcsciP+KVDE/C9EM6EJ3y4jDnQwPHhwrl6r/GLPvDEi8K6GMXQDGgn9NoeowJVrdX0ObEDL2xJO+ZcEbuXUpfYgm/oPbnANyMXn+qbhLwkru1HHRdEhm1u2T1NtgUMhUxS5zMwQUSkJiZSctiKCRKBQU9/+bXcR6C1Wq5sV7JF8kJ1KpSt6TpurS19sd2IpQGEgvk0SRrVSiyaknLhMR9UyU4TiCm7oh1ooBjSAJKmzPKjyDpqXNKOOD6hIJl2/oSkQZKMAgeZARXdxMS08l+Y6AbK1GkviSLZWsCQLKLITxZPaLt5dSMV7bdN6YVxKiBkk1u0Ux9zTHQliethQoU/QoEy7mEiCOtSTpnAepfWiekipuNIlRZcBPQKeBQFmCkaMvAxgBAGLAoCGrrS9kJXSVufdRx5qAx0AF6nK64JbTkwCcPhDBya4Gg0A0cmOB7PwHHO7+A/PpM4nKKM+vI0DwsOMMeom4xabQ6tmShwjg0l7T7lECeeH4UGwcVfds7AvmnAhTb0tAGmOxkbMdw1XfSQo+TuHMVgHAzjWVkOTA8nJzPwROmCz6Ms6uv4T71OQFtc5UB2AzKzJtgpfWy47P7T9FdM53qISPsogE7eOo6SG8w0fnq8P4s6q9o+4H/O4QOqjmLgFP2+kDblGBq2FX7tLS39lShxgClZNwPA6OksdmrGPhluKhNsLRnh64btUpHvVp0V1ag25WQW4lZtlKubCqdc1ZxpeeF8Z7v6anvn4+vy3rvpvFu2nlrPrA2rar2x9qyadWydWcz6XigUSoWV4tfij+LP4q8J9VZheuaJtbCKv/8AopJGIA==</latexit>⌦rd log det(⌦r ),subject to uIp

⌦r

vIp

⌦⇤r = QT[u,v] (d/r)QT

Solution:
T
T
T
Qdiag(r)Q
=
W
⌦
W
W
⌦
W
c
c
where
is the spectral decomposition of<latexit sha1_base64="+hHwFN2S3UiKc205gnj32zKOxow=">AAAF93icfVTLbhMxFHVLAyW8WliyGRFVaqtSkoIECypVopXCAvWVPqRMiBzPTTLqjD31OE/LX8EHsENsYccWvoK/4TpNm8QDWIpy557je8743nEjicJUFYu/5+ZvLeRu31m8m793/8HDR0vLj09T0ZEMTpiIhDxv0BSikMOJClUE54kEGjciOGtcvLP4WRdkGgpeUYMEajFt8bAZMqowVV967idSflz3tr1DP6aq3WjoiqnramejWzOrwYurZFNLs3b4sVJfKhQ3i6PlZYPSOCiQ8TqoLy989wPBOjFwxSKaptVSMVE1TaUKWQQm73dSSCi7oC2oYshpDGlNj97LeCuYCbymkPjjyhtlp3doGqfpIG4g09pMXcwm/4WpdmzcnFVJjTdaMxiSlRBROrvD1s2mqx3VfFPTIU86Cji7eotmJ/KU8GwHvCCUwFQ0wIAyGeJBeKxNJWUK+5Rf8VyJhA6Fyc9IxPQCpBAxnhTlDCI0wKHHRBxTHmg/5IHR1+18bxy0B2Grra4JTd1zCf3+BOy74GAwAQcuOBxOwGFGt/cfzTThY5TRSB9nYSUBphgVl1EuT6FlFwUpcaC036USkjSMBHcIAX5qUwV23QIBNOHSFmB2knEQ+bYrcYkco7enKA5jr59M2rLnKhwdTcAjYxs+jTLRtf6Pw1ZM69JkQHYDMrcnOCldHLjR+4+Pv+iK4yWA6H4MrWz1RLIbzC1+fLA7cT3q2i7gdy7hA6b2E5AUdde1TyVaw7HCf3/DRn8lal+hjYprAN3TiXfqeo9EKwDbPAx8Gzn27cC2qcpOqz0VUy3VUBb6SuOjWS2U1gzeciX3TssGp1ubpZebW4evCjtvx/fdInlKnpFVUiKvyQ4pkwNyQhj5RH6Qn+RXbpD7nPuS+3pFnZ8b73lCZlbu2x8rfyxP</latexit><latexit sha1_base64="yIRV8XfxKOYPbT0YG7OMIqD08K4=">AAAF8XicfVTNbhMxEHahgRL+WjhyWRFVtAhVSUGCA5Uq0UrhgPqXNpWyaeR4J4nVXXtrO7+WH4EH4Ia4coMrPAhvg52EJvEClqLMzvd55vPM2M00plIVi7+Wbtxczt26vXInf/fe/QcPV9cenUneFQROCY+5OG9iCTFlcKqoiuE8FYCTZgzV5uU7h1d7ICTlrKKGKdQT3Ga0RQlW1tVYfXYUKhgoHVHcNhthglWn2dLCbB5dVIKdoBqmggTVi0pjtVDcKo5XkDVKU6OApuuwsbb8LYw46SbAFImxlLVSMVV1jYWiJAaTD7sSUkwucRtq1mQ4AVnX4xOZYN16oqDFhf0xFYy98zs0TqQcJk3LdJKljznnvzDVSYzvc1mkCcZrAbNkxXksF3e4uFl3ratab+qasrSrgJHJKVrdOFA8cLUPIiqAqHhoDUwEtYUISAcLTJTtUH498FOkeMRNfiFFgi9BcJ7YSmFGILYCGPQJTxLMIh1SFhk96WJTvzce2gfa7qg/hJbu+4TBYAYOfHA4nIFDHxyNZuAok7f/n5wyZVOU4FifZGElAOYYFZ9RLs+hZR8FIexA6bCHBaSSxpx5hMhesrkAe36ACFpw5QIQN8l2ENmOn+LKcozemaN4jP1BOmvLvp/h+HgGHhvX8HmU8J7Tf0LbCW4IkwHJNUj8nthJ6dmBG59/Wv6inzwVDj1IoJ2Nbu/+NeYHPzncm6ked20P7D0X8MG6DlIQ2OZ9rkMsrDQ7VvY/fOGsvxJ1qKyMii/Aqscz7djXHvN2BK551gid5cl3A9vBKjutriqmVqrryeNnP81GobRp7CtX8t+0rHG2vVV6ubV99Kqw+3b63q2gJ+gp2kAl9BrtojI6RKeIoI/oO/qBfuZk7lPuc+7LhHpjabrnMVpYua+/AfZGKds=</latexit><latexit sha1_base64="ccOrvLJLQZ5targeLvLu0ZbqcAk=">AAAF0XicfVRNbxMxEHVLAyV8tXDksiKqVBCqsgUJDlSqRCuFA2pp0qYoGyrHO0lWWdtb2/lcrYS4cuQKf4J/w79hnIYm8QKWVjs773nm7czYrSSOtCmXf62s3lgr3Ly1frt45+69+w82Nh+eadlXDE6ZjKU6b1ENcSTg1EQmhvNEAeWtGOqt3luL1wegdCRFzYwTaHLaEVE7YtSg62M9SBTz6p9qFxul8k55ury84c+MEpmt44vNtZ9BKFmfgzAsplo3/HJimilVJmIxZMWgryGhrEc70EBTUA66mU4VZ94WekKvLRU+wnhT7+KOlHKtx7yFTE5NV7uYdf4LM12euT6bRWfedC1hSDZSxnp5h42bdzf6pv26mUYi6RsQ7Oov2v3YM9KztfXCSAEz8RgNylSEhfBYlyrKDHaguOW5KRI6kVlxKQWnPVBScqwUFQxiFCBgyCTnVIRpEIkwSwO7t9VK32UOOoSo0zV/CO106BJGozk4csHxeA6OXXAymYOTXN7hf3LqRMxQRuO0moeNAlhg1FxGpbKAVlwUlMKBSoMBVZDoKJbCIYR4iBYCHLgBQmjDpQ3A7CTjIIo9N8UlcrJ0b4HiMA5Hybwth26Gk5M5eJLZhi+iTA6s/mrU4fRCZTmQXYPM7QlOygAHbvr/s/KX3eSJsugRh04+Op79a8wNXj0+mKuedu0A8JwreI+uowQUxbzP0oAqlIZjhe/gubX+SkwDgzJqrgBUT+faqas9lp0QbPPQCKzlyLcD26UmP622KlnDb2JaGJkUP7Ptkv80w1vOd++0vHG2u+O/2Nn98LK0/2Z2362Tx+QJ2SY+eUX2SYUck1PCCCffyHfyo1AtjAufC1+uqKsrsz2PyNIqfP0NoU8ddg==</latexit>

56

Learning with Adaptive Regularization
Block Coordinate Descent for Adaptive Regularization:-

One SVD in each outer iteration to solve the matrix optimization
problem-

Seamlessly work with end-to-end training with back-propagation57

Outline•

Deep Learning Preliminary•

Learning with Adaptive Regularization•

Experiments•

Conclusion

58

Experiments
Multiclass Classification and Multitask Regression-

Multiclass Classification: MNIST and CIFAR10
CONV5⇥5⇥10⇥20 FC320⇥50- MNIST: CONV5⇥5⇥1⇥10<latexit sha1_base64="4zN8vfsYaY8+/IDpz6yqlukRlks=">AAAGXXichVRdb9MwFE3HVkYZY4MHHnggopq0IaiSjgkemDRpGyoPsLFvqakq17ltrSV25rifVn4F/4R/wxO/gzfsrGtThw9LVU/uOb7nxvfGrSggsXCcH4WFe4tLxfvLD0oPVx6tPl5bf3IRsx7HcI5ZwPhVC8UQEArngogAriIOKGwFcNm63tf8ZR94TBg9E6MIGiHqUNImGAkVaq5980IkunFb7h99uUiacsf2BAkhtqfAnQInefN/tXOHqhn5x30l3q5OuR2T23EyLs21slNx0mXngTsBZWuyjpvri989n+FeCFTgAMVx3XUi0ZCIC4IDSEpeL4YI4WvUgbqCFCmnhkxPL7E3VMS324yrHxV2Gs3ukCiM41HYUsq0ZJPTwb9xohsmZky7xImdrjlOiQVjQTy/Q+fNh+s90X7fkIRGPQEU375FuxfYgtm6z7ZPOGARjBRAmBN1EDbuIo6wUNNQ2rBNiwiNWVKaswjRNXDGQnVSiGIIVAEUBpiFIaK+9Aj1E5l2sdWSnxKDHQDpdMWdoC0HpmA4nJFDkxyNZuTIJMfjGTnO+Q7+4RlHdMJiFMjTPC04QEZxZipqtQxbM1ngXA2U9PqIQxSTgFFD4KsPOpPgwEzgQxtudAKsJ1kNIt01LW6UJpG7GYmhOBxGs7Ycmg4nJzPyJNENz7KY9XX9p6QToiZPciSektjsiZqUvhq49P0nx++Y5hHX7FEInXz2iOMpZyY/PT6YVZ127QDUd87hswodRcCR8n0lPcRVaWqs1L/3WqM/CqUnVBlnZgGqejSrHZm1B6zjg26eAp5GRvl6YLtI5KdVn0pSdxvKFoZCqsdks+xu6VvONe+0PLioVtztSvXr2/Leh8l9t2w9t15am5ZrvbP2rJp1bJ1b2PpVeFHYLGwt/SwuFVeKq7fShcJkz1NrbhWf/QabQ0uH</latexit>-

CIFAR10:
CONV5⇥5⇥3⇥10<latexit sha1_base64="2JPB0oDCm2bCyAaZPo18YdHhcd8=">AAAGfHicnVTZbtNAFHWgDSVsLbwg8WIRKpWlld2yPVBUqS2EB2jpLsVRNBnfJFZtjzszWUf+qf4NP4O446ZZxiwSI0U5vufMvcdzr6eRhIGQjvOjcOPm3Hzx1sLt0p279+4/WFx6eCJYh1M4pixk/KxBBIRBDMcykCGcJRxI1AjhtHG+rfnTLnARsPhIDhKoRaQVB82AEomh+uKlFxHZFk21vfftJK2rN7YngwiEPQYb18B10tV/ql3nGq1PyT9ta7Ez5hD+J4km6otlZ83Jlp0H7giUrdHary/NXXo+o50IYklDIkTVdRJZU4TLgIaQlryOgITQc9KCKsKYYKWayg43tZcx4ttNxvEXSzuLTu9QJBJiEDVQmXk2OR38EyfbUWrGdBWR2tma4VAsGQvF7A6dNx+udmTzfU0FcdKRENOrt2h2QlsyW4+B7QccqAwHCAjlAR6ETduEEypxWErLtlkiIUOWlmZKROQcOGMRnhSJKYRoIIYeZVFEYl95QeynKmtjo6G+pAbbg6DVlteCpuqZgn5/QvZNcjCYkAOTHA4n5DBXt/eXmiKJRywloTrM05IDTCmOTEWlMsVWTBY4x4FSXpdwSEQQstgQ+Pi9TyXYMRP40IQLnYDqScZBjDfNEheoSdXmlMRQ7PaTSVt2zQoHBxPyINUNn2Yp62r/h0ErInWe5kg6JqnZE5yULg5c9v6j43fM4gnX7F4ErXz2hNMxZyY/3N+ZuM66tgP4nXP4iqG9BDjBui+URzhaw7HCf++VRr8VKk+ijSPTALonE+/E9B6ylg+6eQg8jQz7emDbROanVZ9KWnVrWBb6UuFjulJ2n+tbzjXvtDw4WV9zN9bWv78ub30Y3XcL1hPrqbViudY7a8uqWPvWsUULjwsfC58LlfmfxWfFl8XVK+mNwmjPI2tmFd/+AictVew=</latexit>-

CONV5⇥5⇥10⇥20

Multitask Regression: SARCOS
FC256⇥100- SARCOS: FC21⇥256<latexit sha1_base64="zx+73+YEd3J4cTRwYaRj1kINLCk=">AAAGHHicfZTNbhMxEMfd0kAJXy0cuayIKhUE1W6gLQcqVWqLwgG1pJ9SNooc7yRZdXe9tZ1Py6/CQ3DiAbghrkhcuMJjYG/TJvEClqLMzu/vmVnPrJtpFHLhuj/m5m8sFG7eWrxdvHP33v0HS8sPTzjtMgLHhEaUnTUxhyhM4FiEIoKzlAGOmxGcNs93DD/tAeMhTY7EMIV6jNtJ2AoJFtrVWKr6MRYd3pJvd1RDlj3HF2EM3Cmvb6gXs2x94wp6rmtB7RmzTdVYKrlrbracvOGNjRIar4PG8sInP6CkG0MiSIQ5r3luKuoSMxGSCFTR73JIMTnHbahpM8E6UV1mL6+cFe0JnBZl+pcIJ/NO75A45nwYN7Uyq9hmxvkvJjqxsn0mC1dOtmaYFgtKIz67w8TNu2td0Xpdl2GSdgUk5PItWt3IEdQxbXKCkAER0VAbmLBQH4RDOphhInQziyuOnSLFI6qKMylifA6M0lifFE4IRLqABPqExjFOAumHSaBk1sRmU75TFu1D2O6IK0FL9m3BYDCBAxsOhxM4tOFoNIGjXN7+f3LyNBlTgiN5mMeCAUwpjmxFpTJFKzYFxvRASb+HGaQ8jGhiCQL9PU4F2LUDBNCCCxOAmEnWg5hs2SkutEbJrSmJpdgbpJO27NkZqtUJrCrT8GlKaM/Ufxi2Y9xgKgfJNSR2T/Sk9PTAZe8/Pn7XTp4yQ/djaOejp4xcMzv44cHupOqsa7ugv3MG77VrPwWGdd5n0sdMl6bHSv/7z431V6H0hS7jyC5AV48ntWO79oi2AzDN04ZvLKt8M7AdLPLTak5F1by6TgsDIfWjWi15T80t59l3Wt44Ka95L9fKH16Vtt+M77tF9Bg9QavIQ5toG1XQATpGBH1GP9Ev9LvwsfCl8LXw7VI6Pzfe8wjNrML3Pw1zOe8=</latexit>

FC500⇥500

FC50⇥10

FC500⇥500

FC500⇥10

FC100⇥759

Experiments
Multiclass Classification and Multitask Regression

Dropout: randomly inhibiting hidden activations- Batch Normalization (BN): maintain and correct internal
covariate shift- Weight Decay: L2 regularization of weight matrix- DeCov: penalize covariance matrix of hidden representations-

60

Experiments
Generalization performance on MNIST (10 classes)

Batch size = 256-

Batch size = 2048

AdaReg improves generalization with large batch size and small training
size on MNIST

61

Experiments
Generalization performance on CIFAR10 (10 classes)

Batch size = 256-

Batch size = 2048

AdaReg consistently improves generalization on CIFAR10

62

Experiments
Generalization performance on SARCOS (7 tasks)-

Explained variance ~ negative MSE-

For regression tasks, AdaReg consistently outperforms other baselines
in terms of generalization

63

Experiments
AdaReg optimization trajectory on MNIST (train size/batch size)

600/256

600/2048

6000/256

6000/2048

64

Experiments
Stable Rank and Spectral Norm:srank(W ) := ||W ||2F /||W ||22

1  srank(W )  rank(W )<latexit sha1_base64="+6RVU/TSAn78aBiwDq/s7veoMPw=">AAAF83icfVTNbhMxEHZLAyX8tXDksiKq1CJUZQsSHKhUiVYKB9TQfykbRV7vJFnFa29t59faZ+ABuCGuXJC4wnPwNthp2my8gKXVzs73eebzzKzDlMZSVau/l5ZvrZRu31m9W753/8HDR2vrj88k7wsCp4RTLi5CLIHGDE5VrChcpAJwElI4D3vvLH4+ACFjzk7UOIVmgjssbscEK+NqrW35XkDh0gsUjJSWArNetnm+lfNdu1prlep2dbq8ouHPjAqarXprfeV7EHHST4ApQrGUDb+aqqbGQsWEQlYO+hJSTHq4Aw1jMpyAbOrpmTJvw3gir82FeZjypt78Do0TKcdJaJgJVl3pYtb5L0x1k8z12Swy86ZrATNkxTmVizts3KK70VftN00ds7SvgJGrU7T71FPcs9X3olgAUXRsDExEbArhkS4WmCjTo/KG56ZI8YRn5YUUCe6B4DwxlcKMADUCGAwJTxLMIh3ELMp0YPeGoX6fOegQ4k5XXRPaeugSRqM5OHLB8XgOjl1wMpmDk0Le4X9yypTNUIKpPi7CSgDkGCcuo1bLoTUXBSHMQOlggAWkMqacOYTI/Ga5APtugAjacGkDEDvJZhDZrpvi0nAyvZujOIyDUTpvy4Gb4ehoDh5ltuF5lPCB1X8cdxLcElkBJDcgcXtiJmVgBm56/ln5q27yVFj0MIFOMXoqyA3mBj+u789VT7u2D+Y/F/DBuA5TENjkfa4DLIw0M1bmHbyw1l+JOlBGxokrwKjHc+3Y1U55JwLbPGME1nLk24HtYlWcVluVrOE39dUlZz6zzYq/lZlbznfvtKJxtrPtv9ze+fiqsvd2dt+toqfoGdpEPnqN9lAN1dEpIugT+oF+ol+lfulz6Uvp6xV1eWm25wlaWKVvfwBq8Crg</latexit><latexit sha1_base64="sqHXDmPUtsjh5xtbRCY/2W+xCyE=">AAAF83icfVTNbhMxEHZLAyX8tIUjlxVRpRahkgQkOFCpEi0KB9TSv1TKppHjnSSr7Npb25s/Z5+BB+CGuHJB4grPwdtgb0KTeAFLUWbn+zzzeWbsZhT4QhaLv5aWb6zkbt5avZ2/c/fe/bX1jQfngsWcwBlhAeMXTSwg8CmcSV8GcBFxwGEzgGqz+8bg1R5w4TN6KocR1EPcpn7LJ1hqV2N925UwkEpwTLvJVnXb9aAFV854XB2PG28vy88mVvmy3FgvFHeK6XKyRmlqFNB0HTU2Vr65HiNxCFSSAAtRKxUjWVeYS58EkOTdWECESRe3oaZNikMQdZWeKXE2tcdzWozrH5VO6p3foXAoxDBsamaIZUfYmHH+C5OdMLF9JotInHQtYJosGQvE4g4TN+uuxbL1qq58GsUSKJmcohUHjmSOqb7j+RyIDIbawIT7uhAO6WCOidQ9ym86dooIj1iSX0gR4i5wxkJdKUwJBFoAhT5hYYipp1yfeolyzd5mU71LLLQPfrsj/xBaqm8TBoMZOLDB4XAGDm1wNJqBo0ze/n9yiohOUYIDdZKFJQeYY5zajEplDq3YKHCuB0q5PcwhEn7AqEXw9DWbC7BvB0ivhAlAzCTrQaS7doorzUnU7hzFYhwMollbDuwMx8cz8DgxDZ9HCesZ/Sd+O8QNnmRAcg0Suyd6Unp64NLzT8tftJNH3KCHIbSz0SNOrjE7+MnR/kx12rV90Pecw3vtOoyAY533iXIx19L0WOl/96mx/kpUrtQyTm0BWj2eace29oC1PTDN04ZrLEu+GdgOltlpNVVJaqW6mrx/+jPZKpS2E/3Klew3LWucl3dKz3fKH14U9l5P37tV9Ag9RluohF6iPVRBR+gMEfQRfUc/0M9cnPuU+5z7MqEuL033PEQLK/f1N2awKzo=</latexit>

Generalization bound of NNs using spectral norms and stable ranks(Neyshabur et al. ICLR’ 17):
v
!
u L
L
X
uY
srank(Wj )/n
Generalization Error = O t ||Wj ||22
j=1

j=1

<latexit sha1_base64="nzPs7H3xSM/lFs/jD8+AyAQjtcc=">AAAGGXicfZTNbhMxEMfd0kAJXykcuayIKqUIlWxAggORKmilIIFS+i1l08jxOpttdu2t7Xw6fhLeAokH4Ia4ckJc4T2w07RJvIClKLPz+3tm1jPrZhKFXBSLP5aWr61krt9YvZm9dfvO3Xu5tftHnHYZwoeIRpSdNCHHUUjwoQhFhE8ShmHcjPBxs/PG8OMeZjyk5EAME1yPYUDCVoig0K5GruqUnar3OgyCgsfPmZBewqjfkGdlV52+G4+PG2fjcaN0WvJ4N750O57AAyE5g6SjClqy8ZSoSZCNRi5f3CxOlpM23KmRB9O121hb+eT5FHVjTASKIOc1t5iIuoRMhCjCKut1OU4g6sAA17RJYIx5XU7eXDnr2uM7Lcr0jwhn4p3fIWHM+TBuamUMRZvbzDj/xUQ7VrbPZOHKmawFpsWC0ogv7jBx0+5aV7Re1mVIkq7ABF28RasbOYI6pkeOHzKMRDTUBkQs1AfhoDZkEAndyey6Y6dI4Iiq7EKKGHYwozTWJwUJwpEugOA+onEMiS+9kPhKemZvsynfKov2cRi0xaWgJfu2YDCYwYENh8MZHNpwNJrBUSpv/z85eUKmFMFI7qexYBjPKQ5sRaUyRys2xYzpgZJeDzKc8DCixBL4+mOcC7BtB/BxC5+bAMhMsh5EUrZTnGuNkuU5iaXYGSSztuzYGfb2ZnBPmYbPU0R7pv79MIhhg6kURFcQ2T3Rk9LTAzd5/+nxF+3kCTO0GuMgHT1h6IrZwfd3t2dVT7q2jfV3zvB77aommEGd97H0INOl6bHS/94TY/1VKD2hyziwC9DVw1nt0K49ooGPTfO04RnLKt8MbBuK9LSaU1E1ty4vLjz9qAp5d0PpW86177S0cVTadJ9tlj48z2+9mt53q+AheAQKwAUvwBaogF1wCBD4DH6CX+B35mPmS+Zr5tuFdHlpuucBWFiZ738AbF46Mg==</latexit>-

Smaller stable rank and spectral norm implies better generalization

65

Experiments
AdaReg leads to smaller stable ranks v
and spectral norms

!
u L
L
X
uY
2
t
||Wj ||2
srank(Wj )/n
Generalization Error = O
j=1

j=1

<latexit sha1_base64="nzPs7H3xSM/lFs/jD8+AyAQjtcc=">AAAGGXicfZTNbhMxEMfd0kAJXykcuayIKqUIlWxAggORKmilIIFS+i1l08jxOpttdu2t7Xw6fhLeAokH4Ia4ckJc4T2w07RJvIClKLPz+3tm1jPrZhKFXBSLP5aWr61krt9YvZm9dfvO3Xu5tftHnHYZwoeIRpSdNCHHUUjwoQhFhE8ShmHcjPBxs/PG8OMeZjyk5EAME1yPYUDCVoig0K5GruqUnar3OgyCgsfPmZBewqjfkGdlV52+G4+PG2fjcaN0WvJ4N750O57AAyE5g6SjClqy8ZSoSZCNRi5f3CxOlpM23KmRB9O121hb+eT5FHVjTASKIOc1t5iIuoRMhCjCKut1OU4g6sAA17RJYIx5XU7eXDnr2uM7Lcr0jwhn4p3fIWHM+TBuamUMRZvbzDj/xUQ7VrbPZOHKmawFpsWC0ogv7jBx0+5aV7Re1mVIkq7ABF28RasbOYI6pkeOHzKMRDTUBkQs1AfhoDZkEAndyey6Y6dI4Iiq7EKKGHYwozTWJwUJwpEugOA+onEMiS+9kPhKemZvsynfKov2cRi0xaWgJfu2YDCYwYENh8MZHNpwNJrBUSpv/z85eUKmFMFI7qexYBjPKQ5sRaUyRys2xYzpgZJeDzKc8DCixBL4+mOcC7BtB/BxC5+bAMhMsh5EUrZTnGuNkuU5iaXYGSSztuzYGfb2ZnBPmYbPU0R7pv79MIhhg6kURFcQ2T3Rk9LTAzd5/+nxF+3kCTO0GuMgHT1h6IrZwfd3t2dVT7q2jfV3zvB77aommEGd97H0INOl6bHS/94TY/1VKD2hyziwC9DVw1nt0K49ooGPTfO04RnLKt8MbBuK9LSaU1E1ty4vLjz9qAp5d0PpW86177S0cVTadJ9tlj48z2+9mt53q+AheAQKwAUvwBaogF1wCBD4DH6CX+B35mPmS+Zr5tuFdHlpuucBWFiZ738AbF46Mg==</latexit>

MNIST
66

Experiments
AdaReg leads to smaller stable ranks v
and spectral norms

!
u L
L
X
uY
2
t
||Wj ||2
srank(Wj )/n
Generalization Error = O
j=1

j=1

<latexit sha1_base64="nzPs7H3xSM/lFs/jD8+AyAQjtcc=">AAAGGXicfZTNbhMxEMfd0kAJXykcuayIKqUIlWxAggORKmilIIFS+i1l08jxOpttdu2t7Xw6fhLeAokH4Ia4ckJc4T2w07RJvIClKLPz+3tm1jPrZhKFXBSLP5aWr61krt9YvZm9dfvO3Xu5tftHnHYZwoeIRpSdNCHHUUjwoQhFhE8ShmHcjPBxs/PG8OMeZjyk5EAME1yPYUDCVoig0K5GruqUnar3OgyCgsfPmZBewqjfkGdlV52+G4+PG2fjcaN0WvJ4N750O57AAyE5g6SjClqy8ZSoSZCNRi5f3CxOlpM23KmRB9O121hb+eT5FHVjTASKIOc1t5iIuoRMhCjCKut1OU4g6sAA17RJYIx5XU7eXDnr2uM7Lcr0jwhn4p3fIWHM+TBuamUMRZvbzDj/xUQ7VrbPZOHKmawFpsWC0ogv7jBx0+5aV7Re1mVIkq7ABF28RasbOYI6pkeOHzKMRDTUBkQs1AfhoDZkEAndyey6Y6dI4Iiq7EKKGHYwozTWJwUJwpEugOA+onEMiS+9kPhKemZvsynfKov2cRi0xaWgJfu2YDCYwYENh8MZHNpwNJrBUSpv/z85eUKmFMFI7qexYBjPKQ5sRaUyRys2xYzpgZJeDzKc8DCixBL4+mOcC7BtB/BxC5+bAMhMsh5EUrZTnGuNkuU5iaXYGSSztuzYGfb2ZnBPmYbPU0R7pv79MIhhg6kURFcQ2T3Rk9LTAzd5/+nxF+3kCTO0GuMgHT1h6IrZwfd3t2dVT7q2jfV3zvB77aommEGd97H0INOl6bHS/94TY/1VKD2hyziwC9DVw1nt0K49ooGPTfO04RnLKt8MbBuK9LSaU1E1ty4vLjz9qAp5d0PpW86177S0cVTadJ9tlj48z2+9mt53q+AheAQKwAUvwBaogF1wCBD4DH6CX+B35mPmS+Zr5tuFdHlpuucBWFiZ738AbF46Mg==</latexit>

CIFAR1067

Experiments
Correlations learned in the weight matrix:

CNN, Acc: 89.34

AdaReg, Acc: 92.50

MNIST, train size = 600-

The eﬀect of sharing statistical strength, i.e., “learning from the experience ofothers”
68Experiments
Correlations learned in the weight matrix:

CNN, Acc: 98.99

AdaReg, Acc: 99.19

MNIST, train size = 6,000-

The eﬀect of sharing statistical strength, i.e., “learning from the experience ofothers”
69Outline•

Deep Learning Preliminary•

Learning with Adaptive Regularization•

Experiments•

Conclusion

70

Conclusion-

AdaReg: an approximate empirical Bayes method for regularizing NN
training on small datasets-

Learn the preconditioning matrix adaptively from data-Significant improvement in terms of spectral norm and stable rank,
leading to smaller generalization error
71

Thanks
Q&A
Paper available at: https://arxiv.org/abs/1907.06288
To appear @ NeurIPS-19, code available @ https://github.com/yaohungt/AdaptiveRegularization-Neural-Network72