Learning Neural Networks with Adaptive Regularization✝
✝
✝
✝⚕
Han Zhao* , Yao-Hung Hubert Tsai* , Ruslan Salakhutdinov , Geoﬀrey J. Gordan✝Carnegie Mellon University, ⚕Microsoft Research Montreal (*equal contribution)

https://github.com/yaohungt/Adaptive-Regularization-Neural-Network

Overview

Prior of Parameters

Q: How to eﬀectively regularize neuralnetworks given few available training data?

•••

AdaReg: an approximate empirical Bayes
method for regularizing NN training on small
datasets
Learn the preconditioning matrix adaptively
from the dataSignificant improvement in terms of spectral
norm and stable rank, leading to smaller
generalization error

Multivariate Normal Distribution
W ∼ ℳ𝒩(0p×d, Σr, Σc)

min minW,a ⌦r ,⌦cp(W | Σr, Σc) =⌦r

vIp , uId⌦cd log det(⌦r ) + p log det(⌦c )preconditioned matrix

1/2
1/2 2
1/2
1/2
2
∥Ωr WΩc ∥F = ∥(Ωr ⊗ Ωc )vec(W)∥2vIdSolving Ωr
min⌦r<latexit sha1_base64="lRKJ7aSdOiD/NO/jNNpesr1LkOE=">AAAGJHicfVTNbtNAEN6WBkr4a+HIZUVU1PJTJQUJDlSq1FYKB9TSpk1FHKLNepNYtXfd9eZ3ta/DA/AEPAA3xIELR67wCMwmoU7WwEq2x/N98+OZ8TTjMEhUsfhtYfHKUu7qteXr+Rs3b92+s7J69zQRXUnZCRWhkGdNkrAw4OxEBSpkZ7FkJGqGrNo837V4tcdkEgheUcOY1SPS5kEroESBqrHy7iH2ooA3tBdLabB30SU+9pRct++4CneKq+8rGxg/xb4XirbP1BjbwI/BkqhOs6lfG7C3MiWh3jVmQmisFIqbxfHBWaE0FQpoeg4bq0sfPV/QbsS4oiFJklqpGKu6JlIFNGQm73UTFhN6TtqsBiInEUvqelwEg9dA4+OWkHBxhcfaWQtNoiQZRk1g2lQTF7PKf2GqExlXZ6MkBo/PHAZkJUSYzFtYv1l1rataL+s64HFXMU4nX9HqhlgJbNuF/UAyqsIhCITKAAqBaYdIQhU0Nb+G3RAxGQmTnwsRkXMmhYigUoRTFkICnPWpiCLCfe0F3Dc67aSD9lnQ7qg/hJbuu4TBIAUHLjgcpuDQBUejFBxl4vb/EzOJuUkH7jgLK8nYDKPiMsrlGbTsokxKGCjt9YhkcRKEgjsEH/7LGQd7rgOftdiFdUDtJMMg8m03xAVwjN6eoTiM/UGctmXfjXB0lIJHxjZ8FqWiZ/M/DtoRaUiTAeklSN2ewKT0YODG3z8tf9ENbjeF9g4i1s56h31xibnOjw/30qzHXdtj8J9L9gZUBzGTBOI+0h6RkBqMFTy9J1b6K1HDljK64iYA2ZM0d+LmPtlgZix4VnLStwPbISo7rbYqplaqQ1g2UBpezXqhtGFgy5XcnZYVTrc2S882t94+L+y8mu67ZXQfPUDrqIReoB1URofoBFH0Cf1AP9Gv3Ifc59yX3NcJdXFhanMPzZ3c99+EuT10</latexit>

Determine the Parameters in the Prior
1) Empirical Bayes: estimate the parameters of the prior from dataΣr̂ , Σĉ = arg max p(𝒟 | Σr, Σc) = arg max p(𝒟 | W) p(W | Σr, Σc) dW
∫
Σ ,Σ
Σ ,Σ

02@
<latexit sha1_base64="Ue9g0E5iTRMt4sX51DJZfrgVKc0=">AAAGcXicfVTbThsxEN1QSGlaWmifqr5YREhJKTQLldoHIiEBUvrQQiEQpGyIHK83WbE3vM7V8vf0vX/T7+gPdLwJbOKltZRkMud4Zuw5nk7kuTGvVH7nlp4sr+Sfrj4rPH+x9vLV+sbrqzjsM0IvSeiF7LqDY+q5Ab3kLvfodcQo9jsebXRujxTeGFAWu2FQ5+OItnzcDVzHJZiDq73+s4IsN0BWhBl3sWd51OEly2GYCFMKW1qclayIMdSAb4IaN/UyQjvI8sKuTXkCldE2snzMe52O+CrbIrEJ9sSRlFOCxdxuj5dRNY3y0VZRALwRO6a8j6B2fX80RHu9WNmtJAtlDXNmFI3ZOmtvLP+y7JD0fRpw4uE4bpqViLeEOifxqCxY/ZhGmNziLm2CGWCfxi2R3KhEW+CxkRMy+AQcJd75HQL7cTz2O8BUpcY6ppz/wnjPl7pPZYklStYCBmQehl68uEPFzbqbfe58aQk3iPqcBmR6CqfvIR4i1Xtku4wS7o3BwIS5cBGI9DD0moNCCltITxHhSSgLCyl8fEtZGPpwUzgg1IMCAjokoe/jwBYgJVuKVAwaOqRKB/cERwx1wmiUgiMdHI9TcKyDk0kKTjJ5h//JGUeBTAV3kYU5o3SOUdcZtdocWtNRyhgISlgDzGgUu14YaAQbHvlcgGM9gE0deqcCEKVkEGJQ1VPcAUeK6hxFY5yMorQtJ3qG8/MUPJeq4fMoCQeq/gu36+M2kxmQPIBE7wkoZQCCS84/u/6KnhweN6CnPu1mo8OseMD04Bdnx2nVSdeOKbxzRr+B6zSiDEPe98LCDEoDWcGv9UFZjxIFzDkp6noBUD1Oa8d67dMhKBPDUpZWvhJsD/OsWtWtyKbZgrR0xAX8laWiWZYw5Ux9pmWNq71dc39378en4uHBbN6tGu+MTaNkmMZn49CoGWfGpUFya7n93EGuuvIn/zaP8ptT6lJutueNsbDy238BKwZT0w==</latexit>Tr(⌦r W ⌦c W T )d log det(⌦r ) + IC (⌦r ) with C := {A 2 Sp++ | uIp

vIp }

A

<latexit sha1_base64="2kXY2aOtT325DixfBQ243M1PZ3Q=">AAAGDnicfVRLbxMxEHZLAyW8WjhyWRFVKhRVSUGCA5VaNZXSA2rpEykbIq/XSazu2q7XeVr7H/gdHDlwQ1w5ci2/hvE2bRIvYCnK7HyfZz7PjB3IiCW6XL6cm7+1ULh9Z/Fu8d79Bw8fLS0/Pk1EVxF6QkQk1McAJzRinJ5opiP6USqK4yCiZ8H5jsXPelQlTPBjPZS0EeM2Zy1GsAZXc6nqx1h3CI7MTuqHtEUvPN9s+4x7/tFBtWnW1tJP0o9Z6HX3mtKH2AQo29dGzzrT5lKpvF7Olpc3KmOjhMbroLm88NUPBenGlGsS4SSpV8pSNwxWmpGIpkW/m1CJyTlu0zqYHMc0aZjsuKm3Ap7QawkFP669zDu9w+A4SYZxAEx7uMTFrPNfmO7EqeuzWZLUy9YMBmQtRJTM7rBx8+56V7feNgzjsqspJ1enaHUjTwvPNsYLGZRUR0MwMFEMCuGRDlaYaGhfccVzU0g8EmlxJkWMz6kSIoZKYU5oBAI47RMRx5iHBpoapibrdxCYvdRB+5S1O/qa0DJ9lzAYTMCBCw6HE3DogqPRBBzl8vb/kzORfIzaET3Kw1pROsU4dhm12hRac1GqFAyU8XtYUZmwSHCHEMINnApQdQNkN8YGIHaSYRD5ppviAjip2ZyiOIzdgZy0ZdfNcHg4AQ9T2/BplIie1X/E2jFuqjQHkhuQuD2BSenBwGXnH5e/7CaXyqL7MW3no0tFbjA3ODwdE9VZ16oU7rmi78G1L6nCkPeF8bECaTBW8O+/tNZficbXIOPYFQDq8UQ7drVHoh1S2zwwfGs58u3AdrDOT6utSlqvNCAtHWgDn+lqqfLcvnIV903LG6cb65VX6xsfXpe23o3fu0X0FD1Dq6iC3qAtVEMH6AQR9AX9Qpfod+Fz4Vvhe+HHFXV+brznCZpZhZ9/AELrNdI=</latexit>✓

1
Tr(⌦r W ⌦c W T )d

log det(⌦r ) + IC (⌦r )◆= W ⌦c W T /d⌦r 1 + NC (⌦r )W ⌦c W T /d<latexit sha1_base64="TlrqV5Uk4eWYFaTqYCYGYleIAwM=">AAAGCnicfVTNbhMxEHahgRL+WjhyWRFVahEt2YIEBypVaiuFA7S0aVMpm0Ze7yRZdXe99Tq/K78Bz8ED9Ia4cuQKEm/DOA3ZxAtYWu14vs8znz1ju3HgJ7Jc/rVw4+Zi4dbtpTvFu/fuP3i4vPLoNOFdweCE8YCLM5cmEPgRnEhfBnAWC6ChG0DNvdjVeK0HIvF5VJXDGBohbUd+y2dUoqu5vFNzYsGs2nn1hWdtWDgR5+mGrSzHjywnpLLDaJB+UM10OtlVam3KW28ul8qb5fGw8oY9MUpkMg6bK4tXjsdZN4RIsoAmSd0ux7KRUiF9FoAqOt0EYsouaBvqaEY0hKSRjreqrFX0eFaLC/wiaY29sytSGibJMHSRqfUmJqad/8JkJ1SmT2dJlDUecxiSJedBMr9Cx827613ZetNI/SjuSojY9S5a3cCS3NJFsTxfAJPBEA3KhI8HYbEOFZRJLF1x1TJTxHTEVXEuRUgvQHAe4knRiEGAAiLoMx6GNPJSrKanrkvouuk7ZaB98Nsd+YfQSvsmYTDIwIEJDocZODTB0SgDR7m8/f/kTOJIZV13nIelAJhhVE1GpTKDVkwUhMCGSp0eFRAnfsAjg+Dh7ZsJsGcG8KAFlzoA052MjRhtmykukaPS7RmKwdgfxFlZ9s0MR0cZeKR0wWdRxnta/7HfDmlTqBzIpiAza4Kd0sOGG+9/cvxlMznecEQPQmjno+OTMcXM4MeHe5nqcdX2AO+5gPfoOohBUMz7LHWoQGnYVvh3nmvrr8TUkSijagpA9TTTTk3tAW97oIuHhqMtQ75u2A6V+W7Vp6LqdgPTwkCmOFVrJXtd4Stnm29a3jjd2rRfbm59fFXaeTt575bIE/KUrBGbvCY7pEIOyQlh5DP5Tn6Qn4VPhavCl8LXa+qNhcmax2RuFL79Bv9ZM50=</latexit>⌦r 1 2 NC (⌦r 1 )p−1⊤
Optimal Ωr is the Euclidean projection of WΩcW /d onto C := {A 2 S++ | uIp<latexit sha1_base64="2kXY2aOtT325DixfBQ243M1PZ3Q=">AAAGDnicfVRLbxMxEHZLAyW8WjhyWRFVKhRVSUGCA5VaNZXSA2rpEykbIq/XSazu2q7XeVr7H/gdHDlwQ1w5ci2/hvE2bRIvYCnK7HyfZz7PjB3IiCW6XL6cm7+1ULh9Z/Fu8d79Bw8fLS0/Pk1EVxF6QkQk1McAJzRinJ5opiP6USqK4yCiZ8H5jsXPelQlTPBjPZS0EeM2Zy1GsAZXc6nqx1h3CI7MTuqHtEUvPN9s+4x7/tFBtWnW1tJP0o9Z6HX3mtKH2AQo29dGzzrT5lKpvF7Olpc3KmOjhMbroLm88NUPBenGlGsS4SSpV8pSNwxWmpGIpkW/m1CJyTlu0zqYHMc0aZjsuKm3Ap7QawkFP669zDu9w+A4SYZxAEx7uMTFrPNfmO7EqeuzWZLUy9YMBmQtRJTM7rBx8+56V7feNgzjsqspJ1enaHUjTwvPNsYLGZRUR0MwMFEMCuGRDlaYaGhfccVzU0g8EmlxJkWMz6kSIoZKYU5oBAI47RMRx5iHBpoapibrdxCYvdRB+5S1O/qa0DJ9lzAYTMCBCw6HE3DogqPRBBzl8vb/kzORfIzaET3Kw1pROsU4dhm12hRac1GqFAyU8XtYUZmwSHCHEMINnApQdQNkN8YGIHaSYRD5ppviAjip2ZyiOIzdgZy0ZdfNcHg4AQ9T2/BplIie1X/E2jFuqjQHkhuQuD2BSenBwGXnH5e/7CaXyqL7MW3no0tFbjA3ODwdE9VZ16oU7rmi78G1L6nCkPeF8bECaTBW8O+/tNZficbXIOPYFQDq8UQ7drVHoh1S2zwwfGs58u3AdrDOT6utSlqvNCAtHWgDn+lqqfLcvnIV903LG6cb65VX6xsfXpe23o3fu0X0FD1Dq6iC3qAtVEMH6AQR9AX9Qpfod+Fz4Vvhe+HHFXV+brznCZpZhZ9/AELrNdI=</latexit>c

Intractable.
2) Iterative maximization of the joint distribution: sequence of MAP

Two-layer Neural Network(t) (t)W= arg max log p(𝒟 | W) + log p(W | Σr , Σc )
W
(t+1)
(t)
(t+1)
(t+1)
| Σr, Σc )Σr= arg max log p(𝒟 | W) + log p(W
Σr
(t+1)
(t+1)
(t+1)
(t+1)
| Σr , Σc)Σc = arg max log p(𝒟 | W) + log p(W
Σc
(t+1)

T[u,v] (x) := max{u, min{v, x}}<latexit sha1_base64="ijiExb4GF+9dH7z5X6TEaHB/t44=">AAAF/nicfZTNbhMxEMfd0kAJH23hyGVFVKlFUZWUSnCgUiVaKRxQS5t+SNkocryTxOquvfV6k00sSzwHD8ANIXHiyBXegLfBTtMm8QKWoszO72/P2DN2Ow5pIiuV3wuLd5YKd+8t3y8+ePjo8crq2pOzhKeCwCnhIRcXbZxASBmcSipDuIgF4Kgdwnn78q3l530QCeWsLocxNCPcZbRDCZbG1Vrd8SMse+22quuWaqTlflNvZJueH0AHrjwDM1+lZWNQ5qt+2ct87evWaqmyVRkPL29UJ0YJTcZRa23pmx9wkkbAJAlxkjSqlVg2FRaSkhB00U8TiDG5xF1oGJPhCJKmGm9Pe+vGE3gdLsyPSW/snZ2hcJQkw6htlHYzicus819M9iLt+myURHvjMceMWHIeJvMz7Lp5dyOVnddNRVmcSmDkehedNPQk92whvIAKIDIcGgMTQc1BeKSHBSbSlKu47rkhYjziujgXIsKXIDiPzElhRiA0CTAYEB5FmAXKpyzQ6qa+77RDB0C7PXkj6KiBK8iyKcxcOBxO4dCFo9EUjnJxB/+JmcRsQgkO1UkeSwEwo6i7ilpthtZcCkKYhlJ+HwuIExpy5ggCc+NmFth3FxhfC7sAsZ1sGpHtuiGujEar3RmJozjI4mlZDtwIx8dTeKxtwWcp4X2b/wntRrgldA6SW0jcmphO6ZuGG+9/cvwVN3gsLD2MoJtfPRbklrmLnxztT7MeV20fzD0X8N64DmMQ2MR9oXwsTGqmrcy/X7bWX4XKlyaNupuAyR5Pc8du7iHvBmCLZwzfWk76tmF7WOa71Z6KblSbJixkUplPvVGqbtpXruq+aXnjbHur+nJr+8NOae/N5L1bRs/Qc7SBqugV2kM1dIROEUGf0A/0E/0qfCx8LnwpfL2WLi5M5jxFc6Pw/Q9pMi93</latexit>T[2,4] (z)
6
5
4
3
2

thresholding operator

Approximate empirical Bayes and tractable.

p

Hidden layer h ∈ ℝ with output ŷ ∈ ℝ for regression⊤

yi ) +

1/2
1/2 2||⌦r W ⌦c ||F−1−1
Ωr := Σr , Ωc := Σc : precision matrices(2π)pd/2det(Σr)d/2det(Σc)p/2

r

2

i2[n]

<latexit sha1_base64="efN9CQP2zibhOui91CjtUzwg8NM=">AAAGwnicfZTfbxtFEMfPbTHF/GrKI0JaEbWyIQ22QQIJKlUiRekDasiPplLWPe3tje1t7nYvu3uOnfXxxCN/Ca/wz/DfMHt2YmcNrGTfeD7fnZnbHU9SZMLYbvfvxp27995pvnv/vdb7H3z40ccPth6+MqrUHE64ypR+nTADmZBwYoXN4HWhgeVJBqfJ+Y+en05AG6HksZ0VMMjZSIqh4MyiK95qfPaY0FzI2J3u0AlwVi1+0ULrHYLfvCKPUTLUjLte5fqyoqbMYyeokGdyULVpMWvT6TQW35NliA55Qmax6Lzpky8JzbCYlJH53Id843pf9atTH3dhzufxT6h7spTRRIzaKc3UKAXb9js6GKNYc/CO13QIlUqWeQKa0hbBhTVamFpnyuQtcEusqgsvyYu48O8BHC78U1/bEwQ7v9aCdI3zG+5B/GC7u9utF9k0ektjO1qug3jr3u80VbzMQVqeMWPOet3CDhzTVvAMqhYtDRSMn7MRnKEpWQ5m4OqLrMgj9KRkqDR+pCW1d32HY7kxszxBZc7s2ITMO/+L2XFehT6fxVSkXrcYiq1Smbm9w8fddJ+VdvjdwAlZlBYkX7zFsMzwCohvOZIKPFGbzdBgXAs8CMLHDBvKYmO2HpEwRcGuVNW6lSJn56CVyvGkmOSQYQESLrnKcyZTh62YVo76vUniXlQBvQQxGttrwdBdhoLpdAWnIZzNVnAWwqurFbzayHv5PzlNIZeUs8wdbWKrAdYUx6Fif3+N7ocUtMaGcnTCNBRGZEoGghRny1qAvTBACkO48AG472RsRPk0THGBmso9XZMEiufTYnUtz8MMh4creFj5C1+nXE18/UdilLNYVxuQ30Ae3gl2Co4hW7//8vi7YXIcBEhf5jDajO6H3jULgx8d7K2qrm9tD/B/ruFndL0sQDPM+4WjTGNp2Fb4pDve+lehoxbLOA4LqOfwTe0srH0xDKvaoN4KyvcNO2bWdyvOr144rTaNV/3d3te7/V++2X72w3KS3Y8+jT6P2lEv+jZ6Fu1HB9FJxBu/Nf5o/Nn4q7nXfNu8aJqF9E5jueeT6NZqzv8BwYRvaw==</latexit>−1−1 ⊤exp( − Tr(Σr WΣc W )/2)c

1 X
(ŷ(xi ; W, a)2nsubject to uIppdΣr ∈ 𝕊++, Σc ∈ 𝕊++ : row and column covariance matricesrMotivation

Optimizationp×dŷ = a h, h = g(Wx), W ∈ ℝ

0

0

1

2

3

4

5

6 z

Experiments
AdaReg optimization trajectory on MNIST (train size/batch size)
600/256

600/600

6000/256

6000/2048

Loss function12ℓ(W, a) = (ŷ − y)2Update for weight matrix W

Stable Rank and Spectral Norm for Generalization Error(Neyshabur et al. ICLR’17)
v
!
u L
L
X
uY
2
t
=
O
||W
||
srank(Wj )/n
Generalization Error
j 2
j=1

j=1

<latexit sha1_base64="nzPs7H3xSM/lFs/jD8+AyAQjtcc=">AAAGGXicfZTNbhMxEMfd0kAJXykcuayIKqUIlWxAggORKmilIIFS+i1l08jxOpttdu2t7Xw6fhLeAokH4Ia4ckJc4T2w07RJvIClKLPz+3tm1jPrZhKFXBSLP5aWr61krt9YvZm9dfvO3Xu5tftHnHYZwoeIRpSdNCHHUUjwoQhFhE8ShmHcjPBxs/PG8OMeZjyk5EAME1yPYUDCVoig0K5GruqUnar3OgyCgsfPmZBewqjfkGdlV52+G4+PG2fjcaN0WvJ4N750O57AAyE5g6SjClqy8ZSoSZCNRi5f3CxOlpM23KmRB9O121hb+eT5FHVjTASKIOc1t5iIuoRMhCjCKut1OU4g6sAA17RJYIx5XU7eXDnr2uM7Lcr0jwhn4p3fIWHM+TBuamUMRZvbzDj/xUQ7VrbPZOHKmawFpsWC0ogv7jBx0+5aV7Re1mVIkq7ABF28RasbOYI6pkeOHzKMRDTUBkQs1AfhoDZkEAndyey6Y6dI4Iiq7EKKGHYwozTWJwUJwpEugOA+onEMiS+9kPhKemZvsynfKov2cRi0xaWgJfu2YDCYwYENh8MZHNpwNJrBUSpv/z85eUKmFMFI7qexYBjPKQ5sRaUyRys2xYzpgZJeDzKc8DCixBL4+mOcC7BtB/BxC5+bAMhMsh5EUrZTnGuNkuU5iaXYGSSztuzYGfb2ZnBPmYbPU0R7pv79MIhhg6kURFcQ2T3Rk9LTAzd5/+nxF+3kCTO0GuMgHT1h6IrZwfd3t2dVT7q2jfV3zvB77aommEGd97H0INOl6bHS/94TY/1VKD2hyziwC9DVw1nt0K49ooGPTfO04RnLKt8MbBuK9LSaU1E1ty4vLjz9qAp5d0PpW86177S0cVTadJ9tlj48z2+9mt53q+AheAQKwAUvwBaogF1wCBD4DH6CX+B35mPmS+Zr5tuFdHlpuucBWFiZ738AbF46Mg==</latexit>⊤

W ← W − γ(ŷ − y)(a ∘ h′)x

MNISTh′ : component-wise derivative of h w.r.t. its input⊤(ŷ − y)(a ∘ h′)x : gradient is always rank-1→ rows/columns of W are correlated!
A: Update each row/column by taking informationfrom other rows/columns into consideration!

1

CIFAR-10

MNIST

Batch size = 256

Batch size = 2048

Batch size = 256

Batch size = 2048

CIFAR-10

srank(W ) := ||W ||2F /||W ||22<latexit sha1_base64="sqHXDmPUtsjh5xtbRCY/2W+xCyE=">AAAF83icfVTNbhMxEHZLAyX8tIUjlxVRpRahkgQkOFCpEi0KB9TSv1TKppHjnSSr7Npb25s/Z5+BB+CGuHJB4grPwdtgb0KTeAFLUWbn+zzzeWbsZhT4QhaLv5aWb6zkbt5avZ2/c/fe/bX1jQfngsWcwBlhAeMXTSwg8CmcSV8GcBFxwGEzgGqz+8bg1R5w4TN6KocR1EPcpn7LJ1hqV2N925UwkEpwTLvJVnXb9aAFV854XB2PG28vy88mVvmy3FgvFHeK6XKyRmlqFNB0HTU2Vr65HiNxCFSSAAtRKxUjWVeYS58EkOTdWECESRe3oaZNikMQdZWeKXE2tcdzWozrH5VO6p3foXAoxDBsamaIZUfYmHH+C5OdMLF9JotInHQtYJosGQvE4g4TN+uuxbL1qq58GsUSKJmcohUHjmSOqb7j+RyIDIbawIT7uhAO6WCOidQ9ym86dooIj1iSX0gR4i5wxkJdKUwJBFoAhT5hYYipp1yfeolyzd5mU71LLLQPfrsj/xBaqm8TBoMZOLDB4XAGDm1wNJqBo0ze/n9yiohOUYIDdZKFJQeYY5zajEplDq3YKHCuB0q5PcwhEn7AqEXw9DWbC7BvB0ivhAlAzCTrQaS7doorzUnU7hzFYhwMollbDuwMx8cz8DgxDZ9HCesZ/Sd+O8QNnmRAcg0Suyd6Unp64NLzT8tftJNH3KCHIbSz0SNOrjE7+MnR/kx12rV90Pecw3vtOoyAY533iXIx19L0WOl/96mx/kpUrtQyTm0BWj2eace29oC1PTDN04ZrLEu+GdgOltlpNVVJaqW6mrx/+jPZKpS2E/3Klew3LWucl3dKz3fKH14U9l5P37tV9Ag9RluohF6iPVRBR+gMEfQRfUc/0M9cnPuU+5z7MqEuL033PEQLK/f1N2awKzo=</latexit>

1  srank(W )  rank(W )<latexit sha1_base64="+6RVU/TSAn78aBiwDq/s7veoMPw=">AAAF83icfVTNbhMxEHZLAyX8tXDksiKq1CJUZQsSHKhUiVYKB9TQfykbRV7vJFnFa29t59faZ+ABuCGuXJC4wnPwNthp2my8gKXVzs73eebzzKzDlMZSVau/l5ZvrZRu31m9W753/8HDR2vrj88k7wsCp4RTLi5CLIHGDE5VrChcpAJwElI4D3vvLH4+ACFjzk7UOIVmgjssbscEK+NqrW35XkDh0gsUjJSWArNetnm+lfNdu1prlep2dbq8ouHPjAqarXprfeV7EHHST4ApQrGUDb+aqqbGQsWEQlYO+hJSTHq4Aw1jMpyAbOrpmTJvw3gir82FeZjypt78Do0TKcdJaJgJVl3pYtb5L0x1k8z12Swy86ZrATNkxTmVizts3KK70VftN00ds7SvgJGrU7T71FPcs9X3olgAUXRsDExEbArhkS4WmCjTo/KG56ZI8YRn5YUUCe6B4DwxlcKMADUCGAwJTxLMIh3ELMp0YPeGoX6fOegQ4k5XXRPaeugSRqM5OHLB8XgOjl1wMpmDk0Le4X9yypTNUIKpPi7CSgDkGCcuo1bLoTUXBSHMQOlggAWkMqacOYTI/Ga5APtugAjacGkDEDvJZhDZrpvi0nAyvZujOIyDUTpvy4Gb4ehoDh5ltuF5lPCB1X8cdxLcElkBJDcgcXtiJmVgBm56/ln5q27yVFj0MIFOMXoqyA3mBj+u789VT7u2D+Y/F/DBuA5TENjkfa4DLIw0M1bmHbyw1l+JOlBGxokrwKjHc+3Y1U55JwLbPGME1nLk24HtYlWcVluVrOE39dUlZz6zzYq/lZlbznfvtKJxtrPtv9ze+fiqsvd2dt+toqfoGdpEPnqN9lAN1dEpIugT+oF+ol+lfulz6Uvp6xV1eWm25wlaWKVvfwBq8Crg</latexit>