A Representer Theorem for Vector-Valued
Neural Networks: Insights on Weight
Decay Regularization and Widths of DNNs
Joe Shenouda
UW-Madison ECE

Rahul Parhi

Kangwook Lee

Rob Nowak

DP4ML ICML 2023

Motivations• Vector valued NNs are key to understanding deep networks and multi-task
learning.

• Weight decay is the most popular explicit regularizer for training deep neuralff

networks (DNNs) and can have a drastic e ect on the generalization ability of
the network.

Background• Training a DNN with weight decay corresponds to minimizing a data delity
loss plus the sum of squared weights.

Weight Decay ObjectivefiTraining with Weight Decay

Neural Balance Theorem
Theorem (NBT [Yang 2022, Parhi 2023])
Let fθ be a function represented by a homogenous DNN that solves the weightdecay objective.
<latexit sha1_base64="CtNHICdIQBrCDLY7wCLvR8DfUYE=">AAACvXicbVFdb9MwFHXC1ygfK/DIi8WE1IqpSvYACAkxsQeQQGhIdJtUd8FxnM3UdiLb2ahc/wHe+GkgXvgziJt2SF3Llawcn3t8fXKc11JYlyS/o/jK1WvXb2zc7Ny6fefuZvfe/QNbNYbxIatkZY5yarkUmg+dcJIf1YZTlUt+mE/22v7hGTdWVPqTm9Z8rOiJFqVg1AGVdX8RJXTmiSd5JQs7VfDx5yGbbONl5gwYEgImtlGZFy/TcOw/wFZRd8qo9O9Db1k+DZnYxuUl7itw/T5+gklpKPNEgsmCBr/zb+pkPvVdwD0yw6t2MJkdgzRr5TBituZuSdDPulvJIJkXXgfpBdjaffNt7w/5/mM/6/4kRcUaxbVjklo7SpPajT01TjDJQ4c0lteUTegJHwHUVHE79vP0A34MTIHLysDSDs/Z5ROeKtsaBWUbl13tteT/eqPGlc/HXui6cVyzxUVlI7GrcPuUuBCGMyenACgzArxidkohXAcP3oEQ0tVfXgcHO4P06SD9CGm8RovaQA/RI9RDKXqGdtFbtI+GiEUvos+RiL7Er2Iey1gvpHF0ceYBulTx+V/nkuEZ</latexit>{vk }<latexit sha1_base64="0fXHA9YWvCyGtuzQW2EFaRf8n7Y=">AAAB7nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lE1GPRi8cK9gOaUDbbSbt0swm7m0IJ/RFePCji1d/jzX/jts1Bqw8GHu/NMDMvTAXXxnW/nNLa+sbmVnm7srO7t39QPTxq6yRTDFssEYnqhlSj4BJbhhuB3VQhjUOBnXB8N/c7E1SaJ/LRTFMMYjqUPOKMGit1/HzSH/uzfrXm1t0FyF/iFaQGBZr96qc/SFgWozRMUK17npuaIKfKcCZwVvEzjSllYzrEnqWSxqiDfHHujJxZZUCiRNmShizUnxM5jbWexqHtjKkZ6VVvLv7n9TIT3QQ5l2lmULLloigTxCRk/jsZcIXMiKkllClubyVsRBVlxiZUsSF4qy//Je2LundV9x4ua43bIo4ynMApnIMH19CAe2hCCxiM4Qle4NVJnWfnzXlftpacYuYYfsH5+AaMiI+2</latexit>{wk }<latexit sha1_base64="/fNLA/8UvZPVsagCxA38JVAC8hA=">AAAB7nicbVBNS8NAEJ34WetX1aOXxSJ4KomIeix68VjBfkATymY7aZduNmF3o5TQH+HFgyJe/T3e/Ddu2xy09cHA470ZZuaFqeDauO63s7K6tr6xWdoqb+/s7u1XDg5bOskUwyZLRKI6IdUouMSm4UZgJ1VI41BgOxzdTv32IyrNE/lgxikGMR1IHnFGjZXafv7UG/mTXqXq1twZyDLxClKFAo1e5cvvJyyLURomqNZdz01NkFNlOBM4KfuZxpSyER1g11JJY9RBPjt3Qk6t0idRomxJQ2bq74mcxlqP49B2xtQM9aI3Ff/zupmJroOcyzQzKNl8UZQJYhIy/Z30uUJmxNgSyhS3txI2pIoyYxMq2xC8xZeXSeu85l3WvPuLav2miKMEx3ACZ+DBFdThDhrQBAYjeIZXeHNS58V5dz7mrStOMXMEf+B8/gCOEI+3</latexit>min{wk ,v k }

N
X
i=1

L(y i , f (xi )) +

K
X

2

k=1

2
2
(kwk k2 + kv k k2 )Then for any neuron with input weight wk and output weight v k we have kwk k2 = kv k k2
<latexit sha1_base64="nE33XBG1g6CbNDKQKxFD0o9pD6s=">AAACG3icbVDLSsNAFJ34rPUVdelmaBEEoSRdqBuh6MZlBfuAJoTJZNIOnTyYmVRC2r9w4cZfceNCEVeCi/6Nk7ZCbT0w3MM59zL3HjdmVEjDGGsrq2vrG5uFreL2zu7evn5w2BRRwjFp4IhFvO0iQRgNSUNSyUg75gQFLiMtt3+T+60B4YJG4b1MY2IHqBtSn2IkleToVWtouRHzRBqokj2MnD60hk4VXqkC563Br+XoZaNiTACXiTkj5VrJOnsc19K6o39ZXoSTgIQSMyRExzRiaWeIS4oZGRWtRJAY4T7qko6iIQqIsLPJbSN4ohQP+hFXL5Rwos5PZCgQ+YaqM0CyJxa9XPzP6yTSv7QzGsaJJCGefuQnDMoI5kFBj3KCJUsVQZhTtSvEPcQRlirOogrBXDx5mTSrFfO8Yt6pNK7BFAVwDErgFJjgAtTALaiDBsDgCbyAN/CuPWuv2of2OW1d0WYzR+APtO8fehSkpg==</latexit><latexit sha1_base64="79aBhXJEAJT+8Z4Wmb19F0ji97M=">AAAB+XicbVDLSgMxFL1TX7W+RgU3boJFcFVmXKjLUjcuW7APaIchk0nb0ExmSDKFMvRP3LhQRHDlL/gF7tz4LWbaLrT1QMjhnHvJyQkSzpR2nC+rsLa+sblV3C7t7O7tH9iHRy0Vp5LQJol5LDsBVpQzQZuaaU47iaQ4CjhtB6Pb3G+PqVQsFvd6klAvwgPB+oxgbSTftntBzEM1icyVjaf+yLfLTsWZAa0Sd0HK1ZPGN3urfdR9+7MXxiSNqNCEY6W6rpNoL8NSM8LptNRLFU0wGeEB7RoqcESVl82ST9G5UULUj6U5QqOZ+nsjw5HKw5nJCOuhWvZy8T+vm+r+jZcxkaSaCjJ/qJ9ypGOU14BCJinRfGIIJpKZrIgMscREm7JKpgR3+curpHVZca8qbsO0UYM5inAKZ3ABLlxDFe6gDk0gMIYHeIJnK7MerRfrdT5asBY7x/AH1vsPMVuXuQ==</latexit><latexit sha1_base64="6tjKnfGib50RuHePoaJtJuzPSWE=">AAAB+XicbVDLSgMxFL3js9bXqODGTbAIrsqMC3VZ6sZlC/YBbRkymUwbmskMSaZShv6JGxeKCK78Bb/AnRu/xUzbhbYeCDmccy85OX7CmdKO82WtrK6tb2wWtorbO7t7+/bBYVPFqSS0QWIey7aPFeVM0IZmmtN2IimOfE5b/vAm91sjKhWLxZ0eJ7QX4b5gISNYG8mz7a4f80CNI3Nl9xNv6Nklp+xMgZaJOyelynH9m71VP2qe/dkNYpJGVGjCsVId10l0L8NSM8LppNhNFU0wGeI+7RgqcERVL5smn6AzowQojKU5QqOp+nsjw5HKw5nJCOuBWvRy8T+vk+rwupcxkaSaCjJ7KEw50jHKa0ABk5RoPjYEE8lMVkQGWGKiTVlFU4K7+OVl0rwou5dlt27aqMIMBTiBUzgHF66gArdQgwYQGMEDPMGzlVmP1ov1OhtdseY7R/AH1vsPMuKXug==</latexit>

Balanced!

The Secret Sparsity of Weight Decay
Weight decay

Path-norm

After Rebalancing

Group lasso

SecretSparsity

Outline of Contributions
Vector-Valued Variation Spaces(VV Spaces)• Representer Theorem showingfishallow vector-valued NNs trainedwith weight decay solve data tting
problem over the VV Space.

Bounds on Necessary Width• Tighter bounds on the necessary

width for any homogenous layer of
DNN, depending only on the rank of
the internal features.

VV Spaces

Vector Valued Variation Spaces (VV Spaces)
Vector Measurefi(For nite-width networks)

Representer Theorem

Also extends to deep neural networks

N
For any dataset {xi, yi}i=1 and any lower semicontinuous loss function

there

exists a solution to,with the following representation<latexit sha1_base64="tiIJXHD12VZrLLKZAqI6IEInu4M=">AAAB9XicbVDLSgNBEOz1GeMr6tHLYBA8hd0g6jHoRRAkgnlAsgmzk95kyOzDmVklLPkPLx4U8eq/ePNvnCR70MSChqKqm+4uLxZcadv+tpaWV1bX1nMb+c2t7Z3dwt5+XUWJZFhjkYhk06MKBQ+xprkW2Iwl0sAT2PCGVxO/8YhS8Si816MY3YD2Q+5zRrWROjddm7QFPpDbTloedwtFu2RPQRaJk5EiZKh2C1/tXsSSAEPNBFWq5dixdlMqNWcCx/l2ojCmbEj72DI0pAEqN51ePSbHRukRP5KmQk2m6u+JlAZKjQLPdAZUD9S8NxH/81qJ9i/clIdxojFks0V+IoiOyCQC0uMSmRYjQyiT3NxK2IBKyrQJKm9CcOZfXiT1csk5Kzl3p8XKZRZHDg7hCE7AgXOowDVUoQYMJDzDK7xZT9aL9W59zFqXrGzmAP7A+vwBFfKRmg==</latexit>

K0  N2

The dual space of vector-valued continuous functions is the space of vector-valued measures.

sparse neurons

sparse weights

dense weights<latexit sha1_base64="/gVg2WIFg7ahiGTCxh5lGuX83dA=">AAAB9XicbVA9TwJBEN3DL8Qv1NJmIzGxIncUakm0scREwAROsrfMwYa9vcvunIRc+B82Fhpj63+x89+4wBUKvmSSl/dmMjMvSKQw6LrfTmFtfWNzq7hd2tnd2z8oHx61TJxqDk0ey1g/BMyAFAqaKFDCQ6KBRYGEdjC6mfntJ9BGxOoeJwn4ERsoEQrO0EqPJmHaAB2DGAzR9MoVt+rOQVeJl5MKydHolb+6/ZinESjkkhnT8dwE/YxpFFzCtNRNDSSMj9gAOpYqFoHxs/nVU3pmlT4NY21LIZ2rvycyFhkziQLbGTEcmmVvJv7ndVIMr/xMqCRFUHyxKEwlxZjOIqB9oYGjnFjCuBb2VsqHTDOONqiSDcFbfnmVtGpV76Lq3dUq9es8jiI5IafknHjkktTJLWmQJuFEk2fySt6csfPivDsfi9aCk88ckz9wPn8Ay0SSsw==</latexit><latexit sha1_base64="NuR/FFhsDyxMsdoMMosfVUi6OEg=">AAAB9HicbVA9SwNBEN3zM8avqKXNYhCswl0KtQzaWEYwH5AcYW9vkizZ2zt35yLhyO+wsVDE1h9j579xk1yhiQ8GHu/NMDMvSKQw6Lrfztr6xubWdmGnuLu3f3BYOjpumjjVHBo8lrFuB8yAFAoaKFBCO9HAokBCKxjdzvzWGLQRsXrASQJ+xAZK9AVnaCU/BGWAPoEYDNH0SmW34s5BV4mXkzLJUe+VvrphzNMIFHLJjOl4boJ+xjQKLmFa7KYGEsZHbAAdSxWLwPjZ/OgpPbdKSPuxtqWQztXfExmLjJlEge2MGA7NsjcT//M6Kfav/UyoJEVQfLGon0qKMZ0lQEOhgaOcWMK4FvZWyodMM442p6INwVt+eZU0qxXvsuLdV8u1mzyOAjklZ+SCeOSK1MgdqZMG4eSRPJNX8uaMnRfn3flYtK45+cwJ+QPn8wffWpIq</latexit><latexit sha1_base64="WF86FMm4p//67OH70eJ1cpfupzU=">AAAB9XicbVA9SwNBEJ2LXzF+RS1tFoNgFe5SqGXQxjKC+YDkDHubuWTJ3t6xu6eEI//DxkIRW/+Lnf/GTXKFJj4YeLw3w8y8IBFcG9f9dgpr6xubW8Xt0s7u3v5B+fCopeNUMWyyWMSqE1CNgktsGm4EdhKFNAoEtoPxzcxvP6LSPJb3ZpKgH9Gh5CFn1FjpQSdUaSQSUxVL3S9X3Ko7B1klXk4qkKPRL3/1BjFLI5SGCap113MT42dUGc4ETku9VGNC2ZgOsWuppBFqP5tfPSVnVhmQMFa2pCFz9fdERiOtJ1FgOyNqRnrZm4n/ed3UhFd+xmWSGpRssShMBTExmUVABlwhM2JiCWWK21sJG1FFmbFBlWwI3vLLq6RVq3oXVe+uVqlf53EU4QRO4Rw8uIQ63EIDmsBAwTO8wpvz5Lw4787HorXg5DPH8AfO5w/iI5LC</latexit>

K<latexit sha1_base64="qiPhyD5VjJZZm256wGoJdH9vNwY=">AAAB6HicbVDLSsNAFJ3UV62vqks3g0VwFRJpY90V3QhuWrAPaEOZTG/asZNJmJkIJfQL3LhQxK2f5M6/cfpAfB24cDjnXu69J0g4U9pxPqzcyura+kZ+s7C1vbO7V9w/aKk4lRSaNOax7AREAWcCmpppDp1EAokCDu1gfDXz2/cgFYvFrZ4k4EdkKFjIKNFGatz0iyXHdrxquexhx65UvIuqh13bmeOLlNAS9X7xvTeIaRqB0JQTpbquk2g/I1IzymFa6KUKEkLHZAhdQwWJQPnZ/NApPjHKAIexNCU0nqvfJzISKTWJAtMZET1Sv72Z+J/XTXVY9TMmklSDoItFYcqxjvHsazxgEqjmE0MIlczciumISEK1yaZgQvjz8l/SOrNdz3Yb5VLtchlHHh2hY3SKXHSOauga1VETUQToAT2hZ+vOerRerNdFa85azhyiH7DePgEHW40Z</latexit>

D<latexit sha1_base64="wgBYvU2CHnigf/197+/YZopAzL8=">AAAB6HicbVDLSsNAFJ3UV62vqks3g0VwFRJpY90VdeGyBfuANpTJ9KYdO5mEmYlQQr/AjQtF3PpJ7vwbpw/E14ELh3Pu5d57goQzpR3nw8qtrK6tb+Q3C1vbO7t7xf2DlopTSaFJYx7LTkAUcCagqZnm0EkkkCjg0A7GVzO/fQ9SsVjc6kkCfkSGgoWMEm2kxnW/WHJsx6uWyx527ErFu6h62LWdOb5ICS1R7xffe4OYphEITTlRqus6ifYzIjWjHKaFXqogIXRMhtA1VJAIlJ/ND53iE6MMcBhLU0Ljufp9IiORUpMoMJ0R0SP125uJ/3ndVIdVP2MiSTUIulgUphzrGM++xgMmgWo+MYRQycytmI6IJFSbbAomhD8v/yWtM9v1bLdRLtUul3Hk0RE6RqfIReeohm5QHTURRYAe0BN6tu6sR+vFel205qzlzCH6AevtE/ywjRI=</latexit><latexit sha1_base64="x2vSyKfa4Hhii+D+BHdm0XG9PNA=">AAACbnicdVFNT9wwFHRSoDQtsIDUQxGq1RXSclnFCwocUeHAEaQuIG1WK8d5WSwcO7Id1FWUY/8gN34Dl/4EHDaVyteTLI9m5vnZ46QQ3NgwvPf8DwuLSx+XPwWfv6ysrnXWNy6MKjWDIVNC6auEGhBcwtByK+Cq0EDzRMBlcnPc6Je3oA1X8pedFTDO6VTyjDNqHTXp/IkTmHJZUa3prK5YHWSTitS9OFEiNbPcbdXvejeOG37wisdzYe+thvg2Vda0jpNXjiAGmbZzJ51u2A/DQXi4jxsQ7UWkASQkUYSJY5rqorbOJp27OFWszEFaJqgxIxIWduxOs5wJqIO4NFBQdkOnMHJQ0hzMuHqKq8Y7jklxprRb0uIn9v+Oiuamuadz5tRem5daQ76ljUqbHY4rLovSgmTzQVkpsFW4yR6nXAOzYuYAZZq7u2J2TTVl1v1Q4EL491L8PrgY9EnUJ+f73aOfbRzLaAv9QD1E0AE6QqfoDA0RQw/euvfN2/L++l/9bf/73Op7bc8melZ+7xH+M72Z</latexit><latexit sha1_base64="vWETCePq9kDZ6ipa7TzVSb1Xay4=">AAACDHicbVDLSsNAFL3xWeur6tLNYBHqpiQi6kYo6sKdFewDmlAm00k7dPJwZqKUNB/gxl9x40IRt36AO//GSduFth4YOJxzLnPvcSPOpDLNb2NufmFxaTm3kl9dW9/YLGxt12UYC0JrJOShaLpYUs4CWlNMcdqMBMW+y2nD7V9kfuOeCsnC4FYNIur4uBswjxGstNQuFO2hZw/bie1j1SOYJ/U0RWfouvSAbHknVHKZHuiUWTZHQLPEmpAiTFBtF77sTkhinwaKcCxlyzIj5SRYKEY4TfN2LGmESR93aUvTAPtUOsnomBTta6WDvFDoFyg0Un9PJNiXcuC7OpntLKe9TPzPa8XKO3USFkSxogEZf+TFHKkQZc2gDhOUKD7QBBPB9K6I9LDAROn+8roEa/rkWVI/LFvHZevmqFg5n9SRg13YgxJYcAIVuIIq1IDAIzzDK7wZT8aL8W58jKNzxmRmB/7A+PwBPfCbFg==</latexit>

p

K D)
kfkf
kVkVV = O(w<latexit sha1_base64="yyFkYeMy0QdQovMaHZU6EazKJVU=">AAAB83icbVBNS8NAEJ34WetX1aOXxSJ4KomIehGKXjxWsGmhCWWz3bRLN5tldyOUtH/DiwdFvPpnvPlv3LY5aOuDgcd7M8zMiyRn2rjut7Oyura+sVnaKm/v7O7tVw4OfZ1mitAmSXmq2hHWlDNBm4YZTttSUZxEnLai4d3Ubz1RpVkqHs1I0jDBfcFiRrCxUhCM42DczX1/gm66lapbc2dAy8QrSBUKNLqVr6CXkiyhwhCOte54rjRhjpVhhNNJOcg0lZgMcZ92LBU4oTrMZzdP0KlVeihOlS1h0Ez9PZHjROtREtnOBJuBXvSm4n9eJzPxdZgzITNDBZkvijOOTIqmAaAeU5QYPrIEE8XsrYgMsMLE2JjKNgRv8eVl4p/XvMua93BRrd8WcZTgGE7gDDy4gjrcQwOaQEDCM7zCm5M5L8678zFvXXGKmSP4A+fzB6aWkW4=</latexit><latexit sha1_base64="V6/WcJKESFjSMsRS3eHvMW+TXKI=">AAAB6HicdVDJSgNBEO2JW4xb1KOXxiB4GmZitmPQi+AlAbNAMoSeTk3Spmehu0cIQ77AiwdFvPpJ3vwbeyYRVPRBweO9KqrquRFnUlnWh5FbW9/Y3MpvF3Z29/YPiodHXRnGgkKHhjwUfZdI4CyAjmKKQz8SQHyXQ8+dXaV+7x6EZGFwq+YROD6ZBMxjlCgttW9GxZJlWnXbqlZwRuxqLSW1Rrl6gW3TylBCK7RGxffhOKSxD4GinEg5sK1IOQkRilEOi8IwlhAROiMTGGgaEB+kk2SHLvCZVsbYC4WuQOFM/T6REF/Kue/qTp+oqfztpeJf3iBWXsNJWBDFCgK6XOTFHKsQp1/jMRNAFZ9rQqhg+lZMp0QQqnQ2BR3C16f4f9Itm3bNtNuVUvNyFUcenaBTdI5sVEdNdI1aqIMoAvSAntCzcWc8Gi/G67I1Z6xmjtEPGG+f+/GNEg==</latexit><latexit sha1_base64="x2vSyKfa4Hhii+D+BHdm0XG9PNA=">AAACbnicdVFNT9wwFHRSoDQtsIDUQxGq1RXSclnFCwocUeHAEaQuIG1WK8d5WSwcO7Id1FWUY/8gN34Dl/4EHDaVyteTLI9m5vnZ46QQ3NgwvPf8DwuLSx+XPwWfv6ysrnXWNy6MKjWDIVNC6auEGhBcwtByK+Cq0EDzRMBlcnPc6Je3oA1X8pedFTDO6VTyjDNqHTXp/IkTmHJZUa3prK5YHWSTitS9OFEiNbPcbdXvejeOG37wisdzYe+thvg2Vda0jpNXjiAGmbZzJ51u2A/DQXi4jxsQ7UWkASQkUYSJY5rqorbOJp27OFWszEFaJqgxIxIWduxOs5wJqIO4NFBQdkOnMHJQ0hzMuHqKq8Y7jklxprRb0uIn9v+Oiuamuadz5tRem5daQ76ljUqbHY4rLovSgmTzQVkpsFW4yR6nXAOzYuYAZZq7u2J2TTVl1v1Q4EL491L8PrgY9EnUJ+f73aOfbRzLaAv9QD1E0AE6QqfoDA0RQw/euvfN2/L++l/9bf/73Op7bc8melZ+7xH+M72Z</latexit><latexit sha1_base64="x2vSyKfa4Hhii+D+BHdm0XG9PNA=">AAACbnicdVFNT9wwFHRSoDQtsIDUQxGq1RXSclnFCwocUeHAEaQuIG1WK8d5WSwcO7Id1FWUY/8gN34Dl/4EHDaVyteTLI9m5vnZ46QQ3NgwvPf8DwuLSx+XPwWfv6ysrnXWNy6MKjWDIVNC6auEGhBcwtByK+Cq0EDzRMBlcnPc6Je3oA1X8pedFTDO6VTyjDNqHTXp/IkTmHJZUa3prK5YHWSTitS9OFEiNbPcbdXvejeOG37wisdzYe+thvg2Vda0jpNXjiAGmbZzJ51u2A/DQXi4jxsQ7UWkASQkUYSJY5rqorbOJp27OFWszEFaJqgxIxIWduxOs5wJqIO4NFBQdkOnMHJQ0hzMuHqKq8Y7jklxprRb0uIn9v+Oiuamuadz5tRem5daQ76ljUqbHY4rLovSgmTzQVkpsFW4yR6nXAOzYuYAZZq7u2J2TTVl1v1Q4EL491L8PrgY9EnUJ+f73aOfbRzLaAv9QD1E0AE6QqfoDA0RQw/euvfN2/L++l/9bf/73Op7bc8melZ+7xH+M72Z</latexit>

f1 (x)
f2 (x)
f3 (x)..
.

f1 (x)
f2 (x)
f3 (x)..
.

f1 (x)
f2 (x)
f3 (x)..
.

fD (x)fD (x)fD (x)<latexit sha1_base64="IYScbpw9nRTfpSnE5zx49B2juIE=">AAACCnicdVDLSsNAFJ34rPUVdelmtAh1U5JaYjdCURfurGAf0IQymU7aoZOHMxOhpFm78VfcuFDErV/gzr9x0lZQ0QMXDufcy733uBGjQhrGhzY3v7C4tJxbya+urW9s6lvbTRHGHJMGDlnI2y4ShNGANCSVjLQjTpDvMtJyh2eZ37olXNAwuJajiDg+6gfUoxhJJXX1PXvs2eNuYvtIDjBiSTNN4Qm8LNrihsvkPD3s6gWjZBhlo1qBGbGOLDMjpmFaFjSVkqEAZqh39Xe7F+LYJ4HEDAnRMY1IOgnikmJG0rwdCxIhPER90lE0QD4RTjJ5JYUHSulBL+SqAgkn6veJBPlCjHxXdWYXi99eJv7ldWLpVZ2EBlEsSYCni7yYQRnCLBfYo5xgyUaKIMypuhXiAeIIS5VeXoXw9Sn8nzTLJdMqmVeVQu10FkcO7IJ9UAQmOAY1cAHqoAEwuAMP4Ak8a/fao/aivU5b57TZzA74Ae3tEzZlmpc=</latexit>

kfkf
kVkVV = O(D)<latexit sha1_base64="yyFkYeMy0QdQovMaHZU6EazKJVU=">AAAB83icbVBNS8NAEJ34WetX1aOXxSJ4KomIehGKXjxWsGmhCWWz3bRLN5tldyOUtH/DiwdFvPpnvPlv3LY5aOuDgcd7M8zMiyRn2rjut7Oyura+sVnaKm/v7O7tVw4OfZ1mitAmSXmq2hHWlDNBm4YZTttSUZxEnLai4d3Ubz1RpVkqHs1I0jDBfcFiRrCxUhCM42DczX1/gm66lapbc2dAy8QrSBUKNLqVr6CXkiyhwhCOte54rjRhjpVhhNNJOcg0lZgMcZ92LBU4oTrMZzdP0KlVeihOlS1h0Ez9PZHjROtREtnOBJuBXvSm4n9eJzPxdZgzITNDBZkvijOOTIqmAaAeU5QYPrIEE8XsrYgMsMLE2JjKNgRv8eVl4p/XvMua93BRrd8WcZTgGE7gDDy4gjrcQwOaQEDCM7zCm5M5L8678zFvXXGKmSP4A+fzB6aWkW4=</latexit><latexit sha1_base64="wCI4wCPYxmKFjKB4HW8AtSQOOic=">AAACA3icdVDLSsNAFJ3UV62vqDvdDBahbkpSS+xGKOrCnRXsA5pQJtNJO3TyYGYilDTgxl9x40IRt/6EO//GSVtBRQ9cOJxzL/fe40aMCmkYH1puYXFpeSW/Wlhb39jc0rd3WiKMOSZNHLKQd1wkCKMBaUoqGelEnCDfZaTtjs4zv31LuKBhcCPHEXF8NAioRzGSSurpe/bEsye9xPaRHGLEklaawlN4Vbo46ulFo2wYFaNWhRmxji0zI6ZhWhY0lZKhCOZo9PR3ux/i2CeBxAwJ0TWNSDoJ4pJiRtKCHQsSITxCA9JVNEA+EU4y/SGFh0rpQy/kqgIJp+r3iQT5Qox9V3Vmp4rfXib+5XVj6dWchAZRLEmAZ4u8mEEZwiwQ2KecYMnGiiDMqboV4iHiCEsVW0GF8PUp/J+0KmXTKpvX1WL9bB5HHuyDA1ACJjgBdXAJGqAJMLgDD+AJPGv32qP2or3OWnPafGYX/ID29gkqzJcz</latexit>p

kfkf
kVkVV = O( D)<latexit sha1_base64="yyFkYeMy0QdQovMaHZU6EazKJVU=">AAAB83icbVBNS8NAEJ34WetX1aOXxSJ4KomIehGKXjxWsGmhCWWz3bRLN5tldyOUtH/DiwdFvPpnvPlv3LY5aOuDgcd7M8zMiyRn2rjut7Oyura+sVnaKm/v7O7tVw4OfZ1mitAmSXmq2hHWlDNBm4YZTttSUZxEnLai4d3Ubz1RpVkqHs1I0jDBfcFiRrCxUhCM42DczX1/gm66lapbc2dAy8QrSBUKNLqVr6CXkiyhwhCOte54rjRhjpVhhNNJOcg0lZgMcZ92LBU4oTrMZzdP0KlVeihOlS1h0Ez9PZHjROtREtnOBJuBXvSm4n9eJzPxdZgzITNDBZkvijOOTIqmAaAeU5QYPrIEE8XsrYgMsMLE2JjKNgRv8eVl4p/XvMua93BRrd8WcZTgGE7gDDy4gjrcQwOaQEDCM7zCm5M5L8678zFvXXGKmSP4A+fzB6aWkW4=</latexit>

Simple Experiment<latexit sha1_base64="lwvQLUObBQQweaYcgkZRtovqGYo=">AAACc3icbVFNS+NAGJ5k3bV2P6wuePHgi90FPVgSEfWyIO5lwYuCVaGpYTJ5U4fOTMLMpFpC/sD+PG/+Cy9732ktaNUXhnl4nvf7TQrBjQ2CB8//sPDx02Jjqfn5y9dvy62V1QuTl5phl+Ui11cJNSi4wq7lVuBVoZHKROBlMvw90S9HqA3P1bkdF9iXdKB4xhm1jopbf7OtKMlFasbSfdVdvQ2/IEpwwFWVSGo1v6shi0OIIvftQoQqfRacqynldXVSw8ssozoeOoUPJJ3LfltfV+dTba4k7EASD7fjVjvoBFODtyCcgTaZ2Wncuo/SnJUSlWWCGtMLg8L2K6otZwLrZlQaLCgb0gH2HFRUoulX053V8NMxKWS5dk9ZmLIvIyoqzaRD5+mmvTGvtQn5ntYrbXbYr7gqSouKPRXKSgE2h8kBIOUamRVjByjT3PUK7IZqyqw7U9MtIXw98ltwsdsJ9zvh2V776Hi2jgZZJ5tki4TkgByRP+SUdAkjj96at+GB989f9zf9H0+uvjeL+U7mzN/5D2cOvxQ=</latexit>

f1<latexit sha1_base64="8S+6ED26Hh+H/Y9tWu0fCvnewmI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqseiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0EPa9frniVt05yCrxclKBHI1++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXUsljVD72fzUKTmzyoCEsbIlDZmrvycyGmk9iQLbGVEz0sveTPzP66YmvPYzLpPUoGSLRWEqiInJ7G8y4AqZERNLKFPc3krYiCrKjE2nZEPwll9eJa2LqlereveXlfpNHkcRTuAUzsGDK6jDHTSgCQyG8Ayv8OYI58V5dz4WrQUnnzmGP3A+fwDyLY2U</latexit>

K<latexit sha1_base64="mZPhmjadkOJRGBEeawj1GytO00M=">AAAB6XicbZDLSgMxFIbP1Fsdb1WXboJFcFVmXKgbsehGcFPFXqAdSibNtKGZZEgyQhn6Bm5cKOK2D+Pejfg2ppeFtv4Q+Pj/c8g5J0w408bzvp3c0vLK6lp+3d3Y3NreKezu1bRMFaFVIrlUjRBrypmgVcMMp41EURyHnNbD/vU4rz9SpZkUD2aQ0CDGXcEiRrCx1v0taheKXsmbCC2CP4Pi5Yd7kYy+3Eq78NnqSJLGVBjCsdZN30tMkGFlGOF06LZSTRNM+rhLmxYFjqkOssmkQ3RknQ6KpLJPGDRxf3dkONZ6EIe2Msamp+ezsflf1kxNdB5kTCSpoYJMP4pSjoxE47VRhylKDB9YwEQxOysiPawwMfY4rj2CP7/yItROSv5pyb/ziuUrmCoPB3AIx+DDGZThBipQBQIRPMELvDp959l5c96npTln1rMPf+SMfgBZQpA8</latexit>

f2<latexit sha1_base64="Ss9dWoQWiHsxIGMo6zrQu2XcX1I=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKqMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh7Bf65crbtWdg6wSLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjk01IvNTyhbEyHvGupohE3fjY/dUrOrDIgYaxtKSRz9fdERiNjJlFgOyOKI7PszcT/vG6K4bWfCZWkyBVbLApTSTAms7/JQGjOUE4soUwLeythI6opQ5tOyYbgLb+8Slq1qndZ9e4vKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDzsY2V</latexit>f1
f (x) =
=
f2

K
X

T
v k (wk x

bk )No Weight Decay —> No Sharing
3

f1<latexit sha1_base64="8S+6ED26Hh+H/Y9tWu0fCvnewmI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqseiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0EPa9frniVt05yCrxclKBHI1++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXUsljVD72fzUKTmzyoCEsbIlDZmrvycyGmk9iQLbGVEz0sveTPzP66YmvPYzLpPUoGSLRWEqiInJ7G8y4AqZERNLKFPc3krYiCrKjE2nZEPwll9eJa2LqlereveXlfpNHkcRTuAUzsGDK6jDHTSgCQyG8Ayv8OYI58V5dz4WrQUnnzmGP3A+fwDyLY2U</latexit>2

f2<latexit sha1_base64="Ss9dWoQWiHsxIGMo6zrQu2XcX1I=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKqMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh7Bf65crbtWdg6wSLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjk01IvNTyhbEyHvGupohE3fjY/dUrOrDIgYaxtKSRz9fdERiNjJlFgOyOKI7PszcT/vG6K4bWfCZWkyBVbLApTSTAms7/JQGjOUE4soUwLeythI6opQ5tOyYbgLb+8Slq1qndZ9e4vKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDzsY2V</latexit>

b

1

0

°1

°2

°3

0

º/2

º
µ
w = (cos µ, sin µ)

3º/2

2º

0

º/2

º
µ
w = (cos µ, sin µ)

3º/2

2º

Weight Decay —> Neuron Sharing3

f2<latexit sha1_base64="Ss9dWoQWiHsxIGMo6zrQu2XcX1I=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0mKqMeiF48V7Qe0oWy2m3bpZhN2J0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6xEnC/YgOlQgFo2ilh7Bf65crbtWdg6wSLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjk01IvNTyhbEyHvGupohE3fjY/dUrOrDIgYaxtKSRz9fdERiNjJlFgOyOKI7PszcT/vG6K4bWfCZWkyBVbLApTSTAms7/JQGjOUE4soUwLeythI6opQ5tOyYbgLb+8Slq1qndZ9e4vKvWbPI4inMApnIMHV1CHO2hAExgM4Rle4c2Rzovz7nwsWgtOPnMMf+B8/gDzsY2V</latexit>

f1<latexit sha1_base64="8S+6ED26Hh+H/Y9tWu0fCvnewmI=">AAAB6nicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqseiF48V7Qe0oWy2k3bpZhN2N0IJ/QlePCji1V/kzX/jts1BWx8MPN6bYWZekAiujet+O4W19Y3NreJ2aWd3b/+gfHjU0nGqGDZZLGLVCahGwSU2DTcCO4lCGgUC28H4dua3n1BpHstHM0nQj+hQ8pAzaqz0EPa9frniVt05yCrxclKBHI1++as3iFkaoTRMUK27npsYP6PKcCZwWuqlGhPKxnSIXUsljVD72fzUKTmzyoCEsbIlDZmrvycyGmk9iQLbGVEz0sveTPzP66YmvPYzLpPUoGSLRWEqiInJ7G8y4AqZERNLKFPc3krYiCrKjE2nZEPwll9eJa2LqlereveXlfpNHkcRTuAUzsGDK6jDHTSgCQyG8Ayv8OYI58V5dz4WrQUnnzmGP3A+fwDyLY2U</latexit>

2

b

1

0

°1

°2

°3

0

º/2

º
µ
w = (cos µ, sin µ)

3º/2

2º

0

º/2

º
µ
w = (cos µ, sin µ)

3º/2

2º

Tighter Bounds on Necessary Widthsof DNNsBound on Network Width• Suppose fθ is a DNN which solves the weight decay objective

Bound on Network Widthℝ<latexit sha1_base64="zJUj+usB2220Wz+E6U/MbqWpIS0=">AAACfHiclVFNb9QwEHVSPsry0QWOHLBYkLZCrBKEWi6VKrhwQovUbSutQ+Q4k421ThzZk1arKL+Cf8aNn8KlqpPNAVoujGTN03vzNOOZpFLSYhD88vydO3fv3d99MHr46PGTvfHTZ6dW10bAQmilzXnCLShZwgIlKjivDPAiUXCWrD93+tkFGCt1eYKbCqKCr0qZScHRUfH4B7uUKaBUKTRsnsv2iCnIcMmsXBW8x9O+JOfYsESr1G4Kl5rLto3X308oq3IZh8zIVY77lIlUo6X/5/46uLcpGsXjSTAL+qC3QTiACRliHo9/slSLuoASheLWLsOgwqjhBqVQ0I5YbaHiYs1XsHSw5AXYqOmX19I3jklppo17JdKe/dPR8MJ2U7vKgmNub2od+S9tWWP2MWpkWdUIpdg2ympFUdPuEjSVBgSqjQNcGOlmpSLnhgt09+qWEN788m1w+n4WHszCbx8mx5+GdeySF+QVmZKQHJJj8oXMyYII8tt76U29fe/Kf+2/9d9tS31v8Dwnf4V/cA3DMsTg</latexit>

e=

h ⇣

T
bk
w

1

⌘

d

···

ϕi

⇣

T
bk
w

ψi

N

⌘i

ℝ

D

Main Result
Assume fθ is a DNN which solves the weight decay objective,
for any homogenous layer with input features ϕi and output features ψi, such that

There exists another optimal representation of the form

Where,eK  rank( ) · rank( )<latexit sha1_base64="uAhSsLEWt2i/qSIkCTJyfvqNGDE=">AAACJXicbVA9SwNBEN3z2/gVtbRZDELShDsRtbAI2gg2EUwUciHs7U2SJXt75+6cGo78GRv/io2FIoKVf8XNR2HUBwOP92aYmRckUhh03U9nZnZufmFxaTm3srq2vpHf3KqbONUcajyWsb4JmAEpFNRQoISbRAOLAgnXQe9s6F/fgTYiVlfYT6AZsY4SbcEZWqmVP7mgvoRb6iM8YKaZ6g2KftWIEvV5GOO0fi9CQCFDyPxqVwxKrXzBLbsj0L/Em5ACmaDayr/5YczTCBRyyYxpeG6CzYxpFFzCIOenBhLGe6wDDUsVi8A0s9GXA7pnlZC2Y21LIR2pPycyFhnTjwLbGTHsmt/eUPzPa6TYPm5mQiUpguLjRe1UUozpMDIaCg0cZd8SxrWwt1LeZZpxtMHmbAje75f/kvp+2Tsse5cHhcrpJI4lskN2SZF45IhUyDmpkhrh5JE8k1fy5jw5L8678zFunXEmM9tkCs7XNzDNpbg=</latexit>

Questions?