Formulation for a custom regularizer to minimize amount of space taken by weights

username: Ge0rges
cooked: <p>Hello all.</p>
<p>Below is a screenshot of a custom regularizer I have implemented.</p>
<p>The goal of this regularizer is to minimize <em>the total number of space taken by the weights</em> rather than affect the value of any one weight. However by the nature of a regularizer, I require a formulation for my regularizer that is weight dependent, since otherwise backdrop won’t take it into account.</p>
<p>Essentially, if there are too many weights, it should push all weights to zero equally, and if there are too few weights, push the value of the weights to go up.</p>
<p>My current formulation works but is not weight dependent. I have the function that would push towards the points intended given the number of bytes use and number of bytes available, but as said, it’s weight independent.</p>
<p>I appreciate any input.</p>
<p>PS: While I use numpy operations here, I am aware I need to use torch operations on tensors so as to remain in the computation graph. The problem is that resources_used remains a scalar, and the computations remain weight independent.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2adaadc07528896633287d373c65c4afac6e8cfb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2adaadc07528896633287d373c65c4afac6e8cfb" title="IMG_4424"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2adaadc07528896633287d373c65c4afac6e8cfb_2_690x228.jpeg" alt="IMG_4424" data-base62-sha1="676znXZUXYwDXKS5tyK0XHTLNIT" width="690" height="228" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2adaadc07528896633287d373c65c4afac6e8cfb_2_690x228.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2adaadc07528896633287d373c65c4afac6e8cfb_2_1035x342.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2adaadc07528896633287d373c65c4afac6e8cfb_2_1380x456.jpeg 2x" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_4424</span><span class="informations">2435×807 169 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: albanD
cooked: <p>Hi,</p>
<p>That penalty that you compute does not depend on the value of the parameters right? So it is expected that no gradients wrt these values can be computed.</p>

username: Ge0rges
cooked: <p>Exactly, so I’m looking for a formulation that does the same but is weight dependent.</p>

username: albanD
cooked: <p>Ho sorry I missed one paragraph in your question <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Well anything related to the number of entries won’t work as it won’t be differentiable.<br>
You can renormalize each value to get their number because that would give you a gradient of 0.<br>
I guess you could try to center the parameter and use the standard deviation? You measure how much each tensor varies from a mean. In theory, you could represent them as a mean value and their offsets. So the closer they are, the better?</p>

username: googlebot
cooked: <p>Perhaps L1 penalty (see <a href="https://discuss.pytorch.org/t/how-to-create-a-sparse-autoencoder-neural-network-with-pytorch/3703/6">here</a>) proportional to what you compute would work?</p>

username: Ge0rges
cooked: <p>Thank you for both answers have given me some ideas to think about.</p>