Pytorch: Neural Network for classification - Constrain some weights to be chosen from a finite set

KenZhenLin: <p>When training Neural Network for classification in Pytorch, is it possible to put constraints on the weights in the output layer such that they are chosen from a specific finite feasible set?</p>
<p>For example, let’s say W is the weight in the output layer, is it possible to put constraints on W such that the optimal W is selected from the set S={W_1, W_2, …, W_n}, where each W_i is a given feasible value for W? i.e. I will give the values of the W_1,…,W_n to the model</p>
<p>If this is not possible in Pytorch, is there any other ways to achieve this?</p>
<p>Thanks!</p>

KFrank: <p>Hi Zhen!</p>
<aside class="quote no-group" data-username="KenZhenLin" data-post="1" data-topic="107281" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kenzhenlin/48/32790_2.png" class="avatar"> KenZhenLin:</div>
<blockquote>
<p>let’s say W is the weight in the output layer, is it possible to put constraints on W such that the optimal W is selected from the set S={W_1, W_2, …, W_n}, where each W_i is a given feasible value for W?</p>
</blockquote>
</aside>
<p>This is not a particularly natural thing to do in the context of neural<br>
networks and gradient-descent optimization (and I am not aware of<br>
anything built in to pytorch that will do this).</p>
<p>A general note outside of the context of neural networks and gradient<br>
descent:  The kind of constraints you contemplate lead to a so-called<br>
<em>integer-programming</em> problem – a known “hard” problem.</p>
<p>Back to pytorch:  Let’s say your feasible weights are the integers 1<br>
through 10, <code>S = {1, 2, ..., 10}</code>.  Suppose – based on the learning<br>
rate and the backpropagated gradient – your <code>opt.step()</code> wants to<br>
set <code>W = 1.23</code>.  What do you do?  You could round <code>W</code> to <code>1</code>, but that is<br>
not necessarily satisfactory.</p>
<p>Suppose <code>W</code> happens to be <code>3</code> at a certain stage of training, but you<br>
and I magically know that it should end up being <code>5</code>.  But let’s say that<br>
the learning rate is small so that your optimizer is only taking small<br>
steps, say no larger than 0.01.  How is <code>W</code> suppose to get from <code>3</code> to <code>5</code>,<br>
when it can’t take a step of <code>2</code> or even <code>1</code>.</p>
<p>The best I can imagine is to add a penalty to your loss function that is<br>
zero when <code>W = W_i</code>, and greater than zero otherwise.  For example:</p>
<pre data-code-wrap="python"><code class="lang-python">penalty = ((W - W_1) * (W - W_2) * ... * (W - W_n))**2
loss_with_penalty = regular_loss + alpha * penalty
opt.zero_grad()
loss_with_penalty.backward()
opt.step()
</code></pre>
<p>Here <code>alpha</code> is an adjustable parameter that you have start out at<br>
zero, and then have increase as you train.  When you make <code>alpha</code><br>
very large, you will force <code>W</code> to be close to one of the <code>W_i</code>, but  while<br>
<code>alpha</code> is still zero or small, the training is not prevented from moving<br>
from one <code>W_i</code> to another.  But you will have to experiment in order to<br>
determine how to schedule how <code>alpha</code> increases throughout training.</p>
<p>And at the end, when <code>W</code> is close to one of the <code>W_i</code>, you can always<br>
round it to the nearest <code>W_i</code> if it has to be equal to one of the specified<br>
values.</p>
<p>Good luck.</p>
<p>K. Frank</p>

KenZhenLin: <p>Thank you so much! The penalty approach makes a lot of sense!</p>
<p>I figure out that what I want on the weights <code>W</code> can be stated in the following way:</p>
<p>Suppose the weight or kernel matrix is a 3 by 4 matrix <code>W,</code> and its elements are <code>W_ij</code></p>
<p>I would like <strong>for each column j, there is one and only one nonzero <code>W_ij</code>, and <code>W_ij = 1</code>.</strong></p>
<p>What would be a good way to implement this requirement?</p>
<p>One possible solution I can think of is to put the following constraints using the penalty approach you mentioned:</p>
<p><code>W_1j + W_2j + W_3j = 1 for all j = 1,2,3,4</code></p>
<p>and</p>
<p><code>W_ij * (1-W_ij) = 0, for all i, j</code></p>
<p>But is it possible to implement these constraints directly? Or is there any better way to require the property (in bold) that I want?</p>
<p>If this is not possible in Pytorch, can TensorFlow do it?</p>

KFrank: <p>Hi Zhen!</p>
<aside class="quote no-group" data-username="KenZhenLin" data-post="3" data-topic="107281" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kenzhenlin/48/32790_2.png" class="avatar"> KenZhenLin:</div>
<blockquote>
<p>Suppose the weight or kernel matrix is a 3 by 4 matrix <code>W,</code> and its elements are <code>W_ij</code></p>
<p>I would like <strong>for each column j, there is one and only one nonzero <code>W_ij</code>, and <code>W_ij = 1</code>.</strong></p>
</blockquote>
</aside>
<p>Again, this is an integer-programming problem – not a natural fit for<br>
gradient descent.</p>
<p>But why do you want to do this?  Suppose you had weights that didn’t<br>
have this constrained form, but gave a better-performing network, e.g.,<br>
more accurate predictions.  Why would you not prefer those?</p>
<aside class="quote no-group">
<blockquote>
<p>One possible solution I can think of is to put the following constraints using the penalty approach you mentioned:</p>
<p><code>W_1j + W_2j + W_3j = 1 for all j = 1,2,3,4</code></p>
<p>and</p>
<p><code>W_ij * (1-W_ij) = 0, for all i, j</code></p>
</blockquote>
</aside>
<p>I haven’t thought about it carefully, but these constraints do look like<br>
they are consistent with what you described above, and can be turned<br>
into penalties.</p>
<aside class="quote no-group">
<blockquote>
<p>If this is not possible in Pytorch, can TensorFlow do it?</p>
</blockquote>
</aside>
<p>I know less about tensorflow, but I’m not aware of anything built into<br>
it that would handle such constraints for you.  Tensorflow is also a<br>
neural-network framework that is based on backpropagation and<br>
gradient descent, so it is also not the natural tool for this part of your<br>
use case.</p>
<p>A comment:  If your weight matrix, <code>W</code>, really is as described above, you<br>
have 81 (<code>3**4</code>) choices for <code>W</code>.  You could freeze <code>W</code> to be each one of<br>
those choices in succession, and train the rest of your network using<br>
gradient descent while holding the specific choice of <code>W</code> constant.  You<br>
will now have 81 trained networks.  Use the network, with its specific<br>
<code>W</code>, that performs the best.  This is, of course, the “exhaustive search”<br>
algorithm for the integer-programming problem, but with only 81 points<br>
in your search space, it is perfectly doable.  (But it will scale horribly<br>
should <code>W</code> grow in size beyond 3 by 4.)</p>
<p>Good luck.</p>
<p>K. Frank</p>

KenZhenLin: <p>Thanks again for the nice comments!</p>
<p>May I ask one more quick question? How do I freeze only a subset of the variables during the training process?</p>
<p>For example, if I have a layer with weight matrix <code>W</code> and bias vector <code>b</code>, how do I only freeze <code>b</code> and still update the <code>W</code> during the training process?</p>