How to assign fixed two weights to neural network for discrete selection between those during optimization

Waqas_Ahmed: <p>I would like to assign the fixed random element to weight for each connection in neural network. lets say there are only two values available for weight to select from instead to decreasing it constantly. how can i assign those two values ?? any idea?? <a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>You can directly assign any tensor to the trainable parameters by wrapping the assignment into a <code>no_grad()</code> guard as seen here:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10)
with torch.no_grad():
    lin.weight.copy_(torch.ones(10, 10))
    lin.bias.copy_(torch.ones(10))
</code></pre>
<p>Note that I’ve used <code>torch.ones</code> as a simple example, but you can use your “fixed” values.</p>
<p>I’m not sure if I understand the general use case correctly, but I assume you don’t want to “train” the model and allowing it to pick only your fixed values as I wouldn’t know how this should work.</p>

Waqas_Ahmed: <p>Thanks for your response. Let me explain you in better way. I am using 2-layered convolutional neural network to classify Cifar-10. Algorithm needs to assign K number of fixed values to weight per connection. these values are sampled through glorot/xavier uniform distribution. the shape of my first layer looks like [6,3,5,5].  so, we have 450 weights for this layer only. but since we need to assign K number of fixed values per weight (connection) then we can have 450*K weights in total for the first layer. Now each weight is associated with Score function which is updated through backpropagation and only these scores are updated not weights directly. when scores are updated then looking at the scores associated with each weight, we select one weight per connection and thus it is manually assigned to the network, and we test the network using test samples. in summary, we select weights for each connection from the fixed set of numbers only instead to reducing it continuously.  for your reference i am attaching the research whose code i am trying to implement. it would be great if you can give me any idea how i can achieve the desired result. <a href="https://arxiv.org/abs/2101.06475" class="inline-onebox" rel="noopener nofollow ugc">[2101.06475] Slot Machines: Discovering Winning Combinations of Random Weights in Neural Networks</a><br>
Thanks. i would be really greatly if i can get any hint.</p>

ptrblck: <aside class="quote no-group" data-username="Waqas_Ahmed" data-post="3" data-topic="167775">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/waqas_ahmed/48/55483_2.png" class="avatar"> Waqas_Ahmed:</div>
<blockquote>
<p>when scores are updated then looking at the scores associated with each weight, we select one weight per connection and thus it is manually assigned to the network, and we test the network using test samples.</p>
</blockquote>
</aside>
<p>I would assume that my example code snippet should work for the actual assignment.<br>
I’m unsure how the scoring and selection works, but I assume your general workflow might be:</p>
<ul>
<li>create the scores</li>
<li>select the corresponding “discrete” values based on the scores</li>
<li>create the tensor which should be assigned to the parameter</li>
<li>assign the tensor</li>
<li>test the new model with your validation data</li>
</ul>

Waqas_Ahmed: <p>during training process (forward and backward pass), how can i assign two values to a single weight for optimization process? since each connection have two choices for weight hence while forward and backward pass these values should be placed one by one in place of weight. so, if we have 100 parameters then (100*100) possible weight combination should be tested during training(weight optimization) any idea???</p>

ptrblck: <aside class="quote no-group" data-username="Waqas_Ahmed" data-post="5" data-topic="167775">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/waqas_ahmed/48/55483_2.png" class="avatar"> Waqas_Ahmed:</div>
<blockquote>
<p>how can i assign two values to a single weight for optimization process?</p>
</blockquote>
</aside>
<p>I’m not sure I understand the question correctly, but you cannot assign multiple values to a single scalar.<br>
My previous code snippet shows how to assign a tensor to a parameter.</p>

Waqas_Ahmed: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/9623ad5c02bd2ebd2be7fd6c28619a6330faf62c.png" data-download-href="https://discuss.pytorch.org/uploads/default/9623ad5c02bd2ebd2be7fd6c28619a6330faf62c" title="Capture"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/9623ad5c02bd2ebd2be7fd6c28619a6330faf62c_2_624x499.png" alt="Capture" data-base62-sha1="lqc3DcflbrySxZ23WDa3KPGerAU" width="624" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/9623ad5c02bd2ebd2be7fd6c28619a6330faf62c_2_624x499.png, https://discuss.pytorch.org/uploads/default/original/3X/9/6/9623ad5c02bd2ebd2be7fd6c28619a6330faf62c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/6/9623ad5c02bd2ebd2be7fd6c28619a6330faf62c.png 2x" data-dominant-color="F0F3F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture</span><span class="informations">663×531 56.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I need to optimize fully connected 2 layered neural network which has only two choices per weight which are manually assigned as shown in the picture. how can i optimize this sort of network? i hope this picture made it clear to you. during forward pass i would like to give both the values one by one (eg, w1  and w2) to optimize network and in the end i have to select one. my question is  how and where should i declare these values so that it can be considered during optimization process. do i need to make my custom optimizer? or create any sort of loop?</p>

ptrblck: <aside class="quote no-group" data-username="Waqas_Ahmed" data-post="7" data-topic="167775">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/waqas_ahmed/48/55483_2.png" class="avatar"> Waqas_Ahmed:</div>
<blockquote>
<p>my question is how and where should i declare these values so that it can be considered during optimization process. do i need to make my custom optimizer? or create any sort of loop?</p>
</blockquote>
</aside>
<p>Yes, the “optimization” would need a custom logic, since the default optimizers in PyTorch update parameters based on gradients, while your approach needs a “selection” algorithm, which you would need to implement manually (at least I’m not aware of any library or automatic way to do this kind of optimization for you).</p>