How to add bias=True in lienar layer if I have a pretrained model with bias=False

username: Sourabh_Patil
cooked: <p>Hi,<br>
I have a pretrained model which has a linear layer. It had parameter bias=False while training. I want to make bias=True in the same model, as it has some dependencies while converting model to onnx. Is there any way to do it, so that performance stays the same.</p>

username: KFrank
cooked: <p>Hi Sourabh!</p>
<aside class="quote no-group" data-username="Sourabh_Patil" data-post="1" data-topic="189662" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sourabh_patil/48/61679_2.png" class="avatar"> Sourabh_Patil:</div>
<blockquote>
<p>I have a pretrained model which has a linear layer. It had parameter bias=False while training. I want to make bias=True in the same model</p>
</blockquote>
</aside>
<p>The simplest way to do this will be to replace the <code>Linear</code> in question with<br>
a new <code>Linear</code> with <code>bias = True</code> and then initialize the new <code>Linear</code>’s <code>weight</code><br>
(and <code>bias</code>) with the values from the old <code>Linear</code>.</p>
<p>You need to get your hands on the <code>Linear</code> in question somehow.  It will typically<br>
be a property of the model or some sub-model or an element of some collection.<br>
I will illustrate this with an example where it is a element of a <code>Sequential</code>.</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
2.1.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2023)
&gt;&gt;&gt;
&gt;&gt;&gt; seq = torch.nn.Sequential (torch.nn.Linear (3, 2, bias = False), torch.nn.Tanh(), torch.nn.Linear (5, 1))
&gt;&gt;&gt; seq[0]
Linear(in_features=3, out_features=2, bias=False)
&gt;&gt;&gt; seq[0].weight
Parameter containing:
tensor([[-0.0820,  0.2541,  0.5175],
        [-0.0235,  0.0478,  0.5665]], requires_grad=True)
&gt;&gt;&gt; oldLin = seq[0]
&gt;&gt;&gt; seq[0] = torch.nn.Linear (3, 2, bias = True)
&gt;&gt;&gt;
&gt;&gt;&gt; with torch.no_grad():
...     _ = seq[0].weight.copy_ (oldLin.weight)   # copy in pre-trained weight into new layer
...     _ = seq[0].bias.zero_()                   # zero out new bias for consistency with pre-trained model
...
&gt;&gt;&gt; seq
Sequential(
  (0): Linear(in_features=3, out_features=2, bias=True)
  (1): Tanh()
  (2): Linear(in_features=5, out_features=1, bias=True)
)
&gt;&gt;&gt; seq[0]
Linear(in_features=3, out_features=2, bias=True)
&gt;&gt;&gt; seq[0].weight
Parameter containing:
tensor([[-0.0820,  0.2541,  0.5175],
        [-0.0235,  0.0478,  0.5665]], requires_grad=True)
&gt;&gt;&gt; seq[0].bias
Parameter containing:
tensor([0., 0.], requires_grad=True)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: Sourabh_Patil
cooked: <p>Hi,</p>
<p>Thanks for the solution. I also tried out something similar which worked for me.</p>
<p>import copy<br>
####### model without bias i.e. bias=False<br>
model_with_bias = copy.deepcopy(model)</p>
<h6><a name="to-check-which-linear-layer-we-want-to-modify-1" class="anchor" href="#to-check-which-linear-layer-we-want-to-modify-1"></a>to check which linear layer we want to modify</h6>
<p>for name, param in model_with_bias.named_parameters():<br>
print(name)</p>
<p>layer_name = ‘linear’</p>
<p>####### get linear layer weights only<br>
weights = getattr(model_with_bias, layer_name).weight<br>
print(weights.shape)<br>
in_features = weights.shape[1]</p>
<p>####### create a new linear layer with bias = True<br>
new_linear_layer = torch.nn.Linear(in_features, in_features, bias=True)</p>
<p>######## clone the weights to new linear layer’s weights<br>
new_linear_layer.weight.data = weights.clone()</p>
<p>######## make all bias terms as zeros, as they have some initialization<br>
new_linear_layer.bias.data.fill_(0)</p>
<p>######## replace old linear layer with bias=False with new linear layer with bias=True<br>
setattr(model_with_bias, layer_name, new_linear_layer)</p>
<p>Regards,<br>
Sourabh</p>

username: KFrank
cooked: <p>Hi Sourabh!</p>
<aside class="quote no-group" data-username="Sourabh_Patil" data-post="3" data-topic="189662" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sourabh_patil/48/61679_2.png" class="avatar"> Sourabh_Patil:</div>
<blockquote>
<p>######## clone the weights to new linear layer’s weights<br>
new_linear_layer.weight.data = weights.clone()</p>
<p>######## make all bias terms as zeros, as they have some initialization<br>
new_linear_layer.bias.data.fill_(0)</p>
</blockquote>
</aside>
<p>A quick caution:  <code>.data</code> is deprecated and can lead to errors.  (I do believe that<br>
your usage of <code>.data</code> will work the way you want with the current version of<br>
pytorch, but why ask for trouble?)</p>
<p>The preferred way to do this is to mutate the tensors (i.e., use functions that<br>
modify the tensors in place) under the protection of a <code>with torch.no_grad():</code><br>
block (as illustrated in the example I posted).</p>
<p>Best.</p>
<p>K. Frank</p>

username: Sourabh_Patil
cooked: <p>Hi,</p>
<p>Thanks you so much for the feedback. Will keep your inputs noted.</p>
<p>Regards,<br>
Sourabh</p>