How to re-set alll parameters in a network

Sohrab_Salimian: <p>How to re-set the weights for the entire network, using the original pytorch weight initialization</p>

ptrblck: <p>You could create a <code>weight_reset</code> function similar to <code>weight_init</code> and reset the weigths:</p>
<pre><code class="lang-python">def weight_reset(m):
    if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
        m.reset_parameters()

model = = nn.Sequential(
    nn.Conv2d(3, 6, 3, 1, 1),
    nn.ReLU(),
    nn.Linear(20, 3)
)

model.apply(weight_reset)
</code></pre>
<p>Or alternatively just create a new instance of your model.</p>

Sohrab_Salimian: <p>thank you! Solved my issue</p>

Zuanazzi: <p>Is it possible to reset only part of the weights of the model? For instance, the weights of an specific layer, or even some random weights of one layer?</p>

I-Love-U: <p>You can use the <code>weight</code> method of your <code>Conv</code> or <code>Linear</code> to get these parameters…</p>

Brando_Miranda: <h2>This probably works too:</h2>
<p>This is not a robust solution and wont work for anything except core torch.nn layers, but this works:</p>
<pre><code class="lang-auto">for layer in model.children():
   if hasattr(layer, 'reset_parameters'):
       layer.reset_parameters()
</code></pre>
<hr>
<p>credit: <a href="https://discuss.pytorch.org/t/reinitializing-the-weights-after-each-cross-validation-fold/11034/2" class="inline-onebox">Reinitializing the weights after each cross validation fold</a></p>

Axel_Bregnsbo: <p>Solution by <a class="mention" href="/u/brando_miranda">@Brando_Miranda</a> did not work for me. I had to add an extra loop:</p>
<pre><code class="lang-auto">for layers in model.children():
    for layer in layers:
        if hasattr(layer, 'reset_parameters'):
            layer.reset_parameters()
</code></pre>

Alon: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I was looking for a more general way to reset <strong>all</strong> model parameters in case you don’t know the specific layers of the model. Would that do the trick?</p>
<pre><code class="lang-auto">def weight_reset(m):
    reset_parameters = getattr(m, "reset_parameters", None)
    if callable(reset_parameters):
        m.reset_parameters()

model.apply(weight_reset)
</code></pre>

ptrblck: <p>Your approach looks valid and I’m unsure if there is a better way.<br>
You could use e.g. <code>if isinstance(m, nn.Module)</code>, but this would also return <code>True</code> for custom modules, which don’t necessarily implement the <code>reset_parameters()</code> method.</p>

Brando_Miranda: <p>is your response to alon?</p>
<hr>
<p>btw related: <a href="https://stackoverflow.com/questions/63627997/reset-parameters-of-a-neural-network-in-pytorch" class="inline-onebox" rel="noopener nofollow ugc">python 3.x - Reset parameters of a neural network in pytorch - Stack Overflow</a></p>

Brando_Miranda: <p><a class="mention" href="/u/alon">@Alon</a> shouldn’t your code be recursive?</p>

Brando_Miranda: <p>Here is the code with an example that runs:</p>
<pre><code class="lang-auto">def lp_norm(mdl: nn.Module, p: int = 2) -&gt; Tensor:
    lp_norms = [w.norm(p) for name, w in mdl.named_parameters()]
    return sum(lp_norms)

def reset_all_weights(model: nn.Module) -&gt; None:
    """
    refs:
        - https://discuss.pytorch.org/t/how-to-re-set-alll-parameters-in-a-network/20819/6
        - https://stackoverflow.com/questions/63627997/reset-parameters-of-a-neural-network-in-pytorch
        - https://pytorch.org/docs/stable/generated/torch.nn.Module.html
    """

    @torch.no_grad()
    def weight_reset(m: nn.Module):
        # - check if the current module has reset_parameters &amp; if it's callabed called it on m
        reset_parameters = getattr(m, "reset_parameters", None)
        if callable(reset_parameters):
            m.reset_parameters()

    # Applies fn recursively to every submodule see: https://pytorch.org/docs/stable/generated/torch.nn.Module.html
    model.apply(fn=weight_reset)


def reset_all_linear_layer_weights(model: nn.Module) -&gt; nn.Module:
    """
    Resets all weights recursively for linear layers.

    ref:
        - https://pytorch.org/docs/stable/generated/torch.nn.Module.html
    """

    @torch.no_grad()
    def init_weights(m):
        if type(m) == nn.Linear:
            m.weight.fill_(1.0)

    # Applies fn recursively to every submodule see: https://pytorch.org/docs/stable/generated/torch.nn.Module.html
    model.apply(init_weights)


def reset_all_weights_with_specific_layer_type(model: nn.Module, modules_type2reset) -&gt; nn.Module:
    """
    Resets all weights recursively for linear layers.

    ref:
        - https://pytorch.org/docs/stable/generated/torch.nn.Module.html
    """

    @torch.no_grad()
    def init_weights(m):
        if type(m) == modules_type2reset:
            # if type(m) == torch.nn.BatchNorm2d:
            #     m.weight.fill_(1.0)
            m.reset_parameters()

    # Applies fn recursively to every submodule see: https://pytorch.org/docs/stable/generated/torch.nn.Module.html
    model.apply(init_weights)


# -- tests

def reset_params_test():
    import torchvision.models as models
    from uutils.torch_uu import lp_norm

    resnet18 = models.resnet18(pretrained=True)
    resnet18_random = models.resnet18(pretrained=False)

    print(f'{lp_norm(resnet18)=}')
    print(f'{lp_norm(resnet18_random)=}')
    print(f'{lp_norm(resnet18)=}')
    reset_all_weights(resnet18)
    print(f'{lp_norm(resnet18)=}')


if __name__ == '__main__':
    reset_params_test()
    print('Done! \a\n')
</code></pre>
<p>output:</p>
<pre><code class="lang-auto">lp_norm(resnet18)=tensor(517.5472, grad_fn=&lt;AddBackward0&gt;)
lp_norm(resnet18_random)=tensor(668.3687, grad_fn=&lt;AddBackward0&gt;)
lp_norm(resnet18)=tensor(517.5472, grad_fn=&lt;AddBackward0&gt;)
lp_norm(resnet18)=tensor(476.0836, grad_fn=&lt;AddBackward0&gt;)
Done!
</code></pre>
<p>I am assuming this works because I calculated the norm twice for the pre-trained net and it was the same both times before calling reset.</p>
<p>Though I was unhappy it wasn’t closer to the norm of the random net I must admit but I think this is good enough.</p>
<p>related: <a href="https://stackoverflow.com/a/69905671/1601580" class="inline-onebox" rel="noopener nofollow ugc">python 3.x - Reset parameters of a neural network in pytorch - Stack Overflow</a></p>

J_Johnson: <p>I find it better to just</p>
<pre><code class="lang-auto">del loss, model, optimizer
torch.cuda.empty_cache()
</code></pre>
<p>And then recreate the model and optimizer how you did in the first place.</p>
<p>Note that not deleting the loss resulted in cuda memory errors as it is connected to the model’s gradients graphs and thus keeps the whole thing in memory.</p>

Alon: <p>I think that model.apply() is already recursive.<br>
See: <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html" rel="noopener nofollow ugc">Module — PyTorch 1.11.0 documentation</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/52916e25c943f4e326a97c56a3fa228bd63044b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/52916e25c943f4e326a97c56a3fa228bd63044b3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52916e25c943f4e326a97c56a3fa228bd63044b3_2_690x93.png" alt="image" data-base62-sha1="bMqJBbCkrmKBasEjPyRALpj2r2b" width="690" height="93" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/52916e25c943f4e326a97c56a3fa228bd63044b3_2_690x93.png, https://discuss.pytorch.org/uploads/default/original/3X/5/2/52916e25c943f4e326a97c56a3fa228bd63044b3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/2/52916e25c943f4e326a97c56a3fa228bd63044b3.png 2x" data-dominant-color="F4F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">844×114 10.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

chic-hauret: <p>Why is the <code>_reset_parameters</code> method private (have an underscore in its name) for <code>torch.nn.MultiheadAttention</code>?</p>

ptrblck: <p>I’m unsure why some modules define it as a private method while others, such as <a href="https://github.com/pytorch/pytorch/blob/c757647dd8c198fbcca4d4e6c7cba6f1d12b97e6/torch/nn/modules/linear.py#L103"><code>nn.Linear</code></a>, do not. <a class="mention" href="/u/zhangguanheng66">@zhangguanheng66</a> might know more as <code>nn.MultiHeadAttention</code> was merged in <a href="https://github.com/pytorch/pytorch/pull/18334">this PR</a>.</p>