How to initialize weights when using a manual Kfold and custom nn.module (init_weights how to?)

TellonUK1: <p>From the below URL’s I was able to piece together that if im using a manual K-fold (Not skorch) that I need to reinitialize the weights before each new fold, this is the suggestion on how to do that,</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/49433936/how-do-i-initialize-weights-in-pytorch">
  <header class="source">

      <a href="https://stackoverflow.com/questions/49433936/how-do-i-initialize-weights-in-pytorch" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/604734/f%c3%a1bio-perez" target="_blank" rel="noopener nofollow ugc">
    <img alt="F&amp;#225;bio Perez" src="https://www.gravatar.com/avatar/b7b889367f06fa0457298888c3402ee9?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/49433936/how-do-i-initialize-weights-in-pytorch" target="_blank" rel="noopener nofollow ugc">How do I initialize weights in PyTorch?</a>
</h4>

<div class="tags">
  <strong>python, machine-learning, deep-learning, neural-network</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/604734/f%c3%a1bio-perez" target="_blank" rel="noopener nofollow ugc">
    F&amp;#225;bio Perez
  </a>
  on <a href="https://stackoverflow.com/questions/49433936/how-do-i-initialize-weights-in-pytorch" target="_blank" rel="noopener nofollow ugc">04:34PM - 22 Mar 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="quote" data-post="1" data-topic="15035">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neo/48/3968_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-is-the-best-way-to-apply-k-fold-cross-validation-in-cnn/15035">What is the best way to apply k-fold cross validation in CNN?</a> 
  </div>
  <blockquote>
    Hello, 
How can I apply k-fold cross validation with CNN. I do not want to make it manually; for example, in leave one out, I might remove one item from the training set and train the network then apply testing with the removed item. Could you please help me to make this in a standard way. 
Regards,
  </blockquote>
</aside>

<p>I must be missing something though,</p>
<pre><code>class Net(nn.Module):

    def init_weights(self):
        if (type(self) == nn.Linear):
            nn.init.xavier_uniform(self.weight)
            self.bias.data.fill_(0.01)

    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(4, hl)
        self.fc2 = nn.Linear(hl, 3)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

net = Net()
net.apply(init_weights) # reset network weights
</code></pre>
<p>NameError: name ‘init_weights’ is not defined</p>

ptrblck: <p>Define <code>init_weigths</code> outside of your model and pass <code>m</code> instead of <code>self</code>:</p>
<pre><code class="lang-python">def weights_init(m):
    if isinstance(m, nn.Conv2d):
        torch.nn.init.xavier_uniform(m.weight.data)
        torch.nn.init.xavier_uniform(m.bias.data)

net.apply(weights_init)
</code></pre>

josselineperdomo: <p>Suppose my model is a more complex architecture (for example an U-net), what is the best way to restart their data? I am trying to use k folds validation and my model is an U-net but I don’t know how to restart their weights.</p>
<p>I really appreciate your help <a class="mention" href="/u/ptrblck">@ptrblck</a> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks</p>

ptrblck: <p>It depends a bit on your use case.</p>
<p>If you haven’t written a custom <code>weights_init</code> method, but just initialize the model and thus use the default random initializations, I would recommend to just recreate the model.<br>
Note that you should also recreate the optimizer in this case.</p>
<p>On the other hand, if you already defined a custom <code>weights_init</code> method, just reset the model via <code>model.apply(weights_init)</code>.</p>
<p>Also, not sure if this fits your use case, but you could initialize the model once, create a <code>copy.deepcopy</code> of its <code>state_dict</code>, and reload this <code>state_dict</code> for each fold via <code>model.load_state_dict(state_dict)</code>.</p>
<p>Let me know, if one of these approaches would work. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

josselineperdomo: <p>My model use the defaults random initialization, but I think is a best practice restart the weights of my current model instance instead of create a new one. I didn’t understand what do you mean with recreate the model, could you please give me more details? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Let me give you more details, I am trying to fit a UNet model with a dataset of around 1000 images. I thought to apply K folds cross validation due to the size of the dataset, but after I tested my code I noticed I didn’t restart model’s weights when a new folds configuration starts, it is the right way to handle the model in this case?</p>
<p>Thanks for your reply <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <aside class="quote no-group" data-username="josselineperdomo" data-post="5" data-topic="20596">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/eb8c5e/48.png" class="avatar"> josselineperdomo:</div>
<blockquote>
<p>I didn’t understand what do you mean with recreate the model, could you please give me more details</p>
</blockquote>
</aside>
<p>I just meant to use a new instance as shown here:</p>
<pre data-code-wrap="python"><code class="lang-python">model = MyModel()
# perform training on first fold

# recreate model
model = MyModel()
# perform training on second fold ...
</code></pre>
<p>You could of course recreate these instances in a loop, if that’s more convenient.</p>
<aside class="quote no-group" data-username="josselineperdomo" data-post="5" data-topic="20596">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/eb8c5e/48.png" class="avatar"> josselineperdomo:</div>
<blockquote>
<p>but after I tested my code I noticed I didn’t restart model’s weights when a new folds configuration starts, it is the right way to handle the model in this case?</p>
</blockquote>
</aside>
<p>Yes, you should reinitialize the model randomly after it was trained on a specific fold.<br>
If you don’t want to recreate the model instance, you could call the <code>reset_parameters()</code> method on each submodule:</p>
<pre data-code-wrap="python"><code class="lang-python">for name, module in model.named_modules():
    if hasattr(module, 'reset_parameters'):
        print('Resetting ', name)
        module.reset_parameters()
</code></pre>
<p>Note that I would still recommend to recreate the optimizer, as it might store running estimates (e.g. if you are using <code>Adam</code>).</p>

josselineperdomo: <p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"> what would happen if I handle my train and evaluation process through Ignite? Should I define the model, optimizer, trainer, and evaluator per fold?</p>

ptrblck: <p>I’m not sure, how this workflow would look in Ignite, but <a class="mention" href="/u/vfdev-5">@vfdev-5</a> certainly knows! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

vfdev-5: <p>Personnally I would do this via a single script file or method that has as input fold index. In this case, there will be no problems with restarting model, optimizer, lr_scheduler and other state-full object (e.g. amp if nvidia/apex is used).<br>
So, a basic code will be something like that:</p>
<pre><code class="lang-python">from ignite.engine import create_supervised_trainer, create_supervised_evaluator, Events
from ignite.metrics import Accuracy


def train_on_single_fold(*args, fold_index=0, **kwargs):

    train_loader, val_loader = get_dataflow(fold_index=fold_index, **kwargs)
    model, optimizer, criterion, lr_scheduler = initialize_model(**kwargs)
    metrics = {
         "accuracy": Accuracy(),
    }

    trainer = create_supervised_trainer(model, optimizer, criterion)

    @trainer.on(Events.EPOCH_COMPLETED)
    def update_lr_scheduler(_):
        lr_scheduler.step()

    evaluator = create_supervised_evaluator(model, metrics=metrics)

    @trainer.on(Events.ITERATION_COMPLETED(every=validate_every))
    def validate(trainer):
        evaluator.run(val_loader)
        metrics = evaluator.state.metrics
        print("After {} iterations, binary accuracy = {:.2f}"
              .format(trainer.state.iteration, metrics['accuracy']))

    trainer.run(train_loader, max_epochs=kwargs.get(max_epochs, 100))
</code></pre>
<p>Hope this helps. Please, feel free to ask more about ignite usage if needed (and do not forget to set the category “ignite” otherwise our team won’t get the notification).</p>
<p>PS: <a class="mention" href="/u/ptrblck">@ptrblck</a> thanks a lot for mentioning !</p>