How to switch model from training to evaluation?

username: Dudly01
cooked: <p>Hello! I am trying to set up a training script using DistributedDataParallel (DDP) where the model changes between training and evaluation modes. However, when I try to switch into evaluation mode with <code>model=model.eval()</code> <code>model</code> becomes a <code>NoneType</code>. I also tried to use <code>model=model.train(False)</code> but the result was the same.</p>
<p>My issue is reproduceable with modifying the <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="nofollow noopener">DDP example</a>, thus:</p>
<pre><code class="lang-auto">import os
import torch
import torch.distributed as dist
import torch.nn as nn
import torch.optim as optim
import torch.multiprocessing as mp

from torch.nn.parallel import DistributedDataParallel as DDP


def setup(rank, world_size):
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # initialize the process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)


def cleanup():
    dist.destroy_process_group()


class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = nn.Linear(10, 10)
        self.drop1 = nn.Dropout(p=0.6)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(10, 5)

    def forward(self, x):
        return self.net2(self.relu(self.drop1(self.net1(x))))


def demo_basic(rank, world_size):
    print(f"Running basic DDP example on rank {rank}.")
    setup(rank, world_size)

    # create model and move it to GPU with id rank
    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    # Training mode
    print("Training")
    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()
    optimizer.step()

    # Evaluation mode
    print("Evaluating")
    ddp_model = ddp_model.eval()
    outputs = ddp_model(torch.randn(20, 10))

    cleanup()


def run_demo(demo_fn, world_size):
    mp.spawn(demo_fn,
             args=(world_size,),
             nprocs=world_size,
             join=True)


if __name__ == "__main__":
    run_demo(demo_basic, 1)

</code></pre>
<p>What is the proper way of switching between modes DDP? (Or it is not intended to be switched?)</p>
<p>Thank you in advance!</p>

username: Dudly01
cooked: <p>I am pinging this as somebody might have more insight into it than me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: iffiX
cooked: <p>Hello, this issue hasn’t been fixed in 1.5.0, but has been fixed in 1.5.1:</p>
<p>v1.5.0:</p>
<pre><code class="lang-auto">    def train(self, mode=True):
        super(DistributedDataParallel, self).train(mode)
        for module in self._module_copies[1:]:
            module.train(mode)
</code></pre>
<p>is not returning <code>self</code></p>
<p>v1.5.1</p>
<pre><code class="lang-auto">    def train(self, mode=True):
        self.training = mode
        for module in self.children():
            module.train(mode)
        return self
</code></pre>
<p>is returning self.</p>

username: Dudly01
cooked: <p>Thank you for your answer.</p>
<p>Strangely enough I am using the version 1.5.1 and the line returning <code>self</code> is present in the <code>train()</code> function. I even tried to reinstall 1.5.1 after cleaning conda cache. Then I created a a new conda environment and installed pytorch with python 3.8 (as I originally was using 3.7). However, the problem was still there.</p>
<p>The only thing I did not try was to insall the nightly-builds, as I could not download it within 7 minutes and lost patience. <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<p>However, if the intended way of switching is not different from the non DistributedDataParallel case then I am glad. I was just starting out with DistributedDataParallel and was not sure whether its possible to switch modes, or one has to define the mode before using the wrapper or some other magic.</p>

username: mrshenli
cooked: <p>Looks like we still miss that return at least in master. I am not sure whether some earlier changes were applied but got revert or not. Adding it in <a href="https://github.com/pytorch/pytorch/pull/42131" class="inline-onebox" rel="noopener nofollow ugc">Let DDP.train() return self to stay consistent with nn.Module by mrshenli · Pull Request #42131 · pytorch/pytorch · GitHub</a></p>
<blockquote>
<p>I was just starting out with DistributedDataParallel and was not sure whether it’s possible to switch modes, or one has to define the mode before using the wrapper or some other magic.</p>
</blockquote>
<p>DDP’s <code>train()</code> and <code>eval()</code> should work as expected. Just please remember to wrap it with <code>torch.no_grad()</code> when running in eval mode.</p>

username: Dudly01
cooked: <p>I just tried it out with 1.6.0 but it seems your commit did not make it into it (or the issue is elsewhere <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> )</p>
<p>On the other hand, thank you very much for mentioning <code>torch.no_grad()</code>! It was a feature I was not aware of yet, and helped me out tremendously.</p>

username: Dudly01
cooked: <p>Lastly, thank you <a class="mention" href="/u/iffix">@iffiX</a> and <a class="mention" href="/u/mrshenli">@mrshenli</a> for taking your time to answer.<br>
Both of you were a big help!</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="Dudly01" data-post="6" data-topic="90530">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/eada6e/48.png" class="avatar"> Dudly01:</div>
<blockquote>
<p>I just tried it out with 1.6.0 but it seems your commit did not make it into it (or the issue is elsewhere <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> )</p>
</blockquote>
</aside>
<p>It will be included in v1.7. The branch cut date for v1.6 was a few weeks ago.</p>

username: Dudly01
cooked: <p>In the meantime I also realized that as my only intention is to switch my model to evaluation mode, I can also accomplish it with <code>model.eval()</code> and there is no real need for using <code>model = model.eval()</code>.</p>
<p>I leave this here for future reference aiding ppl like me. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

username: ritwik.m07
cooked: <p>In addition to what <a class="mention" href="/u/mrshenli">@mrshenli</a> said</p>
<p>use this</p>
<pre><code class="lang-auto">with torch.no_grad():
    self.model.module.eval()
    out = self.model(input)
</code></pre>
<p>instead of this</p>
<pre><code class="lang-auto">with torch.no_grad():
    self.model.eval()
    out = self.model(input)
</code></pre>