Model.train() and model.eval() vs model and model.eval()

zhidali: <p>Hi I am new to Pytorch.</p>
<p>Is model.train() the same as model for training?<br>
I find they are the same, am I correct?<br>
Thanks.</p>
<h1>RNN Model</h1>
<p>class RNN(nn.Module):<br>
def <strong>init</strong>(self, input_size, hidden_size, num_layers, num_classes):<br>
super(RNN, self).<strong>init</strong>()<br>
self.hidden_size = hidden_size<br>
…                       …      = dropout(…)</p>
<pre><code>def forward(self, x):
</code></pre>
<p>…<br>
…<br>
…<br>
rnn = RNN(input_size, hidden_size, num_layers, num_classes)</p>
<p>criterion = nn.CrossEntropyLoss()<br>
optimizer = torch.optim.Adagrad(…)</p>
<h1>train…</h1>
<p>…</p>
<h1>test</h1>
<p>model.eval()</p>

dys129: <p>Yes, they are the same. By default all the modules are initialized to train mode (self.training = True). Also be aware that some layers have different behavior during train/and evaluation (like BatchNorm, Dropout) so setting it matters.</p>
<p>Also as a rule of thumb for programming in general, try to explicitly state your intent and set model.train() and model.eval() when necessary.</p>

zhidali: <p>thanks for the help~:grinning:</p>

billtubbs: <p>Why do <code>model.train()</code> and <code>model.eval()</code> return a reference to the model.  What is the intended usage for the return value?</p>
<p>I am using as follows:</p>
<p><code>model.train()</code></p>
<p>But this means that in a Jupyter notebook it outputs the model object <strong>repr</strong> which is unwanted:</p>
<pre><code class="lang-auto">LeNet(
  (m): Sequential(
    (0): Sequential(
      (0): Conv2d(1, 20, kernel_size=(5, 5), stride=(1, 1))
      (1): BatchNorm2d(20, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU(inplace)
      (3): MaxPool2d(kernel_size=3, stride=3, padding=0, dilation=1, ceil_mode=False)
      (4): Dropout(p=0.25)
    )
    (1): Sequential(
      (0): Conv2d(20, 50, kernel_size=(5, 5), stride=(1, 1))
      (1): BatchNorm2d(50, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU(inplace)
      (3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
      (4): Dropout(p=0.25)
    )
    (2): View()
    (3): Linear(in_features=200, out_features=500, bias=True)
    (4): BatchNorm1d(500, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (5): ReLU(inplace)
    (6): Dropout(p=0.25)
    (7): Linear(in_features=500, out_features=10, bias=True)
  )
)
</code></pre>

justusschock: <p>to avoid this you could simply do</p>
<pre><code class="lang-auto">model = model.train()
</code></pre>
<p>And the same should work for eval(). Since the reference is assigned to a variable it should not be printed any more</p>

billtubbs: <p>Thanks, I can see how that solves the problem.  But I don’t understand why it was set up this way.  Usually if you are trying to change an attribute or a setting, the method would be called something like <code>set_attribute(value)</code>.  In this case, <code>set_mode("train")</code> maybe or <code>set_trainmode(True)</code>.  A method of this type would not be expected to return a value.  <code>model.train()</code> sounds like it is going to actually train the network.  Not very Pythonic!</p>

justusschock: <p>Maybe it was implemented like this to be consistent with methods like <code>.float()</code> or <code>.cuda()</code> but this is only a guess.</p>

SkonTeam: <p>It returns <code>self</code> so that you can chain different function such as: <code>model.train().cuda()</code> .<br>
It’s called a fluent interface.</p>

billtubbs: <p>Nice!  Thanks for explaining that and elucidating the intended usage.  B.t.w. presumably something like this is possible then: <code>model.eval().do_something().train()</code> (I haven’t tried it).</p>

GiganticRay: <p>i think that is just a programming thumb…, that we always return something</p>

bothmena: <p><code>model.eval().do_something().train()</code> will only work if <code>do_something()</code> return a reference to the model object.</p>
<p>And even if it works, I personally wouldn’t recommend it!<br>
I find it much more readible and clear to do it this way:</p>
<pre><code class="lang-auto">model.eval()
model.do_something()
model.train()
</code></pre>

Shubhankar: <p>You don’t need to do <code>model = model.train()</code><br>
It’s an internal method hence just</p>
<p><code>model.train()</code></p>

souserge: <p>I think, the intention of <code>model = model.train()</code> is to avoid seeing the lengthy output when run in a Jupyter Notebook. But for that, I’d suggest a small trick I recently learned – adding a semicolon (yes, in Python <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:">) at the end of the line. So <code>model.train();</code> will not produce any output <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Gali_Dekel: <p>i have 2 equivalent implementations of a conv network. in one of them the network.eval() works fine. and network(input) gives good predictions. in another it doesnt. what could be the reason??<br>
thanks</p>

asafbenj: <p>Interesting… I wonder why they didn’t mention this point or use <code>train()</code> and <code>eval()</code> in the beginner tutorials, seeing as it is so important. I’m just starting out with PyTorch by adapting the code from these tutorials and could have easily missed this point…</p>