Model .eval() predicts all the same (or nearly same) outputs

KhanMar: <p>Hi!<br>
I’m training the changed DETR transformer model on the custom dataset.<br>
During .train() mode the model is doing normal predictions (all different), but if I run .eval() mode for evaluation - the outputs of the model are all same (or almost same).<br>
What can be the problem?<br>
The LR is 1e-5 and the out layer is linear.<br>
Will appreciate any advice!</p>

KhanMar: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I watched many posts you were helping people with the similar problem - can you have a look in my case as well? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>During the evaluation (and after calling <code>model.eval()</code>) some layers change their behavior, e.g. batchnorm layers are using the running stats to normalize the input activations and dropout layers are disabled.<br>
If the accuracy of the model decreases significantly during the eval phase, noisy batchnorm stats could be the cause and you could either try to increase the batch size during training or change the momentum of the batchnorm layers to smooth the updates.</p>

KhanMar: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your answer. I’m actually using nn.LayerNorm instead of nn.BatchNorm layers as in original DETR code. So I guess the batch size for me doesn’t matter and also  I can’t change momentum for those layers. Am I correct?<br>
My dataset is about 1000 samples, the batch size now is set to 4 and I’m training on 3 GPUs at a time.<br>
Do you have any other suggestions?</p>

ptrblck: <p>Ah OK, that’s interesting as these layers shouldn’t have any running stats registered as buffers.<br>
In any case, you should check which layers could change their behavior between training and validation runs. In case you are using custom layers, check if they are using their internal <code>self.training</code> attribute to change the forward pass.</p>

KhanMar: <p>There are used custom TransformerEncoder and TransformerDecoder layers, but I can’t find any self.training parameters passed in them. Let me post here the structure:</p>
<p>class TransformerEncoderLayer(nn.Module):</p>
<pre><code>def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1,
             activation="relu", normalize_before=False):
    super().__init__()
    self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout)
    # Implementation of Feedforward model
    self.linear1 = nn.Linear(d_model, dim_feedforward)
    self.dropout = nn.Dropout(dropout)
    self.linear2 = nn.Linear(dim_feedforward, d_model)

    self.norm1 = nn.LayerNorm(d_model)
    self.norm2 = nn.LayerNorm(d_model)
    self.dropout1 = nn.Dropout(dropout)
    self.dropout2 = nn.Dropout(dropout)

    self.activation = _get_activation_fn(activation)
    self.normalize_before = normalize_before
</code></pre>
<p>class TransformerDecoderLayer(nn.Module):</p>
<pre><code>def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1,
             activation="relu", normalize_before=False):
    super().__init__()
    self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout)
    self.multihead_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout)
    # Implementation of Feedforward model
    self.linear1 = nn.Linear(d_model, dim_feedforward)
    self.dropout = nn.Dropout(dropout)
    self.linear2 = nn.Linear(dim_feedforward, d_model)

    self.norm1 = nn.LayerNorm(d_model)
    self.norm2 = nn.LayerNorm(d_model)
    self.norm3 = nn.LayerNorm(d_model)
    self.dropout1 = nn.Dropout(dropout)
    self.dropout2 = nn.Dropout(dropout)
    self.dropout3 = nn.Dropout(dropout)

    self.activation = _get_activation_fn(activation)
    self.normalize_before = normalize_before
</code></pre>
<p>So I have no idea of what could cause the differences between Training and Evaluation parts. Can it be because of the Dropout usage? The dropout value is set to 0.1 now.</p>

ptrblck: <p>Yes, <code>dropout</code> will be disabled during evaluation as previously mentioned and would thus change the output. I don’t know if it could be responsible for your issue, but if no other layers change their behavior I would guess so. Could you disable dropout also during training and see if now the training output would also be static?</p>

KhanMar: <p>I did some experiments as you suggested!</p>
<ol>
<li>
<p>TRAINING mode with default dropout = 0.1:<br>
Transformer output:<br>
tensor([[[-1.2072, -0.0247,  1.4827,  …,  1.9502, -1.0014, -0.0777],<br>
[-1.0472,  1.0473, -0.1111,  …,  1.9880, -0.7706, -0.9534],<br>
[-0.9117,  0.6312,  1.5603,  …,  1.7538, -1.7898, -0.0764],<br>
…,<br>
[-1.5139,  0.8651,  1.3959,  …,  1.4869, -0.7368, -0.9495],<br>
[-1.3713,  0.1539,  1.1079,  …,  1.7069,  1.0330, -0.5884],<br>
[-0.9833,  0.9340,  1.4092,  …,  2.1241, -0.9530, -0.0743]]],<br>
device=‘cuda:0’, grad_fn=)<br>
And final output of the model:<br>
{‘pred_logits’: tensor([[-0.0081,  0.2497, -0.3607,  …, -0.0395, -0.2595, -0.0823],<br>
[ 0.0760,  0.3130, -0.2936,  …,  0.1302, -0.7095, -0.1434],<br>
[ 0.0592,  0.2099, -0.3150,  …,  0.1744, -0.8228, -0.6104],<br>
…,<br>
[ 0.0553,  0.0501, -0.2397,  …,  0.2563, -0.5291, -0.3276],<br>
[-0.0884, -0.1168, -0.1549,  …,  0.1795, -0.1136,  0.0661],<br>
[ 0.3371,  0.1840, -0.7856,  …,  0.2078, -0.0932, -0.4215]],<br>
device=‘cuda:0’, grad_fn=)}</p>
</li>
<li>
<p>TRAINING mode with dropout = 0.0 - can I consider it as DROPOUT is off, right?<br>
tensor([[[-0.8847,  0.8008,  1.3410,  …,  1.9398, -0.9303, -0.7386],<br>
[-0.9201,  0.8979,  1.4128,  …,  1.9263, -0.8656, -0.8076],<br>
[-0.8593,  0.8811,  1.3582,  …,  1.8995, -0.9350, -0.7555],<br>
…,<br>
[-0.9158,  0.9414,  1.3551,  …,  1.8739, -0.8665, -0.9009],<br>
[-0.8818,  0.8018,  1.3489,  …,  1.9480, -0.9242, -0.8278],<br>
[-0.8603,  0.7642,  1.3655,  …,  2.0025, -0.9490, -0.8010]]],<br>
device=‘cuda:0’, grad_fn=)<br>
{‘pred_logits’: tensor([[-0.1241,  0.0806, -0.3436,  …,  0.2659, -0.3817, -0.2606],<br>
[-0.0993,  0.0815, -0.3826,  …,  0.2688, -0.3978, -0.2872],<br>
[-0.1544,  0.0542, -0.3570,  …,  0.2584, -0.3570, -0.2809],<br>
…,<br>
[-0.0540,  0.0955, -0.3878,  …,  0.2575, -0.3749, -0.3109],<br>
[-0.1354,  0.0464, -0.3761,  …,  0.2357, -0.3562, -0.2966],<br>
[-0.1206,  0.0465, -0.3985,  …,  0.2530, -0.3951, -0.2766]],<br>
device=‘cuda:0’, grad_fn=)}</p>
</li>
<li>
<p>EVAL mode with or without dropout doesn’t matter as it’s off:<br>
tensor([[[-0.8847,  0.8008,  1.3410,  …,  1.9398, -0.9303, -0.7386],<br>
[-0.9201,  0.8979,  1.4128,  …,  1.9263, -0.8656, -0.8076],<br>
[-0.8593,  0.8811,  1.3582,  …,  1.8995, -0.9350, -0.7555],<br>
…,<br>
[-0.9158,  0.9414,  1.3551,  …,  1.8739, -0.8665, -0.9009],<br>
[-0.8818,  0.8018,  1.3489,  …,  1.9480, -0.9242, -0.8278],<br>
[-0.8603,  0.7642,  1.3655,  …,  2.0025, -0.9490, -0.8010]]],<br>
device=‘cuda:0’, grad_fn=)<br>
{‘pred_logits’: tensor([[-0.1241,  0.0806, -0.3436,  …,  0.2659, -0.3817, -0.2606],<br>
[-0.0993,  0.0815, -0.3826,  …,  0.2688, -0.3978, -0.2872],<br>
[-0.1544,  0.0542, -0.3570,  …,  0.2584, -0.3570, -0.2809],<br>
…,<br>
[-0.0540,  0.0955, -0.3878,  …,  0.2575, -0.3749, -0.3109],<br>
[-0.1354,  0.0464, -0.3761,  …,  0.2357, -0.3562, -0.2966],<br>
[-0.1206,  0.0465, -0.3985,  …,  0.2530, -0.3951, -0.2766]],<br>
device=‘cuda:0’, grad_fn=)}</p>
</li>
</ol>
<p>All of 1-3 were done using training data (same sample).<br>
So 2 and 3 have identical outputs, which are very similar to each other in each column.<br>
It means the problem of .eval() for me are Dropout layers…can it happen because of my data? F.ex. should I scale the dataset?<br>
Then the question is how can I overcome this issue?</p>

KhanMar: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> can you please share ideas, if any? <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Morteza_Janatdoust: <p>Hi,<br>
I have the same problem as <a class="mention" href="/u/khanmar">@KhanMar</a> . I encountered the same output in “.eval()” evaluation mode. My model also includes a transformer. I think this bug is in the transformer itself.</p>

dleemiller: <p>I have encountered the same issue as well. I trained a model with dropout p=0 in the transformer encoder block. Later, if I load the weights, I get correct results from train() with torch.no_grad(), and incorrect results that are uniform regardless of input if I use eval().</p>
<p>I noticed that since 1.12, there were some big structural changes in the transformer code. I will try again using 1.11 and report back.</p>

dleemiller: <p>I believe the bug is here:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/b04b2fa9aa52cacbdc9aaaf477d55b0af845ce81/torch/nn/modules/transformer.py#L449">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/b04b2fa9aa52cacbdc9aaaf477d55b0af845ce81/torch/nn/modules/transformer.py#L449" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/b04b2fa9aa52cacbdc9aaaf477d55b0af845ce81/torch/nn/modules/transformer.py#L449" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/b04b2fa9aa52cacbdc9aaaf477d55b0af845ce81/torch/nn/modules/transformer.py#L449</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="439" style="counter-reset: li-counter 438 ;">
          <li>    src_key_padding_mask: the mask for the src keys per batch (optional).</li>
          <li>
          </li>
<li>Shape:</li>
          <li>    see the docs in Transformer class.</li>
          <li>"""</li>
          <li>
          </li>
<li># see Fig. 1 of https://arxiv.org/pdf/2002.04745v1.pdf</li>
          <li>why_not_sparsity_fast_path = ''</li>
          <li>if not src.dim() == 3:</li>
          <li>    why_not_sparsity_fast_path = f"input not batched; expected src.dim() of 3 but got {src.dim()}"</li>
          <li class="selected">elif self.training:</li>
          <li>    why_not_sparsity_fast_path = "training is enabled"</li>
          <li>elif not self.self_attn.batch_first :</li>
          <li>    why_not_sparsity_fast_path = "self_attn.batch_first was not True"</li>
          <li>elif not self.self_attn._qkv_same_embed_dim :</li>
          <li>    why_not_sparsity_fast_path = "self_attn._qkv_same_embed_dim was not True"</li>
          <li>elif not self.activation_relu_or_gelu:</li>
          <li>    why_not_sparsity_fast_path = "activation_relu_or_gelu was not True"</li>
          <li>elif not (self.norm1.eps == self.norm2.eps):</li>
          <li>    why_not_sparsity_fast_path = "norm1.eps is not equal to norm2.eps"</li>
          <li>elif src_mask is not None:</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">        why_not_sparsity_fast_path = ''
        ...
        elif self.training:
            why_not_sparsity_fast_path = "training is enabled"
        ...

            if (not why_not_sparsity_fast_path) and (src_key_padding_mask is not None):
                convert_to_nested = True
                output = torch._nested_tensor_from_mask(output, src_key_padding_mask.logical_not(), mask_check=False)
                src_key_padding_mask_for_layers = None
</code></pre>
<p>There is a long set of conditionals that will create a different behavior, which is likely to reproduce differently under many circumstances. If you look at the code, <code>why_not_sparsity_fast_path</code> is always set during training, but not for eval.</p>
<p>The eval mode performs a modification of <code>output</code> using <code>src_key_padding_mask</code>. If you are not using src key padding mask, you will not observe the behavior, but if you set it to None when you use eval(), then it works as expected (but without src key padding mask).</p>
<p>This big conditional for <code>why_not_sparsity_fast_path</code> is not used in version 1.11, so I expect that rolling back will fix the issue.</p>
<p>Edit: I can confirm that loading my weights using 1.11 fixes the issue for me.</p>

ptrblck: <p>Could you create an issue on GitHub with your explanation, please, so that the code owners could take a look at it?</p>

dleemiller: <p>I’ve opened an issue here:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/86120">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/86120" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/86120" target="_blank" rel="noopener nofollow ugc">TransformerEncoder src_key_padding_mask does not work in eval()</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2022-10-03" data-time="15:46:50" data-timezone="UTC">03:46PM - 03 Oct 22 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/AdventLee" target="_blank" rel="noopener nofollow ugc">
          <img alt="AdventLee" src="https://avatars.githubusercontent.com/u/101732276?v=4" class="onebox-avatar-inline" width="20" height="20">
          AdventLee
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

Described here: https://discuss.pytorch.org/t/model-eval<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">-predicts-all-the-same-or-nearly-same-outputs/155025/12

Starting with version 1.12, TransformerEncoder uses two different code paths, which perform differently between training and evaluation mode. In training, the `why_not_sparsity_fast_path` will be set, and the sparsity fast path will not be used. During eval(), if using a `src_key_padding_mask`, output will be modified:

https://github.com/pytorch/pytorch/blob/b04b2fa9aa52cacbdc9aaaf477d55b0af845ce81/torch/nn/modules/transformer.py#L271
```
            if (not why_not_sparsity_fast_path) and (src_key_padding_mask is not None):
                convert_to_nested = True
                output = torch._nested_tensor_from_mask(output, src_key_padding_mask.logical_not(), mask_check=False)
                src_key_padding_mask_for_layers = None
```

The output is usually some constant value. It seems like their might be some misalignment with the expected `src_key_padding_mask` and the `logical_not()` operation. Perhaps the mask inversion should not be performed.

### Versions

Python version: 3.8.13 (default, Mar 28 2022, 06:59:08) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.22621-SP0
Is CUDA available: True
CUDA runtime version: 11.6.124
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 516.94
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.7.1
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.3
[pip3] segmentation-models-pytorch==0.3.0
[pip3] torch==1.12.1+cu116
[pip3] torchaudio==0.12.1+cu116
[pip3] torchmetrics==0.9.3
[pip3] torchvision==0.13.1+cu116
[conda] efficientnet-pytorch      0.7.1                    pypi_0    pypi
[conda] numpy                     1.23.3                   pypi_0    pypi
[conda] segmentation-models-pytorch 0.3.0                    pypi_0    pypi
[conda] torch                     1.12.1+cu116             pypi_0    pypi
[conda] torchaudio                0.12.1+cu116             pypi_0    pypi
[conda] torchmetrics              0.9.3                    pypi_0    pypi
[conda] torchvision               0.13.1+cu116             pypi_0    pypi</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


mikekgfb: <p>Thanks so much for opening a defect on this issue.  We’re trying to recreate the problem, but we have not been able to create a test cases that produces incorrect results in the presence of src_key_padding_mask.Thus, I wanted to follow up in on Rui’s request for a repro for this issue at <a href="https://github.com/pytorch/pytorch/issues/86120" class="inline-onebox" rel="noopener nofollow ugc">TransformerEncoder src_key_padding_mask does not work in eval() · Issue #86120 · pytorch/pytorch · GitHub</a></p>
<p>cc: <a class="mention" href="/u/dleemiller">@dleemiller</a></p>

dleemiller: <p>I added a code snippet that I think narrows down the issue.</p>
<p>I think that if the src_key_padding_mask is supplied with a boolean mask, then it resolves this issue; however, using another type (long, float) appears to be problematic.</p>

mikekgfb: <p>This is super helpful.  Thanks so much for narrowing this down, and providing a repro.</p>