Model evaluation fails when using GPUs but works well on CPU

username: DJB
cooked: <p>My model is trained using multiple GPUs and it works well on training. However, the problem happens during evaluation. it is pretty strange to me since the evaluation phase doesn’t show any problem using the CPU. However, when using the GPU, it makes an error like below. plz help me and thanks in advance.</p>
<p>(I’m using torch.transformer_encoder layers and the error message seems like it is related to the encoder mask problem but I don’t know what to do… the shape of mask is well defined and there was no problem on CPU and when training)</p>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
Input In [49], in &lt;cell line: 4&gt;()<br>
10 pes = pes.to(device)<br>
11 pads = pads.to(device)<br>
—&gt; 13 outputs = model(seqs, pes, pads)</p>
<p>File ~/anaconda3/envs/rna_ss/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)<br>
1126 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1127 # this function, and just call forward.<br>
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1129         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1130     return forward_call(*input, **kwargs)<br>
1131 # Do not call functions when jit is used<br>
1132 full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>File ~/RSSP/RSSP/model.py:95, in Model.forward(self, seqs, pes, pads, verbose)<br>
92 def forward(self, seqs, pes, pads, verbose=False):<br>
93     input, pe = self.input_generator(seqs, pes)<br>
—&gt; 95     encoded = self.transformer_encoder(input, src_key_padding_mask = pads[:,:,0])<br>
97     concatenated_encoder_output = torch.cat([encoded, pe], dim=2)<br>
99     mat = matrix_rep(concatenated_encoder_output) # symmetric</p>
<p>File ~/anaconda3/envs/rna_ss/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)<br>
1126 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1127 # this function, and just call forward.<br>
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1129         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1130     return forward_call(*input, **kwargs)<br>
1131 # Do not call functions when jit is used<br>
1132 full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>File ~/anaconda3/envs/rna_ss/lib/python3.9/site-packages/torch/nn/modules/transformer.py:238, in TransformerEncoder.forward(self, src, mask, src_key_padding_mask)<br>
236         output = mod(output, src_mask=mask)<br>
237     else:<br>
 → 238         output = mod(output, src_mask=mask, src_key_padding_mask=src_key_padding_mask)<br>
240 if convert_to_nested:<br>
241     output = output.to_padded_tensor(0.)</p>
<p>File ~/anaconda3/envs/rna_ss/lib/python3.9/site-packages/torch/nn/modules/module.py:1130, in Module._call_impl(self, *input, **kwargs)<br>
1126 # If we don’t have any hooks, we want to skip the rest of the logic in<br>
1127 # this function, and just call forward.<br>
1128 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks<br>
1129         or _global_forward_hooks or _global_forward_pre_hooks):<br>
 → 1130     return forward_call(*input, **kwargs)<br>
1131 # Do not call functions when jit is used<br>
1132 full_backward_hooks, non_full_backward_hooks = [], []</p>
<p>File ~/anaconda3/envs/rna_ss/lib/python3.9/site-packages/torch/nn/modules/transformer.py:437, in TransformerEncoderLayer.forward(self, src, src_mask, src_key_padding_mask)<br>
417     tensor_args = (<br>
418         src,<br>
419         self.self_attn.in_proj_weight,<br>
(…)<br>
430         self.linear2.bias,<br>
431     )<br>
432     if (not torch.overrides.has_torch_function(tensor_args) and<br>
433             # We have to use a list comprehension here because TorchScript<br>
434             # doesn’t support generator expressions.<br>
435             all([(x.is_cuda or ‘cpu’ in str(x.device)) for x in tensor_args]) and<br>
436             (not torch.is_grad_enabled() or all([not x.requires_grad for x in tensor_args]))):<br>
 → 437         return torch._transformer_encoder_layer_fwd(<br>
438             src,<br>
439             self.self_attn.embed_dim,<br>
440             self.self_attn.num_heads,<br>
441             self.self_attn.in_proj_weight,<br>
442             self.self_attn.in_proj_bias,<br>
443             self.self_attn.out_proj.weight,<br>
444             self.self_attn.out_proj.bias,<br>
445             self.activation_relu_or_gelu == 2,<br>
446             False,  # norm_first, currently not supported<br>
447             self.norm1.eps,<br>
448             self.norm1.weight,<br>
449             self.norm1.bias,<br>
450             self.norm2.weight,<br>
451             self.norm2.bias,<br>
452             self.linear1.weight,<br>
453             self.linear1.bias,<br>
454             self.linear2.weight,<br>
455             self.linear2.bias,<br>
456             src_mask if src_mask is not None else src_key_padding_mask,  # TODO: split into two args<br>
457         )<br>
458 x = src<br>
459 if self.norm_first:</p>
<p>RuntimeError: Mask shape should match input shape; transformer_mask is not supported in the fallback case.</p>

username: ptrblck
cooked: <p>It seems that the “fast path” in the transformer encoder is used and fails. Could you create a <a href="https://github.com/pytorch/pytorch/issues">GitHub issue</a> in case it’s still failing in the latest nightly release, please?</p>

username: dalbom
cooked: <p>Met the same error today on the stable version.</p>
<p>input.shape: [4, 4096, 32]<br>
mask.shape: [4, 4096]</p>

username: ptrblck
cooked: <p>Could you check the latest nightly binary and create a GitHub issue if you are still seeing the error?<br>
I don’t know if <a class="mention" href="/u/djb">@DJB</a> followed up on it or not.</p>

username: dalbom
cooked: <p>Hi, confirmed the same problem also in the latest nightly binary.</p>
<p>1.11.0 doesn’t have the problem. I’ll create a GitHub issue.</p>
<ul>
<li>Update<br>
There was already <a href="https://github.com/pytorch/pytorch/issues/83142" rel="noopener nofollow ugc">a report</a>
</li>
</ul>

username: ptrblck
cooked: <p>The latest nightly binary should work, since 83142 was already merged.<br>
If you are still seeing the issue, feel free to update the corresponding issue.</p>

username: Kong
cooked: <p>I am facing the same issue. I read the fix in <a href="https://github.com/pytorch/pytorch/pull/83639" class="inline-onebox" rel="noopener nofollow ugc">Fix issue in softmax.cu with transformer error when mask seqlen &gt; 1024 by erichan1 · Pull Request #83639 · pytorch/pytorch · GitHub</a> but im still unsure of what to do? Do I need a different version of PyTorch? My current version is 1.12.0.</p>
<p>EDIT: Sorry I just created a new environment with the latest version of PyTorch 1.13.1 and I no longer get the error.</p>