Run_backward: expected dtype Float but got dtype Long

juperez999: <p>Trying to run the Variable._execution_engine.run_backward call and I am getting an error:<br>
RuntimeError: expected dtype Float but got dtype Long. Even though both tensors are float32, I have attached a picture below showing the output from the debugger along with the error. <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db7555d4e7429603bd3ba3fa959169c34916eb6f.png" data-download-href="https://discuss.pytorch.org/uploads/default/db7555d4e7429603bd3ba3fa959169c34916eb6f" title="52%20AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db7555d4e7429603bd3ba3fa959169c34916eb6f_2_554x499.png" alt="52%20AM" data-base62-sha1="vjpXfJwPT6Hyz5DbYbkNekWDc5x" width="554" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db7555d4e7429603bd3ba3fa959169c34916eb6f_2_554x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db7555d4e7429603bd3ba3fa959169c34916eb6f_2_831x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db7555d4e7429603bd3ba3fa959169c34916eb6f_2_1108x998.png 2x" data-dominant-color="F4EDEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">52%20AM</span><span class="informations">1758×1584 437 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We are using the torch.nn.MSELoss function and we are getting similar results with the torch.nn.SmoothL1Loss function. Any ideas on what the problem is or what next steps should be?<br>
We are running pytorch version 1.4.0.dev20191111.</p>

albanD: <p>Could you run with anomaly mode enabled to see which forward function creates this error please?</p>

juperez999: <p>So I reran using the following code and still received same error (no changes).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18e1d957ae32392e022f19c82e3f7bdbc3c29388.png" data-download-href="https://discuss.pytorch.org/uploads/default/18e1d957ae32392e022f19c82e3f7bdbc3c29388" title="14%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18e1d957ae32392e022f19c82e3f7bdbc3c29388_2_690x187.png" alt="14%20PM" data-base62-sha1="3y7kZiXTrI6BEhPEsisxVDvjggM" width="690" height="187" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/8/18e1d957ae32392e022f19c82e3f7bdbc3c29388_2_690x187.png, https://discuss.pytorch.org/uploads/default/original/3X/1/8/18e1d957ae32392e022f19c82e3f7bdbc3c29388.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/8/18e1d957ae32392e022f19c82e3f7bdbc3c29388.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">14%20PM</span><span class="informations">1018×276 30.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Should I be running a different way? Also I am leveraging fastai (ver 1.0.59) just in case that matters.</p>

juperez999: <p>Here is the full error stack from the anomaly detect run.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d69a350cfa0eb3005b8509dea453eb006dece3cd.png" data-download-href="https://discuss.pytorch.org/uploads/default/d69a350cfa0eb3005b8509dea453eb006dece3cd" title="51%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d69a350cfa0eb3005b8509dea453eb006dece3cd_2_620x500.png" alt="51%20PM" data-base62-sha1="uCsz6fgtfMF4wx16CdnmlRd1dxP" width="620" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d69a350cfa0eb3005b8509dea453eb006dece3cd_2_620x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d69a350cfa0eb3005b8509dea453eb006dece3cd_2_930x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/6/d69a350cfa0eb3005b8509dea453eb006dece3cd.png 2x" data-dominant-color="F2D2D2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">51%20PM</span><span class="informations">965×777 145 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(can only upload one screen capture at a time)</p>

albanD: <p>You should get two stack traces. One with a function in the forward and one with the backward call.</p>

juperez999: <p>When I run this with detect anomaly on, I get the same error stack trace. I dont have a NaN value problem. My tensors and loss are showing as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/94f2d7daadcc830f2f6089b61d6c4c109a3a929c.png" data-download-href="https://discuss.pytorch.org/uploads/default/94f2d7daadcc830f2f6089b61d6c4c109a3a929c" title="43%20PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/94f2d7daadcc830f2f6089b61d6c4c109a3a929c_2_690x356.png" alt="43%20PM" data-base62-sha1="lfEXc1yzVtSkNcvpX6E2VKvE5uk" width="690" height="356" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/94f2d7daadcc830f2f6089b61d6c4c109a3a929c_2_690x356.png, https://discuss.pytorch.org/uploads/default/original/3X/9/4/94f2d7daadcc830f2f6089b61d6c4c109a3a929c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/4/94f2d7daadcc830f2f6089b61d6c4c109a3a929c.png 2x" data-dominant-color="F5F7F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">43%20PM</span><span class="informations">832×430 58.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We can run the forward function independently entirely and there are no problems as is shown from the output of the debug screen capture above.</p>

juperez999: <p><a class="mention" href="/u/alband">@albanD</a> Just wanted to let you know that this had to do with the shape of the tensors. We were able to fix the problem by using the MSELossFlat loss function provided in the fastai framework. This essentially wraps the torch.nn.MSELoss function in a flattener.<br>
Below is a link to the fastai version of the loss function.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/fastai/fastai/blob/8013797e05f0ae0d771d60ecf7cf524da591503c/fastai/layers.py#L257" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/fastai/fastai/blob/8013797e05f0ae0d771d60ecf7cf524da591503c/fastai/layers.py#L257" target="_blank" rel="nofollow noopener">fastai/fastai/blob/8013797e05f0ae0d771d60ecf7cf524da591503c/fastai/layers.py#L257</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="247" style="counter-reset: li-counter 246 ;">
<li>    return FlattenedLoss(nn.CrossEntropyLoss, *args, axis=axis, **kwargs)</li>
<li>
</li>
<li>def BCEWithLogitsFlat(*args, axis:int=-1, floatify:bool=True, **kwargs):</li>
<li>    "Same as `nn.BCEWithLogitsLoss`, but flattens input and target."</li>
<li>    return FlattenedLoss(nn.BCEWithLogitsLoss, *args, axis=axis, floatify=floatify, is_2d=False, **kwargs)</li>
<li>
</li>
<li>def BCEFlat(*args, axis:int=-1, floatify:bool=True, **kwargs):</li>
<li>    "Same as `nn.BCELoss`, but flattens input and target."</li>
<li>    return FlattenedLoss(nn.BCELoss, *args, axis=axis, floatify=floatify, is_2d=False, **kwargs)</li>
<li>
</li>
<li class="selected">def MSELossFlat(*args, axis:int=-1, floatify:bool=True, **kwargs):</li>
<li>    "Same as `nn.MSELoss`, but flattens input and target."</li>
<li>    return FlattenedLoss(nn.MSELoss, *args, axis=axis, floatify=floatify, is_2d=False, **kwargs)</li>
<li>
</li>
<li>class NoopLoss(Module):</li>
<li>    "Just returns the mean of the `output`."</li>
<li>    def forward(self, output, *args): return output.mean()</li>
<li>
</li>
<li>class WassersteinLoss(Module):</li>
<li>    "For WGAN."</li>
<li>    def forward(self, real, fake): return real.mean() - fake.mean()</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>This an interesting bug because we have not encountered it before in our classification problems. We freely interchange pytorch and fastai loss functions, however, in this case it was a problem. We were working the rossmann sales problem.</p>

ptrblck: <p>Just to clarify the solution:<br>
you were able to avoid the type mismatch by using the FastAI <code>MSELossFlat</code> method, but are still seeing it using plain PyTorch code?</p>

Phanikumar2all: <p>Hello,</p>
<p>Is there any other way to solve this problem?. I am facing the same problem too.</p>

ptrblck: <p>Are you seeing this issue using FastAI as well or plain PyTorch?</p>

Phanikumar2all: <p>I solved that issue by converting my tensors to float using .float(). This worked.</p>