Shapes matched properly, but I get error: RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

superbunny38: <p>Hello,<br>
I have an error and I paste the error message below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afa0cfb17dfc14474e333152536d083a1f507a21.png" data-download-href="https://discuss.pytorch.org/uploads/default/afa0cfb17dfc14474e333152536d083a1f507a21" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/f/afa0cfb17dfc14474e333152536d083a1f507a21.png" alt="image" data-base62-sha1="p3G5P9VTKN5gSHlZnhDIgbrw4fv" width="690" height="398" data-dominant-color="F7F7F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1170×676 12.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Basically, I was doing a classification task with a model, and I get this error when I try to do back propagation. I was trying to solve this issue, so I checked a few posts that say that this error is due to the shape-matching problem.<br>
However, for my error, I doubt that this is a shape-not-matching problem, because forward propagation works without any error. I can calculate loss just fine on cuda without any problems, as you can see in the screenshot.</p>
<p>FYI, the loss is categorical cross entropy loss! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
I will wait for your replies!<br>
Thank you.</p>

ptrblck: <p>Could you run your code on the CPU and check if you are getting a better error message?<br>
We have unfortunately seen a few errors wrongly reported as cuBLAS or cuDNN issues as the actual error was not properly caught and raised.</p>

superbunny38: <p>Hello, thank you so much for your reply! <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"></p>
<p>the one that I got an error was from the sever of my company’s environment.<br>
However, as you could see from the attached screenshot below, I didn’t get the error that I reported, from a different environment from that of my company’s. So I think it’s probably a specific version of a library that is throwing me the error. I’ll check on Monday and figure out which version of which library is problematic.</p>
<p>Thank you so much! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1ca62e552c735e2f22e7f3d54e0aa276ec83c8da.png" alt="image" data-base62-sha1="45rpraU21XMXI1bZTN7nO6EvSWK" width="619" height="246"></p>

ptrblck: <p>Could you update to the latest stable or nightly release, too?<br>
In case you can still reproduce the issue on the other machine, could you post a minimal and executable code snippet, please?</p>