Code change:
Kind of similar to Sometimes our Eigen build fails this way: It doesn't seem to reliably repro. cc
I got some sort of similar-ish error persistently on a test branch:
Not that persistently; a PR stacked on the failing one succeeded. So there is something nondeterministic going on here.
another instance: CI on
Looks like an NVCC bug. The workground is protecting eigen headers with Can we extract some basic code example and compile it against NVCC?
Another one in
Can you try with 10.2?
Mingbo, can you put general information on the issue you are seeing here in this issue?
latest word on this is from mingbo: "from what I can tell, it's a flaky one. might depend on which VS version. I hit the problem before and problem fixed (at least seem so) after upgrading VS 2019 latest version" and "the same error showed up once last week, and re-run the job finished without problem."
Hi. Did anyone have this issue persistently, ie that a rebuild did not "fix" it? We've had this issue in another (closed source) project. Exposing cuda to less includes (such as boost) seems to help but is not predictable.
It's never been persistent for us, but this is mostly from our Windows CI builds where we blow everything away and rebuild from scratch, which may help.
Yet another occurrence in the forum: It seems to be fixed by , which implies there may be a race condition in reading the include files or generating the intermediate source files.
I was able to reproduce the issue with a more verbose output with And then I inspected the variables carefully and found out that the VC env is activated twice. We should really avoid that.
You are SO COOL!!!
I tried to build from scratch several times with this PR and could not reproduce the issue anymore. Let's assume it's fixed.
Looks like it was not resolved.
Another occurrence: This time it seems to claim the error is around the usage of .
This sounds like a good old multiple-instances-of-nvcc-write-to-the-same-temp-file One can invoke to see all temporary filenames it creates. I guess on windows they are more likely to collide with each other than on linux. One solution to the problem would be to invoke nvcc sequentially. Another is to tweak environment (there must be something similar to TMPDIR on Windows, mustn't it?)
Yes, they are called TMP and TEMP.
Is there any example of the latter method on Unix systems?
I've written a utility . Hopefully it can be used to resolve this problem if that's the case.
New occurrence after the previous fix:
seems related to
If this is the case, maybe we could apply to fix that.
unfortunately, this introduces another thrust problem (e.g. ): D:/programs64/cuda101/includethrust/detail/complex/catrig.h(607): error: no instance of function template "thrust::atanh" matches the argument list argument types are: (double) with problematic include chain -thrust/complex.h -thrust/detail/complex/catrig.h (has cplusplus conditions) -thrust/detail/complex/c99math.h where "using ::atanh" is omitted for MSVC. Patching c99math.h (or even thrustdetailconfigcppdialect.h without using /Zc:_cplusplus) seems to work for me, but I'm not sure what's the best way to fix this within pytorch tree and whether c++ dialect is the only problem here.
Thanks for the investigation. BTW, could you please share with us how did you patch c99math.h?
Disregard that, still got same unreproducible C2993 error with /Zc:_cplusplus. Earlier, I had "build" directory reproducibly failing (at ), and hardcoding c++17 in thrustdetailconfigcppdialect.h fixed that. But looks like it is just a symptom of some inconsistence in environment/flags (across nvcc stages maybe?) just "using ::atanh"
Yeah, maybe. I am considering to add a wrapper to retry the build a few times.
I can now stably reproduce a related error by passing in .
By commenting out some part of the code, I find the code that causes the previous error. I don't know why it is executed in the device code.
Well, I figured out the reason. It seems that and are implicitly marked as and . This behavior can be disabled by adding .
Nice work!
Seems to be a nvcc bug
Thanks for reminding. I have commented there.
This is indeed an nvcc bug. There is no known workaround at the moment, but the next release of the CUDA toolkit will contain a fix. Ref thrust/thrust.
Met this problem:
Just retry a few times.
The CUDA TK 11.0 RC is now available: This issue has been fixed in that version of nvcc.
Thanks for the info. Actually, our release binaries rely on the cudatoolkit package in anaconda cloud. So we won't be able to publish the binaries with CUDA 11 now. I'm just curious whether that is maintained by Nvidia? Another question, any plan to backport the changes to exisiting CUDA versions (e.g. cuda 10.1, cuda 10.2)?
I don't have that information unfortunately -- it'd be best to contact the maintainer of that package and see when they plan to update. I'm not aware of any plans to backport this fix.
I have also obtained this error with CUDA 10.1/cuDNN 7.6.4 on Win Server 2019.
this is a cuda bug. Please see above discussion and upgrade to cuda 11.
I fixed the error by installing sscache for use with cuda 10.1 and cuda 10.2, as done here.
it's an intermittent error and the probability of occurrence varies depending on your system. So it's not really fixed, but you may have found a way to reduce the occurrence in your system, which may be sufficient in your case :)
Another instance, this time inside protobuf: Running with I found the syntax error is introduced by which is the cuda compiler front end. operates on the output from the compiler's pre-processor and that output barely even changed to trigger the build error, only line number metadata and some extra whitespace was changed in the pre-processed output. Here is the diff of the two inputs, one of which failed and the other didn't: And the corresponding change in the output: This failure reproduced reliably on the CI machine. Running nvcc manually multiple times, it always fails or passes given the same inputs. I think it's a deterministic failure, but is annoyingly sensitive to any changes in the input file whatsoever.
closing due to age
Wait, do we think this is fixed? It's not clear to me it is...
I think this was related to CUDA 10.2 builds on windows which we don't actually support anymore so it is probably safe to keep closed.

Related discussion:
text: lib_paths = list(filter(bool, [ os.getenv('CUDNN_LIB_DIR'), os.path.join(CUDA_HOME, 'lib'), <del> os.path.join(CUDA_HOME, 'lib64') </del> <ins> os.path.join(CUDA_HOME, 'lib64'), '/usr/lib/x86_64-linux-gnu/', </ins> ])) include_paths = list(filter(bool, [ os.getenv('CUDNN_INCLUDE_DIR'), os.path.join(CUDA_HOME, 'include'), <ins> '/usr/include/' </ins> ])) for path in lib_paths: if path is None or not os.path.exists(path):