SimTk/openmm — pull request #81 by rmcgibbo (2013-07-27 05:45:44+00:00)
Title: WORKAROUND: Titan compute nodes lack write access to /tmp, which is used internally by nvcc during jit compilation.

I don't really want to merge this PR, but I want to submit it (and then close it without merging) to serve as a record for the workaround that I'm using on ORNL titan, and perhaps get some input from others to see if there's a better way.

The problem is that `nvcc` will not run on the compute nodes. You can verify this with `aprun -n 1 -a xt nvcc helloworld.cu` for some simple kernel `helloworld.cu`. The error message is something about being unable to write to /tmp. These nvcc [docs](http://docs.nvidia.com/cuda/cuda-compiler-driver-nvcc) seem to indicate that on linux, nvcc uses /tmp internally for save some data, and that it's not configurable. On windows, it is configurable. Messing with environment variables does not seem to help.

The workaround in this PR is to write out the `.cu` kernels to disk with the first line containing the full command used to do the compilation (commented out). Then, if the compilation fails, we wait in an infinite loop until the ptx file appears on disk. Once it appears, we load it in.

Now before running OpenMM on the compute node, I run a little background process on the PBS MOM node that watches in an infinite loop for the appearance of new `.cu` files in the temporary directory, parses their first line, and compiles them to ptx. That code is [here](https://github.com/rmcgibbo/watchcompiler/blob/master/watchcompiler).