SimTk/openmm — pull request #84 by peastman (2013-07-29 20:31:25+00:00)
Title: Cache the PTX files for compiled kernels

This is a feature I've had in mind for a long time, but I've also been hesitant to implement it. I'd appreciate any thoughts on whether it's a good idea, and how it could be improved.

After compiling kernels to PTX, it saves the resulting files instead of deleting them. The next time you run your program, it can just reuse the file without having to recompile it. This makes a big difference to the Context creation time. It also offers a workaround for cases like Robert's where you can't run nvcc on the compute node, since it can just reuse files compiled elsewhere.

But this needs to be absolutely bulletproof, or it could create all sorts of problems. That's what I'm worried about. I've tried hard to make it robust, but try to think of ways it might fail.

The filename is a SHA1 hash of the kernel source code, plus the compute capability it was compiled for. I think that should be sufficient to guarantee it will never use a cache file it shouldn't. If it tries to load the cache file and gets any sort of error, it just ignores the file and recompiles the kernel. Hopefully that's sufficient to deal with cases where a cache file has been corrupted. It compiles to a unique filename, just as it did before, and only renames it once it has successfully loaded the module, so that should also protect against creating corrupt files in the cache. And if an error occurs while trying to rename the file, it just gives up on caching and deletes it, as it did before.

Currently, it never deletes cache files. They just build up. On Mac and Linux, everything in the temp directory gets deleted automatically whenever you reboot, so that will clean them out. But I don't know how the temp directory is handled on clusters that are rarely rebooted. Also, Windows does not seem to ever delete temporary files. I'm not sure how to deal with this. I don't want temporary files to just build up forever, but any scheme for deleting them creates lots of potential problems.

For reference, running the full suite of unit tests creates about 25 MB of cached files. I think I can decrease that somewhat, though.