Do not leak tempdirs

Be nice and clean up after yo'self!

Fixes #1333, although I can't speak for Ridley. I didn't have time to go digging there.

/cc @stevendanna