Better cleanup of temporary files in tests

Not sure I've squashed them all, but this is a start
> This looks suspect,

It was definitely being used for directories (at least in some cases). Happy to fix the call-sites if you prefer.