Simplify implementation of TemporaryWorkingDirectory.

Closes gh-5160

This means `TemporaryWorkingDirectory` can only be used as a context manager, but that's the only way we were using it anyway.