Backport PR #5163: Simplify implementation of TemporaryWorkingDirectory.

Closes gh-5160

This means `TemporaryWorkingDirectory` can only be used as a context manager, but that's the only way we were using it anyway.
whatsnew 1.2.1