Remove temp folder before and after running all tests

Fixes #4971