Revert "Make sure we always override the temp dir"

This reverts commit f79f400bfd5295adbca6fbb3030a6db1ac9257ae.