Build in temp dir

This part of change is also in #2382 , you might only add it in one Pull request to avoid conflict when merging.