build: Remove temporary directories

Otherwise we ENOSPC quickly.