Only remove the build dir if it exists