Don't remove a build directory that will never be there