Make build less flaky by using set comparison