Change:
As part of our CI we run "cargotest" on a variety of ecosystem crates, making sure they successfully build and run tests. Unfortunately, some of use stochastic tests like quickcheck, which means that a crate passing our CI one day may not necessarily mean it passes CI the next. Perhaps we should audit the tests being run and filter out ones like these? See and cc
Yeah I think we've discussed this to some extent previously within infra team as part of our "what to do about cargotest" discussions. cc in particular Not sure that we came to any conclusions though.
Related:
The test from xsv fails occasionally:
This just bit me and took a lot of time as I triple-checked that after bors failing twice (!) in xsv, the same test actually works fine on my system.
The xsv test suite seems to be full of quickcheck tests, so I think the only way to fix this is to remove xsv from cargotest (or to only run a short list of curated tests).
See where there is some discussion fo changing the cargotest set. I think implies dropping xsv.
Hm, that discussion seems to have stalled though. So to fix the immediate problem, I propose we do something like

Discussion:
packages: &'static [&'static str], features: Option<&'static [&'static str]>, manifest_path: Option<&'static str>, <ins> /// `filters` are passed to libtest (i.e., after a `--` in the `cargo test` invocation). filters: &'static [&'static str], </ins> } const TEST_REPOS: &[Test] = &[

packages: &[], features: None, manifest_path: None, <ins> filters: &[], </ins> }, Test { name: "ripgrep",

packages: &[], features: None, manifest_path: None, <ins> filters: &[], </ins> }, Test { name: "tokei",

packages: &[], features: None, manifest_path: None, <ins> filters: &[], </ins> }, Test { name: "xsv",

packages: &[], features: None, manifest_path: None, <ins> // Many tests here use quickcheck and some of them can fail randomly, so only run deterministic tests. filters: &[ "test_flatten::", "test_fmt::", "test_headers::", "test_index::", "test_join::", "test_partition::", "test_search::", "test_select::", "test_slice::", "test_split::", "test_stats::", "test_table::", ], </ins> }, Test { name: "servo",

packages: &["selectors"], features: None, manifest_path: None, <ins> filters: &[], </ins> }, Test { name: "diesel",

// not any other crate present in the diesel workspace // (This is required to set the feature flags above) manifest_path: Some("diesel/Cargo.toml"), <ins> filters: &[], </ins> }, ];

if let Some(lockfile) = test.lock { fs::write(&dir.join("Cargo.lock"), lockfile).unwrap(); } <del> if !run_cargo_test(cargo, &dir, test.packages, test.features, test.manifest_path) { </del> <ins> if !run_cargo_test(cargo, &dir, test.packages, test.features, test.manifest_path, test.filters) { </ins> panic!("tests failed for {}", test.repo); } }

packages: &[&str], features: Option<&[&str]>, manifest_path: Option<&str>, <ins> filters: &[&str], </ins> ) -> bool { let mut command = Command::new(cargo_path); command.arg("test");

command.arg("-p").arg(name); } <ins> command.arg("--"); command.args(filters); </ins> let status = command // Disable rust-lang/cargo's cross-compile tests .env("CFG_DISABLE_CROSS_TESTS", "1")