Repository: cross-rs/cross
Pull request #873 (2022-06-27 02:35:27)

Add integration tests.
Adds an an integration test for remote Docker support with and without persistent data volumes, and adds it to CI. In addition, we've added integration tests for custom toolchains using `cargo-bisect-rustc`.  Also changes the workflow  to ensure publish depends on fmt, clippy,  and cargo-deny. Fixes a bug when removing data volumes, since it does  not require a target.

Alexhuszagh: Since this would slow down our CI by quite a bit by default (`bisect` took 8 min, `remote` took 4 min), while most of our other actions take ~1m or less, I've changed it so it only runs if we use `bors`, so this works on PRs or if we try. (I'm using an invalid target here intentionally).

bors try --target x
Alexhuszagh: bors try --target xyzabcdef
bors[bot]: ## try

Build failed:
  * [conclusion](https://github.com/cross-rs/cross/runs/7065888414?check_suite_focus=true)
Alexhuszagh: bors try --target xyzabcdef
bors[bot]: ## try

Build failed:
  * [conclusion](https://github.com/cross-rs/cross/runs/7065972743?check_suite_focus=true)
Alexhuszagh: Not sure why the CI failed, maybe it has to do with `build`?
Alexhuszagh: bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Build succeeded:
  * [conclusion](https://github.com/cross-rs/cross/runs/7076841283?check_suite_focus=true)
Alexhuszagh: bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Build succeeded:
  * [conclusion](https://github.com/cross-rs/cross/runs/7078303043?check_suite_focus=true)
Alexhuszagh: Marking as a draft since #878 will affect this (specifically, the docker-in-docker tests).
Alexhuszagh: bors try --target aarch64-unknown-linux-gnu
Alexhuszagh: bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Already running a review
bors[bot]: ## try

Build succeeded:
  * [conclusion](https://github.com/cross-rs/cross/runs/7100020315?check_suite_focus=true)
Alexhuszagh: bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Build succeeded:
  * [conclusion](https://github.com/cross-rs/cross/runs/7100377265?check_suite_focus=true)
Alexhuszagh: bors try --target aarch64-unknown-linux-gnu
Alexhuszagh: bors try-
bors try --target aarch64-unknown-linux-gnu
Alexhuszagh: bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Build succeeded:
  * [conclusion](https://github.com/cross-rs/cross/runs/7152106480?check_suite_focus=true)
Emilgardis: bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Build succeeded:
  * [conclusion](https://github.com/cross-rs/cross/runs/7161341367?check_suite_focus=true)
Emilgardis: Nice, coverage works now!

https://app.codecov.io/gh/cross-rs/cross/compare/873?src=pr&el=h1
Alexhuszagh: We should re-add the docker-in-docker tests here, so we can run the test suite and ensure the volume finder works, and test with custom mounts (so #893 never regresses). We can have a separate test for the cross image itself, I believe.
Alexhuszagh: Currently waiting on a patch for #901.
Emilgardis: bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Build failed:
  * [conclusion](https://github.com/cross-rs/cross/runs/7168912407?check_suite_focus=true)
Alexhuszagh: Mistakenly had an assumption that a container engine would be available for the tests, which isn't a requirement, since the docker-in-docker test needs one to find the proper mount paths. I've gated that test to only run on Linux, if the container engine and hostname are set, and the container engine can inspect the hostname.

bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Build failed:
  * [conclusion](https://github.com/cross-rs/cross/runs/7169365078?check_suite_focus=true)
Alexhuszagh: bors try --target aarch64-unknown-linux-gnu
Alexhuszagh: > I don't understand how the dind test is supposed to help. For example I don't see that it's even tested in CI right now, it just silently fails.

It should be tested. Also, the goal is to prevent a regression like in #901.
bors[bot]: ## try

Build succeeded:
  * [conclusion](https://github.com/cross-rs/cross/runs/7169877000?check_suite_focus=true)
Emilgardis: I mean this one: https://github.com/cross-rs/cross/runs/7169829781?check_suite_focus=true#step:5:46

The unit test. I'm pretty sure it's exiting early, and that because `HOSTNAME` is not defined
Emilgardis: https://app.codecov.io/gh/cross-rs/cross/compare/873/tree/src/docker/shared.rs#D1L891 see coverage
Alexhuszagh: > I mean this one: https://github.com/cross-rs/cross/runs/7169829781?check_suite_focus=true#step:5:46
> 
> The unit test. I'm pretty sure it's exiting early, and that because `HOSTNAME` is not defined

That's intentional, so it doesn't error if docker-in-docker isn't being used. I believe `HOSTNAME` is probably defined, but `docker inspect $HOSTNAME` fails, and so we skip the test. This test, however, shouldn't skip:

https://github.com/cross-rs/cross/runs/7169840511?check_suite_focus=true#step:5:359

Testing it locally with a panic after will fail.

```rust
#[test]
fn test_docker_in_docker() -> Result<()> {
    if !cfg!(target_os = "linux") {
        return Ok(());
    }

    let vars = unset_env();

    let engine = create_engine(MessageInfo::default());
    let hostname = env::var("HOSTNAME");
    if engine.is_err() || hostname.is_err() {
        reset_env(vars);
        return Ok(());
    }
    let engine = engine.unwrap();
    let hostname = hostname.unwrap();
    let output = subcommand(&engine, "inspect")
        .arg(hostname)
        .run_and_get_output(MessageInfo::default())?;
    if !output.status.success() {
        reset_env(vars);
        return Ok(());
    }
    panic!("got past inspect");
    
    ...

    Ok(())
}
```
Alexhuszagh: > https://app.codecov.io/gh/cross-rs/cross/compare/873/tree/src/docker/shared.rs#D1L891 see coverage

Ah that's unusual. Not sure why that's the case, it should be defining `HOSTNAME` inside the image. I guess the diagnostics are good, then.
Emilgardis: Aha, I get it now!

You'll have to forward `RUSTFLAGS` inside the containered container, it's probably doing everything correct just not providing coverage.

See https://github.com/cross-rs/cross/runs/7169840511?check_suite_focus=true#step:19:22 where there's no profraw files and https://github.com/cross-rs/cross/runs/7169840371?check_suite_focus=true#step:10:7 where there is
Alexhuszagh: bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Build succeeded:
  * [conclusion](https://github.com/cross-rs/cross/runs/7170163942?check_suite_focus=true)
Emilgardis: hmm, still not working

```
      Running `CARGO_INCREMENTAL=0 /home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/llvm-profdata merge -sparse -o /home/runner/work/cross/cross/target/cross.profdata`
added 18 packages, and audited 19 packages in 1s
2 packages are looking for funding
  run `npm fund` for details
found 0 vulnerabilities
added 3 packages, changed 3 packages, and audited 22 packages in 360ms
3 packages are looking for funding
  run `npm fund` for details
found 0 vulnerabilities
error: no input files specified. See llvm-profdata merge -help
```

I don't know why.
Alexhuszagh: I have no idea either, I might need to also forward all of these as well:
- `LLVM_PROFILE_FILE`
- `CARGO_INCREMENTAL`
- `RUST_TEST_THREADS`
Emilgardis: oh.. ofc, need to forward `LLVM_PROFILE_FILE` too...
Emilgardis: Incremental would be good to disable, as we don't use the incremental data
Alexhuszagh: > Incremental would be good to disable, as we don't use the incremental data

I've just forwarded everything, since if they're provided by the action, there's probably a good reason for them.

bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Build succeeded:
  * [conclusion](https://github.com/cross-rs/cross/runs/7170342199?check_suite_focus=true)
Alexhuszagh: Code coverage still isn't working. At least the code tests to work on my machine.
Emilgardis: We need to make the target folder available at the same path/the path in LLVM_PROFILE, easiest solution is to do `-v /home/runner/work/cross/cross/target/:/home/runner/work/cross/cross/target/`
Alexhuszagh: bors try --target aarch64-unknown-linux-gnu
bors[bot]: ## try

Build succeeded:
  * [conclusion](https://github.com/cross-rs/cross/runs/7170503381?check_suite_focus=true)
Alexhuszagh: Nothing is working so I'm going to throw in a panic and see if it just "works".
Alexhuszagh: Let's see if it panics:
bors try --target aarch64-unknown-linux-gnu
Emilgardis: It is working, no? https://github.com/cross-rs/cross/runs/7170509736
Alexhuszagh: > It is working, no? https://github.com/cross-rs/cross/runs/7170509736

Wait it is, maybe the code coverage I got was outdated. I'll squash the commits into one.

Patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index b521928..80e89bf 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -321,8 +321,69 @@ jobs:
           LATEST: ${{ needs.check.outputs.is-latest || 'false' }}
         shell: bash
 
+  # we should always have an artifact from a previous build.
+  remote:
+    needs: [shellcheck, test, check]
+    runs-on: ubuntu-latest
+    if: github.actor == 'bors[bot]'
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/setup-rust
+
+      - name: LLVM instrument coverage
+        id: remote-cov
+        uses: ./.github/actions/cargo-llvm-cov
+        with:
+          name: integration-remote
+
+      - name: Run Remote Test
+        env:
+          TARGET: aarch64-unknown-linux-gnu
+        run: ./ci/test-remote.sh
+        shell: bash
+
+  bisect:
+    needs: [shellcheck, test, check]
+    runs-on: ubuntu-latest
+    if: github.actor == 'bors[bot]'
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/setup-rust
+
+      - name: LLVM instrument coverage
+        id: bisect-cov
+        uses: ./.github/actions/cargo-llvm-cov
+        with:
+          name: integration-bisect
+
+      - name: Run Bisect Test
+        env:
+          TARGET: aarch64-unknown-linux-gnu
+        run: ./ci/test-bisect.sh
+        shell: bash
+
+  docker-in-docker:
+    needs: [shellcheck, test, check]
+    runs-on: ubuntu-latest
+    if: github.actor == 'bors[bot]'
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/setup-rust
+
+      - name: LLVM instrument coverage
+        id: docker-in-docker-cov
+        uses: ./.github/actions/cargo-llvm-cov
+        with:
+          name: integration-docker-in-docker
+
+      - name: Run Docker-in-Docker Test
+        env:
+          TARGET: aarch64-unknown-linux-gnu
+        run: ./ci/test-docker-in-docker.sh
+        shell: bash
+
   publish:
-    needs: [build, check]
+    needs: [build, check, fmt, clippy, cargo-deny, remote, bisect, docker-in-docker]
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v3
diff --git a/ci/shared.sh b/ci/shared.sh
new file mode 100755
index 0000000..457861d
--- /dev/null
+++ b/ci/shared.sh
@@ -0,0 +1,23 @@
+#!/usr/bin/env bash
+
+function retry {
+  local tries="${TRIES-5}"
+  local timeout="${TIMEOUT-1}"
+  local try=0
+  local exit_code=0
+
+  while (( try < tries )); do
+    if "${@}"; then
+      return 0
+    else
+      exit_code=$?
+    fi
+
+    sleep "${timeout}"
+    echo "Retrying ..." 1>&2
+    try=$(( try + 1 ))
+    timeout=$(( timeout * 2 ))
+  done
+
+  return ${exit_code}
+}
diff --git a/src/bin/commands/containers.rs b/src/bin/commands/containers.rs
index ffa3e86..a6a3712 100644
--- a/src/bin/commands/containers.rs
+++ b/src/bin/commands/containers.rs
@@ -482,6 +482,7 @@ pub fn remove_persistent_volume(
     engine: &docker::Engine,
     channel: Option<&str>,
 ) -> cross::Result<()> {
+    // we only need a triple that needs docker: the actual target doesn't matter.
     let msg_info = MessageInfo::create(verbose, quiet, color.as_deref())?;
     let triple = cross::Host::X86_64UnknownLinuxGnu.triple();
     let (_, _, dirs) =
diff --git a/src/docker/shared.rs b/src/docker/shared.rs
index 36893b2..0465ee4 100644
--- a/src/docker/shared.rs
+++ b/src/docker/shared.rs
@@ -763,6 +763,162 @@ mod tests {
         }
     }
 
+    mod directories {
+        use super::*;
+        use crate::cargo::cargo_metadata_with_args;
+        use crate::temp;
+
+        fn unset_env() -> Vec<(&'static str, Option<String>)> {
+            let mut result = vec![];
+            let envvars = ["CARGO_HOME", "XARGO_HOME", "NIX_STORE"];
+            for var in envvars {
+                result.push((var, env::var(var).ok()));
+                env::remove_var(var);
+            }
+
+            result
+        }
+
+        fn reset_env(vars: Vec<(&'static str, Option<String>)>) {
+            for (var, value) in vars {
+                if let Some(value) = value {
+                    env::set_var(var, value);
+                }
+            }
+        }
+
+        fn create_engine(msg_info: MessageInfo) -> Result<Engine> {
+            Engine::from_path(get_container_engine()?, Some(false), msg_info)
+        }
+
+        fn cargo_metadata(subdir: bool, msg_info: MessageInfo) -> Result<CargoMetadata> {
+            let mut metadata = cargo_metadata_with_args(
+                Some(Path::new(env!("CARGO_MANIFEST_DIR"))),
+                None,
+                msg_info,
+            )?
+            .ok_or_else(|| eyre::eyre!("could not find cross workspace"))?;
+
+            let root = match subdir {
+                true => get_cwd()?.join("member"),
+                false => get_cwd()?.parent().unwrap().to_path_buf(),
+            };
+            fs::create_dir_all(&root)?;
+            metadata.workspace_root = root;
+            metadata.target_directory = metadata.workspace_root.join("target");
+
+            Ok(metadata)
+        }
+
+        fn home() -> Result<PathBuf> {
+            home::home_dir().ok_or_else(|| eyre::eyre!("could not find home directory"))
+        }
+
+        fn get_cwd() -> Result<PathBuf> {
+            // we need this directory to exist for Windows
+            let path = temp::dir()?.join("Documents").join("package");
+            fs::create_dir_all(&path)?;
+            Ok(path)
+        }
+
+        fn get_sysroot() -> Result<PathBuf> {
+            Ok(home()?
+                .join(".rustup")
+                .join("toolchains")
+                .join("stable-x86_64-unknown-linux-gnu"))
+        }
+
+        fn get_directories(
+            metadata: &CargoMetadata,
+            mount_finder: &MountFinder,
+        ) -> Result<Directories> {
+            let cwd = get_cwd()?;
+            let sysroot = get_sysroot()?;
+            Directories::create(mount_finder, metadata, &cwd, &sysroot)
+        }
+
+        fn path_to_posix(path: &Path) -> Result<String> {
+            #[cfg(target_os = "windows")]
+            {
+                path.as_wslpath()
+            }
+            #[cfg(not(target_os = "windows"))]
+            {
+                path.as_posix()
+            }
+        }
+
+        #[track_caller]
+        fn paths_equal(x: &Path, y: &Path) -> Result<()> {
+            assert_eq!(path_to_posix(x)?, path_to_posix(y)?);
+            Ok(())
+        }
+
+        #[test]
+        fn test_host() -> Result<()> {
+            let vars = unset_env();
+            let mount_finder = MountFinder::new(vec![]);
+            let metadata = cargo_metadata(false, MessageInfo::default())?;
+            let directories = get_directories(&metadata, &mount_finder)?;
+            paths_equal(&directories.cargo, &home()?.join(".cargo"))?;
+            paths_equal(&directories.xargo, &home()?.join(".xargo"))?;
+            paths_equal(&directories.host_root, &metadata.workspace_root)?;
+            assert_eq!(
+                &directories.mount_root,
+                &path_to_posix(&metadata.workspace_root)?
+            );
+            assert_eq!(&directories.mount_cwd, &path_to_posix(&get_cwd()?)?);
+
+            reset_env(vars);
+            Ok(())
+        }
+
+        #[test]
+        #[cfg_attr(not(target_os = "linux"), ignore)]
+        fn test_docker_in_docker() -> Result<()> {
+            let vars = unset_env();
+
+            let engine = create_engine(MessageInfo::default());
+            let hostname = env::var("HOSTNAME");
+            if engine.is_err() || hostname.is_err() {
+                eprintln!("could not get container engine or no hostname found");
+                reset_env(vars);
+                return Ok(());
+            }
+            let engine = engine.unwrap();
+            let hostname = hostname.unwrap();
+            let output = subcommand(&engine, "inspect")
+                .arg(hostname)
+                .run_and_get_output(MessageInfo::default())?;
+            if !output.status.success() {
+                eprintln!("inspect failed");
+                reset_env(vars);
+                return Ok(());
+            }
+
+            let mount_finder = MountFinder::create(&engine, true)?;
+            let metadata = cargo_metadata(true, MessageInfo::default())?;
+            let directories = get_directories(&metadata, &mount_finder)?;
+            let mount_finder = MountFinder::new(docker_read_mount_paths(&engine)?);
+            let mount_path = |p| mount_finder.find_mount_path(p);
+
+            paths_equal(&directories.cargo, &mount_path(home()?.join(".cargo")))?;
+            paths_equal(&directories.xargo, &mount_path(home()?.join(".xargo")))?;
+            paths_equal(&directories.host_root, &mount_path(get_cwd()?))?;
+            assert_eq!(
+                &directories.mount_root,
+                &path_to_posix(&mount_path(get_cwd()?))?
+            );
+            assert_eq!(
+                &directories.mount_cwd,
+                &path_to_posix(&mount_path(get_cwd()?))?
+            );
+
+            reset_env(vars);
+            Ok(())
+        }
+    }
+
     mod mount_finder {
         use super::*;
 


Test patch:
diff --git a/ci/test-bisect.sh b/ci/test-bisect.sh
new file mode 100755
index 0000000..4db9a5a
--- /dev/null
+++ b/ci/test-bisect.sh
@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+# shellcheck disable=SC1091,SC1090
+
+# test to see that custom toolchains work
+
+set -x
+set -eo pipefail
+
+if [[ -z "${TARGET}" ]]; then
+    export TARGET="aarch64-unknown-linux-gnu"
+fi
+
+ci_dir=$(dirname "${BASH_SOURCE[0]}")
+ci_dir=$(realpath "${ci_dir}")
+. "${ci_dir}"/shared.sh
+project_home=$(dirname "${ci_dir}")
+
+main() {
+    local td=
+    local err=
+
+    retry cargo fetch
+    cargo build
+    cargo install cargo-bisect-rustc --debug
+    export CROSS="${project_home}/target/debug/cross"
+
+    td="$(mktemp -d)"
+    git clone --depth 1 https://github.com/cross-rs/rust-cpp-hello-word "${td}"
+
+    pushd "${td}"
+    retry cargo fetch
+    # shellcheck disable=SC2016
+    echo '#!/usr/bin/env bash
+export CROSS_CUSTOM_TOOLCHAIN=1
+exec "${CROSS}" run --target '"${TARGET}" > bisect.sh
+    chmod +x bisect.sh
+
+    if ! err=$(cargo bisect-rustc --script=./bisect.sh --target "${TARGET}" 2>&1 >/dev/null); then
+        if [[ "${err}" != *"does not reproduce the regression"* ]]; then
+            echo "${err}"
+            exit 1
+        fi
+    else
+        echo "should have failed, instead succeeded" 1>&2
+        exit 1
+    fi
+    popd
+
+    rm -rf "${td}"
+}
+
+main
diff --git a/ci/test-docker-in-docker.sh b/ci/test-docker-in-docker.sh
new file mode 100755
index 0000000..3057c68
--- /dev/null
+++ b/ci/test-docker-in-docker.sh
@@ -0,0 +1,56 @@
+#!/usr/bin/env bash
+# shellcheck disable=SC1004
+
+# test to see that running docker-in-docker works
+
+set -x
+set -eo pipefail
+
+if [[ -z "${TARGET}" ]]; then
+    export TARGET="aarch64-unknown-linux-gnu"
+fi
+
+source=$(dirname "${BASH_SOURCE[0]}")
+source=$(realpath "${source}")
+home=$(dirname "${source}")
+
+main() {
+    docker run -v "${home}":"${home}" -w "${home}" \
+        --rm -e TARGET -e RUSTFLAGS -e RUST_TEST_THREADS \
+        -e LLVM_PROFILE_FILE -e CARGO_INCREMENTAL \
+        -v /var/run/docker.sock:/var/run/docker.sock \
+        docker:18.09-dind sh -c '
+#!/usr/bin/env sh
+set -x
+set -euo pipefail
+
+apk add curl
+curl --proto "=https" --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+source "${HOME}/.cargo/env"
+
+# building on release is slow
+apk add libgcc gcc musl-dev
+cargo test --workspace
+cargo install --path . --force --debug
+
+export CROSS_CONTAINER_IN_CONTAINER=1
+
+apk add git
+td="$(mktemp -d)"
+git clone --depth 1 https://github.com/cross-rs/rust-cpp-hello-word "${td}"
+cd "${td}"
+cross run --target "${TARGET}" --verbose
+
+td="$(mktemp -d)"
+git clone --depth 1 https://github.com/cross-rs/test-workspace "${td}"
+cd "${td}"
+cross build --target "${TARGET}" --workspace \
+    --manifest-path="./workspace/Cargo.toml" --verbose
+cd workspace
+cross build --target "${TARGET}" --workspace --verbose
+cd binary
+cross run --target "${TARGET}" --verbose
+'
+}
+
+main
diff --git a/ci/test-remote.sh b/ci/test-remote.sh
new file mode 100755
index 0000000..8fb3583
--- /dev/null
+++ b/ci/test-remote.sh
@@ -0,0 +1,55 @@
+#!/usr/bin/env bash
+# shellcheck disable=SC1091,SC1090
+
+# test to see that remote docker support works.
+
+set -x
+set -eo pipefail
+
+export CROSS_REMOTE=1
+if [[ -z "${TARGET}" ]]; then
+    export TARGET="aarch64-unknown-linux-gnu"
+fi
+
+ci_dir=$(dirname "${BASH_SOURCE[0]}")
+ci_dir=$(realpath "${ci_dir}")
+. "${ci_dir}"/shared.sh
+project_home=$(dirname "${ci_dir}")
+
+main() {
+    local err=
+
+    retry cargo fetch
+    cargo build
+    export CROSS="${project_home}/target/debug/cross"
+    export CROSS_UTIL="${project_home}/target/debug/cross-util"
+
+    # if the create volume fails, ensure it exists.
+    if ! err=$("${CROSS_UTIL}" volumes create 2>&1 >/dev/null); then
+        if [[ "${err}" != *"already exists"* ]]; then
+            echo "${err}"
+            exit 1
+        fi
+    fi
+    cross_test_cpp
+    "${CROSS_UTIL}" volumes remove
+
+    # ensure the data volume was removed.
+    cross_test_cpp
+}
+
+cross_test_cpp() {
+    local td=
+    td="$(mktemp -d)"
+
+    git clone --depth 1 https://github.com/cross-rs/rust-cpp-hello-word "${td}"
+
+    pushd "${td}"
+    retry cargo fetch
+    "${CROSS}" run --target "${TARGET}"
+    popd
+
+    rm -rf "${td}"
+}
+
+main
diff --git a/ci/test.sh b/ci/test.sh
index 4b9b494..fbfadda 100755
--- a/ci/test.sh
+++ b/ci/test.sh
@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# shellcheck disable=SC2086
+# shellcheck disable=SC2086,SC1091,SC1090
 
 set -x
 set -euo pipefail
@@ -10,30 +10,9 @@ set -euo pipefail
 
 ci_dir=$(dirname "${BASH_SOURCE[0]}")
 ci_dir=$(realpath "${ci_dir}")
+. "${ci_dir}"/shared.sh
 project_home=$(dirname "${ci_dir}")
 
-function retry {
-  local tries="${TRIES-5}"
-  local timeout="${TIMEOUT-1}"
-  local try=0
-  local exit_code=0
-
-  while (( try < tries )); do
-    if "${@}"; then
-      return 0
-    else
-      exit_code=$?
-    fi
-
-    sleep "${timeout}"
-    echo "Retrying ..." 1>&2
-    try=$(( try + 1 ))
-    timeout=$(( timeout * 2 ))
-  done
-
-  return ${exit_code}
-}
-
 workspace_test() {
   "${CROSS[@]}" build --target "${TARGET}" --workspace "$@" ${CROSS_FLAGS}
   "${CROSS[@]}" run --target "${TARGET}" -p binary "$@" ${CROSS_FLAGS}