Repository: dragonflyoss/Dragonfly2
Pull request #1860 (2022-11-23 08:01:50)

chore: add e2e with nydus snapshotter


<!--- Provide a general summary of your changes in the Title above -->

## Description
Add a new GitHub actions that run dragonfly + nydus in e2e test.

<!--- Describe your changes in detail -->

## Related Issue

<!--- This project only accepts pull requests related to open issues -->
<!--- If suggesting a new feature or change, please discuss it in an issue first -->
<!--- If fixing a bug, there should be an issue describing it with steps to reproduce -->
<!--- Please link to the issue here: -->

## Motivation and Context

<!--- Why is this change required? What problem does it solve? -->

## Screenshots (if appropriate)

## Types of changes

<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->

- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Documentation Update (if none of the other choices apply)

## Checklist

<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->

- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the **CONTRIBUTING** document.
- [x] I have added tests to cover my changes.


codecov-commenter: # [Codecov](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1860?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) Report
> Merging [#1860](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1860?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) (f15da68) into [main](https://codecov.io/gh/dragonflyoss/Dragonfly2/commit/c0abbf131b650b4242accba3a5c7971b77d5d6d8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) (c0abbf1) will **decrease** coverage by `0.03%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1860/graphs/tree.svg?width=650&height=150&src=pr&token=PUICJYSAT7&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss)](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1860?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss)

```diff
@@            Coverage Diff             @@
##             main    #1860      +/-   ##
==========================================
- Coverage   42.87%   42.84%   -0.04%     
==========================================
  Files         131      131              
  Lines       16601    16601              
==========================================
- Hits         7117     7112       -5     
- Misses       8944     8951       +7     
+ Partials      540      538       -2     
```

| Flag | Coverage Δ | |
|---|---|---|
| Object-compatibility-e2etests | `?` | |
| e2etests | `?` | |
| unittests | `42.84% <ø> (-0.04%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1860?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss) | Coverage Δ | |
|---|---|---|
| [client/daemon/peer/piece\_manager.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1860/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wZWVyL3BpZWNlX21hbmFnZXIuZ28=) | `52.99% <0.00%> (-0.41%)` | :arrow_down: |
| [client/daemon/peer/peertask\_conductor.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1860/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wZWVyL3BlZXJ0YXNrX2NvbmR1Y3Rvci5nbw==) | `56.67% <0.00%> (-0.37%)` | :arrow_down: |
| [...ent/daemon/peer/peertask\_piecetask\_synchronizer.go](https://codecov.io/gh/dragonflyoss/Dragonfly2/pull/1860/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dragonflyoss#diff-Y2xpZW50L2RhZW1vbi9wZWVyL3BlZXJ0YXNrX3BpZWNldGFza19zeW5jaHJvbml6ZXIuZ28=) | `50.45% <0.00%> (+0.91%)` | :arrow_up: |


Patch:
diff --git a/deploy/helm-charts b/deploy/helm-charts
index 6d1eb77f..2bbf9d31 160000
--- a/deploy/helm-charts
+++ b/deploy/helm-charts
@@ -1,1 +1,1 @@
-Subproject commit 6d1eb77fdc69b68167c7cbd15ca972f79eb811f5
+Subproject commit 2bbf9d314947b907bd3bc3fdde27ede7ee94bde6


Test patch:
diff --git a/.github/workflows/e2e-nydus.yml b/.github/workflows/e2e-nydus.yml
new file mode 100644
index 00000000..57f3b801
--- /dev/null
+++ b/.github/workflows/e2e-nydus.yml
@@ -0,0 +1,111 @@
+name: E2E Test with Nydus
+
+on:
+  push:
+    branches: [main, release-*]
+    paths-ignore: ["**.md", "**.png", "**.jpg", "**.svg", "**/docs/**"]
+  pull_request:
+    branches: [main, release-*]
+    paths-ignore: ["**.md", "**.png", "**.jpg", "**.svg", "**/docs/**"]
+  schedule:
+    - cron: '0 4 * * *'
+
+env:
+  GO_VERSION: 1.19
+  KIND_VERSION: v0.12.0
+  CONTAINERD_VERSION: v1.5.2
+  NERDCTL_VER: 0.22.2
+  KIND_CONFIG_PATH: test/testdata/kind/config.yaml
+  DRAGONFLY_CHARTS_PATH: deploy/helm-charts/charts/dragonfly
+  NYDUS_SNAPSHOTTER_CHARTS_PATH: deploy/helm-charts/charts/nydus-snapshotter
+
+jobs:
+  e2e_tests_nydus:
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v3
+        with:
+          submodules: recursive
+
+      - name: Install Go
+        uses: actions/setup-go@v3
+        with:
+          go-version: ${{ env.GO_VERSION }}
+
+      - name: Get dependencies
+        run: |
+          go mod download github.com/go-task/slim-sprig
+          go install github.com/onsi/ginkgo/v2/ginkgo@v2.1.0
+          mkdir -p /tmp/artifact
+
+      - name: Setup buildx
+        uses: docker/setup-buildx-action@v2
+        id: buildx
+        with:
+          install: true
+
+      - name: Setup Kind
+        uses: engineerd/setup-kind@v0.5.0
+        with:
+          version: ${{ env.KIND_VERSION }}
+          config: ${{ env.KIND_CONFIG_PATH }}
+
+      - name: Build images
+        run: |
+          hack/gen-buildx.sh
+          make docker-build docker-build-testing-tools
+          make kind-load
+
+      - name: Setup dragonfly
+        run: |
+          helm install --wait --timeout 10m --dependency-update \
+            --create-namespace --namespace dragonfly-system \
+            -f test/testdata/charts/config.yaml \
+            dragonfly ${{ env.DRAGONFLY_CHARTS_PATH }}
+
+      - name: Setup nydus
+        run: |
+          helm install --wait --timeout 10m --dependency-update \
+            --create-namespace --namespace nydus-system \
+            -f test/testdata/charts/config-nydus.yaml \
+            nydus-snapshotter ${{ env.NYDUS_SNAPSHOTTER_CHARTS_PATH }}
+
+          export ns=nydus-system
+          p=`kubectl -n $ns get pods --no-headers -o custom-columns=NAME:metadata.name`
+          echo "snapshotter pod name ${p}"
+          # helm wait is wait for daemonset, in this e2e test we ensure the pod is ready
+          kubectl -n $ns wait po $p --for=condition=ready --timeout=2m
+      - name: Run E2E test
+        run: |
+          docker exec kind-control-plane curl -L -o nerdctl-${NERDCTL_VER}-linux-amd64.tar.gz https://github.com/containerd/nerdctl/releases/download/v${NERDCTL_VER}/nerdctl-${NERDCTL_VER}-linux-amd64.tar.gz
+          docker exec kind-control-plane tar xzf nerdctl-${NERDCTL_VER}-linux-amd64.tar.gz
+          docker exec kind-control-plane install -D -m 755 nerdctl /usr/local/bin/nerdctl
+
+          # this is only a simple test that run `date` in container
+          docker exec kind-control-plane /usr/local/bin/nerdctl run --snapshotter nydus --network=none ghcr.io/dragonflyoss/image-service/nginx:nydus-latest date
+
+          kubectl apply -f test/testdata/k8s/nydus.yaml
+          kubectl wait po nydus-pod --for=condition=ready --timeout=1m
+          kubectl delete -f test/testdata/k8s/nydus.yaml
+      - name: Dump logs
+        if: ${{ failure() }}
+        continue-on-error: true
+        run: |
+          export ns=nydus-system
+          echo "============ debug nydus pods: ============"
+          for p in `kubectl -n $ns get pods --no-headers -o custom-columns=NAME:metadata.name`; do kubectl -n $ns get pod $p -o yaml; done
+          for p in `kubectl -n $ns get pods --no-headers -o custom-columns=NAME:metadata.name`; do kubectl -n $ns describe pod $p; done
+
+          echo "============ debug kind containerd config ============"
+          docker exec kind-control-plane cat /etc/containerd/config.toml
+          docker exec kind-control-plane containerd config dump
+          echo "============ debug kind containerd log ============"
+          docker exec kind-control-plane journalctl -u containerd
+
+          echo "============ debug nydus snapshotter log: ============"
+          for p in `kubectl -n $ns get pods --no-headers -o custom-columns=NAME:metadata.name`; do kubectl -n $ns logs $p -c nydus-snapshotter || echo "failed to get snapshotter log"; done
+
+          echo "============ debug kind kubelet log ============"
+          docker exec kind-control-plane journalctl -u kubelet
diff --git a/test/testdata/charts/config-nydus.yaml b/test/testdata/charts/config-nydus.yaml
new file mode 100644
index 00000000..b2e34b0b
--- /dev/null
+++ b/test/testdata/charts/config-nydus.yaml
@@ -0,0 +1,26 @@
+name: nydus-snapshotter
+pullPolicy: Always
+hostNetwork: true
+resources:
+  requests:
+    cpu: "0"
+    memory: "0"
+  limits:
+    cpu: "1"
+    memory: "1Gi"
+
+dragonfly:
+  enable: true
+  mirrorConfig:
+    host: "http://127.0.0.1:65001"
+    auth_through: false
+    headers:
+      "X-Dragonfly-Registry": "https://ghcr.io"
+    ping_url: "http://127.0.0.1:40901/server/ping"
+
+
+containerRuntime:
+  initContainerImage: ghcr.io/liubin/toml-cli:v0.0.7
+  containerd:
+    enable: true
+    configFile: "/etc/containerd/config.toml"
diff --git a/test/testdata/charts/config.yaml b/test/testdata/charts/config.yaml
index 6a9766d9..7f009a40 100644
--- a/test/testdata/charts/config.yaml
+++ b/test/testdata/charts/config.yaml
@@ -119,3 +119,7 @@ manager:
       path: /tmp/artifact
   config:
     verbose: true
+
+containerRuntime:
+  containerd:
+    enable: true
diff --git a/test/testdata/containerd/config.toml b/test/testdata/containerd/config.toml
index 4b4ccf48..75315670 100644
--- a/test/testdata/containerd/config.toml
+++ b/test/testdata/containerd/config.toml
@@ -6,7 +6,7 @@ version = 2
 
 [plugins."io.containerd.grpc.v1.cri".containerd]
   # save disk space when using a single snapshotter
-  discard_unpacked_layers = true
+  discard_unpacked_layers = false
   # explicitly use default snapshotter so we can sed it in entrypoint
   snapshotter = "overlayfs"
   # explicit default here, as we're configuring it below
diff --git a/test/testdata/k8s/nydus.yaml b/test/testdata/k8s/nydus.yaml
new file mode 100644
index 00000000..aa3762dc
--- /dev/null
+++ b/test/testdata/k8s/nydus.yaml
@@ -0,0 +1,12 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  name: nydus-pod
+spec:
+  containers:
+    - name: nginx
+      image: ghcr.io/dragonflyoss/image-service/nginx:nydus-latest
+      imagePullPolicy: Always
+      command: ["sh", "-c"]
+      args:
+        - tail -f /dev/null
diff --git a/test/testdata/kind/config.yaml b/test/testdata/kind/config.yaml
index a865ce68..c95f5de4 100644
--- a/test/testdata/kind/config.yaml
+++ b/test/testdata/kind/config.yaml
@@ -14,3 +14,5 @@ nodes:
         containerPath: /etc/containerd/config.toml
       - hostPath: /tmp/artifact
         containerPath: /tmp/artifact
+      - hostPath: /dev/fuse
+        containerPath: /dev/fuse