Repository: moby/moby (language: go)

Issue:
Flaky test: TestAPISwarmServicesMultipleAgents (ppc64le)
Currently seen (sometimes) on ppc64le. 

``` sh
----------------------------------------------------------------------
09:59:37 FAIL: docker_api_swarm_test.go:228: DockerSwarmSuite.TestApiSwarmServicesMultipleAgents
09:59:37 
09:59:37 [d25244000] waiting for daemon to start
09:59:37 [d25244000] daemon started
09:59:37 [d57794000] waiting for daemon to start
09:59:37 [d57794000] daemon started
09:59:37 [d84297000] waiting for daemon to start
09:59:37 [d84297000] daemon started
09:59:37 timeout: d57794000
09:59:37 Attempt #2: daemon is still running with pid 26843
09:59:37 Attempt #3: daemon is still running with pid 26843
09:59:37 Attempt #4: daemon is still running with pid 26843
09:59:37 Attempt #5: daemon is still running with pid 26843
09:59:37 tried to interrupt daemon for 6 times, now try to kill it
09:59:37 docker_api_swarm_test.go:245:
09:59:37     // reconciliation on d2 node down
09:59:37     c.Assert(d2.Stop(), checker.IsNil)
09:59:37 ... value *os.PathError = &os.PathError{Op:"remove", Path:"/go/src/github.com/docker/docker/bundles/1.12.0-dev/test-integration-cli/d57794000/docker.pid", Err:0x2} ("remove /go/src/github.com/docker/docker/bundles/1.12.0-dev/test-integration-cli/d57794000/docker.pid: no such file or directory")
09:59:37 
09:59:37 [d25244000] exiting daemon
09:59:37 [d84297000] exiting daemon
10:00:05 
10:00:05 ----------------------------------------------------------------------
```



Discussion / hints:
Closing this because it hasn't failed since the last swarmkit vendor #23707.

Opening this up again because I've seen it failing again on Docker Master powerpc. Error looks the same.

```sh
20:06:48 FAIL: docker_api_swarm_test.go:222: DockerSwarmSuite.TestAPISwarmServicesMultipleAgents
20:06:48 
20:06:48 [dbc331c396772] waiting for daemon to start
20:06:48 [dbc331c396772] daemon started
20:06:48 
20:06:48 [dcbe4346b30b4] waiting for daemon to start
20:06:48 [dcbe4346b30b4] daemon started
20:06:48 
20:06:48 [de5cb413fbb9d] waiting for daemon to start
20:06:48 [de5cb413fbb9d] daemon started
20:06:48 
20:06:48 [dcbe4346b30b4] daemon started
20:06:48 Attempt #2: daemon is still running with pid 5482
20:06:48 Attempt #3: daemon is still running with pid 5482
20:06:48 Attempt #4: daemon is still running with pid 5482
20:06:48 Attempt #5: daemon is still running with pid 5482
20:06:48 tried to interrupt daemon for 6 times, now try to kill it
20:06:48 docker_api_swarm_test.go:239:
20:06:48     // reconciliation on d2 node down
20:06:48     d2.Stop(c)
20:06:48 daemon/daemon.go:390:
20:06:48     t.Fatalf("Error while stopping the daemon %s : %v", d.id, err)
20:06:48 ... Error: Error while stopping the daemon dcbe4346b30b4 : remove /go/src/github.com/docker/docker/bundles/1.14.0-dev/test-integration-cli/dcbe4346b30b4/docker.pid: no such file or directory
20:06:48 
20:06:48 [dbc331c396772] exiting daemon
20:06:48 [de5cb413fbb9d] exiting daemon
20:07:04 
20:07:04 ----------------------------------------------------------------------
```

ping @thaJeztah for labels

closing again because I haven't seen this in a while
Seeing this fail again a couple of times (in the docker/engine repo); e.g. https://github.com/docker/engine/pull/113
Flaky test was renamed from `TestApiSwarmServicesMultipleAgents ` to `TestAPISwarmServicesMultipleAgents ` with PR #26967 git commit 7fb7a47

Still looks like flakiness is an issue https://github.com/moby/moby/search?q=TestAPISwarmServicesMultipleAgents&type=Issues
Seen failing again on s390x _twice_  https://github.com/moby/moby/pull/39710 (https://github.com/moby/moby/pull/39710#issuecomment-520024978)

```
19:49:57  FAIL: docker_api_swarm_service_test.go:96: DockerSwarmSuite.TestAPISwarmServicesMultipleAgents
19:49:57  
19:49:57  Creating a new daemon
19:49:57  [d4d05ad981139] waiting for daemon to start
19:49:57  [d4d05ad981139] waiting for daemon to start
19:49:57  [d4d05ad981139] daemon started
19:49:57  
19:49:57  Creating a new daemon
19:49:57  [d12b046fb2579] waiting for daemon to start
19:49:57  [d12b046fb2579] waiting for daemon to start
19:49:57  [d12b046fb2579] daemon started
19:49:57  
19:49:57  [d12b046fb2579] joining swarm manager [d4d05ad981139]@0.0.0.0:2477, swarm listen addr 0.0.0.0:2478
19:49:57  Creating a new daemon
19:49:57  [d300aa72b91d4] waiting for daemon to start
19:49:57  [d300aa72b91d4] waiting for daemon to start
19:49:57  [d300aa72b91d4] daemon started
19:49:57  
19:49:57  [d300aa72b91d4] joining swarm manager [d4d05ad981139]@0.0.0.0:2477, swarm listen addr 0.0.0.0:2479
19:49:57  waited for 1.577051822s (out of 30s)
19:49:57  waited for 11.219922ms (out of 30s)
19:49:57  waited for 14.548776ms (out of 30s)
19:49:57  waited for 488.307601ms (out of 30s)
19:49:57  [d12b046fb2579] Stopping daemon
19:49:57  [d12b046fb2579] exiting daemon
19:49:57  [d12b046fb2579] Daemon stopped
19:49:57  waited for 6.438570617s (out of 30s)
19:49:57  docker_api_swarm_service_test.go:120:
19:49:57      waitAndAssert(c, defaultReconciliationTimeout, reducedCheck(sumAsIntegers, d1.CheckActiveContainerCount, d3.CheckActiveContainerCount), checker.Equals, instances)
19:49:57  docker_utils_test.go:435:
19:49:57      c.Assert(v, checker, args...)
19:49:57  ... obtained int = 6
19:49:57  ... expected int = 5
19:49:57  ... output: "7f0fea7cf90a\n1112f21745a1\n33f03ffbbe5d\n", output: "a8410e7b6cbb\nceb51a4b9e83\nec644145951e\n"
19:49:57  
19:49:57  waited for 30.099112756s (out of 30s)
19:49:57  [d4d05ad981139] Stopping daemon
19:49:57  [d4d05ad981139] exiting daemon
19:49:57  [d4d05ad981139] Daemon stopped
19:49:57  [d300aa72b91d4] Stopping daemon
19:49:57  [d300aa72b91d4] exiting daemon
19:49:57  [d300aa72b91d4] Daemon stopped
```

Errors from those daemons:

[d4d05ad981139.log](https://github.com/moby/moby/files/3487668/d4d05ad981139.log)

```
time="2019-08-09T17:48:57.933417645Z" level=error msg="error reading the kernel parameter net.ipv4.neigh.default.gc_thresh1" error="open /proc/sys/net/ipv4/neigh/default/gc_thresh1: no such file or directory"
time="2019-08-09T17:48:57.933436407Z" level=error msg="error reading the kernel parameter net.ipv4.neigh.default.gc_thresh2" error="open /proc/sys/net/ipv4/neigh/default/gc_thresh2: no such file or directory"
time="2019-08-09T17:48:57.933461332Z" level=error msg="error reading the kernel parameter net.ipv4.neigh.default.gc_thresh3" error="open /proc/sys/net/ipv4/neigh/default/gc_thresh3: no such file or directory"
time="2019-08-09T17:48:58.101467482Z" level=error msg="error reading the kernel parameter net.ipv4.vs.expire_nodest_conn" error="open /proc/sys/net/ipv4/vs/expire_nodest_conn: no such file or directory"
time="2019-08-09T17:49:15.007089589Z" level=error msg="failed to update task to REMOVE" error="update out of sequence" module=node node.id=0yarrof2luuo71gkgk3nqhxr1
time="2019-08-09T17:49:45.174094451Z" level=error msg="failed to remove node" error="rpc error: code = Aborted desc = dispatcher is stopped" method="(*Dispatcher).Session" node.id=0yarrof2luuo71gkgk3nqhxr1 node.session=xojyt7pfga4c0rv2tq4odq815
time="2019-08-09T17:49:45.174519751Z" level=error msg="failed to remove node" error="rpc error: code = Aborted desc = dispatcher is stopped" method="(*Dispatcher).Session" node.id=76ojejsxxp7mdnl44cfkwgtj8 node.session=kdl7v7rqug63xv9l049kazuhb
time="2019-08-09T17:49:45.175082951Z" level=error msg="failed to receive changes from store watch API" error="rpc error: code = Unknown desc = context canceled"
```

[d12b046fb2579.log](https://github.com/moby/moby/files/3487672/d12b046fb2579.log)

```
time="2019-08-09T17:48:58.937430090Z" level=error msg="error in agentInit: failed to create memberlist: Could not set up network transport: failed to obtain an address: Failed to start TCP listener on \"0.0.0.0\" port 7946: listen tcp 0.0.0.0:7946: bind: address already in use"
time="2019-08-09T17:48:58.937533028Z" level=error msg="error in agentInit: failed to create memberlist: Could not set up network transport: failed to obtain an address: Failed to start TCP listener on \"0.0.0.0\" port 7946: listen tcp 0.0.0.0:7946: bind: address already in use"
```

[d300aa72b91d4.log](https://github.com/moby/moby/files/3487673/d300aa72b91d4.log)


```
time="2019-08-09T17:48:59.841089225Z" level=error msg="error in agentInit: failed to create memberlist: Could not set up network transport: failed to obtain an address: Failed to start TCP listener on \"0.0.0.0\" port 7946: listen tcp 0.0.0.0:7946: bind: address already in use"
time="2019-08-09T17:49:45.175557220Z" level=error msg="agent: session failed" backoff=100ms error=EOF module=node/agent node.id=76ojejsxxp7mdnl44cfkwgtj8
time="2019-08-09T17:49:45.263181127Z" level=error msg="agent: session failed" backoff=300ms error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2477: connect: connection refused\"" module=node/agent node.id=76ojejsxxp7mdnl44cfkwgtj8
time="2019-08-09T17:49:45.504284729Z" level=error msg="agent: session failed" backoff=700ms error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2477: connect: connection refused\"" module=node/agent node.id=76ojejsxxp7mdnl44cfkwgtj8
time="2019-08-09T17:49:45.807404089Z" level=error msg="agent: session failed" backoff=1.5s error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2477: connect: connection refused\"" module=node/agent node.id=76ojejsxxp7mdnl44cfkwgtj8
time="2019-08-09T17:49:46.069788485Z" level=error msg="agent: session failed" backoff=3.1s error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2477: connect: connection refused\"" module=node/agent node.id=76ojejsxxp7mdnl44cfkwgtj8
```

Fix patch:
diff --git a/Jenkinsfile b/Jenkinsfile
index 6707839168792..d93a24a89f08e 100644
--- a/Jenkinsfile
+++ b/Jenkinsfile
@@ -9,8 +9,6 @@ pipeline {
     }
     parameters {
         booleanParam(name: 'arm64', defaultValue: true, description: 'ARM (arm64) Build/Test')
-        booleanParam(name: 's390x', defaultValue: false, description: 'IBM Z (s390x) Build/Test')
-        booleanParam(name: 'ppc64le', defaultValue: false, description: 'PowerPC (ppc64le) Build/Test')
         booleanParam(name: 'dco', defaultValue: true, description: 'Run the DCO check')
     }
     environment {
@@ -51,406 +49,6 @@ pipeline {
         }
         stage('Build') {
             parallel {
-                stage('s390x') {
-                    when {
-                        beforeAgent true
-                        // Skip this stage on PRs unless the checkbox is selected
-                        anyOf {
-                            not { changeRequest() }
-                            expression { params.s390x }
-                        }
-                    }
-                    agent { label 's390x-ubuntu-2004' }
-
-                    stages {
-                        stage("Print info") {
-                            steps {
-                                sh 'docker version'
-                                sh 'docker info'
-                                sh '''
-                                echo "check-config.sh version: ${CHECK_CONFIG_COMMIT}"
-                                curl -fsSL -o ${WORKSPACE}/check-config.sh "https://raw.githubusercontent.com/moby/moby/${CHECK_CONFIG_COMMIT}/contrib/check-config.sh" \
-                                && bash ${WORKSPACE}/check-config.sh || true
-                                '''
-                            }
-                        }
-                        stage("Build dev image") {
-                            steps {
-                                sh '''
-                                docker build --force-rm -t docker:${GIT_COMMIT} .
-                                '''
-                            }
-                        }
-                        stage("Unit tests") {
-                            steps {
-                                sh '''
-                                sudo modprobe ip6table_filter
-                                '''
-                                sh '''
-                                docker run --rm -t --privileged \
-                                  -v "$WORKSPACE/bundles:/go/src/github.com/docker/docker/bundles" \
-                                  --name docker-pr$BUILD_NUMBER \
-                                  -e DOCKER_EXPERIMENTAL \
-                                  -e DOCKER_GITCOMMIT=${GIT_COMMIT} \
-                                  -e DOCKER_GRAPHDRIVER \
-                                  -e VALIDATE_REPO=${GIT_URL} \
-                                  -e VALIDATE_BRANCH=${CHANGE_TARGET} \
-                                  docker:${GIT_COMMIT} \
-                                  hack/test/unit
-                                '''
-                            }
-                            post {
-                                always {
-                                    junit testResults: 'bundles/junit-report*.xml', allowEmptyResults: true
-                                }
-                            }
-                        }
-                        stage("Integration tests") {
-                            environment { TEST_SKIP_INTEGRATION_CLI = '1' }
-                            steps {
-                                sh '''
-                                docker run --rm -t --privileged \
-                                  -v "$WORKSPACE/bundles:/go/src/github.com/docker/docker/bundles" \
-                                  --name docker-pr$BUILD_NUMBER \
-                                  -e DOCKER_EXPERIMENTAL \
-                                  -e DOCKER_GITCOMMIT=${GIT_COMMIT} \
-                                  -e DOCKER_GRAPHDRIVER \
-                                  -e TESTDEBUG \
-                                  -e TEST_INTEGRATION_USE_SNAPSHOTTER \
-                                  -e TEST_SKIP_INTEGRATION_CLI \
-                                  -e TIMEOUT \
-                                  -e VALIDATE_REPO=${GIT_URL} \
-                                  -e VALIDATE_BRANCH=${CHANGE_TARGET} \
-                                  docker:${GIT_COMMIT} \
-                                  hack/make.sh \
-                                    dynbinary \
-                                    test-integration
-                                '''
-                            }
-                            post {
-                                always {
-                                    junit testResults: 'bundles/**/*-report.xml', allowEmptyResults: true
-                                }
-                            }
-                        }
-                    }
-
-                    post {
-                        always {
-                            sh '''
-                            echo "Ensuring container killed."
-                            docker rm -vf docker-pr$BUILD_NUMBER || true
-                            '''
-
-                            sh '''
-                            echo "Chowning /workspace to jenkins user"
-                            docker run --rm -v "$WORKSPACE:/workspace" busybox chown -R "$(id -u):$(id -g)" /workspace
-                            '''
-
-                            catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE', message: 'Failed to create bundles.tar.gz') {
-                                sh '''
-                                bundleName=s390x-integration
-                                echo "Creating ${bundleName}-bundles.tar.gz"
-                                # exclude overlay2 directories
-                                find bundles -path '*/root/*overlay2' -prune -o -type f \\( -name '*-report.json' -o -name '*.log' -o -name '*.prof' -o -name '*-report.xml' \\) -print | xargs tar -czf ${bundleName}-bundles.tar.gz
-                                '''
-
-                                archiveArtifacts artifacts: '*-bundles.tar.gz', allowEmptyArchive: true
-                            }
-                        }
-                        cleanup {
-                            sh 'make clean'
-                            deleteDir()
-                        }
-                    }
-                }
-                stage('s390x integration-cli') {
-                    when {
-                        beforeAgent true
-                        // Skip this stage on PRs unless the checkbox is selected
-                        anyOf {
-                            not { changeRequest() }
-                            expression { params.s390x }
-                        }
-                    }
-                    agent { label 's390x-ubuntu-2004' }
-
-                    stages {
-                        stage("Print info") {
-                            steps {
-                                sh 'docker version'
-                                sh 'docker info'
-                                sh '''
-                                echo "check-config.sh version: ${CHECK_CONFIG_COMMIT}"
-                                curl -fsSL -o ${WORKSPACE}/check-config.sh "https://raw.githubusercontent.com/moby/moby/${CHECK_CONFIG_COMMIT}/contrib/check-config.sh" \
-                                && bash ${WORKSPACE}/check-config.sh || true
-                                '''
-                            }
-                        }
-                        stage("Build dev image") {
-                            steps {
-                                sh '''
-                                docker build --force-rm -t docker:${GIT_COMMIT} .
-                                '''
-                            }
-                        }
-                        stage("Integration-cli tests") {
-                            environment { TEST_SKIP_INTEGRATION = '1' }
-                            steps {
-                                sh '''
-                                docker run --rm -t --privileged \
-                                  -v "$WORKSPACE/bundles:/go/src/github.com/docker/docker/bundles" \
-                                  --name docker-pr$BUILD_NUMBER \
-                                  -e DOCKER_GITCOMMIT=${GIT_COMMIT} \
-                                  -e DOCKER_GRAPHDRIVER \
-                                  -e TEST_INTEGRATION_USE_SNAPSHOTTER \
-                                  -e TEST_SKIP_INTEGRATION \
-                                  -e TIMEOUT \
-                                  -e VALIDATE_REPO=${GIT_URL} \
-                                  -e VALIDATE_BRANCH=${CHANGE_TARGET} \
-                                  docker:${GIT_COMMIT} \
-                                  hack/make.sh \
-                                    dynbinary \
-                                    test-integration
-                                '''
-                            }
-                            post {
-                                always {
-                                    junit testResults: 'bundles/**/*-report.xml', allowEmptyResults: true
-                                }
-                            }
-                        }
-                    }
-
-                    post {
-                        always {
-                            sh '''
-                            echo "Ensuring container killed."
-                            docker rm -vf docker-pr$BUILD_NUMBER || true
-                            '''
-
-                            sh '''
-                            echo "Chowning /workspace to jenkins user"
-                            docker run --rm -v "$WORKSPACE:/workspace" busybox chown -R "$(id -u):$(id -g)" /workspace
-                            '''
-
-                            catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE', message: 'Failed to create bundles.tar.gz') {
-                                sh '''
-                                bundleName=s390x-integration-cli
-                                echo "Creating ${bundleName}-bundles.tar.gz"
-                                # exclude overlay2 directories
-                                find bundles -path '*/root/*overlay2' -prune -o -type f \\( -name '*-report.json' -o -name '*.log' -o -name '*.prof' -o -name '*-report.xml' \\) -print | xargs tar -czf ${bundleName}-bundles.tar.gz
-                                '''
-
-                                archiveArtifacts artifacts: '*-bundles.tar.gz', allowEmptyArchive: true
-                            }
-                        }
-                        cleanup {
-                            sh 'make clean'
-                            deleteDir()
-                        }
-                    }
-                }
-                stage('ppc64le') {
-                    when {
-                        beforeAgent true
-                        // Skip this stage on PRs unless the checkbox is selected
-                        anyOf {
-                            not { changeRequest() }
-                            expression { params.ppc64le }
-                        }
-                    }
-                    agent { label 'ppc64le-ubuntu-1604' }
-
-                    stages {
-                        stage("Print info") {
-                            steps {
-                                sh 'docker version'
-                                sh 'docker info'
-                                sh '''
-                                echo "check-config.sh version: ${CHECK_CONFIG_COMMIT}"
-                                curl -fsSL -o ${WORKSPACE}/check-config.sh "https://raw.githubusercontent.com/moby/moby/${CHECK_CONFIG_COMMIT}/contrib/check-config.sh" \
-                                && bash ${WORKSPACE}/check-config.sh || true
-                                '''
-                            }
-                        }
-                        stage("Build dev image") {
-                            steps {
-                                sh '''
-                                docker buildx build --load --force-rm -t docker:${GIT_COMMIT} .
-                                '''
-                            }
-                        }
-                        stage("Unit tests") {
-                            steps {
-                                sh '''
-                                sudo modprobe ip6table_filter
-                                '''
-                                sh '''
-                                docker run --rm -t --privileged \
-                                  -v "$WORKSPACE/bundles:/go/src/github.com/docker/docker/bundles" \
-                                  --name docker-pr$BUILD_NUMBER \
-                                  -e DOCKER_EXPERIMENTAL \
-                                  -e DOCKER_GITCOMMIT=${GIT_COMMIT} \
-                                  -e DOCKER_GRAPHDRIVER \
-                                  -e VALIDATE_REPO=${GIT_URL} \
-                                  -e VALIDATE_BRANCH=${CHANGE_TARGET} \
-                                  docker:${GIT_COMMIT} \
-                                  hack/test/unit
-                                '''
-                            }
-                            post {
-                                always {
-                                    junit testResults: 'bundles/junit-report*.xml', allowEmptyResults: true
-                                }
-                            }
-                        }
-                        stage("Integration tests") {
-                            environment { TEST_SKIP_INTEGRATION_CLI = '1' }
-                            steps {
-                                sh '''
-                                docker run --rm -t --privileged \
-                                  -v "$WORKSPACE/bundles:/go/src/github.com/docker/docker/bundles" \
-                                  --name docker-pr$BUILD_NUMBER \
-                                  -e DOCKER_EXPERIMENTAL \
-                                  -e DOCKER_GITCOMMIT=${GIT_COMMIT} \
-                                  -e DOCKER_GRAPHDRIVER \
-                                  -e TESTDEBUG \
-                                  -e TEST_INTEGRATION_USE_SNAPSHOTTER \
-                                  -e TEST_SKIP_INTEGRATION_CLI \
-                                  -e TIMEOUT \
-                                  -e VALIDATE_REPO=${GIT_URL} \
-                                  -e VALIDATE_BRANCH=${CHANGE_TARGET} \
-                                  docker:${GIT_COMMIT} \
-                                  hack/make.sh \
-                                    dynbinary \
-                                    test-integration
-                                '''
-                            }
-                            post {
-                                always {
-                                    junit testResults: 'bundles/**/*-report.xml', allowEmptyResults: true
-                                }
-                            }
-                        }
-                    }
-
-                    post {
-                        always {
-                            sh '''
-                            echo "Ensuring container killed."
-                            docker rm -vf docker-pr$BUILD_NUMBER || true
-                            '''
-
-                            sh '''
-                            echo "Chowning /workspace to jenkins user"
-                            docker run --rm -v "$WORKSPACE:/workspace" busybox chown -R "$(id -u):$(id -g)" /workspace
-                            '''
-
-                            catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE', message: 'Failed to create bundles.tar.gz') {
-                                sh '''
-                                bundleName=ppc64le-integration
-                                echo "Creating ${bundleName}-bundles.tar.gz"
-                                # exclude overlay2 directories
-                                find bundles -path '*/root/*overlay2' -prune -o -type f \\( -name '*-report.json' -o -name '*.log' -o -name '*.prof' -o -name '*-report.xml' \\) -print | xargs tar -czf ${bundleName}-bundles.tar.gz
-                                '''
-
-                                archiveArtifacts artifacts: '*-bundles.tar.gz', allowEmptyArchive: true
-                            }
-                        }
-                        cleanup {
-                            sh 'make clean'
-                            deleteDir()
-                        }
-                    }
-                }
-                stage('ppc64le integration-cli') {
-                    when {
-                        beforeAgent true
-                        // Skip this stage on PRs unless the checkbox is selected
-                        anyOf {
-                            not { changeRequest() }
-                            expression { params.ppc64le }
-                        }
-                    }
-                    agent { label 'ppc64le-ubuntu-1604' }
-
-                    stages {
-                        stage("Print info") {
-                            steps {
-                                sh 'docker version'
-                                sh 'docker info'
-                                sh '''
-                                echo "check-config.sh version: ${CHECK_CONFIG_COMMIT}"
-                                curl -fsSL -o ${WORKSPACE}/check-config.sh "https://raw.githubusercontent.com/moby/moby/${CHECK_CONFIG_COMMIT}/contrib/check-config.sh" \
-                                && bash ${WORKSPACE}/check-config.sh || true
-                                '''
-                            }
-                        }
-                        stage("Build dev image") {
-                            steps {
-                                sh '''
-                                docker buildx build --load --force-rm -t docker:${GIT_COMMIT} .
-                                '''
-                            }
-                        }
-                        stage("Integration-cli tests") {
-                            environment { TEST_SKIP_INTEGRATION = '1' }
-                            steps {
-                                sh '''
-                                docker run --rm -t --privileged \
-                                  -v "$WORKSPACE/bundles:/go/src/github.com/docker/docker/bundles" \
-                                  --name docker-pr$BUILD_NUMBER \
-                                  -e DOCKER_GITCOMMIT=${GIT_COMMIT} \
-                                  -e DOCKER_GRAPHDRIVER \
-                                  -e TEST_INTEGRATION_USE_SNAPSHOTTER \
-                                  -e TEST_SKIP_INTEGRATION \
-                                  -e TIMEOUT \
-                                  -e VALIDATE_REPO=${GIT_URL} \
-                                  -e VALIDATE_BRANCH=${CHANGE_TARGET} \
-                                  docker:${GIT_COMMIT} \
-                                  hack/make.sh \
-                                    dynbinary \
-                                    test-integration
-                                '''
-                            }
-                            post {
-                                always {
-                                    junit testResults: 'bundles/**/*-report.xml', allowEmptyResults: true
-                                }
-                            }
-                        }
-                    }
-
-                    post {
-                        always {
-                            sh '''
-                            echo "Ensuring container killed."
-                            docker rm -vf docker-pr$BUILD_NUMBER || true
-                            '''
-
-                            sh '''
-                            echo "Chowning /workspace to jenkins user"
-                            docker run --rm -v "$WORKSPACE:/workspace" busybox chown -R "$(id -u):$(id -g)" /workspace
-                            '''
-
-                            catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE', message: 'Failed to create bundles.tar.gz') {
-                                sh '''
-                                bundleName=ppc64le-integration-cli
-                                echo "Creating ${bundleName}-bundles.tar.gz"
-                                # exclude overlay2 directories
-                                find bundles -path '*/root/*overlay2' -prune -o -type f \\( -name '*-report.json' -o -name '*.log' -o -name '*.prof' -o -name '*-report.xml' \\) -print | xargs tar -czf ${bundleName}-bundles.tar.gz
-                                '''
-
-                                archiveArtifacts artifacts: '*-bundles.tar.gz', allowEmptyArchive: true
-                            }
-                        }
-                        cleanup {
-                            sh 'make clean'
-                            deleteDir()
-                        }
-                    }
-                }
                 stage('arm64') {
                     when {
                         beforeAgent true


Test patch: