Repository: moby/moby (language: go)

Issue:
ci: buildkit CI is broken; "cgo: malformed DWARF TagVariable entry"
### Description

Since late yesterday evening, BuildKit CI is broken; as far as I can see, it's breaking in a step that does not use the code in this repository (but please double-check if I'm right), so either

- The base image used
- The Dockerfile frontend (it looks to be using `dockerfile-upstream:master`)
- Perhaps some obscure issue, similar to https://github.com/containerd/containerd/issues/6387, where the Git "short commit" is used in the source code?
- Corrupted remote build-cache? (does it use that?)
- something changes on GitHub action hosts ?
- GitHub actions builds of docker / containerd / buildx ?

Failure looks like this: (https://github.com/moby/moby/actions/runs/3600129177/jobs/6064524682)

```
> [containerd-alt-15 1/1] RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache   git fetch origin   && git checkout -q "v1.5.11"   && make bin/containerd   && make bin/containerd-shim-runc-v2   && mv bin /out:
#0 4.935 + bin/containerd
#47 268.1 # github.com/containerd/containerd/vendor/github.com/miekg/pkcs11
#47 268.1 cgo: malformed DWARF TagVariable entry
#47 279.5 make: *** [Makefile:213: bin/containerd] Error 2
```

Which is similar to reports in go1.17 some time ago;

- looks reported in https://github.com/golang/go/issues/53000
- and a fix in https://go-review.googlesource.com/c/go/+/406816 ?


### Reproduce

See https://github.com/moby/moby/pull/44567 and other PRs

### Expected behavior

_No response_

### docker version

```bash
sudo rm /etc/docker/daemon.json
  sudo service docker restart
  docker version
  docker info
  shell: /usr/bin/bash -e {0}
  env:
    BUNDLES_OUTPUT: ./bundles
    BUILDKIT_REF: 4febae4f874bd8ef52dec30e988c8fe0bc96b3b9
Client:
 Version:           20.10.21+azure-1
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f82a10204ec5708d5fbba130ad76cfee49
 Built:             Tue Oct 25 17:53:02 UTC 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
```


### docker info

```bash
Server:
 Engine:
  Version:          20.10.21+azure-1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208812eb5e792fa99736c9167d1e10f4ab49
  Built:            Tue Oct 25 11:44:15 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.14+azure-1
  GitCommit:        b84d0b151c2395a5917996d602b192ce1e0fa461
 runc:
  Version:          1.1.4
  GitCommit:        5fd4c4d144137e991c4acebb2146ab1483a97925
 docker-init:
  Version:          0.19.0
  GitCommit:        
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., 0.9.1+azure-2)
  compose: Docker Compose (Docker Inc., 2.12.2+azure-1)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 21
 Server Version: 20.10.21+azure-1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b84d0b151c2395a5917996d602b192ce1e0fa461
 runc version: 5fd4c4d144137e991c4acebb2146ab1483a97925
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.15.0-1023-azure
 Operating System: Ubuntu 20.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 6.781GiB
 Name: fv-az77-14
 ID: 4IWW:VLI2:7YTM:AFEZ:4NJ5:VCDD:VCJU:3HXK:DSAL:C6S2:OKSN:N3A5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: githubactions
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
```


### Additional Info

The step where things break:

```
shell: /usr/bin/bash -e {0}
  env:
    BUNDLES_OUTPUT: ./bundles
    BUILDKIT_REF: 4febae4f874bd8ef52dec30e988c8fe0bc96b3b9
    CONTEXT: .
    TEST_DOCKERD: 1
    TEST_DOCKERD_BINARY: ./build/moby/binary-daemon/dockerd
    TESTPKGS: ./client
    TESTFLAGS: -v --parallel=1 --timeout=30m --run=//worker=dockerd$
+ docker buildx build --target integration-tests --output type=docker,name=buildkit-tests . --progress=plain
```

Failure with more detail:

```
 > [containerd-alt-15 1/1] RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache   git fetch origin   && git checkout -q "v1.5.11"   && make bin/containerd   && make bin/containerd-shim-runc-v2   && mv bin /out:
#0 5.824 + bin/containerd
#49 311.7 # github.com/containerd/containerd/vendor/github.com/miekg/pkcs11
#49 311.7 cgo: malformed DWARF TagVariable entry
#49 324.2 make: *** [Makefile:213: bin/containerd] Error 2
------
Dockerfile:143
--------------------
 142 |     ARG GO111MODULE=off
 143 | >>> RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache \
 144 | >>>   git fetch origin \
 145 | >>>   && git checkout -q "$CONTAINERD_ALT_VERSION_15" \
 146 | >>>   && make bin/containerd \
 147 | >>>   && make bin/containerd-shim-runc-v2 \
 148 | >>>   && mv bin /out
 149 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c git fetch origin   && git checkout -q \"$CONTAINERD_ALT_VERSION_15\"   && make bin/containerd   && make bin/containerd-shim-runc-v2   && mv bin /out" did not complete successfully: exit code: 2
Error: Process completed with exit code 1.
```

:warning: As mentioned, it looks like it's using

- a container builder
- the `docker.io/docker/dockerfile-upstream:master` frontend (`sha256:0e39c80bd858c49913c85dd3636aae0f6c8266ff1dcc24e5aeabf98e93579843`)

```
#1 [internal] booting buildkit
#1 starting container buildx_buildkit_builder-9473aa25-e043-4654-934c-be4271ea8e3d0
#1 starting container buildx_buildkit_builder-9473aa25-e043-4654-934c-be4271ea8e3d0 0.6s done
#1 DONE 0.6s

#2 [internal] load .dockerignore
#2 transferring context: 56B done
#2 DONE 0.0s

#3 [internal] load build definition from Dockerfile
#3 transferring dockerfile: 11.57kB done
#3 DONE 0.0s

#4 resolve image config for docker.io/docker/dockerfile-upstream:master
#4 ...

#5 [auth] docker/dockerfile-upstream:pull token for registry-1.docker.io
#5 DONE 0.0s

#4 resolve image config for docker.io/docker/dockerfile-upstream:master
#4 DONE 0.8s

#6 docker-image://docker.io/docker/dockerfile-upstream:master@sha256:0e39c80bd858c49913c85dd3636aae0f6c8266ff1dcc24e5aeabf98e93579843
#6 resolve docker.io/docker/dockerfile-upstream:master@sha256:0e39c80bd858c49913c85dd3636aae0f6c8266ff1dcc24e5aeabf98e93579843 done
#6 sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 1.05MB / 11.47MB 0.2s
#6 sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 3.15MB / 11.47MB 0.3s
#6 sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 11.47MB / 11.47MB 0.5s done
#6 extracting sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc
#6 extracting sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 0.2s done
#6 DONE 0.8s
```



Discussion / hints:
/cc @crazy-max 
As `CGO` appears to be involved, this may also be a Go caching issue; I see `--mount=target=/root/.cache,type=cache` is used, and Go is known to have issues with caching if CGO is involved (sometimes getting a cache-hit where it shouldn't etc); if `--mount=target=/root/.cache,type=cache` is shared between stages, then it's possible it's using the wrong cache.


I'm able to repro with:

```console
$ docker buildx build --target integration-tests https://github.com/moby/buildkit.git#4febae4f874bd8ef52dec30e988c8fe0bc96b3b9
...
------
 > [containerd 1/1] RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache   git fetch origin   && git checkout -q "v1.6.2"   && make bin/containerd   && make bin/containerd-shim-runc-v2   && make bin/ctr   && mv bin /out:
#0 2.845 + bin/containerd
#0 3.980 # github.com/miekg/pkcs11
#0 3.980 cgo: malformed DWARF TagVariable entry
#0 3.985 make: *** [Makefile:230: bin/containerd] Error 2
------
```

I will take a closer look.
This looks to be related to the [golang alpine image 1.18](https://github.com/moby/buildkit/blob/4febae4f874bd8ef52dec30e988c8fe0bc96b3b9/Dockerfile#L25). Switching to 1.18.7 and it works now. Any idea @tianon? Maybe an issue with Alpine 3.17 since 1.18.7 alpine image is based on Alpine 3.16.

Edit: confirmed, works fine with `golang:1.18-alpine3.16`.
I know alpine dropped LLVM on s390x (see https://github.com/docker/cli/issues/3822), but that shouldn't be related I think?
It looks fine with `golang:1.19-alpine` on master branch though: https://github.com/moby/buildkit/blob/master/Dockerfile#L25. Switching to `golang:1.18-alpine` on master and same issue. There's smth fishy with this version :thinking:.
Ok this change https://go-review.googlesource.com/c/go/+/406816 is not yet backported to 1.18 branch.
Oh, interesting; I saw it was reported with go1.17, so assumed it was included in 1.18, but I see now it was marked for go1.19.

We should request a backport for that for go1.18
@crazy-max could this one also be related?

- https://github.com/golang/go/issues/53013
Ah, that one was back ported I see https://github.com/golang/go/commit/e727f41930ae9a8bedfbc85eb8ca7268486571ec

So the procedure on their side looks to be to leave a backport request comment? https://github.com/golang/go/issues/53013#issuecomment-1333949358

Can we make an "as minimal as possible" reproducer to post together with that comment?
> So the procedure on their side looks to be to leave a backport request comment?

Done: https://github.com/golang/go/issues/53000#issuecomment-1335095986
So it looks like buildkit is using `golang:1.18-alpine` (no minor version for Go, and no specific alpine version); I see `1.18-alpine` is now an alias for `1.18.8-alpine3.17`, `1.18-alpine3.17`, `1.18.8-alpine`, `1.18-alpine`, so updated to alpine 3.17.

With alpine 3.16:

```bash
docker run -it --rm golang:1.18.8-alpine3.16 sh -c 'apk add --quiet --no-cache clang && clang --version'
Alpine clang version 13.0.1
Target: aarch64-alpine-linux-musl
Thread model: posix
InstalledDir: /usr/bin
```

With alpine 3.17:

```bash
docker run -it --rm golang:1.18.8-alpine3.17 sh -c 'apk add --quiet --no-cache clang && clang --version'
Alpine clang version 15.0.6
Target: aarch64-alpine-linux-musl
Thread model: posix
InstalledDir: /usr/bin
```
Opened https://github.com/moby/moby/pull/44571 to fix CI in the meantime.

Fix patch:
diff --git a/.github/workflows/buildkit.yml b/.github/workflows/buildkit.yml
index 954c006c0d0c4..d9de83a471abe 100644
--- a/.github/workflows/buildkit.yml
+++ b/.github/workflows/buildkit.yml
@@ -72,7 +72,7 @@ jobs:
           ./hack/go-mod-prepare.sh
           # FIXME(thaJeztah) temporarily overriding version to use for tests; remove with the next release of buildkit
           # echo "BUILDKIT_REF=$(./hack/buildkit-ref)" >> $GITHUB_ENV
-          echo "BUILDKIT_REF=4febae4f874bd8ef52dec30e988c8fe0bc96b3b9" >> $GITHUB_ENV
+          echo "BUILDKIT_REF=0bfcd83e6db95e6c6877ee6e5224b994cea62ba1" >> $GITHUB_ENV
         working-directory: moby
       -
         name: Checkout BuildKit ${{ env.BUILDKIT_REF }}


Test patch: