Repository: containers/buildah

Issue #4715: Buildah version 1.29.1 broken fuse-overlayfs in gitlab runner
Created: 2023-04-06T09:15:16Z
Description
After our buildah image was upgraded using the v1.29.1 version we noticed all our pipelines that were using buildah build started to fail on our gitlab runners. Seems like an issue with the fuse-overlayfs package
We're using the quay.io/buildah/stable:latest image, after downgrading to v1.29.0 buildah worked again.
Steps to reproduce the issue:

Upgrade buildah-stable to v1.29.1
Run buildah build in gitlab runner

Describe the results you received:
Buildah build failing to unmount and mount
Command used:
$ buildah build -q -f .docker/Dockerfile -t $CI_REGISTRY_IMAGE:$BUILD_TAG -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
Output:
Line 1 time="2023-04-06T08:20:26Z" level=error msg="Unmounting /var/lib/containers/storage/overlay/de00ab76e27e966fa7c0c0b79a5ad1247cdf765946bc05ada5b6d99be3a42be5/merged: invalid argument"
Line 2 Error: mounting new container: mounting build container "9c4bcdfd7a1a64ae4bb1d399c930ae9a29e52d240c1405e00bf7638cde951901": creating overlay mount to /var/lib/containers/storage/overlay/de00ab76e27e966fa7c0c0b79a5ad1247cdf765946bc05ada5b6d99be3a42be5/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/X3DKLX3WMGVRBGZJYZ4MSULUEY:/var/lib/containers/storage/overlay/l/VLKLEB5WCVLZFQUO2W3ANMY7ZY:/var/lib/containers/storage/overlay/l/YWHPWE4M7KVTL6WXCXO6LBJW52:/var/lib/containers/storage/overlay/l/RJM72NK2LQWGBJU2CB2BWSTGBT:/var/lib/containers/storage/overlay/l/6TW2TMTSHXLBWS4KEJYUFY73CA:/var/lib/containers/storage/overlay/l/MTOGBS4AKRVUFTNZYIFGJLFZCX:/var/lib/containers/storage/overlay/l/ZZXGS22J43ZTSNW3SH2S2L4WRU,upperdir=/var/lib/containers/storage/overlay/de00ab76e27e966fa7c0c0b79a5ad1247cdf765946bc05ada5b6d99be3a42be5/diff,workdir=/var/lib/containers/storage/overlay/de00ab76e27e966fa7c0c0b79a5ad1247cdf765946bc05ada5b6d99be3a42be5/work,nodev,fsync=0,volatile": invalid argument
Describe the results you expected:
A working buildah build command

Duplicate issue #4727: buildah quay.io image now triggers an overlay mount error
Created: 2023-04-10T06:01:31Z
Description
We use the quay.io buildah stable image to build an image and it now fails. It seems the image changed while the buildah version stayed at 1.29.1 and the GL runners as well our personal runners all trigger the problem. Using the same version but not this image is a simple workaround though, more below.
Steps to reproduce the issue:
I'm not totally sure about all conditions that are necessary to trigger the problem.

build an image using quay.io/buildah/stable on a GL runner

Describe the results you received:
time="2023-04-10T05:24:38Z" level=error msg="Unmounting /var/lib/containers/storage/overlay/2ab7e2e69ab745698479320e2ec6a357d878a688e3fcc141c3eb7dc3c55545c1/merged: invalid argument"
Error: mounting new container: mounting build container "cf1cf827797128f6b16338d15d4046733b6a758aa44a5fb760f918b85afcf04e": creating overlay mount to /var/lib/containers/storage/overlay/2ab7e2e69ab745698479320e2ec6a357d878a688e3fcc141c3eb7dc3c55545c1/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/l/UYVRA2Y5XFAFZ2VMMYIES47F5L,upperdir=/var/lib/containers/storage/overlay/2ab7e2e69ab745698479320e2ec6a357d878a688e3fcc141c3eb7dc3c55545c1/diff,workdir=/var/lib/containers/storage/overlay/2ab7e2e69ab745698479320e2ec6a357d878a688e3fcc141c3eb7dc3c55545c1/work,nodev,fsync=0,userxattr,volatile": invalid argument

See https://gitlab.com/osci/prometheus-exporter-exporter-container/-/jobs/4085194541
(I just added a tag command but the build part is unchanged)
I got this error with gitlab.com runners, but can replicate it with our runners, one on F37 kernel 6.0.7-301.fc37.x86_64 and the other F36 and kernel 5.18.9-200.fc36.x86_64.
It looks a little bit like #4704 but it is not the same. Also 1.29.1 worked before.
Describe the results you expected:
It should build the image without problem like it did not long ago with the same version.
See https://gitlab.com/osci/prometheus-exporter-exporter-container/-/jobs/4039291717
If I use a centos stream 9 image, install buildah (which happens to have the same exact version) and do the build, then all works well.
See https://gitlab.com/osci/prometheus-exporter-exporter-container/-/jobs/4085112209
(the CI fails because I made a mistake in the tag command but the build works)
Output of buildah version:
Version:         1.29.1
Go Version:      go1.19.5
Image Spec:      1.0.2-dev
Runtime Spec:    1.0.2-dev
CNI Spec:        1.0.0
libcni Version:  v1.1.2
image Version:   5.24.1
Git Commit:      
Built:           Fri Feb 17 10:05:41 2023
OS/Arch:         linux/amd64
BuildPlatform:   linux/amd64

Closing comment (2023-04-10T06:25:35Z): I have closed this because underlying problem is same as #4715 , either is using native overlay with wrong options or not using fuseoverlay for older kernel, I think quick fix is this one #4722 and long term fix is create correct config for different storage drivers.