Repository: osbuild/osbuild-composer
Pull request #2857 (2022-07-26 09:01:03)

CI: drop /tmp/artifacts upload to Gitlab
Contents of this folder are uploaded to S3 with shutzbot/upload_artifacts.sh
Also fixing epel installation in the mentioned script.


This pull request includes:

- [ ] adequate testing for the new functionality or fixed issue
- [ ] adequate documentation informing people about the change such as
  - [ ] submit a PR for the [guides](https://github.com/osbuild/guides) repository if this PR changed any behavior described there: https://www.osbuild.org/guides/

<!--
Thanks for proposing a change to osbuild-composer!

Please don't remove the above check list. These are things that each pull
request must have before it is merged. It helps maintainers to not forget
anything.

If the reason for ticking any of the boxes is ambiguous, please add a short
note explaining why.

In addition, if this pull request fixes a downstream issue, please refer to
test/README.md and add these additional items:

- [ ] 1st commit of any `rhbz#` related PR contains bug reproducer; CI reports FAIL or
- [ ] PR contains automated tests for new functionality and
- [ ] QE has approved reproducer/new tests and
- [ ] Subsequent commits provide bug fixes without modifying the reproducer; CI reports PASS and
- [ ] QE approves this PR; RHBZ status is set to `MODIFIED + Verified=Tested`

Information regarding our GitLab pipeline (Schutzbot):

CI will not be ran automatically if WIP label is applied or the PR is in DRAFT state, instead only
a link will be provided to the pipeline which can then be triggered manually if desired. To run the
CI automatically either switch the PR to ready or apply WIP+test label.

Outside contributors need manual approval from one of the osbuild-composer maintainers.

Schutzbot will only be triggered if all Tests jobs in GitHub workflow succeed.
-->


jrusz: > We need this artifact, otherwise
> 
> https://github.com/osbuild/osbuild-composer/blob/f3c166c3fd0317a5fc3928b0fd10447cd3adc7e0/test/cases/diff-manifests.sh#L96
> 
> will break.

Oh ok sorry, didn't notice that, thanks! It should be then just that one artifact, I've updated the commit, should work like this..
jrusz: > @jrusz I can see
> 
> ```
> No match for argument: s3cmd
> Error: Unable to find a match: s3cmd
> [2022-07-26T10:48:15+00:00] Job artifacts will be uploaded to: s3://image-builder-ci-artifacts/osbuild-composer/PR-2857/2770471275/
> schutzbot/upload_artifacts.sh: line 24: s3cmd: command not found
> ```
> 
> in https://gitlab.com/redhat/services/products/image-builder/ci/osbuild-composer/-/jobs/2770471275 thinking

Yeah, I had a similar issue in our Upgrade test and adding the `source /etc/os-release` solved that, but here it didn't seem to have helped, investigating....

Patch:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index 4e497f4a3..8d477cba2 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -666,7 +666,7 @@ Manifest-diff:
     GIT_DEPTH: 500
   artifacts:
     paths:
-      - "/tmp/artifacts"
+      - manifests.diff
 
 SCHEDULED_CLOUD_CLEANER:
   stage: cleanup
diff --git a/Schutzfile b/Schutzfile
index b4a444f03..c337e9e34 100644
--- a/Schutzfile
+++ b/Schutzfile
@@ -2,7 +2,7 @@
   "fedora-35": {
     "dependencies": {
       "osbuild": {
-        "commit": "51315a985aa3c26179acead9459db9c249a8b4f6"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     },
     "repos": [
@@ -79,7 +79,7 @@
   "fedora-36": {
     "dependencies": {
       "osbuild": {
-        "commit": "527be5081c21a313afb0c0cdeb4c3edadcf2225c"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     },
     "repos": [
@@ -156,21 +156,21 @@
   "rhel-8.4": {
     "dependencies": {
       "osbuild": {
-        "commit": "51315a985aa3c26179acead9459db9c249a8b4f6"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     }
   },
   "rhel-8.6": {
     "dependencies": {
       "osbuild": {
-        "commit": "51315a985aa3c26179acead9459db9c249a8b4f6"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     }
   },
   "rhel-8.7": {
     "dependencies": {
       "osbuild": {
-        "commit": "51315a985aa3c26179acead9459db9c249a8b4f6"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     },
     "repos": [
@@ -216,14 +216,14 @@
   "rhel-9.0": {
     "dependencies": {
       "osbuild": {
-        "commit": "51315a985aa3c26179acead9459db9c249a8b4f6"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     }
   },
   "rhel-9.1": {
     "dependencies": {
       "osbuild": {
-        "commit": "51315a985aa3c26179acead9459db9c249a8b4f6"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     },
     "repos": [
@@ -269,21 +269,21 @@
   "centos-8": {
     "dependencies": {
       "osbuild": {
-        "commit": "51315a985aa3c26179acead9459db9c249a8b4f6"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     }
   },
   "centos-9": {
     "dependencies": {
       "osbuild": {
-        "commit": "51315a985aa3c26179acead9459db9c249a8b4f6"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     }
   },
   "centos-stream-9": {
     "dependencies": {
       "osbuild": {
-        "commit": "51315a985aa3c26179acead9459db9c249a8b4f6"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     },
     "repos": [
@@ -329,7 +329,7 @@
   "centos-stream-8": {
     "dependencies": {
       "osbuild": {
-        "commit": "51315a985aa3c26179acead9459db9c249a8b4f6"
+        "commit": "376cbffd136bc4ba86fc7c63697fa5b88fe3acef"
       }
     },
     "repos": [
diff --git a/internal/container/client.go b/internal/container/client.go
index 68d942605..cb05b9939 100644
--- a/internal/container/client.go
+++ b/internal/container/client.go
@@ -8,12 +8,14 @@ import (
 	"io"
 	"os"
 	"path/filepath"
+	"strconv"
 	"strings"
 
 	_ "github.com/containers/image/v5/docker/archive"
 	_ "github.com/containers/image/v5/oci/archive"
 	_ "github.com/containers/image/v5/oci/layout"
 	"github.com/osbuild/osbuild-composer/internal/common"
+	"golang.org/x/sys/unix"
 
 	"github.com/containers/common/pkg/retry"
 	"github.com/containers/image/v5/copy"
@@ -33,41 +35,40 @@ const (
 	DefaultPolicyPath = "/etc/containers/policy.json"
 )
 
-var (
-	defaultAuthFile = GetDefaultAuthFile()
-)
-
 // GetDefaultAuthFile returns the authentication file to use for the
 // current environment.
 //
 // This is basically a re-implementation of `getPathToAuthWithOS` from
 // containers/image/pkg/docker/config/config.go[1], but we ensure that
-// the returned path is either accessible or nonexistent. This is needed
-// since any other error than os.ErrNotExist will lead to an overall
-// failure and thus prevent any operation even with public resources.
+// the returned path is either accessible. This is needed since any
+// other error than os.ErrNotExist will lead to an overall failure and
+// thus prevent any operation even with public resources.
 //
 // [1] https://github.com/containers/image/blob/55ea76c7db702ed1af60924a0b57c8da533d9e5a/pkg/docker/config/config.go#L506
 func GetDefaultAuthFile() string {
 
-	dirExistsOrEmpty := func(path string) bool {
-		_, err := os.Stat(path)
-		// document the error case
-		return err == nil || os.IsNotExist(err)
+	checkAccess := func(path string) bool {
+		err := unix.Access(path, unix.R_OK)
+		return err == nil
+	}
+
+	if authFile := os.Getenv("REGISTRY_AUTH_FILE"); authFile != "" {
+		if checkAccess(authFile) {
+			return authFile
+		}
 	}
 
 	if runtimeDir := os.Getenv("XDG_RUNTIME_DIR"); runtimeDir != "" {
-		path := filepath.Join(runtimeDir, "containers", "auth.json")
-		if dirExistsOrEmpty(path) {
-			return path
+		if checkAccess(runtimeDir) {
+			return filepath.Join(runtimeDir, "containers", "auth.json")
 		}
 	}
 
-	path := fmt.Sprintf("/run/containers/%d/auth.json", os.Getuid())
-	if dirExistsOrEmpty(path) {
-		return path
+	if rundir := filepath.FromSlash("/run/containers"); checkAccess(rundir) {
+		return filepath.Join(rundir, strconv.Itoa(os.Getuid()), "auth.json")
 	}
 
-	return filepath.Join("var", "empty", "containers-auth.json")
+	return filepath.FromSlash("/var/empty/containers-auth.json")
 }
 
 // A Client to interact with the given Target object at a
@@ -131,7 +132,7 @@ func NewClient(target string) (*Client, error) {
 
 			OSChoice: "linux",
 
-			AuthFilePath: defaultAuthFile,
+			AuthFilePath: GetDefaultAuthFile(),
 		},
 		policy: policy,
 	}
@@ -144,6 +145,11 @@ func (cl *Client) SetAuthFilePath(path string) {
 	cl.sysCtx.AuthFilePath = path
 }
 
+// GetAuthFilePath gets the location of the `containers-auth.json(5)` file.
+func (cl *Client) GetAuthFilePath() string {
+	return cl.sysCtx.AuthFilePath
+}
+
 func (cl *Client) SetArchitectureChoice(arch string) {
 	// Translate some well-known Composer architecture strings
 	// into the corresponding container ones
diff --git a/internal/distro/rhel8/distro.go b/internal/distro/rhel8/distro.go
index 89688ceb1..39971dd4d 100644
--- a/internal/distro/rhel8/distro.go
+++ b/internal/distro/rhel8/distro.go
@@ -419,7 +419,16 @@ func (t *imageType) PackageSets(bp blueprint.Blueprint, options distro.ImageOpti
 
 	// if we are embedding containers we need to have `skopeo` in the build root
 	if len(bp.Containers) > 0 {
-		mergedSets[buildPkgsKey] = mergedSets[buildPkgsKey].Append(rpmmd.PackageSet{Include: []string{"skopeo"}})
+
+		extraPkgs := rpmmd.PackageSet{Include: []string{"skopeo"}}
+
+		if t.rpmOstree {
+			// for OSTree based images we need to configure the containers-storage.conf(5)
+			// via the org.osbuild.containers.storage.conf stage, which needs python3-pytoml
+			extraPkgs = extraPkgs.Append(rpmmd.PackageSet{Include: []string{"python3-pytoml"}})
+		}
+
+		mergedSets[buildPkgsKey] = mergedSets[buildPkgsKey].Append(extraPkgs)
 	}
 
 	// depsolve bp packages separately
@@ -567,10 +576,8 @@ func (t *imageType) Manifest(customizations *blueprint.Customizations,
 // checkOptions checks the validity and compatibility of options and customizations for the image type.
 func (t *imageType) checkOptions(customizations *blueprint.Customizations, options distro.ImageOptions, containers []container.Spec) error {
 
-	// we support embedding containers on all image types that are not ostree based
-	// since we need to store them outside `/var` since that is not preserved in
-	// commits and then point the container `storage.conf` to that extra location
-	if t.rpmOstree && len(containers) > 0 {
+	// we do not support embedding containers on ostree-derived images, only on commits themselves
+	if len(containers) > 0 && t.rpmOstree && (t.name != "edge-commit" && t.name != "edge-container") {
 		return fmt.Errorf("embedding containers is not supported for %s on %s", t.name, t.arch.distro.name)
 	}
 
diff --git a/internal/distro/rhel8/pipelines.go b/internal/distro/rhel8/pipelines.go
index c5ecad4a6..7939d335e 100644
--- a/internal/distro/rhel8/pipelines.go
+++ b/internal/distro/rhel8/pipelines.go
@@ -439,7 +439,22 @@ func osPipeline(t *imageType,
 
 	if len(containers) > 0 {
 		images := osbuild.NewContainersInputForSources(containers)
-		skopeo := osbuild.NewSkopeoStage(images, "")
+
+		var storagePath string
+
+		// OSTree commits do not include data in `/var` since that is tied to the
+		// deployment, rather than the commit. Therefore the containers need to be
+		// stored in a different location, like `/usr/share`, and the container
+		// storage engine configured accordingly.
+		if t.rpmOstree {
+			storagePath = "/usr/share/containers/storage"
+			storageConf := "/etc/containers/storage.conf"
+
+			containerStoreOpts := osbuild.NewContainerStorageOptions(storageConf, storagePath)
+			p.AddStage(osbuild.NewContainersStorageConfStage(containerStoreOpts))
+		}
+
+		skopeo := osbuild.NewSkopeoStage(images, storagePath)
 		p.AddStage(skopeo)
 	}
 
diff --git a/internal/distro/rhel90/distro.go b/internal/distro/rhel90/distro.go
index 02d9ec58d..f4c60efb3 100644
--- a/internal/distro/rhel90/distro.go
+++ b/internal/distro/rhel90/distro.go
@@ -353,7 +353,16 @@ func (t *imageType) PackageSets(bp blueprint.Blueprint, options distro.ImageOpti
 
 	// if we are embedding containers we need to have `skopeo` in the build root
 	if len(bp.Containers) > 0 {
-		mergedSets[buildPkgsKey] = mergedSets[buildPkgsKey].Append(rpmmd.PackageSet{Include: []string{"skopeo"}})
+
+		extraPkgs := rpmmd.PackageSet{Include: []string{"skopeo"}}
+
+		if t.rpmOstree {
+			// for OSTree based images we need to configure the containers-storage.conf(5)
+			// via the org.osbuild.containers.storage.conf stage, which needs python3-toml
+			extraPkgs = extraPkgs.Append(rpmmd.PackageSet{Include: []string{"python3-toml"}})
+		}
+
+		mergedSets[buildPkgsKey] = mergedSets[buildPkgsKey].Append(extraPkgs)
 	}
 
 	// depsolve bp packages separately
@@ -501,10 +510,8 @@ func (t *imageType) Manifest(customizations *blueprint.Customizations,
 // checkOptions checks the validity and compatibility of options and customizations for the image type.
 func (t *imageType) checkOptions(customizations *blueprint.Customizations, options distro.ImageOptions, containers []container.Spec) error {
 
-	// we support embedding containers on all image types that are not ostree based
-	// since we need to store them outside `/var` since that is not preserved in
-	// commits and then point the container `storage.conf` to that extra location
-	if t.rpmOstree && len(containers) > 0 {
+	// we do not support embedding containers on ostree-derived images, only on commits themselves
+	if len(containers) > 0 && t.rpmOstree && (t.name != "edge-commit" && t.name != "edge-container") {
 		return fmt.Errorf("embedding containers is not supported for %s on %s", t.name, t.arch.distro.name)
 	}
 
diff --git a/internal/distro/rhel90/pipelines.go b/internal/distro/rhel90/pipelines.go
index 15e5cd101..26d631261 100644
--- a/internal/distro/rhel90/pipelines.go
+++ b/internal/distro/rhel90/pipelines.go
@@ -431,7 +431,22 @@ func osPipeline(t *imageType,
 
 	if len(containers) > 0 {
 		images := osbuild.NewContainersInputForSources(containers)
-		skopeo := osbuild.NewSkopeoStage(images, "")
+
+		var storagePath string
+
+		// OSTree commits do not include data in `/var` since that is tied to the
+		// deployment, rather than the commit. Therefore the containers need to be
+		// stored in a different location, like `/usr/share`, and the container
+		// storage engine configured accordingly.
+		if t.rpmOstree {
+			storagePath = "/usr/share/containers/storage"
+			storageConf := "/etc/containers/storage.conf"
+
+			containerStoreOpts := osbuild.NewContainerStorageOptions(storageConf, storagePath)
+			p.AddStage(osbuild.NewContainersStorageConfStage(containerStoreOpts))
+		}
+
+		skopeo := osbuild.NewSkopeoStage(images, storagePath)
 		p.AddStage(skopeo)
 	}
 
diff --git a/internal/osbuild/containers_storage_conf_stage.go b/internal/osbuild/containers_storage_conf_stage.go
new file mode 100644
index 000000000..94635000d
--- /dev/null
+++ b/internal/osbuild/containers_storage_conf_stage.go
@@ -0,0 +1,58 @@
+package osbuild
+
+import "fmt"
+
+type CSCStorageOptions struct {
+	AdditionalImageStores []string `json:"additionalimagestores,omitempty"`
+}
+
+// CSCStorage is short for ContainersStorageConfigStorage
+type CSCStorage struct {
+	Options *CSCStorageOptions `json:"options,omitempty"`
+}
+
+type ContainersStorageConfig struct {
+	Storage CSCStorage `json:"storage,omitempty"`
+}
+
+type ContainersStorageConfStageOptions struct {
+	Filename string                  `json:"filename,omitempty"`
+	Config   ContainersStorageConfig `json:"config"`
+	Comment  []string                `json:"comment,omitempty"`
+}
+
+func (ContainersStorageConfStageOptions) isStageOptions() {}
+
+func NewContainerStorageOptions(filename string, additionalImageStores ...string) *ContainersStorageConfStageOptions {
+	options := ContainersStorageConfStageOptions{
+		Filename: filename,
+	}
+
+	if len(additionalImageStores) > 0 {
+		options.Config.Storage.Options = &CSCStorageOptions{
+			AdditionalImageStores: additionalImageStores,
+		}
+	}
+
+	return &options
+}
+
+func (o *ContainersStorageConfStageOptions) validate() error {
+	if o.Filename == "" {
+		return fmt.Errorf("`Filename` must be set")
+	}
+
+	return nil
+}
+
+func NewContainersStorageConfStage(options *ContainersStorageConfStageOptions) *Stage {
+
+	if err := options.validate(); err != nil {
+		panic(err)
+	}
+
+	return &Stage{
+		Type:    "org.osbuild.containers.storage.conf",
+		Options: options,
+	}
+}
diff --git a/osbuild-composer.spec b/osbuild-composer.spec
index 987ae9c85..4b25687cb 100644
--- a/osbuild-composer.spec
+++ b/osbuild-composer.spec
@@ -308,10 +308,10 @@ The core osbuild-composer binary. This is suitable both for spawning in containe
 Summary:    The worker for osbuild-composer
 Requires:   systemd
 Requires:   qemu-img
-Requires:   osbuild >= 55
-Requires:   osbuild-ostree >= 55
-Requires:   osbuild-lvm2 >= 55
-Requires:   osbuild-luks2 >= 55
+Requires:   osbuild >= 62
+Requires:   osbuild-ostree >= 62
+Requires:   osbuild-lvm2 >= 62
+Requires:   osbuild-luks2 >= 62
 Requires:   %{name}-dnf-json = %{version}-%{release}
 
 %description worker
diff --git a/schutzbot/upload_artifacts.sh b/schutzbot/upload_artifacts.sh
index cb2b739ad..fe7250586 100755
--- a/schutzbot/upload_artifacts.sh
+++ b/schutzbot/upload_artifacts.sh
@@ -10,7 +10,7 @@ ARTIFACTS=${ARTIFACTS:-/tmp/artifacts}
 function greenprint {
   echo -e "\033[1;32m[$(date -Isecond)] ${1}\033[0m"
 }
-
+source /etc/os-release
 # s3cmd is in epel, add if it's not present
 if [[ $ID == rhel || $ID == centos ]] && ! rpm -q epel-release; then
     curl -Ls --retry 5 --output /tmp/epel.rpm \


Test patch:
diff --git a/internal/container/client_test.go b/internal/container/client_test.go
index 8a3f36ec2..7133fad04 100644
--- a/internal/container/client_test.go
+++ b/internal/container/client_test.go
@@ -2,6 +2,8 @@ package container_test
 
 import (
 	"context"
+	"io/ioutil"
+	"os"
 	"testing"
 	"time"
 
@@ -63,3 +65,38 @@ func TestClientResolve(t *testing.T) {
 
 	assert.Error(t, err)
 }
+
+func TestClientAuthFilePath(t *testing.T) {
+
+	client, err := container.NewClient("quay.io/osbuild/osbuild")
+	assert.NoError(t, err)
+
+	authFilePath := client.GetAuthFilePath()
+	assert.NotEmpty(t, authFilePath)
+	assert.Equal(t, authFilePath, container.GetDefaultAuthFile())
+
+	// make sure the file is accessible
+	_, err = ioutil.ReadFile(authFilePath)
+	assert.True(t, err == nil || os.IsNotExist(err))
+
+	t.Run("XDG_RUNTIME_DIR", func(t *testing.T) {
+		runtimeDir := os.Getenv("XDG_RUNTIME_DIR")
+
+		if runtimeDir == "" {
+			t.Skip("XDG_RUNTIME_DIR not set, skipping test")
+			return
+		}
+
+		t.Cleanup(func() {
+			os.Setenv("XDG_RUNTIME_DIR", runtimeDir)
+		})
+
+		os.Unsetenv("XDG_RUNTIME_DIR")
+
+		authFilePath := container.GetDefaultAuthFile()
+		assert.NotEmpty(t, authFilePath)
+		_, err = ioutil.ReadFile(authFilePath)
+		assert.True(t, err == nil || os.IsNotExist(err))
+	})
+
+}
diff --git a/internal/osbuild/containers_storage_conf_stage_test.go b/internal/osbuild/containers_storage_conf_stage_test.go
new file mode 100644
index 000000000..1289b029e
--- /dev/null
+++ b/internal/osbuild/containers_storage_conf_stage_test.go
@@ -0,0 +1,31 @@
+package osbuild
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+)
+
+func TestContainersStorageConfStage(t *testing.T) {
+	expectedStage := &Stage{
+		Type: "org.osbuild.containers.storage.conf",
+		Options: &ContainersStorageConfStageOptions{
+			Filename: "/usr/share/containers/storage.conf",
+			Config: ContainersStorageConfig{
+				Storage: CSCStorage{
+					Options: &CSCStorageOptions{
+						AdditionalImageStores: []string{
+							"/usr/share/containers/storage/",
+						},
+					},
+				},
+			},
+		},
+	}
+
+	actualStage := NewContainersStorageConfStage(
+		NewContainerStorageOptions("/usr/share/containers/storage.conf",
+			"/usr/share/containers/storage/"),
+	)
+	assert.Equal(t, expectedStage, actualStage)
+}
diff --git a/test/cases/diff-manifests.sh b/test/cases/diff-manifests.sh
index 12849730f..f2f2956ed 100755
--- a/test/cases/diff-manifests.sh
+++ b/test/cases/diff-manifests.sh
@@ -2,9 +2,6 @@
 
 set -euo pipefail
 
-artifacts="/tmp/artifacts"
-mkdir -p "${artifacts}"
-
 # Colorful output.
 function greenprint {
     echo -e "\033[1;32m[$(date -Isecond)] ${1}\033[0m"
@@ -12,6 +9,9 @@ function greenprint {
 function redprint {
     echo -e "\033[1;31m[$(date -Isecond)] ${1}\033[0m"
 }
+function revert_to_head {
+   git checkout "$head"
+}
 
 if [[ "${CI_COMMIT_BRANCH}" != PR-* ]]; then
     greenprint "${CI_COMMIT_BRANCH} is not a Pull Request"
@@ -63,6 +63,8 @@ if (( err != 0 )); then
     exit 1
 fi
 
+# revert to $head on exit
+trap revert_to_head EXIT
 greenprint "Checking out merge-base ${mergebase}"
 git checkout "${mergebase}"
 
@@ -90,10 +92,10 @@ if (( err == 0 )); then
 fi
 
 greenprint "Manifests differ"
-echo "${diff}" > "${artifacts}/manifests.diff"
+echo "${diff}" > "manifests.diff"
 greenprint "Saved diff in job artifacts"
 
-artifacts_url="${CI_JOB_URL}/artifacts/browse/ci-artifacts"
+artifacts_url="${CI_JOB_URL}/artifacts/browse"
 
 cat > "${review_data_file}" << EOF
 {"body":"⚠️ This PR introduces changes in at least one manifest (when comparing PR HEAD ${head} with the ${basebranch} merge-base ${mergebase}).  Please review the changes.  The changes can be found in the [artifacts of the \`Manifest-diff\` job [0]](${artifacts_url}) as \`manifests.diff\`.\n\n${merge_base_fail}[0] ${artifacts_url}","event":"COMMENT"}
diff --git a/test/cases/ostree-ng.sh b/test/cases/ostree-ng.sh
index bc0419944..7c894a4d5 100755
--- a/test/cases/ostree-ng.sh
+++ b/test/cases/ostree-ng.sh
@@ -113,6 +113,7 @@ case "${ID}-${VERSION_ID}" in
         OSTREE_REF="fedora/35/${ARCH}/iot"
         OSTREE_OSNAME=fedora
         OS_VARIANT="fedora35"
+        EMBEDED_CONTAINER="false"
         ;;
     "fedora-36")
         CONTAINER_TYPE=fedora-iot-container
@@ -120,22 +121,27 @@ case "${ID}-${VERSION_ID}" in
         OSTREE_REF="fedora/36/${ARCH}/iot"
         OSTREE_OSNAME=fedora
         OS_VARIANT="fedora36"
+        EMBEDED_CONTAINER="false"
         ;;
     "rhel-8.7")
         OSTREE_REF="test/rhel/8/${ARCH}/edge"
         OS_VARIANT="rhel8-unknown"
+        EMBEDED_CONTAINER="true"
         ;;
     "rhel-9.1")
         OSTREE_REF="test/rhel/9/${ARCH}/edge"
         OS_VARIANT="rhel9-unknown"
+        EMBEDED_CONTAINER="true"
         ;;
     "centos-8")
         OSTREE_REF="test/centos/8/${ARCH}/edge"
         OS_VARIANT="centos8"
+        EMBEDED_CONTAINER="true"
         ;;
     "centos-9")
         OSTREE_REF="test/centos/9/${ARCH}/edge"
         OS_VARIANT="centos-stream9"
+        EMBEDED_CONTAINER="true"
         ;;
     *)
         echo "unsupported distro: ${ID}-${VERSION_ID}"
@@ -389,6 +395,14 @@ home = "/home/admin/"
 groups = ["wheel"]
 EOF
 
+# RHEL 8.7 and 9.1 later support embeded container in commit
+if [[ "${EMBEDED_CONTAINER}" == "true" ]]; then
+    tee -a "$BLUEPRINT_FILE" > /dev/null << EOF
+[[containers]]
+source = "quay.io/fedora/fedora:latest"
+EOF
+fi
+
 greenprint "📄 container blueprint"
 cat "$BLUEPRINT_FILE"
 
@@ -560,7 +574,7 @@ EOF
 
 # Test IoT/Edge OS
 greenprint "📼 Run Edge tests on BIOS VM"
-sudo ansible-playbook -v -i "${TEMPDIR}"/inventory -e image_type="$OSTREE_OSNAME" -e ostree_commit="${INSTALL_HASH}" /usr/share/tests/osbuild-composer/ansible/check_ostree.yaml || RESULTS=0
+sudo ansible-playbook -v -i "${TEMPDIR}"/inventory -e image_type="$OSTREE_OSNAME" -e ostree_commit="${INSTALL_HASH}" -e embeded_container="true" /usr/share/tests/osbuild-composer/ansible/check_ostree.yaml || RESULTS=0
 check_result
 
 # Clean up BIOS VM
@@ -645,6 +659,14 @@ name = "kernel-rt"
 EOF
 fi
 
+# RHEL 8.7 and 9.1 later support embeded container in commit
+if [[ "${EMBEDED_CONTAINER}" == "true" ]]; then
+    tee -a "$BLUEPRINT_FILE" > /dev/null << EOF
+[[containers]]
+source = "quay.io/fedora/fedora:latest"
+EOF
+fi
+
 greenprint "📄 upgrade blueprint"
 cat "$BLUEPRINT_FILE"
 
@@ -739,7 +761,7 @@ ansible_ssh_common_args="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/
 EOF
 
 # Test IoT/Edge OS
-sudo ansible-playbook -v -i "${TEMPDIR}"/inventory -e image_type="$OSTREE_OSNAME" -e ostree_commit="${UPGRADE_HASH}" /usr/share/tests/osbuild-composer/ansible/check_ostree.yaml || RESULTS=0
+sudo ansible-playbook -v -i "${TEMPDIR}"/inventory -e image_type="$OSTREE_OSNAME" -e ostree_commit="${UPGRADE_HASH}" -e embeded_container="true" /usr/share/tests/osbuild-composer/ansible/check_ostree.yaml || RESULTS=0
 check_result
 
 # Final success clean up
diff --git a/test/cases/ostree.sh b/test/cases/ostree.sh
index a3efb4af4..25e8d2ee6 100755
--- a/test/cases/ostree.sh
+++ b/test/cases/ostree.sh
@@ -20,6 +20,7 @@ case "${ID}-${VERSION_ID}" in
         OS_VARIANT="fedora35"
         USER_IN_COMMIT="false"
         BOOT_LOCATION="https://mirrors.rit.edu/fedora/fedora/linux/releases/35/Everything/x86_64/os/"
+        EMBEDED_CONTAINER="false"
         ;;
     "fedora-36")
         IMAGE_TYPE=fedora-iot-commit
@@ -27,6 +28,7 @@ case "${ID}-${VERSION_ID}" in
         OS_VARIANT="fedora36"
         USER_IN_COMMIT="false"
         BOOT_LOCATION="https://mirrors.rit.edu/fedora/fedora/linux/releases/36/Everything/x86_64/os/"
+        EMBEDED_CONTAINER="false"
         ;;
     "rhel-8.4")
         IMAGE_TYPE=edge-commit
@@ -34,6 +36,7 @@ case "${ID}-${VERSION_ID}" in
         OS_VARIANT="rhel8.4"
         USER_IN_COMMIT="true"
         BOOT_LOCATION="http://download.devel.redhat.com/released/rhel-8/RHEL-8/8.4.0/BaseOS/x86_64/os/"
+        EMBEDED_CONTAINER="false"
         ;;
     "rhel-8.6")
         IMAGE_TYPE=edge-commit
@@ -41,12 +44,14 @@ case "${ID}-${VERSION_ID}" in
         OS_VARIANT="rhel8-unknown"
         USER_IN_COMMIT="true"
         BOOT_LOCATION="http://download.devel.redhat.com/released/rhel-8/RHEL-8/8.6.0/BaseOS/x86_64/os/"
+        EMBEDED_CONTAINER="false"
         ;;
     "rhel-8.7")
         IMAGE_TYPE=edge-commit
         OSTREE_REF="rhel/8/${ARCH}/edge"
         OS_VARIANT="rhel8-unknown"
         USER_IN_COMMIT="true"
+        EMBEDED_CONTAINER="true"
 
         # Use a stable installer image unless it's the nightly pipeline
         BOOT_LOCATION="http://download.devel.redhat.com/released/rhel-8/RHEL-8/8.6.0/BaseOS/x86_64/os/"
@@ -60,12 +65,14 @@ case "${ID}-${VERSION_ID}" in
         OS_VARIANT="rhel9.0"
         USER_IN_COMMIT="true"
         BOOT_LOCATION="http://download.devel.redhat.com/released/rhel-9/RHEL-9/9.0.0/BaseOS/x86_64/os/"
+        EMBEDED_CONTAINER="false"
         ;;
     "rhel-9.1")
         IMAGE_TYPE=edge-commit
         OSTREE_REF="rhel/9/${ARCH}/edge"
         OS_VARIANT="rhel9-unknown"
         USER_IN_COMMIT="true"
+        EMBEDED_CONTAINER="true"
 
         # Use a stable installer image unless it's the nightly pipeline
         BOOT_LOCATION="http://download.devel.redhat.com/released/rhel-9/RHEL-9/9.0.0/BaseOS/x86_64/os/"
@@ -79,6 +86,7 @@ case "${ID}-${VERSION_ID}" in
         OS_VARIANT="centos8"
         USER_IN_COMMIT="true"
         BOOT_LOCATION="http://mirror.centos.org/centos/8-stream/BaseOS/x86_64/os/"
+        EMBEDED_CONTAINER="true"
         ;;
     "centos-9")
         IMAGE_TYPE=edge-commit
@@ -86,6 +94,7 @@ case "${ID}-${VERSION_ID}" in
         OS_VARIANT="centos-stream9"
         USER_IN_COMMIT="true"
         BOOT_LOCATION="https://odcs.stream.centos.org/production/latest-CentOS-Stream/compose/BaseOS/x86_64/os/"
+        EMBEDED_CONTAINER="true"
         ;;
     *)
         echo "unsupported distro: ${ID}-${VERSION_ID}"
@@ -335,6 +344,14 @@ groups = ["wheel"]
 EOF
 fi
 
+# RHEL 8.7 and 9.1 later support embeded container in commit
+if [[ "${EMBEDED_CONTAINER}" == "true" ]]; then
+    tee -a "$BLUEPRINT_FILE" > /dev/null << EOF
+[[containers]]
+source = "quay.io/fedora/fedora:latest"
+EOF
+fi
+
 # Build installation image.
 build_image "$BLUEPRINT_FILE" ostree
 
@@ -504,6 +521,14 @@ groups = ["wheel"]
 EOF
 fi
 
+# RHEL 8.7 and 9.1 later support embeded container in commit
+if [[ "${EMBEDED_CONTAINER}" == "true" ]]; then
+    tee -a "$BLUEPRINT_FILE" > /dev/null << EOF
+[[containers]]
+source = "quay.io/fedora/fedora:latest"
+EOF
+fi
+
 # Build upgrade image.
 build_image "$BLUEPRINT_FILE" upgrade
 
@@ -570,7 +595,7 @@ ansible_ssh_common_args="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/
 EOF
 
 # Test IoT/Edge OS
-sudo ansible-playbook -v -i "${TEMPDIR}"/inventory -e image_type=${IMAGE_TYPE} -e ostree_commit="${UPGRADE_HASH}" /usr/share/tests/osbuild-composer/ansible/check_ostree.yaml || RESULTS=0
+sudo ansible-playbook -v -i "${TEMPDIR}"/inventory -e image_type=${IMAGE_TYPE} -e ostree_commit="${UPGRADE_HASH}" -e embeded_container="true" /usr/share/tests/osbuild-composer/ansible/check_ostree.yaml || RESULTS=0
 check_result
 
 # Final success clean up
diff --git a/test/data/ansible/check_ostree.yaml b/test/data/ansible/check_ostree.yaml
index e0327bb04..80538e445 100644
--- a/test/data/ansible/check_ostree.yaml
+++ b/test/data/ansible/check_ostree.yaml
@@ -5,6 +5,7 @@
     workspace: "{{ lookup('env', 'WORKSPACE') }}"
     skip_rollback_test: "false"
     fdo_credential: "false"
+    embeded_container: "false"
     total_counter: "0"
     failed_counter: "0"
 
@@ -150,7 +151,7 @@
     - name: check if it is simplified-installer or raw-image
       command: findmnt -r -o SOURCE -n /sysroot
       register: result_encrypted
-    
+
     - set_fact:
         device_name: "{{ result_encrypted.stdout }}"
       when: "'/dev/mapper/luks-' in result_encrypted.stdout"
@@ -342,6 +343,36 @@
       shell: dmesg --notime | grep -i "error\|fail" || true
       register: result_dmesg_error
 
+    - name: check embeded container image with podman
+      command: podman images
+      become: yes
+      register: result_podman_images
+      when:
+        - embeded_container == "true"
+        - (ansible_facts['distribution'] == 'RedHat' and ansible_facts ['distribution_version'] is version('9.0', '>')) or
+          (ansible_facts['distribution'] == 'RedHat' and ansible_facts ['distribution_version'] is version('8.6', '>') and ansible_facts ['distribution_version'] is version('9.0', '!=')) or
+          (ansible_facts['distribution'] == 'CentOS')
+
+    - name: embded container should be listed by podman images
+      block:
+        - assert:
+            that:
+              - "'quay.io/fedora/fedora' in result_podman_images.stdout"
+            fail_msg: "fedora image is not built in image"
+            success_msg: "fedora image is built in image"
+      always:
+        - set_fact:
+            total_counter: "{{ total_counter | int + 1 }}"
+      rescue:
+        - name: failed count + 1
+          set_fact:
+            failed_counter: "{{ failed_counter | int + 1 }}"
+      when:
+        - embeded_container == "true"
+        - (ansible_facts['distribution'] == 'RedHat' and ansible_facts ['distribution_version'] is version('9.0', '>')) or
+          (ansible_facts['distribution'] == 'RedHat' and ansible_facts ['distribution_version'] is version('8.6', '>') and ansible_facts ['distribution_version'] is version('9.0', '!=')) or
+          (ansible_facts['distribution'] == 'CentOS')
+
     # case: check running container with podman
     - name: run ubi8 image
       command: podman run registry.access.redhat.com/ubi8/ubi-minimal:latest cat /etc/redhat-release