Repository: osbuild/osbuild-composer
Pull request #2995 (2022-09-16 01:19:29)

test: Add Fedora 36 and 37 and remove 35 test for ostree.sh
1. Fedora 35 will be EOL, it's removed from `ostree.sh` and enable __Fedora 36__ and __37__ instead.
2. Fedora 35 is removed from `ostree-ng.sh` as well.

henrywang: @achilleas-k @gicmo I found some issues about Fedora 37 support:
1. `/sysroot` is mounted as __ro__ on Fedora 37 with `edge-commit`, but not with `edge-raw-image`
2. `/sysroot` and `/var` are not remounted on Fedora 37 with `edge-commit`, but not with `edge-raw-image`
3. When build `edge-installer` on Fedora 37, I found a very interesting log and an error log:
```
35.7 > 35.6
DEBUG: 35.7 >= 35.6
```
```
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="  org.osbuild.bootiso.mono failure:" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    /usr/lib/tmpfiles.d/journal-nocow.conf:26: Failed to resolve specifier: uninitialized /etc detected, skipping" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    All rules containing unresolvable specifiers will be skipped." jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    Failed to open file \"/sys/fs/selinux/checkreqprot\": Read-only file system" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    Traceback (most recent call last):" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="      File \"/run/osbuild/bin/org.osbuild.bootiso.mono\", line 435, in <module>" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="        ret = main(args[\"inputs\"]," jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="              ^^^^^^^^^^^^^^^^^^^^" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="      File \"/run/osbuild/bin/org.osbuild.bootiso.mono\", line 400, in main" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="        make_isolinux(isolinux, root, info, tree)" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="      File \"/run/osbuild/bin/org.osbuild.bootiso.mono\", line 318, in make_isolinux" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="        install(src, dst)" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="      File \"/run/osbuild/bin/org.osbuild.bootiso.mono\", line 165, in install" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="        shutil.copyfile(src, dst)" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="      File \"/usr/lib64/python3.11/shutil.py\", line 256, in copyfile" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="        with open(src, 'rb') as fsrc:" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="             ^^^^^^^^^^^^^^^" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/lorax/templates.d/99-generic/config_files/x86/grub.conf'" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    " jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
```
achilleas-k: > @achilleas-k @gicmo: Issue #2702 and issue #2703 are fixed and found some issues about Fedora 37 support:
> 
> 1. `/sysroot` is mounted as **ro** on Fedora 37 with `edge-commit`, but not with `edge-raw-image`
> 2. `/sysroot` and `/var` are not remounted on Fedora 37 with `edge-commit`, but not with `edge-raw-image`

The edge-commit (iot-commit) is installed by using our own kickstart in the test, right?  For the raw-image, we configure ostree with `sysroot.readonly = true`.  My guess is the kickstart `ostreesetup` command doesn't do this?

> 3. When build edge image on Fedora 36 and 37, I found a very interesting log:
> 
> from Fedora 37
> 
> ```
> 35.7 > 35.6
> DEBUG: 35.7 >= 35.6
> ```
> 
> and from Fedora 36
> 
> ```
> 35.5 < 35.6
> ```

These are from a function in the tests that checks for the version of `composer-cli` to parse the output appropriately.  They're just debug messages, but we should stop printing them since they're just noise.

> 
> 5. When build `edge-installer` on Fedora 37, it's failed with error log:
> 
> ```
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="  org.osbuild.bootiso.mono failure:" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    /usr/lib/tmpfiles.d/journal-nocow.conf:26: Failed to resolve specifier: uninitialized /etc detected, skipping" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    All rules containing unresolvable specifiers will be skipped." jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    Failed to open file \"/sys/fs/selinux/checkreqprot\": Read-only file system" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    Traceback (most recent call last):" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="      File \"/run/osbuild/bin/org.osbuild.bootiso.mono\", line 435, in <module>" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="        ret = main(args[\"inputs\"]," jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="              ^^^^^^^^^^^^^^^^^^^^" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="      File \"/run/osbuild/bin/org.osbuild.bootiso.mono\", line 400, in main" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="        make_isolinux(isolinux, root, info, tree)" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="      File \"/run/osbuild/bin/org.osbuild.bootiso.mono\", line 318, in make_isolinux" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="        install(src, dst)" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="      File \"/run/osbuild/bin/org.osbuild.bootiso.mono\", line 165, in install" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="        shutil.copyfile(src, dst)" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="      File \"/usr/lib64/python3.11/shutil.py\", line 256, in copyfile" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="        with open(src, 'rb') as fsrc:" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="             ^^^^^^^^^^^^^^^" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    FileNotFoundError: [Errno 2] No such file or directory: '/usr/share/lorax/templates.d/99-generic/config_files/x86/grub.conf'" jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> Sep 16 02:45:55 fedora-37 osbuild-worker[3104]: time="2022-09-16T02:45:55Z" level=info msg="    " jobId=4a28f296-b73d-4e66-ba13-dafac4a5e84d
> ```

This is a known issue and is being worked on: https://github.com/osbuild/osbuild/issues/1115 and https://issues.redhat.com/browse/COMPOSER-1758

henrywang: @achilleas-k Thanks for your update!
`The edge-commit (iot-commit) is installed by using our own kickstart in the test, right? For the raw-image, we configure ostree with sysroot.readonly = true. My guess is the kickstart ostreesetup command doesn't do this?`
Correct. iot-commit uses our own kickstart. But same kickstart used by Fedora 36, it does not have this issue. I suspect the `ostree-remount` service might changed.

Patch:
diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
index c7f0dbfb7..dd9f0c94c 100644
--- a/.gitlab-ci.yml
+++ b/.gitlab-ci.yml
@@ -355,9 +355,8 @@ OSTree:
   parallel:
     matrix:
       - RUNNER:
-          - rhos-01/fedora-35-x86_64
-          # - rhos-01/fedora-36-x86_64
-          # - rhos-01/fedora-37-x86_64
+          - rhos-01/fedora-36-x86_64
+          - rhos-01/fedora-37-x86_64
           - rhos-01/rhel-8.4-ga-x86_64
           - rhos-01/rhel-8.6-ga-x86_64
           - rhos-01/rhel-9.0-ga-x86_64
@@ -375,7 +374,6 @@ New OSTree:
   parallel:
     matrix:
       - RUNNER:
-          - rhos-01/fedora-35-x86_64
           - rhos-01/fedora-36-x86_64
           - rhos-01/fedora-37-x86_64
           - rhos-01/rhel-8.7-nightly-x86_64
diff --git a/internal/blueprint/blueprint.go b/internal/blueprint/blueprint.go
index 7efc9dfed..1c96edf0a 100644
--- a/internal/blueprint/blueprint.go
+++ b/internal/blueprint/blueprint.go
@@ -67,6 +67,10 @@ func (b *Blueprint) DeepCopy() Blueprint {
 
 // Initialize ensures that the blueprint has sane defaults for any missing fields
 func (b *Blueprint) Initialize() error {
+	if len(b.Name) == 0 {
+		return fmt.Errorf("empty blueprint name not allowed")
+	}
+
 	if b.Packages == nil {
 		b.Packages = []Package{}
 	}
diff --git a/internal/cloudapi/v2/handler.go b/internal/cloudapi/v2/handler.go
index 3242ce6d9..ee3813dc3 100644
--- a/internal/cloudapi/v2/handler.go
+++ b/internal/cloudapi/v2/handler.go
@@ -132,7 +132,7 @@ func (h *apiHandlers) PostCompose(ctx echo.Context) error {
 		return HTTPError(ErrorUnsupportedDistribution)
 	}
 
-	var bp = blueprint.Blueprint{}
+	var bp = blueprint.Blueprint{Name: "empty blueprint"}
 	err = bp.Initialize()
 	if err != nil {
 		return HTTPErrorWithInternal(ErrorFailedToInitializeBlueprint, err)
diff --git a/internal/distro/fedora/distro.go b/internal/distro/fedora/distro.go
index ac89a2949..f1795d3d7 100644
--- a/internal/distro/fedora/distro.go
+++ b/internal/distro/fedora/distro.go
@@ -654,7 +654,7 @@ func (t *imageType) Manifest(customizations *blueprint.Customizations,
 	containers []container.Spec,
 	seed int64) (distro.Manifest, error) {
 
-	bp := &blueprint.Blueprint{}
+	bp := &blueprint.Blueprint{Name: "empty blueprint"}
 	err := bp.Initialize()
 	if err != nil {
 		panic("could not initialize empty blueprint: " + err.Error())
diff --git a/internal/store/store.go b/internal/store/store.go
index 400ddfd25..bcc37dfee 100644
--- a/internal/store/store.go
+++ b/internal/store/store.go
@@ -209,17 +209,13 @@ func (s *Store) GetBlueprintChanges(name string) []blueprint.Change {
 
 func (s *Store) PushBlueprint(bp blueprint.Blueprint, commitMsg string) error {
 	return s.change(func() error {
-		if len(bp.Name) == 0 {
-			return fmt.Errorf("empty blueprint name not allowed")
-		}
-
-		commit, err := randomSHA1String()
+		// Make sure the blueprint has default values and that the version is valid
+		err := bp.Initialize()
 		if err != nil {
 			return err
 		}
 
-		// Make sure the blueprint has default values and that the version is valid
-		err = bp.Initialize()
+		commit, err := randomSHA1String()
 		if err != nil {
 			return err
 		}
diff --git a/internal/weldr/api.go b/internal/weldr/api.go
index 73c7dd5f5..aff0364aa 100644
--- a/internal/weldr/api.go
+++ b/internal/weldr/api.go
@@ -1945,6 +1945,17 @@ func (api *API) blueprintsNewHandler(writer http.ResponseWriter, request *http.R
 		}
 	}
 
+	// Make sure the blueprint has default values and that the version is valid
+	err = blueprint.Initialize()
+	if err != nil {
+		errors := responseError{
+			ID:  "BlueprintsError",
+			Msg: err.Error(),
+		}
+		statusResponseError(writer, http.StatusBadRequest, errors)
+		return
+	}
+
 	commitMsg := "Recipe " + blueprint.Name + ", version " + blueprint.Version + " saved."
 	err = api.store.PushBlueprint(blueprint, commitMsg)
 	if err != nil {


Test patch:
diff --git a/internal/weldr/api_test.go b/internal/weldr/api_test.go
index ba4995f5a..483d9a452 100644
--- a/internal/weldr/api_test.go
+++ b/internal/weldr/api_test.go
@@ -568,6 +568,13 @@ func TestBlueprintsChanges(t *testing.T) {
 	test.SendHTTP(api, true, "POST", "/api/v0/blueprints/new", `{"name":"`+id+`","description":"Test","packages":[{"name":"httpd","version":"2.4.*"}],"version":"0.0.0"}`)
 	test.TestRoute(t, api, true, "GET", "/api/v0/blueprints/changes/"+id, ``, http.StatusOK, `{"blueprints":[{"changes":[{"commit":"","message":"Recipe `+id+`, version 0.0.0 saved.","revision":null,"timestamp":""},{"commit":"","message":"Recipe `+id+`, version 0.0.0 saved.","revision":null,"timestamp":""}],"name":"`+id+`","total":2}],"errors":[],"limit":20,"offset":0}`, ignoreFields...)
 	test.SendHTTP(api, true, "DELETE", "/api/v0/blueprints/delete/"+id, ``)
+
+	// Test with an empty Version
+	/* #nosec G404 */
+	id = strconv.Itoa(rand.Int())
+	test.SendHTTP(api, true, "POST", "/api/v0/blueprints/new", `{"name":"`+id+`","description":"Test","packages":[{"name":"httpd","version":"2.4.*"}],"version":""}`)
+	test.TestRoute(t, api, true, "GET", "/api/v0/blueprints/changes/"+id, ``, http.StatusOK, `{"blueprints":[{"changes":[{"commit":"","message":"Recipe `+id+`, version 0.0.0 saved.","revision":null,"timestamp":""}],"name":"`+id+`","total":1}],"errors":[],"limit":20,"offset":0}`, ignoreFields...)
+	test.SendHTTP(api, true, "DELETE", "/api/v0/blueprints/delete/"+id, ``)
 }
 
 func TestBlueprintsDepsolve(t *testing.T) {
diff --git a/test/cases/ostree.sh b/test/cases/ostree.sh
index 70a27649a..ef015cf1e 100755
--- a/test/cases/ostree.sh
+++ b/test/cases/ostree.sh
@@ -15,14 +15,6 @@ fi
 
 # Set os-variant and boot location used by virt-install.
 case "${ID}-${VERSION_ID}" in
-    "fedora-35")
-        IMAGE_TYPE=iot-commit
-        OSTREE_REF="fedora/35/${ARCH}/iot"
-        OS_VARIANT="fedora35"
-        USER_IN_COMMIT="false"
-        BOOT_LOCATION="https://mirrors.rit.edu/fedora/fedora/linux/releases/35/Everything/x86_64/os/"
-        EMBEDED_CONTAINER="false"
-        ;;
     "fedora-36")
         IMAGE_TYPE=iot-commit
         OSTREE_REF="fedora/36/${ARCH}/iot"