Repository: kubevirt/containerized-data-importer
Pull request #3704: uploadserver, tests: include error message for upload failure due to lack of space
Author: login: Acedus
Labels: name: release-note-none
description: Denotes a PR that doesn't merit a release note.
name: size/M
name: dco-signoff: yes
description: Indicates the PR's author has DCO signed all their commits.
name: lgtm
description: Indicates that a PR is ready to be merged.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
Created: 2025-04-17T14:02:02Z | Merged: 2025-04-23T20:57:04Z

<!--  Thanks for sending a pull request!  Here are some tips for you:
1. Follow the instructions for writing a release note from k8s: https://git.k8s.io/community/contributors/guide/release-notes.md
-->

**What this PR does / why we need it**:
The uploadserver features two handler functions for sync and async uploads.

The async upload handler correctly reports an error message if QEMU image size validation errors due to lack of space in the target by appending the error message to the HTTP response. The sync upload handler, however, does not.

This commit makes error handling for failed upload identical between both handlers. Furthermore, it also checks for ENOSPC errors during write as can happen when the image is streamed directly into target instead of going through scratch space first (e.g., raw images). It also consolidates the sync and async upload failure e2e tests into a single DescribeTable and improves upon its clarity.

**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
Fixes:
- https://issues.redhat.com/browse/CNV-57559

**Special notes for your reviewer**:

**Release note**:
<!--  Write your release note:
1. Enter your extended release note in the below block. If the PR requires additional action from users switching to the new release, include the string "action required".
2. If no release note is required, just write "NONE".
-->
```release-note
none
```




Diff:
diff --git a/pkg/uploadserver/uploadserver.go b/pkg/uploadserver/uploadserver.go
index 86f3f5a898..bef30af7c3 100644
--- a/pkg/uploadserver/uploadserver.go
+++ b/pkg/uploadserver/uploadserver.go
@@ -369,25 +369,13 @@ func (app *uploadServerApp) uploadHandlerAsync(irc imageReadCloser) http.Handler
 
 		app.mutex.Lock()
 		defer app.mutex.Unlock()
+		app.uploading = false
 
 		if err != nil {
-			klog.Errorf("Saving stream failed: %s", err)
-			if errors.As(err, &importer.ValidationSizeError{}) {
-				w.WriteHeader(http.StatusBadRequest)
-			} else {
-				w.WriteHeader(http.StatusInternalServerError)
-			}
-
-			_, writeErr := fmt.Fprintf(w, "Saving stream failed: %s", err.Error())
-			if writeErr != nil {
-				klog.Errorf("failed to send response; %v", err)
-			}
-
-			app.uploading = false
+			handleStreamError(w, err)
 			return
 		}
 
-		app.uploading = false
 		app.processing = true
 
 		// Start processing.
@@ -433,8 +421,7 @@ func (app *uploadServerApp) processUpload(irc imageReadCloser, w http.ResponseWr
 	app.uploading = false
 
 	if err != nil {
-		klog.Errorf("Saving stream failed: %s", err)
-		w.WriteHeader(http.StatusInternalServerError)
+		handleStreamError(w, err)
 		return
 	}
 
@@ -586,3 +573,18 @@ func newSnappyReadCloser(stream io.ReadCloser) io.ReadCloser {
 		closers: []io.Closer{stream},
 	}
 }
+
+func handleStreamError(w http.ResponseWriter, err error) {
+	if errors.As(err, &importer.ValidationSizeError{}) || strings.Contains(err.Error(), "no space left on device") {
+		w.WriteHeader(http.StatusBadRequest)
+		err = errors.New("effective image size is larger than the reported available storage. A larger PVC is required")
+	} else {
+		w.WriteHeader(http.StatusInternalServerError)
+	}
+	klog.Errorf("Saving stream failed: %s", err)
+
+	_, writeErr := fmt.Fprintf(w, "Saving stream failed: %s", err.Error())
+	if writeErr != nil {
+		klog.Errorf("failed to send response; %v", err)
+	}
+}
diff --git a/tests/upload_test.go b/tests/upload_test.go
index 9c442d6969..4e96398f1a 100644
--- a/tests/upload_test.go
+++ b/tests/upload_test.go
@@ -286,7 +286,7 @@ var _ = Describe("[rfe_id:138][crit:high][vendor:cnv-qe@redhat.com][level:compon
 			Expect(string(body)).To(ContainSubstring(pvc.Name))
 		})
 
-		DescribeTable("Verify validation error message on async upload if virtual size > pvc size", Serial, func(filename string) {
+		DescribeTable("Verify failure on upload if effective size > pvc size", Label("no-kubernetes-in-docker"), Serial, func(filename, uploadPath string) {
 			By("Verify PVC annotation says ready")
 			found, err := utils.WaitPVCPodStatusReady(f.K8sClient, pvc)
 			Expect(err).ToNot(HaveOccurred())
@@ -299,31 +299,10 @@ var _ = Describe("[rfe_id:138][crit:high][vendor:cnv-qe@redhat.com][level:compon
 			Expect(token).ToNot(BeEmpty())
 
 			By("Do upload")
+			tokenExpiration := 5 * time.Minute
 			Eventually(func() error {
-				return uploadFileNameToPath(binaryRequestFunc, filename, uploadProxyURL, asyncUploadPath, token, http.StatusBadRequest)
-			}, timeout, pollingInterval).Should(BeNil(), "Upload should eventually succeed, even if initially pod is not ready")
-		},
-			Entry("[test_id:4989]fail given a large virtual size QCOW2 file", utils.UploadFileLargeVirtualDiskQcow),
-			Entry("fail given a large physical size QCOW2 file", utils.UploadFileLargePhysicalDiskQcow),
-		)
-
-		DescribeTable("[posneg:negative][test_id:2330]Verify failure on sync upload if virtual size > pvc size", Label("no-kubernetes-in-docker"), Serial, func(filename string) {
-			By("Verify PVC annotation says ready")
-			found, err := utils.WaitPVCPodStatusReady(f.K8sClient, pvc)
-			Expect(err).ToNot(HaveOccurred())
-			Expect(found).To(BeTrue())
-
-			var token string
-			By("Get an upload token")
-			token, err = utils.RequestUploadToken(f.CdiClient, pvc)
-			Expect(err).ToNot(HaveOccurred())
-			Expect(token).ToNot(BeEmpty())
-
-			By("Do upload")
-			Eventually(func() bool {
-				err = uploadFileNameToPath(binaryRequestFunc, filename, uploadProxyURL, syncUploadPath, token, http.StatusOK)
-				return err != nil && strings.Contains(err.Error(), "Unexpected return value 500")
-			}, timeout, pollingInterval).Should(BeTrue())
+				return uploadFileNameToPath(binaryRequestFunc, filename, uploadProxyURL, uploadPath, token, http.StatusBadRequest)
+			}, tokenExpiration, pollingInterval).Should(BeNil(), "Upload should fail with HTTP 400")
 
 			uploadPod, err := utils.FindPodByPrefix(f.K8sClient, f.Namespace.Name, utils.UploadPodName(pvc), common.CDILabelSelector)
 			Expect(err).NotTo(HaveOccurred(), fmt.Sprintf("Unable to get uploader pod %q", f.Namespace.Name+"/"+utils.UploadPodName(pvc)))
@@ -335,17 +314,14 @@ var _ = Describe("[rfe_id:138][crit:high][vendor:cnv-qe@redhat.com][level:compon
 					GetLogs(uploadPod.Name, &v1.PodLogOptions{SinceTime: &metav1.Time{Time: CurrentSpecReport().StartTime}}).
 					DoRaw(context.Background())
 				return string(out), err
-			}, controllerSkipPVCCompleteTimeout, assertionPollInterval).Should(Or(
-				ContainSubstring("is larger than the reported available"),
-				ContainSubstring("no space left on device"),
-				ContainSubstring("qemu-img execution failed"),
-				ContainSubstring("calculated new size is < than current size, not resizing"),
-			))
+			}, controllerSkipPVCCompleteTimeout, assertionPollInterval).Should(ContainSubstring("effective image size is larger than the reported available"))
 		},
-			Entry("fail given a large virtual size RAW XZ file", utils.UploadFileLargeVirtualDiskXz),
-			Entry("fail given a large virtual size QCOW2 file", utils.UploadFileLargeVirtualDiskQcow),
-			Entry("fail given a large physical size RAW XZ file", utils.UploadFileLargePhysicalDiskXz),
-			Entry("fail given a large physical size QCOW2 file", utils.UploadFileLargePhysicalDiskQcow),
+			Entry("with a large virtual size RAW XZ file sync", utils.UploadFileLargeVirtualDiskXz, syncUploadPath),
+			Entry("with a large physical size RAW XZ file sync", utils.UploadFileLargePhysicalDiskXz, syncUploadPath),
+			Entry("with a large physical size QCOW2 file sync", utils.UploadFileLargePhysicalDiskQcow, syncUploadPath),
+			Entry("with a large physical size QCOW2 file, async", utils.UploadFileLargePhysicalDiskQcow, asyncUploadPath),
+			Entry("with a large virtual size QCOW2 file sync", utils.UploadFileLargeVirtualDiskQcow, syncUploadPath),
+			Entry("[test_id:4989]with a large virtual size QCOW2 file async", utils.UploadFileLargeVirtualDiskQcow, asyncUploadPath),
 		)
 	})
 


Reviews:
user: login: akalenyu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-17T14:13:58Z
user: login: Acedus
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T14:18:18Z
user: login: akalenyu
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-17T14:26:00Z
user: login: Acedus
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T14:33:37Z
user: login: akalenyu
body: Looks good!
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-20T11:26:26Z
user: login: Acedus
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-20T11:41:42Z

Review comments:
diff_hunk: @@ -586,3 +573,18 @@ func newSnappyReadCloser(stream io.ReadCloser) io.ReadCloser {
 		closers: []io.Closer{stream},
 	}
 }
+
+func handleStreamError(w http.ResponseWriter, err error) {
+	if errors.As(err, &importer.ValidationSizeError{}) || strings.Contains(err.Error(), "no space left on device") {
path: pkg/uploadserver/uploadserver.go
user: login: akalenyu
body: I think you can extract and reuse this to catch all ENOSPC variations in the wilderness https://github.com/kubevirt/containerized-data-importer/blob/2b2d3781c86e32e1a41075f557f12610710fac7c/pkg/controller/util.go#L392-L396

If you decide to do that, could you also please add "disk quota exceeded" so we get a quick win with https://issues.redhat.com/browse/CNV-51463 :pray:
created_at: 2025-04-17T14:13:58Z
author_association: COLLABORATOR
diff_hunk: @@ -586,3 +573,18 @@ func newSnappyReadCloser(stream io.ReadCloser) io.ReadCloser {
 		closers: []io.Closer{stream},
 	}
 }
+
+func handleStreamError(w http.ResponseWriter, err error) {
+	if errors.As(err, &importer.ValidationSizeError{}) || strings.Contains(err.Error(), "no space left on device") {
path: pkg/uploadserver/uploadserver.go
user: login: Acedus
body: Problem is that it only simplifies the error message so I can't replace the if with it, and I can't replace the actual error message as the returned simplified message is "DataVolume too small to contain image" which is a bit misleading in the case of an upload to a regular PVC.
created_at: 2025-04-17T14:18:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -586,3 +573,18 @@ func newSnappyReadCloser(stream io.ReadCloser) io.ReadCloser {
 		closers: []io.Closer{stream},
 	}
 }
+
+func handleStreamError(w http.ResponseWriter, err error) {
+	if errors.As(err, &importer.ValidationSizeError{}) || strings.Contains(err.Error(), "no space left on device") {
path: pkg/uploadserver/uploadserver.go
user: login: akalenyu
body: Yeah I was thinking you'd only extract the if statement out to a common pkg
IsNoCapacityError or something
created_at: 2025-04-17T14:26:00Z
author_association: COLLABORATOR
diff_hunk: @@ -586,3 +573,18 @@ func newSnappyReadCloser(stream io.ReadCloser) io.ReadCloser {
 		closers: []io.Closer{stream},
 	}
 }
+
+func handleStreamError(w http.ResponseWriter, err error) {
+	if errors.As(err, &importer.ValidationSizeError{}) || strings.Contains(err.Error(), "no space left on device") {
path: pkg/uploadserver/uploadserver.go
user: login: Acedus
body: In that case sure. 
created_at: 2025-04-17T14:33:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -371,23 +371,11 @@ func (app *uploadServerApp) uploadHandlerAsync(irc imageReadCloser) http.Handler
 		defer app.mutex.Unlock()
 
 		if err != nil {
-			klog.Errorf("Saving stream failed: %s", err)
-			if errors.As(err, &importer.ValidationSizeError{}) {
-				w.WriteHeader(http.StatusBadRequest)
-			} else {
-				w.WriteHeader(http.StatusInternalServerError)
-			}
-
-			_, writeErr := fmt.Fprintf(w, "Saving stream failed: %s", err.Error())
-			if writeErr != nil {
-				klog.Errorf("failed to send response; %v", err)
-			}
-
+			handleStreamError(w, err)
 			app.uploading = false
 			return
 		}
 
-		app.uploading = false
path: pkg/uploadserver/uploadserver.go
user: login: akalenyu
body: Just a redundancy thing right? I'd keep it tbh or adjust the other handlers for consistency
created_at: 2025-04-20T11:18:35Z
author_association: COLLABORATOR
diff_hunk: @@ -335,17 +314,14 @@ var _ = Describe("[rfe_id:138][crit:high][vendor:cnv-qe@redhat.com][level:compon
 					GetLogs(uploadPod.Name, &v1.PodLogOptions{SinceTime: &metav1.Time{Time: CurrentSpecReport().StartTime}}).
 					DoRaw(context.Background())
 				return string(out), err
-			}, controllerSkipPVCCompleteTimeout, assertionPollInterval).Should(Or(
-				ContainSubstring("is larger than the reported available"),
-				ContainSubstring("no space left on device"),
-				ContainSubstring("qemu-img execution failed"),
-				ContainSubstring("calculated new size is < than current size, not resizing"),
-			))
+			}, controllerSkipPVCCompleteTimeout, assertionPollInterval).Should(ContainSubstring("effective image size is larger than the reported available"))
 		},
-			Entry("fail given a large virtual size RAW XZ file", utils.UploadFileLargeVirtualDiskXz),
-			Entry("fail given a large virtual size QCOW2 file", utils.UploadFileLargeVirtualDiskQcow),
-			Entry("fail given a large physical size RAW XZ file", utils.UploadFileLargePhysicalDiskXz),
-			Entry("fail given a large physical size QCOW2 file", utils.UploadFileLargePhysicalDiskQcow),
+			Entry("with a large virtual size RAW XZ file sync", utils.UploadFileLargeVirtualDiskXz, syncUploadPath),
+			Entry("with a large physical size RAW XZ file sync", utils.UploadFileLargePhysicalDiskXz, syncUploadPath),
+			Entry("with a large physical size QCOW2 file sync", utils.UploadFileLargePhysicalDiskQcow, syncUploadPath),
+			Entry("with a large physical size QCOW2 file, async", utils.UploadFileLargePhysicalDiskQcow, asyncUploadPath),
+			Entry("with a large virtual size QCOW2 file sync", utils.UploadFileLargeVirtualDiskQcow, syncUploadPath),
+			Entry("[test_id:4989]with a large virtual size QCOW2 file async", utils.UploadFileLargeVirtualDiskQcow, asyncUploadPath),
path: tests/upload_test.go
user: login: akalenyu
body: :+1: 
created_at: 2025-04-20T11:23:26Z
author_association: COLLABORATOR
diff_hunk: @@ -371,23 +371,11 @@ func (app *uploadServerApp) uploadHandlerAsync(irc imageReadCloser) http.Handler
 		defer app.mutex.Unlock()
 
 		if err != nil {
-			klog.Errorf("Saving stream failed: %s", err)
-			if errors.As(err, &importer.ValidationSizeError{}) {
-				w.WriteHeader(http.StatusBadRequest)
-			} else {
-				w.WriteHeader(http.StatusInternalServerError)
-			}
-
-			_, writeErr := fmt.Fprintf(w, "Saving stream failed: %s", err.Error())
-			if writeErr != nil {
-				klog.Errorf("failed to send response; %v", err)
-			}
-
+			handleStreamError(w, err)
 			app.uploading = false
 			return
 		}
 
-		app.uploading = false
path: pkg/uploadserver/uploadserver.go
user: login: Acedus
body: Oversight on my part, thanks.
created_at: 2025-04-20T11:41:41Z
author_association: CONTRIBUTOR

PR comments:
user: login: Acedus
created_at: 2025-04-17T14:02:11Z
body: /cc @akalenyu 
author_association: CONTRIBUTOR
user: login: coveralls
created_at: 2025-04-17T14:12:26Z
body: 
[![Coverage Status](https://coveralls.io/builds/73302178/badge)](https://coveralls.io/builds/73302178)

coverage: 59.212% (+0.04%) from 59.172%
when pulling **64cdac4459f7da1abab1b7eb7cb985d5fc6f908a on Acedus:report-upload-error-message-on-write-error**
into **6d0e96133c39390ed98f55e76c565e27349d6d28 on kubevirt:main**.

author_association: NONE
user: login: akalenyu
created_at: 2025-04-17T16:28:24Z
body: /test pull-containerized-data-importer-fossa
author_association: COLLABORATOR
user: login: Acedus
created_at: 2025-04-20T07:22:27Z
body: [Change](https://github.com/kubevirt/containerized-data-importer/compare/2b2d3781c86e32e1a41075f557f12610710fac7c..d57b16b7566970e42b60f160c5af8267a01ebb6c) - Addressed @akalenyu comments.

I've opted to introduce the error matcher as part of importer/errors as it made the most sense and further polluting util seemed counter-productive. 

EDIT: @akalenyu I'd argue this change warrants a PR of its own so we can use it right away throughout the codebase. 
author_association: CONTRIBUTOR
user: login: Acedus
created_at: 2025-04-20T08:42:04Z
body: [Change](https://github.com/kubevirt/containerized-data-importer/compare/d57b16b7566970e42b60f160c5af8267a01ebb6c..f5e1f9e7b39b268d417d1a7a86dceb5141d7fd7e) - Reverted previous change in favor of introducing it in another PR.
author_association: CONTRIBUTOR
user: login: Acedus
created_at: 2025-04-20T11:42:21Z
body: [Change](https://github.com/kubevirt/containerized-data-importer/compare/f5e1f9e7b39b268d417d1a7a86dceb5141d7fd7e..64cdac4459f7da1abab1b7eb7cb985d5fc6f908a) - Addressed @akalenyu comment.
author_association: CONTRIBUTOR
user: login: akalenyu
created_at: 2025-04-20T11:48:29Z
body: /approve
/cc @mhenriks
author_association: COLLABORATOR
user: login: kubevirt-bot
created_at: 2025-04-20T11:48:33Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubevirt/containerized-data-importer/pull/3704#issuecomment-2817132868" title="Approved">akalenyu</a>*

The full list of commands accepted by this bot can be found [here](https://prow.ci.kubevirt.io/command-help?repo=kubevirt%2Fcontainerized-data-importer).

The pull request process is described [here](https://github.com/kubevirt/community/blob/main/docs/add-merge-automation-to-your-repository.md#example-workflow)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubevirt/containerized-data-importer/blob/main/OWNERS)~~ [akalenyu]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: awels
created_at: 2025-04-23T13:10:14Z
body: /lgtm
author_association: MEMBER
user: login: akalenyu
created_at: 2025-04-24T07:13:50Z
body: /cherrypick release-v1.62
author_association: COLLABORATOR
user: login: kubevirt-bot
created_at: 2025-04-24T07:14:44Z
body: @akalenyu: new pull request created: #3713

<details>

In response to [this](https://github.com/kubevirt/containerized-data-importer/pull/3704#issuecomment-2826609311):

>/cherrypick release-v1.62


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>
author_association: CONTRIBUTOR