Repository: minio/minio (language: go)

Issue:
with mincache enabled `HEAD` request returns different `Content-Length` then matching `GET` on a KMS setup
I have enabled server-side encryption on my bucket and uploaded `Audi cabriolet 01.jpg` of 2,971,514 bytes to the `test` bucket. When I try to copy the file using `mc`, then I got the following error: `Input reader closed pre-maturely. Expected '2972986' bytes, but only received '2971514' bytes.`

I ran the `mc` command using the `--debug` flag and noticed that the `HEAD` request returns a different length then the `GET` request. This is the `HEAD` request:
```
HEAD /test/audi/Audi%20cabriolet%2001.jpg HTTP/1.1
Host: tenant-rdk-hl.tenant-rdk:9000
User-Agent: MinIO (linux; amd64) minio-go/v7.0.70 __debug_bin2996553503/DEVELOPMENT.GOGET
Authorization: AWS4-HMAC-SHA256 Credential=l5m7mQiwUi2GCSH3/20240612/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240612T160723Z
```
`HEAD` response:
```
HTTP/1.1 200 OK
Content-Length: 2972986
Accept-Ranges: bytes
Content-Type: image/jpeg
Date: Wed, 12 Jun 2024 16:07:23 GMT
Etag: "efd738945e94f4dbb8588b788b2ea7c1"
Last-Modified: Wed, 12 Jun 2024 15:21:24 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: d5a265f7265c149d31b79b2568737f9e0d97aab546838b9090d7403265887733
X-Amz-Request-Id: 17D84D79C009F04A
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
```
Then it performs the actual `GET` and that request looks like:
```
GET /test/audi/Audi%20cabriolet%2001.jpg HTTP/1.1
Host: tenant-rdk-hl.tenant-rdk:9000
User-Agent: MinIO (linux; amd64) minio-go/v7.0.70 __debug_bin2996553503/DEVELOPMENT.GOGET
Accept-Encoding: identity
Authorization: AWS4-HMAC-SHA256 Credential=l5m7mQiwUi2GCSH3/20240612/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=**REDACTED**
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240612T160904Z
```
And the response is:
```
HTTP/1.1 200 OK
Content-Length: 2971514
Accept-Ranges: bytes
Content-Type: image/jpeg
Date: Wed, 12 Jun 2024 16:09:04 GMT
Etag: "efd738945e94f4dbb8588b788b2ea7c1"
Last-Modified: Wed, 12 Jun 2024 15:21:24 GMT
Server: MinIO
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin
Vary: Accept-Encoding
X-Amz-Id-2: d5a265f7265c149d31b79b2568737f9e0d97aab546838b9090d7403265887733
X-Amz-Request-Id: 17D84D916E7CC820
X-Amz-Server-Side-Encryption: aws:kms
X-Amz-Server-Side-Encryption-Aws-Kms-Key-Id: arn:aws:kms:tenant-rdk-key
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block
```
This seems to be a bug in Minio, because both the `HEAD` and `GET` requests should return the same content length. I never have seen this issue without encryption.



Discussion / hints:
There is a cache in-between `mc` and Minio. Disabling the cache fixes the issue and Minio does return the proper content-size and encryption headers.
You have uploaded different size at the same object. 
```
root@DESKTOP-QPDA04R:/mnt/c/workspace/go/src/minkms/dist# mc cp minio9000/mytest/dlv.exe dlv.exe 
mc: <ERROR> Failed to copy `http://192.168.2.6:9000/mytest/dlv.exe`. Input reader closed pre-maturely. Expected `19311104` bytes, but only received `10248344` bytes.
root@DESKTOP-QPDA04R:/mnt/c/workspace/go/src/minkms/dist# 
```
No @jiuker you are not understanding the issue. Please DM me.
Restarting mincache will fix the problem? @ramondeklein 
@jiuker I will try to create a Docker image using this PR and check if the problem still exists.
> @jiuker I will try to create a Docker image using this PR and check if the problem still exists.

Yesh. Please sync the result. Maybe you have another case here. @ramondeklein 
this is still not fully fixed yet. 
Please try latest master and also https://github.com/miniohq/mincache/pull/35 @ramondeklein and let us know if it works for you. 

Thanks 

Fix patch:
diff --git a/cmd/bucket-handlers.go b/cmd/bucket-handlers.go
index 60fad18ae82be..dfd600134a277 100644
--- a/cmd/bucket-handlers.go
+++ b/cmd/bucket-handlers.go
@@ -51,7 +51,6 @@ import (
 	sse "github.com/minio/minio/internal/bucket/encryption"
 	objectlock "github.com/minio/minio/internal/bucket/object/lock"
 	"github.com/minio/minio/internal/bucket/replication"
-	"github.com/minio/minio/internal/config/cache"
 	"github.com/minio/minio/internal/config/dns"
 	"github.com/minio/minio/internal/crypto"
 	"github.com/minio/minio/internal/event"
@@ -1342,22 +1341,6 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
 					continue
 				}
 
-				asize, err := objInfo.GetActualSize()
-				if err != nil {
-					asize = objInfo.Size
-				}
-
-				globalCacheConfig.Set(&cache.ObjectInfo{
-					Key:          objInfo.Name,
-					Bucket:       objInfo.Bucket,
-					ETag:         getDecryptedETag(formValues, objInfo, false),
-					ModTime:      objInfo.ModTime,
-					Expires:      objInfo.Expires.UTC().Format(http.TimeFormat),
-					CacheControl: objInfo.CacheControl,
-					Metadata:     cleanReservedKeys(objInfo.UserDefined),
-					Size:         asize,
-				})
-
 				fanOutResp = append(fanOutResp, minio.PutObjectFanOutResponse{
 					Key:          objInfo.Name,
 					ETag:         getDecryptedETag(formValues, objInfo, false),
@@ -1452,22 +1435,6 @@ func (api objectAPIHandlers) PostPolicyBucketHandler(w http.ResponseWriter, r *h
 		w.Header().Set(xhttp.Location, obj)
 	}
 
-	asize, err := objInfo.GetActualSize()
-	if err != nil {
-		asize = objInfo.Size
-	}
-
-	defer globalCacheConfig.Set(&cache.ObjectInfo{
-		Key:          objInfo.Name,
-		Bucket:       objInfo.Bucket,
-		ETag:         etag,
-		ModTime:      objInfo.ModTime,
-		Expires:      objInfo.ExpiresStr(),
-		CacheControl: objInfo.CacheControl,
-		Metadata:     cleanReservedKeys(objInfo.UserDefined),
-		Size:         asize,
-	})
-
 	// Notify object created event.
 	defer sendEvent(eventArgs{
 		EventName:    event.ObjectCreatedPost,
diff --git a/cmd/object-handlers.go b/cmd/object-handlers.go
index e0100249c1ccb..e5ac51a2de7e1 100644
--- a/cmd/object-handlers.go
+++ b/cmd/object-handlers.go
@@ -610,8 +610,6 @@ func (api objectAPIHandlers) getObjectHandler(ctx context.Context, objectAPI Obj
 			w.Header().Set(xhttp.AmzServerSideEncryptionCustomerAlgorithm, r.Header.Get(xhttp.AmzServerSideEncryptionCustomerAlgorithm))
 			w.Header().Set(xhttp.AmzServerSideEncryptionCustomerKeyMD5, r.Header.Get(xhttp.AmzServerSideEncryptionCustomerKeyMD5))
 		}
-		// Never store encrypted objects in cache.
-		update = false
 		objInfo.ETag = getDecryptedETag(r.Header, objInfo, false)
 	}
 
@@ -620,39 +618,6 @@ func (api objectAPIHandlers) getObjectHandler(ctx context.Context, objectAPI Obj
 		hash.AddChecksumHeader(w, objInfo.decryptChecksums(opts.PartNumber, r.Header))
 	}
 
-	var buf *bytebufferpool.ByteBuffer
-	if update {
-		if globalCacheConfig.MatchesSize(objInfo.Size) {
-			buf = bytebufferpool.Get()
-			defer bytebufferpool.Put(buf)
-		}
-		defer func() {
-			var data []byte
-			if buf != nil {
-				data = buf.Bytes()
-			}
-
-			asize, err := objInfo.GetActualSize()
-			if err != nil {
-				asize = objInfo.Size
-			}
-
-			globalCacheConfig.Set(&cache.ObjectInfo{
-				Key:          objInfo.Name,
-				Bucket:       objInfo.Bucket,
-				ETag:         objInfo.ETag,
-				ModTime:      objInfo.ModTime,
-				Expires:      objInfo.ExpiresStr(),
-				CacheControl: objInfo.CacheControl,
-				Metadata:     cleanReservedKeys(objInfo.UserDefined),
-				Range:        rangeHeader,
-				PartNumber:   opts.PartNumber,
-				Size:         asize,
-				Data:         data,
-			})
-		}()
-	}
-
 	if err = setObjectHeaders(ctx, w, objInfo, rs, opts); err != nil {
 		writeErrorResponse(ctx, w, toAPIError(ctx, err), r.URL)
 		return
@@ -665,6 +630,12 @@ func (api objectAPIHandlers) getObjectHandler(ctx context.Context, objectAPI Obj
 
 	setHeadGetRespHeaders(w, r.Form)
 
+	var buf *bytebufferpool.ByteBuffer
+	if update && globalCacheConfig.MatchesSize(objInfo.Size) {
+		buf = bytebufferpool.Get()
+		defer bytebufferpool.Put(buf)
+	}
+
 	var iw io.Writer
 	iw = w
 	if buf != nil {
@@ -696,6 +667,31 @@ func (api objectAPIHandlers) getObjectHandler(ctx context.Context, objectAPI Obj
 		return
 	}
 
+	defer func() {
+		var data []byte
+		if buf != nil {
+			data = buf.Bytes()
+		}
+
+		if len(data) == 0 {
+			return
+		}
+
+		globalCacheConfig.Set(&cache.ObjectInfo{
+			Key:          objInfo.Name,
+			Bucket:       objInfo.Bucket,
+			ETag:         objInfo.ETag,
+			ModTime:      objInfo.ModTime,
+			Expires:      objInfo.ExpiresStr(),
+			CacheControl: objInfo.CacheControl,
+			Metadata:     cleanReservedKeys(objInfo.UserDefined),
+			Range:        rangeHeader,
+			PartNumber:   opts.PartNumber,
+			Size:         int64(len(data)),
+			Data:         data,
+		})
+	}()
+
 	// Notify object accessed via a GET request.
 	sendEvent(eventArgs{
 		EventName:    event.ObjectAccessedGet,
@@ -1939,22 +1935,6 @@ func (api objectAPIHandlers) CopyObjectHandler(w http.ResponseWriter, r *http.Re
 		Host:         handlers.GetSourceIP(r),
 	})
 
-	asize, err := objInfo.GetActualSize()
-	if err != nil {
-		asize = objInfo.Size
-	}
-
-	defer globalCacheConfig.Set(&cache.ObjectInfo{
-		Key:          objInfo.Name,
-		Bucket:       objInfo.Bucket,
-		ETag:         objInfo.ETag,
-		ModTime:      objInfo.ModTime,
-		Expires:      objInfo.ExpiresStr(),
-		CacheControl: objInfo.CacheControl,
-		Size:         asize,
-		Metadata:     cleanReservedKeys(objInfo.UserDefined),
-	})
-
 	if !remoteCallRequired && !globalTierConfigMgr.Empty() {
 		// Schedule object for immediate transition if eligible.
 		objInfo.ETag = origETag
@@ -2343,9 +2323,8 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
 			data = buf.Bytes()
 		}
 
-		asize, err := objInfo.GetActualSize()
-		if err != nil {
-			asize = objInfo.Size
+		if len(data) == 0 {
+			return
 		}
 
 		globalCacheConfig.Set(&cache.ObjectInfo{
@@ -2355,7 +2334,7 @@ func (api objectAPIHandlers) PutObjectHandler(w http.ResponseWriter, r *http.Req
 			ModTime:      objInfo.ModTime,
 			Expires:      objInfo.ExpiresStr(),
 			CacheControl: objInfo.CacheControl,
-			Size:         asize,
+			Size:         int64(len(data)),
 			Metadata:     cleanReservedKeys(objInfo.UserDefined),
 			Data:         data,
 		})
@@ -2712,22 +2691,6 @@ func (api objectAPIHandlers) PutObjectExtractHandler(w http.ResponseWriter, r *h
 			scheduleReplication(ctx, objInfo, objectAPI, dsc, replication.ObjectReplicationType)
 		}
 
-		asize, err := objInfo.GetActualSize()
-		if err != nil {
-			asize = objInfo.Size
-		}
-
-		defer globalCacheConfig.Set(&cache.ObjectInfo{
-			Key:          objInfo.Name,
-			Bucket:       objInfo.Bucket,
-			ETag:         objInfo.ETag,
-			ModTime:      objInfo.ModTime,
-			Expires:      objInfo.ExpiresStr(),
-			CacheControl: objInfo.CacheControl,
-			Size:         asize,
-			Metadata:     cleanReservedKeys(objInfo.UserDefined),
-		})
-
 		// Notify object created event.
 		evt := eventArgs{
 			EventName:    event.ObjectCreatedPut,
diff --git a/cmd/object-multipart-handlers.go b/cmd/object-multipart-handlers.go
index 2f86bc55c323a..fe267b6ee9c3b 100644
--- a/cmd/object-multipart-handlers.go
+++ b/cmd/object-multipart-handlers.go
@@ -36,7 +36,6 @@ import (
 	sse "github.com/minio/minio/internal/bucket/encryption"
 	objectlock "github.com/minio/minio/internal/bucket/object/lock"
 	"github.com/minio/minio/internal/bucket/replication"
-	"github.com/minio/minio/internal/config/cache"
 	"github.com/minio/minio/internal/config/dns"
 	"github.com/minio/minio/internal/config/storageclass"
 	"github.com/minio/minio/internal/crypto"
@@ -1054,22 +1053,6 @@ func (api objectAPIHandlers) CompleteMultipartUploadHandler(w http.ResponseWrite
 	}
 	sendEvent(evt)
 
-	asize, err := objInfo.GetActualSize()
-	if err != nil {
-		asize = objInfo.Size
-	}
-
-	defer globalCacheConfig.Set(&cache.ObjectInfo{
-		Key:          objInfo.Name,
-		Bucket:       objInfo.Bucket,
-		ETag:         objInfo.ETag,
-		ModTime:      objInfo.ModTime,
-		Expires:      objInfo.ExpiresStr(),
-		CacheControl: objInfo.CacheControl,
-		Size:         asize,
-		Metadata:     cleanReservedKeys(objInfo.UserDefined),
-	})
-
 	if objInfo.NumVersions > int(scannerExcessObjectVersions.Load()) {
 		evt.EventName = event.ObjectManyVersions
 		sendEvent(evt)


Test patch: