Repository: fsouza/fake-gcs-server (go)

Issue:
Content-Type being set incorrectly for resumable uploads.
Hi! 

First of all, thank you for developing this. It's an invaluable tool for local development :) 

We are using the latest version (`1.44.0`) with Ruby/Rails where the Google SDK [always does resumable uploads](https://cloud.google.com/storage/docs/resumable-uploads#behavior).

Generally, it's working well but the content type of all uploaded files is set to `application/json`. This can be verified by `exec`ing into the Docker container and running `attr` (might have to `apk --update add` it first) on the file.

```
# attr
A filename to operate on is required
Usage: attr [-LRSq] -s attrname [-V attrvalue] pathname  # set value
       attr [-LRSq] -g attrname pathname                 # get value
       attr [-LRSq] -r attrname pathname                 # remove attr
       attr [-LRq]  -l pathname                          # list attrs
      -s reads a value from stdin and -g writes a value to stdout
# attr -g metadata /storage/bucket/6d97b1f7-f3cb-49db-89a3-6039d01f24ba/b17d0638-275e-4864-b868-a13131131f69/original_file_0.jpg
Attribute "metadata" had a 463 byte value for /storage/bucket/6d97b1f7-f3cb-49db-89a3-6039d01f24ba/b17d0638-275e-4864-b868-a13131131f69/original_file_0.jpg:
{"ContentType":"application/json","ContentEncoding":"","Crc32c":"p7Um+Q==","Md5Hash":"r2DdsnVOdPUZweG4crCjog==","Etag":"\"r2DdsnVOdPUZweG4crCjog==\"","ACL":[{"Entity":"projectOwner-test-project","EntityID":"","Role":"OWNER","Domain":"","Email":"","ProjectTeam":null}],"Metadata":null,"Created":"2023-03-16T12:26:21.005992Z","Deleted":"0001-01-01T00:00:00Z","Updated":"2023-03-16T12:26:21.005996Z","CustomTime":"0001-01-01T00:00:00Z","Generation":1678969581005998}
```

Now this happens regardless of whether `content_type` is added to the original upload. Here is the output of the SDK with no `content_type` parameter. As you can see, no `content_type` is present in the initial upload, but the actual uploaded file ends up with a content type of `application/json`.

```
Intiating resumable upload command to http://localhost:4080/upload/storage/v1/b/bucket/o?name=b6e3a967-f145-40e1-89ed-e0b524c67431%2Fdbaa0927-be5f-4993-b314-f4a51ae7aa1c%2Foriginal_file_0.jpg
Success - #<Google::Apis::StorageV1::Object:0x00007f8bcf8f7d90
 @acl=
  [#<Google::Apis::StorageV1::ObjectAccessControl:0x00007f8bcf8f6d78
    @bucket="bucket",
    @entity="projectOwner-test-project",
    @object=
     "b6e3a967-f145-40e1-89ed-e0b524c67431/dbaa0927-be5f-4993-b314-f4a51ae7aa1c/original_file_0.jpg",
    @project_team=
     #<Google::Apis::StorageV1::ObjectAccessControl::ProjectTeam:0x00007f8c1efbb7e0>,
    @role="OWNER">],
 @bucket="bucket",
 @generation=0,
 @id=
  "bucket/b6e3a967-f145-40e1-89ed-e0b524c67431/dbaa0927-be5f-4993-b314-f4a51ae7aa1c/original_file_0.jpg",
 @kind="storage#object",
 @name=
  "b6e3a967-f145-40e1-89ed-e0b524c67431/dbaa0927-be5f-4993-b314-f4a51ae7aa1c/original_file_0.jpg",
 @size=0>

Sending upload command to http://[::]:4080/upload/storage/v1/b/bucket/o?uploadType=resumable&name=b6e3a967-f145-40e1-89ed-e0b524c67431%2Fdbaa0927-be5f-4993-b314-f4a51ae7aa1c%2Foriginal_file_0.jpg&upload_id=9b19140d2d0e510b73c61a0fe8947d30
Success - #<Google::Apis::StorageV1::Object:0x00007f8c3fadab88
 @acl=
  [#<Google::Apis::StorageV1::ObjectAccessControl:0x00007f8c3fad9b70
    @bucket="bucket",
    @entity="projectOwner-test-project",
    @object=
     "b6e3a967-f145-40e1-89ed-e0b524c67431/dbaa0927-be5f-4993-b314-f4a51ae7aa1c/original_file_0.jpg",
    @project_team=
     #<Google::Apis::StorageV1::ObjectAccessControl::ProjectTeam:0x00007f8c3fbc3950>,
    @role="OWNER">],
 @bucket="bucket",
 @content_type="application/json",
 @crc32c="9YirKw==",
 @etag="\"t9JShAgcze7Jkm4cKJtasQ==\"",
 @generation=1678974975332775,
 @id=
  "bucket/b6e3a967-f145-40e1-89ed-e0b524c67431/dbaa0927-be5f-4993-b314-f4a51ae7aa1c/original_file_0.jpg",
 @kind="storage#object",
 @md5_hash="t9JShAgcze7Jkm4cKJtasQ==",
 @name=
  "b6e3a967-f145-40e1-89ed-e0b524c67431/dbaa0927-be5f-4993-b314-f4a51ae7aa1c/original_file_0.jpg",
 @size=410714,
 @time_created=Thu, 16 Mar 2023 13:56:15 +0000,
 @updated=Thu, 16 Mar 2023 13:56:15 +0000>
```

Now, as I mentioned, this even happens when you pass the correct `content_type` to the upload command.


```
Intiating resumable upload command to http://localhost:4080/upload/storage/v1/b/bucket/o?name=2b708cc5-f130-4be0-8baf-ccf902095b8e%2Fc0033f4b-38a8-423c-b47d-109cbd07003c%2Foriginal_file_0.jpg
Success - #<Google::Apis::StorageV1::Object:0x00007f8544ee73b8
 @acl=
  [#<Google::Apis::StorageV1::ObjectAccessControl:0x00007f8544ee4320
    @bucket="bucket",
    @entity="projectOwner-test-project",
    @object=
     "2b708cc5-f130-4be0-8baf-ccf902095b8e/c0033f4b-38a8-423c-b47d-109cbd07003c/original_file_0.jpg",
    @project_team=
     #<Google::Apis::StorageV1::ObjectAccessControl::ProjectTeam:0x00007f8535c22650>,
    @role="OWNER">],
 @bucket="bucket",
 @content_type="image/jpeg",
 @generation=0,
 @id=
  "bucket/2b708cc5-f130-4be0-8baf-ccf902095b8e/c0033f4b-38a8-423c-b47d-109cbd07003c/original_file_0.jpg",
 @kind="storage#object",
 @name=
  "2b708cc5-f130-4be0-8baf-ccf902095b8e/c0033f4b-38a8-423c-b47d-109cbd07003c/original_file_0.jpg",
 @size=0>

Sending upload command to http://[::]:4080/upload/storage/v1/b/bucket/o?uploadType=resumable&name=2b708cc5-f130-4be0-8baf-ccf902095b8e%2Fc0033f4b-38a8-423c-b47d-109cbd07003c%2Foriginal_file_0.jpg&upload_id=0f02645f3d2c2029c9a1c05425ae1c09
Success - #<Google::Apis::StorageV1::Object:0x00007f8574cc6658
 @acl=
  [#<Google::Apis::StorageV1::ObjectAccessControl:0x00007f8524d1faf0
    @bucket="bucket",
    @entity="projectOwner-test-project",
    @object=
     "2b708cc5-f130-4be0-8baf-ccf902095b8e/c0033f4b-38a8-423c-b47d-109cbd07003c/original_file_0.jpg",
    @project_team=
     #<Google::Apis::StorageV1::ObjectAccessControl::ProjectTeam:0x00007f8524d1c4b8>,
    @role="OWNER">],
 @bucket="bucket",
 @content_type="application/json",
 @crc32c="rSseRg==",
 @etag="\"4g3RHUNbHQr3TfVps1zKJw==\"",
 @generation=1678973375548755,
 @id=
  "bucket/2b708cc5-f130-4be0-8baf-ccf902095b8e/c0033f4b-38a8-423c-b47d-109cbd07003c/original_file_0.jpg",
 @kind="storage#object",
 @md5_hash="4g3RHUNbHQr3TfVps1zKJw==",
 @name=
  "2b708cc5-f130-4be0-8baf-ccf902095b8e/c0033f4b-38a8-423c-b47d-109cbd07003c/original_file_0.jpg",
 @size=237839,
 @time_created=Thu, 16 Mar 2023 13:29:35 +0000,
 @updated=Thu, 16 Mar 2023 13:29:35 +0000>
```

If we restart the GCS server and it reloads the files from disk, the content type is set correctly (again, can be verified by running `attr` on the file as well). I assume this is because of https://github.com/fsouza/fake-gcs-server/issues/531.

It would however be great if the content type would also be correct for newly uploaded files, when using resumable uploads!

It looks like this was addressed in https://github.com/fsouza/fake-gcs-server/issues/532 and supposedly fixed in https://github.com/fsouza/fake-gcs-server/pull/924 but it does not seem to be working.

I also tried passing in the `ContentType` via `metadata`, according to the fix mentioned here: [#924](https://github.com/fsouza/fake-gcs-server/pull/924/files#diff-0569e7aafa398a922d8b95d609b84e24cd72ab510b7b0ab775f99392e246e20cR390).


```
Intiating resumable upload command to http://localhost:4080/upload/storage/v1/b/bucket/o?name=4dd472d2-9668-44a8-acb0-bb7126a2b0cf%2F1db2bb8b-73aa-410a-9d74-007bf111325d%2Foriginal_file_0.jpg
Success - #<Google::Apis::StorageV1::Object:0x00007fd4afa404a8
 @acl=
  [#<Google::Apis::StorageV1::ObjectAccessControl:0x00007fd4b800ef08
    @bucket="bucket",
    @entity="projectOwner-test-project",
    @object=
     "4dd472d2-9668-44a8-acb0-bb7126a2b0cf/1db2bb8b-73aa-410a-9d74-007bf111325d/original_file_0.jpg",
    @project_team=
     #<Google::Apis::StorageV1::ObjectAccessControl::ProjectTeam:0x00007fd4afc2b678>,
    @role="OWNER">],
 @bucket="bucket",
 @content_type="image/jpeg",
 @generation=0,
 @id=
  "bucket/4dd472d2-9668-44a8-acb0-bb7126a2b0cf/1db2bb8b-73aa-410a-9d74-007bf111325d/original_file_0.jpg",
 @kind="storage#object",
 @metadata={"ContentType"=>"image/jpeg"},
 @name=
  "4dd472d2-9668-44a8-acb0-bb7126a2b0cf/1db2bb8b-73aa-410a-9d74-007bf111325d/original_file_0.jpg",
 @size=0>

Sending upload command to http://[::]:4080/upload/storage/v1/b/bucket/o?uploadType=resumable&name=4dd472d2-9668-44a8-acb0-bb7126a2b0cf%2F1db2bb8b-73aa-410a-9d74-007bf111325d%2Foriginal_file_0.jpg&upload_id=3abe67bc7326fca3d517d792e42f07ad
Success - #<Google::Apis::StorageV1::Object:0x00007fd4afa208b0
 @acl=
  [#<Google::Apis::StorageV1::ObjectAccessControl:0x00007fd49862f178
    @bucket="bucket",
    @entity="projectOwner-test-project",
    @object=
     "4dd472d2-9668-44a8-acb0-bb7126a2b0cf/1db2bb8b-73aa-410a-9d74-007bf111325d/original_file_0.jpg",
    @project_team=
     #<Google::Apis::StorageV1::ObjectAccessControl::ProjectTeam:0x00007fd4afabb1a8>,
    @role="OWNER">],
 @bucket="bucket",
 @content_type="application/json",
 @crc32c="jnk9vw==",
 @etag="\"GwotjEH5otIJKB7HArHxLw==\"",
 @generation=1678978075907447,
 @id=
  "bucket/4dd472d2-9668-44a8-acb0-bb7126a2b0cf/1db2bb8b-73aa-410a-9d74-007bf111325d/original_file_0.jpg",
 @kind="storage#object",
 @md5_hash="GwotjEH5otIJKB7HArHxLw==",
 @metadata={"ContentType"=>"image/jpeg"},
 @name=
  "4dd472d2-9668-44a8-acb0-bb7126a2b0cf/1db2bb8b-73aa-410a-9d74-007bf111325d/original_file_0.jpg",
 @size=46951,
 @time_created=Thu, 16 Mar 2023 14:47:55 +0000,
 @updated=Thu, 16 Mar 2023 14:47:55 +0000>
```

As you can see, the _metadata_ is even saved correctly, but the _actual_ `ContentType` is still set to  `application/json`.

```
# attr -g metadata /storage/danni-dev/4dd472d2-9668-44a8-acb0-bb7126a2b0cf/1db2bb8b-73aa-410a-9d74-007bf111325d/original_file_0.jpg
Attribute "metadata" had a 486 byte value for /storage/danni-dev/4dd472d2-9668-44a8-acb0-bb7126a2b0cf/1db2bb8b-73aa-410a-9d74-007bf111325d/original_file_0.jpg:
{"ContentType":"application/json","ContentEncoding":"","Crc32c":"jnk9vw==","Md5Hash":"GwotjEH5otIJKB7HArHxLw==","Etag":"\"GwotjEH5otIJKB7HArHxLw==\"","ACL":[{"Entity":"projectOwner-test-project","EntityID":"","Role":"OWNER","Domain":"","Email":"","ProjectTeam":null}],"Metadata":{"ContentType":"image/jpeg"},"Created":"2023-03-16T14:47:55.90744Z","Deleted":"0001-01-01T00:00:00Z","Updated":"2023-03-16T14:47:55.907445Z","CustomTime":"0001-01-01T00:00:00Z","Generation":1678978075907447}
```

Pull request:
Propagate ContentType field on create/modify ops
This PR tries to address 2 bugs that where found when using some of the create/modify operations GCS supports in our integration tests that used the following interactions:
* Rely on a streamable upload operations of GCS to upload of a File.
* Submit an update on the blob to override the metadata content-type information without modifying the bytes of the blob.

To crosscheck, we ran the same tests against the real GCS API to verify the behaviour to expect:
1. Streamable upload, by default if no `content-type `is provided; GCS sets it up to` 'application/octet-stream'` or uses the one provided by the user in the BlobInfo (This is the expected behaviour for this operation in GCS).
2. For an update, when we merge metadata information regarding the content-type, it should override and reflect the change on the update response.

Also this behaviour is referenced on the API spec, that the `content-type` should be set to `application/octet-stream` on object creation if it is not defined:

1. [GCS Insert Request Body](https://cloud.google.com/storage/docs/json_api/v1/objects/insert#request-body)
2. [GCS Update Request Body](https://cloud.google.com/storage/docs/json_api/v1/objects/update#request-body)

Also should hopefully address the following issue https://github.com/fsouza/fake-gcs-server/issues/1098 :slightly_smiling_face: 

Fixes #1098.

Patch:
diff --git a/examples/node/package-lock.json b/examples/node/package-lock.json
index 0a7568d..1d147d3 100644
--- a/examples/node/package-lock.json
+++ b/examples/node/package-lock.json
@@ -24,9 +24,9 @@
       "integrity": "sha512-z1CjRjtQyBOYL+5Qr9DdYIfrdLBe746jRTYfaYU6MeXkqp7UfYs/jX16lFFVzZ7PGEJvqZNqYUEtb1mvDww4pA=="
     },
     "@google-cloud/storage": {
-      "version": "6.9.5",
-      "resolved": "https://registry.npmjs.org/@google-cloud/storage/-/storage-6.9.5.tgz",
-      "integrity": "sha512-fcLsDA8YKcGuqvhk0XTjJGVpG9dzs5Em8IcUjSjspYvERuHYqMy9CMChWapSjv3Lyw//exa3mv4nUxPlV93BnA==",
+      "version": "6.10.0",
+      "resolved": "https://registry.npmjs.org/@google-cloud/storage/-/storage-6.10.0.tgz",
+      "integrity": "sha512-MaFNtMxPpnv6c43HcRsJTUiYhXgcjy+mshLyZpfGKMpE2vJ8C1mBFK/ZrlcPBt47ZK0tz9p/mNTyvi8dRsdKPw==",
       "requires": {
         "@google-cloud/paginator": "^3.0.7",
         "@google-cloud/projectify": "^3.0.0",
@@ -180,16 +180,16 @@
       }
     },
     "google-auth-library": {
-      "version": "8.7.0",
-      "resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-8.7.0.tgz",
-      "integrity": "sha512-1M0NG5VDIvJZEnstHbRdckLZESoJwguinwN8Dhae0j2ZKIQFIV63zxm6Fo6nM4xkgqUr2bbMtV5Dgo+Hy6oo0Q==",
+      "version": "8.8.0",
+      "resolved": "https://registry.npmjs.org/google-auth-library/-/google-auth-library-8.8.0.tgz",
+      "integrity": "sha512-0iJn7IDqObDG5Tu9Tn2WemmJ31ksEa96IyK0J0OZCpTh6CrC6FrattwKX87h3qKVuprCJpdOGKc1Xi8V0kMh8Q==",
       "requires": {
         "arrify": "^2.0.0",
         "base64-js": "^1.3.0",
         "ecdsa-sig-formatter": "^1.0.11",
         "fast-text-encoding": "^1.0.0",
         "gaxios": "^5.0.0",
-        "gcp-metadata": "^5.0.0",
+        "gcp-metadata": "^5.2.0",
         "gtoken": "^6.1.0",
         "jws": "^4.0.0",
         "lru-cache": "^6.0.0"
diff --git a/examples/node/package.json b/examples/node/package.json
index ab59467..18ab88a 100644
--- a/examples/node/package.json
+++ b/examples/node/package.json
@@ -10,6 +10,6 @@
   "author": "",
   "license": "ISC",
   "dependencies": {
-    "@google-cloud/storage": "^6.9.5"
+    "@google-cloud/storage": "^6.10.0"
   }
 }
diff --git a/fakestorage/object.go b/fakestorage/object.go
index 62bfd86..5430a48 100644
--- a/fakestorage/object.go
+++ b/fakestorage/object.go
@@ -731,6 +731,9 @@ func (s *Server) rewriteObject(r *http.Request) jsonResponse {
 	}
 	defer created.Close()
 
+	if vars["copyType"] == "copyTo" {
+		return jsonResponse{data: newObjectResponse(created.ObjectAttrs)}
+	}
 	return jsonResponse{data: newObjectRewriteResponse(created.ObjectAttrs)}
 }
 
@@ -993,9 +996,10 @@ func (s *Server) updateObject(r *http.Request) jsonResponse {
 	}
 
 	var payload struct {
-		Metadata   map[string]string `json:"metadata"`
-		CustomTime string
-		Acl        []acls
+		Metadata    map[string]string `json:"metadata"`
+		ContentType string            `json:"contentType"`
+		CustomTime  string
+		Acl         []acls
 	}
 	err := json.NewDecoder(r.Body).Decode(&payload)
 	if err != nil {
@@ -1009,7 +1013,7 @@ func (s *Server) updateObject(r *http.Request) jsonResponse {
 
 	attrsToUpdate.Metadata = payload.Metadata
 	attrsToUpdate.CustomTime = payload.CustomTime
-
+	attrsToUpdate.ContentType = payload.ContentType
 	if len(payload.Acl) > 0 {
 		attrsToUpdate.ACL = []storage.ACLRule{}
 		for _, aclData := range payload.Acl {
diff --git a/fakestorage/server.go b/fakestorage/server.go
index 21db772..f93c883 100644
--- a/fakestorage/server.go
+++ b/fakestorage/server.go
@@ -255,8 +255,7 @@ func (s *Server) buildMuxer() {
 		r.Path("/b/{bucketName}/o/{objectName:.+}/acl/{entity}").Methods(http.MethodPut).HandlerFunc(jsonToHTTPHandler(s.setObjectACL))
 		r.Path("/b/{bucketName}/o/{objectName:.+}").Methods(http.MethodGet, http.MethodHead).HandlerFunc(s.getObject)
 		r.Path("/b/{bucketName}/o/{objectName:.+}").Methods(http.MethodDelete).HandlerFunc(jsonToHTTPHandler(s.deleteObject))
-		r.Path("/b/{sourceBucket}/o/{sourceObject:.+}/copyTo/b/{destinationBucket}/o/{destinationObject:.+}").Methods(http.MethodPost).HandlerFunc(jsonToHTTPHandler(s.rewriteObject))
-		r.Path("/b/{sourceBucket}/o/{sourceObject:.+}/rewriteTo/b/{destinationBucket}/o/{destinationObject:.+}").Methods(http.MethodPost).HandlerFunc(jsonToHTTPHandler(s.rewriteObject))
+		r.Path("/b/{sourceBucket}/o/{sourceObject:.+}/{copyType:rewriteTo|copyTo}/b/{destinationBucket}/o/{destinationObject:.+}").Methods(http.MethodPost).HandlerFunc(jsonToHTTPHandler(s.rewriteObject))
 		r.Path("/b/{bucketName}/o/{destinationObject:.+}/compose").Methods(http.MethodPost).HandlerFunc(jsonToHTTPHandler(s.composeObject))
 		r.Path("/b/{bucketName}/o/{objectName:.+}").Methods(http.MethodPut, http.MethodPost).HandlerFunc(jsonToHTTPHandler(s.updateObject))
 	}
diff --git a/fakestorage/upload.go b/fakestorage/upload.go
index 1fbb1ef..fe1cceb 100644
--- a/fakestorage/upload.go
+++ b/fakestorage/upload.go
@@ -485,7 +485,12 @@ func (s *Server) uploadFileContent(r *http.Request) jsonResponse {
 	obj.Crc32c = checksum.EncodedCrc32cChecksum(obj.Content)
 	obj.Md5Hash = checksum.EncodedMd5Hash(obj.Content)
 	obj.Etag = fmt.Sprintf("%q", obj.Md5Hash)
-	obj.ContentType = r.Header.Get(contentTypeHeader)
+	contentTypeHeader := r.Header.Get(contentTypeHeader)
+	if contentTypeHeader != "" {
+		obj.ContentType = contentTypeHeader
+	} else {
+		obj.ContentType = "application/octet-stream"
+	}
 	responseHeader := make(http.Header)
 	if contentRange := r.Header.Get("Content-Range"); contentRange != "" {
 		parsed, err := parseContentRange(contentRange)


Test patch:
diff --git a/fakestorage/object_test.go b/fakestorage/object_test.go
index 48c1263..ac3cd0a 100644
--- a/fakestorage/object_test.go
+++ b/fakestorage/object_test.go
@@ -1669,6 +1669,57 @@ func TestServerClientObjectUpdateCustomTime(t *testing.T) {
 	})
 }
 
+func TestServerClientObjectUpdateContentType(t *testing.T) {
+	const (
+		bucketName  = "some-bucket"
+		objectName  = "data.txt"
+		content     = "some nice content"
+		contentType = "some content-type"
+	)
+	objs := []Object{
+		{
+			ObjectAttrs: ObjectAttrs{
+				BucketName:  bucketName,
+				Name:        objectName,
+				ContentType: contentType,
+			},
+			Content: []byte(content),
+		},
+	}
+	url := fmt.Sprintf("https://storage.googleapis.com/storage/v1/b/%s/o/%s", bucketName, objectName)
+	runServersTest(t, runServersOptions{objs: objs}, func(t *testing.T, server *Server) {
+		client := server.HTTPClient()
+		jsonBody := []byte(`{"ContentType": "another content-type"}`)
+		bodyReader := bytes.NewReader(jsonBody)
+		req, err := http.NewRequest(http.MethodPost, url, bodyReader)
+		if err != nil {
+			t.Fatal(err)
+		}
+		resp, err := client.Do(req)
+		if err != nil {
+			t.Fatal(err)
+		}
+		defer resp.Body.Close()
+		if resp.StatusCode != http.StatusOK {
+			t.Errorf("wrong status returned\nwant %d\ngot  %d", http.StatusOK, resp.StatusCode)
+		}
+		data, err := io.ReadAll(resp.Body)
+		if err != nil {
+			t.Fatal(err)
+		}
+		var respJsonBody ObjectAttrs
+		err = json.Unmarshal(data, &respJsonBody)
+		if err != nil {
+			t.Fatal(err)
+		}
+		updatedContentType := respJsonBody.ContentType
+		expectedContentType := "another content-type"
+		if updatedContentType != expectedContentType {
+			t.Errorf("unexpected content type time\nwant %q\ngot  %q", expectedContentType, updatedContentType)
+		}
+	})
+}
+
 func TestServerClientObjectPatchCustomTime(t *testing.T) {
 	const (
 		bucketName  = "some-bucket"