Repository: minio/minio (language: go)

Issue:
GCS Tier OFFLINE after upgrade to RELEASE.2024-06-06T09-36-42Z
After minio upgrade to version: quay.io/minio/minio:RELEASE.2024-06-06T09-36-42Z, caused Tier status went Offline.

```
  tenant:
    image:
      repository: quay.io/minio/minio
      tag: RELEASE.2024-05-28T17-19-04Z   --->   tag: RELEASE.2024-06-06T09-36-42Z
```       

## Expected Behavior
![expected_minio_tier](https://github.com/minio/minio/assets/66746460/1a7e0c21-53e0-48f5-91fe-c255b20209e4)


## Current Behavior
![current_minio_tier](https://github.com/minio/minio/assets/66746460/9f9eee81-7e63-421e-bb45-861ca84f76f5)


## Context
Objects moved to tier storage are not accessible.
`mc: <ERROR> Failed to copy https://minio-api-sbx.... Error response code XGCSInternalError.`

## Your Environment
* Version used (`minio --version`):
MinIO version: quay.io/minio/minio:RELEASE.2024-06-11T03-13-30Z

* Server setup and configuration:
helm deployed minio operator and tenant version: 5.0.15
https://min.io/docs/minio/kubernetes/upstream/operations/install-deploy-manage/deploy-operator-helm.html
https://min.io/docs/minio/kubernetes/upstream/operations/install-deploy-manage/deploy-minio-tenant-helm.html#deploy-tenant-helm

* Operating System and version (`uname -a`):
GKE k8s cluster with tier setup to GCS bucket access by service account with permissions "Storage Object Admin" on bucket level.
https://min.io/docs/minio/kubernetes/upstream/administration/object-management/transition-objects-to-gcs.html#configure-the-remote-storage-tier



Discussion / hints:
Please share the server logs from hot-tier.
in the log are error messages like this one:

```
 API: SYSTEM.transition()
 Time: 12:35:25 UTC 06/13/2024
 DeploymentID: 9efbc796-0ce2-4f19-9a3f-ff72cb43c9c3
 Error: Transition to GCS-HOT-TIER failed for test-bucket/test-file.zip version:99e3068e-ad2e-428e-be47-1ff85e7bcf6d with googleapi: Error 401: Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist)., required (*fmt.wrapError)
        3: internal/logger/logger.go:268:logger.LogIf()
        2: cmd/logging.go:116:cmd.transitionLogIf()
        1: cmd/bucket-lifecycle.go:495:cmd.(*transitionState).worker()
```  

service account for tiering has storage.object.admin access on GCS bucket and it worked before with this setup.
I tried activate this service account on local and was able to move file to/from GCS bucket.
> in the log are error messages like this one:
> 
> ```
>  API: SYSTEM.transition()
>  Time: 12:35:25 UTC 06/13/2024
>  DeploymentID: 9efbc796-0ce2-4f19-9a3f-ff72cb43c9c3
>  Error: Transition to GCS-HOT-TIER failed for test-bucket/test-file.zip version:99e3068e-ad2e-428e-be47-1ff85e7bcf6d with googleapi: Error 401: Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist)., required (*fmt.wrapError)
>         3: internal/logger/logger.go:268:logger.LogIf()
>         2: cmd/logging.go:116:cmd.transitionLogIf()
>         1: cmd/bucket-lifecycle.go:495:cmd.(*transitionState).worker()
> ```
> 
> service account for tiering has storage.object.admin access on GCS bucket and it worked before with this setup. I tried activate this service account on local and was able to move file to/from GCS bucket.

It looks like your credentials are wrong.
Please list your tier targets in --json form using mc and share the output.

```
mc ilm tier ls alias/ --json
```

```
mc ilm tier info alias/ GCS-HOT-TIER
```

> Please list your tier targets in --json form using mc and share the output.
> 
> ```
> mc ilm tier ls alias/ --json
> ```
> 
> ```
> mc ilm tier info alias/ GCS-HOT-TIER
> ```


```
❯ mc ilm tier ls gcp-test --json
{"status":"success","tiers":[{"Version":"v1","Type":"gcs","Name":"GCS-HOT-TIER","S3":{},"Azure":{"SPAuth":{}},"GCS":{"Endpoint":"https://storage.googleapis.com/","Creds":"REDACTED","Bucket":"minio-test-standard-tier","Prefix":"gcp-test","Region":"europe-west4"},"MinIO":{}}]}

❯ mc ilm tier info gcp-test GCS-HOT-TIER
┌────────────┬───┬────┬──────┬───────┬────────┐
│ Tier Name  │AP…│Typ…│Usage │Object…│Version…│
├────────────┼───┼────┼──────┼───────┼────────┤
│GCS-HOT-TIE…│gc…│war…│69 Gi…│   4471│    4471│
└────────────┴───┴────┴──────┴───────┴────────┘
``` 



Are your creds correct? @minarikpeto can you verify them via gsutil - outside of MinIO? 
can you recreate the tier as a different name and see if that is seen as online or offline which works independently in gsutil? 
got same issue with `RELEASE.2024-06-13T22-53-53Z`

```
❯ mc ilm tier check minio-dev TIER-GCS-MINIO-DEV-SBK
mc: <ERROR> Unable to verify remote tier target: failed to perform PUT: googleapi: Error 401: Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist)., required.
```
<img width="1499" alt="image" src="https://github.com/minio/minio/assets/17775940/c90f695c-68c1-481b-8393-a8ae95969691">

and event cant create tier with gcs anymore
the credentials is never changed, and permission scope is `Storage Admin`

![image](https://github.com/minio/minio/assets/17775940/45fa3083-6d48-42ba-a762-7f0270512c2b)

> Are your creds correct? @minarikpeto can you verify them via gsutil - outside of MinIO?

tried that and was able to download/upload file in GCS bucket.

```
❯ gsutil cp gs://minio-standard-tier/gcp-test/b7e16db891f4a200/28/5f/file-100m.test .

Copying gs://minio-standard-tier/gcp-test/b7e16db891f4a200/28/5f/file-100m.test...
| [1 files][100.0 MiB/100.0 MiB]    3.1 MiB/s
Operation completed over 1 objects/100.0 MiB.
```  
> can you recreate the tier as a different name and see if that is seen as online or offline which works independently in gsutil?

this is the same service account that works for gsutil.
```
❯ mc ilm tier add gcs gcp-test GCS-HOT-TIER_02 --credentials-file /Users/pminarik/Downloads/gcp-test-5bb02e299f3f.json --bucket minio-standard-tier --storage-class STANDARD --region europe-west4 --prefix gcp-test
mc: <ERROR> Unable to configure remote tier target: failed to perform PUT: googleapi: Error 401: Anonymous caller does not have storage.objects.create access to the Google Cloud Storage object. Permission 'storage.objects.create' denied on resource (or it may not exist)., required.
``` 
yeah looks like the credentials JSON is not getting honored after the SDK upgrade, will check why. 
@minarikpeto can you retry this on a separate test setup on RELEASE.2024-05-28T17-19-04Z  and try creating the tier again? 

Fix patch:
diff --git a/cmd/warm-backend-gcs.go b/cmd/warm-backend-gcs.go
index 029c2550be09a..3246fd1c8f63f 100644
--- a/cmd/warm-backend-gcs.go
+++ b/cmd/warm-backend-gcs.go
@@ -22,7 +22,6 @@ import (
 	"errors"
 	"fmt"
 	"io"
-	"net/http"
 
 	"cloud.google.com/go/storage"
 	"github.com/minio/madmin-go/v3"
@@ -118,14 +117,9 @@ func newWarmBackendGCS(conf madmin.TierGCS, tier string) (*warmBackendGCS, error
 		return nil, err
 	}
 
-	clnt := &http.Client{
-		Transport: globalRemoteTargetTransport,
-	}
-
 	client, err := storage.NewClient(context.Background(),
 		option.WithCredentialsJSON(credsJSON),
 		option.WithScopes(storage.ScopeReadWrite),
-		option.WithHTTPClient(clnt),
 		option.WithUserAgent(fmt.Sprintf("gcs-tier-%s", tier)+SlashSeparator+ReleaseTag),
 	)
 	if err != nil {


Test patch: