Repository: googleapis/google-cloud-go (Go)

Issue:
logging: detectKubernetesResource() should set region not zone for location
**Client**

Logging

**Environment**

on GKE

**Go Environment**

N/A

**Code**

N/A

**Expected behavior**

detectKubernetesResource() set region for location

**Actual behavior**

detectKubernetesResource() set zone for location

**Screenshots**

N/A

**Additional context**

`detectKubernetesResource()` currenly set zone(like "us-central1-b") for `location`.
But default stackdriver exporter in GKE uses region(like "us-central1") for `location` and GCP UI generates Cloud Logging URL which uses region for `location` at Pod logs page.
So, when developers want to switch from stackdriver exporter to direct exporter by this library, location value is not compatible between changes and default detection is not useful.

Log entry of resource type "k8s_container":  "location" label set incorrectly
When using the Go library to create container log entry (resource type `k8s_container`), the logging `location` label is set to the Node zone instead of the cluster location.

https://github.com/googleapis/google-cloud-go/blob/477ccee8eb210a0e0f9c71513318e0fbbd08928c/logging/resource.go#L187

Cloud Monitoring documentation covers this:

https://cloud.google.com/monitoring/api/resources#tag_k8s_container
> location: The physical location of the cluster that contains the container.

For a GKE regional cluster `location` should be set in the cluster region and the same logic applies for the zonal cluster. 

Note this is set correctly for logs going through the GKE logging pipeline as well as for some of other client library such as ruby and C++:

* https://github.com/googleapis/google-api-ruby-client/blob/3c1c3311a9179bbb3e3b17f1939d180748e5e6d0/generated/google-apis-monitoring_v3/lib/google/apis/monitoring_v3/classes.rb#L1694
* https://github.com/googleapis/google-cloud-cpp/blob/850564f330ac6e918d6d40ac1ab17a200282a196/google/cloud/opentelemetry/internal/monitored_resource.cc#L88

I believe the reason why the  `location` label is set to the Cluster location in the first place is that this along with the cluster name is required to uniquely identify the cluster. GKE allows the cluster to have the same name if the location is different, so the name alone is not enough.

The `location` label being set incorrectly can result in the container log not showing up as expected in Cloud console due to them not matching the expected log filter. In some cases logs could potentially be attributed to the wrong cluster.

**Expected behavior**

Log entry created using `k8s_container` resource type have the `location` label set to the cluster location 

**Actual behavior**

Log entry created using `k8s_container` resource type and the `location` label set to the Node zone





Discussion: @tamayika can you please clarify what do you call "default stackdriver exporter" and to share an example of the resource metadata from the logs that are ingested by the default stackdriver exporter?

thank you
This is the example of metadata which default stackdriver exporter uploads.(i.e. stdout or stderr logs metadata)

```yaml
resource:
  labels:
    cluster_name: "my-cluster"
    container_name: "my-container"
    location: "us-central1"
    namespace_name: "my-namespace"
    pod_name: "my-pod"
    project_id: "my-project"
  type: "k8s_container"
```

This is the example of metadata which current logging library uploads.
```yaml
resource:
  labels:
    cluster_name: "my-cluster"
    container_name: "my-container"
    location: "us-central1-b"
    namespace_name: "my-namespace"
    pod_name: "my-pod"
    project_id: "my-project"
  type: "k8s_container"
```

So, only `location` is different.
@tamayika according to [documentation](https://cloud.google.com/monitoring/api/resources#tag_k8s_container) for the resource metadata for `k8s_container` types  the value for `location` can be either a region or a zone. While I agree with you that it is unclear/ambiguous what value should go into `location`, the default stackdriver exporter is a legacy item that we will not support with an integration such as this one that will break the existing experience for other customers as well.
Thanks for consideration.
I will propose to change GKE logs URL Parameter at GCP Support Page.
@gkevinzheng The doc you mentioned regarding the location label (for `k8s_container`) says:

>  The physical location of the cluster that contains the container.

To me this means that location should be set to a zone in the case of a GKE zonal cluster and to the region in the case of a GKE regional cluster. 

It seems this is the exact behavior that happens when going through the default GKE logging pipeline. I'm not sure what "stackdriver exporter" / legacy is meant to refer to exactly.

I think the reported issue here is that the logging library doesn't set "location" to the Cluster zone or region but the zone where the node exists. This seems like a problematic behavior.
@arekkusu Ahh, this makes sense. I'll add logic to the library for distinguishing those two use-cases then. 
@tamayika thank you for [providing](https://github.com/googleapis/google-cloud-go/issues/9070#issuecomment-1839889969) data. I am right to assume that by "default stackdriver exporter" your reference the implicit logging agent of GKE?
@minherz Yes, that's right.





Fix (patch):
diff --git a/logging/resource.go b/logging/resource.go
index 3a53a86ae3bb..737c6d0392f7 100644
--- a/logging/resource.go
+++ b/logging/resource.go
@@ -201,8 +201,8 @@ func detectKubernetesResource() *mrpb.MonitoredResource {
 	if projectID == "" {
 		return nil
 	}
-	zone := detectedResource.metadataZone()
 	clusterName := detectedResource.attrs.Metadata("instance/attributes/cluster-name")
+	clusterLocation := detectedResource.attrs.Metadata("instance/attributes/cluster-location")
 	namespaceName := detectedResource.attrs.ReadAll("/var/run/secrets/kubernetes.io/serviceaccount/namespace")
 	if namespaceName == "" {
 		// if automountServiceAccountToken is disabled allow to customize
@@ -217,7 +217,7 @@ func detectKubernetesResource() *mrpb.MonitoredResource {
 		Type: "k8s_container",
 		Labels: map[string]string{
 			"cluster_name":   clusterName,
-			"location":       zone,
+			"location":       clusterLocation,
 			"project_id":     projectID,
 			"pod_name":       podName,
 			"namespace_name": namespaceName,