Repository: SeldonIO/seldon-core (language: python)

Issue:
Pipeline Gateway Podmonitor label selector does not match Pipeline Gateway labels
<!-- Welcome and thank you for helping us make Seldon Core better!

## Describe the bug
<!-- A clear and concise description of what the bug is. -->

It looks like the configuration of the prometheus podmonitors were not updated correctly when the [SeldonRuntime CRD was introduced](https://github.com/SeldonIO/seldon-core/pull/4847/files#diff-608c783a8c14298e3151fc91b57a38a686e07bf59a44b19ad0caa725490ede5c).

Previously, we used labels to configure which resources our podmonitors would scrape metrics from, e.g. for the [pipelinegateway](https://github.com/SeldonIO/seldon-core/blob/79afceae01540a08cca158535540189b381f79da/prometheus/monitors/pipelinegateway-podmonitor.yaml#L8):
```
spec:
  selector:
    matchLabels:
      app: pipelinegateway
```

I can see from the the reconcilers code that we are adding the labels to the deployment spec like so
```
templateLabels := utils.MergeMaps(map[string]string{constants.KubernetesNameLabelKey: name}, labels)
```
where the `constants.KubernetesNameLabelKey` is defined as
```
const (
	AppKey                    = "app"
	KubernetesNameLabelKey    = "app.kubernetes.io/name"
```

From what I can tell, name is typically defined as `seldon-pipelinegateway` from the [operator api definition](https://github.com/SeldonIO/seldon-core/blob/79afceae01540a08cca158535540189b381f79da/operator/apis/mlops/v1alpha1/seldonconfig_types.go#L38) which results in the following label `app.kubernetes.io/name=seldon-pipelinegateway` which does not match our existing podmonitors.

This results in prometheus being unable to scrape things like the `pipelinegateway` and its [metrics](https://docs.seldon.io/projects/seldon-core/en/v2/contents/metrics/operational.html#list-of-scv2-metrics) being missing.

As far as I can tell, the server metrics happen to be fine because [they use the `constants.AppKey`](https://github.com/SeldonIO/seldon-core/blob/79afceae01540a08cca158535540189b381f79da/operator/controllers/reconcilers/server/statefulset_reconciler.go#L73) so have not changed. I'm not sure if the [envoy ServiceMonitor](https://github.com/SeldonIO/seldon-core/blob/79afceae01540a08cca158535540189b381f79da/prometheus/monitors/envoy-servicemonitor.yaml#L8) was similarly affected


## To reproduce
<!-- Steps required to reproduce the issue. For example:
1. define model ...
2. build image ...  (especially what wrapper version is used)
3. deploy ...
-->

## Expected behaviour
<!-- A clear and concise description of what you expected to happen. -->
Either the podmonitor label selectors should match the ones created by the Seldon Runtime Reconciler, or the reconciler adds the labels that the podmonitor currently expects



Discussion / hints:

Fix patch:
diff --git a/k8s/helm-charts/seldon-core-v2-setup/templates/seldon-v2-components.yaml b/k8s/helm-charts/seldon-core-v2-setup/templates/seldon-v2-components.yaml
index 0c1845b45a..6c7f9bb139 100644
--- a/k8s/helm-charts/seldon-core-v2-setup/templates/seldon-v2-components.yaml
+++ b/k8s/helm-charts/seldon-core-v2-setup/templates/seldon-v2-components.yaml
@@ -1663,6 +1663,8 @@ spec:
       - containerPort: 9000
         name: server-http
         protocol: TCP
+      - containerPort: 8082
+        name: server-metrics
       readinessProbe:
         httpGet:
           path: /v2/health/live
diff --git a/operator/controllers/reconcilers/server/statefulset_reconciler.go b/operator/controllers/reconcilers/server/statefulset_reconciler.go
index a6ff801430..0e10bf1a35 100644
--- a/operator/controllers/reconcilers/server/statefulset_reconciler.go
+++ b/operator/controllers/reconcilers/server/statefulset_reconciler.go
@@ -70,9 +70,9 @@ func toStatefulSet(meta metav1.ObjectMeta,
 	scaling *mlopsv1alpha1.ScalingSpec,
 	labels map[string]string,
 	annotations map[string]string) *appsv1.StatefulSet {
-	labels[constants.AppKey] = constants.ServerLabelValue
-	metaLabels := utils.MergeMaps(map[string]string{constants.AppKey: constants.ServerLabelValue}, labels)
-	templateLabels := utils.MergeMaps(map[string]string{constants.ServerLabelNameKey: meta.Name, constants.AppKey: constants.ServerLabelValue}, labels)
+	labels[constants.KubernetesNameLabelKey] = constants.ServerLabelValue
+	metaLabels := utils.MergeMaps(map[string]string{constants.KubernetesNameLabelKey: constants.ServerLabelValue}, labels)
+	templateLabels := utils.MergeMaps(map[string]string{constants.ServerLabelNameKey: meta.Name, constants.KubernetesNameLabelKey: constants.ServerLabelValue}, labels)
 	ss := &appsv1.StatefulSet{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:        meta.Name,
diff --git a/operator/pkg/constants/constants.go b/operator/pkg/constants/constants.go
index 9be66483a8..a4025df219 100644
--- a/operator/pkg/constants/constants.go
+++ b/operator/pkg/constants/constants.go
@@ -39,7 +39,6 @@ var (
 
 // Label selector
 const (
-	AppKey                    = "app"
 	KubernetesNameLabelKey    = "app.kubernetes.io/name"
 	ServerLabelValue          = "seldon-server"
 	ServerLabelNameKey        = "seldon-server-name"
diff --git a/prometheus/monitors/agent-podmonitor.yaml b/prometheus/monitors/agent-podmonitor.yaml
index 4ba2dd32a5..62d8f9d86b 100644
--- a/prometheus/monitors/agent-podmonitor.yaml
+++ b/prometheus/monitors/agent-podmonitor.yaml
@@ -5,7 +5,7 @@ metadata:
 spec:
   selector:
     matchLabels:
-      app: seldon-server
+      app.kubernetes.io/name: seldon-server
   namespaceSelector:
     matchNames: []
     any: false
diff --git a/prometheus/monitors/envoy-servicemonitor.yaml b/prometheus/monitors/envoy-servicemonitor.yaml
index 455a865bff..e2b3e31b32 100644
--- a/prometheus/monitors/envoy-servicemonitor.yaml
+++ b/prometheus/monitors/envoy-servicemonitor.yaml
@@ -5,7 +5,7 @@ metadata:
 spec:
   selector:
     matchLabels:
-      app: seldon-mesh
+      app.kubernetes.io/name: seldon-envoy
   namespaceSelector:
     matchNames: []
     any: false
diff --git a/prometheus/monitors/pipelinegateway-podmonitor.yaml b/prometheus/monitors/pipelinegateway-podmonitor.yaml
index d5c58a0825..87dd888338 100644
--- a/prometheus/monitors/pipelinegateway-podmonitor.yaml
+++ b/prometheus/monitors/pipelinegateway-podmonitor.yaml
@@ -5,7 +5,7 @@ metadata:
 spec:
   selector:
     matchLabels:
-      app: pipelinegateway
+      app.kubernetes.io/name: pipelinegateway
   namespaceSelector:
     matchNames: []
     any: false
diff --git a/prometheus/monitors/server-podmonitor.yaml b/prometheus/monitors/server-podmonitor.yaml
index 6f95166517..d5ea6a2752 100644
--- a/prometheus/monitors/server-podmonitor.yaml
+++ b/prometheus/monitors/server-podmonitor.yaml
@@ -5,7 +5,7 @@ metadata:
 spec:
   selector:
     matchLabels:
-      app: seldon-server
+      app.kubernetes.io/name: seldon-server
   namespaceSelector:
     matchNames: []
     any: false


Test patch:
diff --git a/operator/controllers/reconcilers/server/statefulset_reconciler_test.go b/operator/controllers/reconcilers/server/statefulset_reconciler_test.go
index 38f8ff6a51..21caa02fcc 100644
--- a/operator/controllers/reconcilers/server/statefulset_reconciler_test.go
+++ b/operator/controllers/reconcilers/server/statefulset_reconciler_test.go
@@ -42,7 +42,7 @@ import (
 func TestStatefulSetReconcile(t *testing.T) {
 	g := NewGomegaWithT(t)
 
-	stsJson := `{"metadata":{"name":"mlserver","namespace":"ns1","creationTimestamp":null,"labels":{"app":"seldon-server","app.kubernetes.io/managed-by":"Helm"},"ownerReferences":[{"apiVersion":"mlops.seldon.io/v1alpha1","kind":"Server","name":"mlserver","uid":"7a9dc74b-552c-49da-8e45-09a6af752319","controller":true,"blockOwnerDeletion":true}]},"spec":{"replicas":1,"selector":{"matchLabels":{"seldon-server-name":"mlserver"}},"template":{"metadata":{"name":"mlserver","namespace":"ns1","creationTimestamp":null,"labels":{"app":"seldon-server","app.kubernetes.io/managed-by":"Helm","seldon-server-name":"mlserver"}},"spec":{"volumes":[{"name":"downstream-ca-certs","secret":{"secretName":"seldon-downstream-server","optional":true}},{"name":"config-volume","configMap":{"name":"seldon-agent"}},{"name":"tracing-config-volume","configMap":{"name":"seldon-tracing"}}],"containers":[{"name":"rclone","image":"docker.io/seldonio/seldon-rclone:latest","ports":[{"name":"rclone","containerPort":5572,"protocol":"TCP"}],"resources":{"limits":{"memory":"128Mi"},"requests":{"cpu":"50m","memory":"128Mi"}},"volumeMounts":[{"name":"mlserver-models","mountPath":"/mnt/agent"}],"readinessProbe":{"tcpSocket":{"port":5572},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":5,"successThreshold":1,"failureThreshold":3},"lifecycle":{"preStop":{"httpGet":{"path":"terminate","port":9007}}},"imagePullPolicy":"IfNotPresent"}],"terminationGracePeriodSeconds":120,"serviceAccountName":"seldon-server","securityContext":{"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true,"fsGroup":1000}}},"volumeClaimTemplates":[{"metadata":{"name":"mlserver-models","creationTimestamp":null},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}}},"status":{}}],"serviceName":"mlserver","podManagementPolicy":"Parallel","updateStrategy":{}},"status":{"replicas":0,"availableReplicas":0}}`
+	stsJson := `{"metadata":{"name":"mlserver","namespace":"ns1","creationTimestamp":null,"labels":{"app.kubernetes.io/name":"seldon-server","app.kubernetes.io/managed-by":"Helm"},"ownerReferences":[{"apiVersion":"mlops.seldon.io/v1alpha1","kind":"Server","name":"mlserver","uid":"7a9dc74b-552c-49da-8e45-09a6af752319","controller":true,"blockOwnerDeletion":true}]},"spec":{"replicas":1,"selector":{"matchLabels":{"seldon-server-name":"mlserver"}},"template":{"metadata":{"name":"mlserver","namespace":"ns1","creationTimestamp":null,"labels":{"app":"seldon-server","app.kubernetes.io/managed-by":"Helm","seldon-server-name":"mlserver"}},"spec":{"volumes":[{"name":"downstream-ca-certs","secret":{"secretName":"seldon-downstream-server","optional":true}},{"name":"config-volume","configMap":{"name":"seldon-agent"}},{"name":"tracing-config-volume","configMap":{"name":"seldon-tracing"}}],"containers":[{"name":"rclone","image":"docker.io/seldonio/seldon-rclone:latest","ports":[{"name":"rclone","containerPort":5572,"protocol":"TCP"}],"resources":{"limits":{"memory":"128Mi"},"requests":{"cpu":"50m","memory":"128Mi"}},"volumeMounts":[{"name":"mlserver-models","mountPath":"/mnt/agent"}],"readinessProbe":{"tcpSocket":{"port":5572},"initialDelaySeconds":5,"timeoutSeconds":1,"periodSeconds":5,"successThreshold":1,"failureThreshold":3},"lifecycle":{"preStop":{"httpGet":{"path":"terminate","port":9007}}},"imagePullPolicy":"IfNotPresent"}],"terminationGracePeriodSeconds":120,"serviceAccountName":"seldon-server","securityContext":{"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true,"fsGroup":1000}}},"volumeClaimTemplates":[{"metadata":{"name":"mlserver-models","creationTimestamp":null},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}}},"status":{}}],"serviceName":"mlserver","podManagementPolicy":"Parallel","updateStrategy":{}},"status":{"replicas":0,"availableReplicas":0}}`
 	sts := appsv1.StatefulSet{}
 	err := json.Unmarshal([]byte(stsJson), &sts)
 	g.Expect(err).To(BeNil())
@@ -271,7 +271,7 @@ func TestStatefulSetReconcile(t *testing.T) {
 				ObjectMeta: metav1.ObjectMeta{
 					Name:      "foo",
 					Namespace: "default",
-					Labels:    map[string]string{constants.AppKey: constants.ServerLabelValue},
+					Labels:    map[string]string{constants.KubernetesNameLabelKey: constants.ServerLabelValue},
 				},
 				Spec: appsv1.StatefulSetSpec{
 					ServiceName: "foo",
@@ -281,7 +281,7 @@ func TestStatefulSetReconcile(t *testing.T) {
 					},
 					Template: v1.PodTemplateSpec{
 						ObjectMeta: metav1.ObjectMeta{
-							Labels:    map[string]string{constants.ServerLabelNameKey: "foo", constants.AppKey: constants.ServerLabelValue},
+							Labels:    map[string]string{constants.ServerLabelNameKey: "foo", constants.KubernetesNameLabelKey: constants.ServerLabelValue},
 							Name:      "foo",
 							Namespace: "default",
 						},
@@ -417,8 +417,8 @@ func TestToStatefulSet(t *testing.T) {
 					Name:      "foo",
 					Namespace: "default",
 					Labels: map[string]string{
-						constants.AppKey: constants.ServerLabelValue,
-						"l1":             "l1val"},
+						constants.KubernetesNameLabelKey: constants.ServerLabelValue,
+						"l1":                             "l1val"},
 					Annotations: map[string]string{"a1": "a1val"},
 				},
 				Spec: appsv1.StatefulSetSpec{
@@ -430,8 +430,8 @@ func TestToStatefulSet(t *testing.T) {
 					Template: v1.PodTemplateSpec{
 						ObjectMeta: metav1.ObjectMeta{
 							Labels: map[string]string{constants.ServerLabelNameKey: "foo",
-								constants.AppKey: constants.ServerLabelValue,
-								"l1":             "l1val"},
+								constants.KubernetesNameLabelKey: constants.ServerLabelValue,
+								"l1":                             "l1val"},
 							Annotations: map[string]string{"a1": "a1val"},
 							Name:        "foo",
 							Namespace:   "default",