Repository: kubernetes-sigs/kubebuilder (language: go)

Issue:
Unable to visually metrics in prometheus following steps provided
### What broke? What's expected?

After following https://book.kubebuilder.io/reference/metrics.html, the metrics from the controller don't show up in the prometheus UI.

I can see metrics in the namespace, but they come from the control plane (apiserver, kubelet, kcm, etc)

By default, the kube-prometheus installation can only monitor default, kube-system and monitoring namespaces. The solution is to add an extra Role and RoleBinding for your namespace, similar to these ones: https://github.com/prometheus-operator/kube-prometheus/blob/main/manifests/prometheus-roleSpecificNamespaces.yaml

The Role and RoleBinding could be included in the scaffolding.

### Reproducing this issue

In a brand new kubebuilder project, follow the steps in https://book.kubebuilder.io/reference/metrics.html using the latest kube-prometheus (v0.10.0)

### KubeBuilder (CLI) Version

3.3.0

### PROJECT version

3

### Plugin versions

_No response_

### Other versions

_No response_

### Extra Labels

_No response_


Discussion / hints:
HI @alculquicondor, 

In the e2e tests, we install the Prometheus and we check the metrics:

- See that we grant the permissions:
https://github.com/kubernetes-sigs/kubebuilder/blob/master/test/e2e/v3/plugin_cluster_test.go#L192-L197
- Then, we apply the ServiceMonitor:
https://github.com/kubernetes-sigs/kubebuilder/blob/master/test/e2e/v3/plugin_cluster_test.go#L209-L221
- By the latest, we check the metrics:
https://github.com/kubernetes-sigs/kubebuilder/blob/master/test/e2e/v3/plugin_cluster_test.go#L287-L292
 
Could you please check and let us know why we cannot face the issue raised by you?
Have we a way to reproduce your scenario with the tests? 


That's curling the metrics directly from `e2e-foo-controller-manager-metrics-service`. That works fine.

The problem is that prometheus can't list the Service in the controller's namepace. So maybe the E2E test could additionally query the metrics from the `prometheus-k8s` service?
Hi @alculquicondor, 

It shows me that the extra roles required are related to how to configure the Prometheus. Therefore it is not something that we can address in the default scaffolds because users might have a Prometheus installed/configured in another way. 

However,  we could add to the doc: https://book.kubebuilder.io/reference/metrics.html the required steps via a note. Something like: 
> If you installed the Prometheus using the Operator ...

Also, we should probably add a link to the Prometheus Operator documentation which describes that it only has access by default to the namespace informed and also how to fix it. However, I could not find this documentation there either. So, I'd like to know if you can help us address this issue by:

a) Creating an issue to the Prometheus Operator project asking for it to be properly documented 
> In POV, it would be nice we have the Prometheus Operator maintainer's input to be sure that we are making the right assumptions before we add the note in the Kubebuilder doc. They might be able to let us know what exactly the file/manifest should be applied to and the best way to do it. 

b) Then, we need a PR to the Kubebuilder docs with this note and the approach/steps confirmed/shared by them.

WDYT? 
Have you any other suggestions to address this issue?
Could you help us out here address this need?



I don't see it too different from creating a ServiceMonitor in the scaffolds.

But yes, I'll open the issue in prometheus-operator.

/assign
Thank you for your help 👍 
WDYT about we first check the behaviour with Operator-Prometheus maintainers (what is their recommendation to address the needs and default behaviour) then we can discuss if the best approach is to solve it via docs or scaffold? Also, could you please add here the link for the issue open there?


Hi, There are a few resources that are missing in order for Prometheus to be able to scrape the metrics and alerts:
1. Update file config/rbac/kustomization.yaml - Add 2 lines:
```
- prometheus_role.yaml
- prometheus_role_binding.yaml
```
$ vi config/rbac/kustomization.yaml
```
resources:
# All RBAC will be applied under this service account in
# the deployment namespace. You may comment out this resource
# if your manager will use a service account that exists at
# runtime. Be sure to update RoleBinding and ClusterRoleBinding
# subjects if changing service account names.
- service_account.yaml
- role.yaml
- role_binding.yaml
- prometheus_role.yaml
- prometheus_role_binding.yaml
- leader_election_role.yaml
- leader_election_role_binding.yaml
# Comment the following 4 lines if you want to disable
# the auth proxy (https://github.com/brancz/kube-rbac-proxy)
# which protects your /metrics endpoint.
- auth_proxy_service.yaml
- auth_proxy_role.yaml
- auth_proxy_role_binding.yaml
- auth_proxy_client_clusterrole.yaml
```
2. Create file config/rbac/prometheus_role.yaml

$ vi config/rbac/prometheus_role.yaml
```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-role
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
```
3. Create file config/rbac/prometheus_role_binding.yaml

$ vi config/rbac/prometheus_role_binding.yaml
```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus-role
subjects:
- kind: ServiceAccount
  name: prometheus-k8s
  namespace: monitoring
```

@alculquicondor @sradco @machadovilaca,

@joelanford make a good point related to this one:

**Why we should not scaffold the RBAC**

> It assumes that Prometheus operator is installed in a specific namespace (monitoring) and uses a specific SA (prometheus-k8s)

**Therefore, why we scaffold the ServiceMonitor and not the RBAC?**
Because that is specific to the Operator itself.

Then, the best approach to move forward here is only to ensure that we have the info in the [docs](https://book.kubebuilder.io/reference/metrics.html). Thereby, others will not get stuck. 

Such as: 

```
<aside class="warning">
<h2>Permissions required for Prometheus Operator have access to the metrics</h2>

Note that for Promethues Operator be able to use the metrics it needs to have access to 
the ServiceMonitor which is scaffold under `config/promethues` directory in your Project.
In this way, if you have not Promethues installed already ensure that you will create the Cluster Rule
and Cluster Rule Binding described in the [Promethues Operator documentation](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/rbac.md).

</aside>

```
Are you saying that the scaffold for ServiceMonitor should be removed too? That sounds consistent to me.
Hi @alculquicondor, 

> Are you saying that the scaffold for ServiceMonitor should be removed too? That sounds consistent to me.

No. We should not remove the ServiceMonitor. It makes sense because it is specific to the Operator/Project.

However, we should NOT scaffold the RBAC rules because it is not specific for the project and the cluster which has the Prometheus Operator installed would have them also applied to be able to have access to the ServiceMonitor(s) available on the cluster. 

Therefore, the covey is we just add the note to docs as for example suggested in the comment https://github.com/kubernetes-sigs/kubebuilder/issues/2646#issuecomment-1190255943



Fix patch:
diff --git a/docs/book/src/reference/metrics.md b/docs/book/src/reference/metrics.md
index 0e44165f333..5e30d93199d 100644
--- a/docs/book/src/reference/metrics.md
+++ b/docs/book/src/reference/metrics.md
@@ -51,6 +51,7 @@ Follow the steps below to export the metrics using the Prometheus Operator:
    We recommend using [kube-prometheus](https://github.com/coreos/kube-prometheus#installing)
    in production if you don't have your own monitoring system.
    If you are just experimenting, you can only install Prometheus and Prometheus Operator.
+
 2. Uncomment the line `- ../prometheus` in the `config/default/kustomization.yaml`.
    It creates the `ServiceMonitor` resource which enables exporting the metrics.
 
@@ -69,6 +70,20 @@ NAME                                         AGE
 monitor-controller-manager-metrics-monitor   2m8s
 ```
 
+<aside class="warning">
+<h2>If you are using Prometheus Operator ensure that you have the required
+permissions</h2>
+
+If you are using Prometheus Operator, be aware that, by default, its RBAC
+rules are only enabled for the `default` and `kube-system namespaces`. See its
+guide to know [how to configure kube-prometheus to monitor other namespaces using the `.jsonnet` file](https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/monitoring-other-namespaces.md).
+
+Alternatively, you can give the Prometheus Operator permissions to monitor other namespaces using RBAC. See the Prometheus Operator
+[Enable RBAC rules for Prometheus pods](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/getting-started.md#enable-rbac-rules-for-prometheus-pods)
+documentation to know how to enable the permissions on the namespace where the
+`ServiceMonitor` and manager exist.
+</aside>
+
 Also, notice that the metrics are exported by default through port `8443`. In this way,
 you are able to check the Prometheus metrics in its dashboard. To verify it, search
 for the metrics exported from the namespace where the project is running


Test patch: