npm package: @marcos_feitoza/grafana
Latest version: 1.1.1

# Grafana (Observability UI)


### Grafana

#### Install
```
helm pull grafana/grafana --version 8.4.1
```
```
 helm template grafana ./grafana -n monitoring > grafana.yaml
 ```
## SSL Certificates Using cert-manager for Granafa


1. Generate the Certificate and Key:
```
# Generate a private key
openssl genrsa -out key.pem 2048

# Generate a self-signed certificate
openssl req -new -x509 -key key.pem -out cert.pem -days 365 -subj /CN=grafana.local
```
2. Create a Kubernetes secret using these files:

```
kubectl create secret tls grafana-tls --cert=cert.pem --key=key.pem -n monitoring
```
```
k describe secrets grafana-tls-wh6ws -n monitoring                                                                                          
Name:         grafana-tls-wh6ws
Namespace:    monitoring
Labels:       cert-manager.io/next-private-key=true
Annotations:  <none>

Type:  Opaque

Data
====
tls.key:  1704 bytes
```
3. Create Certificate Resource
```
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: grafana-cert
  namespace: monitoring
spec:
  secretName: grafana-tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: grafana.local.com
  dnsNames:
    - grafana.local.com
    - '*.grafana.local.com'
```
4. Configure Ingress for Grafana

Set up an Ingress resource for Grafana to use the SSL certificate. This configuration will direct traffic to Grafana and apply the SSL/TLS settings.
```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: grafana-ingress
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  ingressClassName: nginx
  rules:
    - host: grafana.local.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: grafana
                port:
                  number: 80
  tls:
    - hosts:
        - grafana.local.com
      secretName: grafana-tls
```
5. Verify the Setup

#### secret
```
k get secret -n monitoring 
NAME                 TYPE     DATA   AGE
grafana              Opaque   3      22h
grafana-cert-n4r8f   Opaque   1      13h
grafana-tls-wh6ws    Opaque   1      13h
```
```
k describe secrets grafana-tls-wh6ws -n monitoring                                                                                                                                                                                    (pik8s/default)
Name:         grafana-tls-wh6ws
Namespace:    monitoring
Labels:       cert-manager.io/next-private-key=true
Annotations:  <none>

Type:  Opaque

Data
====
tls.key:  1704 bytes
```
#### Certifiticates
```
kubectl get certificates -n monitoring
NAME           READY   SECRET        AGE
grafana-cert   False   grafana-tls   13h
grafana-tls    False   grafana-tls   13h
```
```
k describe certificates.cert-manager.io  grafana-cert -n monitoring
Name:         grafana-cert
Namespace:    monitoring
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2024-08-06T01:24:34Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:commonName:
        f:dnsNames:
        f:issuerRef:
          .:
          f:kind:
          f:name:
        f:secretName:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2024-08-06T01:24:33Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:nextPrivateKeySecretName:
    Manager:      cert-manager-certificates-key-manager
    Operation:    Update
    Subresource:  status
    Time:         2024-08-06T01:24:34Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
          .:
          k:{"type":"Issuing"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
            f:type:
    Manager:      cert-manager-certificates-trigger
    Operation:    Update
    Subresource:  status
    Time:         2024-08-06T01:24:34Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"Ready"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
            f:type:
    Manager:         cert-manager-certificates-readiness
    Operation:       Update
    Subresource:     status
    Time:            2024-08-06T01:30:33Z
  Resource Version:  805312
  UID:               7839ce65-26c8-4bf7-b139-06b27c84c6ff
Spec:
  Common Name:  grafana.local.com
  Dns Names:
    grafana.local.com
    *.grafana.local.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-staging
  Secret Name:  grafana-tls
Status:
  Conditions:
    Last Transition Time:        2024-08-06T01:24:34Z
    Message:                     Issuing certificate as Secret was previously issued by Issuer.cert-manager.io/
    Observed Generation:         1
    Reason:                      IncorrectIssuer
    Status:                      True
    Type:                        Issuing
    Last Transition Time:        2024-08-06T01:24:34Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
  Next Private Key Secret Name:  grafana-cert-n4r8f
Events:                          <none>
```

## Pods Overview: filtro `pod_regex`

No dashboard `Pods Overview`, o filtro de texto `pod_regex` permite filtrar pods por regex no nome.

Exemplos:

- Somente Grafana:
  - `^grafana-.*`
- Pods com `personal` no nome:
  - `.*personal.*`
- Pods com `crypto` no nome:
  - `.*crypto.*`
- Somente monitoring (grafana/loki/vector):
  - `^(grafana|loki|vector)-.*`
- Pod específico:
  - `^grafana-5d4c8dffdd-nrvk6$`

Para funcionar nos painéis e variáveis, use o filtro como regex:

- `=~"$pod_regex"`
- ou `/$pod_regex/`

#### Verify Ingress: Check that the Ingress resource is correctly routing traffic and applying SSL/TLS.
```
kubectl logs -n cert-manager deployment/cert-manager
I0806 15:15:50.701719       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="grafana.local.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-wft84" "related_resource_namespace"="monitoring" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="grafana-tls-2fghh-3829399934-1227087645" "resource_namespace"="monitoring" "resource_version"="v1" "type"="HTTP-01"
I0806 15:15:50.701905       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="grafana.local.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-npc6l" "related_resource_namespace"="monitoring" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="grafana-tls-2fghh-3829399934-1227087645" "resource_namespace"="monitoring" "resource_version"="v1" "type"="HTTP-01"
I0806 15:15:50.702059       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="grafana.local.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-ww52x" "related_resource_namespace"="monitoring" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="grafana-tls-2fghh-3829399934-1227087645" "resource_namespace"="monitoring" "resource_version"="v1" "type"="HTTP-01"
E0806 15:15:50.710047       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://grafana.local.com/.well-known/acme-challenge/nikrW0u9cZ9I9redAqzjlcJqKpfEfMn8B6KpnNEkcoo': Get \"http://grafana.local.com/.well-known/acme-challenge/nikrW0u9cZ9I9redAqzjlcJqKpfEfMn8B6KpnNEkcoo\": dial tcp: lookup grafana.local.com on 10.96.0.10:53: no such host" "dnsName"="grafana.local.com" "resource_kind"="Challenge" "resource_name"="grafana-tls-2fghh-3829399934-1227087645" "resource_namespace"="monitoring" "resource_version"="v1" "type"="HTTP-01"
```

#### Troubleshooting
Certificate Status Issues: If the certificate is not issued, check the logs of cert-manager:
```
kubectl logs -n cert-manager deployment/cert-manager
```
* SSL Handshake Errors: Ensure the backend-protocol annotation matches Grafana’s service configuration and that Grafana is not set to serve HTTPS unless configured to do so. 

### How to reset grafana's admin password (installed by helm)

```
namespace=monitoring
kubectl exec --namespace $namespace -it $(kubectl get pods --namespace $namespace -l "app.kubernetes.io/name=grafana" -o jsonpath="{.items[0].metadata.name}") -- grafana cli admin reset-admin-password yourNewPasswordHere

INFO[01-21|10:24:17] Connecting to DB                         logger=sqlstore dbtype=sqlite3
INFO[01-21|10:24:17] Starting DB migration                    logger=migrator

Admin password changed successfully ✔
```


Este repositório entrega o Grafana da stack de observabilidade do homelab.

## Papel na arquitetura

- Fonte de métricas: `Prometheus`
- Fonte de logs: `Loki`
- Coleta de logs no cluster: `Vector -> Loki`
- Dashboards versionados em Git (provisioning)

## Fluxo de integração (métricas + logs)

1. Apps e componentes do cluster expõem `/metrics`.
2. Prometheus faz scrape e armazena séries temporais.
3. Vector coleta logs dos pods (DaemonSet), normaliza labels (`namespace`, `pod`, `container`, `node`, `app`) e envia para Loki.
4. Grafana consulta Prometheus (métricas) e Loki (logs) com dashboards provisionados.

## Diagrama da stack

### Diagrama visual (Mermaid)

```mermaid
flowchart LR
  U[User] --> G[Grafana]

  subgraph K8s[Cluster Kubernetes]
    A[Pods de Aplicacao]
    M[Componentes de Infra\nArgoCD, kube-state-metrics, node-exporter]
    V[Vector DaemonSet]
    L[Loki]
    P[Prometheus]
  end

  A -->|/metrics| P
  M -->|/metrics| P
  A -->|stdout/stderr logs| V
  M -->|stdout/stderr logs| V
  V -->|labels + logs| L

  G -->|PromQL| P
  G -->|LogQL| L
```

### Código `python-diagrams` (para gerar imagem)

> Requer `python3` + pacote `diagrams` e Graphviz instalado no host.

```python
# diagrams/observability_stack.py
from diagrams import Cluster, Diagram
from diagrams.onprem.monitoring import Grafana, Prometheus
from diagrams.onprem.logging import Loki
from diagrams.k8s.compute import Pod

with Diagram("observability-stack", show=False, filename="grafana/docs/observability-stack"):
    user = Pod("user")

    with Cluster("kubernetes"):
        apps = Pod("apps")
        infra = Pod("infra metrics")
        vector = Pod("vector ds")
        loki = Loki("loki")
        prom = Prometheus("prometheus")
        grafana = Grafana("grafana")

    apps >> prom
    infra >> prom
    apps >> vector >> loki
    infra >> vector

    user >> grafana
    grafana >> prom
    grafana >> loki
```

Comando:

```bash
python3 grafana/diagrams/observability_stack.py
```

## Estrutura relevante

- `helm/prod-values.yaml`: configuração principal para produção.
- `helm/dashboards/*.json`: dashboards custom versionados.
- Datasources provisionados:
  - `Prometheus`: `http://prometheus-server.monitoring.svc.cluster.local`
  - `Loki`: `http://loki.monitoring.svc.cluster.local:3100`

## Dashboards principais

- `Nodes Overview`
- `Pods Overview`
- `Logs Overview`
- `Personal Finance Logs Overview`
- `ArgoCD`

## Filtro `pod_regex` no Pods Overview

Use regex para filtrar pods por nome:

- Somente grafana: `^grafana-.*`
- Tudo que contém `personal`: `.*personal.*`
- Tudo que contém `crypto`: `.*crypto.*`
- Somente monitoring core: `^(grafana|loki|vector)-.*`
- Pod exato: `^grafana-5d4c8dffdd-nrvk6$`

As queries devem usar:

- `pod=~"$pod_regex"`

## Operação (runbook rápido)

- Ver pods Grafana:

```bash
kubectl -n monitoring get pods -l app.kubernetes.io/name=grafana -o wide
```

- Ver logs:

```bash
kubectl -n monitoring logs -l app.kubernetes.io/name=grafana --tail=200
```

- Ver provisioning carregado no startup:

```bash
kubectl -n monitoring logs <grafana-pod> --tail=300 | grep -E "provisioning|dashboard|datasource"
```

## Troubleshooting comum

- Dashboard sem dados em painel de sync/atividade:
  - Validar janela de tempo (eventos podem ser raros em `30m`).
  - Testar mesma query no Explore.
- Pod novo do Grafana em `0/1` durante rollout:
  - Conferir probes e tempo de startup.
  - Verificar saúde do nó/CNI (`NodeNotReady`, flannel, etc.).
- Logs no Loki com `app=unknown`:
  - Ajustar labels no `vector/helm/templates/configmap.yaml`.

## Convenções

- Mudanças em dashboard/datasource sempre via Git + PR.
- Nada manual persistente no UI sem versionar JSON correspondente.