npm package: @marcos_feitoza/prometheus
Description: #### Install .. .. .. ..
Latest version: 1.0.4


### PROMETHEUS

#### Install
..
..
..
..

## SSL Certificates Using cert-manager for Granafa


1. Generate the Certificate and Key:
```
# Generate a private key
openssl genrsa -out key.pem 2048

# Generate a self-signed certificate
openssl req -new -x509 -key key.pem -out cert.pem -days 365 -subj /CN=prometheus.local
```
2. Create a Kubernetes secret using these files:

```
kubectl create secret tls prometheus-tls --cert=cert.pem --key=key.pem -n monitoring
```
```
k describe secrets prometheus-tls-wh6ws -n monitoring                                                                                          
Name:         prometheus-tls-wh6ws
Namespace:    monitoring
Labels:       cert-manager.io/next-private-key=true
Annotations:  <none>

Type:  Opaque

Data
====
tls.key:  1704 bytes
```
3. Create Certificate Resource
```
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: prometheus-cert
  namespace: monitoring
spec:
  secretName: prometheus-tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: prometheus.local.com
  dnsNames:
    - prometheus.local.com
    - '*.prometheus.local.com'
```
4. Configure Ingress for prometheus

Set up an Ingress resource for prometheus to use the SSL certificate. This configuration will direct traffic to prometheus and apply the SSL/TLS settings.
```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: prometheus-ingress
  namespace: monitoring
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-staging
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  ingressClassName: nginx
  rules:
    - host: prometheus.local.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: prometheus
                port:
                  number: 80
  tls:
    - hosts:
        - prometheus.local.com
      secretName: prometheus-tls
```
5. Verify the Setup

#### secret
```
k get secret -n monitoring 
NAME                 TYPE     DATA   AGE
prometheus              Opaque   3      22h
prometheus-cert-n4r8f   Opaque   1      13h
prometheus-tls-wh6ws    Opaque   1      13h
```
```
k describe secrets prometheus-tls-wh6ws -n monitoring                                                                                                                                                                                    (pik8s/default)
Name:         prometheus-tls-wh6ws
Namespace:    monitoring
Labels:       cert-manager.io/next-private-key=true
Annotations:  <none>

Type:  Opaque

Data
====
tls.key:  1704 bytes
```
#### Certifiticates
```
kubectl get certificates -n monitoring
NAME           READY   SECRET        AGE
prometheus-cert   False   prometheus-tls   13h
prometheus-tls    False   prometheus-tls   13h
```
```
k describe certificates.cert-manager.io  prometheus-cert -n monitoring
Name:         prometheus-cert
Namespace:    monitoring
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2024-08-06T01:24:34Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:commonName:
        f:dnsNames:
        f:issuerRef:
          .:
          f:kind:
          f:name:
        f:secretName:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2024-08-06T01:24:33Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:nextPrivateKeySecretName:
    Manager:      cert-manager-certificates-key-manager
    Operation:    Update
    Subresource:  status
    Time:         2024-08-06T01:24:34Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
          .:
          k:{"type":"Issuing"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
            f:type:
    Manager:      cert-manager-certificates-trigger
    Operation:    Update
    Subresource:  status
    Time:         2024-08-06T01:24:34Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        f:conditions:
          k:{"type":"Ready"}:
            .:
            f:lastTransitionTime:
            f:message:
            f:observedGeneration:
            f:reason:
            f:status:
            f:type:
    Manager:         cert-manager-certificates-readiness
    Operation:       Update
    Subresource:     status
    Time:            2024-08-06T01:30:33Z
  Resource Version:  805312
  UID:               7839ce65-26c8-4bf7-b139-06b27c84c6ff
Spec:
  Common Name:  prometheus.local.com
  Dns Names:
    prometheus.local.com
    *.prometheus.local.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-staging
  Secret Name:  prometheus-tls
Status:
  Conditions:
    Last Transition Time:        2024-08-06T01:24:34Z
    Message:                     Issuing certificate as Secret was previously issued by Issuer.cert-manager.io/
    Observed Generation:         1
    Reason:                      IncorrectIssuer
    Status:                      True
    Type:                        Issuing
    Last Transition Time:        2024-08-06T01:24:34Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
  Next Private Key Secret Name:  prometheus-cert-n4r8f
Events:                          <none>
```

#### Verify Ingress: Check that the Ingress resource is correctly routing traffic and applying SSL/TLS.
```
kubectl logs -n cert-manager deployment/cert-manager
I0806 15:15:50.701719       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="prometheus.local.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-wft84" "related_resource_namespace"="monitoring" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="prometheus-tls-2fghh-3829399934-1227087645" "resource_namespace"="monitoring" "resource_version"="v1" "type"="HTTP-01"
I0806 15:15:50.701905       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="prometheus.local.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-npc6l" "related_resource_namespace"="monitoring" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="prometheus-tls-2fghh-3829399934-1227087645" "resource_namespace"="monitoring" "resource_version"="v1" "type"="HTTP-01"
I0806 15:15:50.702059       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="prometheus.local.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-ww52x" "related_resource_namespace"="monitoring" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="prometheus-tls-2fghh-3829399934-1227087645" "resource_namespace"="monitoring" "resource_version"="v1" "type"="HTTP-01"
E0806 15:15:50.710047       1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://prometheus.local.com/.well-known/acme-challenge/nikrW0u9cZ9I9redAqzjlcJqKpfEfMn8B6KpnNEkcoo': Get \"http://prometheus.local.com/.well-known/acme-challenge/nikrW0u9cZ9I9redAqzjlcJqKpfEfMn8B6KpnNEkcoo\": dial tcp: lookup prometheus.local.com on 10.96.0.10:53: no such host" "dnsName"="prometheus.local.com" "resource_kind"="Challenge" "resource_name"="prometheus-tls-2fghh-3829399934-1227087645" "resource_namespace"="monitoring" "resource_version"="v1" "type"="HTTP-01"
```

#### Troubleshooting
Certificate Status Issues: If the certificate is not issued, check the logs of cert-manager:
```
kubectl logs -n cert-manager deployment/cert-manager
```
* SSL Handshake Errors: Ensure the backend-protocol annotation matches prometheus’s service configuration and that prometheus is not set to serve HTTPS unless configured to do so. 

### How to reset prometheus's admin password (installed by helm)

```
namespace=monitoring
kubectl exec --namespace $namespace -it $(kubectl get pods --namespace $namespace -l "app.kubernetes.io/name=prometheus" -o jsonpath="{.items[0].metadata.name}") -- prometheus cli admin reset-admin-password yourNewPasswordHere

INFO[01-21|10:24:17] Connecting to DB                         logger=sqlstore dbtype=sqlite3
INFO[01-21|10:24:17] Starting DB migration                    logger=migrator

Admin password changed successfully ✔
```

# Prometheus (Métricas)

Este repositório entrega o Prometheus da stack de observabilidade do homelab.

## Papel na arquitetura

- Coleta métricas do cluster, nodes e aplicações.
- Serve dados PromQL para Grafana.
- Faz scrape de serviços de infraestrutura (incluindo ArgoCD via `extraScrapeConfigs`).

## Integração com Grafana, Vector e Loki

- Grafana consulta Prometheus para métricas (CPU, memória, disco, rede, status de pods, ArgoCD).
- Vector/Loki não dependem do Prometheus para logs, mas compartilham labels de contexto (`namespace`, `pod`, `node`, `app`) para correlação em dashboards.
- Fluxo operacional:
  - Métricas: `App/Infra -> Prometheus -> Grafana`
  - Logs: `Pods -> Vector -> Loki -> Grafana`

## Configuração principal

Arquivo de referência:

- `helm/prod-values.yaml`

Pontos críticos:

- `serverFiles.prometheus.yml`: jobs padrão de scrape Kubernetes.
- `extraScrapeConfigs`: jobs extras (ex.: ArgoCD).
- `kube-state-metrics.prometheusScrape`: atualmente `false` (importante para dashboards que dependem dessas métricas).

## Scrape ArgoCD

Já existe configuração para:

- `argocd-application-controller`
- `argocd-server`
- `argocd-repo-server`
- `argocd-applicationset-controller`

Origem: `extraScrapeConfigs` em `helm/prod-values.yaml`.

## Métricas essenciais para cluster

- CPU: `node_cpu_seconds_total`
- Memória: `node_memory_MemAvailable_bytes`
- Disco: `node_filesystem_avail_bytes`
- Rede: `node_network_receive_bytes_total`

## Verificação rápida

- Status do pod:

```bash
kubectl -n monitoring get pods -l app=prometheus,component=server -o wide
```

- Targets up/down (via API interna):

```bash
kubectl -n monitoring exec <prometheus-pod> -c prometheus-server -- \
  sh -c "wget -qO- 'http://127.0.0.1:9090/api/v1/targets'"
```

- Query de sanidade ArgoCD:

```bash
kubectl -n monitoring exec <prometheus-pod> -c prometheus-server -- \
  sh -c "wget -qO- 'http://127.0.0.1:9090/api/v1/query?query=argocd_app_info'"
```

## Troubleshooting comum

- Dashboard sem métrica:
  - Confirmar target `UP` em `/targets`.
  - Confirmar série existe via API `/api/v1/query`.
  - Revisar labels esperadas na query (ex.: `namespace`, `job`, `phase`).
- Métrica ausente de pod/app:
  - Checar annotations de scrape (`prometheus.io/scrape`, `prometheus.io/port`, `prometheus.io/path`).
- ArgoCD sem dados:
  - Validar serviços `argocd-metrics` e `argocd-server-metrics` no namespace `argocd`.

## Convenções

- Alterações de scrape somente via Git.
- Fixar versão de chart/imagem e evitar drift manual.