Project: Tracetest (Observability and Analysis / Observability)
File: setup-k3s.md

This example shows how one can use Tracetest to do trace based testing for kubectl commands. This works by leveraging the `Audit-Id` header sent back from the kubernetes api server for every request. the `kubectl` command allows to enable a very verbose output, where it includes the response headers. This example script parses that info and maps it to the corresponding Trace ID. This setup is a bit limited at the moment. It only works with Jaeger backend, becasue it provides a search API that allows us to search spans that include the correct `Audit-Id` tag. For everything to work, we need the following environment configured: A Kubernetes cluster with `APIServerTracing` enabled and configured A OtelCollector that can be accessed by the k8s api server to send its tracing data A Jager backend to store the traces k3s can be used to easily setup a cluster with the correct configuration. It's easy enough to be configured in any dev environment, even a CICD. The otelcollector cannot be deployed in the target cluster because it complicates the networking setup needed to have the k8s apiserver communicating with the collector. An easy solution for this is to have docker compose starting the otelcol/jaeger services, so their ports can be exposed directly to the host network. create a `docker-compose.yaml` file that includes both services: ```yaml services: jaeger: healthcheck: test: CMD wget --spider localhost:16686 timeout: 3s interval: 1s retries: 60 image: jaegertracing/all-in-one:latest restart: unless-stopped ports: 16686:16686 otel-collector: command: --config /otel-local-config.yaml depends_on: jaeger: condition: service_started image: otel/opentelemetry-collector:0.54.0 ports: 4317:4317 volumes: ./otel-collector.yaml:/otel-local-config.yaml ``` We need to create a config file in the same directory for the otel-collector so it can communicate with Jaeger: ```yaml receivers: otlp: protocols: grpc: http: processors: probabilistic_sampler: hash_seed: 22 sampling_percentage: 100 batch: timeout: 100ms exporters: logging: logLevel: debug jaeger: endpoint: jaeger:14250 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [probabilistic_sampler, batch] exporters:

# logging is optional, ``` With both files created on the same director, we can start them: ``` sudo docker compose up -d ``` We need to start a k3s cluster, but we need some special configurations: the k8s version needs to be 1.27+ for traces to work we need to enable the tracing feature flag we need to pass the otel-collector address to the apiserver First, we'll create the tracing config files, because it's required for the apiserver to start. Make sure that the directory exists by running: `sudo mkdir -p /etc/kube-tracing/` ```yaml apiVersion: apiserver.config.k8s.io/v1beta1 kind: TracingConfiguration endpoint: localhost:4317 samplingRatePerMillion: 1000000 # 100% EOF ``` ```yaml apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration featureGates: KubeletTracing: true tracing: endpoint: host.k3d.internal:4317 samplingRatePerMillion: 1000000 ``` This settings are documented . The only difference is that we are setting the `samplingRatePerMillion` value to 1.000.000 (meaning, 100%) so that all traces are send to the collector. This effectively disables sampling. Use this setting with care, and you probablly shouldn't do this in a prod environment. Now we can install the k3s cluster: ```sh curl -sfL https://get.k3s.io | \ INSTALLK3SVERSION="v1.27.1+k3s1" \ INSTALLK3SEXEC="--kube-apiserver-arg=feature-gates=APIServerTracing=true --kube-apiserver-arg=tracing-config-file=/etc/kube-tracing/apiserver-tracing.yaml --kubelet-arg=config=/etc/kube-tracing/kubelet-tracing.yaml" \ sh -s - server --cluster-init ``` We use the `INSTALLK3SVERSION` env var to set the correct version, and the `INSTALLK3SEXEC` to pass flags and settings to the k8s apiserver. That's it! We have a k8s apiserver ready to do trace based testing. With our it's very easy. You only need to set the correct jaeger UI url and the command you want to run. For example: ```sh JAEGERUIURL="http://127.0.0.1:16686" ./test-kubectl.bash ./test.yaml "kubectl get pods -A" ``` The `test.yaml` file can be named as you want, but it needs to look like this: ```yaml type: Test spec: id: ze90fyU4R # can be anything name: kubectl # can be anything trigger: type: traceid # MUST BE traceide traceid: id: ${var:asd} # MUST BE in line 8 specs: name: List span exists selector: span[tracetest.span.type="general" name="List"] assertions: attr:tracetest.selected_spans.count = 1 ``` You also need to have the `tracetest` cli configured to talk with the correct tracetest server. The tracetest server also needs to be configured to use the same Jaeger instance as the k8s for its datastore