Project: K8sGPT (Observability and Analysis / Observability)
File: README.md

The serve commands allow you to run k8sgpt in a grpc server mode. This would be enabled typically through `k8sgpt serve` and is how the in-cluster k8sgpt deployment functions when managed by the The grpc interface that is served is hosted on and the repository for this is A fantastic tool for local debugging and development is `grpcurl` It allows you to form curl like requests that are http2 e.g. ``` grpcurl -plaintext -d '{"namespace": "k8sgpt", "explain" : "true"}' localhost:8080 schema.v1.ServerService/Analyze ``` ``` grpcurl -plaintext localhost:8080 schema.v1.ServerService/ListIntegrations { "integrations": [ "trivy" ] } ``` ``` grpcurl -plaintext -d '{"integrations":{"trivy":{"enabled":"true","namespace":"default","skipInstall":"false"}}}' localhost:8080 schema.v1.ServerService/AddConfig ``` ub.com%2Fk8sgpt-ai%2Fk8sgpt?ref=badge_shield) ](https://opensource.org/licenses/Apache-2.0) ](https://github.com/k8sgpt-ai/k8sgpt) ](https://codecov.io/github/k8sgpt-ai/k8sgpt) `k8sgpt` is a tool for scanning your Kubernetes clusters, diagnosing, and triaging issues in simple English. It has SRE experience codified into its analyzers and helps to pull out the most relevant information to enrich it with AI. Out of the box integration with OpenAI, Azure, Cohere, Amazon Bedrock, Google Gemini and local models. <a href="https://www.producthunt.com/posts/k8sgpt?utmsource=badge-featured&utmmedium=badge&utmsouce=badge-k8sgpt" target="blank"><img src="https://api.producthunt.com/widgets/embed-image/v1/featured.svg?post_id=389489&theme=light" alt="K8sGPT - K8sGPT&#0032;gives&#0032;Kubernetes&#0032;Superpowers&#0032;to&#0032;everyone | Product Hunt" style="width: 250px; height: 54px;" width="250" height="54" /></a> <img src="images/demo4.gif" width=650px; /> ``` brew tap k8sgpt-ai/k8sgpt brew install k8sgpt ``` <details> <summary>RPM-based installation (RedHat/CentOS/Fedora)</summary> 32 bit: <!x-release-please-start-version--> ``` curl -LO https://github.com/k8sgpt-ai/k8sgpt/releases/download/v0.3.31/k8sgpt_386.rpm sudo rpm -ivh k8sgpt_386.rpm ``` <!x-release-please-end--> 64 bit: <!x-release-please-start-version--> ``` curl -LO https://github.com/k8sgpt-ai/k8sgpt/releases/download/v0.3.31/k8sgpt_amd64.rpm sudo rpm -ivh -i k8sgpt_amd64.rpm ``` <!x-release-please-end--> </details> <details> <summary>DEB-based installation (Ubuntu/Debian)</summary> 32 bit: <!x-release-please-start-version--> ``` curl -LO https://github.com/k8sgpt-ai/k8sgpt/releases/download/v0.3.31/k8sgpt_386.deb sudo dpkg -i k8sgpt_386.deb ``` <!x-release-please-end--> 64 bit: <!x-release-please-start-version--> ``` curl -LO https://github.com/k8sgpt-ai/k8sgpt/releases/download/v0.3.31/k8sgpt_amd64.deb sudo dpkg -i k8sgpt_amd64.deb ``` <!x-release-please-end--> </details> <details> <summary>APK-based installation (Alpine)</summary> 32 bit: <!x-release-please-start-version--> ``` curl -LO https://github.com/k8sgpt-ai/k8sgpt/releases/download/v0.3.31/k8sgpt_386.apk apk add k8sgpt_386.apk ``` <!x-release-please-end--> 64 bit: <!x-release-please-start-version--> ``` curl -LO https://github.com/k8sgpt-ai/k8sgpt/releases/download/v0.3.31/k8sgpt_amd64.apk apk add k8sgpt_amd64.apk ``` <!x-release-please-end-->x </details> <details> <summary>Failing Installation on WSL or Linux (missing gcc)</summary> When installing Homebrew on WSL or Linux, you may encounter the following error: ``` ==> Installing k8sgpt from k8sgpt-ai/k8sgpt Error: The following formula cannot be installed from a bottle and must be built from the source. k8sgpt Install Clang or run brew install gcc. ``` If you install gcc as suggested, the problem will persist. Therefore, you need to install the build-essential package. ``` sudo apt-get update sudo apt-get install build-essential ``` </details> Download the latest Windows binaries of k8sgpt* from the tab based on your system architecture. Extract the downloaded package to your desired location. Configure the system path* variable with the binary location To install within a Kubernetes cluster please use our `k8sgpt-operator` with installation instructions available This mode of operation is ideal for continuous monitoring of your cluster and can integrate with your existing monitoring such as Prometheus and

Currently, the default AI provider is OpenAI, you will need to generate an API key from You can do this by running `k8sgpt generate` to open a browser link to generate it Run `k8sgpt auth add` to set it in k8sgpt. You can provide the password directly using the `--password` flag. Run `k8sgpt filters` to manage the active filters used by the analyzer. By default, all filters are executed during analysis. Run `k8sgpt analyze` to run a scan. And use `k8sgpt analyze --explain` to get a more detailed explanation of the issues. You also run `k8sgpt analyze --with-doc` (with or without the explain flag) to get the official documentation from Kubernetes. K8sGPT uses analyzers to triage and diagnose issues in your cluster. It has a set of analyzers that are built in, but you will be able to write your own analyzers. [x] podAnalyzer [x] pvcAnalyzer [x] rsAnalyzer [x] serviceAnalyzer [x] eventAnalyzer [x] ingressAnalyzer [x] statefulSetAnalyzer [x] deploymentAnalyzer [x] cronJobAnalyzer [x] nodeAnalyzer [x] mutatingWebhookAnalyzer [x] validatingWebhookAnalyzer [x] hpaAnalyzer [x] pdbAnalyzer [x] networkPolicyAnalyzer [x] gatewayClass [x] gateway [x] httproute [x] logAnalyzer Run a scan with the default analyzers ``` k8sgpt generate k8sgpt auth add k8sgpt analyze --explain k8sgpt analyze --explain --with-doc ``` Filter on resource ``` k8sgpt analyze --explain --filter=Service ``` Filter by namespace ``` k8sgpt analyze --explain --filter=Pod --namespace=default ``` Output to JSON ``` k8sgpt analyze --explain --filter=Service --output=json ``` Anonymize during explain ``` k8sgpt analyze --explain --filter=Service --output=json --anonymize ``` <details> <summary> Using filters </summary> List filters ``` k8sgpt filters list ``` Add default filters ``` k8sgpt filters add [filter(s)] ``` Simple filter : `k8sgpt filters add Service` Multiple filters : `k8sgpt filters add Ingress,Pod` Remove default filters ``` k8sgpt filters remove [filter(s)] ``` Simple filter : `k8sgpt filters remove Service` Multiple filters : `k8sgpt filters remove Ingress,Pod` </details> <details> <summary> Additional commands </summary> List configured backends ``` k8sgpt auth list ``` Update configured backends ``` k8sgpt auth update $MYBACKEND1,$MYBACKEND2.. ``` Remove configured backends ``` k8sgpt auth remove -b $MYBACKEND1,$MYBACKEND2.. ``` List integrations ``` k8sgpt integrations list ``` Activate integrations ``` k8sgpt integrations activate [integration(s)] ``` Use integration ``` k8sgpt analyze --filter=[integration(s)] ``` Deactivate integrations ``` k8sgpt integrations deactivate [integration(s)] ``` Serve mode ``` k8sgpt serve ``` Analysis with serve mode ``` grpcurl -plaintext -d '{"namespace": "k8sgpt", "explain": false}' localhost:8080 schema.v1.ServerService/Analyze ``` </details> K8sGPT uses the chosen LLM, generative AI provider when you want to explain the analysis results using --explain flag e.g. `k8sgpt analyze --explain`. You can use `--backend` flag to specify a configured provider (it's `openai` by default). You can list available providers using `k8sgpt auth list`: ``` Default: openai Active: Unused: openai localai azureopenai cohere amazonbedrock amazonsagemaker google huggingface noopai googlevertexai ``` For detailed documentation on how to configure and use each provider see . To set a new default provider ``` k8sgpt auth default -p azureopenai Default provider set to azureopenai ``` <details> With this option, the data is anonymized before being sent to the AI Backend. During the analysis execution, `k8sgpt` retrieves sensitive data (Kubernetes object names, labels,

This data is masked when sent to the AI backend and replaced by a key that can be used to de-anonymize the data when the solution is returned to the user. <summary> Anonymization </summary> Error reported during analysis: ```bash Error: HorizontalPodAutoscaler uses StatefulSet/fake-deployment as ScaleTargetRef which does not exist. ``` Payload sent to the AI backend: ```bash Error: HorizontalPodAutoscaler uses StatefulSet/tGLcCRcHa1Ce5Rs as ScaleTargetRef which does not exist. ``` Payload returned by the AI: ```bash The Kubernetes system is trying to scale a StatefulSet named tGLcCRcHa1Ce5Rs using the HorizontalPodAutoscaler, but it cannot find the StatefulSet. The solution is to verify that the StatefulSet name is spelled correctly and exists in the same namespace as the HorizontalPodAutoscaler. ``` Payload returned to the user: ```bash The Kubernetes system is trying to scale a StatefulSet named fake-deployment using the HorizontalPodAutoscaler, but it cannot find the StatefulSet. The solution is to verify that the StatefulSet name is spelled correctly and exists in the same namespace as the HorizontalPodAutoscaler. ``` Note: Anonymization does not currently apply to events. Anonymization does not currently apply to events. In a few analysers like Pod, we feed to the AI backend the event messages which are not known beforehand thus we are not masking them for the time being. The following is the list of analysers in which data is being masked:- Statefulset Service PodDisruptionBudget Node NetworkPolicy Ingress HPA Deployment Cronjob The following is the list of analysers in which data is not being masked:- RepicaSet PersistentVolumeClaim Pod Log _*Events_ *Note: k8gpt will not mask the above analysers because they do not send any identifying information except Events analyser. Masking for Events analyzer is scheduled in the near future as seen in this . Further research has to be made to understand the patterns and be able to mask the sensitive parts of an event like pod name, namespace etc. The following is the list of fields which are not being masked:- Describe ObjectStatus Replicas ContainerStatus _*Event Message_ ReplicaStatus Count (Pod) *Note: It is quite possible the payload of the event message might have something like "super-secret-project-pod-X crashed" which we don't currently redact (scheduled in the near future as seen in this ). The K8gpt team recommends using an entirely different backend (a local model) in critical production environments. By using a local model, you can rest assured that everything stays within your DMZ, and nothing is leaked. If there is any uncertainty about the possibility of sending data to a public LLM (open AI, Azure AI) and it poses a risk to business-critical operations, then, in such cases, the use of public LLM should be avoided based on personal assessment and the jurisdiction of risks involved. </details> <details> <summary> Configuration management</summary> `k8sgpt` stores config data in the `$XDGCONFIGHOME/k8sgpt/k8sgpt.yaml` file. The data is stored in plain text, including your OpenAI

Config file locations: | OS | Path | | - | | | MacOS | ~/Library/Application Support/k8sgpt/k8sgpt.yaml | | Linux | ~/.config/k8sgpt/k8sgpt.yaml | | Windows | %LOCALAPPDATA%/k8sgpt/k8sgpt.yaml | </details> <details> There may be scenarios where caching remotely is preferred. In these scenarios K8sGPT supports AWS S3 or Azure Blob storage Integration. <summary> Remote caching </summary> <em>Note: You can only configure and use only one remote cache at a time</em> Adding a remote cache AWS S3 As a prerequisite `AWSACCESSKEYID` and `AWSSECRETACCESSKEY` are required as environmental variables. Configuration, ``` k8sgpt cache add s3 --region <aws region> --bucket <name> ``` Minio Configuration with HTTP endpoint ``` k8sgpt cache add s3 --bucket <name> --endpoint <http://localhost:9000>``` Minio Configuration with HTTPs endpoint, skipping TLS verification ``` k8sgpt cache add s3 --bucket <name> --endpoint <https://localhost:9000> --insecure``` K8sGPT will create the bucket if it does not exist Azure Storage We support a number of to authenticate against Azure Configuration, ``` k8sgpt cache add azure --storageacc <storage account name> --container <container name> ``` K8sGPT assumes that the storage account already exist and it will create the container if it does not exist It is the user* responsibility have to grant specific permissions to their identity in order to be able to upload blob files and create SA containers (e.g Storage Blob Data Contributor) Google Cloud Storage As a prerequisite `GOOGLEAPPLICATIONCREDENTIALS` are required as environmental variables. Configuration, ``` k8sgpt cache add gcs --region <gcp region> --bucket <name> --projectid <project id>``` K8sGPT will create the bucket if it does not exist Listing cache items ``` k8sgpt cache list ``` Purging an object from the cache Note: purging an object using this command will delete upstream files, so it requires appropriate permissions. ``` k8sgpt cache purge $OBJECT_NAME ``` Removing the remote cache Note: this will not delete the upstream S3 bucket or Azure storage container ``` k8sgpt cache remove ``` </details> <details> <summary> Custom Analyzers</summary> There may be scenarios where you wish to write your own analyzer in a language of your choice. K8sGPT now supports the ability to do so by abiding by the and serving the analyzer for consumption. To do so, define the analyzer within the K8sGPT configuration and it will add it into the scanning process. In addition to this you will need to enable the following flag on analysis: ``` k8sgpt analyze --custom-analysis ``` Here is an example local host analyzer in When this is run on `localhost:8080` the K8sGPT config can pick it up with the following additions: ``` custom_analyzers: name: host-analyzer connection: url: localhost port: 8080 ``` This now gives the ability to pass through hostOS information ( from this analyzer example ) to K8sGPT to use as context with normal analysis. See the docs on how to write a custom analyzer </details> Find our official documentation available Please read our . Find us on <a href="https://github.com/k8sgpt-ai/k8sgpt/graphs/contributors"> <img src="https://contrib.rocks/image?repo=k8sgpt-ai/k8sgpt" /> </a> ](https://app.fossa.com/projects/git%2Bgithub.com%2Fk8sgpt-ai%2Fk8sgpt?ref=badge_large)