Fairwinds Insights: CI Pipeline to Protect Your Kubernetes Clusters
Tags: Ci, Kubernetes, Continous Deployment, Continuous Integration

About a year ago, we built Fairwinds Insights to make our lives easier. We were managing hundreds of clusters across dozens of organizations, and needed a way to keep track of how healthy those clusters were in terms of security, efficiency, and reliability. So we built a platform for running different Kubernetes auditing tools and aggregating the results into a single dashboard.

Having a birds-eye view of the health of our clusters gives us — and a growing number of customers — the visibility needed to run Kubernetes confidently. It gives the ability to easily add new report types whenever there is something new we want to audit.

But our customers quickly pointed out that Insights could only catch mistakes after they were made! Since it could only analyze a running cluster, any issues it found were already affecting production.

Ideally we’d be able to catch these issues early in the process, and stop them from ever reaching a live cluster. So we built two new ways to run Fairwinds Insights: a Continuous Integration pipeline, which can uncover problems in Infrastructure-as-Code during peer review, and an Admission Controller, which can prevent problematic resources from entering the cluster.

Fairwinds Insights in Continuous Integration

The best time to catch security vulnerabilities and other problems is in peer review and before they make it into the main branch of your code repository. As folks edit Dockerfiles and Kubernetes manifests, they may inadvertently install vulnerable software, introduce security flaws, or omit critical pieces of configuration like resource requests or health probes.

Some of these problems can be detected by a thorough code review from a seasoned Kubernetes expert, but that process can be tedious and error-prone. Fairwinds Insights’ CI integration automates many of the most important checks to help reviewers discover problems quickly.

Insights can be configured to show a warning when new problems are introduced, or to prevent developers from merging their changes so long as issues are present.

When issues are found, developers can visit Fairwinds Insights to learn more about what the problems are, and what they can do to fix them:

Currently, Insights is able to check Kubernetes manifests for configuration issues using Polaris, as well as scan Docker images for known CVEs using Trivy. In the near future, we will add support for building custom configuration policies using OPA, as well as detecting deprecated resources using Pluto.

Of course, not every resource in your cluster goes through a CI process — some are created automatically by Kubernetes controllers, and sometimes engineers will create or edit resources manually using kubectl and Helm. This is why having an Admission Controller is important.

Insights Admission Controller

Ideally, everything in your cluster could be traced back to version-controlled Infrastructure-as-Code (IaC). Teams that utilize IaC have a much easier time auditing changes, rolling back mistakes, and understanding what’s running in their cluster.

But there are always ways for things to get into your cluster without having gone through IaC review. Sometimes you’ll need to hot-fix things, or maybe an engineer accidentally ran kubectl apply in the wrong context. So in addition to CI checks, it’s critical to have a fallback mechanism for preventing security, efficiency, and reliability issues from getting into your cluster. Admission Controllers are the Kubernetes-native solution here.

The Fairwinds Insights Admission Controller runs every time a new resource is added to the cluster. If the resource violates your organization’s policies, the Admission Controller will reject it and notify the client (typically, a developer running kubectl or helm) what they need to change.

Fairwinds Insights allows you to control Admission Controllers in each of your clusters from a single, centralized location. If you want to implement a new policy, or create an exemption, you only need to touch one system, and your changes are automatically federated out everywhere. You can also configure unique policies for certain clusters, e.g. if you want to be stricter in prod versus staging.

Shifting Left

Of course, Fairwinds Insights still does a great job of scanning everything currently running inside your Kubernetes cluster to surface existing issues, and this will always be the truest view into how secure, efficient, and reliable your infrastructure is. But by shifting left into CI pipelines and Admission Control, we’re able to prevent problems before they make it into the cluster.

Furthermore, Fairwinds Insights is able to apply the same policies (or variations of them) from CI to Admission to production, across every cluster in your fleet. This way, everyone is held to the same standard, and those standards are applied consistently throughout the development cycle.

With the new CI and Admission Controller functionality, Ops teams can rest easy knowing that once they’ve gotten their clusters into a good state, Fairwinds Insights will make sure they stay that way.

Resources