Establishing a network perimeter
By: Adam Dunstan
Tags: Kubernetes, Network Security, Networking

This may seem a little obvious however one of the most important decisions you can make in the design of a k8s system is identifying the need for a system perimeter. All networks have perimeters, including public cloud services, and understanding how the k8s system fits will ensure that the level of security you require will be maintained.

The perimeter is the network boundary where access is controlled to the hosts that reside within. In Enterprises these are often called zones and access to and between these zones is controlled with firewalls. In a non-cloud native environment, those network devices are independently managed, either manually or using network-specific orchestration systems. In a cloud-native system, the k8s system is used for this function.

The most important question is do you need one? If your system is a lab system, is already within an enterprise security zone and access from other zones including the external internet is not required, there are few constraints on your network design. However, if your applications require external or intra-zone access the solution can get significantly more complex, especially if network administration is outside of our control, the choices you have to integrate your k8s system and build a cloud-native infrastructure may be limited.

What access is required?

One of the great things about k8s is that once installed, routine access to the underlying infrastructure is only required by the infra-team.

kubectl. All system users require access to the command line tool kubectl for operation and connecting to a running container to get a shell prompt where necessary. Installed on remote machines it used TLS for all API traffic, with API authorization dependent upon your configuration and API authorization provided by RBAC.

Services. Services are required to expose any POD running in the complex. Services exposed with NodePort require access to the host interface, those exposed via LoadBalancer can be configured to avoid direct access to the hosts as can externalIPs.

Boundary Alternatives

Rely on the existing infrastructure. The existing infrastructure will provide at least the same level of perimeter security that previously existed. Your cloud native workflow will be impacted without some additional work on your behalf. When a service is created to expose an application that requires access from outside the perimeter, firewall rules will need to be adjusted and without some automation will need to be fairly permissive. Creating an operator to manage your external access is an alternative or if you have a BGP capable device a solution like metallb can help although it depends on specific functionality not present in some popular routers. Of course access for kubectl is still required.

Develop a new boundary. A new boundary is the best solution, even if it is contained within the existing infrastructure. A new boundary protects the k8s hosts, provides access to applications controlled by k8s integrated with workflow, and ensures access is controlled by k8s. With a new boundary established, the new boundary devices can be attached to any existing network without security concern.

CNI role in the Perimeter

Many CNI’s have default configurations designed to avoid getting into network administration, these CNI’s default to tunneling. The purpose described by the CNI is to provide connectivity between POD networks on hosts however, there is nothing unique about routing between hosts, just involvement from the networking team is required.

However the CNI can become more important should an existing perimeter be used and the k8s hosts are interspersed with other hosts. As the other hosts may have access to the nodes, host firewall rules are required. Manually implementing these rules is impossible as services are dynamically defined and k8s manages host forwarding. Network policy is included as part of k8s but requires support from the CNI, however not all have this support. Using network policy is good practice anyway in a layered approach to security.

A simple design

A simple design that we have previously uses creates a perimeter around all of the k8s hosts and its required service hosts.

In this design, storage is provided by Ceph and the storage nodes are separated from the k8s system (we didn’t use rook) to reduce fate sharing. The storage system could be used independently of k8s via an s3 gateway. To provide access to kubectl and S3, a number of proxies were deployed using haproxy and TLS was passed though to allow k8s and ceph to control access.

In addition to the k8s and ceph nodes, there are other service hosts deployed inside the perimeter. These included the hosts necessary to install hosts and switches as well as other basic functions such as external DNS and important for ceph, NTP. These hosts required access by the infrastructure team so a VPN gateway was added with access for only the infrastructure team.

Services are exposed using the LoadBalancer mechanism. Within the k8s system there are a number of ingresses including a service mesh. We used MetalLb in BGP to mode allocate and advertise IP addresses for external services and updated the external DNS using ExternalDNS for Kubernetes. This automation enabled a service to be exposed and DNS entries created based upon a service definition.

The specifics of routing will vary based upon your environment. If directly connected to a public service provider, all of the traffic would be routed via the same router used by the services.

Summary

Thinking about the perimeter structure during the development of k8s infrastructure is critical, especially if you are trying to create a similar development experience to public cloud. Getting stuck with legacy firewall and security processes will have your developers quickly moving back to the convenience of public cloud.