Preserving Client IPs through Google Cloud’s global TCP and SSL Proxy Load Balancers
By: Arend Dittmer
Tags: Cloud Computing, Load Balancing, Google Cloud Platform, Networking

When you are building network architectures that include reverse proxies — systems that connect to backend servers on behalf of clients — there is a good chance that the IP address of an originating client request is ‘lost’ at the proxy. This is a problem if you need the client IP address on the backend server for IP based white/black-listing of clients (e.g. for game servers), for secondary client IP based authentication, for logging of client requests or for troubleshooting purposes.

Google Cloud offers reverse proxies for global load balancing for TCP and SSL. In this article I will first show you step-by-step how to configure these Load Balancers with a configuration that preserves a client’s IP address in the TCP payload. Then we will see how we can use an OSS solution called go-mmproxy in conjunction with the GCP proxy Load Balancer to transparently pass that client IP to an application running on a backend server.

GCP TCP and SSL Proxy Load Balancing

Google Cloud’s TCP Proxy Load Balancer and SSL Proxy Load Balancer are great for making applications and services available at global scale. User connections are terminated at the load balancing layer and then proxied to backend instances. For the SSL Proxy Load Balancer you can choose between SSL and TCP for the connection from the proxy to the backend server. Note that even for TCP connections between proxy and backend server, traffic to the backend server will be encrypted.

GCP’s global TCP and SSL Proxy Load Balancers offer the following benefits

With support for anycast IPs you can use a single external IP address for all users worldwide, while traffic is being routed to backends that are geographically closest to the user

If a region is at capacity new connections are automatically directed to another region with available capacity

The TCP as well as the SSL Proxy Load Balancer can terminate IPv6 connections

SSL Proxy Load Balancer can secure applications that don’t provide native SSL support if you use TCP for the proxy-to-backend traffic

If you are not sure if the TCP Proxy Load Balancer is the right offering for your architecture, this page from the official GCP documentation can help you identify the best solution for your use case.

TCP and SSL Proxy Load Balancing are not well suited for HTTP(S) traffic. For HTTP(S) traffic use external HTTP(S) Load Balancing. Preserving the original client IP is also not an issue when using the HTTP(S) Load Balancer as it transparently sets the IP address of the client in the X-Forwarded-For header.

GCP TCP and SSL Proxy Load Balancing Concepts

Unlike illustrated in the simplified illustration above the load balancer actually comprises multiple components. Let’s zoom in on the GCP constructs that we will be configuring.

Incoming requests to a reserved External IP on a configured port are forwarded to the Target TCP/SSL Proxy as defined in a Global Forwarding Rule. The Target TCP/SSL Proxy terminates the TCP/SSL connection and forwards traffic to the associated backend service. If configured accordingly, the Target TCP/SSL Proxy preserves the client IP in a proxy header. The Global Backend Service defines how traffic is distributed, routing traffic to an Instance Group in the geographically closest region with resource availability. The Global Backend Service is associated with instance groups, that are a zonal grouping of all VMs that are accepting requests for a service. Instance groups are added to the backend service individually and the Load Balancing scheme for each instance group can be specified when it is added to the backend service. The individual nodes are monitored through a health check that updates the backend service.

For more details have a look at the GCP documentation providing a TCP Proxy Load Balancing overview and an SSL Proxy Load Balancing overview.

Configuring and Testing TCP and SSL Proxy Load Balancing

To keep things simple we will set up a single backend server behind the Load Balancer. This basic configuration can then easily be scaled out by adding instance groups and VMs in different geographies as per the Google Cloud documentation for setting up the TCP and SSL Proxy Load Balancers.

To follow, you must be able to create instances and modify a network in a project. You must be either a project owner or editor or you must have all of the following Compute Engine IAM roles

TCP and SSL Proxy

We first create the backend VM instance

gcloud compute instances create my-backend --image-project centos-cloud --image-family centos-7 --tags tcp-lb --zone us-central1-b

Then we create an instance group

gcloud compute instance-groups unmanaged create my-instance-group --zone us-central1-b

We create a named TCP port for the instance group, which is an alias for the port the Load Balancer will direct traffic to. We will use this named port later when we set up the backend service. We chose port 110 for this example. As go-mmproxy does not support SSL/TLS we are using TCP for the proxy-to-backend connection for the SSL proxy.

gcloud compute instance-groups set-named-ports my-instance-group --named-ports tcp110:110 --zone us-central1-b

We add the backend VM to the instance group. In a real world use case we would be adding multiple instances here.

gcloud compute instance-groups unmanaged add-instances my-instance-group --instances my-backend --zone us-central1-b

Now that have an instance group with an instance group exposing port 110 we can configure the Load Balancer. We create the health check on port 110.

gcloud compute health-checks create tcp my-tcp-health-check --port 110

We create the global backend service for TCP for the named port tcp110 that we created for the instance group

gcloud compute backend-services create my-tcp-lb --global-health-checks --global --protocol TCP --health-checks my-tcp-health-check --timeout 5m --port-name tcp110

Now we can add our instance group to the backend service. We specify UTILIZATION as the Load Balancing scheme for balancing traffic across instances within the instance group. max-utilization defines the maximum target for average utilization of backend instances in the backend instance group. We choose 80%, which means that traffic will be redirected to other instance groups if the average utilization exceeds 80%. For more details have a look at the gcloud documentation for adding a backend.

gcloud compute backend-services add-backend my-tcp-lb --global --instance-group my-instance-group --instance-group-zone us-central1-b --balancing-mode UTILIZATION --max-utilization 0.8

TCP Proxy

We create the target TCP proxy specifying that we need the client IP preserved in the proxy header

gcloud compute target-tcp-proxies create my-tcp-lb-target-proxy --backend-service my-tcp-lb --proxy-header PROXY_V1

SSL Proxy

For SSL we need to create a GCP SSL certificate from our certificate files. fullchain.pem is my publicly trusted x509 certificate for my domain mypocketxp.com, generated by the let’s encrypt certificate authority. privkey.pem is the corresponding private key.

gcloud compute ssl-certificates create my-ssl-cert --certificate fullchain.pem --private-key privkey.pem

We create an SSL policy that specifies the server-side support for SSL features for connections between clients and the SSL proxy Load Balancer. The MODERN profile supports a wide set of SSL features, allowing modern clients to negotiate SSL. For more details have a look at the gcloud documentation on SSL policies.

gcloud compute ssl-policies create my-ssl-policy --profile MODERN --min-tls-version 1.0

We create the SSL target proxy preserving the client IP through the proxy header, linking our certificate and applying our SSL policy

gcloud compute target-ssl-proxies create my-ssl-lb-target-proxy --backend-service my-tcp-lb --ssl-certificates my-ssl-cert --ssl-policy my-ssl-policy --proxy-header PROXY_V1

TCP and SSL Proxy

For SSL and TCP proxy we reserve our global anycast IP. For this example we are only allocating an IPv4 address

gcloud compute addresses create lb-static-ipv4 --ip-version=IPV4 --global

We look up the IP address that was reserved and configure the global forwarding rule

gcloud compute addresses list

TCP Proxy

gcloud compute forwarding-rules create my-tcp-lb-ipv4-forwarding-rule --global --target-tcp-proxy my-tcp-lb-target-proxy --address <reserved IP address> --ports 110

SSL Proxy

gcloud compute forwarding-rules create my-ssl-lb-ipv4-forwarding-rule --global --target-ssl-proxy my-ssl-lb-target-proxy --address <reserved IP address> --ports 110

TCP and SSL Proxy

For the SSL and TCP proxy we need to enable traffic from the proxy to the backend VMs. We create a firewall rule that whitelists the source IPs the GCP proxy uses for the proxy-to-vm connection well as for health checks.

gcloud compute firewall-rules create allow-lb-and-health --source-ranges 130.211.0.0/22,35.191.0.0/16 --target-tags tcp-lb --allow tcp:110

TCP Proxy

For a basic test of TCP forwarding we log into the back end server and install iperf, a widely used tool for network performance measurement and tuning, on our backend server

gcloud compute ssh my-backend --zone us-central1-b sudo yum install -y iperf

Then we start iperf in server mode on the backend server

sudo iperf -s -p 110

After about five seconds you will see entries like these showing connection attempts from the health check on port 110.

Now we can connect from the client i.e. our local system to check for connectivity to the backend through the proxy

iperf -c <reserved address> -p 110

and we see after about 5–10 seconds that a connection was established and data was exchanged for the performance test.

On the server side you will see a log entry like the one below indicating that a connection was made with a peer. We won’t see the source IP though as iperf ignores the proxy header in the TCP payload and only sees a connection from an IP within the IP range used by the proxy (35.191.0.0/16).

SSL Proxy

As iperf does not support SSL we need to test the SSL proxy differently. With the small python client code below we can send “Hello, world” to the backend server.

The code uses the Python SSL module that provides a TLS/SSL wrapper for socket objects. The server_name has to match the Common Name (CN) in the ‘Subject’ line or the ‘Subject Alternative Name’ (SAN) of your x509 certificate. You can check the content of your certificate with

openssl x509 -in <certificate file> -text -noout

As we want to validate the root CA we pass the file containing the root CA’s certificate to the SSL context with load_verify_locations().

To identify the x509 certificate file of your root CA you can run

openssl s_client -connect <reserved IP address>:110

with the IP address that we reserved and attached to the forwarding rule above. This will show you the certificate chain up to the root of trust. In my example I see a chain with one intermediate certificate: ‘DST ROOT CA X3’→’Let’s Encrypt Authority X3’→’mypocketxp.com’.

On my laptop (Ubuntu 18.04.5 LTS) I can find the certificate file DST_Root_CA_X3.pem for the root CA in /etc/ssl/certs. If you cannot find the root CA certificate file you can search sites like IdenTrust using the root CA’s CN, in our example ‘DST Root CA X3’, and download the file. Then you can install the certificate as outlined here. You can also find a good write up on certificate verification in this Medium article.

To test connectivity we run ncat as a ‘server’ on the backend. ncat that is installed with the nmap package

yum install -y nmap

ncat listens on port 110. The -k option ensures that ncat does not terminate after a connection is closed

sudo ncat -v -l -k -p 110

When we run our client program on the local host we see the proxy header showing our client IP 209.52.88.4 as well as our message. The source IP 35.191.10.112 is an IP from the proxy range. The connections shown above and below our message are health check connections.

If you want ncat to bind to a privileged port without running it as root you can set the Linux capability cap_net_bind_service for ncat