The Gofrendly magic trick, or how to rewrite a live backend without users noticing it
By: Erwan Lemonnier
Tags: Gofrendly, Pymacaron, Flask, Docker, Gke

About Gofrendly and justifying a mad journey of technical debt cleansing

This article describes the process of rewriting Gofrendly's backend from a monolithic architecture based on PHP and mysql running on single virtual machines, to a dockerized Flask application with no-SQL databases and autoscaling in a GKE kubernetes cluster. Without a single user noticing it (almost).

Gofrendly, a friendship app for women

Gofrendly is a Swedish friendship app for women. Its goal is to help you find friends and get you out of your couch and into a real life full of social interactions and new experiences.

Ever moved to a new town and found yourself totally alone at home one evening? Moved to a new country where you didn’t know anyone? Or went through a separation to realise you needed new friends? That’s when you use Gofrendly.

And why girls only? Because a mixed-gender friendship app would immediately turn into a dating app.

Why did Gofrendly need a new backend?

As Gofrendly reached its first hundred thousand users, the team took a couple of steps back and contemplated their tech stack: 2 mobile apps (ios and android) and a PHP backend with a mysql database, both running on heavily muscled instances in Google Cloud. All outsourced to 2 different companies, one in charge of mobile development, the other in charge of the backend API and its implementation. Would that scale up? How would this setup behave with ten times more users? A hundred times? How fast can new features be rolled out? How fast can we start a/b testing new features? Can we keep outsourcing and still move fast enough?

Some answers were clear: to scale up, Gofrendly needed a more modern stack, needed to own its APIs and to develop its backend in-house.

Mission Impossible

With about 10.000 daily active users, the backend rewrite needed to happen with little or no downtime, and without loosing a single user. Oh, and fast too: 2 months to rewrite the core API on a brand new tech stack. Anyone up for it?

Sounds mad? I loved it. That’s when I joined the team, as its first in-house developer.

Choosing a new tech stack

After long discussions, we agreed on the following tech stack for the new Gofrendly backend:

A python Flask app, running behind gunicorn and using swagger/OpenAPI to spawn its endpoints and data models. Because I had spent the last 4 years battle-testing that stack with my Pymacaron project.

Deployed as a Docker container

In a Google GKE Kubernetes cluster with multi-zone support and node and pod autoscaling for high-availability and fault tolerance

Using Google Firestore in Datastore mode as its no-SQL database, partly replicated to an Elasticsearch cluster for advanced querying. Switching from SQL to no-SQL, easy, right?

What matters and what does not

The legacy backend exposes a REST API, documented via swagger, so we have a relatively clear description of what requests and responses look like. The new backend’s first requirement is to implement that exact same API.

One important strategic decision we took early on was to NOT do a one-time flip-the-switch shut-your-eyes-and-pray apocalyptic rollout. Instead, we would rollout the new backend progressively and as often as possible. Why? To minimise deployment risk and increase failure traceability at every release.

Another vital decision was to not let the backend rewrite stand in the way of developing new features. In a startup like Gofrendly, where the product is the main motor of growth, we just cannot afford delaying the rollout of new features.

And to have any chance at succeeding in this endeavour, we needed a reliable and well covering set of tests validating the API.

The master plan: little proxy becomes big boss

So how can this be achieved in practice? In short: by hijacking all traffic between mobile apps and backend. In fact, the new backend started out as just a proxy, which grew more and more self-sufficient until effectively handling all traffic by itself.

The Gofrendly legacy backend can be divided into 5 main logic areas: authentication, user profile, user matching, user activities and live chat. The idea was to re-implement each in turn into the proxy, and somehow let live users seamlessly use a mix of locally re-implemented logic and older logic handled by the legacy backend. That meant having a split-brain situation where some user data (the user’s profile for example) was hosted in the new no-SQL database while some (ex: the user’s activities) was hosted in the legacy backend’s mysql database. Exactly how we performed this progressive rollout while maintaining data consistency is hard to explain without going into very verbose and potentially staggeringly boring implementation details. So we simply won’t dig into it.

Here is what the roadmap looked like:

Step 1: a proxy with API tests

Write a proxy that serves the same API as the legacy backend, but actually calls the legacy backend on each request it gets. Implement that proxy using the new tech stack: Flask, gunicorn, docker. Write automated tests targeting the backend API, one test suite for every endpoint, as well as test suites covering all major user stories. Those tests must pass against both the proxy and the legacy backend, hence validating that they implement the same API and follow the same business logic.

Write a proxy that serves the same API as the legacy backend, but actually calls the legacy backend on each request it gets. Implement that proxy using the new tech stack: Flask, gunicorn, docker. Write automated tests targeting the backend API, one test suite for every endpoint, as well as test suites covering all major user stories. Those tests must pass against both the proxy and the legacy backend, hence validating that they implement the same API and follow the same business logic. Step 2: GKE clusters

Setup two Google Kubernetes Engine clusters, a staging and a live one, both with the same setup. Write a deployment pipeline releasing the proxy on the staging, then the live cluster, using only kubectl and the gcloud command. Stresstest the proxy in the clusters to validate the GKE setup’s stability and autoscaling configuration. Make sure to have reasonable monitoring on the GKE clusters with Google Stackdriver Monitoring, and optionally a Prometheus/Grafana setup.

Setup two Google Kubernetes Engine clusters, a staging and a live one, both with the same setup. Write a deployment pipeline releasing the proxy on the staging, then the live cluster, using only kubectl and the gcloud command. Stresstest the proxy in the clusters to validate the GKE setup’s stability and autoscaling configuration. Make sure to have reasonable monitoring on the GKE clusters with Google Stackdriver Monitoring, and optionally a Prometheus/Grafana setup. Step 3: Swap the legacy backend with the proxy

Remember when I wrote ‘no flip-the-switch’ rollout? I was lying. At some point we need to redirect all live traffic from the legacy backend to the new proxy. To make it worse, this is an all-or-nothing situation: we cannot let a load balancer enforce a gradual split of the traffic between legacy backend and proxy, since user data is split between both on a per user basis. So at some point, preferably in the middle of the night when traffic is low, we need to edit the DNS and point ‘api.gofrendly.com’ to the live cluster’s public IP instead of the legacy server’s IP. And have all live traffic start flowing through the new proxy, which in itself is a crucial validation step for the whole endeavour, one that we want early on in the rewrite process.

Remember when I wrote ‘no flip-the-switch’ rollout? I was lying. At some point we need to redirect all live traffic from the legacy backend to the new proxy. To make it worse, this is an all-or-nothing situation: we cannot let a load balancer enforce a gradual split of the traffic between legacy backend and proxy, since user data is split between both on a per user basis. So at some point, preferably in the middle of the night when traffic is low, we need to edit the DNS and point ‘api.gofrendly.com’ to the live cluster’s public IP instead of the legacy server’s IP. And have all live traffic start flowing through the new proxy, which in itself is a crucial validation step for the whole endeavour, one that we want early on in the rewrite process. Step 4: Rewrite the authentication logic

The legacy backend uses session tokens, making it necessary to do a database lookup for every request requiring access control. The new implementation relies instead on JWT tokens to identify the caller and its access rights.

The legacy backend uses session tokens, making it necessary to do a database lookup for every request requiring access control. The new implementation relies instead on JWT tokens to identify the caller and its access rights. Step 5: Rewrite the user profile functionality

User registration and all endpoints that update the user’s personal data. To start with, only a few users will be using the new profile code, then more and more, as the code gets stable and complete. This is enforced by letting the proxy look at the user’s JWT token upon each API call and decide whether that particular user should use the new implementation or be proxied to the legacy one. When a user gets green-lighted to use the new implementation, its data gets imported from the legacy database upon the first API request from this user. Data migration therefore happens gradually, and can be restarted when needed, until stability is achieved and the code is ready to handle all users, at which point all the remaining data gets imported in a long batch run. To enforce data consistency, the new implementation keeps importing user data on each request for a while, until the data of all users is properly imported and no legacy code may alter the legacy data. This transition period impacts the endpoint’s response times and must therefore kept short, though in practice it can last for days.

User registration and all endpoints that update the user’s personal data. To start with, only a few users will be using the new profile code, then more and more, as the code gets stable and complete. This is enforced by letting the proxy look at the user’s JWT token upon each API call and decide whether that particular user should use the new implementation or be proxied to the legacy one. When a user gets green-lighted to use the new implementation, its data gets imported from the legacy database upon the first API request from this user. Data migration therefore happens gradually, and can be restarted when needed, until stability is achieved and the code is ready to handle all users, at which point all the remaining data gets imported in a long batch run. To enforce data consistency, the new implementation keeps importing user data on each request for a while, until the data of all users is properly imported and no legacy code may alter the legacy data. This transition period impacts the endpoint’s response times and must therefore kept short, though in practice it can last for days. Step 6: Rewrite the user matching logic

Gofrendly lets users find friends based on an extensive list of criteria, so we need a flexible search API to query user profiles. Each time a user edits her profile, the updated profile gets stored into Firestore and, asynchronously, into an Elasticsearch index. The matching logic must therefore be rewritten from SQL to Elasticsearch queries: a completely different approach!

Gofrendly lets users find friends based on an extensive list of criteria, so we need a flexible search API to query user profiles. Each time a user edits her profile, the updated profile gets stored into Firestore and, asynchronously, into an Elasticsearch index. The matching logic must therefore be rewritten from SQL to Elasticsearch queries: a completely different approach! Step 7: Rewrite the user activity logic

All the endpoints letting users create and update activities, join them, invite people to them and so on. Using the same progressive data migration strategy explained above

All the endpoints letting users create and update activities, join them, invite people to them and so on. Using the same progressive data migration strategy explained above Step 8: And the journey goes on…

The new backend is now in place and completely backward-compatible with the legacy backend. All traffic is flowing through and users didn’t notice a thing. Time to build for the future and start rolling out new features!

Reality check

Sounds like a reasonable roadmap? Well to some extent, it does, since it mostly worked. Reality, though, is always full of surprise, especially in a startup and under time-pressure: the roadmap of the actual real-life rollout ended up looking more like a mishmash of the clean list above, with many of its steps happening in parallel or out of order. In fact, we started implementing the next version of the backend API (step 8) already while building the initial proxy (step 1 and 2), before we even started rewriting the legacy endpoints (steps 5 to 7). When that happened, there really was no turning back…

It took about 1 month to get a dummy proxy in place, running on the new stack, and redirecting live traffic through it. That included building a deployment pipeline against GKE, committing all infrastructure to code and stress testing the cluster to figure out its optimal size.

It took another two months to rewrite the authentication layer and user profile functionality, together with API tests covering all affected endpoints and ensuring backward compatibility for all user scenarios in the app.

Then we had to look forward and start implementing a few new features, such as email registration, AI-based selfie verification and in-app video calls.

All in all, it took a year to completely shutdown the legacy backend, but by that time the mobile apps were running against a completely redesigned API, optimised for performance at scale, and boasting loads of new functionality.

And for the fun facts on the new backend, have a look here. Thanks for reading this far!