Taking back control: conscious uncoupling from Cloudflare
Tags: Cloudflare, Technology, Ecommerce Web Development, Web Development

Taking back control: conscious uncoupling from Cloudflare

The challenge of working with third-party products without losing your autonomy

Let me begin by saying that we love our third-party products and we love Cloudflare. It’s a great platform and it really helps us to improve our website’s performance. Over time we have put quite some stuff in there, but this also meant we lost quite a bit of independence. What would happen if Cloudflare gets taken over by an evil investment company and prices suddenly soar? What if someone decides to pull the plug on the whole service for whatever reason possible? We’d be severely unhappy and maybe even a little screwed. So yes, we do like, and use, third party products, but we also really, really like our autonomy. Therefore, we decided to take another good look at our Cloudflare set-up and go on a quest to gain control.

Photo by Esteban Lopez on Unsplash

New toys

We work in an industry with constant innovation. This means that interesting new languages, frameworks and third-party products pop-up on a daily basis. Thankfully I work at a company where developers get space and time to investigate and try all these pretty new things, which makes work a lot more interesting. Sometimes these experiments are successful and grow larger and larger. Until problems start to arise. And I guess that is what happened with us and Cloudflare. We never sat down to decide what parts we would use, who would get access and who would be responsible. It just sort of naturally grew. And now we have arrived at a point where we need to prune it back a little bit. We already created a project to deal with Cloudflare workers and use an external DNS tool in our Kubernetes cluster to control DNS-records. Now it was time for the next step.

Page rules gone wild

One of our favorite features on Cloudflare is the page rule section. Over time we collected a fair amount of them, filled with nifty redirects and glorious caching mechanisms. Unfortunately, there is no ‘explain yourself’ field for these rules and we have now reached the point where half of them don’t really make sense anymore. The original creators have moved on and nobody knows for sure what will happen if we delete any of it. Do you know that sinking feeling when you discover you made a major boo-boo? I just recently had the pleasure of experiencing this when I created a faulty page rule. It turned out that I misread the existing page rules, causing me to cache things that most definitely should not be cached. So yeah, this seemed like a good place to continue our quest.

Photo by Mike Dorner on Unsplash

Setting it up

The goal of this project was to move the page rules to our own repository and clean them up while we were at it. By creating a dedicated project with a nice pipeline, we could ensure that the creating, editing and deleting of rules would be registered and could always be traced back. It’s also a plus that all changes will require a pull request and a second pair of eyes checking for good measure.

Another change we planned on making was adding a ‘description’ field to the rules. This should make it easier for us to use and maintain them and hopefully prevent aforementioned boo-boos. Now there was a possibility to use Terraform to manage Cloudflare configuration in code, but we decided to go for a more lightweight solution and take the opportunity to learn a new language while we were at it. Therefore, we created a shiny new Kotlin microservice and started coding away.

The good, the bad and the messy

Cloudflare has a pretty good, well documented API that served pretty much all our needs, so we started by setting up a client with Ktor to talk to the endpoints. Ktor is a nice framework for building asynchronous servers and clients in Kotlin.

The second step was putting our existing page rules in our own repository. We chose a JSON format, excluded fields that didn’t seem relevant and added the all-important ‘description’ field. Adding all these descriptions by hand proved to be time-consuming, but in the end we managed to find one for almost all rules (some mysteries still remain).

Next, we started to add some checks and made it possible to use our own application to add page rules to Cloudflare for all our domains. Which is right about the time when our ‘access right conundrum’ first came up.

Photo by Nick Fewings on Unsplash

Access denied

You see, to make the project work we would need to restrict access to the page rule section of Cloudflare, so that only one or two people would be left with editing rights. This would be the only way to ensure that people use our own repository and not edit anything directly in the Cloudflare dashboard. At the moment there are quite a lot of people who have access and, although we trust our co-workers dearly, we are aware of the ‘I’ll just quickly do it in the dashboard now and edit it in our repository later’-feeling, that often results in forgetting about it and messing things up. So ideally we would revoke access for most users.

99 problems

Unfortunately, a problem arose: it turned out to be impossible to revoke rights to only the page rule part of Cloudflare. It’s part of a bigger permissions package and would mean people would lose access to most other Cloudflare setting as well. Now that decision seemed to be a bit out of scope for our page rule project. Because who would be responsible for these sections then? Would we only give one or two people access rights? That would mean a lot more work for them, and longer waiting times for developers to get things done. Is that something we wanted? Or should we trust our developers and hope that they wouldn’t make any mistakes? A lot of questions that need an answer before we can really continue.

Photo by Daniel Cheung on Unsplash

The next steps

We didn’t want to address these issues in our page rule project, so we decided to build in some extra checks and functionality for the time being. This way we can use it safely without revoking any access. But for our Cloudflare independence quest it meant having to take a step back and looking at the bigger picture. And the bigger questions: what functions do we want to use? How do we want to use them? Who should have access? Who is responsible? Once we decide on that, we can continue with our Cloudflare independency quest and start looking at our other third-party integrations to see how much we rely on them.

Learning as we go

The questions that arose during this project aren’t just applicable to Cloudflare. It’s a bigger issue involving all our dependencies. It’s about the balance between giving people freedom to experiment and deciding on more rigid things, like responsibilities, security and dependency management. It also has to do with growing as a department. Where once there were only a handful of people and everybody knew what was going on, there now is multi-team environment where we might not even know the names of all the people we work with (this might just be me, I’m horrible when it comes to remembering names). But anyway, we learn as we go. And as we grow, I guess. We’ll keep experimenting with new toys, but we will also start to look more deeply into how we use them. And we will continue our quest for independence. To freedom!

PS Do you have any experience with these issues? Any ideas about the subject? Tips on how to remember names of all the people you work with? Please let me know in the comments. I’m curious to know how other people/companies deal with this.

PPS Thank you to Anna Shepeleva, maarten blokker, Martijn Zwennes and Adam Semenenko for the mental support.