GasBuddy’s Technical Transformation
By: Max Metral
Tags: Postgres, Scaling, Gasbuddy, Cloud Computing, Kubernetes

GasBuddy’s Technical Transformation

A morning minute of payments at GasBuddy

A few years back I wrote a story about transforming GasBuddy’s technology stack. We’ve implemented a number of new services, business lines and tools as well as migrating a host of others from our legacy stack. In that time we’ve learned a great deal, validated some decisions, rued others and open sourced a bunch of it. This article provides an update on how our decisions turned out, what tools we developed to stitch this all together, and perhaps tells you some pitfalls to avoid. There is another version of this article where I would say what we would do if we were starting from scratch, and it would resemble this article but certainly not mirror it. There’s also another version of it where we go all-in on AWS-native services and worry about costs later, but given that GasBuddy starts off at the top tier of most traffic based pricing models, that’s a much different equation that a company starting from scratch where time to market is so much more valuable.

All of the technology decisions we made and continue to make are about improving iteration efficiency — which is phrased intentionally to emphasize that speed does not equal efficiency even though they correlate. The strong service boundaries with an ability to deploy canaries, coupled with detailed logging and tracing makes it less important that we have exhaustive tests (IMHO), but it doesn’t mean we just throw some code out and watch the error counters.

I’m not going to give node.js or AWS their own sections. Node has been great for us, but I get that it’s partly preference. The JS package system is just too deep, and we’re not doing a ton of ML which would tip the scale to python (so far). Our adoption of React was both encouraged by and supporting of our decision to go with Javascript. 99.9% of the Javascript written at GasBuddy is ES6+ using the same babel preset and AirBnB inspired lint settings. AWS is expensive compared to other clouds, AWS does everything, you are not the first person to do that on AWS, and nobody got fired for choosing AWS.

Microservices

Serverless is all the rage these days, and for many good reasons (both the specific framework and the general technology). Personally, I’m not yet sold in the generic use case. We get too much value out of the persistent local data and startup/teardown infrastructure, and Kubernetes makes almost all of the developer pain go away (it doesn’t make the ops pain go away and serverless has many attractive characteristics on this front). We made a decision early on to have a specific service architecture — front tier APIs/web projects, back tier services and then persistence layers (Elastic/Postgres) which are typically accessed by only one service. Our deployable units are all named something-api, something-web or something-serv. I think this architecture has served us very well in terms of making things composable, secure, traceable and etc. I can agree with the proposal that all our -apis and -webs should be serverless. The persistence layers are moving towards serverless support (e.g. stateless postgres calls), but I still see value in the “server” for -servs. Additionally, the vendor lock-in required to do more than basic serverless right now is high — it gets lower every day and by the time I finish this article maybe the argument doesn’t hold, but the summary is that I continue to think microservices are a great choice for GasBuddy, even though if I were to do it again today we would have a lot more serverless components.

The “micro” part of our problem decomposition has been key, in that it compartmentalizes bad decisions. So long as the shared components have a high standard, the technical debt in the implementation of a single microservice is limited because we can just rewrite the whole damn thing. This allows teams to learn more linearly and for us to tailor the teams working on a project to the match between skills, project lifetime, demand curve and use of shared resources (like postgres). We do have several cases where an api just proxies to a serv, and that feels wasteful and something we’d like to come up with a better answer for. The main symptom of this disease is Swagger specifications that reference or copy other specs. Both reference and copy have horrific versioning problems. Some sort of “enterprise object specification repo” would help, as could some sort of GraphQL coordinated effort, but the learning and adoption curve on those sorts of things were beyond my appetite given the changes we already had in front of us.

One downside of all these small composable components is that a great deal of the logic in our overall flows is embedded in the connections and data flows, and the timing and ownership of that logic is not easily discoverable. We strive to make our staging environment provide a testable framework for the full process, but this gets very difficult with external partners. Partners like Dwolla do an amazing job of covering all the edge cases in their test environments and documentation, but that is clearly the exception and not the rule.

Kong

I made a joke in the first article about how we might dislike Kong if it switched from Lua to Java. Funnily enough, part of the reason we dislike Kong is Lua. It’s not a language we have reason to use in our stack (some in fluent and nginx, but minimal), and Kong was not baked enough to ignore the language choice. In the end, I’d argue the main problem with Kong was the fact that its oAuth credential implementation is not especially well suited to non-third-party developer use cases. We used Kong oAuth for our mobile app, web login, and everything else. This results in multiple database rows per login. There was, in the version we use, a half baked token reclamation process, but it brought Postgres to its knees and as a result we have had times with hundreds of millions of rows in multiple tables that are essentially wasted space and effort.

So, we have moved to Ambassador as a front-end API gateway (the move isn’t all the way done yet, but we’ve managed to build it in such a way that Kong tokens will continue to work and we can slow-roll off and then eventually shut down Kong and kill the database). In addition to being Kubernetes-native and built on a rock solid Envoy foundation, Ambassador does authentication in such a way that we can build our own token scheme, for which we use cryptography instead of remote persistence — meaning that each login does NOT generate a database row necessarily and that tokens can be centrally verified with a mathematical operation and not a storage access operation. The Kong team are a good bunch, and I’m sure there are use cases where they are a better match for people than Ambassador, but we are not one of them. The Ambassador team has been attentive and busy, and also gives us other very interesting tools like Telepresence that we are starting to use.

Ambassador, like Kubernetes, is mostly a configuration management tool — it leaves the application-specific lifting to you or to the underlying tool (k8s and envoy). Because of that approach, we don’t worry so much about how battle tested it is — the things it does are so easy to test in other environments before doing them in production. The underlying Envoy proxy has a very active community, and Ambassador composes many features into high level capabilities like rate limiting, canary deployments and external auth.

I’ve seen mail flying by recently about Kong and Serverless, and that certainly does change the job of an API gateway, so that may change the calculation for other projects.

Kubernetes

When we made this decision, it was anything but clear. The last thing we wanted was a low level cluster infrastructure that’s unstable and has few users. Now, much like the generic concept of containers, I think Kubernetes is an obvious choice. For me, the key capability is that I don’t think about the deployment process. We can deploy anything anytime anywhere and I don’t worry that the mechanics of rollout will cause a problem or drop a single request. That confidence also took a number of modifications in our application infrastructure, such as node.js socket draining, robust logging and attention to cleanup code so that the service doesn’t just get killed. But Kubernetes does a great job of representing configuration to make a deployment flow that works for our team and workloads.

Kubernetes hasn’t been all roses though. Our initial cluster was handcrafted with loving care, because Kops was barely real and made some decisions we didn’t love at the time. Kops has come a long way, mainly around cluster upgrades and expanding choices of underlying infrastructure components. We have three clusters (production, payments, and staging) and two of them are now using Kops. However, the one with stateful services (RabbitMQ, etcd) is still on a very old version of Kubernetes. There have been several upgrades to Kubernetes that require meaningful planning and risk downtime that we just didn’t want to invest the resources to mitigate. We kept telling ourselves we’d just move to Kops and get two birds with one stone, yet here we are a year later barely closer to upgrading that cluster. That is mostly our failing, for sure, but kubernetes didn’t help with so many breaking changes.

We also lack a real job scheduling system. Kubernetes cron jobs are a joke when compared to Jenkins or similar (for example, if the job fails a number of times, it will just stop running that job, even when the next schedule comes around). Jenkins is anything but Kubernetes-native, and requires way too much management to securely deploy in a cluster. There is a desperate need for proper job scheduling in Kubernetes.

Postgres

When I started, MongoDB was still the rage even though the shine was coming off. I knew the parts of the system I was most concerned about were going to need to be fully ACID compliant, so I figured Postgres was at least a part of the solution. As we built more components, we were constantly surprised at how flexible Postgres is. We have databases running v8 to transform complex data structures with transactional integrity and a single database roundtrip, we have partial indexes that dive into JSONB columns on tables with tens of millions of rows, and we also have traditional transactional payment systems that are perfectly idempotent and have read replicas and all that normal database stuff. I’m not sure I’ve ever thought of a data storage or access problem Postgres can’t solve, and usually in a way where we look at the developer next to us and say “check this crazy s**t out!”

We are currently still running Postgres RDS as opposed to Aurora Postgres for our production databases. We had bad experiences with auto-incrementing values when migrating from PG to Aurora/PG and never tried again for fear of what might happen. RDS performance in AWS isn’t great, and we made a mistake by trying to stuff all our logical databases on a single instance. Moving them would require downtime or significant investment in code changes or tooling to avoid the downtime, and so far it’s always been easier to click the “pay Amazon more for a bigger thing” button. There have been a few datasets where it would probably be more cost efficient to store in a document oriented store (like our “Drives” feature that allows you to log all your drives, which are often a frequent and large set of points and speeds and such).

Tooling could be better for Postgres — writing functions is not as easy as it was in SQL Server, between the complexities of source control and versioning, limitations on return types and number of return values, and etc. The performance monitoring tools also aren’t especially mature so on the occasion we have a burning fire from a bad query it takes longer than it should to find exactly which query is doing it and why the plan is the way it is. IRC has been an important “support channel” for us and there are some all-knowing oracles (hah!) like RhodiumToad that seem to be online all the time and willing to point in the right direction.

Elastic Search

There are search workloads that didn’t initially seem a good fit with postgres (I think we were wrong about several of them) — usually with massive infrequently accessed datasets (e.g.logs), or with text or complex geo search patterns for which we felt something like Elastic Search would be a better match. The truth is I haven’t actually had to use ES much because others on the team have worked on those projects, but I have been the victim of ES more times than I can count. Our ES cluster(s) are the most expensive component of our AWS infrastructure (or #2 at best). If you add in the time the ops and data teams have to spend managing it, it’s not even close (which I’ll discuss a bit more in Logging). I would rather have expected more from Postgres in cases like POI search and either used a managed ES cluster for high volume data or used another system like RedShift or S3. It has not been easy to deal with ES during development and testing either, especially when compared to postgres/redis/etcd. The query language is powerful but highly specialized, which makes it harder for developers to get going and easy for them to blow things up by making hard things look easy. In fairness, I think if we doubled down on ES we would be a lot happier about it, but sometimes you don’t want to invest that amount of your constrained resources on the tool instead of the product.

Logging

We had one thing right — a common logging framework with identifiers that allow full visibility into the path of a request was crucial. We baked it into our open source opinionated service module. Initially we used Winston but recently moved to pino for speed and to take advantage of container and cluster logging infrastructure rather than going directly to a data store. We’ve built a number of tools to monitor the logs in realtime and search for specific requests and such, though they are mostly half baked. We use kibana as well, which is three quarter baked — it doesn’t visualize the traffic well and search isn’t super well tuned for the use case (the other day a search didn’t work because I put quotes around a number by accident). BUT, the real headache has been Elastic Search because the volume of log data is gigantic and heterogenous. We’ve had various problems dropping logs because the data type ES saw in that field first was numeric, and some other service used the same field name for a string. Often, ES would just fall over and then we’re not only having to deal with that outage but we’re losing logs or backing up the log shippers until they bring a cluster to its knees.

We didn’t use Loggly/Datadog or similar because that volume was large from the get go and I didn’t want people worrying about the costs of logging and thus logging less (we currently run about 15TB/month and 41 billion events). In the end, we’ve spent so much time and money on logging infrastructure that I think we may have been better off using a service at the beginning — mostly to avoid the workload but also for the improved analysis and discovery tooling. Conversely, at this point we’ve got it working well enough that it would be a big expense to move it and the steady state costs would be massive. Based on published rates, external providers would be something between $25,000 and $50,000 per month. Our logging infrastructure used to be the most computationally expensive bit of our system, and Jean-Charles did a ton of work moving away from java-based logstash to a combination of fluentbit (C) and fluentd (Ruby); and moving from network delivery (via winston and logstash) to stdout delivery (via pino and k8s/docker scraping) and other trickery to reduce logging load by an order of magnitude or more.

Wercker

Our entire build process for our “new stack” — CI/CD/container deployment — is run by Wercker. This has been true since the beginning — It’s probably done 100,000 builds for us. The main reason we chose it was the ability to use a docker-compose like syntax to bundle services like postgres into a CI pipeline. It’s been great for us — it rarely goes down and if and when it does there’s usually someone around to tell us what’s up. It has been the highest leverage tool we have because we have not paid a penny for it. The problem is that it only runs 2 jobs at a time on the free tier, and the next higher tier is thousands of dollars a month. The pricing is just completely and totally broken, and Oracle (which purchased Wercker) seems to be trying to convince people to use their cloud by teasing Wercker integration. The chance of us leaving AWS for Wercker is -100%. I would hate to have to manage our own Concourse or equivalent, but that’s the next best option. We would pay a couple hundred bucks a month without question for a dedicated pipeline in wercker. But they won’t take our money, because Oracle is Oraclin’. Next they are going to propose a billing model that has to do with how many lines of code are in my project.

React

The web properties were the last “major technology category” we modernized at GasBuddy — i.e. we spent most of our time on persistence layers, back end services and APIs. We chose React for a bunch of reasons that I would just be parroting from the dozens of articles elsewhere on the net. However, there are still dozens of decisions inside React that are much more nuanced — to redux or not to redux? How should bundles be shared across independent projects? How do you package react components to minimize repetition in both work and bundle size? What UI frameworks do we use or do we build our own? Which CSS flavor (modules, JSS, PostCSS, Less) should we use? How do we test (Jest, tap, enzyme)? Should we make storybooks for everything?

We are at a point now where we don’t all agree on these answers. I hate redux, because this isn’t Facebook. We don’t have hundreds of engineers and thousands of complex flows to share. I strongly prefer unstated, which reduces the number of files I have to touch whenever I want a button click handler to at most 2 instead of at least 4 (action enums, reducers, action creators, and the component). As we try and share more code in our mobile apps for non-high traffic and non-input heavy features, bundle size is crucial, and I even like preact if I can get away with it. We started out using semantic-ui which was convenient for our internal UIs, but way too big for mobile views and we’ve built our own components now (in most cases). My views have changed on these answers in the eighteen months since we started Reacting, and our code reflects all of our learning curves and changing opinions. React itself is also changing rapidly— especially the idioms in common use. Hooks and suspense are going to make a huge impact on React code in the next year.

I think React is a great framework, and I use it for a variety of personal projects including a “coding dojo” for my after school coding classes. But, we have work ahead of us to agree on an approach at GasBuddy and as importantly, to modernize the components that don’t use that approach. I’m going to go out an a limb and say our internal support and admin tool is never going to get migrated.

RabbitMQ

Message queues are an under-appreciated part of large software systems. Guaranteed delivery, ordering, idempotency, scalable workers and other patterns are hard to get right with databases or pub/sub tools, especially when it comes to domains like payments (for those not familiar with GasBuddy, we have processed hundreds of millions of dollars of fuel purchases and given away millions of dollars of free gas, as well as being a crowdsourced gas price resource). AWS has SQS and other cloud providers have similar offerings, but as of a couple years ago, they had limitations (in addition to vendor lock in) that made them unusable for our use case. Enter the wonderfully configurable and well-tested RabbitMQ. It can do single delivery, it can do topic routing, it can be run in clusters, it can slice bread… But it’s also quite old, and was not written with modern infrastructure in mind(e.g. containers/Kubernetes). Network partitions happen more frequently than we would like, and it is not easy to build a RabbitMQ cluster that handles this well (it’s gotten a bit better over the past year with cluster formation and automatic recovery). As a result, we’ve coded on the assumption that RabbitMQ is good at its job (it is) but never the system of record. All our operations assume that they may have run already, or that they may be running concurrently in some other queue processing worker (we contributed to the etcd-based microlock project mainly to provide control over that concurrency). So perhaps SQS or similar would have been fine given that freedom. If I was starting from scratch today, I would still use RabbitMQ, but I would be a lot less excited about all the problems it was going to make go away.

Spaces, Not Tabs

I’m just kidding (everyone knows it’s spaces) but on a related note, we made a mistake having a public API convention of snake cased identifiers (account_id) but lint rules requiring camelcase (accountId). In general, we went with airbnb’s style guide, which is fine so far is it goes, so we probably should’ve echoed that convention on the API side. There has been some traditional hand wringing about linters and what a reasonable line length should be, and etc. but I think that’s more human nature than substantive issue. We diverged from airbnb in a few places, and published our configurations as a public module/repo called eslint-config-gasbuddy.

The Team

The GasBuddy engineering team has generally been welcoming and supportive of these changes and I think we all view it as a learning opportunity that will payoff beyond our time here. Given that it was primarily a .Net shop before, it’s a fairly large culture shock to go from the best typed language ever invented (C#, fight!) to the most productive but least safe (ES6, fight!); from SQL Server to Postgres; from colocation to AWS… the list goes on. It was not for everyone, and we lost some good engineers along the way. When you’re planning a transformation like this, I think you have to assume this will happen and put that into the risk equation. Like most companies, the jury is still out on GasBuddy, and we’re still just as hard at work building and rebuilding for scale and quality.

I didn’t touch on mobile much in this article because there hasn’t been nearly the amount of change in that code base (although there has been plenty). There are no shortage of problem areas for our mobile apps that have had material impact on the quality of the user experience and thus on our happiness with our own product. Most of those issues are not about code but about process and trying to grow a free app into a growing business. This can be jarring for consumers that have become used to the old GasBuddy. Those “long time GasBuddies” are of course crucial to our success and we need to be better for them as well as the newcomers. This is a strong focus area for us in 2019, and hopefully that will become clear in the next couple of months (for example, we’re finally going to do search-along-route, because duh).

GasBuddy Open Source Packages

We choose the MIT License by default, because Go Beavers! Almost all of our open source modules are exclusively used by us. In other words, basically none of them became widely used or modified by others. That’s fine by us — we still hope it showed up in Google searches for people having problems or that it led us to contribute back to other open source efforts. Perhaps more importantly, it set the tone for our expectations of ourselves in terms of hacks and writing code you hope nobody ever looks at. What it did NOT do is improve our documentation habits. Modern code inherently needs less comments (IMHO), but that doesn’t mean it documents itself from a usage perspective. Had we had more external engagement on our open source efforts, we probably would’ve been forced to be better about that. The learning curve for our various platform decisions was steep, and it would’ve been better with a more cohesive documentation strategy, but that would have reduced iteration speed and undoubtedly led to documentation becoming out of date and misleading without significant investment.

babel-preset-gasbuddy: A single babel@7 preset that is used to build our server code and React code. We add optional chaining, class properties, and CSS module handling to a @babel/preset-env base. It also supports “React modules” such that we still get tree shaking (removing unused code) from client bundles when using a shared node module with React code in it.

eslint-config-gasbuddy: a linter configuration based on AirBnB’s style guide with a few minor deviations around line length, loops and parameter reassignment. Most of the deviations were recognizing existing patterns that weren’t worth changing rather than intentional rejection of AirBnB’s decisions. Except for the 100 character line length limit, because it’s not 1995 anymore.

@gasbuddy/service: This express-based service infrastructure module runs beneath everything we have, even when it doesn’t expose an HTTP endpoint (like queue workers and batch jobs). It manages inter-service connections, persistent storage (ES, postgres, and more), logging and infrastructure interfacing (e.g. k8s). It is based loosely on PayPal’s Kraken, but adds a configuration based “shared object store” like Express’s middleware chain. We hang this off the express request, via things like req.gb.logger and req.gb.rabbitmq . These hydrated objects come to life via @gasbuddy/hydrate and we wrap a variety of infrastructure components in shims that allow them to be configured from JSON configuration files. These include configured-pino, configured-etcd-client, configured-swaggerize-express, configured-prometheus-client, configured-swagger-client, configured-elasticssearch-client, configured-rabbitmq-client, configured-redis-client, and configured-postgres-client.

Conclusion

I’ll let you know the conclusion when there is one. For now, we’re continuing to evaluate new and old technologies, and we are certainly at a point where the delta from existing decisions matters as much as the quality of the shiny new object. We need to continue to move more of the high frequency activities to the new stack — including system-of-record for credentials and user names, price validation and some other big parts of our solution. We still have a big SQL Server HA cluster in our colo facility, although the load on it has been significantly reduced since the early days. These systems have never risen to the top of the list of pain points, and given the risks of even moving them logically as-is, they will hang on for a long time. Hopefully my next article will be about how we finally closed down our cage.

And in case it wasn’t clear, the title references the fact that the previous article, like this one, had some decisions that were right and good advice (IMHO) for the reader — hopefully >66% — and some that were not. Happy hunting!