PORK: A Technology Resilience Framework
By: Rocket Mortgage Technology
Tags: Technology, Developer Tools, Software Development, Software Engineering, Software Architecture

The story of technology today is complexity. Increasingly complicated software deployed in complex and unpredictable environments that seem to be slipping more and more out of our control. Daunting stuff, this complexity.

A Prediction: The code you’re writing today won’t be the same code running a year from now. The machines hosting your services today won’t be the same machines in a year. Your services will fail occasionally. Your business partners will change their minds about features. Hardware will need security patches to address vulnerabilities. Network blips will disrupt and change the topology of your system. Your end users will interact with your applications in unexpected ways. Your business will scale significantly (hallelujah!), but load will be uneven. Intraday releases will mean services temporarily restart at inopportune times

More examples of complexity, you ask? No problem! Distributed systems, consensus algorithms, machine learning. Team collaborations, human emotions and miscommunications. Feature enhancements, bug fixes, new competition, vendor integrations and organizational restructuring. All this, and we’re just scratching the surface of the complex factors that could impact our technology systems.

Engineers are working within a truly unpredictable and complex environment from a technology, business and human perspective. We should acknowledge this head-on by building systems to be responsive to an uncertain future.

Hungry For Resilient Software

Given the new reality of complexity and constant change, our approach to building technology systems must adapt. At Rocket Mortgage Technology, we’re striving to transform the mortgage industry as well as the technology that runs it. In my role as a Software Architect, I work with a suite of amazing teams that build custom software solutions for our Capital Markets and Treasury business partners. These business areas are critical to the success of our organization, so we need our systems to reflect that criticality. Due to the volatility of the financial markets and the speed with which the mortgage industry changes, this is an exciting and ever-present challenge. Our mission in architecting systems, therefore, is to build systems that are both resilient and flexible enough to turn on a dime.

Our mission in architecting systems, therefore, is to build systems that are both resilient and flexible enough to turn on a dime.

What Is Resilient Software?

Let’s define resiliency as the measure of how a system withstands stress and responds to unforeseen errors and urgent feature requests. More broadly, resiliency is a measure of how a system embraces change. As the needs of our clients change, our systems must change accordingly.

This article details my teams’ approach to building resilient systems through our PORK Resiliency Framework. More than 100 billion dollars of loan originations flow through our systems each year, so the stakes are high. Our framework is constructed of four principles designed to take the fear out of decision-making in this complex and business critical environment. For several years now, these four principles have been the cornerstone of the systems we build and the measuring stick to guide our technology decisions. Engineers, architects, product owners and leadership have rallied around a shared goal to deliver resilient software, and these principles have led the way.

The Four Principles Of PORK

Our teams strive for fast, frequent and confident releases of our software, but how do we balance the safety of our system with the rapid change that comes with frequent releases? Well, imagine if you could say to your engineers, “Don’t be afraid of making mistakes because we have the tools in place to quickly find and easily correct them!”

This is the essence of our framework. We strive for resiliency by observing errors as early as possible, then recovering quickly (ideally before impacting our clients in the first place). With this approach, we’re giving our engineers the runway to experiment, innovate and make mistakes. At the same time, we’re relieving the pressure from our engineering teams as they no longer need to aim to be perfect, bug free or predict the future. To liberate our team members from the fear of mistakes is to empower them to move at a higher velocity and deliver business value faster.

So, what are the four principles of the PORK resiliency framework?

Predictability

Observability

Recoverability

Keep it simple

Predictability

Just because we’re working in an unpredictable and complex environment doesn’t mean our code should be unpredictable, too. Our first core principle asserts that we value predictability over correctness. This may appear counter-intuitive at first glance, but an engineering team should design a system to behave the same way in all circumstances. If a system is wrong but wrong in the same way every time, then we’re well-positioned to quickly diagnose and fix the problem. Predictability also enables the team to better explain, support and evolve the system over time.

We believe that the same inputs fed into a system in the same order should always result in the same output. Why is this deterministic behavior important? Replicating a bug is often the hardest part of an engineer’s job. We have great talent on our engineering teams, and they’re adept at writing code to fix a bug — that’s typically the easy part. Most of the battle can be the task of reproducing the problem in the first place. With a focus on predictability, we can take a notoriously tricky task and make it trivial. If something goes wrong in a predictable system, it’s simple to pass the same inputs through our code again and replicate the behavior so we can diagnose what went wrong. From there, we can leverage our automated toolchain to fix it.

This approach may sound simple, but there are a lot of forces at play here. To make your system predictable, you’ll need to delve deep into the underpinnings of your system design, likely pursuing advanced techniques, such as immutability, retry policies, idempotency, event-driven programming and other techniques.

Challenge yourself and your teams to make engineering choices that reinforce predictability throughout your technology stack, from code to infrastructure and the space between. How do our teams achieve this? We invest in several tools and patterns, which I’ll discuss later in the article.

Observability

Broadly speaking, observability means we’re designing systems with visibility and diagnostic tooling in mind. Lack of a cohesive and reliable observability strategy complicates how we triage issues, impacts system stability and ultimately erodes trust between a technology team and its clients.

If we’re confident we can identify when our system is behaving normally (healthy), then we should also be able to identify when something is behaving out of the norm (unhealthy). Upon observing unhealthy behavior, we can alert our support teams proactively and correct it before it impacts our clients. By contrast, if we don’t have the appropriate visibility into what our system is doing, then we don’t know when something is misbehaving, and we don’t know when it warrants our attention to fix it. This almost certainly leads to unfortunate client impact, and what’s worse, our beloved clients are the ones notifying us that the system is broken. Shouldn’t we already know?

Observability is about quick and effective root cause analysis, not a few flashy visualizations to impress your leadership team. Rather than getting distracted by polished graphs and charts, I recommend you focus on a chain of diagnostic tools. We want three types of tools in this toolchain:

Proactive alerts that call out unhealthy trends

Tools to form a hypothesis of what has gone wrong (e.g. dashboards for logs, metrics, traces)

Tools to confirm or deny the hypothesis

Once you have confirmed your hypothesis, you can leverage our principle of Predictability to replicate the problem and test your bug fix. Voila!

If we’re aware we have a problem (Observability), and we can quickly replicate and diagnose the problem (Predictability), then we are in good shape to correct the problem quickly.

Recoverability

When building a system, engineers have a tendency to try to make it bulletproof against failure. This seems reasonable at first glance, but ultimately proves to be unreasonably harsh and restrictive. Engineers should be thoughtful and defensive in their coding practices, but if they’re not careful, bulletproofing can lead teams down a path of fear, long-lived testing environments, days or weeks of extensive testing and planned maintenance windows. This approach will undoubtedly manifest itself in brittle systems and slow software delivery.

As we shift our mindset toward embracing unpredictability, rather than avoiding failures at all costs, we should instead be thinking about how quickly we can respond to inevitable failures. You cannot avoid failure all together, but you can very much control how quickly a team responds to it. We need not bulletproof our systems if we can quickly observe system issues and repair them before our business partners or clients feel the impact.

Recoverability is particularly important when considering how teams manage the data flowing through their systems. Data, in many ways, is a system’s most precious resource and the backbone of our business. If your system encounters an outage, chances are you’ll lose some data and need to recover it. Have a recovery plan in place for this in advance! Have a plan to repopulate your data in caches, event streams and data stores. For example, can you work with the system of record to provide a full “state of the world” snapshot to repopulate all your data on-demand? You should have layers of protection in place and, ideally, you should practice how you use them. Don’t wait for a production outage to hatch a recovery plan. Do it now.

Don’t wait for a production outage to hatch a recovery plan. Do it now.

We leverage many additional techniques for recovery, as well, such as approaching system design with a disposable mindset. The disposable mindset encourages teams to avoid building precious, brittle systems. Instead, support teams can quickly tear down a misbehaving application and spin up a fresh new application in its place. During critical outages, we favor this approach for quick recovery so our clients can get back to work as quickly as possible. Later, once the urgency has diminished, our engineers will delve deep into the observability data we have collected to explain what led to the outage in the first place. A disposable mindset often requires stateless services, a careful plan around managing your data (as discussed above) as well as some modern tooling like Docker containers and container orchestration, which I’ll briefly discuss later.

My recommendation is to shoot for the moon and build a reset button that will quickly restore an unhealthy system to health in an automated fashion. This will certainly take a lot of work, but it’s a noble goal. Remember, due to the unpredictable nature of our complex environments, you can’t avoid failure, but with principles such as these, we learn to build resilient systems from unreliable parts.

Keep It Simple

Simple, elegant solutions are less brittle, less likely to break and easier to fix when they do break.

As we design and implement systems, we’re typically addressing multiple goals across multiple time horizons at once. These include the initial delivery of a new system, but also the ease of maintaining the system for the years to come and the ease with which our system can evolve over time to meet our clients’ evolving needs. Given the time-shifting, shape-shifting nature of these goals, unless we have a healthy respect for complexity, the software project is almost certain to fail. In my experience, the single biggest indicator in the success of a software project is simplicity. It’s also the single biggest factor in reasoning about complex systems when you’re urgently alerted to a production issue at 3:00 a.m. Simplicity allows for a shared mental model across your team. Can every engineer on your team draw a reasonably accurate diagram of how your system is architected? I hope so.

Our systems act as a record of all the hundreds and thousands of decisions we’ve made along the way. Build the simplest thing that will solve the problem. Be deliberate in your decision-making process and be able to justify your decisions. Because we must live with these decisions, maintain them, support them, debug them — it is in our collective best interest that we keep it simple.

Tools For The Job

As described, the principles of our PORK Resiliency Framework are conceptual in nature, thus not overtly opinionated with regards to programming languages, utilities or technology stacks.

That said, I’ll share some tooling and best practices that my teams leverage today to achieve the goals of our framework. We expect these tools will change over time, but we also expect the four core principles will continue to apply, nevertheless.

Infrastructure As Code (IAC)

We use Terraform, which helps provision predictable infrastructure to avoid hard to debug environmental drift issues, which arise from manual configuration across multiple environments.

Continuous Integration/Continuous Delivery (CI/CD)

No surprise here. Tooling in this space is widely available to introduce automation, which offers us predictable and immutable build artifacts (for example, Docker images) and predictable promotion across environments. We use CircleCI and custom tooling to achieve this across our systems.

Docker

In our world, Docker refers to the containers as well as the orchestration layer that manages those containers. We love Docker as it promotes a disposable mindset, which ensures all dependencies are documented in code, and prevents snowflake environments. If a Docker container dies, another will be spun up dynamically by the container orchestrator (Docker Swarm, Kubernetes, ECS) to replace it. Because containers are immutable, the new one is just as good as the prior. The orchestration platform also offers us readily available infrastructure (compute on demand), not to mention a variety of predictability-focused tooling such as software defined networking, declarative files to describe the desired state of your system (Docker Compose, Kubernetes manifests) and a number of other abstractions to help with engineering productivity and resilient system design.

F#

We use C# in many scenarios, and C# is used more broadly across our company, but we love F# in this specific arena. F# is a functional language that addresses many of our predictability concerns elegantly (through first-class constructs around immutability and side-effect-free processing). That said, a functional language is not mandatory. You can certainly achieve predictability through C#, Python or other languages, though it will require more deliberate decision making and the engineering discipline to stay the course when the predictability path is not the easiest way forward. A functional language makes sense to us because we can emphasize immutability, testability and other predictability concerns at the code level, just as we already do at the service level and the system level. This allows for a cohesive predictability strategy up and down our technology stack.

Kafka

We love Kafka because, like the other tools mentioned here, it makes good engineering practices simpler to implement. It’s easy to integrate and deprecate services, so your system can evolve as needed over time. We believe immutable data is easier to manage, and Kafka is a durable log of immutable events kept in strict order (per partition). Your teams don’t need to reinvent the wheel and write custom code to achieve resiliency through buffering, redundancy, scalability and fault tolerance. These and other useful patterns and principles come out of the box with Kafka.



Furthermore, Kafka allows us to process data as events happen in the real world, rather than wait for a user request. Thus, we’re processing earlier and can observe anomalies sooner. This flexibility allows us to fix issues even before the user invokes a request (thus, no client impact). When we do encounter a bug, Kafka allows us to rewind the consumer offset and replay the data inputs we received earlier. This functionality allows us to diagnose the problem easily and get back to health, as described in our predictability and recoverability principles.

Splunk

We use Splunk for several reasons across our organization, but in the context of PORK, Splunk helps us achieve our observability goals using logs. At a high level, Splunk helps us describe, explain and alert on the health of our systems. We use Splunk for building dashboards and alerts that show anomalies. We leverage the Splunk DSL to quickly drill down into request-level context to troubleshoot and diagnose production issues as they’re happening.

Grafana

Metrics help us describe and alert on the health of our system, but unlike request-level logs, it struggles to explain the state of our system because metrics intentionally strip away the context of individual requests. We love Grafana, nonetheless, but we focus it where it brings the most value — quickly and efficiently graphing trends that don’t require drilling down into request-level data. We find that Grafana is great at visualizing resource utilization (CPU, RAM, disk) and HTTP endpoint status codes.

Summary

With the PORK Resiliency framework, we’re attempting to address the increasing complexity of technology and our environment by turning the design of a system on its head, such that we no longer over-emphasize the initial delivery of a system. Instead, we ask ourselves: How will our system respond to an urgent business need down the road? How will we react when we have a bug? Do we have the right tooling in place to respond quickly and confidently? The answers to these questions will reveal how resilient a team’s systems truly are. Resiliency, after all, is a quality that applies not only to the systems but to the teams that run them.

How do you ensure resiliency in your systems? Let me know in the comments!