In the beginning, there was void *
By: Vincent Vanhoucke
Tags: API, Machine Learning, Software Engineering, Coding, Software Development

In the beginning, there was void *

A tale of evolving code and unintended consequences

In honor of the imminent release of ‘Software Engineering at Google’, which I highly recommend, I thought I’d relate a tale of how software evolution and feature creep can go wrong in ways which, while feeling great at every step of the journey, yield a net outcome a decade later that is a disaster with a surprisingly large blast radius. This is my modest parallel to Tony Hoare’s ‘billion dollar mistake’, or Dennis Richie’s ‘most expensive one-byte mistake’ and, coincidentally, it involves both pointers and zero-terminated data-structures.

It all begins innocently enough. The year is 2009. A colleague of mine has a large number of dot products that need to be sped up to improve the scalability of a large-scale optimization problem. At the time, floating-point computation isn’t a common bottleneck for our workloads, and we don’t have off-the-shelf solutions readily available: bringing in a BLAS library is overkill, and there are simpler and definitely more fun things to do for someone like myself, who loves to write gnarly math code and mess around with vectorized Intel instructions.

No sweat, I’ll hack together a quick dot-product implementation using SSE intrinsics, and some hand-tuned unrolling for good measure. There is a very clever way to leverage the terrifyingly beautiful Duff’s device and express unrolling without abusing the pre-processor, at the cost of an equally scandalous abuse of the C syntax. I love it. One wrinkle: SSE instructions like to operate on 16-byte aligned data. Our toolchain doesn’t always provide that guarantee at the time, and compiler hints are still somewhat compiler-specific. No problem: I’ll just allocate a bigger buffer and add a data pointer to the closest 16-byte boundary to enforce alignment.

But wait, if I’m going to put a restriction on how the data gets allocated, it is now very tempting to make sure the contract is enforced via proper typing. So let’s wrap that naked pointer into a new data structure: float * becomes something like aligned_scoped_array<float>. Hey, this opens up new possibilities: dealing with edge effects, when using SSE instructions, is always most of the actual code, and easily alleviated when doing linear algebra by zero-padding the data up to multiples of 16 bytes, which plays nicely with my data alignment tricks. Great: let’s enforce that in the data structure, since we now control it. Done, it’s all very very fast now. Moving on.

Wait. Did someone say linear algebra? Can we process those dot products in batches instead? By then we’re really into BLAS territory, but let’s not rewrite everything just for a basic matrix multiply: just add another dimension to the array, a function that performs the 2D multiplication, and we’re good to go. Aligning and padding the data also means we can step through it without any further pointer indirection, which makes the code short and elegant. Of course, anyone who has read any of the literature on BLAS, notably the seminal Goto paper on the topic, knows that short and elegant probably leaves a ton of performance on the table, because considerations around cache performance can get quite complicated. The right answer is likely not to try to roll out your own implementation, but … it’s fun and it feels like progress, right?

Hang on. Can you do the same for quantized representations? Oh, but yes! In fact, I wrote this nifty paper on how mixing unsigned and signed fixed-point representations works amazingly well for neural networks. BLAS has nothing to offer in the space of fixed-point math, while all alignment and padding considerations still apply. So I’ll just add a specialization for int8*uint8 products, and we’re done. Magic.

By now, you’re probably beginning to see a pattern. Feature creep, slightly suboptimal decisions at every step of the way, the illusion of progress and decisions that are increasingly difficult to walk back from. Fast forward a few years down the road, and my one-off wrapper for a naked pointer has become a fully-fledged, tentacular matrix library, pedantically called FastMatrix. It supports x86, ARM and CUDA. It handles float, double, int8, uint8, int16 and int32. It compiles for Android and iOS. It has moved to a general ‘util’ directory in our shared codebase. It is being used by deep learning codebase as well as a growing number of projects, most of which I have never heard of. It is also sitting in a respectable position on the list of biggest consumers of CPU cycles on our Google-wide profiling dashboard. What’s not to love?

Well. It also has a terrible, maddeningly inconsistent API, in part because different hardware vendors have optimized for different data types and layouts, and I failed to abstract away those implementation details. It is also stuck in an evolutionary dead-end: forcing control of the data layout just to save a few CPU cycles and lines of code turns out to be a terrible idea, because much of the benefit disappears in all the data movements that it typically forces the users to go through. Its mere existence impedes progress towards better-supported, more comprehensive, BLAS-aware solutions such as the awesome Eigen library, which has gained support for fixed-point operations in the meantime. Its CUDA port is full of antipatterns, which I cringe at seeing cut-and-pasted elsewhere in the codebase. We also made a misguided attempt at supporting dynamic dispatch of some of the algorithms across instruction sets, which, if the caller isn’t careful, may cause leakage of compilation units for the wrong platform into one’s binary, and cause obscure crashes that are near impossible to tie back to the original root cause because the binary doesn’t even have to invoke the offending code to be tainted. Oops. It’s all fun and games until ads SRE come a-knocking.

Did I ever mention Google that rejected me when I first applied as a software engineer? They may have had a point. That library needed to die.

But by then, all these major design flaws had also paid short-term dividends: it is pretty fast. It’s there. Due to the somewhat loose, C-like API, it’s easy to add features to it, or copy-paste it piecemeal. Removing some of the most egregious maintenance nightmares would likely either cost thousands of CPU cores, or hundreds of software engineer hours: performance-sensitive code is extremely difficult to reason about, because while it’s safe to assume it has decent coverage for correctness, performance coverage is very difficult to achieve: a 10% speed regression could cost a mere microsecond or overload a datacenter, and you may not be able to tell the difference until someone deploys a binary with your changes a few months after the change is committed. By then, other repositories that I had no visibility into had started to rely on it, including its gnarliest implementation details, and I’d reached pandemic-levels of infection.

The year is now 2020. It’s been a decade. The monster is not entirely dead, though I’ve hacked a great number of its heads. Its demise recently got a significant boost from a die-hard fan of the library, who has fortuitously decided to leave the company. Don’t tell them: I gleefully deleted all their code the very same day. In my 12 years here, I have proudly removed more than 1.5M lines of code from our repository, yet these tendrils stubbornly remain. In retrospect, the best decision I made in recent years was to insist that TensorFlow not depend on any of it.

Donald Knuth famously said that no code is ever fully debugged until its last user has died. Obviously, Knuth doesn’t work at Google, or he might have said ‘… its last user and all of their colleagues …’. Even if the code is dead, some of its antipatterns survive, either in people’s heads or via good old-fashioned copypasta. Titus Winters, co-author of the aforementioned book, loves to say that ‘Software Engineering is programming integrated over time.’ One issue is that the time constants involved can be measured in decades, and rarely does someone stay in any job long enough to see their mistakes unfold. Decisions made locally, incrementally, can quickly poison an entire codebase and become terrifyingly difficult to walk back from, particularly as people move on to other, shinier problems.

Coda

When I started working on the DistBelief codebase a number of years later, I immediately noticed something surprising: the innermost structure that held the neural networks was a naked ‘void *’, nothing more. I looked up who wrote that heresy, and, of course, that turned out to be Jeff Dean. Oh. I timidly asked: ‘Jeff … but whyyy???’ His answer was simply: ‘well, we don’t know what we’re doing yet, so let’s not bake any assumptions into the data structures that we can’t walk back from.’ Sure enough, the right data layout to use in this context was extremely non-obvious: you can use one to optimize for forward inference on CPU, but it penalizes the speed of the backward pass during training, and the backward pass is roughly 2X more expensive than the forward pass. But in production you mostly just run inference, so that may be the right tradeoff? It depends, because on GPU it’s almost exactly the other way around. These tradeoffs took some time to settle, and eventually we codified a type system, a layout and removed the naked pointer. The lesson? Sometimes it is better to let Hyrum’s law (another noted co-author of the upcoming book) work in your favor and defer API contracts until you understand how your users will (ab)use them, instead of committing to an API early and becoming its victim.