by jacquesm | 2025-11-26 20:48:42+00:00
> I was not making a statement that I am ignorant.

ok.

> I was saying I believe the proposal to model general software engineering after avionics is misguided and inviting you to clarify your position.

But we are not talking about 'general software engineering', we are talking about Cloudflare specifically and that makes a massive difference.

> It is certainly valid to ask what CloudFlare or anyone else for that matter could learn from avionics engineering or from NASA or from civil engineering focused on large scale projects or anywhere else that good engineering practices might come from. However, there is a persistent undercurrent in discussions around software reliability and general software engineering that ignore the fact that there are major trade-offs made for different engineering efforts.

I think we are all aware of those trade offs. We are focusing on a specific outage here that cost an absolute fortune and that used some very specific technical constructs and we are wondering if there would have been better alternatives either by using different constructs or by using different engineering principles.

> “Oh, look how reliable avionics are. We should just copy that.”

> Cool, except I would bet avionics cost 100 times as much to build per line of code as anything CloudFlare has ever shipped.

And there is a pretty good chance that had they done that that they would have come out ahead.

> The design constraints are just fundamentally different.

Yes, but not quite that different that lessons learned can not be transported. The main reason why aviation is different is because it is a regulated industry and - at least in the past - regulators have teeth, and without their stamp of approval you are simply not taking off with passengers on board.

> Avionics are built for a specific purpose in an effectively unchanging environment.

That is very much not the case. The environment aircraft are subject to are - and increasingly so due to climate change - dynamic to a point that would probably surprise you.

What is not changing is this: the price for unexpected outcomes in that industry is that at some point global air travel will no longer be seen as safe and that once that happens one of the engines behind our economies will start failing. In that sense the differences with Cloudflare are in fact not that large.

> If Cloudflare built their offerings in the same way, they would never ship new features, the quality of their request filtering would plummet as adversaries adjusted faster than CloudFlare could react, and realistically they would be overtaken by a competitor within a few years at most. They aren’t building avionics, so they shouldn’t engineer as if they are. Their engineering practices should reflect the reality of the environment in which they are building a product.

I do not believe that you are correct here. They could, they can afford it and they have reached a scale at which the door is firmly closed against competitors, this is not a two bit start-up anymore.

> This is no different than people who ask, “Why don’t we build software the way we build bridges?” Because we’re not building bridges. Most bridges look exactly like some other bridge that was built 10 miles away. That’s nothing like building new software. That’s far more like deploying a new instance of existing software with slightly different config.

This too does not show deep insight into the kind of engineering that goes into any particular bridge. That they look the same to you is just the outside, the interface. But how a particular bridge is anchored and engineered can be a world of a difference from another bridge in a different soil situation, even if they look identical. The big trick is that they all look like simple constructs, but they're not.

> The software was poorly designed. No doubt it was implemented the spec. Does that change the fact that the sum total of the engineering yielded a deadly result? There is no papering over the fact that “building to avionics standards” led direct to the deaths of 346 people in this case.

That is not what happened and that is not what the outcome of the accident investigation led to conclude.

Boeing fucked up, not some software engineer taking a short-cut. This was a top down managed disaster with multiple attempts to cover up the root cause and a complete failure of regulatory oversight.