The Three Levels of Software Safety
By: Marianne Bellotti
Tags: Cybersecurity, Software Development, Safety, Distributed Systems, Technology

The more software eats the world, the more critical safety is … but what exactly does that mean?

Hacker image by catalyststuff

Software engineers are bad at safety because software engineers are not used to the idea that software can injure. All around the industry, the mantel of technical leadership has been passed to people about my age, perhaps a few years older. We grew up when computers weren’t so powerful, when their use was an optimization rather than a necessity, when their first commercial successes were in toys. We don’t think about safety as being a relevant issue for software, and we need to change our perspective on that.

But what does it mean for software to be safe? It’s easy to conceptualize how a car could be safe or unsafe. Easy to understand how a medical instrument could be safe of unsafe. But code?

I like to think of software safety as being about three levels of concerns. Understanding where what you are building fits on those three levels, will tell you how best to focus your time and attention in a safety conversation.

Level 1: Safety as a Synonym for Security

For years, the only “safety” software developers thought about was “memory safety.” People will still jump to that conclusion, treating safety as a synonym for security.

The connection between memory and safety isn’t just a conflation of terms. Safety is ultimately about preventing a system from reaching dangerous states. In software, the principle clearinghouse of state change is memory. So the first line of defense preventing a program from reaching a dangerous state is controlling what can access its memory and how that memory can be accessed.

The need to manage what parts of memory a process can access traces its roots back to mainframe timesharing. Resource management and memory safety started as a straight forward customer service issue (if you’re paying for time and someone else is running something that eats up all the memory, it’s a problem). Computer scientists theorized that these annoyances could be weaponized and they almost immediately were starting with the Morris worm in 1988 and continuing to this day.

Level 2: Safety as Predictability

But as software grew in complexity and sophistication, state stopped relying on memory so much. A program could be “stateless” by passing state without storing it from one system to another in a transaction. The more common these kinds of designs became the more concurrency bugs and pure functions became part of conversations with software engineers. Undesirable states could be triggered by transactions happening in the wrong order (race conditions), or by depending on other states which were also in flux.

Safety at this level of software becomes about predictability (or determinism). The ecosystem that has grown around it includes type systems and formal verification. These were not techniques that were invented the solve safety problems, but the renewed interest from academia and the penetration of such approaches in developer tooling is a direct result of increase awareness of how dangerous unpredictable systems can be.

Level 3: Safety as Ergonomics

The area of software safety I find the most interesting is also its newest frontier. It’s the unsafe states that can occur when computers become participants in other larger human based systems. Like the issues in level 2, the problems and techniques of level 3 are not new discoveries. Modern technologies — particularly AI — have made them relevant in ways they were not before.

Software has always been present in human systems, but there is a difference between being present in the system and being a participant in the system. When you use a database to keep track of inventory, software is present in the system. When you build software that automatically reorders items based on that database, the software is now a participant.

The key difference is whether software is recording state or altering state, because safety is ultimately about our likelihood of reaching undesirable states. As software automation becomes more and more prevalent, the issue of how software-initiated state changes can effect other human participants in the process is moving into the forefront of the conversation around technology.

Many describe this issue as software ethics, but for me it is a safety issue. Ethics assumes unsafe states are foreseeable. When a design decision causes an airplane to crash, we do not call ethics into question unless someone can demonstrate that the people who made that decision had information that suggested that outcome was a possibility. Otherwise it’s just an accident.

So far the tools we have to handle Level 3 are user centric design and problem setting processes. User centric design helps broaden our perspectives so we can see more potential outcomes on different potential interactions. Problem setting forces us to set boundaries with our technology. The same way memory safety restricts what states a program can access, problem setting asks software engineers to treat these powerful technologies as scalpels and restrict what parts of the process they can participate in.

Safety as a Scaling Problem

Safety, like risk, is ultimately about scale. At each level, problems that were always there take on new significance and the undesirable states that were always possible become more common. We shift our attention and develop new safety tools and techniques to accommodate these different priorities.

When a program runs on a single machine with a single user, the most relevant safety issues are how that program affects the state of the computer itself. Concurrency issues are still there; ergonomic issues are still there, but the impact either of those issues have is not as significant. If a single isolated computer has concurrency issues all the data needed to it sort out and determine the correct state is on that single computer. If a single user has a negative outcome from using the program it is unlikely to spread to other users unless the program is shared.

When a single computer grows to a network the first level of safety is still important, but the second level of safety becomes more dangerous. State moves from computer to computer and the blast radius of undesirable state changes increases.

When digital systems stop merely supporting processes but become load bearing participants of that process, the third level of safety moves to the front of the line. Again the blast radius of undesirable state changes increases, damaging not just machines but causing people to take actions that are misinformed and potentially injurious.

The first step in developing a safety practice is to determine what scale what you are building will reach in the immediate future and focus on building out best practices around that level of safety. Then as you scale the technology, continue to evolve your approach to reflect the concerns of higher levels.