Using the Cynefin framework during IT major incident response
By: Sundeep Singh
Tags: Cynefin Framework, Incident Response, Complex Systems, It Service Management

The importance of our decisions

Being able to make quick and effective decisions is a key attribute for major incident leads — the ones responsible for seeing an incident through to its resolution. Uninformed and poorly timed decisions could result in delays to resolve incidents and adverse outcomes. Cynefin can be used to make sense of the place from where we make decisions, to improve our judgement and situational awareness.

Incidents with most critical business impact are where IT systems are relied upon, but don’t perform as expected for most, if not all users (e.g. unexpected outages, severe performance degradation or security beaches) – referred to as “Sev 1” category.

These types of incidents can carry a heightened sense of pressure for those involved. Initially when they occur we might find we are confused as to how to respond; this is known as the disorder domain. The simplest definition of the disorder domain is the state of not knowing which domain you are in. In this domain, we are lost, lack situational awareness and apply little thought, so tend to rely on instinct and might take action based on past thinking, behaviours and previous decisions. We revert to using techniques we are comfortable with which can create a false sense of confidence resulting in ineffective decision making. For example all of our IT systems suddenly become inaccessible for users. Knowing if we are under a sustained denial of service attack or if there’s a power outage on the internal network, is the difference between being in disorder or chaos.

Without an awareness of the domain we are in, we risk oversimplification or overcomplicating a situation — an incident lead who has had success previously comfortably leading through chaos, may unawarely perceive a situation that would be better responded by use of good or best practice, as a crisis.

The primary goal in disorder, is to gather basic information so we can quickly transfer to another domain.

Making sense of chaos

Making sense of my behaviour and how I perceive a situation as major incident lead, helps me make decisions with greater confidence. Typically, once someone takes lead of a major incident with a brief understanding of the situation, they will be in the chaos domain. It’s not possible to know the cause and effect of a situation in this domain otherwise known as the place of “unknown-unknowables”.

Cynefin suggests the best way to approach a chaotic situation is to “act-sense-respond”. Act to establish order, sense where there is stability, and respond to transition a chaotic into situation into a complex one. The incident leads primary focus is on triaging and getting a feel of the situation, to stop the bleeding and establish some order and control. Providing reliable information to those that need to know is crucial in chaotic situations. Direct communication through a single channel to avoid miscommunication or rumour, is what’s needed. Usually this isn’t the time to ask for input, but to inform who needs to know, there is a problem.

Panic and uncertainty is common in this domain; as incident lead its key to act quickly and decisively to address what’s most pressing and navigate through chaos. For example where security incidents occur with potential data compromise across multiple systems, doing a quick risk analysis, check recent changes, temporarily shutting down access to systems, communicating what’s happening to a closed group, followed by crisis or continuity plans — may be a way to immediately stop the bleeding. The faster we transition from the chaotic to complex domain, the better chance we have at understanding the cause-effect relationship, limiting adverse business impact. During chaos we will require creative thinking and it’s in this domain that we learn and develop “novel” (new and not already known) ideas and practices. For example we may come across a security incident on our systems presenting a new type of attack vector that was unfamiliar to us previously, learning how to respond.