Emergent behaviour and dashboarding in Instana
By: Robert Barron
Tags: Sre, IBM, Observability, API, DevOps

Introduction

They say that the game of rugby was invented in 1823 when a young boy, William Webb Ellis, studying at Rugby Public School, “with a fine disregard for the rules of football as played in his time, first took the ball in his arms and ran with it.” (This story is apocryphal and the rules of football were probably not well regarded, in any case. But why let mere facts get in the way of a good story with an excellent lesson?)

The game of basketball was certainly invented in 1891 by James Naismith, who wanted to exercise his students with a strenuous indoor game. He adapted a game from his youth called “Duck on a Rock” and the rest, as they say, is history.

These are examples of emergent behaviour or emergent game play.

Essentially, emergent behaviour in a game is when the game environment is flexible enough to allow players to choose their own directions and challenges beyond those which the designers of the game had in mind. The myriad varieties of card games, all using the same 52 cards, showcase just how flexible that environment is. On the other hand, a checkers board and checkers tokens don’t have the same flexibility, and there are far fewer varieties of checkers. Some types of computer games are explicitly called “sandbox games” where gamers have the flexibility and the tools to find creative ways to fulfill goals — no matter what the original designers may have thought was the best way to do something.

Emergent behaviour in operational tools is the equivalent of this sandbox, where sysadmins, operators, DevOps and Site Reliability Engineers can customize their tools to solve the problems that they have in front of them, not merely the problems the tools were originally designed to solve.

Emergent behaviour — some clever examples

In this article I’ll discuss how IBM Observability by Instana APM encourages emergent behaviour, specifically in the design, development and usage of dashboards.

One of my favourite examples of emergent behaviour comes from a feature called Maintenance Mode in IBM Cloud Pak® for Watson AIOps/Event Manager (previously Netcool Operations Insight/NOI). The designed purpose of this feature is to mark specific computers as “in maintenance”. This means that if a problem occurred during this period, then the system would consider it a legitimate, even expected, issue and therefore not escalate the problem for human attention or try to resolve it automatically. Since the computer was in maintenance, any problems which arose in this period were expected.

My client, a bank, used this mechanism in an ingenious way.

Part of its responsibilities was to track the opening of certain safes. So the bank hooked up the safes to emit a message whenever they were opened, and put the safes in maintenance mode during regular working hours. This meant that the error message “the safe has been opened” would be automatically quashed during working hours and an alarm would be raised out of working hours.

Were there other ways to solve this problem? Of course. But these engineers found it easier to repurpose a tool they were familiar with than to develop a new solution using other tools.

Instana infrastructure dashboard — You can build a diagram of your datacenter/cloud environment with the skillful use of tags and labels.

Another example is that of a public transport company which used IBM monitoring tools, not to track the performance of IT transactions, but rather the speed of escalators in underground stations. They wanted to track anomalies in escalator performance to be able to send maintenance workers to fix the problem.

My colleague Matthew Duggan has used the topology mapping component of Watson AIOps (originally designed to map the relationships between IT components) to map out London Transport and the relationships between Super Mario characters!