Creating experimentation as a habit
By: Bithika Mehra
Tags: Experiment, Experimentation, Cro, Conversion Optimization, A B Testing

A couple of months back, I heard about the book Atomic Habits By James Clear on The Mindvalley Podcast. Atomic Habits provides a blueprint for creating good habits and sticking to them.

I have struggled with creating good habits. Well, who hasn’t? So, I listened intently, nodding my head vigorously during my commute back home. Needless to say the book has stayed with me and I have revisited the concepts repeatedly.

“Habits are like atoms of our lives. Each one is a fundamental unit that contributes to our overall improvement.”

The book lists steps to create good habits, which are applicable to creating an experimentation culture as well. In this post, I will draw those parallels and I hope to provide ideas to practitioners to drive adoption.

Identify cues.

When we walk into a dark room, we don’t think of what to do next. We instinctively reach for the light switch. And this is pretty much what we do every time we walk into a dark room.

Habits begin with a cue, or a trigger to act. Walking into a dark room cues us to perform an action that will enable sight. Next comes a craving for a change in state — in this case, to be able to see. Then comes our response which is the action to flick the light switch on. The final step in the process is the reward — the mild relief from being able to see our surroundings.

We know that behaviours that give satisfying consequences tend to be repeated until they become automatic.

So how do we apply this to experimentation? What could be the cues? As per Ton Wesseling (Founder, Online Dialogue and Instructor at CXL Institute), the following are the instances when we should use testing:

Deployment: At the time of any deployment, release the change to a certain percentage of the users to ensure that there is no negative impact of the change.

Research: When you are looking for signals to see what elements on the page or in the flow are working or if a change will adversely impact user experience.

Optimization: When you are looking to make improvements in bottomline metrics and are looking for the wins.

It’s necessary to establish testing as a part of the process to guide decision making at all these junctures.

Have a plan.

Most of us tend to be too vague about our intentions. We say “I’m going to eat better.” and simple hope we’ll follow through. Instead, the author suggests having an implementation intention. An implementation intention introduces a clear plan of action, setting out when and where we’ll carry out the habit we’d like to cultivate.

In the case of experimentation, this means having a roadmap of hypotheses originating from the cues identified above and consequently the tests you are looking to run over the next few months. Simply saying, “yeah, we gotta test more” or “we should start testing” is not enough. Have a plan for what, when, where, why and how you will test. Don’t have a testing tool or capability? Start with adding that to the roadmap first!

Be consistent.

The author gives an example of a plane taking off from Los Angeles en route to New York that adjusts course 3.5 degrees to the south resulting in the plane’s nose moving just a few feet. This change isn’t noticeable on board till the passengers alight in Washington DC and not New York, much to their dismay!

Such is the power of small changes. You can make tiny changes to your behaviour which when repeated time and time again, become habits that could lead to big results.

It is no different for organizations starting on the path of experimentation. It is important to get your team and the organization in the habit of testing — not major changes requiring months of development work but in the true agile spirit, small iterative changes.

Set a goal for a number of tests for the team. But be sure to empower your team with the necessary tools, training and support required for testing or this could lead to frustration.

Small habits doesn’t only relate to running small tests, but also to making small changes in your processes to drive testing. It could be a very simple exercise where a project kick-off includes listing down all the assumptions being made. The team can then figure out how to prove or disprove each through data.

Make it easy.

This is a no-brainer. We like easy. Want to eat healthier? Put fruits on the counter and chips and cookies on the top most shelf in the pantry. (Bonus: Hide the step stool.)

How can we make experimentation easy? It certainly does not appear to be easy with questions ranging from am I testing the right thing to will this break something on the website or app to am I analyzing the test correctly! There are so many things to plan for when planning to run tests.

This is where organizations that have managed to get experimentation in their DNA really shine. Organizations like Booking.com have managed to democratize access to experimentation at all levels. It is no longer a tool of the chosen few but all employees right from copywriters to developers to product owners are equipped to run tests.

On the flip side, make it harder to not test. Make experimentation results mandatory for big changes. There should be evidence not opinions backing any consumer facing changes being made.

“Many people think they lack motivation when what they really lack is clarity.”

Attach rewards.

We live in a delayed-return environment. You go to the gym or have a salad for lunch, but you don’t lose weight overnight. Our brains, however, have evolved to cope with an immediate-return environment of earlier humans who weren’t thinking of long term returns like saving for retirement. They were focused on immediate concerns such as finding food and staying safe.

Now how can this be applied to experimentation? Testing, like most things, functions in a delayed-return environment. You set up a test now and have to let it “cook” for 2–4 weeks before analyzing the performance. Further, you are running iterative tests that when combined yield a higher return, potentially attributable to your experimentation program at the end of a quarter or a year.

Recognition and celebrating milestones such as first test, tenth test etc. through some sweet treats could be some effective ways to build in rewards. Who doesn’t like cookies or doughnuts around the office? Remote teams will need to get more creative.

It is important to celebrate testing and learnings, not the wins or the losses.

Track your habit.

A habit tracker using a simple calendar or diary and crossing off every day that you stick to your new habit is an effective way of staying on track. For experimentation to become a habit, dashboard, gantt charts, roadmaps are all good ways to see how you are doing against your goals. Are you running the tests you had on your roadmap? How many tests are you running?