A swiss cheese model for reducing biases in user research
By: Biswajeet Das
Tags: User Research, Usability Testing, Research, UX Research, Cognitive Bias

A swiss cheese model for reducing biases in user research

Fig: Swiss Cheese Model for preventing Corona Virus infection.

Every accident has underlying factors behind its occurrence. Those factors are usually human or design-related which causes slips and mistakes, which cause the accident. Sometimes we detect the right causes and sometimes we become biased and find the wrong causes behind the accidents due to various reasons like— time constraints, cognitive biases, institutional pressure, social pressure, etc.

The Reasons Swiss cheese model of accident causation is a model used in risk analysis and risk management, including aviation safety, engineering, healthcare, and emergency service organizations.

Each slice of cheese in the cheese model are precautions. The holes in the cheese slices are the loopholes or chances of imperfection in each precaution slice of cheese. When these cheese slices of precautions get aligned in such a way that the line of hazard/error can pass through it easily an accident is caused. According to the Reasons Swiss Cheese Model these accidents or errors can be prevented if we take enough precautions.

So the idea or the aim of the Reason’s Swiss Cheese Model for Accidents is to suggest us to not try and find errors/hazards behind accidents to rectify them as there are high chances that we end up finding the wrong errors causing the accident. But rather it suggests us to try and minimize the accidents by adding more cheese slices(precautions), not allowing cheese slices to align in such a way that line of hazard/error can pass through to cause an accident and try and reduce the size of the holes(loopholes/imperfections) in each slice so chances of line of hazard/error to pass through becomes more difficult and hence increases chances of preventing an accident. All layers are important because no layer can be perfect, so working on these cheese layers(precautions) and adding more layers is what the Swiss Cheese model suggests us to avoid accidents.

An example of a Swiss Cheese model is one which was developed during this COVID-19 pandemic by WHO to prevent people from getting infected by the deadly Corona Virus. Without this model, the number of Positive Cases would have been even more than what it is now. People are still trying to add more layers to it so that the effect of the virus reduces significantly. Things like working on a Vaccine, healthy diet plans, etc. are just a few to mention.

Applying it for User Research

In the scramble, while looking for insights we often don’t realize that we get cognitive biases, they are sort of like optical illusions, but not only they interfere with the way we see things, they also get in the way of our behavior when we make decisions and solve problems. This can affect the product or the service we are doing the research for immensely as we will not end up solving the right problem solving which the experience of the product will improve hence get new customers or fully satisfy existing customers. Biases will never let us see the real picture but a blurred picture of which we ourselves won’t be fully sure of. Being aware of these biases can help us make better decisions. Training our own minds to not get biased while doing user studies is what we should do. How do we do it? Is not that easy but not impossible.

So for User Research to reduce the effect of cognitive biases. We suggest a Swiss Cheese Model of Avoiding Biases for the User Researchers which would reduce the biases which occur during User Research and help us to develop more User-Centric products and services.

User Research gives direction to the Product Development at each stage — Discovery, Building and Testing.

How do we know or assume that there is biasness in User Research? The Metrics?

a) When Customer Experience of the Product or the Service is not upto the mark. We know this through user feedbacks on product as well as through qualitative and quantitative research methods like User interviews, Usability Tests, etc.

b) When the User Experience Strategy fails. UX Strategy includes bringing together technical know-how, business strategy, customer needs, and your vision for the experience you want the user to have.

c) Checking Consumer Data based on users' behavior and engagement with the service or the product in activities like — Drop-offs, New customer acquisitions, use of functions, etc. If these things are not like what was intended there are chances that there were biasses in User Research.

User Researchers are humans too and humans can make mistakes. We all are prey to Cognitive Biases in our day to day life in some way or the other no matter how wise we are. Whether it was biased while taking the decision of buying a stock in the stock market or buying the right medicine. So User Researchers are no different.

The different types of biases we see during User Research are:

I’ve collected a list of 17 Biases. It’s good to know them. It will be really helpful to be aware of them during research to stay bias-free.

1.Confirmation Bias — Ignoring user pain points because they don’t fit into our existing assumptions.

A typical example would be a researcher/product manager asking the participant — “Have you ever done/considered doing Action X through App Y”, and the participant says, “Yes! All the time!”. Hooray! you have hit a jackpot and your evidence is confirmed. Users love taking that action through your app. But in reality that might be far away from the truth.

2. Anchoring Bias — Information we receive before going out and doing our User Research can anchor us.

We can my experience as an example, when I was working as an intern in one E-commerce company this incident happened: I was asked to find out user behaviors and attitudes while reselling using the companies app. While transcribing the recorded interviews I completely omitted all other parts like the users motivations, their views and their expectations thinking that those were not very necessary because I was so focused only on users behaviors and attitudes while reselling. My senior caught this at the right time and made me re do the transcribing and documenting as I was missing many Important things which can be very useful to our research.

3. Hindsight Bias — In order to create synchronicity and order to the ebb and flow of time, we try and find reasons for the events that happened in our past, without having any factual evidence for it. It is often called the “I knew it all along” phenomenon.

For Example : A user was complaining about his business not running well due to the oncoming of the internet and people buying things online rather than coming to the stores. So when I posed the counter question of why he was not getting into e-Commerce, I received a rather surprising answer: “The websites are not taking good care of customers and there are delivery issues and most of my customers are not tech savvy and they are not very comfortable with online payment”.

4. Overconfidence Effect — The team can be let to believe that one thing a user said is the most important thing and applies to all.

For Example: During Usability Testing the moderator asked people why they are not able to use the filter function to search products on the app. Few said they could not understand how it is used, some said they could not discover it. So the moderator just asked “Have you used the filter function”? If the User said No, he would take an assumption that it must have been because he/she couldn’t understand its function or might not have discovered it. The cause for the other users behind not using Filter option could be — not being able to search relevant products under category while searching using filters or their perceived usefulness of the function is low.

5. Social Desirability — We are social animals, and this means that our actions and words are presented in a way that makes us look good amongst others, even though they might be inaccurate. This is so deep-rooted in our behavior that we even disdainfully label those who don’t follow these norms as anti-social.

For Example: “ Some middle aged housewives who are not very tech savvy sometimes do not admit that they face problems while using the app when interviewed. They actually take their children’s help to use the app”. When we did Usability Testing we found out that what they say they do in the app and how they actually do it is very different. Why didn’t she tell me the truth? Because she is image conscious, she don’t like it when people find out that she doesn’t know something.

6. Blind-spot bias — Most people believe the people around them are more biased than they themselves are. This is Blind-spot bias. Our ability to perceive bias in others is actually pretty good. Our ability to perceive bias in ourselves is generally dreadful. This leads to bias that can damage the way we work and an inability to tackle it — because we can’t see it.

For Example — While doing User Research the Researcher sometimes think that they are on the right track and solving the right problem. They prioritize things which is not so important to solve over other thing which are more important.

7. Culture Bias — This happens when a researcher interprets results based on their own cultural beliefs or attitudes rather than from a neutral point of view. It can also involve a moderator making involuntary suggestions to a participant which affects the way a participant answers.

For Example — “The moderator thought that since she speaks English she would be able to use the app better than the non English speaking ladies. But in reality although she speaks English she was not very tech savvy”.

8. Hawthorne Effect — The participant becomes conscious during an interview or an usability test and behaves differently.

For Example — “ As a regular user of the app to sell sarees, Mrs. Dixit got very excited when someone from the company took her app Usability Test. She got very conscious and ended up making mistakes by trying to show off her on the app skills to the moderator”.

9. Availability Bias — Compromises made while selecting sample for studies and selecting the wrong sample just for the sake of research due to lack of time, lack of proper screener questions or availability of users.

For Example — A research was performed on the users of a reselling to know how they are using the app to do their business. We contacted Mr. Raghu as our CX team gave us his number saying he is a prominent user of the app. “When Mr. Raghu was asked how he is using the app, he said that he don’t use the app anymore and that he gave his account to his brother living in the other city who is using the app really well, but Mr. Raghu’s number is linked with the app, and since we couldn’t get his brother on the line who is currently using the app we ended up talking to Mr. Raghu who said he will talk in behalf of his brother and tell us how he use the app.

10. Sunk Cost Fallacy — As researchers, we invest a lot of time into conducting research and collecting data. Over a period of time, this data can become a burden rather than being helpful. Obsessing over the findings, we can easily get lost and miss the bigger picture of what we really need to achieve and deliver.

For Example: Once I was conducting a User Research for an engineering and business solution company in I had to understand how the stakeholders of the Pharmaceutical Cycle worked and their knowledge about counterfeit drugs and how they tackle it. The company was developing a New Blockchain Technology to trace counterfeit drugs. I interviewed around 20 pharmacists, 5 drug manufacturers in the city. I invested around a weeks time to do all the interviewing, documenting and finally presenting the insights. Later most of the insights were not used as most of the answers from the stakeholders were under social pressure and were hence biased. I was so obsessed with the findings that for a moment I was not able to achieve the bigger picture. I realized that rather than asking people directly whether they have sold counterfeit drugs by mistake or asking them details about their manufacturers or customers who have complained about drugs I should have rather framed the questions a little differently to have received more unbiased insights from them because these are sensitive issues which people won’t like to disclose if asked by an outsider.

11. The Illusion of Transparency— In interviews, many participants try to convey their emotions through body language, pauses, and other non-verbal cues. The illusion of transparency makes it difficult for them to really know whether the message is being conveyed rightly.

For Example — During some interviews some participants showed very were very happy that we called and they gave remarkable insights and feedback about our app and some showed that they were very busy. But we convinced the busy users to join the study as it was important for us to talk to them so that we can solve their problems. Their being busy to give us just 15 minutes might indicate their desire to use the service/product, it might also tell us how useful they think it is to talk to us. Here we should try to find out pain points of the user. The participant might not be able to tell us exactly why he/she is not able to use our service or why he/she is not willing to use it properly due to various reasons. Reasons like social desirability to portray them as tech savvy or business Gurus. In such cases we should try to open them up framing the questions tactfully(that’s what they wanted).

12. Fundamental Attribution Error — Humans tend to attribute errors to internal characteristics even when it is situational/caused by an external force

For Example: When you conduct usability tests and you hear a user talking about making a ‘mistake’ doing a certain task, pay attention! Ask him to perform the task and try to understand his situation, background, etc which is making him do the mistake.

13. Wording Bias — Framing effect takes place when we do not ask open ended questions. We sometimes tend to unknowingly force the participant to answer our question or perform an activity in a particular manner in which we assume things should be like. That we we don’t get the real answer to our question.

For Example: Asking a question like “ How do you use search option while using the app”? is far better than asking “How do you use search option to search for products”? as asking them how they use it leaves it open to them to say things like “ I use search option only when I cannot find a product in the widgets or in the banner”, “ I use have used search option just once” or even tell you his/her steps of using it. But if you ask how do you use it to search products(because everyone knows search is used to find products) the answer might be biased, you will only know the steps of using search option which we would anyways have chances of seeing during usability testing.

14. Funding Bias — If respondents know what is being researched and who is trying to prove what, it will most likely influence the way they answer questions.

For example — Participants when interviewed by some higher level executive from the company to get their honest feedback about the service usually hesitate to tell minute negative things about the service because they think that they’ll appear ungrateful. Unless prompted to give honest feedback through well framed questions using assumptions they would not give any negative feedback about the service if they are earning well enough by reselling through the e-commerce service in front of an top level guy from that company. They would rather feel more comfortable to tell about their problems to the customer care over the phone.

15. Friendliness Bias — If respondents know what is being researched and who is trying to prove what, it will most likely shape the way they answer the questions. Here people providing feedback tell you the answers they think you want to hear.

For Example — If the respondent know that the moderator is from the research team from the company and the moderator started of the aim of the research like “ I want to see how our expert users behave with the product” just to make the respondent happy as everyone likes to be praised. Here the respondent will try to do everything correctly and won’t show him those steps where he/she gets stuck or is unable to understand.

I have added bias — 16 to the list from my own experience as an User Researcher with an E-Commerce start up where I had to interview people who were not very comfortable with our language and we neither had a translator in some cases.

16. Language Bias — Sometimes the participant doesn’t properly understand the language of the interviewer. That affects the answers which we get out of the questions. The participant tends to give shorter answers and the interviewer tends to try and guide the participant towards an answer which he thinks is suitable.

For example: I was doing a Handicraft Research in Meghalaya, India with villagers who didn’t speak English nor Hindi the national language of the country. So I had to use an interpreter who knew a little bit of their native language. Although I noted down what my interpreter narrated for me. Most of the insights were quite biased. I couldn’t get to their real problem other than the very common problems like government funding unavailability and their current economic condition.

There are more biases to cover but to avoid making this article too long I’ve listed just 17 here.You can learn about more biases which I have not mentioned here as you keep conducting User Research. You can refer to Dan Kahneman’s “ Thinking Fast and Slow” to learn about more of these. It’s a great book for all User Researchers.

Working on the Swiss Cheese Model

Step 1:

Let’s Build the Cheese (Guidelines/Precautions) for the Swiss Cheese Model

We will have 6 pieces of cheese slices in our model.

Secondary Research Guidelines User Sampling Guidelines Preparation Guidelines Execution Guidelines Behavioral Guidelines Data Analyzing Guidelines

Step 2:

After that lets compile the list of Do’s and Don’ts for the User Research and club them under the Guidelines. These will be the holes in the Cheese slices.

List of Do’s and Don’ts:

1.Secondary Research Guidelines: Evaluate the secondary data if there is any — ask these questions:

a. What was the research provider’s purpose in presenting the data?

b. Who collected the data?

c. When was the data collected?

d. How was the data collected?

e. What type of data was collected?

f. Is the data consistent with data from other sources?

Note down if there are any assumptions/hypothesis to validate before beginning the study.

The Right Research Approach — Decide these ahead:

a. Problem Statement — What problem will you solve by doing this research?

b. Decide which Methods you will use and in what sequence — quantitative or qualitative.

c. What criteria will you use for defining actionable insights?

d. What metrics do you plan to use to analyze user behavior (i.e., ideal time-on-task, expected bounce rate, etc.)

e. How will you categorize user behavior quantitatively (i.e., what percentage of test participants should successfully complete the flow)?

f. How will you present the Insights.

2. User Sampling Guidelines: choose participants only who are representative of your target audience. Do not compromise on your participants.

a. Decide on the number of users required and how you will categorize them according to the need of the research. If you have any criteria’s like — age, behavior or motivation with the product/service, new/old users, sex, etc. than plan all that ahead. For Example: “ I need 10 ladies from the semi-urban areas who have recently signed up (New Users) with our app. I want to study how ladies use the app when they are New. I want to know how their children or husbands help them if they ever help them”.

b. Get their numbers from the Customer Experience team and call them or send them a message asking their permission for a screening or try calling them directly. Keep extra number of participants so that if some participants backs out at the last minute you will still have more people for the research.

c. Do the screening process: Screening is very important. Ask the participant if he or she is willing to volunteer for the study, if they say yes ask them for a time when they are free and tell them how much time you need from them. Know about their background — place,occupation, age , sex, etc before conducting the study. Use consumer data to know how the participant had been using the product or the service till now if it is available.

Remember: No compromise on sample size and quality.

3. Preparation Guidelines

Write an easy to understand User Test Script/questionnaire for interview. Keep it short and precise.

Think carefully about the sequence of questions, and consider alternating the order from interview to interview. This isn’t necessary for all questions, mainly ones that might trigger the anchoring effect.

Let the questions be mostly open ended, keep clear wordings, prepare a list of follow up questions. Keep them with you.

Questions should not be worded similarly, and should not lead to an answer.

Discuss the questions for the scripts and questionnaires with members of other department’s as well to get their point of view too.

Do a practice user interviews and usability tests with team members to see how it works out.

Keep a notepad to take notes during the test, a print of the script/questionnaire ready with you.

4. Execution Guidelines

Ask open ended questions, present task, users should never be pushed into confirming a specific outcome or problem, Clear wordings, Neutral and Straightforward, follow up questions — very important(prepare ahead but go with the flow) — so that we understand what is important for them.

Let just one person ask the questions, others take notes or ask a follow up only if necessary — why’s, how’s, etc.

Respondents should not know who exactly is doing the research.

5. Behavioral Guidelines

Check your emotions. You can’t necessarily control your grumpy mood, but try to be aware of how you’re feeling when going into an interview — write it down and, when analyzing your findings, consider whether your mood could have had an effect.

Talk less, listen and observe instead. Ask a question and then get out of the way. Don’t rush to fill silences with your own observations or further question.

Watch your body language. Keep a poker face. Be neutral, careful not to respond positively or negatively in what you say or how you act.

6. Data Analyzing Guidelines

For every piece of feedback, ask yourself how you’re framing the data. The awareness of a process will help you understand why your perspective feels positive or negative.

Take into consideration additional perspective from competitor research and other team member inputs.

Step 3:

First, Let’s break the User Research into 5 Phases:

Secondary Research Phase or the Discovery phase— During this phase we do our background secondary research to form a foundation for the research. We check existing data, form assumptions to validate. Approach Phase — During this phase we plan the research — the problem statement, the script, aims, metrics, methodologies, sample size, how we will use the collected data, etc. Preparation Phase — During this phase we prepare for user research — writing questionnaires, usability testing scripts, mobilizing the users, etc. Primary Research or Execution Phase — In this phase we perform the research — User Interviews, Usability Tests, etc. Data Analyzing Phase — In this phase we analyze the data, filter out useful actionable insights from both qualitative and quantitative data accumulated during the research.

The 5 User Research Phases for the Swiss Model of Biases

Step 4:

Building the Swiss Cheese Model of Biases — Understanding how it works.

Our 6 Cheese slices(which are the precautions in the form of guidelines)of the model are going to pass through these 5 phases in the Swiss Cheese Model. We are adding these phases here so that the researcher knows exactly when to take precaution at which phase of the research and know exactly at all times and be aware of what kind of biases may happen at a particular phase and the necessary guidelines/precautions they should ideally follow to avoid biases. The guidelines/precautions will be the cheese slices and To DO’s and Not to DO’s will be the holes in the cheese slice through which line of biases will pass.

In each guideline(represented by the cheese slices)we are to follow a set of rules as precautions which are the — Do’s and Don’ts(represented by the holes in the cheese). If you follow all the Do’s the size of the holes will decrease and if you don’t follow the Do’s the size of the holes will increase. If you don’t do the don’ts the size of the holes will decrease and if you do the don’ts the size of the holes will increase hence allowing some biases to happen by allowing the biasness lines to pass through the guidelines/precaution slices through the holes and finally giving us biased insights of the research.