User interviews done wrong
By: Adéla Tofflová
Tags: Mistakes, Research, User Research, Interview, UX Research

The not-so-hidden call for affirmation

So you have this awesome new feature, idea or a product (let’s call it a thing) and you want to run it by your (potential) customers first. And that’s great! It’s fantastic to get feedback and work with it to further improve your thing.

However, there is one sneaky little problem that every designer, PM (or whoever wants to validate anything) has to deal with. And that’s asking for honest opinions on their little baby feature or a product that they already love and cherish.

Let’s take a look at a common mistake you can make when it comes to this, with our interviewer (I) and the respondent (R):

I: “You like this, right?” R: “Sure, it’s great! I definitely love it and would use it all the time!” I: “Wow that’s awesome. Thank you!”

And now it’s very easy for you to think that this is a definite win, people love it and that you should ship it right away and make millions!

Although… not quite.

Of course, it could be true, and everyone really loves the thing, but more often than not, the reality is not that great. This is a classic example of a pleasing answer. The respondent can see that you really like your own thing and wants to make you feel good. Even if they did not like it one bit, they might feel pressured to tell you that it’s good.

Because you asked them to.

The other thing that comes into play here is confirmation bias — because you really want people to like the thing, you would ask in a way and look for such data that would support your opinion. Don’t worry, it’s natural, and the best thing you can do about it is to be aware of it (and carefully prepare your questions ahead of time).

Be careful and give the respondent the option to answer freely.

Nevertheless, there are times when even a question like this is acceptable. For example, after the respondent already said on his own that he likes something, it’s okay to repeat it back to him and ask for reassurance, and maaaaybe get a bit more information in the process as well.

A better way to do this would be to have someone who is a bit less attached to the thing lead the interview. Then it’s completely okay for the interviewer to tell the respondent “hey I did not make this, so feel free to tell me your honest opinion”. Other than that, it’s cool to ask the respondent for the way they have been doing it until now (aka the process) first, and then show them your thing. Then, you can simply ask “what do you think?”.

But don’t forget to ask additional questions — don’t be satisfied with the first thing the respondent tells you.

The hypothetical bliss

This is a big one. People tend to get lost in this and often fail to realize that they should be asking for respondent’s own experience instead of some constructed future possibilities.

Imagine your thing is doing quite well and you are considering adding another feature to the set. Let’s say you want to go and ask your customers about it first. You want to find out if they really need it, what are the problems, use-cases and come up with an elegant solution.

So far so good.

However, there are some things that could go wrong — you could be really set on doing the feature and make the mistakes described above, or you could ask the participant to predict the future.

I: “If we add this awesome feature to our product, would you use it more often?” R: “Uh… sure? Yeah.”

In this scenario, the respondent cannot accurately tell you what will happen in the future, unless they’re psychic. And even if they are, there is a pretty big chance that it won’t come true.

(I am STILL waiting for my prince charming and the house on the beach.)

The other thing that could go wrong here is making the participant validate your thoughts by asking clumsy questions.

I: “Did you ever run into this problem?” R: “No, not really.” I: “…Okay but let’s say you did, how would you solve it? Would you do this and this?” R: “Oh, well.. I guess I would do this and this.”

Notice that the first question is rather nice, asking for participant’s own experience is good.

But with the second one, you are basically forcing the participant to acknowledge the problem even though they did not ever encounter it. They would, of course, try to answer and come up with a solution, if you asked. But would that really reflect their needs and experience?

Not really.

This strategy would make them agree with your preconceptions, and you could go home, feeling giddy that your idea was successfully “validated” by a customer.

But that’s all it is… you did not learn anything about their real needs and problems; you merely got them to agree with your idea.

So you might end up getting a solution for something that’s not a problem in the first place and would be a waste that nobody would use once finished.

But that’s the worst case scenario… A good way around this is to ask your participant to describe their process and ask about any discrepancies or possible problems they face. Oh and ask about how are they solving them now. That’s a winner.

The please-don’t-take-it-away-from-me

This one is easy. People hate when you take things away from them. Even if they never ever ever use something, they still want it (remember how they wanted to scrap MS Paint?).

So if you ever wanted to remove something from your thing (be it a button or a whole feature) asking your customers if they would be okay with it, is never a good idea.

I: “How would you feel if we got rid of this thing?” R: “What? Why?” I: “Well, we can see that it’s not very popular…” R: “But I need it. What if I need to use it real soon?”

So yeah, this is not a very useful question to ask. It would be better to look deeper into how your customers use the thing. If there’s a feature only a fraction of people use, reach out to them and ask them about their use cases. Understand the process.

The “awkward” silence

The last thing I want to talk about today is silence.

I: “What do you think about this?” R: … I: “I mean this, the thing that we talked about for 20 minutes now. It’s this and this and I personally think it’s cool and all… My mom also likes it, you see.”

I have seen it many times — people always feel the pressure of silence and especially in the role of the interviewer. Because interviewers should be driving the interview, right? So in order to avoid the awkwardness, people tend to overly explain the task or their question, or even sabotage the research by spilling out their own views, expectations, etc.

Remember, the respondent also needs some time to process the question and formulate their answer.

In the ideal world, the participant should be talking way more than you. But that’s not always the case, and that’s ok.

And as bad as long stretches of silence seem they are there to help you.

So instead of filling them with potentially harmful blabber, use them to your advantage and let the participant talk, observe their body language or think about the best way to ask your next question.