Why are there data bias on your survey?
By: Jennifer Hall
Tags: Surveys, Bias, User Research, UX Research, Quantitative Research

As we all know, it is significant to recruit the right users, especially when you are recruiting users for filling a survey. Because survey is a quantitative research method, and it requires rigorous and strict data. If the respondents are not the right users, it won’t bring you the valuable results, and may even lead you into a wrong direction on the product development. So, we have to recruit the users from our target market, with the typical features of characteristics which can represent most of our users. But how to target your typical users? So here are 6 things that you should consider before fielding a survey.

1.Motivation for filling the survey

Just think: when there is a survey, no matter online or offline，what is your motivation to fill it? If you don’t have enough time，or you have more important things to complete，or maybe you are not interested in the topic of the survey，or the survey can’t bring you any benefits or profit, nor help you solve the problems you are facing, you will ignore the survey and then leave. Meanwhile, it is obvious that respondents are the ones who want to earn money/commission. Or the ones who are very emotional, which means that these kind of respondents have lots of things to complain about the product，and they need a way to express their feelings and ideas and emotions, so your survey becomes their way to complain.

2.The time and date/month you publish your survey

If you publish your survey on weekends，it is possible that people will ignore your survey because they will relax on weekends. If you publish your survey on the time when your users are very busy，it is also possible that people will ignore again. The time lag should also be taken into consideration. For example, EST and PST. When is the time that the users frequently used your website? This is the time that you can get highly responses from the users.

As for the date, for example, your product is a shopping website，the user data collected from the survey, which was published before Christmas Day or after Christmas Day, will be different. Specifically, people will buy gifts for their friends and family members before Christmas Day, and after Christmas Day, they are less motivated to visit the shopping websites.

3.The places or the ways that survey was published

Your survey may be published online or offline. If you publish your survey online, the people who see your survey are the ones who are online. How about the offline users? What is the proportion of the offline users? And what is the proportion of the online users? Is it possible to know the key characteristics of the typical users if the offline users are ignored? And, you have to check whether your survey is fluent enough to be loaded. In other words, how long will be the time of loading the survey page that the respondents have to wait for?

About publishing your survey online:

First of all, what ways to choose for publishing your survey online?

●If you publish your survey on your own website, the respondents and visitors on your website are the ones who are interested in your website. What are their demographics? For example, their career, their income, their gender and so on. The role of the users always influence their needs, their goals and their behaviors. In addition, in what kind of context, what time and date, and in what kind of motivation or reason, that they become the visitors or even buyers of your website? What’s the proportion of them in the total number of typical users?

●Then, if you choose Facebook groups or other online communities to publish your survey, what are the types of these members in these groups? Are they your typical users? How will your differentiate these users? What’s the proportion of them in the total number of typical users? Is it possible that they will provide you with the valuable data? Valuable data means that you will be able to collect more than 85% typical characteristics from these respondents. Thus, before you publish your survey in this social groups, you have to know the ingredients or, say, the parts of these social groups very clearly.

●Last but not least, if you publish your survey to the email addresses of the users, who are these users? Are they your typical users? If yes, what’s the proportion of them? Is there a possibility that your email are spammed by them?

Second, if you publish your survey offline, which places should be chosen? For example, if you are fielding a survey for finding out the user data of sports events booking platform, it is likely to touch your typical users and the largest proportion of them in sports facility complexes, other than the places like shopping malls. Because your typical users are soccer players and volleyball players, they often go to the sports facilities, and the people in the shopping malls are the ones who want to purchase goods, or experience customers’ lives. So before you publish your survey, you should consider which places contains the most majority of your typical users.

4.Expert users, senior users and junior users

In most cases, expert users and senior users can provide you with over 85% of the typical characteristics of the majority of the users. However, how will you define expert user, senior user and junior user? It depends on the frequency that they use your product, their experiences of using your product, their understanding on your product. The situation could be different as the industries changes and the market changes. To know more details, you can learn from the marketing and advertising departments of your company.

5.The anticipation and imagination before users filling your survey

The words and expressions you use in your survey, including the titles, the sentences and the paragraphs, the order and the logic(causes and effects) of the questions, will all be able to lead to different understandings. So when you are editing the questions, don’t use terms or some professional or complex words, don’t ask misleading questions, and don’t combine more than one topics in the same question, one question, one topic. These will help you minimize the bias on the anticipation and imagination of the users when they are reading your survey.

6.If your competitors fill the survey with false data on purpose

There is also a possibility that your competitors will give you the false data to mislead you into a wrong direction on the product development with some technical ways.

So, how to solve this issue and minimize the bias of collecting the data when you are fielding a survey?

1)You can create a proto-persona based on the basic data of the typical users, and write the recruitment criterias based on the characteristics of the typical users at the same time.

2)Then you can recruit 1–2 users to have an in-depth interview, to check whether the survey you designed can help you find out the valuable data or not.

3)After editing your survey, you can pre-publish your survey out and recruit 5–10 users, to check whether your survey is designed effectively. The purpose of this step is to help you avoid wrong direction.

Thank you for reading here. I’ll recommend some books, courses and tools for sophisticating and improving your quantitative research skills.

Books:

Survey Research Methods by Earl Babbie

The Handbook of Online Marketing Research by Joshua Grossnickle and Oliver Raskin

Online courses:

UX Research at Scale: Surveys, Analytics, Online Testing from University of Michigan

中国人可以看这两个课程：

问卷调查设计及研究方法 — — 浙江大学

社会调查与统计分析 — — 大连理工大学

Tools:

Sparklit is an ad-based polling service that allows you to ask basic multiple-choice questions and then tabulates the results as simple tables.

Zoomerang offers a larger set of question types, a selection of premade surveys, and the option for more sophisticated analysis tools for a fee.

SurveyMonkey similarly offers a larger selection of question types and more survey options.

Other tools:

http://doopoll.co/

http://www.google.com/forms

http://surveys.google.com/

https://mindswarms.com/

https://qualaroo.com/

http://www.responster.com/

http://spssau.com/front/spssau/

http://www.surveypal.com/

https://surveynuts.com/

http://www.survicate.com/

https://www.typeform.com/

https://www.yesinsights.com/

The tools for publishing your survey:

https://www.qualtrics.com/

https://www.alchemer.com/survey-software-features/survey-branding/