Opening Up the Open Question
How To Get More From Your Open Questions
Eric Greenberg, Japan Country ManagerSurvey Sampling International#SSIwebinar @SSItweets© 2016 Survey Sampling International

You have 5 minutes with the President…..

You should
more benefit
havebe more
kept likeyourtheforeconamy
n/acollegewords
clintonand not
students
just speak#SSIwebinar @SSItweets© 2016 Survey Sampling International2

You have 5 minutes with the President
You have done a great job.
People have VERY shortmemories of all the good you
have done for our country,
and just love to pick on youfor every wrong that
happens in this country.
Please continue with yourcoverage of health care andnot listen to the naysayers.

#SSIwebinar @SSItweets© 2016 Survey Sampling International

3

Why can’t the world be more like this?

#SSIwebinar @SSItweets© 2016 Survey Sampling International

4But let’s not get carried away (yet)
•
•
•
•
•
•

Who could make a video?
Who does make a video?
Increasing conversion
Does it work?
Is video data comparable?
Are there biases….?

#SSIwebinar @SSItweets© 2016 Survey Sampling International

5

Who Does It?

#SSIwebinar @SSItweets© 2016 Survey Sampling International

6

Who could make a video?
• Need camera and microphone• Have camera = 54%• Have mic = 59%• Have both = 48%#SSIwebinar @SSItweets© 2016 Survey Sampling International

7

Who could make a video?

#SSIwebinar @SSItweets© 2016 Survey Sampling International

8

Who does make a video?

#SSIwebinar @SSItweets© 2016 Survey Sampling International

9Who does make a video?

#SSIwebinar @SSItweets© 2016 Survey Sampling International10Factors influencing conversion• Planned usage— Coding only
— Display at MR events (client presentation)— Public display• Language style— Legalese vs plain language#SSIwebinar @SSItweets© 2016 Survey Sampling International

11

Planned usage#SSIwebinar @SSItweets© 2016 Survey Sampling International

12

Language style#SSIwebinar @SSItweets© 2016 Survey Sampling International

13

Language style within planned usage#SSIwebinar @SSItweets© 2016 Survey Sampling International

14

Gaining informed consent• You have to say something!
• It is ‘expected’“Q7_1 We’d like you to video yourself answering the question that
follows, your camera and microphone will be automaticallyswitched on. Is that okay?”• 49% acceptance (lowest of all)

#SSIwebinar @SSItweets© 2016 Survey Sampling International

15

Does It Work?

#SSIwebinar @SSItweets© 2016 Survey Sampling International

16

Experiment #1• Ordinary panel members from SSI panels• Doing normal (dull) survey about home heating• Two open questions:
1. Easy question: Why do you prefer your preferred heating fuel?
2. Difficult question: Why would you switch/not switch to electricity if
it came from sustainable resources?

• Random assignment to Video : Non-video group• Video group self defined into Made Video : Refused Video#SSIwebinar @SSItweets© 2016 Survey Sampling International17

Q1 results: character count• 6x more characters!

#SSIwebinar @SSItweets© 2016 Survey Sampling International18

Q1 results: answer count• 1.5 x more answers!

#SSIwebinar @SSItweets© 2016 Survey Sampling International19

Q2 results: character count• 5x more characters!

#SSIwebinar @SSItweets© 2016 Survey Sampling International20

Q2 results: answer count• 1.33 x more answers!

#SSIwebinar @SSItweets© 2016 Survey Sampling International

21

Same top answers#SSIwebinar @SSItweets© 2016 Survey Sampling International

22

Boost in importance of secondary answers#SSIwebinar @SSItweets© 2016 Survey Sampling International

23

Unexpected boost in minority answer….

#SSIwebinar @SSItweets© 2016 Survey Sampling International

24

Similar top answers#SSIwebinar @SSItweets© 2016 Survey Sampling International

25

Small boost in secondary#SSIwebinar @SSItweets© 2016 Survey Sampling International

26

Response quality: video vs. text
Why do you prefer <HEATING FUEL>?

Vox Pop
Hi, I like gas because I like to cook with it, with arange top. It's fast, it's easy, you know it's on, youknow what the temperature is based on how big theflame is. I like gas because it's quiet. I like gas
because you don't smell it. I think it's cleaner thanoil. I know it's less expensive than oil. I like gas, it's
maintained better than having to call the oilcompany to come get it

Standard OE
Clean, no smell, easy
control
Electricity does not explode after
an earthquake; there are no fuellines to rupture.

Gas is a lot cheaper and more efficient I think, thanusing anything else. It's there at the click of a button,
like for cooking. Just click it and it's on. And, for thegas heating, it's a lot easier, because you just turnthe gas radiators on or the heater on, and it's there.
You don't have to worry about whether you've got
any money or wood or anything to actually heat yourhouse.
#SSIwebinar @SSItweets© 2016 Survey Sampling International27

Response quality: video vs. text
Why would you switch/not switch to sustainable electricity?

Vox PopAlthough I like the idea of sustainable sourcing to
generate electricity and making that a more viable
source for heating, I've mentioned in my previous
response that I had electric heat once before and I do
have some issues with allergies and such and electricheat just makes the air very dry and it seems toaggravate my allergies. For that reason, I probably
would avoid electric heat unless there was someother system in place that would allow it to be lessdry. But I do like the idea of it being a sustainablesource of fuel that doesn't rely on other forms of fuelsuch as coal or nuclear energy to be generated.

Standard OE
save the earth resources
helps the planet and futuregenerations
It is cheaper and less damaging toour atmosphere

I would choose sustainable sourcing because wind
power - wind is regenerated all the time. Solar panels- the sun comes up everyday, so it regenerates. And it
would be free. Can't beat that.
#SSIwebinar @SSItweets© 2016 Survey Sampling International28

Who does a video?

#SSIwebinar @SSItweets© 2016 Survey Sampling International29

Who does a video?

#SSIwebinar @SSItweets© 2016 Survey Sampling International|30

Experiment #2• Ordinary panel members from SSI panels• Doing “divisive” survey about political issues• One open question:
1. If you had the opportunity to talk to the President for just 5minutes, what would you want to tell him? What should heconcentrate on for the future? Where is he going right? Where is hegoing wrong?

• Assignment to Video: No-video based on equipment• Video group split into Video: Refused based on consent#SSIwebinar @SSItweets© 2016 Survey Sampling International31

Who does video?

#SSIwebinar @SSItweets© 2016 Survey Sampling International32

Practical• About half can do a video• Over a half of these will do a video• Demographically biased• Be careful with permissions• Equipment problem will go away
• Consent issue may get better with experience#SSIwebinar @SSItweets© 2016 Survey Sampling International33

Data quality• High quality colour and depth• And completeness• But do not take alone• Data is attitudinally biased• Allow a type-in alternative• Measure saliency/interest to balance#SSIwebinar @SSItweets© 2016 Survey Sampling International34

Do• Check for presence of camera and microphone• Check both are working• Get informed consent• Warn against videoing others• Be aware of situation— At work— Others around#SSIwebinar @SSItweets© 2016 Survey Sampling International35

Don’t••••••Expect a video response when they say “yes”
Frighten off with excessive permissions
Use plain language
Expect “bright young things” that you could drop into an advert
Over-analyse the best looking videos
These are just normal respondents in their own homes, in theirown time, with their own lighting…

#SSIwebinar @SSItweets© 2016 Survey Sampling International

36

Summary: The value of video• Solves a real problem with online self-completion, the paucity ofdata in open questions• Gives rich and deep insights• Video OE is a major step forward#SSIwebinar @SSItweets© 2016 Survey Sampling International37

www.surveysampling.com#SSIwebinar @SSItweets© 2016 Survey Sampling International38