Designing with Machine Learning
By: Jonathan Remulla
Tags: Machine Learning, Technology, User Experience, Artificial Intelligence, Product Design

Janice just started her job at a brand new start-up three weeks ago. She’s the very first customer success specialist and wants to make a good impression on her new bosses. They haven’t done any research to understand what customers are saying about their product, so the founder suggests using SurveyMonkey to send a survey to their 5,000+ clients …tomorrow. “Sure!” says Janice, through the teeth of her contrived smile. The problem is Janice has never created a survey before, let alone sent one to 5,000+ people. Janice thinks to herself, “Holy cow, how am I going to do this?”

Fear, anxiety, lack of confidence — all common emotions new customers of SurveyMonkey experience when writing their first survey. You can’t un-send a survey, so customers need a way to confidently create a professional survey that will return valuable data that can inform better customer understanding or to make a decision.

My product manager and I tried to answer three questions:

How might we leverage existing SurveyMonkey expertise and data?

How might we increase confidence and reduce anxiety?

How might we automatically create a professional survey for users?

SurveyMonkey has over 60 million registered users. Of those, 16 million are active users. We receive over 20 million responses per day. We definitely had the data we needed to begin designing features driven by Machine Learning.

What is the difference between AI and Machine Learning?

Artificial Intelligence, or AI, has been at the tip of our tech-tongues for a while now. There is still a lot of confusion around the difference between AI and Machine Learning. In some instances, you’ve probably heard them used interchangeably.

AI is the broader concept of machines being able to carry out tasks in a way that we would consider “smart.” Machine Learning is an application of AI based around the idea that we should really just be able to give machines access to data and let them learn for themselves (Source).

AI = The Concept. ML = The Application of AI. For SurveyMonkey’s purposes, we set out to use ML to provide predictions and recommendations.

The ‘Genius’ feature that started it all

In early 2017, we created a feature called ‘SurveyMonkey Genius’ which scored a customer’s survey and provided an estimate for the survey completion rate and time to complete. At the time, completion rate and time to complete were valuable insights we could confidently provide customers.

When we released the feature, it was a hit because it showed a massive 10% increase in survey deploy rate (deploy = a survey sent that returns at least 5 responses). We helped make users feel more confident, but we knew we could do even more. Our next step was to build off of this win and figure out a way to provide meaningful value throughout the whole survey creation process.

Next up, question type prediction.

Over 30% of customers already have their questions typed up even before they start using SurveyMonkey for the first time. They have an idea of what they want to ask. But SurveyMonkey has about 20 different question types (Multiple Choice, Comment Box, Matrix, Matrix of Dropdown Menus, etc.) and a lot of the time, customers don’t know which to choose. Sometimes, they might choose a question type that’s not quite right for their use case, which can affect the quality of the answers they get back.

We set out to design a feature that could recommend the right question type for you.

SurveyMonkey is able to predict the right question type with 65% accuracy.

Our team for this feature consisted of a product manager, a data scientist for a couple hours a week, and a front-end engineer. With over 16 million active users, we knew we had mountains of data about the question types they selected and the actual text customers typed into the question-text input field. The data scientist analyzed the content of the input text to develop a data model that could predict the right question type for customers.

A data model is what data scientists create by analyzing data to make a prediction or recommendation.

When we launched this in Q1 2018, we were able to predict the right question type with 65% accuracy. We reduced the time it took for customers to write a survey and increased their confidence because we provided a prediction that SurveyMonkey can stand behind.

One step further: populating answer choices

It takes time for customers to type in a 5-point answer scale like ‘Very satisfied, Satisfied, Neither satisfied nor dissatisfied, Dissatisfied, and Very dissatisfied.’ Go ahead, time yourself. I clocked in at about 47 seconds, including correcting typos. Multiply that time by about 9 questions and you’re looking at 7 minutes.

Since we’d already predicted the right question type, how might we automatically fill out answer options? This was a no-brainer, but we had limited data. We never tracked and tried to correlate question text, question type, and answer text. We needed to build a feature to create a Scale Effect to collect the data we needed to inform a data model to make confident answer recommendations. This feature would also need to deliver customer value.

How we reached the scale of data we needed

To create this Scale Effect, the data scientist gathered the top answer-scale options, about 15 in all. Early Q2 2018, we launched a feature called ‘Answer Genius’ which allowed customers to select an answer-scale from a simple drop-down list. Every time a customer selected an answer-scale from the drop-down, it was a piece of data for us. We were building our mountain of data — and it took until the end of Q2.

A Scale Effect means you need a large amount of data for Data Models to be considered confident in their predictions and recommendations (Source).

Once Q3 rolled around, the data scientist was able to design a data model that suggested a single answer-scale option that was correlated with the customer input question-text. Shortly after, we launched an ML-powered answer-prediction feature.

What if we could automatically build a survey for our customers?

SurveyMonkey has over 100+ expert-written templates for customers to use. New customers just don’t know which one is right for them. So at the end of 2018, we began working on an MVP that guides customers to select the best possible survey. We called it ‘Build it for me’ mode. This would allow us to collect the data we needed. However, it was risky because we were significantly disrupting the flow.

To build this feature, we had a content strategist and a survey researcher on our team. ‘Build it for me’ mode asks customers questions about their audience, survey goal, and use case. Based on those answers, we create the best-possible template for them. We also built a ‘Genius Assistant’ panel that makes recommendations and guides the customers through the rest of the survey-creation process. However, the ‘Genius Assistant’ was not built into the survey-sending process.