Writer in tech? Here is why you should run A/B tests
By: Bar Zukerman
Tags: UX, Marketing, Content, A B Testing, Product Management

By Amelia Bartlett

The role of the writer in tech has evolved. Today, writers are expected to be more than just excellent writers who can whip up witty, cool, or professional text. They must utilize their craft in a way that aligns, advances, and achieves the company’s strategic business goals.

They receive briefs for blog posts, ad campaigns, scripts, and user experience flows that are backed with data and have clear business goals and KPIs. In their writing, they should integrate strategic messaging, mobilize users to take action, and customers to develop positive emotions and opinions towards the brand and product they are writing about.

A seasoned writer in tech isn’t just an excellent writer. They understand user experience. They know how to conduct customer and competitor research. They are familiar with business KPIs and how to positively impact them using words. They know what makes for a successful marketing campaign and effective product flow. They are a lot more than just writers.

Managers in tech companies looking to hire writers already look for these qualities in their top candidates. Candidate tests and assignments must showcase a product, UX, and business understanding. In interviews, writers will be asked questions like “how would you rephrase this sentence to increase CTR? And how would you change it to decrease churn?”.

An effective tool that helps turn writers into more than just writers, but also masters of product, marketing, and business is content A/B tests. Unfortunately, it’s a tool that isn’t used enough by content professionals. Product professionals for example, never release a feature without experimenting with at least 2 versions. So why is content any different?

There are some possible reasons why writers in tech aren’t running any, or as many, A/B tests as they should:

It might be “above their pay grade”

They aren’t necessarily numbers people

They don’t have the time

It can be technically challenging

It’s hard to get others on board

And despite that, there are at least 7 reasons why it’s worth it for any writer in tech to try.

#1: It’s really cheap

Planning and executing a product A/B test that involves varying product capabilities and UIs can take weeks and requires effort from Product Managers, UX Designers, Developers, and Analysts. Changing the wording on a button or a title can take as little as a few hours from the moment of ideation to execution. In some companies, such experiments can even be launched without any Dev efforts. Netflix, for example, developed Shakespeare, a tool that lets writers run light copy tests without heavy engineering support.

Content A/B tests require minimal resources, but their impact potential doesn’t fall short of other types of experiments (we’ll talk numbers in a bit).

#2: You’ll build strong relationships with your colleagues

If you only focus on writing, you’ll always be left out of the conversation. In tech companies, stakeholders work together as a team to build products, launch campaigns, and close business deals. If you fail to see the big picture and use your craft to help the team achieve its common business goal, you’ll simply single yourself out and you won’t be valued as much by your colleagues.

#3: You’ll contribute to the company’s business goals

Take time to understand your company’s business model and product. Break down the user journey and find out what KPIs matter at every step. Talk to users, read testimonials, talk to product managers and salespeople. Immediately, you’ll start getting ideas for how you can optimize existing content to improve performance.

Let’s take a look at this example from Hubspot. Knowing that users who engage with search have a 163.8% higher blog lead conversion rate, the company set out to improve user engagement with the search bar which at the time of testing was very low. Variant C showed a 6.46% increase in users who engage in the search bar.

Variant A

2. Variant B (same text: “search by topic”) with different UI.

3. Variant C

#4: You’ll improve the user/customer experience

There isn’t a product in the world that is 100% frictionless. There is always a moment in the flow where users get confused, frustrated, or discouraged. Explaining complex information with simple words or making sure users understand what a certain button does, can significantly minimize negative emotions in users and contribute to a more pleasant experience.

This example is from a Netflix landing page with multiple buttons that lead to the same page. The version on the left is a landing page with 2 buttons that contain the same text: “JOIN FREE FOR A MONTH”. The version on the right shows 4 buttons with different text: 1 — “JOIN FREE FOR A MONTH”, 2 — “TAKE US ANYWHERE”, 3 — “TRY WITHOUT COMMITTING”, 4 — “WATCH IT ALL”. Users responded better to the version on the left. The assumption is that consistent text reduces confusion.

This example is from Jakub Linowski’s blog post on Good UI.

#5: It’s a sign of true ownership

The role of the writer doesn’t end when a blog post or ad campaign is published nor when a new product feature is launched. Tracking the performance of your content and optimizing it when needed is a sign of true ownership of the content you’ve written. You don’t just write it, send it off to Dev/Design and then forget about it. You care about whether or not it achieves the business goal it was meant to achieve. If it doesn’t, you get back to the drawing board and come up with ideas for how to optimize it. With A/B tests, you can be confident that your changes were effective.

#6: You’ll make science-backed decisions

Writing is an iterative process. You can always make more edits. Every time you come back to your text you’ll notice something you want to change. When we write with a business goal in mind, it’s hard to narrow down our options to the best one. How can we know which of our options will compel more users to take action? A/B tests are the only way of getting a science-backed answer.

For this reason, A/B tests are an integral part of the culture at Booking.com: “it removes the hierarchical nature of decision making by creating a focus on data-driven decisions.” Here’s an example of an A/B test that helped Booking.com optimize and choose the right text for its notifications screen (B won).

This example is taken from a blog post by Hadas Harush from Booking.com’s product blog.

#7: You’ll become a better writer

The more A/B tests you launch, you’ll start collecting data about the voice & tone and terminology that work best for your product, brand, target audience, and various steps in the user journey. You will also understand what type of content doesn’t work. Naturally, in your writing process, you will start implementing these lessons.

Did I miss anything? If you have more good reasons for writers in tech to run A/B tests, please share them below.