CXL Scholarship — 8th Week Review
By: Mohammad Sammak
Tags: Google Tag Manager, Statistics, Google Analytics, A B Testing, Conversion Optimization

Always track the money. Look for ways to affect the bottom line of the company. Even if you are tracking the engagement, go after those that will finally bring money to the bank account. When you are doing an A/B test, test the funnel steps that are important from this point of view.

Always consider the element of seasonality in your test. Never stop a test after a few days, just because you think one variation is the obvious winner. Users show different behaviors on different days of the week. So always take this into account that each test has to run for at least a week. But never stop a test unless you reach a confidence level.

You might need to run multiple A/B tests simultaneously. In these cases, if the pages might overlap or be parts of a funnel, make sure that the traffic has been evenly divided.

If you use Optimizely, VWO, or Monetate, you need to send your data to google analytics. It is always better to look at your data in two places and check if they match up or not.

You are running a test with one control version and multiple variations. After running the test for a few days, you see that one of the variations is performing much lower than the others. The matter of fact is that you shouldn’t stop that variation, because it might skew your data. If you want to get rid of it, first stop the test completely, remove the variation, and then start the whole test from the beginning.

The next topic is about a concept that everyone seems to know about, but in my opinion, nobody really knows what it is all about. I am talking about buyer persona, avatar, or whatever you name them. It is so scary to me that I really don’t know how to come up with them in the first place. It seems to be in the hands of a small group of elites.

Peep says that we have two things: a customer theory and a buyer persona. The theory part is about what we think we know about the customer. It also consists of our learnings about what has worked in the past, what hasn’t, and what can be done. Actually buyer persona is a subset of customer theory.

Customer theory needs to change after every test. You learn new things about your customers and change your assumptions about the customers.

Forming a good hypothesis is rather hard. You need to know what is the problem, what you think is the solution for that problem and after that, how are you going to measure the results.

A good and educated hypothesis will lead you through the test. The better the hypothesis is, the better the results would be and consequently, you will learn more after the test. All of these will lead you to form a better hypothesis again and this loop continues itself.

When there are a lot of problems that need addressing, what will you do? How do you prioritize these problems and how will you find which one is the most important one? Peep introduced three models for prioritization. The PIE (Potential, Importance, Ease) and ICE (Impact, Confidence, Ease) models were almost the same. But we had another model that is developed by CXL itself which is named as PXL.

A lot of things might skew your test results. For example we have something called history effect. In the history effect, other elements outside of your control will have an impact on your results. For example a part of your business is related to operations in the middle east, some weird country like Iran decides to run an atomic missile program and the whole world goes crazy about it!

Another thing is the instrumentation effect. You choose a test to run, but you don’t check the cross device or cross browser. The test might not work as planned on a specific device or browser and the whole thing will blow up!

The next thing is selection effect. You run a ppc campaign and get good results with it. After spending all of your ppc budget, you think that kind of result is also achievable using SEO or social media. But it doesn’t work that way.

In running an A/B test, there is some common error known as the Flicker Effect. It happens when the user sees the control version before the variation for a few seconds. This is bad news for you, because it will open your hand before the users’ eyes. There are a lot of factors that can have an effect on the flicker effect.

If you want to implement the A/B test tag using tag managers like GTM, don’t do this. This is a bad practice and is prone to result in a flicker effect.

16- Testing Strategies by Peep Laja

So, what should I test? What should I change first and then measure the effectiveness? This is a big and also important question that you need an answer to.

You know, you have to start the whole testing thing after a hypothesis. You have to know what is wrong or what might be wrong and then try to be creative and think about possible solutions to that problem.

If your landing page doesn’t convert enough traffic, maybe it is because of a weak CTA or a long page. Maybe the users get confused and can’t find her path on the page. Maybe your imagery isn’t good enough. You have to guess different things based on the problem that you know is there.

Well, I am officially confused. Peep says if you want to run A/B tests, you’d better not only change one thing on the page unless you have a website with millions of visitors. Is this rational?

He says you can change multiple items together and then look if the changes made an impact or not. If it did, then you can go back and turn off those changes one by one to see which one actually did the job.

I have always thought that in A/B testing, we are only allowed to change one thing at a time, because only then we can attribute the results to the change. But I think Peep said this because startups usually are short in time and don’t have the luxury of time bi enterprises possess.

Should I use A/B test or MVT? This is heavily dependent on the traffic volume you have. If you have a low traffic website, start with A/B tests and stick with it. But if you have a decent amount of traffic (>100,000 / month), you can think about MVT.

Most agencies run 1 MVT after every 10 A/B tests. This isn’t a general rule, but it can give you a hint.

I just learned about a notion and I just love it. It appears that we have something called a Bandit test in some testing tools that allow us to dynamically allocate bigger traffic to the variation that seems to be performing better.

In bandit tests, you don’t need to personally supervise the process. It is based on constant learning and implementation and the outlier will be awarded with more traffic automatically.

You might have a bunch of sections on your website or on a specific landing page. By using Existence Testing, you are going to test if a section is necessary to be or not. You simply delete that section off the page and run an A/B test. If the conversion rate didn’t change, it simply means that the specific section didn’t play any roles on the page.

You know what? Peep talked about two concepts that I didn’t get very much. One was Iterative testing and the other was Innovative testing. All of those concepts seemed very like the ones that I had previously known here in this course.

Right at the last part of the course which normally doesn’t convey new messages, Peep introduced a new concept called Split Path testing. In this method, you take the user to a completely different path. You can suppose that the user has gone through a whole different funnel and the designer intends to find out which path leads to more conversions.

17- Statistics for A/B Testing by Georgi Georgiev

This lesson is all about statistics, at least based on the preview I just watched. The tutor has a new accent and you have to focus to understand what he is saying, but he is likable.

He says marketers just look at the numbers and don’t know what to do with them. For example they don’t know how to do risk management or making an estimation. So are we going to get some data-related knowledge? I hope so.

To be honest, I didn’t understand what the guys talked about. He said that data needs to be used for making business decisions and this seemed understandable to me. But after that, All those mean, deviations, p-value and stuff came in and I couldn’t take my eyes off the subtitles to understand what he was talking about.

He also talked about the difference between correlation and causation. I previously knew about this topic and hence, I could barely understand the topic.

He promised that everything is going to be meaningful in the next lessons. He’d better be right, because this course has a final exam! Scary!

The next part wasn’t any different. He tried to explain something called z-score and p-value using mathematical formulas. But I swear to god I didn’t understand a single one of them.

The Only thing that I understood was the relationship between p-value and statistical significance. The smaller the p-value is, the bigger or better the statistical significance will be.

He also talkd about something called statistical intervals and I didn’t understand it either. No surprises!

Giorgi Giorgiev

You can’t believe how boring and difficult it was going through this course. I can definitely tell you it was the worst part of my experience with this whole program. I just wanted to finish it.

Believe me or not, I didn’t learn anything. It is better to tell you that it was kind of not understandable at all! So as a sign of objection, I decided no to write any points of it down.

Wrap up

I don’t know what to say. It is a saying that people tend to evaluate their experience based on the last thing they have had. So I don’t want to fall into that trap. I know that the last thing I watched was terrible in quality and ruined my experience, But now I am over it. It is gone and there are a lot of things to come. I am eager to learn more practical stuff.

Let’s move on.