Experimental Studies
By: Parveen Khurana
Tags: Data Science Training, Experimental Study, Lurking Variable, Data Science, Data Science Courses

Statistics plays a very very important role in experimental studies. All the news that we hear around the COVID vaccine being tested, and we hear about control groups and volunteers and all that, those are studies being conducted and when the results of such studies are interpreted, a lot of statistics goes into it. In this article, we discuss how to conduct such an experiment.

There are two types of studies: one is observational studies and the other is experimental studies.

In observational studies, the observer is neutral say if we are trying to measure the average height of students in a class, then as an observer, we’re not doing anything to students, we’re just going to go and measure the height so things are already happening there in nature, we’re just observing that for example observing the height and then recording it and then computing the average or variance.

Compare this to the case where we are giving some treatment to the students for example some drug which will enhance height or stamina or some other medical factor in the human body and so on. So, here is a treatment and now we’re no longer observers, we’re actually interfering in the process. We are giving certain treatment(s) and make certain observations, this is known as an experimental study.

So, in this context, an experiment means applying treatment to experimental units. In most cases, the experimental units are human subjects but we could have other experimental units as well as per the scenario.

There are certain terminologies associated with such an experiment. Let’s understand the terminology by taking an example;

It is known that certain types of nuts help to reduce cholesterol levels. Now suppose we want to systematically study this and it might be related to quantity as well so in this example/study, we take the quantity to vary between 4 to 10.

In any experiment, there is a response variable that we try to track. In this case, the response variable is cholesterol.

Experimental units, in this case, are human subjects but a very specific type of human subjects, we only want to consider heart patients.

Then there are factors(factors are things which can have an impact on the response variable) involved in an experiment, there are two factors in this case that affects the response variable: Quantity and the types of nuts, and then there are factor levels for example for quantity could be in between 4 to 10, similarly, the types of nuts could have three levels/categories(almond, cashews, walnuts).

Once we have the factors and factor levels, we have a treatment. Treatment is a combination of these factor levels for example for the ‘Quantity’ factor there are 4 different levels(say we want to take only 4, 6, 8, 10 from the 4–10 range) and for ‘type of nuts’ there are 3 possible levels, so overall there are 4*3 = 12 combinations possible as treatment.

The most important thing is to be careful of lurking variables. When conducting the study, we need to ensure that the lurking variables do not contribute to some effects or side effects and we end up making wrong conclusions.

Lurking variables, in this case, could be age, diet, exercise for example if we select the treatment to be 4 almonds but the sample contains units that consume a lot of oily food, junk food then the effect of 4 almonds will not show up. We are trying to assess the importance of one variable which is almond and its quantity but we have these lurking variables which could have an effect on the response variable which is the cholesterol level.

So, we have to make sure that when we are applying this treatment, all the groups which are getting different types of treatments have a random distribution, and people are assigned to it randomly so that we don’t end up having a situation where there was one group which were given 6 almonds and all the people in this group had a very bad diet. On the other hand, there was another group that was given 4 cashews and all of them had a very good diet which might have been contributing to the lower cholesterol levels instead of the treatment(4 cashews in this case). And we don’t want to have that. So, lurking variables are something in an experiment that we need to be sure that we are controlling for.

Let’s talk about collecting samples for treatment:

As discussed above, there are 12 treatments possible and for every treatment, we want to take a random sample from the population. So, the units are randomly assigned to each study group and that is one way of controlling for the lurking variables. Since we have done a random assignment, it is very unlikely that there was one group that was super-fit, very healthy or there was one group with all people that consumes junk food. So, randomness is one way of controlling for lurking variables.

The other way is to specifically control for the lurking variables for example we could make sure that the age distribution of people across these different groups is the same so we have the same number of young people, the same number of old people, middle-aged people in all the groups. Similarly, we need to ensure that during the treatment they all follow a very similar diet plan and have the same exercise part as well.

Population

A good treatment should follow the three basic principles

Randomization: people assigned to different treatment groups should be randomly selected, it should not be like we select a unit/person, and if that person does not like cashews so we put that unit/person in the almonds group and so on. It should be completely random and that is very important.

Repetition: The experiment should also have repetition and that’s why we don’t select a sample size of 1 for every group, we select a larger sample so say the same treatment is applied to say 100 or 200 units(sample size). So, every treatment group should have more than 1 person.

Control: we should control the effect of lurking variables. One way of doing this is by randomization, the other way is to explicitly control it by ensuring that whatever is the lurking variable that is controlled across all the groups(they have the same diet, same exercise, same sleeping pattern, and same age distribution) to conduct a good experiment otherwise the results of the experiments would not be reliable.

Another important concept when designing experiments is the Placebo effect:

Suppose a company comes up with a new treatment say a new drug for controlling the sugar level and they want to try it, so there is an experiment group and they want to try it on the group.

Placebo effect — the medicine may not be effective but just because a person is taking medicine, something is psychologically triggered in the body and the person start feeling better and even start showing results in terms of quantifiable parameters so just because a person is taking a medicine, it is possible that the stress level of the person goes down(with the rationale, satisfaction that at least the person is taking some initiative, action in terms of doing something about the problem(say sugar problem) and since the person is doing that maybe he is taking less stress about it and just because the person is taking less stress, the sugar level might actually improve even though the treatment was just blank pill, nothing medically sound in that pill).

Now it is possible that the people are showing improvement just because of the placebo effect and not because of the actual medicine. So, in such cases, we need a baseline to compare with and that is known as the control group.

So, whenever we do an experiment, the actual treatment is provided to one group and an empty pill(which looks exactly like the actual pill given to the experimental group, and people in the control group should not know that they are in the control group) is provided to the control group. And now if we see a drastic difference between the medical parameters(with all the statistics, hypothesis) of the two groups then we can say with surety if the treatment is effective or not.

Both groups must have some sort of treatment for the lurking variables.

There are two types possible here:

Single-blind experiment — In this type of experiment, the subjects/units are not made aware of if they belong to the control group or the experimental group. This is the default choice when conducting an experiment.

Double-blind experiment — Here both the experimenter as well as the subjects do not know which group is a control group or which one is an experimental group.

So, this is what a randomized control experiment looks like. The idea here is to look at certain parameters of these two groups and then come up with a hypothesis that distinguishes the two samples and to prove it statistically or argue the same using hypothesis testing.

This is the process of collecting the data/assigning subjects for different groups and controlling the effects of lurking variables for an experiment, if the data collection process is flawed then the downstream conclusion(using inferential statistics) that we draw will also be flawed.