Experimental Design in Data Science | by Benedict Neo | Towards Data Science

Full seriesPart 1 - What is Data Science, Big data and the Data Science processPart 2 - The origin of R, why use R, R vs Python and resources to learnPart 3 - Version Control, Git & GitHub and best practices for sharing code.Part 4 - The 6 types of Data AnalysisPart 5 - The ability to design experiments to answer your Ds questionsPart 6 - P-value & P-hacking Part 7 - Big Data, it's benefits, challenges, and future
This series is based on the Data Science Specialization offered by John Hopkins University on Coursera. The articles in this series are notes based on the course, with additional research and topics for my own learning purposes. For the first course, Data Scientist Toolbox, the notes will be separated into 7 parts. Notes on the series can also be found here.
Asking the right questions before solving a problem is a great start, it points you towards the right direction in the midst of complexity. But to really solve your problems effectively, you also need proper planning and good experimental design. Similar to conducting experiments in the scientific method, which entails concise series of steps to follow for conducting a safe experiment — the name of the equipment and measurements for certain things, you’ll also have to design your Data Science experiments. Then only can a clear path be paved towards breaking down your problem part by part, meticulously, and with the proper methodology? Here’s more about experimental design.
It generally means organizing an experiment so that you have the right data to effectively answer your DS questions.
Before you begin solving your DS problems, a few problems can arise.
What’s the best way to answer the question?
What are you measuring and how?
What is the right data?
How can I collect that data?
What tools and libraries do I use?
etc...
All these problems are a crucial part of a DS experiment and have to be answered right from the start. Without proper planning and designing, you’ll be facing many of these issues throughout your workflow, making it unproductive.
So what’s the right flow in an experiment?
1. Formulate question
2. Design experiment
3. Identify problems and sources of error
4. Collect data
The process starts with clearly formulating your questions before any data collection, then designing the best set-up possible to gather the data to answer your questions, identifying problems or sources of error in your design, and only then, collecting data.
Bad Data → Wrong analysis → Wrong conclusions
Erroneous conclusions can have sweeping effects that have a trickle effect (citations in papers eventually is applied in real medicinal cases)
For studies that have high stakes, such as determining cancer patients' treatment plans, experimental design is crucial.
Papers with bad data and wrong conclusions will be retracted and have a bad reputation.
The variable that is manipulated, not dependent on other variables
Variable expected to change as a result of changes in the independent variable
An educated guess as to the relationship between variables and outcome of the experiment
Here’s an example of experimental design in action testing the correlation between books read and literacy.
Hypothesis: As books read increases, literacy also increases
X-axis: Books read
Y-axis: Literacy
Experimental set-up: I hypothesize the literacy level depends on books read
Design of experiment — Measure books read and literacy of 100 individuals
Sample size (n) — Number of experimental subjects included in the experiment
Before collecting data to test your hypothesis, you first have to consider the problems that can cause errors in your result, one of them being a confounder.
extraneous variables that may affect the relationships between dependent and independent variables
For this example, a confounder can be the age variable, because it can both affect the number of books read and literacy. Any relationship between books read and literacy may be caused by age.
Hypothesis: Books read → Literacy
Confounder: Books read ← Age → Literacy
Having confounders in mind, and designing your experiment that controls those confounders is crucial so that it won’t affect your results, and that you’re designing your experiments correctly.
There are three ways you can deal with confounders:
ControlRandomizationReplication
Control
Randomization
Replication
Going back to the book-literacy experiment, to control for the effects of age on the result, we can measure the age of each individual to take into account the effects of age on literacy.
This creates two groups in your study :
(1) Control group & (2) Treatment group, where:
control group → participants of fixed ages
treatment group →participants of the range of ages.
results are compared, know whether the fixed age group differs in literacy from the group of various ages.
Another general example is with drug testing, to test the effect of a drug on patients, a control group won’t receive the drugs, while the treatment group is given the drugs, and the effects are compared.
This is a bias where a person believes psychologically that a certain treatment is positively affecting them, even though no treatment was given at all. This is commonly seen in medicine, where a placebo drug can substitute the effects of a real drug.
To tackle this bias, subjects would be blinded, meaning they won’t know what group they are assigned to:
Control group with mock treatment (sugar pill but told drug)
treatment group with the actual treatment
if placebo present, both groups will experience it equally
Randomization basically assigns individuals randomly to distinct groups, this is great for two reasons, (1) you don’t know the confounder variables, and (2) it lessens the risk of biasing one group to be enriched for confounder
Taking the drug testing example
A number of subjects sample, randomization was done
Distribute participants over two main groups, one confounded and the other randomized, each with their own control groups and treatment group
It’s from the randomized group that you can tell whether there are biases present.
Replication is basically repeating your experiment, but this time with different subjects, this is very important, as it shows the reproducibility of your experiment.
Replication is also necessary mainly because conducting just one experiment might be of chance, a result of many factors, such as:
confounders being unevenly distributed
systemic error in data collection
outliers
If replication is done(with a new set of data) and it produces the same conclusions, this shows that the experiment is strong and has a good design.
Moreover, the heart of replication is all about the variability of data, which relates to p-value, which has a golden rule of (P ≤ 0.05) that many strive for in statistical hypothesis testing. More about that in the next article
Building a city like New York required blueprints and very precise planning, which created what you see today. Designing experiments in Data Science should be the same.
This is the basics of experimental design, which is fundamentally about precise planning and design to ensure that you have the appropriate data and design for your analysis or studies so that erroneous conclusions can be prevented.
A cool mnemonic for the flow is QDPC- Question Data Plan Carefully
Formulating questions → Designing experiment → Identifying problems and errors → Collecting data.
Erroneous conclusions will have trickle effects as they’re cited and used in real-life applications especially in medicine.
Independent variable (x-axis) — manipulated and not affected by other variables
Dependent variable (y-axis) — change in x results in y
Hypothesis — Educated guess about the relationship between X and Y
Confounder — Extraneous variable that affects the relationship between independent and dependent
Control — fixing confounder in an experiment or taking into account and measuring
Control groups — a group of subjects that have an independent variable measured but don’t receive treatment. Acts as a control for the experiment, compare changes and effect
Blinded subjects — blindly assigned participants to groups to test the placebo effect
Placebo effect — One’s belief of an effect created by a placebo treatment
Randomization — randomly assign participants to present biases and confounding effects on group
Replication — repeating the experiment to strengthens the conclusion of the result
FiveThirtyEight
xkcd comic
yale statistics experimental design
medium.com
towardsdatascience.com
towardsdatascience.com
medium.com
towardsdatascience.com
If you want to be updated with my latest articles follow me on Medium.
Follow my other social profiles too!
Linkedin
Twitter
GitHub
Reddit
Be on the lookout for my next article and remember to stay safe!