6 bits of advice for Data Scientists
By: Rahul Agarwal
Tags: Machine Learning, Artificial Intelligence, Data Science, Towards Data Science, Data Visualization

To err is human.

And who is more human than us data scientists when measured by that metric.

The important thing is to look at our mistakes. And learn from them.

A data scientist needs to be critical and always on a lookout of something that others miss.

But sometimes in our day to day job and coding perse, we get lost in our train of thought and fail to look at the overall picture.

In the end, our business partners have only hired us to generate value, and we won’t be able to generate value unless we develop business critical thinking.

So here is some advice that one can include in the day to day data science work to be more diligent and more impactful at the same time.

1. Beware of the Clean Data Syndrome

How many times it happens that we start working straight on the data we get. Start creating models? Or even present descriptive analytics generated automatically to our business counterparts?

But, do you ever ask — Does this data make sense?

Falsely assuming that the data is clean could lead you towards wrong Hypotheses.

You actually can discern a lot of important patterns by looking at discrepancies in the data.

For example, if you notice that a particular column has more than 50% values missing, one might think about dropping the column.

But what if some data collection instrument has some error? You could have helped the business to improve the process.

Or let us say you have a distribution of Male vs. Female as 90:10 in a Female Cosmetic business.

One may assume clean data and show the results as it is or they can use common sense and ask their business partner if the labels are switched.

2. Be Aware

We all know fab.com. For those who don’t, it is a website that sells “CURATED HEALTH, FITNESS & WELLNESS PRODUCTS”.

But it was not always so.

Fab.com started up as fabulis.com, a site to help gay men meet people.

One of the site’s popular features was the “Gay deal of the Day”.

One day the deal was for Hamburgers — and half of the buyers were women. Why were women on the site?

The above fact caused the data team to realize that there was a market for selling goods to women. So Fabulis.com changed its business model to fab.com as a sale site for designer products.

Be on the lookout for something out of the obvious. Be ready to ask questions. If you find something you may have hit gold.

Data can help a business to optimize revenue, but sometimes data has the power of changing the direction of the company as well.

Another example, Flickr started up as a Multiplayer game. Only when the founders noticed that people were using it as a photo upload service, did they pivot to a photo sharing app.

There are countless examples as such.

Try to make a good example out of your company’s business as well.

3. Start Focusing on the right metrics

What do we want to optimize for?

Most of the businesses fail to answer this simple question.

Every business problem is a little different, and it should be optimized differently.

For example, A website owner might ask you to optimize for active users. But is it the right metric? It is just a vanity metric, which will always increase.

I would instead try to optimize the percentage of users that are active to know how my product is performing.

Another Example, We all have created classification models. A lot of time we have tried to increase accuracy for our models. But do we want accuracy as a metric of our model performance?

What if we are predicting the number of asteroids that will hit the earth.

Just say zero all the time. And you will be 99% accurate. My model can be reasonably accurate, but not at all valuable. A better metric would be the F score.

Designing a Data Science project is much more important than the modeling itself.

4. Statistics Lie sometimes, maybe a lot of times