How to avoid frustration and stay happy in your data science job
By: Martin Šiklar
Tags: Machine Learning, Efficiency, Happiness, Self Improvement, Data Science

In my experience, a good baseline usually already performs at about 90% of the best possible model (depending on the use case of course). In my opinion, getting to the 90% fast with an easy solution that is interpretable to me is more impressive than spending days or weeks fine-tuning a neural network that is hard to explain, slower to run and only performs 2% better.

Once you have a baseline established, you have something to present to your colleagues and clients. You will have a better understanding of the data, the underlying problem and it will take some pressure off you because you already have a first working solution! Great! Try to get some feedback from your colleagues and clients, to see how this compares to their experiences and expectations. These insights will prove themselves to be very valuable in the next iteration when you will try to improve your model. When your colleagues or clients do not have any valuable input, try to find literature and compare your first results to existing solutions. Before you start working on an improved version of the model, make sure you are working with the right amount of data (read below)!

Subset your data

Another pit I dug myself a couple of times in the past was, to perform my tests and modelling on the entire data set. No, I am not talking about the importance of training and test set here, but about the importance of downsizing! Having a lot of data (is usually) a great position to be in as a data scientist, but when you start working on your problem, don’t make the mistake to use all of it in the beginning.

It’s a trap — taken from giphy.com

Every record in your data set, makes each iteration of improvement and test run slower. Start off, by setting up your machine learning workflow and model architecture on as few data records as possible, just to have a running proof of concept, even if the model performance (in terms of accuracy) is poor!

Once your pipeline is working, increase your data set size continuously and see if the model performance is increasing. If this is not the case, then very likely your pipeline has some unforeseen errors. Not working with the whole data set, will let you iterate and debug much faster!

Subset your training set into a manageable small dataset. If you work with structured tabular data, 30 samples per class are enough to get you started. If you work with structured data such as images, maybe even 1 or 2 images can be enough to be able to set up a proof of concept.

Gradually add complexity

Photo by Timo Volz on Unsplash

Once you have your baseline, it is time to gradually improve your model! The most important thing is, do this gradually! Do not change too many parameters at once! This one is really hard to follow, I know! To be honest, I hardly ever adhere to this principle myself, but here is why we should:

While we are moving to more and more complex models, it is important to understand why the increased complexity works!

More complexity means more moving parts. The more changes we make at once, the harder it is to keep track of what change actually caused the improvement! “Was it the new feature I added or the hyperparameter I adjusted?”

By changing too much too fast, we will likely end up with a sort of black box that seems to work just fine but simultaneous lose explainability which hinders future improvement. A strategy that many data scientist seem to follow, is to increase the model's complexity and parameters until the model starts overfitting. Then, they pull the handbrake by implementing some kind of regularization to prevent this. We should however always strive to understand how the complexity we added improved the model and if it is really needed.

I found a quote that sums this up on the MLCMU website:

“When designing a new model, we should be constantly aware that we are not building a complex model for the sake of its complexity but for its expressiveness. Our inability to address this issue on both the intuitive and theoretical level is an intellectual debt that we will eventually need to pay off.”

Prioritize what to work on

Photo by Andrew Seaman on Unsplash

In literature, this is often referred to as “finding the bottleneck”. Machine learning models usually have a lot of screws that we can use to tune and improve performance. Due to a lack of understanding or wrong prioritization, however, many Data Scientists focus on the wrong parameters to tune and waste a lot of time improving the wrong thing.

The funniest thing however is, that often we know that the screw we are tuning is not the most important one, but we keep fiddling around with (unimportant) details, just to keep ourselves busy so we don’t have to engage with the bigger problems.

So how do we find this one magic screw? One way is, to avoid overly complex workflows (see the chapter above). When our model is simple we usually have a good understanding of why it is not working and what to do to improve its performance. However, when things get more complex, we lose this overview and start guessing, turning, pushing and pulling with the expectation that something will eventually happen that will solve our problems! Try to avoid this “technique” since it very often results in frustration and an even more complex model. Instead, try to spend some time on error analysis!

Error analysis

When you don’t know why your model is performing poorly, try to look into the data points that were misclassified by your model (for classification) or show the largest residuals (for regression). Sometimes these observations are outliers but very often (if you look at enough data points) you will observe a pattern that will give you a better understanding of what's happening: Maybe the misclassified data points all belong to the same class? If you have multiple data sources, maybe they all come from a certain sensor that returned flawed data points? Maybe they show more noise than the others?

Furthermore, it is important to know, how well a human would perform on these tasks or more generally what the maximum expected accuracy (e.g. based on literature) for this task is. If you have a reference on what's theoretically possible in terms of accuracy, it will help you prioritize correctly.

Photo by Colton Sturgeon on Unsplash

Take-home messages for a happier life as a Data Scientist

Start with a simple model as a baseline. This helps to understand the data better and gives you a first feeling of accomplishment.

Don’t be afraid to use simple methods to get you started. They are a great benchmark for future improvements.

Gradually progress towards more complexity. Only change one parameter at a time.

Do not add unnecessary complexity. Only add complexity if you can explain why it is needed.

Subset your data, this enables you to iterate over your model more quickly and fix bugs.

Prioritize what to work on!

Analyze your model's performance and assess priorities by thorough error analysis.

Further material: