7 Tips for a Successful Data Science Project in the Industry
By: Aviad Atlas
Tags: Office Hours, Tech, Data Science, Machine Learning, Industry

1. Define the problem and the goal

Manager: “I need you to build a model that will identify bad data flowing in the system.” Me: “Hmm, sure, no problem. Just let me know in which DB the data is located, and I’ll get right to it.”

But what exactly is “bad data”? Why do we want to filter it out? What is the business value in this project? It’s important to define what the problem is exactly, and what is our goal. What is the cost of a False-Positive in this case? Is it equivalent to a False-Negative? Ask all questions needed in order to reach a clear target, and create a clear metric for your project, preferably one which is a single value. It’s a shame if you perform a whole cycle of building a model, only to find out that you didn’t fully comprehend what the problem actually was. You might not exactly be back to square one, but that was a lot of effort spent in the wrong direction.

There are some cases where you’re dependant on other personnel in your company to fully understand the project. Don’t procrastinate, move forward toward a clear definition of what is required in your project.

Since we’re discussing the beginning of the project, I’ll add another note related to this stage of a project. Your manager doesn’t necessarily understand the difficulty of the problem at hand. It’s important to reflect an initial outcome of the project (what accuracy you will reach, etc.). You'll probably be off a bit, but when you come back with results, you prefer not to get an “oh…” reaction. Be aligned from the beginning.

2. Fully understand the data

Manager: “There are missing values in the label column. Those indicate that the annotater didn’t find anything in the image.”

That sounds logical. But turns out that missing values also indicate cases that the annotator simply couldn’t decide which of the 6 possible labels is the most accurate one, so he left it empty. These two scenarios are completely different, yet if both types of missing values are treated the same, this will harm your trained model.

It’s crucial to make sure you understand your data well. You might engineer some feature, and later find out that it was engineered wrongly (e.g. you took a “zip code” column and bucketed it to different regions in a certain country, but turns out that not all the zip codes belong to that same specific country).

Whether it’s the label column, or a column you’ll use as a feature, make sure that your understanding of the data is as accurate as you want your model to be. You can even consult the annotators themselves or their managers. Whatever it takes.

3. What will be the situation in production?

Manager: “The relevant data is in table1, table2, table3, and table4. A simple join will get you all the data.”

Cool, you got everything you need. You move forward with your model, get great results, and you’re ready to deploy it to production. You just need to make final adjustments.

Me: “Say, which source provides field A?” Manager: “Field A? That’s some enrichment we do to the data once it finished the full flow in the system.”

Oops, that’s not good. It turns out this is even a feature that doesn’t add too much to the model’s accuracy. But your model was already trained using it. Make sure you understand what will be the situation when your model will be in production, so your model will have all the data it needs, and it will be suitable for that specific setting in production.

4. Keep it simple

Me: “I think I’ll try the super-duper-really-cool-model first. What do you think?” Manager: “You’re the one with the knowledge. Go for it!”

As said, generally your goal is to bring results relatively quickly. Your manager doesn’t always have a sense of what the possible solutions might be. Usually, simple solutions are more valuable when the time invested is taken into account. And very often they’ll even perform better than the super complex solution you think will be cool here, regardless of the time invested. It’s, of course, not always the case, but at least first try the more “simpler” solutions before moving on to the “cool” ones.

In addition, very often it’s possible to solve the problem at least partially with a simple solution. When doing that, you’re already creating value. Once done, you can move forward to the more complex solutions, if still needed.

5. Consult others

Me: “Yeah, I tried algorithm A. It looks like a good solution here.” Colleague: “That’s interesting, it might work well here. Though I think I would try algorithm B here.” Me: “Huh… Didn’t think of that. Good idea!”

Hopefully you’re not a single Data Scientist in your company. If so, it’s a great opportunity to learn from others and hear how they would tackle your problem. Don’t feel uncomfortable. There’s no one who can come up with all possible solutions, there’s no one that knows everything. Besides, very often, as a result of brainstorming together, you’ll come up with an idea yourself that you wouldn’t think about without this consultation.

With regards to this topic, I would also add the importance of planning your project and breaking your work into tasks. The outcome of a group brainstorm should be just that. This will enable you to stay focused and not get caught up too much researching a specific algorithm that is a potential solution.

6. Error Analysis

Manager: “I wonder why the model performs well for scenarios A and B, but not for scenario C.” Me: “I think the other algorithm will work better. And also training on more data will probably do the trick.”

Yeah, this is probably not what you were dreaming about when you landed this job. But it’s extremely important. Looking into several (or many) mistakes manually can give you a good intuition about what’s wrong. Don’t feel as if this “isn’t something I should be doing”. This will probably be more effective than just throwing more data at your model.

7. Documentation

Me: “There was a reason I filtered out those rows from the data. I can’t remember why I did that…”

This is sometimes a step that’s neglected, or done only partially. The current project you’re working on has elements that you’ll forget about next month when you move on to your next project. Document what steps you took throughout the project, what data was selected and why, the queries you ran on your DB (and your code, of course!). The more, the better. This will be extremely useful when you might have to revise the project and refine your solution, or perhaps when you will be working on a different project that is related to this one. You can even benefit from documentation while still working on the project (“why did I decide to take route A instead of route B again..?”).