5 Tips to Succeed in Delivering a Data Science Project
By: Carlos Rosas
Tags: Team Collaboration, Data Science, Project Management, Volunteering, Best Practices

5 Tips to Succeed in Delivering a Data Science Project to the Client Carlos Rosas Follow Apr 10 · 8 min read

In this article, we will take you through the lessons learned and tips to avoid falling into commonly overlooked pitfalls in data science projects. How to avoid them and deliver your project in a professional way? Here are 5 lessons we learned during our 8 weeks project at Omdena.com.

Authors: Salma Nasir, Carlos Rosas

Photo by Doran Erickson on Unsplash

We recently participated in the Omdena AI Project “Accelerating Not-For-Profit Funding to Facilitate Positive Change” in collaboration with the social enterprise “Our Community”.

The purpose of the project was to build an AI solution to find, monitor, and provide real-time analytics for grants in Australia, New Zealand, UK, the USA, Canada, and South Africa.

1. Bottom-up approach — Don’t put the cart before the horse

A typical data science project would follow these steps

Source: Omdena

Invest adequate time to understand the problem at hand rather than focus on what the end solution would look like and what models to build or BI tools to use.

A correct understanding of the scope of work of the project is the most critical step to organize and plan your project. It will also help you to define the right data you need to collect to reach the goal of the project.

Where to start?

An initial meeting with your client to go through the scope of work is critical for the project to identify their needs, expectations and ultimately understand the problem to be solved.

The success of a data science project requires you to constantly check your scope of work to make sure that the solution you are working on is aligned to the problem of your client.

Iterate…This is how we did it

During the course of the project, we revisited the scope of work several times. This process encouraged the active participation of all team members whose experience and knowledge in different fields such as project management, data analysis, software development, natural language processing, user experience, or economics helped to understand the problem better.

Photo by Paul Skorupskas on Unsplash

This multi-perspective review (technical, user experience, quality of the data) helped us to track progress in reaching the different goals of the project. It also allowed us to realize that sometimes we were working on things that had not been required by our client.

Our first lesson learned: Focus on delivering things required by your client.

2. Understand your data

Using web scraping, our team managed to collect a large amount of structured and unstructured data from different public and private organizations to build the main dataset we used in the solution for our customers.

With such a huge amount of data, it is crucial to ensure all team members can see the big picture and understand the data structure and attributes before rolling sleeves and diving into the preprocessing and analysis phase.

Ensure Team Alignment…

Whether working in a small or big team it’s beneficial to review the definition of each variable. In our case, none of the team members had experience in funding activities and therefore, one of our biggest challenges was to use the collected information the right way.

Photo by You X Ventures on Unsplash

We were lucky with the commitment of “Our Community” representatives to the project. At the beginning of the project, they provided us with a glossary with the main terms used in funding activity.

…And Keep Updating the documents as you go

We used this document as a basis to create a list of the minimum variables or attributes to be included in the CSV files generated from the web scraping process. These guidelines gave our team direction in the data collection process.

The team also made the effort of using the names of the variables correctly in our daily communications. This exercise helped us to identify if any of our team members had difficulties understanding a concept.

When we released the final version of the dataset, we also drafted a README document with the definitions of each of the variables included. We also went through this list in one of our weekly meetings to ensure everyone was on the same page.

Source: Omdena

Don’t forget to verify your definitions with your client; at the end of the day, they will be the main user of the solution you provide. As mentioned, we received a lot of support and guidance from “Our Community”. They helped us to validate definitions of the dataset’s attributes and suggested changes to improve the clarity of the concepts.

Our second lesson learned: Time spent getting acquainted with your dataset is a time well spent.

3. Adapt your data

Building a dataset is an ongoing process in which you may need to redefine the existing attributes or add new ones. Questioning the data will help you to define the next steps to follow.

One of the questions to be answered by our solution was to estimate the total annual grant volume in each of the 6 countries of interest. We initially expressed all the grant amounts in USD. However, our client flagged that the results were expected to be analyzed on a country level and, therefore, expressing the results in local currencies made more sense.

Play the devil’s advocate

Data analysis is a must-have skill in data science projects. The results of your solution or model depend on the quality of the data you provide. For this reason, it is critical you analyze each of the attributes of your dataset via exploratory data analysis (EDA) to verify that you have the right information to answer the business problem of your client.

We criticized our data and we realized the need of creating new variables using the current data. We made this decision in order to be able to answer the questions set up in the scope of work.

It’s also important to pay attention to the impact of feature engineering on the overall dataset. We used feature engineering to create new variables from existing ones.

Keep an eye on data integrity

Our team used a tool provided by “Our Community” named “CLASSIE”, an acronym for “Classification of Social Sector Initiatives and Entities”.

The purpose of using this tool was to classify the different grants collected by our team using the taxonomy developed by “Our Community”. This step delivered the information collected in the format required by our client. As a result of the application of CLASSIE to our data, we created new attributes such as :

Subject: the sector that a grant fits into.

Population: the beneficiary group of the grant.

Activities: activities supported by the grant.

After applying CLASSIE to our data, we realized that some grants fell under multiple subjects (e.g. health, education ). As our customer was interested in having visibility per category we exploded the dataset to accommodate for that.

In order to maintain the integrity of our data, we had to create new variables to avoid having false information regarding the counts of grants (by adding unique identifiers). We also split the amount awarded per grant category to avoid misleading information of overall value that could result from summation.

Source: Omdena

Our third lesson learned: Make sure that your dataset is able to provide answers to the questions set in the scope of work of your project.

4. Keep an Active communication with your Customer

You may be a Data Science expert, but your customer is the SME (Subject Matter Expert) in the project. Working closely with your customer will drive your project in the right direction.

During the development of the project, we had a continuous communication flow with our clients to verify that our solution was adapted to their needs.

Active communication with your customer will help you to understand their needs beyond the written scope of work. You also will learn about how they expect to use your solution. In order to maintain active communication, we used weekly update meetings, messages on slack channels, and ad hoc meetings.

Agree on a Communication plan

During weekly meetings, the team presented concise and clear updates for what’s accomplished and what’s planned to be done during the upcoming week; discussed challenges and required support/clarifications as well as setting priorities.

We also get to communicate with customers by posting direct and specific questions, get their feedback on results and arrange for short ad hoc meetings when required to further elaborate on certain points. Presentation skills, storytelling, and effective listening are key skills data scientists need to acquire and develop over time.

Our fourth lesson learned: Fluent communication with your customer is critical to ensure that your solution is built in the right direction.

5. Build a Customer-centric Solution

As data scientists, we are often tempted with creating nice charts, tables, or maps. However, the selection of visualizations and tools should be designed based on your client’s needs.

Before delivering your final solution, revisit your problem statement and verify which goals have (not) been accomplished. Then allow your client to iterate with your solution, this is the optimum way to allow them to validate your solution and suggest any changes.

What does our customer want?

At the final stages of the project when the team started building the dashboard, several ideas and approaches were pitched. The debate was easily settled when we put ourselves in the customer’s shoes. Our solution was a dashboard where our client was able to navigate from the most general information to the most specific details.

Starting from a global view for grants and per country, then filtering using any attribute e.g. per subject, beneficiary groups or grantee country, etc. Also, we included a section that illustrates a customized detailed report for the “GrantMakers” as per the preselected filters from other dashboards.

Source: Omdena

Our final lesson learned: Don’t forget that your customer is the end-user of your solution.

Wrapping up…

Data science projects are much more beyond building a certain model, applying an algorithm, or creating fancy visualizations. Whether working solo or in a team, think of it more as tailoring solutions that address specific problems or fulfill a customer’s need, leveraging both technical and soft skills.

Adapting our mindset around this idea, and iteratively applying the lessons learned in each subsequent part of the project helped us to improve our skills and performance as individuals and most importantly as a team.