What I Learned After One Year Working As A Data Scientist
By: Roberto Sannazzaro
Tags: Business Science, Data Science, Data, Technology, Programming

What I Learned After One Year Working As A Data Scientist

It turns out there are many things I wished I knew before…

Photo by Ruslan Burlaka from Pexels

Intro

In October 2019 started my journey as a data scientist. Here I want to share my experience, tips, and things I wished I knew before to avoid you getting burned.

At the beginning of October, I will turn one year working as a data scientist at Evo. Creating awesome tools with cutting-edge technologies applied to the retail and supply chain industry.

This year has been an incredible rollercoaster, I have learned a lot of things, and I consider myself very lucky to be part of an awesome team, which strives for perfection while being helpful, motivating, and competent.

You are the Average of the 5 People You Spend the Most Time With.

Well, I don’t know if it’s true or not, but if true I am definitely going for it.

One of the things that scared me the most was when the pandemic (and when the lockdown) hit: I was afraid I would not be able to deliver the same way as before, since the lack of the office environment, colleagues, and the possibility to ask for help to the colleague next to you simply disappeared from one day to another.

Photo by cottonbro from Pexels

Luckily, as often, it turned out it was just a matter of getting used to it. As often we are naturally scared of new things, however, once we fall into the water either we (try to) swim or we sink.

What I wished I knew before

Doing Data Science outside the academia or, to be clear, in a professional environment has many differences that not always are taught or explained:

In a professional environment, you are not focusing on a single technology (eg. Python), rather you are in the middle of an ensemble of different technologies (SQL, Docker, R, Python, Bash…).

focusing on a technology (eg. Python), rather you are in the middle of an of different (SQL, Docker, R, Python, Bash…). Sometimes the old spreadsheet is the way: managers are not always acquainted with development tools such as pandas , dplyr , or matplotlib .

Photo by energepic.com from Pexels

You have to be able to demonstrate and measure everything: sometimes you have tasks you need to work alone, and your teammates do not know much about the task, or even the technology you involved in solving the task. Be ready to write down results, and be able to deliver reports measuring the impact of your work.

and everything: sometimes you have tasks you need to work alone, and your teammates do not know much about the task, or even the technology you involved in solving the task. Be ready to write down results, and be able to deliver reports measuring the impact of your work. Machine learning is not a black-box : if delivering a solution that includes some machine learning running in the back, you must able to explain and understand each prediction created by the model.

: if delivering a solution that includes some machine learning running in the back, you must able to and each prediction created by the model. Everyone has its own opinions and points of view: be able to listen and respect other people’s views, and always be ready to change your mind.

and respect other people’s views, and be ready to change your mind. Very often you need to take a step back: when working on a single feature of a product it is very easy to get overwhelmed and lose the focus on the project. This is very dangerous. Keep always in mind why you are working on a particular thing and consider it always as a piece of a puzzle. Not a standalone entity.

Photo by Ivan Bertolazzi from Pexels

Do not be a fanboy: we all have our favorite technologies, libraries, cloud providers, or frameworks, however sometimes, we won’t have the possibility to work with our beloved tools, hence be ready and embrace the situations in which you need to use (and learn from scratch) a different stack.

Remember, as a junior, you don’t have to know everything about a little, rather a little about everything!

You will very often need to do different operations using spreadsheets , last but not least: spreadsheets are (still) often the to-go reporting tool. Get comfortable with operations such as pivot tables, VLOOKUP , IF , creating (and formatting) charts .

, last but not least: spreadsheets are (still) often the to-go tool. Get comfortable with operations such as tables, , , creating (and formatting) . SQL: I could not mention how important is to (at least) know the basics of SQL. Data is very often stored in databases meaning that being familiar with SQL is a must. Get comfortable with joins, select statements, group by with least one SQL flavor (MySQL, MSSQL, Redshift).

Source: Wikipedia

Understand how different technologies communicate with each other. While working on a task you might need to connect to a database to get the data, get comfortable with libraries to connect to databases in your to-go language.

Bash/Powershell scripting: there is a good chance you will encounter things such as CRON jobs. Familiarize yourself with the Linux and bash environment, and learn how to launch scripts using arguments from the terminal, this is how often scripts are launched.

Docopt, you will definitely hear of that one day…

Another very common situation is when data comes from an API , it is worth understanding how APIs works , more specific concepts such as token authentication, JSON, and HTTP codes.

, it is worth understanding how APIs , more specific concepts such as token authentication, JSON, and HTTP codes. If data is not stored on a database, or coming from an API there is a good chance you will work with Buckets, buckets are simple storage services provided by AWS where users can store different files such as .csv , images, and so on.

An example of how an S3 Bucket can be used.

Working with big data: while learning or at university we are taught how to read data from files such as .csv , we call a small function such as .read_csv and automagically we have our data ready for manipulation. In real life, often, you won’t be able at all to perform something like this. Data collections are too big to fit in memory, and different strategies, such as reading files in chunks , using technologies such as spark , or working on smaller samples of data.

data: while learning or at university we are taught how to read data from files such as , we call a small function such as and automagically we have our data ready for manipulation. In real life, often, you at all to perform something like this. Data collections are too to fit in memory, and different strategies, such as reading files in , using technologies such as , or working on smaller of data. Docker: to keep the whole team on track, avoid dependencies issues between dev and production environment, or simply to avoid the infamous phrase “it works on my machine”, Docker provides and infrastructure to manage environments (called containers). I would advise to familiarize yourself with Docker and its basic commands such as docker-compose and docker-build and understand what a registry is. This will save you a lot of time.

Source: RSAConference 2019

Conclusions

As previously mentioned later, very often, it is just a matter of getting used to it. Do not be afraid to learn new things or getting your hands dirty, keep in mind to always accept critics and listen to others, and finally: do not be afraid to “break things”.

Finally, I understand it is very easy to get overwhelmed when learning something new, there are thousands of resources, articles, videos, courses, and when the menu is too long choosing becomes harder and harder, however: