An amateur data scientist: learn programming with a hobby project
By: Yulia Lukashina
Tags: Data Science Training, Learning Programming, Data Science Careers, Programming, Data Science Skills

Jupiter is not only a planet © Yulia Lukashina

An amateur data scientist: learn programming with a hobby project

My own “research” project helped me to become confident with programming, and prepare myself for the next career level. An overview of the tools and languages involved.

With almost 32 DataCamp chapters finished I still could not call myself a data scientist. It became clear during the multiple job interviews. Although DataCamp and Code Academy offered seemingly a lot of programming practice, I could not write any ad hoc code. That was a big problem.

Start a hobby project

I needed more practice. I checked on the projects that DataCamp suggested, but I did not like them somehow. One of the R beginner’s courses used the International Movie Database (known as IMDB) to explain descriptive statistics and basic data cleaning.

Since I am a movie fan, I got the row data from the IMDB, and started to “play around” with it, as they call it on DataCamp.

I did not have a particular research question in mind. But I must say that making the IMDb data tidy alone took a lot of time and effort.

Practically, every user can add a movie on IMDB. And then rate any movie. This led to a lot of irrelevant entries, like movies nobody watched, or thousands of episodes of the same TV soap opera.

Finding your dataset: Kaggle and Co.

You have actually a big choice of publicly available data sets, like data on economic developments from OECD or Eurostat.

Another source of big data sets for beginners is Kaggle. It is an online platform where you can create your profile, and do all the things you know from other social networks: give likes, follow people, feel happy if someone likes you.

As a perk, big companies publish their data on Kaggle as part of numerous competitions, where you can either win money or get employed by the company.

You may want to use Kaggle as a cloud computing service for free if your PC cannot level the data.

From description to correlation

But let me get back to my IMDB experiment. In the beginning, it took me quite a time to stop making “grammar” mistakes. That only proved the efficiency of my self-learning method.

As soon as I started to keep all commas and brackets in place without consulting Google every time, the coding became very fluent and pleasant. I could go to the next level. The project was developing itself from a full mess into something meaningful.

I started to ask research questions. First, very descriptive ones. For instance, which genre is the most representative? Which genre gets the highest scores?

What is very important is that I found out that “Shawshank Redemption” is the best movie ever.

Then I moved to question about correlations. The database was launched in 1990 (according to Wikipedia). The movies that were released after that date, and especially after the Internet access became a routine part of our life, have much much much more chances to get scored.

That nevertheless does not apply to all movies. Not to “Shawshank Redemption”, for example. It was out in 1994.

Data viz for deeper insights

By plotting data and using other dataviz knowledge from the online courses, I discovered more dependencies. For instance, correlations between genre and rating score, or genre and decade. Before the 1980s you could barely speak about science fiction as a serious movie genre. But after 2000 the genre exploded.

Discovering relationships between movie scores and year of production

The box plots gave me some not so intuitive results. Interesting enough, the meaning of the score is different for different genres. A comedy with 6 is still a good one, whereas a drama with 6 is a very poor and not worth watching at all.

The visualization part of my project made me look for new packages that we did not cover in the online courses. It also helped me to start with generalizations and conclusions.

Movie scores and their relationships to their genres

Never-ending learning

Two years later I turned back to my IMDB notes. I went through the same data cleaning and plotting procedures, but this time wrote the code in Python. I also switched to the Jupyter notebook.

I also used a GitHub directory to track and backup changes in my scripts. I did it more to get familiar with the platform and the concept of branches and merges. Then my project was migrated to Jupyter, and both GitHub and Kaggle were abandoned. I found it easier to track changes by simple copy-paste in Jupyter, and it is fast enough to keep all versions of the code in one workspace without making your PC getting out of breath.

Beyond the data science

The last but not the least, this (ongoing) project helped me to get to know API calls.

One day I read through the IMDB copyright guidelines more carefully. It stated that the results of data manipulations are not allowed to be handled to any third party and can only be used privately.

For this reason, I started to look for some other substantial movie database. I came across The Movie DB. It allowed only API calls, no CSV download, which the IMDB offered. In addition to standard API packages, I got closer to the mysterious JSON format.

I also had to work on my data cleaning script again. Sometimes the data I received made no sense to me. Not due to a syntax mistake, but rather since I either sent a not so well-elaborated request, or failed to get the right data from the JSON.

Two lessons I learned here apart from the coding itself. First, always check the JSON structure exactly. Missing a level prevented me from getting the entire data I needed. Second and quite banal, always read through the documentation carefully.

Old data, new angle

In the end, my IMDB project was such fun that I decided to do more. I polished my research question and turned it into the following:

“How good is the rating of a queried movie in comparison to all others of the same genre?”

Titanic’s (1997) score and its place in the drama genre

As a solution, I got the data on a movie, including the genre, then queried all movies of that genre and then built a box plot, showing descriptive statistics as a box plot with one movie marked as a star.

It was supposed to help in assessing the real value of that movie. It weighted the score and gave the factual span of the scores for similar movies, not just its place within an abstract 1-to-10 scale.

After I finished the data cleaning and descriptive part, I was so excited that I decided to program a small one function Python app to query movie titles and then display them as buttons. A user would then press a button to get a box plot for a movie. But this is another story.

Conclusion

During all this, I learned to write R and Python code fluently, without checking any source for syntax or a function. Fluency is very important to stay in the programming flow.

It is also very helpful if you have a skills test during a job interview.

To put it all together, if you want to get closer to your data science dream, but your current job does not offer you a lot of practice you may go through these steps:

finish an online course

find a dataset with a sufficient amount of a coherent data

get some practice with it using open source tools and languages

publish your results in a data science community

use it as your portfolio

start writing your data science job applications.

Good luck!