Data Engineering — Udacity Nanodegree
By: Martyn Woodison
Tags: Python, Postgres, Data Engineering, NoSQL, Udacity Nanodegree

Why this course and why write about it?

I currently work for a Technology company, I have several years of experience with database technologies and everything that goes around this, my skills have evolved over the years to take on development (front and backend) work, but I would describe myself as self-taught (with a lot of help from some amazing colleagues over the years). I read a great post on Stack Exchange that gave me some confidence around my journey which stated when it comes to development, what you know and can demonstrate is more important than how you learned it, its an expectation of a good developer that you will be able to learn and find resources without needing to be handheld throughout. With all that said, we are currently in lockdown again, so I thought it couldn’t hurt to have something on the CV that validates my experience before demonstrating.

The bigger question, why am I writing this. I have a few different reasons, and not to go into excessive details as it's not relevant to this post, but I will highlight a couple. Firstly, several studies highlight the learning-by-teaching effect is a great way to improve understanding and improve knowledge retention, I would also love to improve my writing style, and the only way to do that is to practice, so why not start by writing articles on Medium. The most compelling reason came from another medium post I read recently. Shaunta Grimes had written a post on starting to write on medium in 2020, it was nicely written, and the piece I took away came from a section where she spoke about Seth Godin. The story references a statement he made on a podcast encouraging people to write, not because you are going to make money from it, but because of the benefits you will gain from doing it.

“….But to have a blog because of the discipline it gives you, to know that you’re going to write something tomorrow. Something that might not be read by many people — it doesn’t matter — it will be read by you.”

There was something that resonated with me from that so I thought I would give it a go.

Welcome to the Nanodegree Program

To start with I won’t dwell too much on the cost, this seems to change month to month, when I signed up it cost £165 (50% discount) a month but looking on the website today the discount has disappeared. I would say £330 a month does seem excessive from the initial content I have seen.

The setup for the course is smart, what you get that you don’t necessarily get with Udemy is a working environment set for you with access to different instructors (there are pros and cons to this), mentor help, peer chat, careers team and a careers portal. These are all nice to have features and a good intro to the course, but at this point, it is hard to judge how useful these features are.

Within this section, it covers the prerequisites for the course in more detail. Having gone through the first sections of the course, I would like to cover a couple of pieces off to start.

With regards to SQL requirements, it states you should be able to write advanced SQL join functions, potentially working with queries every day, I wouldn’t consider anything I have seen on the course so far requiring advanced knowledge, advanced beginner maybe but not advanced.

The Python skills, if you are a novice, would be more of a challenge. You will require all of the following to get a proper understanding of the different sections and specifically ETL work; understanding this well is a must.

Define functions.

Use loops and conditional statements.

Write and use classes in your functions.

Work with various data structures.

Use data-related libraries, like NumPy and Pandas for ETL purposes.

Nothing within the course should be considered a barrier but be aware if you haven’t learnt the basics the first project submission may seem daunting.

Data Modeling

Within the first section of the course, we cover the starting points of Data Modeling. The opening projects look at modelling data for PostgreSQL and Apache Cassandra. It was a nice introduction to the course as I have had a lot of exposure to Postgres over the last few years from the relational side, I have less experience with NoSQL, and I had never used Apache Cassandra, so I was looking forward to covering that section.

Each lecture is supported by key points and takeaways plus additional reading; the examples provided are good although fairly basic. There is some good initial grounding and reading around ACID transactions and when a relational database is the right fit. The explanations are in-depth, although a lot come from free material readily available to all. For example, the section on ACID transactions uses Wikipedia as its source for each of the explanations:

“Consistency: Only transactions that abide by constraints and rules are written into the database; otherwise, the database keeps the previous state. The data should be correct across all rows and tables. Check out additional information about consistency on Wikipedia.”

The exercises are OK, I’ve read some posts on the exercises not always working, that wasn’t my experience, and I found the notebooks to all work as expected. The one frustration I had was around the amount of repetitive typing to carry out insert statements in some of the earlier examples, I’m sure with a few tweaks this could be a more interesting learning experience, but that's just my personal opinion.

ETL for Data Modeling with Postgres

I enjoyed the project for the relational data modeling, it was a nice step up from some of the earlier examples provided, and I finished it feeling like I had come away with a better understanding of the key concepts and some useful tips I could use.

This section then closes off by covering lectures on NoSQL data models and a similar project using the fictional Sparkify music streaming company with Apache Cassandra. The course covers off the best fit for NoSQL, listed below:

Need high availability.

Large amounts of data.

Need linear scalability.

Low latency.

Fast read and writes are required.

There were some good examples provided around the primary key, covering how the partition key determines the distribution of data and how the clustering column determines the order of data. The final project in this section ensured this was understood by working through three fictional business questions requiring a good understanding of partition key vs composite key vs clustering Columns.

Example Apache Cassandra query

Conclusion

I’m a good chunk through the course, and I’ve enjoyed the opening sections, I’m working my way through the remainder and will make my code available on Github and summarise the benefits when I finish. From what I’ve seen, the course is well put together, I’ve refreshed on some core elements, picked up some tips and come away with project examples to enhance my Github page.

For the amount I’ve paid and the fact I’ve covered the majority of the course in the first month whilst working full time feels like reasonable value, I’m not sure I would feel the same if I had paid the full price, some of the earlier sections are too basic and slightly repetitive. Still, as a course, it provides a good initial exposure to concepts and different database technologies.

Writing my thoughts down has also been very beneficial, I recapped on topics that I may not have done previously and feel l have left with a better understanding as a result.