Should You Become a Data Engineer in 2021? | by Nicholas Leong | Towards Data Science

All of us have heard about the title — Data Scientist, the sexiest job of the 21st Century.But is it really?
For many people who are new to the data industry, Data Science will be the first thing they hear about, and the title just sticks. Suddenly, everyone is either already a Data Scientist, or aspiring to be one.
However, that is not the only job title in the industry.
There are many more job titles that are related to data.Data Scientists have been on the frontier of data for way too long, it is time for the emerging of Data Engineers and other data-related roles.
Looking at 2020 data, it is already happening.Data Science might be slowing down.
According to the 2021 Data Science Interview Report by interviewquery.com that takes in over 10,000 Data Science interview experiences, Data Science interviews grew by 10% compared to Data Engineering interviews which grew by 40% in 2020.
On job growth, data was taken from over 450+ tech companies. For the data scientist role, there was actually a 15% decrease in job growth in 2020 versus 2019. This dip was offset by the growth in other data-related roles like business analysts and data engineers.
Hence, stop spreading the news that Data Scientist is still the sexiest job.There are ‘hotter’ roles in data right now, and Data Engineer is one of them.
Let’s talk about job growth and the demand for data engineers.According to DICE’s 2020 Tech Job Report, Data Engineer is the fastest-growing job in 2019, growing by 50% YoY. Data Scientist is also up there on the list, growing by 32% YoY.
But that’s not all, other salary studies have also found similar results about the Data Engineer role —
Hired State of Software Engineers Report— 45% YoY Job Growth
LinkedIn’s Emerging Job Report 2020–33% YoY Job Growth
Burning Glass Nova Platform — 88% YoY Job Growth
As data practitioners, we understand that data doesn’t lie. The data states that Data Engineering is one of the fastest-growing jobs in recent years, even as far as surpassing Data Science roles.
According to the IDG Cloud Survey, which collects data from 551 large tech buyers, states that only 38% of their total IT environments are on the cloud as of today (2020).
In 18 months, these IT environments are expected to hit 59% on the cloud. The same tech buyers are also allocating 32% of the IT budget to cloud computing, which amounts to $73.8M per Tech Firm (59% increase from 2018).
After a huge influx of Data Scientists, companies realized they needed the proper data infrastructure to perform any meaningful analysis on the data. Companies have started to spend money on proper cloud infrastructure, data architecture, and hiring new data engineering talents.
One solid proof of demand for Data Engineers is the growth in big data.If you’ve been living under a rock, data is the new oil.
New data is being generated every day, and it will only continue to grow. With the insane amount of data, there will be an increase in the demand for engineers to handle data.
Another giveaway is the growth in big data engineering services provided by consulting firms like Accenture and other tech companies like Cognizant. With the increase in data, proper data engineering services is increasing as well. The data engineering services market is experiencing an 18% growth p.a, which is expected to hit 31% p.a by 2025.
Data Engineers' salaries are very high. Even higher than Data Scientists’ in many cases. Many salary studies (summarized below), have shown that Data Engineers are among the highest-paid talent, and their salaries do not see any slowing down anytime soon.
According to Hired’s 2019 State of Software Engineers report, average salaries for data engineers increased 7% in New York and 6% in the Bay area, vs. an average of 3.5% for the rest of Americans.
Here are the salary studies on Data Engineers —
As seen from the data, Data Engineers comfortably make anywhereabove $110k which is considered a respectable number by most people. Furthermore, the salaries are expected to increase due to the shortage of supply of Data Engineers.
Based on a study done by Ori Rafael (CEO of Upsolver), there are about 2.5 candidates per job listing on LinkedIn for the Data Engineer role compared to the other roles which have much more applicants.
The number suggests that there is a shortage in the supply of Data Engineers. Hence, applicants are much more likely to negotiate a higher salary range for themselves in the hiring process.
It goes without saying the hiring process is different for everyone, and the negotiation range depends on a wide range of factors. These factors include but are not limited to —
Years of Experience
Skills
Previous Achievements
Location of Company
Industry
Speaking from experience, I have personally seen companies that value their data greatly offer higher salaries compared to other companies. A good example of these companies are companies that offer some sort of data product, as well as having some sort of goals for their data.
So, what do Data Engineers do?Well, I’ve written all about it here in one of my previous posts.
towardsdatascience.com
Essentially, Data Engineers manage the core foundation on how the data of the company is structured in the data warehouse. Tasks range from designing table structures to building ETL/ELT pipelines to ensure a scalable, high-performance infrastructure to satisfy all data needs of the company.
They also deal with pretty much anything that has to do with automation with data. One of the examples I have worked on before is the Gmail Data Pipeline I created.
towardsdatascience.com
Data Engineers are able to answer questions like —
When an event happens on the site, what happens in the master database?
Why are the tables in the data warehouse structured like that?
Why do we extract data on a daily basis instead of hourly?
More advanced Data Engineers also work on streaming data pipelines that ingest data into a data sink in real-time. This is extremely useful for Machine Learning products that take in data and perform something with it in real-time.
There is a wide range of skills required to be a Data Engineer. Besides mastering programming languages, Data Engineers need to understand how databases work, as well as how to properly design pipelines to load data into databases. I’ll make it easier for you by summarizing it into 5 main competencies —
Programming Language — Python and SQL
SQL and NoSQL Databases
ETL/ELT Technologies — Apache Airflow, Hadoop
Streaming — Apache Beam
Infrastructure — Cloud Infrastructure
These skills combine perfectly to form a Data Engineer.To really get a feel on how to build pipelines, take a look at a guide I wrote on how to build your first pipeline using Apache Airflow.
towardsdatascience.com
If you have made it this far, you must be really interested in Data Engineering.
What’s there not to like?
It is one of the fastest-growing jobs in one of the fastest-growing industries while having one of the highest average salaries in the world. It almost sounds too good to be true. In this article, we have discussed the following on the position —
- Job Outlook- Pay- Responsibilies- Skills
As usual, I end with a quote.
You can have data without information, but you cannot have information without data — Daniel Keys Moran
You can also support me by signing up for a medium membership through my link. You will be able to read an unlimited amount of stories from me and other incredible writers!
I am working on more stories, writings, and guides in the data industry. You can absolutely expect more posts like this. In the meantime, feel free to check out my other articles to temporarily fill your hunger for data.
Thanks for reading! If you want to get in touch with me, feel free to reach me at nickmydata@gmail.com or my LinkedIn Profile. You can also view the code for previous write-ups in my Github.