Move from Engineering to Data Science | Towards Data Science

A few years ago, I made a big decision in my life when leaving the job at a Germany tech giant to pursue a higher degree in Data Science and shifting my career toward this field. I probably didn’t think this through when I decided and might be I was so obsessed with all the great innovative ideas of many sales guys from Europe. Although those visions provided me with a conceptual understanding about the potential of AI across many industries and inspired me to make the move, I haven’t seen many of these ideas coming true in reality so far. Just like one of my friend used to believe that if he knew how to build a good Machine Learning model, he would be able to predict the stock price and become a billionaire. This was what motivated him to explore Data Science and with this passion, he won a lot of ML competitions for his powerful models. Yet he is not a billionaire so far, despite the extremely high salary he is being paid for his talent. I will let you know if one day he is successful in tracking and predicting the indices.
Back to the story, this career switching, nevertheless, gave me bunches of opportunities for exploring possibilities, expanding my knowledge, living in another country and more importantly, the experience of working on a new field with many different aspects as compared to the engineering field I was working on. So what are the differences?
“Fail Fast, Fail Often”
This is the key concept of Agile, a project management methodology widely used in software development and adopted in the data science field. The idea here is that failing fast will allow us to learn quickly from the mistakes while failing often helps us perform this learn-and-error loop more often to deliver a better product in the next version. Agile is not the best but so far having a happy marriage with ML projects. Basically, everything re-doable and iterable is mostly perfect for Agile. Models can be re-trained for better accuracy and performance, data can be processed in another method to have a cleaner output, visualisations can be re-touched to deliver a better message.
However, due to the nature of the product, Waterfall or doing thing in a sequence is more compatible with the engineering projects. For instance, you cannot destroy and redo level 2 of the tower when you are already constructing at level 20. Furthermore, failures in engineering have to be avoided at all cost even during development. When the electrical cables have been buried a few meters underground, the only way to learn from mistakes is to wait until the next project. Or when you are managing a construction site, you don’t want this “fail fast, fail often” thing happens because someone might be injured by this failure.
“404 Error Page Not Found”
I have to confess that I literally blew up many things before with bugs in my code during the previous engineering career. You might wonder why software can cause explosions and guys, I am not talking about making a bomb here. What is meant here is the fact that as an engineer, everything you do will have direct physical impacts and therefore a software bug could mean severe physical damages. Let’s take look at the simple code below:
IF (#currentPressure > #pressureHighLimit) THEN    #pumpOut := 0;END_IF;
This is the code used to protect a pump system. It looks simple to many software developers as they can even explain it without knowing this programming language. The code says: “if the current pressure is greater than the threshold then the pump has to be stopped”. I guess you may now have a little bit understanding about the situation here and why we need this code. If the pressure keeps building up inside the pipe due to a clog on the other side, the pipe might be exploded. Even the algorithm is obviously straight forward, knowing how the actual physical components, controlled by the software, interacting with each other is extremely important. It is relatively easy to just miss one condition and everything might go completely wrong when a software bug can create an explosion and someone might be injured.
In Data Science, a bug would be an unhandled exception, data processing pipeline errors, choosing suboptimal model parameters or backdoor that leads to data breaches. These “bugs” might create negative impacts on customer experience, affect the business directly and even cause reputation damages to the organisation. Therefore, the “fail fast, fail often” strategy is needed to catch these failures as soon as possible in the development before deploying the software into the production environment and scaling up. Even if the bugs come later during the operation, it still can be fixed by a software update.
When Apple was in a rush to release iOS 13 on schedule in 2019, many users experienced several problems such as battery drain, unreliable Bluetooth connection, Apple pay and mobile data service issues, etc. Soon after, they released certain patches to fix the bugs and quickly improve the OS after several cycles of updates.
However, also in 2019, Boeing didn’t have the chance to do that with their 737 Max accidents due to a bug in the software used to control the stability of the aeroplane. They also recalled all the aeroplanes for the software updates but it was too late. 189 people died in the crashes, 737 Max aeroplanes have been globally grounded, $4.3 billions of market value was erased and the CEO had to step down. All started with a few lines of buggy code when the team was trying to meet the deadline.
When incidents happen in engineering, human lost would come first and then financial damages arrive later whereas in software development or data science, the impacts on the business would be the first thing to come and human lost might happen if someone jumps off the building because of the stock price drop.
“Safety First”
No system is perfect and failures will always happen. However, when these events occur, the priority for handling them is different between engineering and data science.
You may have seen “Safety First” signs when driving past by a construction site or visiting a factory before. Indeed, safety is the top priority in the engineering field and the rule of protection is human first. For example, the protection circuit breaker must shut the power off when there is a dangerous failure that can cause someone electrocuted even the business might lose billions dollar due to this downtime.
When it comes to Data Science, data is the subject of protection. Data governance and security will be given the highest priority and all applications must comply with them. An AI application would never be deployed into the production no matter how accurate the model is if it violates the governance framework and poses a threat of data breaches. These are the first and biggest problem data scientists have to deal with when sourcing data in a data science project, especially with sensitive and highly valuable data.
“ISO 45001”
Engineers have to follow certain standards when designing their products while conventions are more popular in the data science world.
For instance, variables in Python should be named this way: first_variable, second_variable, etc. Or “X” and “y” should be used to name features set and target variable respectively in a supervised machine learning problem. These are conventions that have never been written in any legal documents and we all know that it is not illegal to name variable like this: “thisismyvariable” although you might become a weirdo to your developer fellows and reviewers.
In contrast, when it is written in the national standard that: “general power outlet shall not be mounted less than 75 millimetres from the floor” then in the design, all engineers and contractors must follow this regulation. If they try to install the power socket lower than the standard distance and someone is electrocuted later due to this incorrect installation, serious legal actions would take place.
“Which job pays higher?”
I guess this is the most interesting part to most of you. Instead of providing numbers and visualisations as well as reports for salary insights, I will tell you my personal experience even though it might be subjective.
Although facing higher risks with lots of strict regulations to follow, the average salary of engineers is lower than that of data scientists. I am an engineer working in the factory automation field for more than 8 years and my last annual income as an engineer was only a bit higher than my current salary as an entry-level data scientist.
However, engineering consultants might earn a lot more than mid-senior data scientists. Some of the offers for control engineer positions, working in mega-projects in UAE and Northern Europe, provide a package which is twice the average salary of a data scientist role in Australia. Imagine you only have to work for six months to receive the income that you could have to work for the entire year to get it. This could mean half-year working and half-year on vacation.
“It’s not just about money”
Engineering can be considered as a traditional industry in which experience is often given more weight than creativity. Unless you move to management levels, the top positions in engineering have been saturated and there are not many changes in human resource allocation. As engineers are always restricted by national and regional standards, working in another country using different standards would never be easy for them. For this reason, they are less likely to become a global employee and end up staying in the same position for more than ten years.
On the other hand, since Data Science is an emerging field with promising future and opportunities, working as a data scientist will give the employees more room to develop their career and expand the possibilities. Besides, Data scientists are not often restricted by local regulations except data protection policies. For example, Python programming language is spoken the same way no matter where the data scientist comes from. In other words, the skillsets are transferable across the borders and it is thus easier for data scientists to apply for jobs in other countries.
“What did the sales guys say?”
I believe that there will be an intersection between data science and engineering where the boundaries between the two field become blurry. Industry 4.0 is, in fact, the combination of Data Science and Automation Engineering. The results from the analytics of Big Data resources could provide insights for the Automation System to make its own decisions for production and the manufacturing process will be performed dynamically as compared to the traditional control system with fixed or structured programs. Although the idea of using AI as the brain and the mechanical systems as the muscles haven’t been successfully and widely implemented, the merging process of data science and engineering has begun.
Some engineering projects are now able to apply agile model by using simulations. The engineers can quickly design virtual prototypes and use the data from simulations to improve their design in the next iterations. By this way, they can foresee most of the potential failures in the design and reduce the time-to-market of the product. Three years ago, I had a chance to use a technology called digital-twin to test my program before running it on the real plant. I can build a virtual model of the machine I am working with and run my program to control the virtual mechanical components of that machine. By using the virtual plant, I am able to quickly test my program without blowing up the actual pipe and therefore the risk during the implementation phase is dramatically reduced.
Another example is predictive maintenance applications in which data from sensors around the device can be collected and analysed to early predict the point when this device fails to perform necessary maintenances beforehand. I can use data from flow meters, pressure transmitters and temperature sensors around the pump to evaluate the current performance and forecast the upcoming mechanical failure of the pump. This would help the management become more initiative in maintenance planning, manufacturing scheduling and decreasing the downtime.
Although this story seems to give more credit to Data Science, I am not encouraging you to leave your engineering job and switch your career. Sometimes, it’s not just about drawing a line between the two and choose one of them but more importantly, it’s about how you would see yourself in the next 5 or 10 years and bridge the skill gaps in the future.
I just hope that this story based on my personal experience would give you a clear overview of the two fields so that you can take it as a reference to plan your next steps. For example, if you are a data scientist, you can choose to learn more about using electronics boards to develop your IoT and Big Data applications. Or if you are currently an electrical engineer, you can start researching about time series model which can be used to predict power consumptions of households.
Finally, always remember that your passion will lead you to a better future after all.