5 Post-Pandemic Trends in Data Science
By: Tobia Tudino
Tags: Business Science, Business, AI, Covid 19, Data Science

5 Post-Pandemic Trends in Data Science

How Data Scientists Have Adapted to Meet Urgent Business Needs in the Face of the Covid-19 Crisis

Photo by engin akyurt on Unsplash

At the beginning of this year, I was interviewed for an article about critical trends in data science I predicted for 2020. When I said that changes were happening so rapidly that it was practically impossible to fully anticipate what would happen in the field of data science over the next year, I had no idea how true that would be for 2020.

With the Covid-19 pandemic upending everything around the world this year, the major trends in data science, especially data science applied to the business world, have changed significantly. Here are five post-pandemic trends that I have seen emerge in recent months.

1. Analytics and data science have become less siloed.

Cross-functional teams have become increasingly common, yet most businesses still kept data scientists and analytics teams separate from the teams making critical business decisions. Only the most data-driven companies had effectively integrated data science into regular business operations. The pandemic quickly changed this. Businesses created cross-functional crisis-response teams to create analytics-driven solutions. For many companies, this has revealed data science as a valuable tool to improve decision-making. Post-Covid, more of these teams are likely to be the norm.

Data scientists must, in turn, have a better cross-functional understanding. Data scientists must understand the business context to contribute to these discussions beyond just providing the data. It’s not enough to be a coding expert anymore; data scientists need to be business scientists who can contribute actionable insights. As data experts become less siloed, they must have a more comprehensive understanding of the industries where they work and business operations.

Photo by Carlos Muza on Unsplash

2. Data visualization skills are a must.

Customers and investors need to feel secure that the businesses they are interacting with are committed to safe practices, so more companies are releasing more data than they had in the past. Raw data, however, is useless. That’s why data scientists skilled in data visualization have come in infinitely higher demand. Data visualization skills are no longer a bonus on a resume. They are required. At the beginning of the year, I highlighted the increasing importance of being able to explain how your algorithms work and how you came to your conclusions. Now data scientists must show, not tell.

People everywhere are experiencing incredibly high levels of stress. They aren’t willing to invest a lot of time in understanding your point. You need to make your results clear in seconds; a useful graph or chart is the only way. Data scientists without data visualization skills are at a considerable disadvantage post-Covid.

3. Automated decision intelligence has become a priority.

Only the most agile companies are surviving the economic devastation of the pandemic. Every decision matters, and leaders must respond to changes in the market faster than ever. Automating key decisions using real-time data helps prevent costly mistakes. In these unprecedented times, decisions based on past performance and gut-feelings no longer suffice; everything has changed. Automated decision intelligence helps fill the gaps in our knowledge by using real-time data relevant to our current economic reality.

Automated decisions based on AI models allow companies to react more nimbly to new patterns in the data. The best intelligence comes from models that can anticipate trends long before humans could on their own. Data scientists now must prioritize models that can deliver this kind of accurate intelligence.

Interestingly, the pressures of Covid-19 have also reduced employee resistance to these kinds of AI-augmented automated decisions. Accenture research indicated that one of the biggest roadblocks to implementing this kind of technology in 2019 was a lack of employee adoption and negative sentiment. That is already changing. The pressure put on the workforce during the pandemic has made it more apparent than ever that we need effective technology to help us make better decisions and survive during difficult economic times. As businesses recover, employees are likely to continue to use decision intelligence to make their work easier.

Photo by Nong Vang on Unsplash

4. Businesses have focused even more on the customer, as have algorithms.

Historically, businesses offered what they believed customers should want in a business-centric “push” model. Few alternatives were usually available, so there was less risk of alienating the customer. Over the past decade, we have rapidly seen businesses forced to adopt a more customer-centric approach. With the internet making consumers better informed and more demanding, the economy has shifted to a customer-driven “pull” model.

The coronavirus has only accelerated this trend. Consumer spending has dropped overall, and impulse shopping in stores has virtually ended. Customers are only purchasing goods and services online that precisely serve their needs. This makes it vital for the algorithms that dictate pricing, customer service, and supply chain decisions to align with that same goal. Data scientists must pivot to increasingly customer-driven models in order to meet the needs of businesses.

5. Data scientists must be more agile in responding to data imperfections and model drift.

Drastic deviations from historical patterns in consumer behaviour and economic activity have challenged every business’s models’ robustness. Data scientists found themselves getting unexpected errors and even nonsense recommendations. The disruptions have revealed fatal design flaws in far too many models. After all, a good model should use disruption as an opportunity for automated learning. If the models drift in response, then they were not properly set up for self-learning in the first place. Such a discovery is never encouraging, but it’s even direr amidst a crisis.

Data scientists have had to respond quickly to ensure that even imperfect models were still delivering the most accurate information possible as they were reworked to adapt to a dynamic economic reality. This crisis requires an agile response to existing data imperfections and preventing deviations from becoming model drift. Not only are data scientists working to use imperfect data to get actionable results, but they are also forced to rework models to better learn from and adjust to continued disruptions moving forward. Since there is no precedent for this pandemic, data scientists must think creatively and quickly — something that makes many typically methodical data scientists uncomfortable.

Yet data scientists must adapt to be more comfortable with meeting crises with action. Although we can hope that another worldwide pandemic does not happen anytime soon, smaller disasters and other problems with data are inevitable. Data scientists must prepare themselves to respond as agilely as possible to whatever problem they face next to avoid model drift — and continue to provide data-driven insights companies can depend on. This crisis has revealed the weak points in everyone’s systems; now data scientists must shore up those weaknesses to avoid breaking down come the next disruption.

Photo by Fatos Bytyqi on Unsplash

Covid-19 may have changed data science for good

These trends are not merely interesting deviations from the past; they are likely to dictate how data scientists operate for many years to come. While it’s easy to ignore these trends in the aftermath of the crisis, we should all learn from what’s happening, so we can be prepared for how they change our profession in the long run.

Special thanks to Kaitlin Goodrich.