Why data science has failed to deliver on its promises
By: Andy Greatorex
Tags: Innovation, Disruption, Jobs, Data Science, Machine Learning

DSaaB — Data science as a bubble

It’s a familiar story: a new technology breakthrough kicks things off and media interest fuels hype, even while no practical products exist. Premature publicity produces success stories which become popular, while scores of failures go unnoticed and unlogged.

Data science became the hip, new kid on the block around the time AlexNet was released back in 2012, and its rise, much like economic bubbles, was predicated on a tide of speculation and C-suite FOMO.

In the ensuing years, open positions for data scientists exploded, an entire cottage industry around bootcamps and courses emerged online and at universities, conferences devoted to the subject were born, online demos wowed the world, and barriers to entry fell with the democratisation of machine learning platforms through plug-and-play solutions.

And while some companies had success in their data science operation, many — sadly — did not. Executives assumed all they would need were an elite team of ‘unicorn’ data scientists capable of end-to-end data pipelines. Unsurprisingly, this dream failed, and the companies who employed it did not see the data science enabled gains they’d expected to find.

Slowly, slowly, companies are maturing beyond the desperate, speculative default position that hiring a crack team of data scientists will unlock wider margins and amass prosperity. Executives are waking up to what the baseline requirements are for supporting an effective data science operation. They’ve been wounded before by a promised land vision of models and statistics and humungous gains in this-and-that metric, and should now be more battle-hardened and skeptical of a skin-deep data strategy.

One important side effect of the ballooned capital and intellectual investment in data science over the past decade or so is that the data infrastructure necessary to support its ascent has been assembled in lines of code, the classrooms of academic institutions, and in the headspace of executives. This is clear in the hiring surges for data engineers over data scientists, despite the oversupply of junior data scientists entering the job market relative to demand.

A speculative bubble on a disruptive technology is built on the gamble that it will one day become market dominant. What’s typically overlooked is that innovation in one field is often contingent on the innovation of a supporting industry. For electric vehicles this is batteries. The phenomenal improvement in the quality of smart phone cameras was necessitated by the rise of, unsurprisingly, smartphones.

For data science, the partnering innovation is a conglomerate of items amounting to a well rounded data strategy. This must include: well-defined data ownership with personal accountability; ongoing reviews of the existing data model and data quality; a centralised data governance process; a deep focus on data architecture; pushing for a data-driven culture via forums, working groups, and discussions; automated data pipeline testing; well defined processes for ML model deployment with ongoing monitoring, maintenance, and ownership.

Once the strategy is adopted and worshipped as gospel, only then can data scientists produce useful work year after year. Until then, expect POCs and fragile implementations.

The bubble of data science has been depressingly unproductive against its intended aims, but in its wake it’s helped to enact a new dawn of data enabled culture and better data strategy.

So what’s the lesson here? It should be clear. The prerequisite for any organisation seeking value creation via data science is to offset in-advance the technical debt accrued during model development and deployment. You can’t run out and hire a data science team without an airtight information processing pipeline and a data first culture.