Why are Google Bigquery, Snowflake, Redshift and other cloud data warehouses slower than most expect? — Part 0
By: Lori Lu
Tags: Big Data Analytics, AWS, Snowflake, Google Big Query, Cloud Data Warehouse

Why “ Performance at Scale ” Matters?

C-Level Dashboard Shame

If a leadership dashboard is not fast or blazing-fast, then it’s broken.

I’ve heard of quite a few funny stories on leadership dashboards from some of my VIP clients. Each story was “formed” in its unique shape, but all came down to the same storyline — an ambitious department commits to delivering a real-time, responsive, self-service dashboard to empower the leadership team to change to a data-driven decision-making process. Therefore, they spend months designing the metrics/KPIs, visuals & formating and so forth. When the dashboard finally goes alive, they have to give up the idea of doing a live demo for stakeholders in a performance review meeting. What if the dashboard loading fails halfway through the live demo? Would it lead to a catastrophic collapse of trust?

Another story from another VIP customer — one regional leader measured leadership dashboard latency one by one with millisecond accuracy to push the engineering team to make it faster. Still, the performance stays the same even after an expensive 2-week consultation with a performance specialist from Microsoft Azure Analytics Services Team.

Data Democratization

“Data democratisation means that everybody has access to data and there are no gatekeepers that create a bottleneck at the gateway to the data. The goal is to have anybody use data at any time to make decisions with no barriers to access or understanding,” says Bernard Marr, bestselling author of “Big Data in Practice.”

I believe this term, Data Democratization, has gone viral for a while. You can find its presence across all sorts of digital transformation marketing blogs and case studies.

Ultimately, for entrepreneurs striving to lay the foundation for future growth and keep leading the market, data democratisation is a must. The question on their minds should not be if data democratisation is coming, but how they can create it in their organisation quickly and efficiently.

So, in the process of realising the “HOW” question, what are the key points decision-makers cannot miss marks on when hunting for the right technical solution?

Imagine, when breaking down information silos, making it manageable from a single place, and then connecting those previously siloed data to allow access to any tier of your company, without a super powerful analytics tool in place, what would happen?

Every morning, non-technical data consumers “squeeze” into the analytics platform ( like “being packed into subway cars like sardines” ), exploring a variety of data sources, comparing KPIs of this year to the last couple of years and then witnessing Slowliness, Timeout, Failed request, Errors and so forth…

Sooner or later, people across the organisation would realise that the concept of self-service analytics is coming, not existing as of now.

As you can tell from the stories from early cloud adoptors who have poured their blood, sweat, and tears into those modern cloud data warehouse solutions but still failed to make it work, most current analytics platforms are still incapable of handling this impossible situation.

Technically, data democratisation is more than simply thinking about the number of data-driven consumers and data volume but also handling the complexity of varying user behaviours, put in another way, meeting their dynamic, unpredictable, sometimes, over-creative analytical needs — looking at data from different perspectives over several years.

Techy Generation Y, Generation Z

Millennials & The Generation after Millennials

“ Getting goods that fast is certainly advantageous in many ways, but it has also taught the millennial generation — and people from other age groups — that there’s no need to wait for things they want and need. ” See the full article here — “Why More Millennials Should Practice Patience”

Millennials have a reputation for job-hopping. Unattached to organisations and institutions, people from this generation — born between 1980 and 1996 — are said to move freely from company to company, more so than any other generation.

Why do they contribute such an ever high employee turnover rate in history?

One of the striking reasons ( based on some observations of my peers) is the massive disconnect between our experience in our personal lives and our experience at work with data.

One of the founders of Dremio shared his view on this as well when he was sharing the story regarding why they created Dremio.

He said, all these people have this great experience in their personal lives where they go home and they can ask a question on Google, get an answer 1 second later, and then they come to work, and it takes a month to be able to create a new visualisation or a new dashboard, or creating a new machine learning model. So in our personal lives, we have this instant gratification, this amazing experience with data. And we open our smartphones, use applications to book travels within minutes. And so forth.

Are those young, less patient, but highly talented folks going to stay if everything is 100x slower than Google speed?

Millennials and the generations coming after have been advised to practice patience, and patience is a virtue. But, this is how we were growing up and how we’ve been “nurtured” by the world around us.