The Growing Data Opportunity
By: Ry Walker
Tags: Cloud Computing, Wearables, Data Science

Computers are Everywhere

Because we all have super computers in our pocket, on our desk, and soon in everyday objects all around us, responsibility for information technology has become dispersed. People are taking responsibility for their own technology, and IT departments are willingly releasing power, especially as the “bottom-up” SaaS product strategy gains popularity (i.e. Slack).

As more organizations master their own data, the population of people inside organizations learning to wield that data will grow.

Companies are certainly becoming more ambitious with data but are the people being empowered? Most people don’t have access to relevant customer data and if they do it’s usually well after the fact.

Cloud computing

Companies are moving applications and databases to the cloud in droves. Cloud computing is in a deployment period.

For example, GE has already moved 1,000 apps to AWS and the question of moving the remaining 75% of their apps is one of when not if.

As more organizations adopt cloud computing, a side effect is that more of their data will be in the cloud, and this data is now staged to flow with more fluidity.

“Big data in the cloud democratizes the use of big data beyond large companies.” — Peter Levine

Data in the cloud gives you the ability to give access to anyone. And if everyone has access to the organization’s real-time data, the organization becomes more agile, allowing it to iterate more rapidly. This has an effect of flattening of the organization, fostering a collaborative culture, and making decisions quicker.

SaaS/Microservices Proliferation

We are moving towards a tipping point where more of an organization’s data is generated and stored outside of the company than within, but this varies by company.

With SaaS, this effect is fractal. If you look inside the company that you think has your data, the majority of their data (which is really your data) is in their SaaS tools, and so on down the rabbit hole (a few levels, at least.)

And even within your organization, your developers are likely adopting microservice architectures, using “the right database for the job” (read up on Hadoop, MongoDB, mySQL, Redshift, Cassandra, InfluxDB — databases are exploding). Using the right database for the job is a technically sound decision but there are side effects. There can be several databases per application, and usually these applications will not share databases with each other (a concept known as an “anti-pattern”).

This is a hard truth for some but a truth nonetheless: your data is becoming spread out and there’s no going back to centralization.

Computer cognition

Finally, the most powerful trend that is really just starting — software can learn from data. Once we see self-driving cars in everyday life, we will scrutinize every job that humans do and ask ourselves “can a self-learning computer do this work?”

In the near future, analytics and business intelligence will be “table stakes” because computers are ready to do real work for us. And machine learning is very data hungry. It wants data from as many sources as you can provide. A lot of the time, providing more data is more effective than trying to improve algorithms. There is a hunger for all data we can gather.

“Just as all real-world workflows became software, all software will become analytical. Computers will have cognition; machines will have intelligence.”

— Zetta Venture Partners

Will you isolate?

You have a choice: be on an isolated island or put your data on the wire. It reminds me of those early days of the Internet. When I would have serious conversations with large companies on whether they really needed to have a website (especially the brick and mortar companies)—conversations that I often lost (but they all came around eventually).

I see the same thing now with data. Any company that creates value also creates valuable data, and the smart ones will be prepared to share it.

“Data is incredibly valuable. It helps create superior products, it forms a barrier to entry, and it can be directly monetized.” — Leo Polovets

99.9% of companies aren’t even close to being prepared to capitalize on this data opportunity. A good early move is to begin gathering your data into a flexible, real-time firehose that you can wield in an agile way.

This is why we’re building Astronomer — to facilitate this vision.