How Palantir renews the promise of big data
By: Miaoling Liang
Tags: Technology, Data Analytics, Palantir, Scaleup

Data analytics has been around since the late 1960s when computing and processing costs dropped. In the last decade, we saw another boom of data analytics through the rise of mobile technology, social media, and big data. The new era promised greater analytical powers, by which algorithms far surpassed the human ability to process information; it expanded the need for intelligent assistance, delivered through a capable system like the Iron Man’s JARVIS. Big data revolutionized decision-making: businesses and organizations desire ever greater “visibility” and chase after “data-driven” insights, to improve strategy, operations, and supply chain. Born in the Silicon Valley that started it all, Palantir is a uniquely positioned company catering to the renewed appetite for data analytics.

Palantir started out as a service company, but its business model today is built around a futuristic technology. Despite recent growth in data analytics, few organizations have unlocked the responsive potentials of big data; in fact, many are still on a journey to technological transformations that are fraught with friction. Today’s big data calls for a full suite of data system, query, dashboarding and optimization tools, while companies founded in the last century (e.g. IBM, Oracle, SAP) remain dominant data solutions providers.

Typical data systems and solutions.

Although a number of smaller point solutions have grown to fill in gaps of these ERP-focused players, they remain disparate systems and interrupt the native flow between end-to-end processes. Any one system relies on heavy integrations to communicate with another system; when integrations are not possible, users fall back on manual importing or exporting of data, reverting back to the old days of operating offline. Palantir set out to solve this problem.

For the last decade, technology consulting has been helping disrupted organizations transition to the new era. They help clients translate extensive knowledge and sophisticated operations into “requirements”, a list of capabilities essential for operations, which technology providers take to configure in a standard solution. But not every requirement can be met, because organizations often have a vast amount of human knowledge and expertise that took decades to develop into a competitive advantage. If they want to onboard all operations onto a new system, the alternative is to customize the solution at much higher costs. However, there is a catch: the customized solution is not future proof. It does not get upgraded, increasing the risk of its obsolescence against the speed technology advances today. In addition to trapping the organization in a sunk cost fallacy, customized solutions tend to reinforce existing siloes and rigid processes rather than changing them. For these reasons, technology consultants and their clients tend to stick with a package of standard solutions, while using manual workarounds to accommodate for unique needs that have not been addressed.

More ambitious companies are aware of the limitation of off-the-shelf solutions and decide that the path to cost-efficient and integrated analytics is to do it themselves. They would hire talents to build an analytics center of excellence (CoE) inhouse. Expectedly, demand for data engineers and analysts have soared in recent years, and it is projected to grow by over 30% in the near future.

Nonetheless, there are significant investment risks to inhousing. First, technology development is not the forte of commercial or public organizations; no matter how well staffed, non-tech companies would have limited resources to match the capability of leading technology vendors. Second, the reality of internal software development is often subpar products, delayed timelines, and over-budget costs. Typically, it takes 5 to 10 years for a company with legacy technology to build a full-fledged CoE — by the time it is fully functional and integrated into operations, the market may already be on the next generation of products and capabilities.

The onset of COVID-19 signals to organizations and enterprises that the need to transform and pivot becomes ever more imminent. Yet the challenge remains: rapid changes in the world today often outpaces the ability to adapt. This is why Palantir is a game-changer in the data solutions market. It effectively delivers an analytics CoE with a substantially shorter timeline (2 years or less) by lending a powerhouse of developers and software engineers to its clients. It propels the client to the latest capabilities, rather than leaving them to play catch up every few years. Of course, the challenge to this type of deployment approach is the steep learning curve, which has been reported as the system being “not user friendly” and requiring “technical backgrounds” to use. Palantir has tried to mitigate these risks by lowering the bar for using the solution (e.g. with no code workflows) and providing training services as part of the solution stack.

What Palantir also changed is the bad reputation around customization. Thanks to its subscription model, Palantir is able to achieve “customization without legacy”. They co-develop a wealth (hundreds) of custom applications with the client, and sustain the delivery of updates to them via Apollo.

By design, frequent updates entice clients to the product as its enhanced powers become addictive: clients “found that the more data [Palantir] touches, the more powerful and relevant and useful it becomes.” Combined, customization and the subscription model highly increase the cost of switching, thereby giving Palantir massive incumbent advantage.

Palantir’s final advantage is its Silicon-Valley strategy: scaling. On Investor Day, Palantir announced that its vision is to scale their solution across industries. Scaling customized solutions used to be a challenge because highly specialized industries (e.g. defense, law enforcement, advanced manufacturing, pharma) have complex operations and unique needs that they share much less with the larger base of consumer business. Traditionally, standard solutions scaled by being the lowest common denominator across industries; but Palantir has chosen a different path. It started in the most sophisticated environments and managed a large number of scenarios. As a result its product is more robust and flexible to support a diversity of needs. It would hold a significant advantage over point solutions players in various industries today, because it would bring to the table not just an ETL, a machine learning algorithm, or an optimizer. It seeks to replace all of those tools, while offering comparable, of not better, analytical power.

Iron Man gets intelligence at his finger tips, via JARVIS.

As long as the promise of big data analytics is alive, Palantir will do well, because the market today still wants a smartphone for enterprises — a vertically integrated, one-stop-shop system that underpins end-to-end operations. Not to mention, the future will see the need for automating analysis with technology and programming, as much of data analysis today remains costly and repetitive. While Palantir’s corporate product, Foundry, may join the competition against current leading data solutions (such as Alteryx and Snowflake), its priority and niche will probably remain with the most complex organizations for a while. In spite of the risk of customer concentration, the best environment for Palantir to grow is where players are few and challenges are many. The development and implementation of Palantir’s products will continue to burn capital — as is the case with any highly customized solutions — but when it succeeds in making JARVIS, it would capture the most valuable and untouchable customers for years to come.