Building a Cloud Data Application with Snowflake
By: Tomáš Sobotík
Tags: Data Analytics, Cloud, Business Intelligence, AWS, Snowflake

Project technical progress

We have come a long way since project start. We started as small POC with focus on fast delivery of first use case to validate the idea behind. We got 10 weeks at the beginning to deliver first insights to customers. 10 weeks where we had to build everything, starting with infrastructure part, over Data pipelines and DB work, up to dashboard development which goes hand in hand with building up the whole reporting platform. Main focus was on speed. There was no time to do fancy things like long discussion about how we gonna do whole DevOps or building strong automation behind the scenes.

But as time goes we try to move to more automated solution with less maintenance. We are getting there step by step. We have started to build automated test cases using Selenium which helps us to speed up the deployment process as well as the testing itself. We have been adding more and more logging for auditing purposes. We have implemented auto scaling to be prepare for thousands of customers. We have added load balancer to eliminate application downtime. We are slowly getting into solution which will be more like CI/CD pipeline than traditional deployment model. We have still a long journey ahead of us to get there.

Use case description

What kind of problem does this project solve? In general it tries to provide valuable data insights about invoices (timelines & trends, cost segmentation, services overview) and usage data overview, aggregated through various dimensions. All of that with many filters which makes finding the proper data set for the analysis even easier.

In usage data area the most valuable is roaming cost data analysis because in today’s world which is full of unlimited subscriptions for mobile services, the roaming is one of very few items on your invoice how you can significantly affect your bill. Analysis of roaming data in terms of finding outliers and subscribers on incorrect price plan can bring serious cost savings.

Why would any company make such data available to customers? And even helping them to get correct understanding of the data? This is about building trust partnership between both parties. Enterprise appreciate their customers and trying to provide them the best service they can in order to make them happy. Using available data to improve own services seems to be right move. The other side — customers then strengthen their trust in company when they see additional effort to maximally adjust the provided service even though it might end up with lower revenue for the service provider but it will brings much bigger customer satisfaction and happiness. Then there is no reason to leave. 👉🏻 Win : Win

Data monetization is the new Holy Grail of data apps

Everyone speaks about it, everyone wants it, but in the end not many corporate projects are able to achieve that. We are talking about data monetization. Many project I have been part of in last 6 or 7 years have this in the pipeline as something nice to have and point which we want to reach once we finish this and that. We have never reach that point.

Even though our use case is more than suitable for monetization you might be saying that it is not your case. You might have data where the potential is not clearly visible. But I believe that data monetization is much easier in today’s cloud world and especially with features which for instance Snowflake offers to support it. Features like Snowflake Data MarketPlace or Snowflake Data Exchange. Maybe you do not have cool project and unique data but definitely you have something what is important for your partners or customers. Thanks to those features it is easier than ever to share your data and now it is even possible to offer them publicly. There might be more organizations and projects who might consider your data interesting and the data may help them to solve their own challenges. It could be just one of your dimension, just a small dataset.

Next Move

In last 14 months we have built robust foundation which is prepared to serve to thousands of customers. Solution is already maximally secured by custom data encryption with data separation thanks to Snowflake reader accounts. Whole architecture is ready to scale to meet all customer demands where scaling is prepared in all architecture layers starting with ECS auto-scaling for our application docker image , up to Snowflake scaling possibilities where we are ready to scale up (bigger virtual warehouse) same as scale out thanks to Multi-cluster Warehouses. But we are not done yet. There is plenty of things which waits for us.

More automation

We want to slowly get more into CI/CD and DataOps. It requires more automation from our end. We have already focused on automated testing where we use Selenium and Jypyter Notebooks . On daily basis we run defined test cases to validate the application functionality same as data correctness. There is still room to improve our data quality verifications by leveraging the features which Dataiku offers for this (metrics, checks, statistics) or adding another special tool like Amazon Deequ.

In terms of deployment pipeline we need to improve the part related to deployment of Snowflake objects which is now heavily manual and very similar situation is with Dataiku projects.

UI Improvements

There is always something to improve on UI to make the app more usable for users. We are planning to work more on filter area where we can try to offer saved filters or more intuitive filtering. We have also plans for improvements in app navigation and home page organization.

Data Science

We have a best tool for Data Science and Machine Learning as a part of our architecture but we have not been using its ML possibilities yet. There is a big room for ML use cases.

New dashboards

We are constantly getting the customer feedback and requirements for new features. There will be definitely more dashboards.

Wrap-up

And that’s it! I wanted to show and tell how the small idea at the beginning can grow into real product within the year thanks to mainly cloud based technologies, decoupled architecture approach and DataOps way of working. I think we have been able to perfectly combine the right tools and services to create a great product in reasonable amount of time. Starting with AWS — industry leader in cloud infrastructure with hundreds of services to choose from. Then Snowflake — simple, scalable, cost effective DWH as a service with many great features and possibilities where the scalability works as a charm! With Dataiku Data Science studio we can create data pipelines quickly and easily. Last one, Apache Superset — open source reporting platform with zero cost where you can build whatever you want because it is open source. 🤟🏻

Last but not least, nothing would be possible without great team behind the scenes! So Thank you Ruchi, Gustavo, Arul and Jan. It is really a pleasure to be part of the team. 🙏 And thanks to Heine for his endless support without that there would be no project at all.