How we process and store terabytes of data
By: Jan Antala
Tags: Scaling, Startup, Cassandra, Big Data, Startup Lessons

Spark and Cassandra

As Kafka is a gateway to Pygmalios cloud, Spark and Cassandra is a heart. We use Cassandra cluster and all tables are also replicated across nodes using NodeSync service. It is fully automatic, no manual intervention are needed and also provides repair service.

Storing Raw Data

Kafka Topic messages are read in real-time to our Cassandra database and saved in raw data tables by Spark streaming jobs. Casandra has great write performance and is horizontally scalable so we are ready to increase throughput by adding additional nodes. We store whole history, every message received, to SSD disks in Google Cloud Platform. This data are automatically partitioned by client venue and day, sorted by timestamp and deduplicated by Cassandra clustering keys.

Raw Data Backups: Google Cloud Storage

We are transferring raw data from Cassandra tables to Google Cloud Storage buckets every day in Spark night batch jobs. We read whole Cassandra partitions from last days as data frames and transform them to parquet file format which has direct Spark support and provides efficient data compression so it is great for storing terabytes of data. This jobs are monitored and we can see if everything is ok.

Backup jobs monitoring

You can even use various storage classes for Google Cloud buckets for better cost effectiveness depending on your needs.

The restore service is however a key benefit of cloud storage. We can load backed up data directly from cloud storage bucket and transfer them back to Cassandra or recalculate metrics directly.

And the last benefit of this backup service are statistics. We can track daily usage for all Pygmalios analytics modules by our customers and see how many events occurred and how they perform.

Processing Data

Data are processed both in real-time and in every-day batch jobs, grouped by sessions and stored in processed data Cassandra tables. They are used for final aggregations and stored in aggregated data tables. Aggregated data are relatively small, but thats our goal. All jobs are also monitored and we can see everything runs smoothly.

We also use separate rest API for batch jobs so we can re-run them as we need.

Clearing old data

We can’t store a whole history on Cassandra SSDs, it is expensive and nobody would pay for it so we have to be cost effective. We have separate cleaning batch Sparks jobs and automatically run them on a regular base as we have all raw data backed up in the cloud storage, which is cheaper. We delete all raw and processed data older then 3 months from Cassandra and keep only aggregated data in Cassandra database. We also have batch jobs ready for GCS data read so that is no problem in case of historical data recalculation. It only have to be a part of premium service as it is more expensive.

Aggregated Data Backups: Datastax OpsCenter

We use Datastax backup and restore service for aggregated data only. We have to restore data very fast for our customers in case of emergency. I hope this won’t happen but we have to be ready and perform regular tests. Aggregated data are relatively small so it is easy to achieve it.

DSE OpsCenter Backup service

Conclusion

Scaling is one of the most challenging phases of startup life. It is hard and very long term process. And data processing and storage is the key part. Pygmalios can process and store terabytes of data in various services and is ready for more scaling.