Data Lake with AWS S3 — Part 3/3. AWS Data Tiering, Data Format, Automate…
By: Piyush M Agarwal
Tags: Aws S3, AWS, S3 Glacier, AWS Lambda, Data Lake

Data Lake with AWS S3 — Part 3/3

Optimising Cost with Data Tiering

If you think about data, data typically has different access patterns over its life cycle. Real-time data streaming in is what we call hot data. You’re going to want to process it with very low latency in near real time, t hat’s going to require high performance compute, high performance storage that is going to be more costly

It’s just a fundamental law of IT that the faster you want to go, the more you’re going to pay.

So but then as that data ages, and you start to collect more and more bulk data and maybe analyse it more and frequently, you can store that in a cooler and cooler tier, that’s much more cost optimised and ultimately you may want to just archive lots of datasets because you’re going to not analyse them in real time.

Or if you move into things like machine learning, you may train and build your models and then periodically you’ll want to go back and maybe retrain those models. As you get more sophisticated you’ll realise there are other data sets you want to add in to them, or, your business is changing. What you can do is you can archive massive amounts of data in glacier or deep archive and then restore it whenever you want to rebuild your models. So ultimately you want to set up a tiering strategy.

AWS Data Tiering

That’s going to encompass all of these and give you a way to optimise cost over a data’s life cycle. An example would be, you could use EMR with Hadoop on local EBS (Elastic Block Store) storage for the very highest performance levels, put bulk data into S3 for frequent processing (maybe migrated to infrequent access), but keep it online for infrequent processing and then archive massive data sets that you might use for machine learning and store those in glacier and only restore them when you need them.

AWS S3 Storage Classes

AWS has been focusing on making S3 better for these types of workloads by introducing more storage classes that are more optimised for these different scenarios. S3 standard is the workhorse for most data-lakes applications. As the data gets colder, you may want to lifecycle it to infrequent access, then subsequently set up a life cycle to glacier deep archive so that you can really cost optimise over time. S3 allows you to automate that and choose the right storage class that matches your requirements.

AWS S3 Intelligent Tiering

AWS recently introduced S3 Intelligent Tiering, where the storage is smart enough to monitor how your data is used, learn how the data is used and tear the data automatically to save money, without AWS account manager having to actively management it.

AWS Deep Archive

This is a great place to store historical data that you either might have tried to write to tape and store off site because you wanted to keep it but you couldn’t afford to. With deep archive, which is about a dollar per terabyte per month, you can now afford to keep essentially all of your data. You don’t have to manage tape, it has the same level of durability and reliability of all of AWS S3 storage services, and you can recover that data in hours.

For example, if you’re building self-driving cars, you’ve collected hundreds of terabytes of data and built your models, post that you don’t want to keep all that historical data online because you’re not going to use it anytime soon. If a new camera comes out, a new type of sensor, a new algorithm you want to test, you could go back and recall a massive data set out of deep archive, train your models and deploy new models and then re-archive that data.

When you think about use cases like machine learning, where the more data you can use to train the model, the more effective you’re going to be, deep archive becomes a very important component of your data lifecycle strategy.

Data Formats

Data formats are important for a data-lake. There’s no such thing as the best data format, all have some form of trade-off depending on the workloads and tools. While a lot of data comes in in unstructured formats or loosely structured formats like CSV or JSON or XML, there are a couple of optimised formats you’re probably going to want to consider. AWS prescribes columnar compressed formats like Parquet or ORC.

Most of the industry standardised around parquet, both because it has inherent compression. If you take a given a JSON file and convert it to parquet, you’re probably going to get about 1/4th compression ratio, which means much less cost to store that data over time. But ultimately because these are self-describing formats, they’re much more efficient to process so you scan much less data.

Setting up Automated Data Pipelines

So in the spirit of wanting to help you build a more optimised data-lake, with AWS you can tackle this problem head on and really cut down the time it takes from having raw data to having data that you can actually drive analytics and machine learning with,

ETL Pipelines with AWS Glue

AWS Glue comes essentially it allows you to set up a catalog prep and ultimately transform your data in an automated serverless fashion so you define your jobs, schedules on how you’re going to crawl and discover data and put it in a catalog and apply metadata to that. And then once you define all that it’s automated, you don’t have to worry about managing physical infrastructure. You’re essentially setting up automated data pipeline and only paying for the resources that you use to do that.

Event-diven Glue Pipeline

Also data’s going to come in concurrently and on very different schedules. So ultimately you’re probably going to want to build an event driven pipeline to do this data transformation and get it ready for use. And so essentially Lambda, AWS’s serverless compute capability, is going to be the engine to do that. So when your data arrives, you’re probably not going to want to kick off a glue job for every bit of data that comes in, but you may want to batch it and do it like every 10,000 objects that come in.

So what you can do is set up Lambda functions that will count and decide when to intelligently start crawl jobs and Glue will discover that data. You can then trigger an optimised job to start to transform it into a more optimised format like parquet catalog and load them into the data-lake. If you’re doing something like a data warehouse, maybe load those into the data warehouse and do that all in a very automated event, driven fashion, where you can write intelligent event triggers to do this in a way that matches your business flow processes.

Data-Lake Security

There are a lot of challenges to building a secure data-lake. This is one of the hardest aspects of doing it successfully. You’ve not just got to control access to the data, prevent the data from being lost, or accidentally deleted, or overwritten and be able to track and audit. So these breakdown into data challenges and management challenges, and AWS has a wide number of tools to both control access to data, meet compliance requirements, as well as manage and audit and track this whole process. This topic deserves deeper insight into various AWS services viz. AWS CloudTrail, Macie, GuardDuty, etc. Given the depth of this sub-topic let’s park it for now 😊

Final Thoughts…

Finally to wrap all this up this all sounds complex, and for a lot of people it is when you think about building a data-lake.

Typical Steps for Building Data-Lake

Even though AWS has tried to automate a lot of this, provide a lot of tools, it’s still not easy. You’ve got to set up your storage foundation layer in S3, apply security there, start to set up all those ingest policies, then set up cleansing and data prep and cataloging processes, start to apply role-based security and then you know ultimately start to analyse that data.

Those are all very typical steps and it can take weeks to months to start to do this at large scale all in the spirit of democratising data-lakes and making it easier for people to get value out of their data more quickly getting to time to results.

Continue Reading…

…

[1] Amazon S3 https://aws.amazon.com/s3/

[2] Amazon QuickSight https://aws.amazon.com/quicksight

[3] Amazon S3 Glacier https://docs.aws.amazon.com/amazonglacier/latest/dev/introduction.html

[4] Amazon S3 pricing https://aws.amazon.com/s3/pricing/

[5] Data Formats https://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-dataformats.html

[6] Amazon Macie https://aws.amazon.com/macie

[7] Set up centralized monitoring for DDoS events and auto-remediate noncompliant resources https://aws.amazon.com/blogs/security/set-up-centralized-monitoring-for-ddos-events-and-auto-remediate-noncompliant-resources/