Creating Data Lake & Discovering Your Data Using AWS Lake Formation Capabilities
By: Adit Modi
Tags: Data Lake, Big Data, Analytics, AWS, Big Data Analytics

Creating Data Lake & Discovering Your Data Using AWS Lake Formation Capabilities Adit Modi Follow Oct 16 · 9 min read

[Article 2 of 6 in series Big Data Applications on AWS]

Previous Article in the Series

AWS Lake Formation makes it easy for you to set up, secure, and manage your data lakes also data discovery using the metadata search capabilities of Lake Formation in the console, and metadata search results restricted by column permissions.

My Background: Cloud and Big Data Enthusiast | 4x AWS Certified | 3x Oracle Cloud Certified | 3x Azure Certified | Big Data Certified.

Data lakes are an increasingly popular way to create a single repository to store and analyze both structured and unstructured data. AWS Lake Formation makes it easy for you to set up, secure, and manage data lakes. This Article walks you through the creation and exploration of a data lake using Lake Formation:

Creating the data lake

o Adding data to your data lake

o Creating catalog databases

o Adding tables from Amazon S3 to catalog databases

Editing and adding metadata within the catalog

o Editing standard metadata

o Adding custom metadata

Steps

Create the data lake

In the AWS Lake Formation console, in the left navigation pane, choose Register and ingest, Data lake locations. Select a single S3 bucket to house several independent data sources in your data lake.

Add data to your data lake

Now that you have an S3 bucket configured as a storage resource for Lake Formation, you must add data to your data lake. You can add data to your data lake’s S3 bucket storage resource using AWS SDKs, AWS CLI, the S3 console, or a Lake Formation blueprint.

With Lake Formation, you can discover and set up the ingestion of your source data. When you add a workflow that loads or updates the data lake, you can choose a blueprint or template of the type of importer to add. Lake Formation provides several blueprints on the Lake Formation console for common source data types to simplify the creation of workflows. Workflows point to your data source and target and specify the frequency that they run.

Sample Datasets are provided as follows

New York City Taxi and Limousine Commission (TLC) Trip Record Data Amazon Customer Reviews

Add Amazon customer reviews to your data lake

AWS hosts a registry to help people share and discover a variety of datasets. For this post, copy a subset of the Amazon customer reviews dataset into your data lake. You don’t have to copy the complete reviews dataset, only the smaller 226-MB portion of watch reviews. You need two copies of this data in your data lake to simulate separate “production” and “test” databases.

If you have not already, install and configure the AWS CLI with IAM user access keys that include permission to read from S3 and write to your Lake Formation S3 bucket. Copy the source to your data lake:

# Replace with your bucket name: YOUR_BUCKET=lf-metadata-demo aws s3 cp \ s3://amazon-reviews-pds/parquet/product_category=Watches/ \ s3://$YOUR_BUCKET/amazon-reviews-prod/amazon-reviews/ --recursive aws s3 cp \ s3://amazon-reviews-pds/parquet/product_category=Watches/ \ s3://$YOUR_BUCKET/amazon-reviews-test/amazon-reviews/ --recursive

3. In the S3 console, confirm that your S3 bucket now contains your two Amazon reviews datasets.

4. Inspect the contents of the folders. The datasets are in Parquet format.

Add New York taxi ride history to your data lake

Much as you did with the Amazon customer reviews dataset, copy a small subset of New York taxi ride history from the Registry of Open Data on AWS into your data lake:

Copy the source data to your data lake:

# Replace with your bucket name: YOUR_BUCKET=lf-metadata-demo aws s3 cp "s3://nyc-tlc/trip data/green_tripdata_2018-02.csv" s3://$YOUR_BUCKET/ny-taxi/trip-data/green_tripdata_2018-02.csv

2. In the S3 console, validate that your S3 bucket contains CSV data for NY taxi trips.

Create catalog databases

You have created an S3 bucket to act as your data lake storage backend and added data to the bucket. However, this data is not readily available in Lake Formation until you catalog the data.

Lake Formation maintains a Hive-compatible data catalog of data within your data lake. Before you can catalog data within your S3 storage backend or use Lake Formation data importers (discussed later) to push data to S3, you must first create a database.

A Lake Formation database is a logical construct to which you later add tables. Each table contains a mapping to one or more objects in S3 that, collectively, represent that table. Tables also contain basic metadata including but not limited to file format, S3 location, column headings, and column types. Lake Formation users can also optionally define arbitrary key-value pairs for tables and columns to better describe the data and act as query-able attributes for data discovery.

You can create one or more databases and populate their tables either manually in the console, programmatically using the AWS SDKs or AWS CLI, or automatically by defining AWS Glue crawlers.

define three logical databases:

o amazon-reviews-prod

o amazon-reviews-test

o ny-taxi

Add tables from S3 to your catalog databases

you created three databases in your Lake Formation catalog. However, these catalog databases are empty and do not yet provide information about the specific tables, schema, file formats, or object paths in S3. To add this information, use one of the following two methods:

Manually define your tables in the catalog using the console, SDKs, or AWS CLI.

Use an AWS Glue crawler to search S3 and automatically add discovered tables to your catalog.

For this article, create and manually run one AWS Glue crawler for each of your three datasets in S3 and databases in the Lake Formation data catalog. A detailed walkthrough is outside the scope of this Article. For guidance, see Working with Crawlers on the AWS Glue Console.

As you proceed, please bear the following in mind:

Create one crawler for each of your three datasets. You should be able to accept most of the default crawler settings. However, the S3 path for your crawlers should read:

o s3://YOUR_BUCKET/amazon-reviews-prod/amazon-reviews

o s3://YOUR_BUCKET/amazon-reviews-test/amazon-reviews

o s3://YOUR_BUCKET/ny-taxi/trip-data

Before you run the crawlers to populate your catalog, you must assign them an IAM role. The role grants them permission to read from your data lake’s S3 bucket, write crawler logs to Amazon CloudWatch, and update your data catalog. Regardless of whether you create a new role, or use an existing role, make a note of the IAM role name. You need this information for the next step.

In addition to permissions defined within IAM, you must also explicitly grant IAM principals (roles or users) the ability to modify your Data Catalog from within Lake Formation itself. Conceptually, this is similar to the concept of bucket policies in S3 used with IAM. In the Lake Formation console, under Permissions, choose Data permissions.

Grant your AWS Glue crawlers the ability to modify your Data Catalog. Configure the following fields:

For IAM users and roles, select the IAM roles that you previously used for your AWS Glue crawlers.

For Database, select the amazon-reviews-prod, amazon-reviews-test, and ny-taxi databases.

For Database permissions, select all permissions.

Leave all Grantable permissions unselected.

After your AWS Glue crawlers have permission to modify your Lake Formation data catalog, return to the AWS Glue console and manually run your three crawlers. After a few minutes, the crawlers should complete their runs. Each should add one table to your data catalog:

o amazon-reviews

o amazon-reviews

o trip-data

Metadata search in the console

Search by classification Search by keyword Search by tag: attribute Multiple filter searches Metadata search results restricted by column permissions

In this Article, we demonstrate the catalog search capabilities offered by the Lake Formation console:

Search by classification

Search by keyword

Search by tag: attribute

Multiple filter searches

Search by classification

Using the metadata catalog search capabilities, search across all tables within your data lake. Two share the name amazon_reviews but separately belong to your simulated “prod” and “test” databases, and the third is trip-data.

In the Lake Formation console, under Data catalog, choose Tables. In the search bar, under Resource Attributes, choose Classification, type CSV, and press Enter. You should see only the trip_data table, which you formatted as CSV in your data lake. The amazon_reviews tables do not appear because they are in Parquet format. In the Name column, choose trip_data. Under Table details, you can see that the classification CSV is correctly identified by the metadata search filter.

Search by keyword

Next, search across your entire data lake filtering metadata by keyword.

To refresh the list of tables, under Data catalog, choose Tables again. From the search bar, type star_rating , and press Enter. Now that you have applied the filter, you should see only the amazon_reviews tables because they both contain a column named star_rating. By choosing either of the two tables, you can scroll down to the Schema section, and confirm that they contain a star_rating column.

Search by tag: attribute

Next, search across your data lake and filter results by metadata tags and their attribute value.

To refresh the list of tables, under Data catalog, choose Tables. From the search bar, type department:research , and press Enter. Now that you have applied the filter, you should see only the trip_data table because this is the only table containing the value of ‘research’ in the table property of ‘department’. Select the trip_data table. Under Table details, you can see the tag: attribute of department | research listed under Table properties.

Multiple filter searches

Finally, try searching across your entire data lake using multiple filters at one time.

To refresh the list of tables, under Data catalog, choose Tables. In the search bar, choose Location, type S3 , and press Enter. For this post, all of your catalog tables are in S3, so all three tables display. In the search bar, choose Classification, type parquet , and press Enter. You should see only the amazon_reviews tables because they are the only tables stored in S3 in Parquet format. Choose either of the displayed amazon_reviews tables. Under Table details, you can see that the following is true.

Location: S3

Classification: parquet

Metadata search results restricted by column permissions

The metadata search capabilities return results based on the permissions specified within Lake Formation. If a user or a role does not have permission to a particular database, table, or column, that element doesn’t appear in that user’s search results.

To demonstrate this, first create an IAM user, dataResearcher, with AWS Management Console access. Make sure to store the password somewhere safe. Next, create an IAM group with the AWSLakeFormationDataAdmin policy attached. Finally, add the dataResearcher user to this new group. We will use this user below to sign into our account with the group’s limited access.

you allowed Everyone to view the tables that the AWS Glue crawlers created. Now, revoke those permissions for the ny-taxi database.

In the Lake Formation console, under Permissions, choose Data permissions. Scroll down or search until you see the Everyone record for the trip_data table. Select the record and choose Revoke, Revoke.

Now, your dataResearcher IAM user cannot see the ny-taxi database or the trip_data table. Resolve this issue by setting up Lake Formation permissions.

Under Permissions, choose Data Permission, Grant. Select the dataResearcher user, the ny-taxi database, and the trip_data table. Under Table permissions, check Select and choose Grant. Log out of the console and sign back in using the dataResearcher IAM user that you created earlier. In the Lake Formation console, choose Tables, select the trip_data table, and look at its properties:

You have learned how to use the metadata search capabilities of Lake Formation. By defining specific user permissions, Lake Formation allowed you to grant and revoke access to metadata in the Data Catalog as well as the underlying data stored in S3. Therefore, you can discover your data sources across your entire AWS environment using a single pane of glass. To learn more, see AWS Lake Formation.

hope all of you Understood AWS Architecture and How to Use Big Data with AWS. If you have any doubt regarding the examples provided or are unable to understand how AWS Services make use of Big Data Applications you can contact me.

LinkedIn : https://www.linkedin.com/in/adit-modi-2a4362191/

Github : https://github.com/AditModi

You can view my badges on:

https://www.youracclaim.com/users/adit-modi/badges

Next Article in the Series