How to Easily Perform Pandas Operations on S3 With AWS Data Wrangler | by Ahmed Besbes | Towards Data Science

I’ve always found it a bit complex and non-intuitive to programmatically interact with S3 to perform simple tasks such as file readings or writings, bulk downloads or uploads or even massive file deletions (with wildcards and stuff).
My typical use-cases deal with Pandas dataframes I download from (or save to) S3 in any of these three formats: CSV, JSON and Parquet for analysis and processing.
While searching for an alternative to boto3 (which is, don’t get me wrong, a great package to interface with AWS programmatically) I came across AWS Data Wrangler, a python library that extends the power of Pandas to AWS by connecting dataframes to AWS data related services such as Athena, Glue, Redshift, DynamoDB, EMR ... and most importantly (for me, at least) S3.
In this post, I’ll be reviewing AWS Data Wrangler’s functionalities to easily interface Pandas with S3.
If you’re not an AWS user, S3 stands for Amazon Simple Storage Service. It’s a suite of web services that you can use to store any amount of data, at any time, from anywhere on the web.
It provides any developer access to highly scalable, reliable, fast and inexpensive data storage.
→For a complete overview of S3, have a look at the official documentation.
→To be able to use S3 and reproduce the following code snippets you need to set up an AWS account.
Before using AWS Data Wrangler, you’ll have to create an IAM user.
IAM stands for Identity and Access Management: it’s a service that enables you to manage access to AWS services and resources securely. IAM allows you to create users and groups and use permissions to allow or deny access to AWS resources.
Once you create an IAM user, AWS will provide you with two credentials: Access key IDand Secret access key. You'll have to use them to configure an AWS profile on your machine.
AWS Data Wrangler will then use this profile to programmatically access AWS services on your behalf.
To create an IAM user, go to the AWS console and search for IAM. On the top-left sidebar, click on Users .
This will list the users you’ve already created. To add a new one, click on Add users.
Enter a name for this user and assign it Programmatic access.
Then, add it to a group to define its set of permissions. For simplicity, give it Administrator Access, but you may want to limit this to a restricted set of permissions in general.
Skip the tag steps and validate the user creation.
Once the user is created, credentials will appear on the screen: Access key ID and Secret access key. We'll use them to configure an AWS profile.
To configure an AWS profile, you’ll have to first install the AWS CLI by following this guide and then enter the following command
aws configure --profile ahmed
A prompt will ask you to enter the previous credentials. Just do it and leave the region name and output format to their default values.
Now, the AWS profile is properly configured.
To make AWS Data Wrangler use it, we’ll have to set up a default session with boto3 at the start of our experiments: just leave these lines at the beginning of your script or notebook.
The subsequent commands will use the permission given to this profile.
First things first, let’s install AWS Data Wrangler.
pip install awswrangler
Before running any command to interact with S3, let’s look at the current structure of my buckets.
Nothing really fancy, just a bunch of buckets I use for personal projects.
AWS Data Wrangler can perform basic operations on these buckets:
checks if an object exists:
>>> wr.s3.does_object_exist("s3://ahmedbesbes.com/plot.html")True
recursively lists objects inside a bucket:
>>> wr.s3.list_objects("s3://ahmedbesbes.com")['s3://ahmedbesbes.com/app.mp4', 's3://ahmedbesbes.com/plot.html', 's3://ahmedbesbes.com/pyldavis.html']
lists directories inside a bucket
>>> wr.s3.list_directories("s3://sozy")['s3://sozy/assets/', 's3://sozy/images/', 's3://sozy/saved_images/']
shows the region name of each bucket
>>> wr.s3.get_bucket_region("ahmedbesbes.com")'eu-west-3'
get information about each object
>>> wr.s3.describe_objects("s3://ahmedbesbes.com/plot.html")
This will output a JSON object describing the file. It includes attributes such as the content type, size as well as permission metadata.
The describe_objectsmethod can also take a folder as input. In that case, it will return a list of JSON objects, each one describing each file in the folder.
Now comes the fun part where we make Pandas perform operations on S3.
Read files
Let’s start by saving a dummy dataframe as a CSV file inside a bucket.
This is as simple as interacting with the local file system. One cool thing here: if the /csv/sub-folder/ didn't already exist, AWS Data Wrangler will create it automatically.
Read multiple CSV files at once:
This is actually quite simple. All you have to do is to pass a list of remote paths. Behind the scenes, AWS Data Wrangler will concatenate the four dataframes along the row axis (axis=0).
Use prefixes
Assume that you have 1000 CSV files inside a folder and you want to read them all at once in a single dataframe. To do this, you can pass the path to the folder to the read_csv method. AWS Data Wrangler will look for all CSV files in it.
wr.s3.read_csv(f"s3://{bucket}/csv/")
Delete objects
You can delete objects by using the delete_objects methods
This method accepts Unix shell-style wildcards in the path argument.
Assume, for example, that you want to delete all the 1000 CSV files that are in a specific folder.
Here’s how you’d do it:
wr.s3.delete_objects(f"s3://{bucket}/folder/*.csv")
You can perform these same operations on JSON and Parquet files as well
Just replace with:
wr.s3.read_csv with wr.s3.read_json or wr.s3.read_parquet
wr.s3.to_csv with wr.s3.to_json or wr.s3.to_parquet
Download objects
AWS Data Wrangler makes it very easy to download objects from S3. This is the main reason why I’ve started using it.
To download a remote object, just use the download method. This method accepts as the first argument the remote path to the object and as the second argument the local path where the object will be saved.
Alternatively, you can download an object in binary mode.
upload objects
Uploading objects follows the same logic.
This post is a collection of notes I took while using AWS Data Wrangler to per. I hope you found them useful.
In later posts, I’ll be reviewing other AWS services such as DynamoDB and EMR and how to interact with them using AWS Data Wrangler.
For the time being, here are some resources worth the read if you want to dig deeper.
AWS S3: https://aws.amazon.com/s3/?nc1=h_ls
AWS IAM: https://aws.amazon.com/iam/?nc2=type_a
AWS Data Wrangler documentation: https://aws-data-wrangler.readthedocs.io/
Github: https://github.com/awslabs/aws-data-wrangler
See you next time!