How I Improved Performance Retrieving Big Data With S3-Select | by Anna Geller | Towards Data Science

I’ve recently come across a feature in S3 that is particularly useful when working with Big Data. You can write a simple SQL query to select specific columns and filter for specific rows to retrieve only the data you need for your application. In this article, I will demonstrate how you can accomplish that using Python’s boto3 library.
Imagine the following scenario:
You are regularly getting a single large CSV file that is stored in an S3 bucket. This file includes marketing data about all countries you’re online shop is active in.
However, you have an application running inside of AWS Lambda that only needs marketing data from one particular country.
Normally, you would have to download this entire large file and filter out data inside of your application. The problem is that your Lambda function may not have enough RAM to read this entire large file into memory. There are some ways to tackle this, such as reading and filtering data in chunks or moving the function to a Docker container, but the simplest and most cost-effective solution in this scenario would be to use the S3-Select feature. Here is how it works.
First, since I actually don’t have any marketing data, we will use data about taxi trips from New York [1], and we will filter for suspicious records with unknown payment type, encoded as payment_type==5according to the data dictionary.
Let’s explore how many rows the entire file has as compared to only those with payment_type==5.
This means that we are only interested in 12 out of half a million records. It would be a massive waste of resources to download and read the entire dataset into memory just to get those 12 rows!
To demonstrate S3-Select in action, we first upload our large CSV file [1] to an S3 bucket:
Now we can use S3-Select to get only data with payment type equal to 5, i.e., we retrieve from S3 only the data we are interested in — data with unknown payment type. And the best part is that it’s all defined in a simple SQL query:
in our query, we select only columns we need for our use case
we filter only for payment_type='5' — note that in S3, all columns in the flat file are considered text, so make sure to enclose your value with quotation marks '5'.
In the code snippet above, we had to define the InputSerialization='CSV' to specify that this is the format of our S3 object. Also, we set FileHeaderInfo to 'Use', which ensures that we can use column names in our S3-select query.
By using the OutputSerialization parameter, we define that we want our output to be comma-separated, which allows us to store the result in a separate CSV file. If you want to use S3-Select in an API, you might prefer the JSON format.
Some extra caveats:
The S3-select only returns the records, not the column names (header), so on line 27, I ensured that the same columns that I defined in the query are also included as the first row in our TARGET_FILE.
S3 select returns a stream of encoded bytes [2], so we have to loop over the returned stream and decode the output: .decode('utf-8').
Let’s cross-check the TARGET_FILE — it should only have 12 rows.
The result:
Nr of rows: 12    ID  distance  tip  total0    1       9.7    0  31.801    1      10.0    0  30.802    1       0.4    0   7.803    1      13.6    0  40.804    1       0.5    0   7.805    1       1.7    0  11.806    1       6.4    0  22.807    1       4.0    0  16.808    1       1.7    0  11.309    1       5.6    0  22.8010   1       7.3    0  30.4211   1       6.7    0  24.80
You may ask: if S3 is so easy to use, why do we need Athena to query a data lake? Here are the main differences between those two:
Athena can query multiple objects at once, while with S3 select, we can only query a single object (ex. a single flat file)
With Athena, we can encapsulate complex business logic using ANSI-compliant SQL queries, while S3-select lets you perform only basic queries to filter out data before loading it from S3.
Athena supports more file formats and more forms of file compression than S3-Select. For instance, S3-select supports only CSV, JSON, and Parquet, while Athena additionally allows TSV, ORC files, and more.
S3-select works only with the S3 API (ex. by using Python boto3 SDK), while Athena can be queried directly from the management console or SQL clients via JDBC.
Athena allows many optimization techniques for better performance and cost-optimization, such as partitioning, columnar storage, while S3-select is a very rudimentary query that just nothing but filtering data.
S3-select can be queried directly, while Athena requires the definition of a schema.
In short, the benefits of this API are:
reduce IO — thus better performance
reduce costs due to smaller data transfer fees.
In this article, we discussed S3-select, which allows filtering data stored in S3. S3-select should only be used when dealing with a single file and when you only need to select specific columns and filter for only specific rows from a flat-file.
If this article was helpful, follow me to see my next articles.
In the article linked below, I discuss various options to transfer large amounts of data between S3 buckets.
medium.com
Resources:
[1] TLC Trip Record Data: https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page
[2] Boto3 Docs: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.select_object_content
[3] AWS Docs: https://docs.aws.amazon.com/AmazonS3/latest/API/API_InputSerialization.html and https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html