Load and Query CSV File in S3 with Presto | by Yifeng Jiang | Towards Data Science

It is such a simple and common task in big data that I thought folks must have done this a thousand times, so when a customer asked me this, I went straight to the internet trying to find some good examples to share with the customer. Guess what? I couldn’t find one! So I decided to write one myself.
A typical data ETL flow with Presto and S3 looks like:
Upload CSV files into S3.Load the CSV files on S3 into Presto.(optional) Convert to analytics optimised format in Parquet or ORC.Run complex query against the Parquet or ORC table.
Upload CSV files into S3.
Load the CSV files on S3 into Presto.
(optional) Convert to analytics optimised format in Parquet or ORC.
Run complex query against the Parquet or ORC table.
In this blog, I use the NewYork City 2018 Yellow Taxi Trip Dataset. The dataset has 112 million rows, 17 columns each row in CSV format. Total size is 9.8GB.
Here is some example data:
head -n 3 tlc_yellow_trips_2018.csvVendorID,tpep_pickup_datetime,tpep_dropoff_datetime,passenger_count,trip_distance,RatecodeID,store_and_fwd_flag,PULocationID,DOLocationID,payment_type,fare_amount,extra,mta_tax,tip_amount,tolls_amount,improvement_surcharge,total_amount2,05/19/2018 11:51:48 PM,05/20/2018 12:07:31 AM,1,2.01,1,N,48,158,2,11.5,0.5,0.5,0,0,0.3,12.81,05/19/2018 11:22:53 PM,05/19/2018 11:35:14 PM,1,1.3,1,N,142,164,2,9,0.5,0.5,0,0,0.3,10.31,05/19/2018 11:37:02 PM,05/19/2018 11:52:41 PM,1,2.2,1,N,164,114,1,11,0.5,0.5,3.05,0,0.3,15.35
I assume you have completed a basic Presto and S3 setup. You also need to set up the Hive catalog in Presto for it to query data in S3. If you haven’t, please take a look at my blog Presto with Kubernetes and S3 — Deployment.
Create a directory in S3 to store the CSV file. We can use any S3 client to create a S3 directory, here I simply use the hdfs command because it is available on the Hive Metastore node as part of the Hive catalog setup in the above blog.
Run the below command from the Hive Metastore node. Change bucket name to match your environment. Note I specified s3a:// as the directory path schema so that hdfs command creates the directory on S3 instead of HDFS.
hdfs dfs -mkdir -p s3a://deephub/warehouse/nyc_text.db/tlc_yellow_trips_2018
Upload the CSV file to S3 under the directory we just created. Any S3 client should work, I use s5cmd, a very fast S3 client, to upload the CSV file into my S3 directory. Here I use FlashBlade S3, so I specified endpoint-url to my FlashBlade data VIP.
s5cmd --endpoint-url=http://192.168.170.12:80 cp tlc_yellow_trips_2018.csv s3://deephub/warehouse/nyc_text.db/tlc_yellow_trips_2018/tlc_yellow_trips_2018.csv
In order to query data in S3, I need to create a table in Presto and map its schema and location to the CSV file.
Launch Presto CLI:
presto-cli --server <coordinate_node:port> --catalog hive
Create a new schema for text data using Presto CLI.
presto> CREATE SCHEMA nyc_text WITH (LOCATION = 's3a://deephub/warehouse/nyc_text.db');
Create an external table for CSV data. You can create many tables under a single schema. Notes:
CSV format table currently only supports VARCHAR data type.
I set skip_header_line_count = 1 to the table property so that first line header in our CSV file is skipped. Remove this property if your CSV file does not include header.
Presto’s CSV format support requires metastore.storage.schema.reader.impl=org.apache.hadoop.hive.metastore.SerDeStorageSchemaReader in the metastore-site.xml Hive Metastore configuration file.
presto> CREATE TABLE hive.nyc_text.tlc_yellow_trips_2018 (    vendorid VARCHAR,    tpep_pickup_datetime VARCHAR,    tpep_dropoff_datetime VARCHAR,    passenger_count VARCHAR,    trip_distance VARCHAR,    ratecodeid VARCHAR,    store_and_fwd_flag VARCHAR,    pulocationid VARCHAR,    dolocationid VARCHAR,    payment_type VARCHAR,    fare_amount VARCHAR,    extra VARCHAR,    mta_tax VARCHAR,    tip_amount VARCHAR,    tolls_amount VARCHAR,    improvement_surcharge VARCHAR,    total_amount VARCHAR)WITH (FORMAT = 'CSV',    skip_header_line_count = 1,    EXTERNAL_LOCATION = 's3a://deephub/warehouse/nyc_text.db/tlc_yellow_trips_2018');
Now I can query the CSV data.
presto> SELECT * FROM nyc_text.tlc_yellow_trips_2018 LIMIT 10;
At this point, I can connect Tableau to visualise data in the Presto table. However, because CSV format table only supports VARCHAR data type, it may expose limits to Tableau. If this is the case, convert CSV to Parquet or ORC format (see below). Parquet or ORC tables generally have better performance than Text/CSV tables.
This is an optional task, but it is recommended if the data will be queried multiple times. By converting text data into analytics optimised format in Parquet or ORC, it does not only improve query performance, but also reduce server and storage resource consumption.
Presto is good for simple conversions that can be done in SQL. For those with complex business logics that cannot be easily done with SQL (e.g., requires Java/Python programming), it is better to use Apache Spark. In this example, I simply convert text to Parquet format without introducing any complex business logic, so I will use Presto for the conversion.
A common practice for managing data in Presto is to use different schemas for your raw text (CSV/TSV) tables and optimised (Parquet/ORC) tables. So I will create a new schema nyc_parq for the Parquet table.
Create a S3 directory for the new schema. Change bucket name to match your environment.
hdfs dfs -mkdir -p s3a://deephub/warehouse/nyc_parq.db
Create the nyc_parq schema in Presto CLI.
presto> CREATE SCHEMA nyc_parq WITH (LOCATION = 's3a://deephub/warehouse/nyc_parq.db');
Create a Parquet table, convert CSV data to Parquet format. You can change the SELECT cause to add simple business and conversion logic.
presto> CREATE TABLE hive.nyc_parq.tlc_yellow_trips_2018COMMENT '2018 Newyork City taxi data'WITH (FORMAT = 'PARQUET')ASSELECT     cast(vendorid as INTEGER) as vendorid,    date_parse(tpep_pickup_datetime, '%m/%d/%Y %h:%i:%s %p') as tpep_pickup_datetime,    date_parse(tpep_dropoff_datetime, '%m/%d/%Y %h:%i:%s %p') as tpep_dropoff_datetime,    cast(passenger_count as SMALLINT) as passenger_count,    cast(trip_distance as DECIMAL(8, 2)) as trip_distance,    cast(ratecodeid as INTEGER) as ratecodeid,    cast(store_and_fwd_flag as CHAR(1)) as store_and_fwd_flag,    cast(pulocationid as INTEGER) as pulocationid,    cast(dolocationid as INTEGER) as dolocationid,    cast(payment_type as SMALLINT) as payment_type,    cast(fare_amount as DECIMAL(8, 2)) as fare_amount,    cast(extra as DECIMAL(8, 2)) as extra,    cast(mta_tax as DECIMAL(8, 2)) as mta_tax,    cast(tip_amount as DECIMAL(8, 2)) as tip_amount,    cast(tolls_amount as DECIMAL(8, 2)) as tolls_amount,    cast(improvement_surcharge as DECIMAL(8, 2)) as improvement_surcharge,    cast(total_amount as DECIMAL(8, 2)) as total_amountFROM hive.nyc_text.tlc_yellow_trips_2018;
Depending on the data size, this conversion may take time. Once it is done, I can query the Parquet data.
presto> SELECT * FROM nyc_parq.tlc_yellow_trips_2018 LIMIT 10;
Confirm the Parquet table’s schema. Note columns are with desired type in this table.
presto> describe nyc_parq.tlc_yellow_trips_2018;        Column         |     Type     | Extra | Comment-----------------------+--------------+-------+--------- vendorid              | integer      |       | tpep_pickup_datetime  | timestamp    |       | tpep_dropoff_datetime | timestamp    |       | passenger_count       | smallint     |       | trip_distance         | decimal(8,2) |       | ratecodeid            | integer      |       | store_and_fwd_flag    | char(1)      |       | pulocationid          | integer      |       | dolocationid          | integer      |       | payment_type          | smallint     |       | fare_amount           | decimal(8,2) |       | extra                 | decimal(8,2) |       | mta_tax               | decimal(8,2) |       | tip_amount            | decimal(8,2) |       | tolls_amount          | decimal(8,2) |       | improvement_surcharge | decimal(8,2) |       | total_amount          | decimal(8,2) |       |(17 rows)
Finally, I configure my analytics application / Tableau to use the optimised nyc_parq schema and Parquet tables.
As data size grows larger (e.g., over TBs), organising data in a way that is optimal for query performance becomes more important. Using Parquet or ORC formant is one optimisation, there are others such as:
Predicate pushdown.
Partitioned table.
Data sorting in Parquet and ORC.
Join strategy and cost based optimisation.
These topics are not Presto specific, they apply to most storages and query engines, including S3 and Presto. The details of these topics are beyond the scope of this blog. Stay tuned on my blogs.