Loading Data From Amazon S3 To Amazon Redshift | by Shafiqa Iqbal | Towards Data Science

In this tutorial, you walk through the process of loading data into your Amazon Redshift database tables from data files in an Amazon S3 bucket from beginning to end.
In this tutorial, you’ll do the following:
Connect to Amazon Redshift using SQL Workbench/J.
Launch an Amazon Redshift cluster and create a database table and schema.
Use COPY commands to load the table from the data files on Amazon S3.
Step 1: Sign in to your AWS account and go to Amazon Redshift Console.
Step 2: On the navigation menu, choose CLUSTERS, then choose Create cluster. The Create cluster page appears.
Step 3: Choose dc2.large for the node type in the Compute-optimized section. Then choose 1 for the Nodes.
Step 4: In the Cluster details section, specify values for Cluster identifier, Database port, Master user name, and Master user password.
Step 5: Go to IAM roles and create a new role with AmazonS3ReadOnlyAccess policy.
Step 6: Choose create cluster.
Step 7: In order to be able to connect to your redshift cluster, make it publically accessible. Go to Network and Security, edit publically accessible tab, Allow instances and devices outside the VPC connect to your database through the cluster endpoint and attach an Elastic IP.
Step 8: Open the security group attached to your cluster (in Network and security) and allow access to your Public IP in inbound rules.
Step 1: Go to https://www.sql-workbench.eu/downloads.html and download generic package for all systems.
Step 2: Open exe file in windows and jar file in linux/mac.
Step 3: Once SQL Workbench/J is opened, Choose File, and then choose Connect window.
Step 4: Choose Create a new connection profile. Choose Manage Drivers. The Manage Drivers dialog opens. In the Name box, type a name for the driver.
Step 5: Add your jar file by importing it from computer.
Step 6: Enter your JDBC URL copied from Redshift Console’s Connection details.
Step 7: Enter your username and password and select Auto-commit checkbox to True.
Step 8: Test your connection. It will connect to your AWS Redshift cluster.
Here are the final configurations.
Step 1: Download allusers_pipe.txt file from here. Create a bucket on AWS S3 and upload the file there.
Step 2: Create your schema in Redshift by executing the following script in SQL Workbench/j.
create schema schema-name authorization db-username;
Step 3: Create your table in Redshift by executing the following script in SQL Workbench/j.
create table schema-name.users(userid integer not null distkey sortkey,username char(8),firstname varchar(30),lastname varchar(30),city varchar(30),state char(2),email varchar(100),phone char(14),likesports boolean,liketheatre boolean,likeconcerts boolean,likejazz boolean,likeclassical boolean,likeopera boolean,likerock boolean,likevegas boolean,likebroadway boolean,likemusicals boolean);
Step 4: Run the following COPY command. The COPY command includes a placeholder for the Amazon Resource Name (ARN) for the IAM role, your bucket name, and an AWS Region, as shown in the following example.
copy users from 's3://<myBucket>/allusers_pipe.txt'iam_role 'aws_iam_role=<iam-role-arn>'delimiter '|' region '<aws-region>';
you can create multiple tables with this structure and populate them accordingly.
Step 5: Now try the example query, you should see the following result.
select * from users;
In this tutorial, we loaded S3 files in Amazon Redshift using Copy Commands. We connected SQL Workbench/J, created Redshift cluster, created schema and tables.
For upcoming stories, you should follow my profile Shafiqa Iqbal.
That’s it, guys! Have fun, keep learning & always coding!