How to create and run an EMR cluster using AWS CLI | by Tran Nguyen | Towards Data Science

“Apache Spark is a unified analytics engine for large-scale data processing”. Spark is considered as “the king of the ‘big data’ jungle” with various applications in data analysis, machine learning, streaming and graph analytics, etc. There are 4 different Spark modes: (1) Local mode: Spark on a single machine such as a laptop for learning syntax and prototyping the project; the other 3 modes are cluster manager modes: (2) Stand-alone mode is used for working on a private cluster; (3) YARN and (4) Mesos modes are used when sharing cluster with a team. With stand-alone mode, Spark is deployed on a private cluster such as EC2 on Amazon Web Service AWS. A Spark cluster includes multiple machines. To use Spark code on each machine, Spark and its dependencies need to be downloaded and installed manually. With Elastic Map Reduce service, EMR, from AWS, everything is ready to use without any manual installation. So instead of using EC2, we use the EMR service to set up Spark clusters.
I did spend many hours struggling to create, set up, and run the Spark cluster on EMR using AWS Command Line Interface, AWS CLI. Although there are a few tutorials for this task that I found or were provided through courses, most of them are so frustrating to follow. Some are not clear enough; some miss some crucial steps; or assume the learners know some prior knowledge about AWS, CLI configurations, etc. After successfully setting up and getting the cluster running, I realized that this task is actually not that complicated; and we should get it done with ease. I don’t want to see people struggling with this anymore. Therefore, I decided to make this tutorial.
This article assumes that you already have some working knowledge of Spark, PySpark, command-line environment, and AWS. Specifically, this article is for readers who know why they need to create the Spark cluster :). For more about Spark, please read the reference here.
This is a long and detailed tutorial. In brief, all the steps include:
Create an AWS accountCreate an IAM userSet up credentials in EC2Create an S3 bucket to store log files produced by the clusterInstall AWS CLI package awscliSet up AWS CLI environment (create the credentials and config files)Create an EMR clusterAllow SSH AccessCreate an SSH connection with the master node of the clusterStart using the EMR cluster
Create an AWS account
Create an IAM user
Set up credentials in EC2
Create an S3 bucket to store log files produced by the cluster
Install AWS CLI package awscli
Set up AWS CLI environment (create the credentials and config files)
Create an EMR cluster
Allow SSH Access
Create an SSH connection with the master node of the cluster
Start using the EMR cluster
Please feel free to skip any step that you already know. The Jupyter notebook version of this tutorial, together with other tutorials on Spark and many more data science tutorials could be found on my Github. Now, let’s dive in!
I am so glad that many of you found this tutorial useful. It is my honor to discuss any issue you encountered during the EMR creating process with you all. Based on some of our discussions, here are some updates for this tutorial.
Some cautions:
1. For this tutorial, AWS regular account should be used instead of AWS Educate account. The AWS regular account provides users the full access to AWS resources and IAM roles; while educate account has some limited access.
2. This is your responsibility for monitoring usage charges on the AWS account you use. Remember to terminate the cluster and other related resources each time you finish working. I have implemented the EMR cluster many times; and this tutorial should produce no cost or cost less than $0.5 on AWS.
3. The AWS console and Udacity content are subjected to change/upgrade through time, so I would recommend you to search AWS site and your Udacity’s lessons for any updated tutorials/guidelines. This tutorial was generated according to the AWS console in July, 2020.
4. This tutorial is produced using Chrome and Mac OS X. It should not be so much different on Windows platform.
Some of the materials are from the Data Engineer nanodegree program on Udacity.
Some ideas and issues were collected from the Knowledge — Udacity Q&A Platform and the Student Hub — Udacity chat platform.
Creating a regular AWS account if you don’t have one already. The instruction is very easy to follow on the AWS site.
Log into your AWS account.
(Optional) To increase security on your AWS resources, you can configure and enable a virtual multi-factor authentication (MFA) device according to the easy AWS guide here.
From the AWS console, click onService, type ‘IAM’ to go to IAM console:
=> Choose User => Add user => Enter a user name such as 'emr_user' , Choose Access type as Programmatic access, and then Next: Permissions.
Click on Attach existence policies directly page, type and set permission as Administrator Access and then choose Next: Tags.
Skip this tag page and choose Next: Review => Choose Create user => Save the user name, Access Key and Secret Access Key.
We will use this IAM user and the access and secret key to set up and access AWS via AWS CLI.
From the AWS console, click on Service, type 'EC2' to go to EC2 console
Choose Key Pairs in Network & Security on the left panel => Choose Create key pair
Type the name for the key pairs, such as "emr-cluster", File format: pem => Choose Create key pair . After this step, a .pem file will be automatically downloaded, in this case, the file name is emr_cluster.pem. We will use this file in Step 6.
This will be the AWS bucket to store the log files produced by the cluster we will create and set up. If we don’t specify an S3 bucket, an S3 bucket will be automatically created for us when an EMR cluster is created and run.
From the AWS console, click on Service, type 'S3' and go to S3 console => Choose Create bucket => Enter a name for the bucket such as 's3-for-emr-cluster', choose your preferred region, such as ‘US West(Oregon). Keep other options at default to create a bucket.
Note that for the best performance and to avoid any error, remember to use the same AWS region/sub-region for all of your work (on S3, EC2, EMR, etc.)
On the terminal, install awscliusing the commandpip install awscli
Type aws help to check if the installation is correct. The installation should be successful if the output is as below:
This step will help us automatically access AWS on the awscli environment using the user credentials we got on Step 2 above.
There are 2 approaches for this set-up: manually creating the credentials and config files (approach 1) or creating these files using theaws command (approach 2). You can use either one.
Approach 1:
Create the credentials file on the terminal as follow (you can use nano or any other text editor of your choice):
On the terminal, navigate to desire folder, usually the root directory and create a hidden directory such as aws :
$mkdir .aws (period to denote a hidden directory)
Change to that directory $cd .aws
Create acredentials file using nano: $nano credentials Type the content of the credentials file as followed (Replace the key ‘EXAMPLE_ID’ and secret key ‘EXAMPle_Key’ with the ones produced for the user ‘emr-user’ in Step 2):
Use Ctrl + X, and then Y to save the file and exit nano.
Create the config file:
$nano config Type the content of the config file as followed (The region we use would be the same we use to create an S3 bucket in Step 4):
[default]region=us-west-2
Use Ctrl + X, and then Y to save the file and exit nano.
On the terminal, type $aws configure and type in the required information as followed:
AWS Access Key ID [None]: (Enter your access key from the user 'emr-user' created in Step 2)AWS Secret Access Key [None]: (Enter your secret key from the user 'emr-user' created in Step 2)Default region name [None]: us-west-2 (The same region used in Step 4)Default output format [None]:
The 2 files will be automatically created in the hidden folder .aws, usually located at root directory as follow:
$cd ~/.aws
Type$ls to verify the existence of the 2 files. The information should be the same as in Approach 1, we can check the content by typing:
$cat credentials
$cat config
To connect to the cluster, we need the .pem file created in step 3. Move the .pem file we downloaded in Step 3 to a desire location used for the project. For me, I put it to the same location as credentials and config files in the hidden folder .aws located at the root directory ~/.aws/emr-cluster.pem )
$ mv ~/Downloads/emr-cluster.pem .
When using this .pem file to set up the ssh, there would be an error if the .pem file is too open. An example of warning would be: “Permissions 0644 for ‘~/.aws/emr-cluster.pem’ are too open. It is required that your private key files are NOT accessible by others. This private key will be ignored.”
In that case, we need to change the permission for this .pem file using the command: sudo chmod 600 ~/.aws/emr-cluster.pem
To verify if we successfully install awscli and set up the credentials, type some aws commands such as
List all IAM user: $aws iam list-users
List all buckets in s3: $aws s3 ls This command will list all the s3 bucket that we have in AWS:
Now we are ready to create our EMR cluster on the terminal. On the terminal, type the command:
aws emr create-cluster --name test-emr-cluster --use-default-roles --release-label emr-5.28.0 --instance-count 3 --instance-type m5.xlarge --applications Name=JupyterHub Name=Spark Name=Hadoop --ec2-attributes KeyName=emr-cluster  --log-uri s3://s3-for-emr-cluster/
The explanation for the EMR script components:
--name: the name for the cluster, in this case, is 'test-emr-cluster'
--use-default-roles: use the default service role (EMR_DefaultRole) and instance profile (EMR_EC2_DefaultRole) for permissions to access other AWS services
--release-label emr-5.28.0: build a cluster with EMR version 5.28.0
--instance-count 3 and--instance-type m5.xlarge: build 1 Master node and 2 Core nodes of type m5.xlarge
--applications Name=JupyterHub Name=Spark Name=Hadoop: install JupyterHub, Spark, and Hadoop on this cluster
--ec2-attributes KeyName=emr-cluster: configures Amazon EC2 instance configurations, KeyName is the EC2 instance name that we set up in step 3 (Set up credentials in EC2 and get the .pem file). In this case, the name is emr-cluster.
--log-uri s3://s3-for-emr-cluster/: specify the S3 bucket where you want to store log files. In this case, the S3 bucket is 's3-for-emr-cluster'. This field is optional (as explained in Step 4).
Since EMR clusters are costly, we can put the option --auto-terminate to auto-terminate the cluster when all the actions on the cluster are done. For this to work, we also need to specify the bootstrap action in the command using --bootstrap-actions Path="s3://bootstrap.sh". When you use auto-termination, the cluster starts, runs any bootstrap actions that you specify, and then executes steps that typically input data, process the data, and then produce and save the output. When the steps finish, Amazon EMR automatically terminates the cluster Amazon EC2 instances. If we don't put any bootstrap action, we should remove this field. The reference for bootstrap could be found in detail on the AWS site.
Desired output after we create an EMR cluster would be:
Using the ClusterId to check status and information of the cluster (j- EXAMPLECLUSTERID) using the command:
aws emr describe-cluster --cluster-id j-EXAMPLECLUSTERID
We should wait for a few minutes for the cluster to be available (Status changes to ‘available’) before proceeding to the next step.
***Update 1: If you run into the problem that "EMR_DefaultRole is invalid" or "EMR_EC2_DefaultRole is invalid" error when creating an Amazon EMR cluster, you may have to delete the roles and the instance profile; then recreate the roles as this instruction. Thank Trevor S. for discussing this issue.
***Update 2: To make the cluster available for the notebook on EMR, we may need to include the SubnetIds in — ec2-attributes:
aws emr create-cluster --name test-emr-cluster --use-default-roles --release-label emr-5.28.0 --instance-count 3 --instance-type m5.xlarge --applications Name=JupyterHub Name=Spark Name=Hadoop --ec2-attributes SubnetIds=subnet-YOURSUBNET,KeyName=emr-cluster  --log-uri s3://s3-for-emr-cluster/
(Thank Saverio G. for discussing me about this issue.)
How to find SubnetIds?
Click on the VPCSubnets tab on the EMR console to choose from the list.
Another safe way for the beginner: Create a similar EMR cluster using the AWS console. Then look at the “Network and hardware” session on the Summary page of that cluster to see the Subnet ID:
You can access Subnets via VPC Dashboard on AWS. More about subnet and VPC could be found on the AWS site.
From the AWS console, click on Service, type EMR, and go to EMR console.
=> Choose Clusters => Choose the name of the cluster on the list, in this case, is test-emr-cluster
On the Summary tab, scroll down to see the part Security and access, choose the Security groups for Master link
Choose the Security group ID for ElasticMapReduce-master
Scroll down on Inbound rules, Edit inbound rules => For safety, delete any SSH rule if have, then choose Add Rule => Choose Type: SSH, TCP for Protocol and 22 for Port Range => For source, select My IP => Choose Save .
Approach 1:
On the terminal, check the id of the cluster using the command
aws emr list-clusters
Use this command to connect to the cluster. Remember to specify the right path of the .pem file:
aws emr ssh --cluster-id j-EXAMPLECLUSTERID --key-pair-file ~/.aws/emr-cluster.pem
If we see the screen with the EMR letter as below, congratulations, you successfully create, set up, and connect to the EMR cluster using AWS CLI !!!!
Close the connection to the cluster using the command $logout .
Approach 2:
From the AWS console, click on Service, type EMR, and go to EMR console.
Choose Clusters => Click on the name of the cluster on the list, in this case test-emr-cluster => On the Summary tab, Click the link Connect to the Master Node Using SSH.
Copy the command shown on the pop-up window and paste it on the terminal.
Remember to replace ~/emr-cluster.pem with the location and filename of the private key file (.pem) we have set up. For example
ssh -i ~/.aws/emr-cluster.pem hadoop@ec2-xx-xxx-xx-xx.us-west-2.compute.amazonaws.com
If we see the screen with the EMR letter, congratulations, you successfully create, set up, and connect to the EMR cluster using AWS CLI !!!!
Create a simple Spark task, for example, create a Spark data frame containing the time as type string and then convert that column to a different format.
On the cluster terminal, create the file test_emr.py
$nano test_emr.py
Copy and paste this script to test_emr.py
### Content of the file 'test_emr.py'from pyspark.sql import SparkSession, functions as Fif __name__ == "__main__":    """        example of script submited to spark cluster    """    spark = SparkSession.builder.getOrCreate()        df = spark.createDataFrame([('01/Jul/1995:00:00:01 -0400', ),('01/Jul/1995:00:00:11 -0400',),('26/Jul/1995:17$                            ('19/Jul/1995:01:56:43 -0400',), ('11/Jul/1995:12:50:18 -0400',)], ['TIME'])    df = df.withColumn("date", F.unix_timestamp(F.col('TIME'), 'dd/MMM/yyyy:HH:mm:ss Z').cast('timestamp'))    # show the dataframe    df.show()        #### stop the spark, otherwise the program will be hanged     spark.stop()
The content of the file as shown on nano text editor:
Submit the script on the cluster terminal using the command:
$spark-submit --master yarn ./test_emr.py
When the task runs, it may get overwhelming with any information on the terminal, which is normal in Spark. The output could be found among the info logging:
Remember to terminate the cluster when not using it anymore.
Type logout to log out of the cluster. Then terminate the cluster using this command:
$aws emr terminate-clusters --cluster-id j-EXAMPLECLUSTERID
Notice that we can easily create a cluster with the same configurations again using the AWS CLI command, which could be found when clicking on the AWS CLI export tab when we choose the terminated cluster `test-emr-cluster` on the EMR console:
I hope no one has to struggle to create, set up, and run the Spark cluster on EMR using AWS CLI anymore. Please let me know if you have any issues or find anything wrong with this tutorial.
The Jupyter notebook version of this tutorial, together with other tutorials on Spark and many more data science tutorials could be found on my Github. Please enjoy!