Getting Zeppelin to Work with EMR
Tags: Python, Apache Zeppelin, AWS, Apache Spark

It’s not as easy as it may seem

Apache Zeppelin is a great notebook tool for interactive coding with Scala and Python. It’s the only sane way I’ve found to develop with Apache Spark — definitely more convenient than mucking about with local Spark installations, or (God forbid) pasting code into the interactive shell for testing.

That said, getting everything set up to connect to Zeppelin on Amazon EMR from my Windows laptop was a bit of a hassle, and once everything was set up I did have a few unresolved quibbles.

EMR Cluster Creation

I like Amazon’s Elastic MapReduce service (EMR) because it comes packaged with most everything you need to get started.

Software Configuration Options for EMR

I chose “Advanced Options” on the EMR Console to create the cluster, because I needed to customize the bootstrap action (more on that later).

I unchecked all the software options except for Hadoop, Spark and Zeppelin. That should get me set up for basic Spark development.

If I was in an enterprise setting or repeatedly using the same schemas, I’d definitely consider setting up AWS Glue Data Catalog, which gives you a centralized place to store metadata describing your files. For now I’m just working on a simple one-off project, so I didn’t bother with that.

On Step 2 (Hardware), I ran into my first unexpected error: “instance type m3.xlarge is not supported in the requested availability zone”.

Error: The requested instance type is not supported

It turns out that certain instance types are only supported in certain availability zones — for example, within the Virginia region there are availability zones “us-east-1a”, “us-east-1b”, etc. I couldn’t find clear documentation of which instance types are available in which regions, but I was able to solve it pretty quickly with trial and error.

Specifically, I was using the Singapore region, and the “m3.xlarge” instance type failed in the Availability Zone “ap-southeast-1c”. Changing the AZ to “ap-southeast-1b” got rid of that error.

On step 3, I configured my Bootstrap Action, which was a .sh script I needed to call to install a Python library I planned on using. It’s important to install the necessary libraries in the bootstrap script, because they need to be installed on all the nodes, and bootstrap scripts are the easiest way to deploy and run a script across the whole cluster.

Bootstrap Actions Dialogue

My script basically consisted of one line of code:

#!/bin/bash sudo pip install sklearn

I put that in a .sh file and uploaded it to S3, making sure that the newlines are Linux-style & not Windows-style. On the console, I added that by clicking “Configure and add” under “Bootstrap Actions”.

On Step 4, I chose an EC2 key pair I had already created, and chose some security groups that I had already set up to allow inbound SSH traffic on port 22 coming from my public IP. If you don’t already have the security groups set up, you can let the console automatically create them for you, then edit the Inbound IP Whitelist to accept traffic from the proper CIDR group.

Once that’s done, you can click “Create” and go grab a coffee or equivalent non-caffeinated beverage.

SSH Tunnel Setup

If, like me, you are a Windows user, you have two main options for connecting to EMR. One is to use the charmingly-capitalized PuTTY, the other is to install Cygwin & use the more elegant Linux-style syntax.

I used Cygwin, and this is the command I used to set up an SSH tunnel from an arbitrary port on my machine to the cluster.

ssh -v -i ./PRIVATE_KEY_FILE.pem -ND 8157 hadoop@ec2-52-77-213-141.ap-southeast-1.compute.amazonaws.com

This tunnel is necessary because for security reasons Zeppelin is available only on the master node of the cluster, so without an SSH tunnel you won’t be able to connect. (Note that it may be possible to configure your EMR cluster to allow public access to Zeppelin — but using an SSH tunnel seems much more secure).

You’ll also need to set up FoxyProxy on your browser. Extensions are available for most of the major browsers, so go ahead and install it.

The AWS instructions for configuring FoxyProxy give you the needed configuration file, which can be loaded using the “Import/Export” feature in the FoxyProxy options. Copy/paste the following text in a file, then follow the instructions from that link:

<?xml version="1.0" encoding="UTF-8"?> <foxyproxy> <proxies> <proxy name="emr-socks-proxy" id="2322596116" notes="" fromSubscription="false" enabled="true" mode="manual" selectedTabIndex="2" lastresort="false" animatedIcons="true" includeInCycle="true" color="#0055E5" proxyDNS="true" noInternalIPs="false" autoconfMode="pac" clearCacheBeforeUse="false" disableCache="false" clearCookiesBeforeUse="false" rejectCookies="false"> <matches> <match enabled="true" name="*ec2*.amazonaws.com*" pattern="*ec2*.amazonaws.com*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" /> <match enabled="true" name="*ec2*.compute*" pattern="*ec2*.compute*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" /> <match enabled="true" name="10.*" pattern="http://10.*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" /> <match enabled="true" name="*10*.amazonaws.com*" pattern="*10*.amazonaws.com*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" /> <match enabled="true" name="*10*.compute*" pattern="*10*.compute*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false" /> <match enabled="true" name="*.compute.internal*" pattern="*.compute.internal*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false"/> <match enabled="true" name="*.ec2.internal* " pattern="*.ec2.internal*" isRegEx="false" isBlackList="false" isMultiLine="false" caseSensitive="false" fromSubscription="false"/> </matches> <manualconf host="localhost" port="8157" socksversion="5" isSocks="true" username="" password="" domain="" /> </proxy> </proxies> </foxyproxy>

When you have everything enabled and configured, you should be able to type http://your-master-public-dns-name:8890/ directly into your web browser, and the Zeppelin interface should show up.

The Zeppelin interface

Coding with Spark in Zeppelin

Now that you have Zeppelin working, go ahead and create a notebook.

You’ll notice that Zeppelin is organized into “interpreter groups”, which offer varying API options for the same framework. For example, Spark has separate interpreter groups for the Scala, Python, and R APIs.

For this particular project I wanted to code in Python, which I discovered can be done by putting %pyspark at the beginning of a block of code.

It turns out that by default you need to put the %pyspark at the beginning of each and every block — otherwise it will try to use the default interpreter, which uses the Scala API.

I’m embarrassed to admit how long I spent trying to change the settings to default to Pyspark first. Although you can “edit” the interpreter group via the GUI, it doesn’t provide an option for editing the order of the interpreters.

Several StackOverflow answers (1) (2) seemed to provide promising solutions — however, due to the nature of open-source software, the answers didn’t work in the version of Zeppelin that I happened to be running today, March 21, 2019, in Amazon EMR. I edited zeppelin-site.xml , I edited interpreter.json , I restarted Zeppelin… trust me, I tried.

Incidentally, there seems to be an interesting ̵b̵u̵g̵ feature in AWS that causes interpreter.json to be overwritten any time Zeppelin is restarted. In exploring this, I ended up on some arcane Zeppelin forums, at which point I decided I could just suck it up and deal with typing out %pyspark every time.

With that out of the way, everything worked nicely. I was able to code & troubleshoot my Spark scripts in real time, running them block by block on the EMR cluster.

Conclusion

Have you figured out a better Zeppelin setup than this one? If so, please comment — I’d love to learn how to make the setup process simpler.

On the plus side, now that I know what the EMR cluster settings need to look like, I can just export the CLI command and automate that part of the process.

So if there is pain in the configuration, it’s a one-time thing, and very much worth it for the pleasure of having a nice environment to work on my notebooks in.