Data Science for Startups: R -> Python
By: Ben Weber
Tags: Startup, Python, Data Science, Towards Data Science, AWS

Source: Yuri_B at pixabay.com

One of the pieces of feedback I received for my blog series Data Science for Startups was that Python would be a better choice for data scientists joining a startup. This makes a lot of sense if Python is already your go to language for performing data science tasks. In my case, I have much more experience in R and wanted to provide an introduction to working with startups using a language that I’ve previously used to solve problems.

Now that I’ve completed the series and turned it into a book, I want to start digging into Python as a scripting language for data science. For now I still prefer Java for productizing models, using DataFlow, but that preference may change as I become more familiar with the language. I’d like to port some of my previous articles to Python from R, to provide an introduction for a broader audience. Here’s my main motivation for exploring Python:

Startup Tooling: Many startups are already using Python for production, or portions of their data pipelines. It makes sense to also use Python for performing analysis tasks.

Many startups are already using Python for production, or portions of their data pipelines. It makes sense to also use Python for performing analysis tasks. PySpark: R and Java don’t provide a good transition to authoring Spark tasks interactively. You can use Java for Spark, but it’s not a good fit for exploratory work, and the transition from Python to PySpark seems to be the most approachable way to learn Spark.

R and Java don’t provide a good transition to authoring Spark tasks interactively. You can use Java for Spark, but it’s not a good fit for exploratory work, and the transition from Python to PySpark seems to be the most approachable way to learn Spark. Deep Learning: I’m interested in Deep Learning, and while there are R bindings for libraries such as Keras, it’s better to code in the native language of these libraries. I previously used R to author custom loss functions, and debugging errors was problematic.

I’m interested in Deep Learning, and while there are R bindings for libraries such as Keras, it’s better to code in the native language of these libraries. I previously used R to author custom loss functions, and debugging errors was problematic. Python Libraries: In addition to the deep learning libraries offered for Python, there’s a number of other useful tools including Flask and Bokeh. There’s notebook environments that can scale including Google’s Colaboratory and AWS SageMaker.

There’s two additional topics that I’d like to cover that I did not provide content for in the initial series:

Virtualization: Once you start running large jobs, you need a better environment for scaling up to work with large data sets. I used Google’s DataFlow in the initial series, but want to present tools that are useful for scaling up analysis when working interactively.

Once you start running large jobs, you need a better environment for scaling up to work with large data sets. I used Google’s DataFlow in the initial series, but want to present tools that are useful for scaling up analysis when working interactively. Spark: I’d like to explore more of the Spark ecosystem, including tools such as the recently announced MLflow. Spark provides a nice environment for working with large scale data, and more easily moving from exploration to production tasks.

To start, I plan on revisiting my previous posts that were R heavy, and provide a port of these posts to Python. Here are the topics in my original series that need to be translated to Python:

Many of these sections can be translated directly, but posts such as Business Intelligence will require using different libraries, such as Bokeh instead of Shiny. I won’t update the sections on DataFlow, since those are authored in Java. However, it is possible to write DataFlow tasks using Python. Instead of porting Java to Python, I’ll explore new tools for productizing work such as Spark and SageMaker.

The goal of this post is to motivate my transition to Python and to provide an introduction to getting up and running with a Jupyter notebook. Given my new focus on virtualization, I also wanted to show how to work with a remote machine on AWS. The remainder of this post discusses how to spin up an EC2 instance on AWS, set up Jupyter notebooks for remote connections, and query data from BigQuery in Python.

Setting up Jupyter

There’s a number of great IDEs available for Python, such as PyCharm. However, I’m going to be focusing on Jupyter, since it’s a notebook environment and many of the tools that are used for scalable data science are based on notebooks, such as DataBricks for Spark, Colaboratory, and SageMaker. It may be useful to start with an IDE when learning the basics of the language, but it’s good to become familiar with notebook environments given the popularity of this type of environment for large-scale tools.

One of the common tasks discussed when getting started with Python is setting up a virtual environment in order to install Python and any necessary libraries, using tools such as virtualenv. It’s a good practice to set up a virtual environment when using Python, because there may be conflicts between libraries, you may need to run multiple versions of Python, or you may want to create a fresh install to start over. Docker is another option, but is much more heavyweight that virtualenv. For this post, I’ll discuss launching an EC2 instance on AWS for setting up a Python 3 environment. This is also much more heavyweight than using virtualenv, but it provides the ability to scale up the size of the machine if necessary when working with larger data sets. It’s also a good opportunity to become more familiar with AWS and getting started with virtualizing data science tasks.

Security is another important consideration when setting up a notebook environment, since you don’t want your workspace to be open to the world. The most secure way of connecting to a Jupyter notebook when using AWS is to set up an SSH tunnel with port forwarding, which ensures that clients can only connect to the notebook if they have the required private key. Another option is to open up the notebook to the open web, but restrict which machines can connect to the EC2 instance. I’ll present the later approach in this post, since it requires fewer steps, but strongly recommend the former approach for any real task.

Launching an EC2 Instance

This post assumes that you’ve already created an AWS account. AWS provides a number of free-tier options that you can use to become familiar with the platform. EC2 is a service that you can use to spin up and connect to virtual machines. We’ll spin up an EC2 instance and use it to host Jupyter notebooks. Documentation on using EC2 is available here.

Launching an EC2 instance on AWS

Perform the following steps from the EC2 Dashboard to launch a machine:

1. Click “Launch Instance”

2. Select “Amazon Linux AMI 2018.03.0”

3. Select “t2.micro”, which is free tier eligible

4. Click “Review and Launch”

5. Click “Launch” and then select a key for connecting via SSH

6. Click “Launch Instances” and then “View Instances”

We’ll also need to edit the machine’s configuration in order to allow inbound Jupyter connections on port 8888. By default, an EC2 instance only allows inbound connections on port 22 using a private key for authentication. Documentation on configuring security groups is available here.

Setting up an exception for port 8888 for Jupyter.

We’ll allow inbound connects to the EC2 instance on port 8888 for only the host machine. Perform the following steps from the EC2 dashboard:

1. Select your EC2 instance

2. Under “Description”, select the security (e.g. launch-wizard-1)

3. Click “Actions” -> “Edit Inbound Rules”

4. Add a new Rule: change the port to 8888, under source, select “My IP”

5. Click “Save”

After performing these steps, you now have an EC2 instance up an running, with an open port available for connecting to Jupyter. In order to connect to your instance, you’ll need a tool such as Putty. Instructions for Windows users are available here. Another option is to using Java to connect directly to your instance. However, I haven’t used this before, and it’s deprecated in Chrome.

Installing Jupyter

Once you’re able to connect to your instance, you’ll need to set up Python 3 and Jupyter. The instance should already have Python 2.7 installed, but we want to use a newer version. Run the following commands to install Python 3, pip, and Jupyter:

sudo yum install -y python36

python36 --version

curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

sudo python36 get-pip.py

pip3 --version

pip3 install --user jupyter

The Amazon linux distro is based on RedHat, so yum is used to install software. Pip is Python’s package manager, which we’ll use to install libraries in a later step. I’ve also included statements to check the install versions.

Private IP and Public IP details for my test instance.

By default, Jupyter only accepts connections from the local machine. This can be changed by using the --ip command. For this to work on an EC2 instance, you’ll need to use the private IP of the machine. This is 172.31.60.173 in the figure above. You can enable remote connections and launch Jupyter using the following command:

jupyter notebook --ip Your_AWS_Prive_IP

When Jupyter launches, it mentions a specific URL to copy into your browser in order to run notebooks. Since we configured Jupyter to use the Private IP, this is what will be printed out when launching Jupyter. To connect to the machine, you’ll need to copy the link, but also modify the IP from the private IP to the public IP, which is 18.206.244.128 in the above figure.



The Jupyter Notebook is running at:

http://Your_AWS_Private_IP:8888/?token=dd921a6c4ebc3a3bb90fcd8ac44394692a80685705da30e3 # output from running the commandThe Jupyter Notebook is running at:

http://Your_AWS_Public_IP:8888/?token=dd921a6c4ebc3a3bb90fcd8ac44394692a80685705da30e3 # switch the internal IP to external to run in a browser

If everything was successful, you should now see Jupyter in your browser.

The Jupyter landing page on my EC2 instance.

You now have a notebook running for authoring interactive python 3 scripts!

Connecting to BigQuery

Now that we have a Jupyter notebook up and running, the first step I wanted to explore before digging into Business Intelligence and EDA was figuring out how to pull data from BigQuery to a Pandas DataFrame for analysis. While Google does provide a library for working with GCP in Python (google-cloud-storage), we don’t need to use it for this specific use case. The pandas-gbq library makes it easy to accomplish the task of pulling result sets locally for analysis. To set up this library, as well as matplotlib for plotting, run the following commands in the terminal:

pip3 install --user pandas-gbq

pip3 install --user matplotlib

We can switch back to the Jupyter environment and start working with BigQuery public data sets. The first line in the snippet below loads the pandas library, which is installed as a dependency when installing pandas-gbq. The second line identifies the product id to use when running queries. The third command runs a query that pulls a data sample from the Natality data set, and assigns the result set to a pandas DataFrame. The final step displays information about the data types in the df object. When you first run this script, the notebook will prompt you for a key as part of authenticating with Google. Click on the provided link and copy the key to proceed.

import pandas as pd projectid = "your_project_id" df = pd.read_gbq("""SELECT *

FROM `bigquery-public-data.samples.natality` limit 100""",

projectid, dialect = "standard") df.info()

For R users, info() in Python is similar to str(), and describe() is similar to summary(). If you want to get a feel for the shape of the data set, you can use some of the built-in plotting functions for DataFrame objects, which generate plots using matplotlib. The snippet below shows how to select two columns from the DataFrame, and plot histograms for these attributes:

df[['weight_pounds', 'mother_age']].hist()

This generates the following matplotlib visualization:

Histograms of attributes in the Natality data set.

We now have an environment set up where we can pull data from BigQuery into pandas DataFrames for analysis. The workbook is available on Github here and a rendered version is available here. I’ve already been using Jupyter notebooks for some of my analysis, so having a Python connection set up with BigQuery in this environment is a good first step to porting my R analysis to Python. The next step is learning how to perform exploratory data analysis, visualization, and modeling using Python libraries.

Conclusion

As a next step in my series on Data Science for Startups, I’m exploring the use of Python instead of R for scripting tasks. Python provides many great libraries that motivate this port, including deep learning, a more gentle introduction to Spark, and scalable notebook environments. This post showed how to set up an EC2 instance and remotely work with Jupyter notebooks. It also showed how to pull data from BigQuery. Now that I’ve laid some groundwork for getting started with Python, I can start porting some of my posts to Python from R.