Python done the R (Markdown) way
By: Vít Gabrhel
Tags: Rstudio, Pandas, Data Science, Plotly, Python

Photo by Vit Gabrhel

Introduction

Starting to work with a new tool may not be easy, especially when that “tool” means a new programming language. At the same time, there might be an opportunity to build upon something already known and so make the transition smoother and less painful.

In my case, it was the transition from R to Python. Unfortunately, my former colleagues, pythonians, did not share my genuine fondness of R. Also, whether R enthusiasts like it or not, Python is a widely used tool in data analysis/engineering/science and beyond. So, I concluded that learning at least some Python is a reasonable thing to do.

For me, the first steps were maybe the most difficult ones. Residing in the comfort of RStudio, IDEs like Pycharm or Atom did not feel familiar. This experience led to the decision to begin in the well-known environment and test its limits when it comes to working with Python.

To tell the truth, I did not end up using RStudio as the weapon of choice for using Python in a general setting. Hopefully, the following text will deliver the message why. However, I am convinced that for some use-cases, like integrating R and Python in an ad hoc analysis R Markdown way, RStudio still represents a viable way to go.

More importantly, it could be a convenient starting line for people with the primary background in R.

So, what did I find?

Analysis

Packages and environment

First of all, let us set the environment and load the required packages.

Global environment:

# Globally round numbers at decimals

options(digits=2) # Force R to use regular numbers instead of using the e+10-like notation

options(scipen = 999)

R libraries:

# Load the required packages.

# If these are not available, install them first.

ipak <- function(pkg){

new.pkg <- pkg[!(pkg %in% installed.packages()[, "Package"])]

if (length(new.pkg))

install.packages(new.pkg,

dependencies = TRUE)

sapply(pkg,

require,

character.only = TRUE)

} packages <- c("tidyverse", # Data wrangling

"gapminder", # Data source

"knitr", # R Markdown styling

"htmltools") # .html files manipulation ipak(packages) tidyverse gapminder knitr htmltools

TRUE TRUE TRUE TRUE

In this analysis, I will be working with the reticulate library, a package developed at RStudio. However, feel free to look for alternatives.

Also, I am going to import the reticulate package separately for the purposes of a clearer flow.

Python activation:

# The "weapon of choice"

library(reticulate) # Specifying which version of python to use.

use_python("/home/vg/anaconda3/bin/python3.7",

required = T) # Locate and run Python

Python libraries:

import pandas as pd # data wrangling

import numpy as np # arrays, matrices, math

import statistics as st # functions for calculating statistics

import plotly.express as px # plotting package

One of the limits of working with Python in R (Studio) is that in some cases you do not receive the Python traceback by default. That is a problem because, if I oversimplify things, a traceback helps you to identify where is the problem. Meaning it helps you fix it. So, consider it as an error message (or the lack of it).

For example, when calling a library that you do not have installed, the Python chunk in R Markdown gives you green lights (so everything looks up and running), but this does not mean that the code ran the way you would expect (e.g. it imported a library).

To deal with that, I suggest making sure you have your libraries installed in Terminal . If they are not, you can always install them and import afterwards.

For example, I will first import the json package which is already installed on my machine. I will do so by using Terminal here in RStudio. In addition, let me try to import the TensorFlow package.

From the following picture, you can see that there is no package TensorFlow . So, let me switch back to bash and install the package:

Then go to the directory where is the newly installed package TensorFlow installed, switch to Python and import the package once again:

All right. For more information, take a look at the Installing Python Modules page.

Data

As we now have R and Python set, let us import some data to play with. I will be working with a sample from the Gapminder, an intriguing project related to socio-demography of the world’s population. To be more specific, I will be working with the latest available data within the GapMinder library.

Data import in R:

# Let us begin with the most recent set of data

gapminder_latest <- gapminder %>%

filter(year == max(year))

So, now we have a data loaded in R. Unfortunately, it is not possible to access the R objects (e.g. vectors od tibbles) directly by Python. So, we need to convert the R object(s) first.

Data import in Python:

# Convert R Data Frame (tibble) into the pandas Data Frame

gapminder_latest_py = r['gapminder_latest']

One important thing to realise when working with Python objects (e.g. arrays or pandas Data Frame) is that they are not explicitly stored in the environment as the R objects are.

In other words, if we want to know what is stored in the workspace, we must call functions like dir() , globals() or locals() :

['R', '__annotations__', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'contact_window_days', 'contact_window_days_style', 'fig', 'gapminder_latest_count_py', 'gapminder_latest_max_lifeExp_py', 'gapminder_latest_mean_lifeExp_py', 'gapminder_latest_median_lifeExp_py', 'gapminder_latest_min_lifeExp_py', 'gapminder_latest_py', 'gapminder_latest_shape', 'gapminder_latest_stdev_lifeExp_py', 'lifeExpHist', 'np', 'pd', 'px', 'r', 'st', 'sys', 'variable', 'variable_grouping', 'variable_name']

Great, among the present objects, we can clearly see the data ( gapminder_latest_py ) or libraries (e.g. px ).

So, let us explore the data a bit!

Life Expectancy

For the demonstration purposes, I will focus on the life expectancy or the average number of years a person is expected to live.

Descriptive statistics

Let’s begin with calculating some descriptive statistics like mean, median or the number of rows in the data using Python:

# Descriptive statistics for the inline code in Python ## Data Frame Overview ### Number of rows

gapminder_latest_shape = gapminder_latest_py.shape[0]

### Number of distinct values within the life expectancy variable

gapminder_latest_count_py = gapminder_latest_py['lifeExp'].nunique() ## Life Expectancy ### Median (Life Expectancy)

gapminder_latest_median_lifeExp_py = st.median(gapminder_latest_py['lifeExp'])

### Mean

gapminder_latest_mean_lifeExp_py = st.mean(gapminder_latest_py['lifeExp'])

### Minimum

gapminder_latest_min_lifeExp_py = min(gapminder_latest_py['lifeExp'])

### Maximum

gapminder_latest_max_lifeExp_py = max(gapminder_latest_py['lifeExp'])

### Standard deviation

gapminder_latest_stdev_lifeExp_py = st.stdev(gapminder_latest_py['lifeExp'])

Nice. Unfortunately, we are not able to use the Python objects for inline coding, one of the key features of literate coding in R Markdown. So, if we want to use the results for inline codes, we need to transform the Python objects back to R:

# Descriptive statistics for the inline code in Python - transformed to R ## Data Frame Overview ## Number of rows

gapminder_latest_nrow_r = py$gapminder_latest_shape

### Number of distinct values within the life expectancy variable

gapminder_latest_count_r = py$gapminder_latest_count_py ## Life Expectancy ### Median (Life Expectancy)

gapminder_latest_median_lifeExp_r = py$gapminder_latest_median_lifeExp_py

### Mean

gapminder_latest_mean_lifeExp_r = py$gapminder_latest_mean_lifeExp_py

### Minimum

gapminder_latest_min_lifeExp_r = py$gapminder_latest_min_lifeExp_py

### Maximum

gapminder_latest_max_lifeExp_r = py$gapminder_latest_max_lifeExp_py

### Standard deviation

gapminder_latest_stdev_lifeExp_r = py$gapminder_latest_stdev_lifeExp_py

So, what can we say about life expectancy in 2007?

First of all, there were 142 countries on the list. The minimum value of life expectancy was 39.61 years, the maximum 82.6 years.

The average value for life expectancy was 67.01 years and 50% or median hope to live 71.94 years or more. Lastly, the standard deviation was 12.07 years.

Graphs (using Plotly)

Okay, let’s move to something else, like graphs.

For example, we can take a look at how is the life expectancy distributed across the globe using Plotly :

fig = px.histogram(gapminder_latest_py, # package.function; Data Frame

x="lifeExp", # Variable on the X axis

range_x=(gapminder_latest_min_lifeExp_py,

gapminder_latest_max_lifeExp_py), # Minimum and maximum values for the X axis

labels={'lifeExp':'Life expectancy - in years'}, # Naming of the interactive part

color_discrete_sequence=['#005C4E']) # Colour of fill lifeExpHist = fig.update_layout(

title="Figure 1. Life Expectancy in 2007 Across the Globe - in Years", # The name of the graph

xaxis_title="Years", # X-axis title

yaxis_title="Count", # Y-axis title

font=dict( # "css"

family="Roboto",

size=12,

color="#252A31"

)) lifeExpHist.write_html("lifeExpHist.html") # Save the graph as a .html object

Unfortunately, it is not possible to print interactive Plotly graphs in R Markdown via Python. Or, to be more precise, you will receive a Figure object by printing (e.g. print(lifeExpHist) ) it:

Figure({

'data': [{'alignmentgroup': 'True',

'bingroup': 'x',

'hovertemplate': 'Life expectancy - in years=%{x}<br>count=%{y}<extra></extra>',

'legendgroup': '',

'marker': {'color': '#005C4E'},

'name': '',

'offsetgroup': '',

'orientation': 'v',

'showlegend': False,

'type': 'histogram',

'x': array([43, 76, 72, 42, 75, 81, 79, 75, 64, 79, 56, 65, 74, 50, 72, 73, 52, 49,

59, 50, 80, 44, 50, 78, 72, 72, 65, 46, 55, 78, 48, 75, 78, 76, 78, 54,

72, 74, 71, 71, 51, 58, 52, 79, 80, 56, 59, 79, 60, 79, 70, 56, 46, 60,

70, 82, 73, 81, 64, 70, 70, 59, 78, 80, 80, 72, 82, 72, 54, 67, 78, 77,

71, 42, 45, 73, 59, 48, 74, 54, 64, 72, 76, 66, 74, 71, 42, 62, 52, 63,

79, 80, 72, 56, 46, 80, 75, 65, 75, 71, 71, 71, 75, 78, 78, 76, 72, 46,

65, 72, 63, 74, 42, 79, 74, 77, 48, 49, 80, 72, 58, 39, 80, 81, 74, 78,

52, 70, 58, 69, 73, 71, 51, 79, 78, 76, 73, 74, 73, 62, 42, 43]),

'xaxis': 'x',

'yaxis': 'y'}],

'layout': {'barmode': 'relative',

'font': {'color': '#252A31', 'family': 'Roboto', 'size': 12},

'legend': {'tracegroupgap': 0},

'margin': {'t': 60},

'template': '...',

'title': {'text': 'Figure 1. Life Expectancy in 2007 Across the Globe - in Years'},

'xaxis': {'anchor': 'y', 'domain': [0.0, 1.0], 'range': [39.613, 82.603], 'title': {'text': 'Years'}},

'yaxis': {'anchor': 'x', 'domain': [0.0, 1.0], 'title': {'text': 'Count'}}}

})

So, we import the previously created .html file instead (e.g. using the includeHTML function from the htmltools package):

htmltools::includeHTML("lifeExpHist.html") # Render the graph

So, here comes a basic, yet interactive histogram made in the Python version of Plotly .

However, producing this graph in RStudio required quite a workaround. At the same time, a size of the graphs produced this way could easily be tens of MBs.

Consequently, a .html report containing such graphs would require a lot of data to download for a reader and it would take more time to render the page.

Summary tables (using pandas)

One of the common use-cases for pandas is to provide a data description. The respective code runs just fine. However, the output cannot be styled as you are used from styling in pandas .

When speaking of pandas , we can easily create a summary table for the commonly used statistics like mean or standard deviation of life expectancy for individual continents:

# Create a pandas Data Frame object containing the relevant variable,

# conduct formatting.



gapminder_latest_py['continent'] = gapminder_latest_py['continent'].astype(str)

variable = 'continent'

variable_name = 'Continents'



gapminder_latest_py['lifeExp'] = gapminder_latest_py['lifeExp'].astype(int)

variable_grouping = 'lifeExp'



contact_window_days = gapminder_latest_py.groupby([

pd.Grouper(key=variable)])\

[variable_grouping]\

.agg(['count',

'min',

'mean',

'median',

'std',

'max'])\

.reset_index()



contact_window_days_style = contact_window_days\

.rename({'count': 'Count',

'median': 'Median',

'std': 'Standard Deviation',

'min': 'Minimum',

'max': 'Maximum',

'mean': 'Mean',}, axis='columns')

Output:

continent Count Minimum Mean Median Standard Deviation Maximum

0 Africa 52 39 54.326923 52.0 9.644100 76

1 Americas 25 60 73.040000 72.0 4.495183 80

2 Asia 33 43 70.151515 72.0 7.984834 82

3 Europe 30 71 77.100000 78.0 2.916658 81

4 Oceania 2 80 80.500000 80.5 0.707107 81

Also, note that the Python environment settings override those from R. For example, take a look at the number of digits for Mean or Standard Deviation. There are six digits instead of two set at the beginning.

Closing remarks

Okay, that’s enough for now. If you are hungry for more advanced things like unsupervised learning using scikit-learn package in R, take a look.

However, before closing this post, let me just say that if you think about switching to Python as such and using it often, consider IDE alternatives to RStudio.

Many analysts swear on Jupyter notebooks for the interactivity, integration of markdown or option to run code in various languages like R , Julia or JavaScript . JupyterHub is a platform based on Jupyter notebooks, adding version control. Usually, users run analyses in a containerised environment). Another take on interactivity and collaboration could be Colab, basically, Jupyter notebooks running on Google Cloud.

Last but not least, there is a great piece of software called Visual Studio Code. It not only allows you to create and run code in a plethora of languages or seamless flow between pure Python code and interactive Jupyter notebooks. And maybe even more importantly, it provides you with very efficient version control management (like Git integration and extensions). If you choose this IDE, you can set up VS Code for Python development like RStudio.

But no matter what path to Python you choose, don’t forget that it is a tool suitable for some situations and maybe not so suitable to others. Just like R. Try to leverage the best of it while being aware of the pros of different tools.