Python & Pandas in Node.js Application | by Joe T. Santhanavanich | Towards Data Science

In Data Sciences, we all know that Python is one of the best tools to handle data management, analytics, visualization, ML, and many more. In the same way, Node.js is the JavaScript runtime environment for server-side which is one of the most popular tools for web architecture according to its economically beneficial, quick response time, etc. See from the Popularity of Programming Language Index (PYPL): Python and JavaScript are the top 1&3 popular languages.
From my experience, I see most Python fans try to write everything using Python. And most Node.js Volk tries to do data cleaning, manipulation in JavaScript... (a bit painful with all that callbacks ...) Several blogs on the internet tried to compare between the Node.js and Python finding out that Node.js is awesome for Web Development and Python for Data Sciences. Actually, we don't need to always stick with the same programming language as there are ways to use them both together. In this article, I will show you an example of how to use a Python script from the Node.js web application.
In this article, I will show an example of how to create a simple API web application where users can request the COVID-19 dataset and time-series aggregation interval with the Node.js and Express framework. The interesting thing here is that we will integrate the time-series aggregation done by Python and Pandas to the application as well.
Later on, developers can use such API applications for easily rendering chart/ map/ dashboard in web application with any JavaScript library; such as Apexchart.js, Highchart.js, or Chart.js. The below figure shows an example of using simple Apexchart for the daily COVID-19 dataset.
An overall structure of this application is shown in the following figure:
First, let’s get started with the Python application which I will take an example from my article about the COVID-19 Time Series Analysis with Pandas in Python. In short, we will use a simple Python script to read and analyze the COVID-19 dataset from JHU CSSE to the Pandas dataframe. Then, group the dataset by ‘Country/Region’ level and transpose the dataframe and set the date/time column as an index column and sort the countries with most confirmed COVID-19 cases and select top countries according to the input country number. After that, resample the dataset with the input time interval and return .json output to the users. The overall script is shown in the following script:
This script takes 2 arguments; python [script].py arg1 arg2 the first argument (arg1) is the number of countries with the highest confirmed COVID-19 cases and the second argument (arg2) is the aggregated time interval. For example, if we want to get confirmed COVID-19 case from the top three countries and aggregated data in monthly, then we can run the Python script with the following command:
$ python covid19aggregator.py 3 'M'
Then, you will get the following result (as of 10th May 2020):
You can see that the time index is written as timestamp which we will keep it this way as it is easy to use in the web-based visualization tool later.
In this part, we gonna start by creating a simple Node.js & Express application. You may initiate your project and install express with the following commands:
$ npm init$ npm install express --save
Then, create a server.js file for the Node.js web server with the example script below. If you are new to Express, please take a look at the documentation here. In part 1, we create a simple express app on port 3000. You may change to any port you like. In part 2, we define a GET request routing to ‘http://localhost:3000/covid_19_timeseries’ and use child_process to spawn the Python child process and also passing the URL query parameters ‘numberOfCountries’ and ‘aggregationInterval’ to the Python process. You may see the overall Node.js server script here:
It is done! Now, this Node.js server is complete and it spawns the Python child process when users make a call to get the COVID 19 data. Let’s test it by running a server with:
$ node server.jsserver running on http://localhost:3000
Let’s check the result on the web browser:(Here you may pass any number of Countries/ aggregation interval you want)
http://localhost:3000/covid_19_timeseries?numberOfCountries=5&aggregationInterval=M
So, that’s about it. Now, you already have a simple API application built with Node.js and Python. Please note that this example is to show a concept of integrating them together, the scripts can be improved in Node.js for better API route structure and in Python for more complex data manipulation workflow.
This article shows an example of how to use NodeJS to spawn a Python process with Pandas and return a result directly to the API call. In this way, we are flexible to integrate all cool Data Python scripts with the popular web frameworks in JavaScript. I hope you like this article and found it useful for your daily work or projects. Feel free to leave me a message if you have any questions or comments.
About me & Check out all my blog contents: Link
Be Safe and Healthy! 💪
Thank you for Reading. 📚