Python libraries for reading large datasets faster ,every data Scientist should know
By: Saurav Anand
Tags: Artificial Intelligence, Machine Learning, AI, Data Science, Data Analysis

Fast decision making process is the first way to success — Dilip kumar

Introduction

In most of the scenarios . as a data scientist we encounter data sets which are huge and cannot be processed using commonly used libraries like pandas and numpy. We can ignore the data points as it will affect our prediction model. Also it is always good to have more datasets for the training as well as testing purpose. In order to overcome these challenge , we need to make use of certain libraries which we are going to discuss in this blog.

Please note that reading the data is not the end of the story as data goes through lot of tasks and processes after that in order to achieve the business goal. In this blog we are not going into the details of it , but only making the readers familiar with these libraries.

Motivation

I took part in a hackathon where the challenge was of regression type . The training datset and testing dataset was huge (4 lakhs and 50k data points respectively). It was not possible to read the entire dataset with pandas as there was memory error when ML algorithms were applied due to the limitation of the RAM. I tried taking the sample but the prediction results were not good. The error metric was huge. So , I started searching for the libraries which will make me task easier and I can process them with ease.

Let’s start discussing these libraries in detail :

1.Dask

Dask is open source and freely available. It is developed in coordination with other community projects like Numpy, Pandas, and Scikit-Learn.Dask is composed of two parts:

Dynamic task scheduling optimized for computation. This is similar to Airflow, Luigi, Celery, or Make, but optimized for interactive computational workloads. “Big Data” collections like parallel arrays, dataframes, and lists that extend common interfaces like NumPy, Pandas, or Python iterators to larger-than-memory or distributed environments. These parallel collections run on top of dynamic task schedulers.

Dask Dataframe mimics Pandas

Pandas Implementation

import pandas as pd

df = pd.read_csv('pima_indians.csv')

df.groupby(df.blood_pressure).value.mean()

Dask Dataframe Implementation

Dask dataframes scale Pandas workflows, enabling applications in time series, business intelligence, and general data munging on big data.

import dask.dataframe as dd

df = dd.read_csv('pima_indians.csv')

df.groupby(df.blood_pressure).value.mean().compute()

Documentation : https://docs.dask.org/en/latest/dataframe.html

Dask Array mimics NumPy

Numpy Implementation

import numpy as np

f = h5py.File('myfile.hdf5')

x = np.array(f['/small-data'])

x - x.mean(axis=1)

Dask Array Implementation

Dask arrays scale Numpy workflows, enabling multi-dimensional data analysis in earth science, satellite imagery, genomics, biomedical applications, and machine learning algorithms.

import dask.array as da

f = h5py.File('myfile.hdf5')

x = da.from_array(f['/big-data'],chunks=(1000, 1000))

x - x.mean(axis=1).compute()

Documentation :https://docs.dask.org/en/latest/array.html

Dask Bag mimics iterators, Toolz, and PySpark

import dask.bag as db

b = db.read_text('my_file.gz').map(json.loads)

b.pluck('name').frequencies().topk(10, lambda pair: pair[1]).compute()

Documentation :https://docs.dask.org/en/latest/bag.html

Dask Delayed mimics for loops and wraps custom code

from dask import delayed

L = []

for fn in filenames: # Use for loops to build up computation

data = delayed(load)(fn) # Delay execution of function

L.append(delayed(process)(data)) # Build connections between variables



result = delayed(summarize)(L)

result.compute()

Documentation: https://docs.dask.org/en/latest/delayed.html

Dask concurrent.futures interface

It provides general submission of custom tasks.

from dask.distributed import Client

client = Client('scheduler:port')



futures = []

for fn in filenames:

future = client.submit(load, fn)

futures.append(future)



summary = client.submit(summarize, futures)

summary.result()

Documentation: https://docs.dask.org/en/latest/futures.html

Features of Dask

Familiar for Python Users : Dask uses existing Python APIs and data structures to make it easy to switch between Numpy, Pandas, Scikit-learn to their Dask-powered equivalents.

: Dask uses existing Python APIs and data structures to make it easy to switch between Numpy, Pandas, Scikit-learn to their Dask-powered equivalents. Scale up to clusters : Dask’s schedulers scale to thousand-node clusters and its algorithms have been tested on some of the largest supercomputers in the world. Dask ships with schedulers designed for use on personal machines. Many people use Dask today to scale computations on their laptop, using multiple cores for computation and their disk for excess storage.

: Dask’s schedulers scale to thousand-node clusters and its algorithms have been tested on some of the largest supercomputers in the world. Dask ships with schedulers designed for use on personal machines. Many people use Dask today to scale computations on their laptop, using multiple cores for computation and their disk for excess storage. Customizable : It enables you to parallelize internal systems. Dask exposes lower-level APIs letting us build custom systems for in-house applications. This helps open source leaders parallelize their own packages and helps business leaders scale custom business logic.

: It enables you to parallelize internal systems. Dask exposes lower-level APIs letting us build custom systems for in-house applications. This helps open source leaders parallelize their own packages and helps business leaders scale custom business logic. Integrates with existing projects : Dask is open source and freely available. It is developed in coordination with other community projects like Numpy, Pandas, and Scikit-Learn.

: Dask is open source and freely available. It is developed in coordination with other community projects like Numpy, Pandas, and Scikit-Learn. Libraries Powered by Dask : Libraries such as scikit-learn , scikit-image ,XGBoost,XArray, Rapids , snorkel etc are powered by it.

2.datatable

This is a Python package for manipulating 2-dimensional tabular data structures (aka data frames). It is close in spirit to pandas or SFrame; however we put specific emphasis on speed and big data support. As the name suggests, the package is closely related to R’s data.table and attempts to mimic its core algorithms and API.

Installation

On mac-OS, Linux and Windows systems installing datatable is as easy as

# datatable installation with internet

# !pip install datatable==0.11.0 > /dev/null



# datatable installation without internet

!pip install ../input/python-datatable/datatable-0.11.1-cp37-cp37m-manylinux2010_x86_64.whl > /dev/null

Import datatable and read file

import datatable as dt

data = dt.fread("pima_indians.csv")

print("Train size:", data.shape)

Documentation Link :-

Github Link :- https://github.com/h2oai/datatable

Documentation:https://datatable.readthedocs.io/en/latest/start/index-start.html#

3.Rapids

It is a great option to scale data processing on GPUs. With a lot of machine learning modelling moving to GPUs, Rapids enables to build end-to-end data science solutions on one or more GPUs.

Installation

# rapids installation (make sure to turn on GPU)

import sys

!cp ../input/rapids/rapids.0.16.0 /opt/conda/envs/rapids.tar.gz

!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz > /dev/null

sys.path = ["/opt/conda/envs/rapids/lib/python3.7/site-packages"] + sys.path

sys.path = ["/opt/conda/envs/rapids/lib/python3.7"] + sys.path

sys.path = ["/opt/conda/envs/rapids/lib"] + sys.path



Import rapids and read file

import cudf

data = cudf.read_csv("pima_indians.csv")

Conclusion

Each library which is discussed above has some pros and cons . But in the end they are very useful when dealing with large datasets

Dask can be slow at times especially with transformations that cannot be parallelized.

can be slow at times especially with transformations that cannot be parallelized. Datatable doesn’t have a very exhaustive set of data processing functions.

doesn’t have a very exhaustive set of data processing functions. Rapids is not useful if you don’t have a GPU.

It ‘s always good to first understand your requirements and check which of these libraries fit best .

References

Happy Reading ,

Saurav Anand

Gain Access to Expert View — Subscribe to DDI Intel