How to handle BigData Files on Low Memory? | by Puneet Grover | Towards Data Science

This is one of the post from my posts from Tackle category, which can be found on my github repo here.
(Edit-31/01/2019) — Added info on dask.distributed.LocalCluster for BigData
(Edit-12/4/2019) — Added new sections on dataset size reduction and File Type Usage [Not Complete yet, still you can take ideas from there to apply.]
IntroductionColumn Memory Reduction (pd.Series.astype()) [:Not Completed]File Types (for less memory usage) [:Not Completed]Data ExplorationPreprocessingIncremental Learning (with Pandas)Dask (Explore+Prep+FitPredict)Further ReadingReferences
Introduction
Column Memory Reduction (pd.Series.astype()) [:Not Completed]
File Types (for less memory usage) [:Not Completed]
Data Exploration
Preprocessing
Incremental Learning (with Pandas)
Dask (Explore+Prep+FitPredict)
Further Reading
References
NOTE:This post goes along with Jupyter Notebook available in my Repo on Github:[HowToHandleBigData]  (with dummy data)and Kaggle:[HowToHandleBigData] (with Kaggle competition data)
With data size increasing exponentially, we are not able to fit our data, and even our models, within our PC’s memory. And we all can’t afford to buy a high end assembled desktop beasts.
For example a recent kaggle competition has dataset which is not able to fit in 17GB of RAM in kaggle kernels or in Colab. It has nearly 2 million rows, and to top that some columns have very large JSON data as strings. How should we tackle this? To whom can we turn to?
Incremental Learning and/or Dask comes to rescue!
You might already know that neural networks are Incremental Learners by nature, so we can tackle this problem there. And many of sklearn’s models provide a method called partial_fit using which we can fit to model in batches. And some of Boosting libraries like XGBoost and LightGBM provide a way to learn incrementally to tackle big data.
Here we will look into incremental solutions provided by some of the Boosting algorithms. And then we will use Dask on the same dataset and use its models to predict.
^
# I don't know who the original author of this function is,# but you can use this function to reduce memory# consumption by 60-70%!def reduce_mem_usage(df):    """     iterate through all the columns of a dataframe and     modify the data type to reduce memory usage.            """    start_mem = df.memory_usage().sum() / 1024**2    print(('Memory usage of dataframe is {:.2f}'                      'MB').format(start_mem))        for col in df.columns:        col_type = df[col].dtype                if col_type != object:            c_min = df[col].min()            c_max = df[col].max()            if str(col_type)[:3] == 'int':                if c_min > np.iinfo(np.int8).min and c_max <\                  np.iinfo(np.int8).max:                    df[col] = df[col].astype(np.int8)                elif c_min > np.iinfo(np.int16).min and c_max <\                   np.iinfo(np.int16).max:                    df[col] = df[col].astype(np.int16)                elif c_min > np.iinfo(np.int32).min and c_max <\                   np.iinfo(np.int32).max:                    df[col] = df[col].astype(np.int32)                elif c_min > np.iinfo(np.int64).min and c_max <\                   np.iinfo(np.int64).max:                    df[col] = df[col].astype(np.int64)              else:                if c_min > np.finfo(np.float16).min and c_max <\                   np.finfo(np.float16).max:                    df[col] = df[col].astype(np.float16)                elif c_min > np.finfo(np.float32).min and c_max <\                   np.finfo(np.float32).max:                    df[col] = df[col].astype(np.float32)                else:                    df[col] = df[col].astype(np.float64)        else:            df[col] = df[col].astype('category')    end_mem = df.memory_usage().sum() / 1024**2    print(('Memory usage after optimization is: {:.2f}'                               'MB').format(end_mem))    print('Decreased by {:.1f}%'.format(100 * (start_mem - end_mem)                                              / start_mem))        return df
TK
^
HDF5, parquet etc.
Note: If you don’t want to fall into this section, you can just look into HDF5. That should be sufficient. And move to next section.
Hierarchical Data Format — Wikipedia (HDF5)
Apache Parquet
Hadoop File Formats: Its not just CSV anymore — Kevin Haas (Don’t scare away after reading Hadoop, you can get away with just using Parquet format only and by using pd.DataFrame.to_parquet and pd.DataFrame.read_parquet methods.)
Also, if you are interested: Parquet vs Avro [I would recommend you to watch it anyway.]
TK
Firstly we will get a feel of what our data looks like by looking at first few rows by using the command:
part = pd.read_csv("train.csv.zip", nrows=10)part.head()
By this you will have basic info on how different columns are structured, how to process each column etc. Make a lists of different type of columns as numerical_columns, obj_columns, dictionary_columns, etc. which will hold all corresponding columns.
Now to explore data we will go column by column like this:
# For dictionary columns you can do:# 'idx' is index of corresponding column in DataFrame.# You can find it by using np.where(col==df.columns)for col in dictionary_columns:    df = pd.read_csv("train.csv.zip", usecols = [idx], converters={col: json.loads})    column_as_df = json_normalize(df[col])    # ... plot each column ...    # ... check if you want to drop any column ...
You can keep a dictionary with all the column names as keys and methods applied to it in a list as a pipeline for that column. You can also pickle dump your dictionary, for future use:
with open("preprocessing_pipeline.pickle", "wb") as fle:  pickle.dump(preprocessing_pipeline, fle)
If even one of the column is too big for your memory, which actually was case for one of the rows in kaggle competition I mentioned above. You can even open one column Incrementally and can do basic stuff like calculating mean, standard deviation etc. manually. Or you can also use Dask for it and use nearly the same API as pandas and calculate them using it. See last section for Dask.
For preprocessing data we will make use of the dictionary we made earlier, which has info on which columns we want to keep (as keys) and what methods to apply to each column (as values), to make a method.
This method will be called for each batch of data during the incremental learning process.
Now one thing to notice here is that we fitted methods (like LabelEncoder’s, Scalars’s etc.) during exploration of whole data column and we will use that to transform data at every incremental step here. Because, in each batch, there might be some data missing and if we had used different LabelEncoder’s, Scalar’s etc. for each batch, these methods wouldn’t have given same result for same category (say). That’s why we already have fitted to whole columns during exploration.
Here is how you can preprocess data:
def preprocess(df):  df.reset_index(drop=True, inplace=True)    # For dict columns:  for col in dict_columns:    col_df = json_normalize(df[col])    # json.loads during pd.read_csv to convert string to dict.                                    col_df.columns = [f"{col}.{subcolumn}" for subcolumn in col_df.columns]    # Select all columns which we selected before.    selected_columns = [c for c in dictionary.keys() if c in col_df.columns()]    to_drop = [c for c in col_df.columns if not in selected_columns]        # Drop all previously unselected columns.    col_df = col_df.drop(to_drop, axis=1)                                               df = df.drop(col, axis=1).merge(col_df, right_index=True, left_index=True)      # And so on...    # And then apply all Scalars, LabelEncoder's to all columns selected...    return df
To read data file incrementally using pandas, you have to use a parameter chunksize which specifies number of rows to read/write at a time.
incremental_dataframe = pd.read_csv("train.csv",                        chunksize=100000) # Number of lines to read.# This method will return a sequential file reader (TextFileReader)# reading 'chunksize' lines every time. To read file from # starting again, you will have to call this method again.
Then you can train on your data incrementally using XGBoost1 or LightGBM. For LightGBM you have to pass in a argument keep_training_booster=True to its .train method and three arguments to XGBoost's .train method.
# First one necessary for incremental learning:lgb_params = {  'keep_training_booster': True,  'objective': 'regression',  'verbosity': 100,}# First three are for incremental learning:xgb_params = {  'update':'refresh',  'process_type': 'update',  'refresh_leaf': True,  'silent': False,  }
On each step we will save our estimator and then pass it as an argument during next step.
# For saving regressor for next use.lgb_estimator = Nonexgb_estimator = Nonefor df in incremental_dataframe:  df = preprocess(df)    xtrain, ytrain, xvalid, yvalid = # Split data as you like    lgb_estimator = lgb.train(lgb_params,                         # Pass partially trained model:                         init_model=lgb_estimator,                         train_set=lgb.Dataset(xtrain, ytrain),                         valid_sets=lgb.Dataset(xvalid, yvalid),                         num_boost_round=10)    xgb_model = xgb.train(xgb_params,                         dtrain=xgb.DMatrix(xtrain, ytrain),                        evals=(xgb.DMatrix(xvalid, yvalid),"Valid"),                        # Pass partially trained model:                        xgb_model = xgb_estimator)    del df, xtrain, ytrain, xvalid, yvalid  gc.collect()
CatBoost's incremental learning method is in progress.2
To speed things up a bit more and if your chunks a still sufficiently big, you can parallelize your preprocessing method using Python's multiprocessing library functions like this:
n_jobs = 4for df in incremental_dataframe:  p = Pool(n_jobs)  f_ = p.map(preprocess, np.array_split(df, n_jobs))  f_ = pd.concat(f_, axis=0, ignore_index=True)  p.close()  p.join()    # And then your model training ...
For an introduction on Parallel programming in Python read my post here.
Dask helps to tap into big data resources in a sequentially-parallel manner.
For introduction to Dask read my post here.
You can apply functions from pandas API here in a similar manner. You can check if there is some null value like this:
df.isnull().sum().compute()
To scale a column you can convert them to array and use its Scaler functions in a similar manner:
rsc = dask_ml.preprocessing.RobustScaler()result = rsc.fit_transform(X[:,i].reshape(-1, 1)) # for ith column
To handle JSON or any other semi structured data like log files etc. you can use functions provided by Dask's Bag container.
df[key] = df[dict_col].to_bag().pluck(key).to_dataframe().iloc[:,0]
And after preprocessing you can use one of Dask's models to train your data.
For full code readDask section in Jupyter Notebook here.
Note:You should only use Dask in case of Big Data, where it is not able to fit in your memory. Otherwise in-memory learning with pandas and sklearn will be lot faster.Note: (Local Cluster)You can perform almost any BigData related query/tasks with the help of LocalCluster. You can, specifically, use 'memory_limit' parameter to constrict Dask's memory usage to a specific amount. Also, at times you might notice that Dask is exceeding memory use, even though it is dividing tasks. It could be happening to you because of the function you are trying to use on your dataset wants most of your data for processing, and multiprocessing can make things worse as all workers might try to copy dataset to memory. This can happen in aggregating cases.In these cases you can use Dask.distributed.LocalCluster parameters and pass them to Client() to make a LocalCluster using cores of your Local machines.from dask.distributed import Client, LocalClusterclient = Client(n_workers=1, threads_per_worker=1, processes=False,                memory_limit='25GB', scheduler_port=0,                 silence_logs=False, diagnostics_port=0)client'scheduler_port=0' and 'diagnostics_port=0' will choose random port number for this particular client. With 'processes=False' dask's client won't copy dataset, which would have happened for every process you might have made.You can tune your client as per your needs or limitations, and for more info you can look into parameters of LocalCluster.You can also use multiple clients on same machine at different ports.
https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-updhttps://github.com/dmlc/xgboost/issues/3055#issuecomment-359505122https://www.kaggle.com/ogrellier/create-extracted-json-fields-datasethttps://github.com/Microsoft/LightGBM/blob/master/examples/python-guide/advanced_example.pyReducing DataFrame memory size by ~65% | Kaggle7 Ways to Handle Large Data Files for Machine Learning — Machine Learning Mastery
https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd
https://github.com/dmlc/xgboost/issues/3055#issuecomment-359505122
https://www.kaggle.com/ogrellier/create-extracted-json-fields-dataset
https://github.com/Microsoft/LightGBM/blob/master/examples/python-guide/advanced_example.py
Reducing DataFrame memory size by ~65% | Kaggle
7 Ways to Handle Large Data Files for Machine Learning — Machine Learning Mastery
https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-updhttps://www.kaggle.com/ogrellier/create-extracted-json-fields-datasethttps://gist.github.com/goraj/6df8f22a49534e042804a299d81bf2d6https://github.com/dmlc/xgboost/issues/3055https://github.com/catboost/catboost/issues/464https://github.com/Microsoft/LightGBM/issues/987https://gist.github.com/ylogx/53fef94cc61d6a3e9b3eb900482f41e0
https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd
https://www.kaggle.com/ogrellier/create-extracted-json-fields-dataset
https://gist.github.com/goraj/6df8f22a49534e042804a299d81bf2d6
https://github.com/dmlc/xgboost/issues/3055
https://github.com/catboost/catboost/issues/464
https://github.com/Microsoft/LightGBM/issues/987
https://gist.github.com/ylogx/53fef94cc61d6a3e9b3eb900482f41e0
Suggestions and reviews are welcome.Thank you for reading!
Signed:
Some rights reserved