carefree-learn: Tabular Datasets ❤️ PyTorch
Tags: Machine Learning, Automl, Tabular Data, Deep Learning, Pytorch

Why carefree-learn?

If you are familiar with machine learning, you may already heard about scikit-learn and some other automl frameworks. The motivation behind creating carefree-learn was two fold:

Leverage PyTorch in the field of machine learning.

Provide a truly CAREFREE experience for both users and developers.

By saying carefree we mean that both using and developing carefree-learn could be finished in one line of code:

Please refer to Quick Start and Build Your Own Models for more details

You may argue that other libraries, like scikit-learn, also support building models in one line of code. But think about the messy stuffs (such as reading data from files, exploring data, performing data pre-processing, etc.) we need to do before we actually use these libraries, and contributing algorithms to these libraries is often much harder than writing your own ones. In carefree-learn, we’ve tried hard to ease these two procedures for you.

Carefree using

As mentioned, other libraries (e.g. scikit-learn) can often only support numpy arrays or DataFrame as input, and have pretty much constraints (e.g. cannot contain nan values). In carefree-learn, however, we’ve tried hard to help you deal with almost ANY kind of tabular datasets, no matter how dirty and messy it is. This means carefree-learn actually provides an end-to-end pipeline on tabular datasets, including AUTOMATICALLY deal with:

Detection of redundant feature columns which can be excluded (all SAME, all DIFFERENT, etc).

Detection of feature columns types (whether a feature column is string column / numerical column / categorical column).

Encoding of string columns and categorical columns (Embedding or One Hot Encoding).

Pre-processing of numerical columns (Normalize, Min Max, etc.).

Imputation of missing values.

And much more…

Therefore, carefree-learn is possible to procss files directly (file-in, file-out):

This is mainly handled by carefree-data, part of the cf* ecosystem

Please refer to Quick Start for more details.

Carefree developing

Thanks to the great modularization provided by PyTorch, carefree-learn was able to design its (tabular datasets oriented) pipeline in a user-friendly as well as a developer-friendly way.

The basic component in carefree-learn is called a pipe , which corresponds to one of those branches which takes in all / part of the inputs, apply some transform , extract some features with extractor , and then feed the final network ( head ) with these features. Here's an example:

A model with three pipes. Please refer to our document to learn more about pipes.

Since most of the deep learning models (neural networks) used in tabular datasets could be represented with pipe , developers can therefore focus on implementing one of its components (namely extractor and head ), instead of having to care about the whole pipeline.

We’ve provided a detailed documention on how to build your own models, as well as an interesting example to guide you step by step. Please feel free to check them out and enjoy yourself!

Why PyTorch?

The reason why I chose PyTorch has already been mentioned above — it is perfectly modularized and customizable. Another reason is that although deep learning outshines in CV and NLP fields, it is not as popular when it comes to the tabular datasets. We think the main reason is that tabular datasets require so many messy steps before we could actually jump into the algorithm part, and there has not existed a modularized framework to organize these stuffs. That’s why we tried our best to handle all of these for you and hope that it could help you focus on developing the core algorithms. And among the deep learning frameworks, PyTorch was truly the most elegant one and we were deeply attracted by its simplicity and power. Since there lacks a satisfying ‘carefree’ solution for tabular datasets, we decided to take advantage of our knowledges and build one ourselves. So here comes the carefree-learn, which aims to provide out of the box tools to train and develop neural networks on tabular datasets with PyTorch.

Getting Started with Iris

We’ve provided an Installation Guide as well as some real life Examples to walk you through the basic / advanced usages of carefree-learn. We’ve also provided a Production Guide to show how could we pack the whole pipeline in carefree-learn efficiently into a zip file. In this section, we’ll use the famous Iris dataset to demonstrate the power of carefree-learn.

We’ve provided the complete source code here as well as a jupyter notebook here.

Inspect the Iris Dataset

Here are some of the information provided by the official website:

This is perhaps the best known database to be found in the pattern recognition literature. The data set contains 3 classes of 50 instances each, where each class refers to a type of iris plant. Predicted attribute: class of iris plant.

And here’s the pandas-view of the raw data:

We didn’t use pandas in our code, but it is convenient to visualize some data with it though 🤣

You can download the raw data (iris.data) with this link.

Basic Usages

Traditionally, we need to process the raw data before we feed them into our machine learning models (e.g. encode the label column, which is a string column, into an ordinal column). In carefree-learn, however, we can train neural networks directly on files without worrying about the rest:

What’s going under the hood is that carefree-learn will try to parse the iris.data automatically (with the help of carefree-data), split the data into training set and validation set, with which we’ll train a fully connected neural network (fcnn).

We can further inspect the processed data if we want to know how carefree-learn actually parsed the input data:

It shows that the raw data is carefully normalized into numerical data that neural networks can accept. You may also notice that the first elements are not identical with the first line of the raw data, this is caused by the auto-shuffle mechanism introduced in carefree-data.

What’s more, by saying normalized, it means that the input features will be automatically normalized to mean=0.0 and std=1.0 :

This means we first normalized the data before we actually split it into train & validation set.

After training on files, carefree-learn can predict & evaluate on files directly as well. We’ll handle the data parsing and normalization for you automatically:

Which yields

This output may vary due to the randomness.

Benchmarking

As we know, neural networks are trained with stochastic gradient descent (and its variants), which will introduce some randomness to the final result, even if we are training on the same dataset. In this case, we need to repeat the same task several times in order to obtain the bias & variance of our neural networks. Fortunately, carefree-learn introduced repeat_with API, which can achieve this goal easily with only a few lines of code:

With num_repeat=3 specified, we’ll train 3 models on iris.data.

Which yields

This output may vary due to the randomness.

We can also compare the performances across different models:

With models=[“linear”, “fcnn”], we’ll train both linear models and fcnn models.

Which yields

This output may vary due to the randomness, but fcnn should be able to beat linear.

It is worth mentioning that carefree-learn supports distributed training, which means when we need to perform large scale benchmarking (e.g. train 100 models), we could accelerate the process through multiprocessing:

In carefree-learn, distributed training doesn’t mean training your model on multiple GPUs or multiple machines. Instead, distributed training in carefree-learn means training multiple models at the same time. Please refer to our documentation for more details.

With num_jobs=2, we will launch 2 processes to run the tasks in a distributed way.

On iris dataset, however, launching distributed training will actually hurt the speed because iris dataset only contains 150 samples, so the relative overhead brought by distributed training will be too large. Please refer the CAUTION section of our documentation for more details.

Advanced Benchmarking

But this is not enough, because we want to know whether other models (e.g. scikit-learn models) could achieve a better performance than carefree-learn models. In this case, we can perform an advanced benchmarking with the Experiment helper class.

Notice that we specified run_command="python run_sklearn.py" for scikit-learn tasks, which means Experiment will try to execute this command in the current working directory for training scikit-learn models. The good news is that we do not need to speciy any command line arguments, because Experiment will handle those for us.

Here is basically what a run_sklearn.py should look like (source code):

With run_sklearn.py defined, we could run those tasks with one line of code:

After finished running with this, we should be able to see the following file structure in the current working directory:

As we expected, carefree-learn models are saved into zip files, while scikit-learn models are saved into sk_model.pkl files. Since these models are not yet loaded, we should manually load them into our environment:

After which we can finally perform benchmarking on these models:

Which yields

This output may vary due to the randomness.

Seems that scikit-learn models are better than carefree-learn models! This is not surprising because neural networks often require more data than traditional machine learning algorithms. However, we can boost carefree-learn models with AutoML, as shown in the next section.

AutoML on Iris

As mentioned above, carefree-learn is actually a minimal Automatic Machine Learning (AutoML) solution for tabular datasets. Up till now we haven’t mentioned any AutoML stuffs yet, so we’ll illustrate how to perform AutoML on Iris dataset in carefree-learn, as well as how to pack the AutoML results into production.

Since carefree-learn has provided the cflearn.Auto API for out-of-the-box usages, AutoML in carefree-learn could be achieved in two lines of code:

We can make predictions directly with auto.predict :

The accuracy should be around 0.97

And of course, we can compare it with other models:

Which yields

This output may vary due to the randomness.

Bravo! Our AutoML model beats the scikit-learn models 🥳

If we are satisfied with the results, we can pack the models up into a zip file

This will generate a pack.zip file.

which could be used on our production environments / machines easily:

Please refer to our documentation for more details.

Conclusion

Contained in this article is just a subset of the features that carefree-learn offers, but we’ve already walked through many basic & common steps we’ll encounter in real life machine learning tasks. Additional capabilities include Fast Embedding, One Hot Encoding Caching, highly customizable modules, specify / customize optimizers / lr schedulers, export to ONNX, and more.

To learn more about carefree-learn, check out the Examples with jupyter notebooks included, as well as the Developer’s Guide to see how do we customize models with carefree-learn. We sincerely hope that carefree-learn could help you either deal with tabular datasets easier or develop new algorithms on tabular datasets easier, and any contributions to carefree-learn would truly be welcomed.