7 Best Python Packages Kagglers Are Using Without Telling You | Towards Data Science

Kaggle is a hot spot for what is trending in data science and machine learning.
Due to its competitiveness, the top players are constantly looking for new tools, technologies, and frameworks that give them an edge over others. If a new package or an algorithm delivers actionable value, there is a good chance it receives immediate adoption and becomes popular.
This post is about 7 of such trendies packages that are direct replacements for many tools and technologies that are either outdated or need urgent upgrades.
Above is a 100k row dataset with 75 features projected to 2D using a package called UMAP. Each dot represents a single sample in a classification problem and is color-encoded based on their class.
Massive datasets like these can make you miserable during EDA, mainly because of the computation and time expenses they come with. So, it is important that each plot you create is spot-on and reveals something significant about the data.
I think that’s one of the reasons why UMAP (Uniform Manifold Approximation and Projection) is so well-received on Kaggle. It is efficient, low-code, and lets you take a real “look” at the data from a high-dimensional perspective:
When I look at plots like these, they remind me of why I got into data science in the first place — data is beautiful!
https://umap-learn.readthedocs.io/en/latest/
https://github.com/lmcinnes/umap
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
UMAP offers an easy, Sklearn-compatible API. After importing the UMAP module, call its fit on the feature and target arrays (X, y) to project them to 2D by default:
The most important parameters of UMAP estimator are n_neighbors and min_dist (minimum distance). Think of n_neighbors as a handle that controls the zoom level of the projection. min_dist is the minimum distance between each projected point.
If you wish to project to a higher dimension, you can tweak n_components just like in Sklearn's PCA.
As dataset sizes are getting bigger, people are paying more attention to out-of-memory, multi-threaded data preprocessing tools to escape the performance limitations of Pandas.
One of the most promising tools in this regard is datatable, inspired by R's data.table package. It is developed by H2O.ai to support parallel-computing and out-of-memory operations on big data (up to 100 GB), as required by today's machine learning applications.
While datatable does not have as large a suite of manipulation functions as pandas, it is found to heavily outperform it on most common operations. In an experiment done on a 100M row dataset, datatable manages to read the data into memory in just over a minute, 9 times faster than pandas.
https://github.com/h2oai/datatable
https://datatable.readthedocs.io/en/latest/?badge=latest
The main data structure in datatable is Frame (as in DataFrame).
>>> type(frame)datatable.Frame
A simple GroupBy operation:
Lazypredict is one of the best one-liner packages I have ever seen.
Using the library, you can train almost all Sklearn models plus XGBoost and LightGBM in a single line of code. It only has two estimators — one for regression and one for classification. Fitting either one on a dataset with a given target will evaluate more than 30 base models and generate a report with their rankings on several popular metrics.
An insight like this will free you from the manual task of selecting a base model, a time much better spent on tasks like feature engineering.
https://lazypredict.readthedocs.io/en/latest/index.html
https://github.com/shankarpandala/lazypredict
One of the more recent libraries I have added to my skill-stack is Kagglers’ favorite — Optuna.
Optuna is a next-generation automatic hyperparameter tuning framework designed to work on virtually any model and neural network available in today’s ML and Deep learning packages.
It offers several advantages over similar tools like GridSearch, TPOT, HyperOPT, etc.:
Platform-agnostic: has APIs to work with any framework, including XGBoost, LightGBM, CatBoost, Sklearn, Keras, TensorFlow, PyTorch, etc.
A large suite of optimization algorithms with early stopping and pruning features baked in
Easy parallelization with little or no changes to the code
Built-in support to visually explore tuning history and the importance of each hyperparameter.
My most favorite feature is its ability to pause/resume/save search histories. Optuna keeps track of all previous rounds of tuning, and you can resume the search for however long you want until you get the performance you want.
Besides, you can make Optuna RAM-independent for massive datasets and searching by storing results in a local or a remote database by adding an extra parameter.
https://github.com/optuna/optuna
https://optuna.readthedocs.io/en/stable/
Optuna: A Next-generation Hyperparameter Optimization Framework
For the sake of simplicity, we are trying to optimize the function (x — 1)2 + (y + 3)2. As you can see, the tuned values for x and y are pretty close to the optimal (1, -3).
Hyperparameter tuning for real estimators is a bit more involved, so why don’t you check out my detailed guide:
towardsdatascience.com
Explainable AI (XAI) is one of the strongest trends in the ML and AI sphere. Companies and businesses are starting to be fussy over the adoption of AI solutions due to their “black box” nature.
Hey, no one can blame them. If data scientists themselves are coming up with tools to understand the models they created, the worries and suspicions of business owners are entirely justified.
One of those tools that often come up in grandmasters’ notebooks on Kaggle is SHAP.
SHAP (Shapley Additive exPlanations) is an approach to explain how a model works using concepts from game theory. At its score, SHAP uses something called Shapley values to explain:
Which features in the model are the most important
The model’s decisions behind any single prediction. For example, asking which features led to this particular output.
The most notable aspects of SHAP are its unified theme and unique plots that break down the mechanics of any model and neural network. Here is an example plot that shows the feature importances in terms of Shapley values for a single prediction:
Trust me, SHAP has way cooler plots. It is such a powerful tool that the Kaggle platform has an entire free course built around it.
https://shap.readthedocs.io/en/latest/index.html
https://github.com/slundberg/shap
A Unified Approach to Interpreting Model Predictions
From local explanations to a global understanding with explainable AI for trees
Explainable machine-learning predictions for the prevention of hypoxemia during surgery
Here is a short snippet to create a bee swarm plot of all predictions from the classic Diabetes dataset:
If you thought GPUs are deep learning-exclusive, you are horribly mistaken.
The cuDF library, created by the open-source platform RAPIDs, enables you to run tabular manipulation operations on one or more GPUs.
Unlike datatable, cuDF has a very similar API to Pandas, thus offering a less steep learning curve. As it is standard with GPUs, the library is super fast, giving it an edge over datatable when combined with its Pandas-like API.
The only hassle when using cuDF is its installation — it requires:
CUDA Toolkit 11.0+
NVIDIA driver 450.80.02+
Pascal architecture or better (Compute Capability >=6.0)
If you want to try out the library without installation limitations, Kaggle kernels are a great option. Here is a notebook to get you started.
https://docs.rapids.ai/api/cudf/stable/
https://github.com/rapidsai/cudf
Here is a snippet from the documentation that shows a simple GroupBy operation on the tips dataset:
Normally, I am against any library or tool that takes a programmer away from writing actual code. But, since auto-EDA libraries are all the rage now on Kaggle, I had to include this section for completeness.
Initially, this section was supposed to be only about AutoViz, which uses XGBoost under the hood to display the most important information of the dataset (that’s why I chose it). Later, I decided to include a few others as well.
Here is a list of the best auto EDA libraries I have found:
DataPrep — the most comprehensive auto EDA [GitHub, Documentation]
AutoViz — the fastest auto EDA [GitHub]
PandasProfiling — the earliest and one of the best auto EDA tools [GitHub, Documentation]
Lux — the most user-friendly and luxurious EDA [GitHub, Documentation]
If you want to see how each package performs EDA, check out this great notebook on Kaggle.
The available tools and packages to execute data science tasks are endless. Everyone has the right to be overwhelmed by this.
But I sincerely hope the tools outlined in this post helped narrow down your focus to what is trending and delivering results in the hands of members of the largest ML community.