Title: INTERPRETABLE COMPACT CATEGORICAL FEATURES ENCODING FOR SUPERVISED LEARNING
Primary area: interpretability and explainable AI
Keywords: Supervised Learning, Categorical Feature Encoding, Compression, Embedding

Abstract: In supervised learning, encoding techniques for continuous features are well studied. Despite the fact that categorical features often carry significant, if not a dominant portion of, feature information in a supervised learning problem the prevailing encoding techniques today are still one-hot encoding and target encoding in real world applications after decades. One hot encoding is known for its curse of dimensionality issue, especially when categorical features are of high cardinality and/or sparse. Such problem not only increases the problem size but may also introduce instability for numerical solvers. Target encoding is often used for categorical data with high cardinality. It is considered an ad-hoc approach that works well in many but not all cases. There is continued effort to study efficient methods to reduce dimension and retain interpretability. However, so far new algorithms developed by the research community are either too complicated or do not exhibit better performance than existing approaches. Our goal is to introduce an algorithm to address such practical issue. In this paper, we propose a polynomial  algorithm called Interpretable Compact Categorical Feature Encoding for Supervised Learning (ICFESL) to compress categorical features by collapsing categorical feature levels. Our encoding ensures no information loss for the regression problem and minimum information loss for the classification case. We prove that applying K-means clustering to the OLS or MLE coefficients yields optimal clustering schemas for the dimension reduction problem. We test our algorithm on larger real world datasets in both regression and classification problems and the results show that its performance is comparable or superior to existing methods.

TL;DR: Interpretable categorical feature encoding