The Right Way to Use Deep Learning for Tabular Data | Entity Embedding | by Benjamin Lau | Towards Data Science

Recently I took part in a Deep Learning course where I was tasked to work on a project that utilized deep learning techniques. Immediately, the thought of using deep learning or neural networks for tabular data struck my mind as I had worked on structured data for the first 6 months of my career.
The use of deep learning in tabular data was not a new idea. People have been using fully-connected neural networks for these data for the longest time but it came with a few disadvantages;
Large amount of data needed for neural networksInsignificant gains in model’s performance compared to more popular machine learning algorithms (e.g Random forest, Gradient boosting trees)Lack of interpretability
Large amount of data needed for neural networks
Insignificant gains in model’s performance compared to more popular machine learning algorithms (e.g Random forest, Gradient boosting trees)
Lack of interpretability
As such, using deep learning for tabular data never really took off. This is true until practitioners transfer the idea of embedding networks used in Natural Language Processing (NLP) to tabular data.
This is how it works.
Traditionally, categorical variables such as ‘Gender’ or ‘Race’ were processed using One-hot encoding, where each entity in the variable was given a unique column to track the presence of the entity.
This method, although convert the data into a suitable format for machine learning algorithms, it assumes independence among entity and results in sparse matrices.
Entity embedding on the other hands uses a vector to represent each entity instead of a binary value. This has a few advantages;
Remove the sparse matrices problem of inefficient computationProduce vectors that show the relationship between each entity (derive additional insights instead of treating them as independent)
Remove the sparse matrices problem of inefficient computation
Produce vectors that show the relationship between each entity (derive additional insights instead of treating them as independent)
Hence, I decided to do a project using tabular to demonstrate the use of entity embeddings. The dataset I use is the IEEE-CIS Fraud Detection data from Kaggle which you can find here.
Here are the step-by-step codes (including Google Colab specific code as I worked on colab).
Firstly, to check the GPU allocated to you in colab you can run the following code.
!nvidia-smi
Next to mount to my google drive,
from google.colab import drivedrive.mount('/content/drive')
Download the dataset from Kaggle, you would need your Kaggle API token for this. If you need any help downloading dataset from Kaggle, this might help.
!mkdir /root/.kaggle!echo '{"username":"USERNAME","key":"KEY"}' > /root/.kaggle/kaggle.json!chmod 600 /root/.kaggle/kaggle.json!kaggle competitions download -c ieee-fraud-detection# unzip all files!unzip train_transaction.csv.zip!unzip test_transaction.csv.zip
Then, read the csv files into pandas dataframe
train = pd.read_csv("train_transaction.csv")test = pd.read_csv("test_transaction.csv")
As this is a fraud detection dataset, having an imbalance data shouldn’t be surprising.
train["isFraud"].mean() #  0.03499000914417313
As data exploration and feature engineering is not the purpose of this post, I will use minimum features to predict the fraud label. To make sure you can replicate my code, here are my processing steps.
# generate time of daytrain["Time of Day"] = np.floor(train["TransactionDT"]/3600/183)test["Time of Day"] = np.floor(test["TransactionDT"]/3600/183)# drop columnstrain.drop("TransactionDT",axis=1,inplace=True)test.drop("TransactionDT",axis=1,inplace=True)# define continuous and categorical variablescont_vars = ["TransactionAmt"]cat_vars = ["ProductCD","addr1","addr2","P_emaildomain","R_emaildomain","Time of Day"] + [col for col in train.columns if "card" in col]# set training and testing setx_train = train[cont_vars + cat_vars].copy()y_train = train["isFraud"].copy()x_test = train[cont_vars + cat_vars].copy()y_test = train["isFraud"].copy()# process cont_vars# scale valuesfrom sklearn.preprocessing import StandardScalerscaler = StandardScaler()x_train["TransactionAmt"] = scaler.fit_transform(x_train["TransactionAmt"].values.reshape(-1,1))x_test["TransactionAmt"] = scaler.transform(x_test["TransactionAmt"].values.reshape(-1,1))# reduce cardinality of categorical variablesidx_list = x_train["card1"].value_counts()[x_train["card1"].value_counts()<=100].index.tolist()x_train.loc[x_train["card1"].isin(idx_list),"card1"] = "Others"x_test.loc[x_test["card1"].isin(idx_list),"card1"] = "Others"# fill missingx_train[cat_vars] = x_train[cat_vars].fillna("Missing")x_test[cat_vars] = x_test[cat_vars].fillna("Missing")for cat, index in categories.items():test[cat] = pd.Categorical(test[cat],categories=categories[cat],ordered=True)
After the processing steps have been done, now we can convert the categorical variables into integers.
# convert to numerical value for modellingdef categorify(df, cat_vars):categories = {}for cat in cat_vars:df[cat] = df[cat].astype("category").cat.as_ordered()categories[cat] = df[cat].cat.categoriesreturn categoriesdef apply_test(test,categories):for cat, index in categories.items():test[cat] = pd.Categorical(test[cat],categories=categories[cat],ordered=True)# convert to integerscategories = categorify(x_train,cat_vars)apply_test(x_test,categories)for cat in cat_vars:x_train[cat] = x_train[cat].cat.codes+1x_test[cat] = x_test[cat].cat.codes+1
Due to the higly imbalanced dataset, I have to artificially generate more fraud data using a technique called Synthetic Minority Over-sampling Technique (SMOTE). The documentation can be found here.
from imblearn.over_sampling import SMOTEsm = SMOTE(random_state=0)x_sm, y_train = sm.fit_resample(x_train, y_train)x_train = pd.DataFrame(x_sm,columns=x_train.columns)
Finally, we can work on modelling the data. This implementation of the entity embedding took the idea and best practices from Jeremy Howard’s ‘Introduction to Machine Learning for Coders’ course. Hence, many of the technical specifics such as embedding sizes and hidden layers were selected by Jeremy through his research and experiences. Anyway, if you have not seen the course, I highly recommend you take the course and listen to Jeremy approach to machine learning. It is completely free too.
# get embedding size for each categorical variabledef get_emb_sz(cat_col,categories_dict):num_classes = len(categories_dict[cat_col])return int(min(600,round(1.6*num_classes**0.56)))
Now to define the neural networks. The architecture of the networks is simply the concatenation of continuous variables with embedding layers for each categorical variables.
# define the neural networksfrom tensorflow.keras.layers import Input, Embedding, Dense, Reshape, Concatenate, Dropout, BatchNormalizationfrom tensorflow.keras import Modeldef combined_network(cat_vars,categories_dict,cont_vars, layers):inputs = []embeddings = []emb_dict ={}# create embedding layer for each categorical variablesfor i in range(len(cat_vars)):emb_dict[cat_vars[i]] = Input(shape=(1,))emb_sz = get_emb_sz(cat_vars[i],categories_dict)vocab = len(categories_dict[cat_vars[i]]) +1embedding = Embedding(vocab,emb_sz,input_length=1)(emb_dict[cat_vars[i]])embedding = Reshape(target_shape=(emb_sz,))(embedding)inputs.append(emb_dict[cat_vars[i]])embeddings.append(embedding)# concat continuous variables with embedded variablescont_input = Input(shape=(len(cont_vars),))embedding = BatchNormalization()(cont_input)inputs.append(cont_input)embeddings.append(embedding)x = Concatenate()(embeddings)# add user-defined fully-connected layers separated with batchnorm and dropout layersfor i in range(len(layers)):if i ==0:x = Dense(layers[i],activation="relu")(x)else:x = BatchNormalization()(x)x = Dropout(0.5)(x)x = Dense(layers[i],activation="relu")(x)output = Dense(1,activation="sigmoid")(x)model = Model(inputs,output)return model
Now to initialize the model.
layers = [200,100]model = combined_network(cat_vars,categories,cont_vars, layers)opt = tf.keras.optimizers.Adam(0.0001)model.compile(optimizer=opt,loss='binary_crossentropy',metrics=["accuracy"])
Process the input for the neural networks.
# process x_train input to fit modelinput_list = []for i in cat_vars:input_list.append(x_train[i].values)input_list.append(x_train[cont_vars].values)# modify x_test input to fit modeltest_list = []for i in cat_vars:test_list.append(x_test[i].values)test_list.append(x_test[cont_vars].values)
Train the model~
model.fit(input_list,y_train,epochs=10)
Making prediction
y_pred = model.predict(test_list)# choose a optimal thresholdy_pred = y_pred>0.1
Lastly, the competition ranks leaderboard using ROC-AUC. Hence we compute our ROC-AUC to see how we fare against others.
roc = metrics.roc_auc_score(y_test,y_pred)roc  0.8787572838641191
Not that bad for a model using only some of the features.
This is it. Using entity embedding for Kaggle tabular data competition. Hope you enjoy it.