Package: pyradox-tabular
pyradox-tabularState of the Art Neural Networks for Tabular Deep LearningTable of Contentspyradox-tabularTable of ContentsInstallationUsageData PreparationDeep Tabular NetworkWide and Deep Tabular NetworkDeep and Cross Tabular NetworkTabTansformerTabNetDeep Neural Decision TreeDeep Neural Decision ForestNeural Oblivious Decision TreeNeural Oblivious Decision EnsembleFeature Tokenizer TransformerTabular ResNetReferencesInstallationpipinstallpyradox-tabularUsageData Preparationpyradox-tabular comes with its ownDataLoaderClass which can be used to load data from a pandasDataFrame.
We provide a utilityDataConfigclass which stores the configuration of the data, which are then required by the model for feature preprocessing.
We also provide seperateModelConfigclasses for the different models, which ae required to store the model hyperparamers.frompyradox_tabular.dataimportDataLoaderfrompyradox_tabular.data_configimportDataConfigdata_config=DataConfig(numeric_feature_names=["numerical","column","names"],categorical_features_with_vocabulary={"column":["label","encoded","unique","values","as","strings"],},)data_train=DataLoader.from_df(x_train,y_train,batch_size=1024)data_valid=DataLoader.from_df(x_valid,y_valid,batch_size=1024)data_test=DataLoader.from_df(x_test,batch_size=1024)This library provides the implementations of the following tabular deep learning models:Deep Tabular NetworkIn principle a neural network can approximate any continuous function and piece wise continuous function. However, it is not suitable to approximate arbitrary non-continuous functions as it assumes certain level of continuity in its general form.Unlike unstructured data found in nature, structured data with categorical features may not have continuity at all and even if it has it may not be so obvious.Deep Tabular Network use the entity embedding method to automatically learn the representation of categorical features in multi-dimensional spaces which reveals the intrinsic continuity of the data and helps neural networks to solve the problem.frompyradox_tabular.model_configimportDeepNetworkConfigfrompyradox_tabular.nnimportDeepTabularNetworkmodel_config=DeepNetworkConfig(num_outputs=1,out_activation='sigmoid',hidden_units=[64,64])model=DeepTabularNetwork.from_config(data_config,model_config,name="deep_network")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)Wide and Deep Tabular NetworkThe human brain is a sophisticated learning machine, forming rules by memorizing everyday events and generalizing those learnings to apply tothings we haven't seen before. Perhaps more powerfully, memorization also allows us to further refine our generalized rules with exceptions.By jointly training a wide linear model (for memorization) alongside a deep neural network (for generalization) Wide and Deep Tabular Networks combine the strengths of both to bring us one step closer to teach computers to learn like humans do.frompyradox_tabular.model_configimportWideAndDeepNetworkConfigfrompyradox_tabular.nnimportWideAndDeepTabularNetworkmodel_config=WideAndDeepNetworkConfig(num_outputs=1,out_activation='sigmoid',hidden_units=[64,64])model=WideAndDeepTabularNetwork.from_config(data_config,model_config,name="wide_deep_network")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)Deep and Cross Tabular NetworkFeature engineering has been the key to the success of many prediction models. However, the process is nontrivial and often requires manual feature engineering or exhaustive searching. DNNs are able to automatically learn feature interactions; however, they generate all the interactions implicitly, and are not necessarily efficient in learning all types of cross features.Deep and Cross Tabular Network explicitly applies feature crossing at each layer, requires no manual feature engineering, and adds negligible extra complexity to the DNN model.frompyradox_tabular.model_configimportDeepAndCrossNetworkConfigfrompyradox_tabular.nnimportDeepAndCrossTabularNetworkmodel_config=DeepAndCrossNetworkConfig(num_outputs=1,out_activation='sigmoid',hidden_units=[64,64],n_cross=2)model=DeepAndCrossTabularNetwork.from_config(data_config,model_config,name="deep_cross_network")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)TabTansformerTabTransformer is built upon self-attention based on Transformers. The Transformer layers transform the embeddings of categorical features into robust contextual embeddings to achieve higher prediction accuracy.The contextual embeddings learned from TabTransformer are highly robust against both missing and noisy data features, and provide better interpretability.frompyradox_tabular.model_configimportTabTransformerConfigfrompyradox_tabular.nnimportTabTransformermodel_config=TabTransformerConfig(num_outputs=1,out_activation='sigmoid',num_transformer_blocks=3,num_heads=4,mlp_hidden_units_factors=[2,1])model=TabTransformer.from_config(data_config,model_config,name="tab_transformer")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)TabNetTabNet uses sequential attention to choose which features to reason from at each decision step, enabling interpretability and better learning as the learning capacity is used for the most salient features.It employs a single deep learning architecture for feature selection and reasoning.frompyradox_tabular.model_configimportTabNetConfigfrompyradox_tabular.nnimportTabNetmodel_config=TabNetConfig(num_outputs=1,out_activation='sigmoid',feature_dim=16,output_dim=12,num_decision_steps=5)model=TabNet.from_config(data_config,model_config,name="tabnet")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)Deep Neural Decision TreeDeep Neural Decision Trees unifies classification trees with the representation learning functionality known from deep convolutional network. These are essentially a stochastic and differentiable decision tree model.frompyradox_tabular.model_configimportNeuralDecisionTreeConfigfrompyradox_tabular.nnimportNeuralDecisionTreemodel_config=NeuralDecisionTreeConfig(depth=2,used_features_rate=1,num_classes=2)model=NeuralDecisionTree.from_config(data_config,model_config,name="deep_neural_decision_tree")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)Deep Neural Decision ForestA Deep Neural Decision Forest is an bagging ensemble of Deep Neural Decision Trees.frompyradox_tabular.model_configimportNeuralDecisionForestConfigfrompyradox_tabular.nnimportNeuralDecisionForestmodel_config=NeuralDecisionForestConfig(num_trees=10,depth=2,used_features_rate=0.8,num_classes=2)model=NeuralDecisionForest.from_config(data_config,model_config,name="deep_neural_decision_forest")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)Neural Oblivious Decision Treefrompyradox_tabular.model_configimportNeuralObliviousDecisionTreeConfigfrompyradox_tabular.nnimportNeuralObliviousDecisionTreemodel_config=NeuralObliviousDecisionTreeConfig()model=NeuralObliviousDecisionTree.from_config(data_config,model_config,name="neural_oblivious_decision_tree")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)Neural Oblivious Decision EnsembleNODE architecture generalizes ensembles of oblivious decision trees, but benefits from both end-to-end gradient-based optimization and the power of multi-layer hierarchical representation learning.frompyradox_tabular.model_configimportNeuralObliviousDecisionEnsembleConfigfrompyradox_tabular.nnimportNeuralObliviousDecisionEnsemblemodel_config=NeuralObliviousDecisionEnsembleConfig()model=NeuralObliviousDecisionEnsemble.from_config(data_config,model_config,name="neural_oblivious_decision_ensemble")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)Feature Tokenizer TransformerIt is a simple adaptation of the Transformer architecture for the tabular domain. In a nutshell, Feature Tokenizer Transformer transforms all features (categorical and numerical) to embeddings and applies a stack of Transformer layers to the embeddings.Thus, every Transformer layer operates on the feature level of one object.frompyradox_tabular.model_configimportFeatureTokenizerTransformerConfigfrompyradox_tabular.nnimportFeatureTokenizerTransformermodel_config=FeatureTokenizerTransformerConfig(num_outputs=1,out_activation='sigmoid',num_transformer_blocks=2,num_heads=8,embedding_dim=32,dense_dim=16)model=FeatureTokenizerTransformer.from_config(data_config,model_config,name="feature_tokenizer_transformer")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)Tabular ResNetTabular Resnet is a ResNet like architecture containing skip connection but instead of Convolutional Layers, it consists of Linear Layers.frompyradox_tabular.model_configimportTabularResNetConfigfrompyradox_tabular.nnimportTabularResNetmodel_config=TabularResNetConfig(num_outputs=1,out_activation='sigmoid',hidden_units=[64,64])model=TabularResNet.from_config(data_config,model_config,name="deep_network")model.compile(optimizer="adam",loss="binary_crossentropy")model.fit(data_train,validation_data=data_valid)preds=model.predict(data_test)ReferencesEntity Embeddings of Categorical Variables (2016, April)Wide & Deep Learning: Better Together with TensorFlow (2016, June)Deep & Cross Network for Ad Click Predictions (2017, August)TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, December)TabNet: Attentive Interpretable Tabular Learning (2020, December)Deep Neural Decision Forests (2015, December)Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data (2019, September)Revisiting Deep Learning Models for Tabular Data (2021, June)