TabCaps: A Capsule Neural Network for Tabular Data Classification with BoW Routing
Authors: Jintai Chen
KuanLun Liao
Yanwen Fang
Danny Chen
Jian Wu
Venue: ICLR 2023 poster (2023)
Keywords: capsule neural network

Abstract: Records in a table are represented by a collection of heterogeneous scalar features. Previous work often made predictions for records in a paradigm that processed each feature as an operating unit, which requires to well cope with the heterogeneity. In this paper, we propose to encapsulate all feature values of a record into vectorial features and process them collectively rather than have to deal with individual ones, which directly captures the representations at the data level and benefits robust performances. Specifically, we adopt the concept of "capsules" to organize features into vectorial features, and devise a novel capsule neural network called "TabCaps" to process the vectorial features for classification. In TabCaps, a record is encoded into several vectorial features by some optimizable multivariate Gaussian kernels in the primary capsule layer, where each vectorial feature represents a specific "profile" of the input record and is transformed into senior capsule layer under the guidance of a new straightforward routing algorithm. The design of routing algorithm is motivated by the Bag-of-Words (BoW) model, which performs capsule feature grouping straightforwardly and efficiently, in lieu of the computationally complex clustering of previous routing algorithms. Comprehensive experiments show that TabCaps achieves competitive and robust performances in tabular data classification tasks.

Published as a conference paper at ICLR 2023
TABCAPS: A CAPSULE NEURAL NETWORK FOR TABU- LAR DATA CLASSIFICATION WITH BOW ROUTING
Jintai Chen', Kuanlun Liao', Yanwen Fang", Danny Z. Chen', Jian Wu
'College of Computer Science and Technology, Zhejiang University Department of Statistics & Actuarial Science, The University of Hong Kong
Department of Computer Science and Engineering, University of Notre Dame Department of Public Health, Zhejiang University School of Medicine
ABSTRACT
Records in table are represented by a collection of heterogeneous scalar features. Previous work often made predictions for records in paradigm that processed each feature as an operating unit, which requires to well cope with the heterogeneity. this paper, propose to encapsulate all feature values of record into vectorial features and process them collectively rather than have to deal with individual ones, which directly captures the representations at the data level and benefits robust performances. Specifically, we adopt the concept of capsules to organize features into vectorial features, and devise novel capsule neural network called TABCAPS to process the vectorial features for classification. In TABCAPS, a record is encoded into several vectorial features by some optimizable multivariate Gaussian kernels in the primary capsule layer, where each vectorial feature represents a specific profile of the input record and is transformed into senior capsule layer under the guidance of a new straightforward routing algorithm. The design of routing algorithm is moti- vated by the Bag-of- Words (BoW) model, which performs capsule feature grouping straightforwardly and efficiently, in lieu of the computationally complex clustering previous routing algorithms. Comprehensive experiments show that TABCAPS achieves competitive and robust performances in tabular data classification tasks. Codes are available at witttpss: /ttpt::////tuubb// com/ watoo///Waathhhsaaaaaaaaa
INTRODUCTION
Tabular data are ubiquitous in real world applications, which records abundantly meaningful informa- tion such as medical examination results (Hassan et al. 2020) and company financial statements (Addo) et al. 2018). Previous methods often processed record by treating the scalar feature values the operating units. For example, decision tree based methods (Breiman et al.) 1984 Chen & Guestrin 2016) used one tabular feature in each decision step. and neural networks (Gorishniy et al.) 1 Chen et al. 2022) elaborately executed saie interactions to capture higher-level semantics.
However, it intractable design effective Ceature-wise interaction approaches (Grinsztajn et al. 2022 Ng. 2004) due to the heterogeneity among features. In this paper, we propose a novel paradigm for supervised tabular learning, which encapsulates all feature values of records into vectorial features and directly conducts on the vectorial feature level. Such design utilizes the sufficient representation space of the vectorial feature format to probably learn the comprehensive data level semantics, and avoids executing complex interactions among heterogeneous features.
To this end, we borrow the concept of capsules (Sabour et al.) 2017) to organize vectorial features, and propose novel capsule neural network (CapsNett called TABCAPS for tabular data classification. In TABCAPS, several optimizable multivariate Gaussian kernels encode all the features of each record into the primary capsules, in which features in vector format represent the marginal likelihoods of the record in reference to the corresponding multivariate Gaussian distributions. We set the scale and location parameters of these Gaussian kernels learnable, thus allowing these kernels to model some plausible data patterns for the dataset and each primary capsule feature represents a specific profile
Corresponding author. E-mail: wuji 1 an2.000 100000zjj....... www.ju..du...om edu.ch.

---

Published as a conference paper at ICLR 2023
Neguarsii alecie adial Mosioo
ticaano stald
V/ order)
PC. Foatures
Tabular Foature Extraction
Concalenation
Sporse Projection by Entmaxs (W))
PC, Votes
Multiveraae Goussion Kernels
intollzedd s.C. Features y
Computing Average
PC Features
optimiuble threshold
Space
Sporse Tronsformation Routing
RC. Votes Selecting
%
S.C. Features
Voting
Vating
Voting
Features Lindoting
BoW Routing Alg.
Predicted Probebilties
F.Ge
(a) An illustration the proposed TABCAPS.
(b) The process of the j-th Senior Capsule Feature Synthesis.
Figure 1: Illustrating (a) our proposed TABCAPS (without the decoder) and (b) the process of senior capsule feature synthesis from features in primary capsules (taking features the j-th senior capsule example). "P.C" denotes "primary capsule" and "S.C." denotes "senior capsule".
of the input record that measures the likelihood to these data patterns. Unlike previous CapsNNts that used one senior capsule to predict the belonging probability for one class, we allot multiple senior capsules for class, motivated by ensemble learning. In previous CapsNets, primary capsule features were transformed into senior capsules by an affinity projection and a routing algorithm (Sabour al., 2017 Hinton et al. 2018) that groups similar primary capsule features by clustering processes. In TABCAPS, a novel sparse projection method and a novel straightforward routing algorithm are proposed to perform the feature transformation from primary to senior capsules.
Our proposed routing algorithm is much more efficient than previous routing algorithms, since the primary capsules in previous CapsNets captured some unknown semanties from unstructured data (e.g., images) and had to apply feature clustering in an iterative process to attain higher-level semantics. In our TABCAPS, features in primary capsules are the likelihood w.r.t. the Gaussian distri- butions, which represents stable semantics and allows to perform a straightforward routing algorithm. Motivated by the bag-of-wo wo-ds (BoW) model (Salton & Lesk 1965) which is efficient in similar information search (Pineda et al., 2011), we propose a straightforward routing (called BoW Routing) for TABCAPS. The proposed BoW Routing computes similarities between primary capsule features and the initialized senior capsule features by counting the co-occurrences of "words" (implemented to learnable templates), and incorporates those primary capsule features whose similarities are beyond adaptive threshold to update senior capsule features. Similar to previous CapsNets, also present tabular-data suited decoder, which tailored for tabular feature reconstruction tasks (e.g., missing value imputation).
Contributions. (i) For the first time, we propose to encapsulate the features of a record in table into vectorial features as operating units, which avoids inefficient interactions between heterogeneous tabular features and directly learns the data-level semantics. (ii) We propose a CapsNet tailored for supervised tabular learning, in which multivariate Gaussian kernels and conduct a novel straightforward routing to synthesize senior capsule features low costs. (iii) Experiments on real-world datasets validate that TABCAPS attain robust performances in tabular data classification.
devise
type of primary capsule with learnable
RELATED WORK
Capsule Neural Networks. Capsule neural networks (Ribeiro 2022 were first proposed to deal with image data, where a capsule represents an object whole or object part in vector or matrix format. Firstly, primary capsules capture the basic object parts by using templates (Kosiorek et al. 2019), optical flow analysis (Sabour et al., (0211, or location-basee
operations (Hinton et al. 201 8). Features in primary capsules are linearly projected to transpose the poses of object parts, so as to synthesize senior capsule features (representing object wholes) guided by a routing algorithm. Most routing algorithms performed iteration processes at high computational costs (Sabour et al. 2017 Hinton al.) 2018). since the primary capsules handle different semantics in dealing with different samples. To avoid iterations, some straightforward approaches were proposed (Choi et al. 2019
Ahmed & Torresai 2019 Chen et al. 2021 Ribeiro al. 2020). However, these straightforward
2

---

Published as a conference paper at ICLR 2023
approaches were for image tasks and some of them got rid of the part-to-whole learning assumption.
At present, CapsNets have been widely used for image processing Rajasegaran et al. 2019 Hinton
et al. 2018; Chen et al. 202I). point clouds (Zhao et al.] 2019b 2020), recommendation shop- ping (Li et al. 2019). and language tasks (Zhao et al. 2019a). However, to our best knowledge, no bespoke CapsNets were proposed for general supervised tabular learning tasks. This paper presents novel CapsNet for tabular data, in which primary capsules learn some specific kinds of data patterns for the dataset rather than the data-adaptive "object parts".
Supervised Tabular Learning.
Various methods were proposed for tabular data processing, mainly
including decision tree algorithms (Breiman et al. [984] Quinlan 1979 Chen & Guestrin, 2
Prokhorenkova et al., 2018 Ke et al. (017) and neural network (NN) based methods (Arik & Pfister
2020:Poppv al. 2019 Chen et al. 2022 Abutbul et al. 2021). Different from tree based methods
that split tabular feature spaces by some information metrics (Quinlan 1979| 2014 Breiman et al.
1984) and sought for possible decision paths, NN based methods often introduced attention and
grouping mechanisms (Chen et al. 2022 Arik & Pfister 2020] Guo et al.) 2017 Wang et al.) 2017]
Gorishnly al. [021]] Yan et al. 2002) for tabular feature interactions yield high-level features.
However, tabular features in table are heterogeneous and are relation-agnoste and thus it is hard to
conduct all the feature interactions in the right way.
3
METHODOLOGY
We propose to classify each record in table by processing a vectorial feature containing all its feature values in order. See the pipeline of our TABCAPS in (ig.(Ta), to better depict an input record, its raw features are first processed by a simple function (e.g., an Abstract Layer (Chen et al., 2022) or a linear projection layer) to yield more comprehensive features, which are then concatenated with raw features and we obtain the basal vectorial feature of size m, E TR. Then, the vectorial feature is projected into primary capsules (in a vector format) by various learnable multivariate Gaussian kernels. Each vectorial feature (in a primary capsule) is transformed into feature votes for senior capsules by the proposed sparse projection (see Sec. 3.2.1), D. and senior capsule features are finally obtained by selecting and merging feature votes, guided by our BoW Routing (see Sec. 3.2.2).
3.1
HOw TO ENCODE RECORDS INTO PRIMARY CAPSULES?
Unlike the known CapsNets for images that utilized various primary capsules to represent different parts of objects, in TABCAPS, different primary capsules represent various profiles for just one record. Formally, a record represented by a vectorial feature x E RT is transformed into feature u, E R for the i-th primary capsule by multivariate Gaussian kernel hi, as:
U1 = (2; E,) cxp(- 1 (x 14 E (x -Me))
(I)
(2T (mr))/2p2 E,
where the location parameter vector μ4 E R" and the scale parameters E, = diag (1 ................................ of the multivariate Gaussian kernel k; are set to be learnable, so that TABCAPS allows to use proper multivariate Gaussian kernels in different datasets. Since features in table are typically regarded as independent, we set E as diagonal matrix. After Eq. (T). a layer normalization is performed on each u; (i 1, ... Np. N, is the count of the primary capsules) for normalization following Tsai et al. (2019). Specially, we also encapsulate the original feature values (denoted by x) to be an additional capsule feature.
Tm)
This design can be interpreted that a primary capsule models an data pattern by determining the parameters (p, Σ) of the multivariate Gaussian kernels, and various primary capsule features of an input record are represented as the likelihoods in reference to these data patterns. It is beneficial to find some dapaset-specific data patterns, which allows implicitly model the tabular feature relations.
3.2 HOW TO SYNTHESIZE SENIOR CAPSULE FEATURES?
Motivated by ensemble learning, we use multiple senior capsules to learn the semantics for one target class and obtain the final prediction by voting, which also meets the need of irregular target pattern fitting (Grinsztajn et al.] (022). To attain the senior capsule features, primary capsule features are transformed by sparse weight matrix into feature votes, and newly proposed BoW Routing selects feature votes to synthesize senior capsule features. Note that the elements in vectorial features are linearly transformed (reversible), and the non-linear mapping (the routing) is performed by taking vectorial features as operating units.
3

---

Published as a conference paper at ICLR 2023
3.2.1 SPARSE PROJECTION
The affiinity projection in previous work (Sabour al. 2017 uses learnable weight W Wi E R to linearly map the i-th primary capsule Teature ;; E RT to yield a feature vote ūzja E IR" for the
j-th senior capsule, by Agj; u, j|c- We modify the weight matrix by adding sparse operation entmax (Peters et al. 2019). and the transformation is computed by:
KgJs u, Wijo W = entmax (Waje),
(2)
where a = 1.5 as default. In this way, the weight matrix Wije shall be sparse and only parts of elements of Uj i is used to synthesize the feature voles and the target-irrelevnt elements are excluded. Different from sparse attention operations, the enimax is applied to the weight matrix. Thus, the capsule feature component selection is consistent for different records.
3.2.2 BOW ROUTING
No. Given the synthesized feature votes ūg]: for the j-th senior capsule (where i = ..,..... j ..2..... N₁, N, and N₂ are the amounts of primary capsules and senior capsules respectively), we use routing algorithm to synthesize senior capsule features by selecting and merging újji- Note that the vanilla routing algorithm was a clustering-like approach (Sabour et al. (017). because a primary capsule may capture any kinds of object parts from images and thus the "cluster centroids" are unknown before the routing step. In our model, each primary capsule learns concrete semantics that represent record profiles in reference to specific multivariate Gaussian distribution, making it possible to predict the initialized senior capsule features ("cluster centroids") directly using a straightforward function w.r.t. semantics (features) captured by primary capsules. Here simply use the average of the feature votes uززs to initialize senior capsule features v/ by v/j = Σ.'' ugx:Npp
To select some feafure voles for senior capsule feature synthesis, we measure the similarities between the initialized senior capsule feature v/ and the primary capsule feature vote ūgJ(- An intuitive solution is to compute the similarity directly. However, directly computing similarity might hinder the performance, which overlooks the special data patterns and importance differences of different vectorial feature elements. Motivated by the success of the bag-of-words, (BoW) model in similarity measure for information retrieval, we introduce the BoW model to the routing algorithm. A classical BoW model first clusters the given features, and collects word dictionary whose words the centroids of feature clusters. Then, a feature can be quantized as the "occurrence" of various words, and feature similarities can be computed by counting the co-occurring words. In this way, the similarity is computed in the semantic space guided by the "words" in discrete steps. Our differentiable BoW Routing depicted as follow.
Occurrence status computing. The dictionary words in the traditional BoW model are computed according to the extracted features. Motivated by Chen et al. (202I), we directly implement the word dictionary with learnable parameter set of size M, {smm Sm E R" (of the identical size as the feature votes (gjp). To compute the "occurrences" of the "words", an occurrence status (a scalar) between ūزز; and the initialized senior capsule feature vi for the "wordd" sm is defined by:
1.2..... M). with each "word"
Pjlim sigmoid (1);: (/x)) (0, 1), Pjm = sigmoid (o)² s'm)) (0,
(3)
where Pylim and Pj.m represent the degrees (i.c., the occurrence statuses) in which udjje and o/TT reveal the semantics of Sm. we let Pyje = [P Pjli:.... ....:....... ...j..,........d and p, [p Pj:2... Pj.M] represent semantic codes of ūz and v/. In computing Eq. (3). the "word" 8m is normalized into 8'm by the l/ norm avoid the impact of the vector lengths. In traditional BoW models, an occurrence status is represented as binary (0 or 1), but by our routing algorithm. we generalize the discrete binary status to a continuous status form (in (0, 1)) using the "sigmoid" function.
Feature vote merging. To compute the similarity between the occurrence statuses P and Pj- traditional BoW models may use the Jaccard distance. Since the occurrence statuses are continuous values in routing algorithm, to which the Jaccard distance cannot be simply applied, we compute the pairwise distances dy:s between Pyjı and p, using the squared Euclidean distance function D:
dzA = D(p,)i i p,) = >>M
m=1 (Pili. Ppmm)?
(4)
4

---

Published as a conference paper at ICLR 2023
Note that the squared Euclidean distance has been proved to be effective approximation of the Jaccard distance (Reina et al., 2014). Based on the pairwise distances dzj= of feature votes and initialized senior capsule features, can directly obtain the feabb votes úzj; to update the features in the j-th senior capsule by performing k-Nearest Neighbor (k-NN) search. However, we find that k-NN can lead instability in training, since the distributions of feature votes are unknown and N-NN might select some feature votes with extreme values.
To gain training stability, we propose to use an adaptive threshold to determine a range around the initialized senior capsule feature v/ (see Fig. [b)). We set this threshold as a learnable positive value
ya O, and the selection of feature votes is determined by:
(5) where e(-) is the Heaviside function, and if and only if qz/z 1, it indicates that the feature vote from the i-th primary capsule, uzjs is selected. Then, we compute the weights for the feature voles and obtain the final senior capsule feature vy by a weighted sum, as:
931: =(>² dgje),
W = softmax (dyle 31)), Vg w E, Ug)1
(6)
where the "softmax" operation guarantees all wyss > 0 (giving small weights to unselected votes), avoiding possible failure if no feature vote is selected to yield vy. In this way, our proposed BoW Routing performs straightforward process to obtain the senior capsule features.
Remarks. Distinct from previous straightforward routing algorithms (Chen et al. 2021: Ribeiro et al., 2020) for images, the efficiency of our BoW Routing partially results from the innovative definition of the primary capsules that learns concrete semantics, allowing to predict initialized senior capsule features directly, in lieu of learning unknown object parts and requiring clustering- like routing algorithms to group them. Our BoW Routing algorithm, with its strong ability to extract semantic patterns and flexible measurement of similarity, allows us to utilize straightforward average method for initializing the senior capsule features.
3.3 SENIOR CAPSULE VOTING FOR CLASSIFICATION
Similar to the known CapsNets, we let the vector lengths of the senior capsule features represent the existence probabilities of one semanties, and this work models each target class by multiple senior capsules motivated by ensemble learning. Suppose there are K classes. We divide all the senior capsules into K groups, such that the features in each group vote for one particular class. The classification outcomes are defined by:
(7) where the vector c represents the predicted probabilities for all the K classes, and ÿ denotes the predicted label. A value 4 in Eq. denotes the mean of the senior capsule feature vector lengths for the k-th class (i.e., in the group GR), which is defined by: 4 = Ejeg, [tsy|le/||G</l Note that we dropout 20% senior capsules in a group before voting in training. We use the margin loss in training our TABCAPS following Sabour et al. (2017), as:
c softmax (l....., (k)) á = ar)m
K
C
Th max e+ 4.) + A(1 Tk) max(0, -t
(8)
where TB = if the k--th class is the target, and otherwise T = O. We set = 0.9, t- = =0.1, and and
0.5 following Sabour et al. (2017).
3.4 A DECODER FOR TABULAR FEATURE RECONSTRUCTION
We build a decoder based on the Abstract Layers CCen al. 2022) for tabular feature reconstruction. We stack two Abstruct Layers in sequence with a shorteut performing feature concatenation, as shown in Fig. 2 Finally. a fully connected layer is on top of the decoder to produce the reconstructed tabular features. Since there are multiple senior capsules voting for a target class, the decoder input is the concatenation of features in the group of senior capsules that votes for the correct class. We do not use the decoder as regularization like previous CapsNets (Sabour et al. 2017) and TABCAPS without the decoder can perform well on classification tasks (see Fig. 3). The decoder is used when we attempt to reconstruct raw features for some tasks (e.g., missing value imputation), which is trained under the specification of the mean square error (MSE) that is scaled down by x 10-6 and jointly works with the loss L (in Eq. (8).
5

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Datasets. We conduct experiments on 8 real- world tabular datasets, including Heart Failure Prediction (Heart) (Chicco & Jurman 2020),
Click (Cup 2012), Diabetes (Ukani (020), EEG
Eye State (EEG) (Dua & Graff][0155 Gas Con-
Correct
Voting
Voting
Voting
centrations (Gas) (Vergara
al.] 2012), Hill-
Class Concotenation
Valley (Hill) (Lee & Franz| 2008). Higgs-small (Higgs) (Baldi al.) 20114) and Epsilon (Chal-
lenge, 2008). All of these datasets have few or no categorical features, eliminating the impacts of categorical feature embedding approaches in model comparison. In pre-processing, data are standardized by the z-score normalization. We follow the train-test split and the data pre- processing in Popoo et al. 2019) for the Click dataset. For the other datasets, we randomly di- vide 80% of the data into the training sets, and the rest 20% are for test. We further sample 15% training data for validation. Experiments were run for 5 times with regard to random seeds.
Abstrect
Concani
S.C. Fectures
Rec. Tobular Feature
Predicted Class Probobility
The decoder for reconstructing raw tabular features. The decoder input is the senior capsule features in the group that votes for the correct class. "Rec." "Reconstructed" and "S.C." "Senior Capsule".
Figure 2:
Implementation details and comparison baselines. Our method is implemented with Py PyTorch on Python 3.7, and is run on GeForce RTX 2080 Ti. In training, we set the batch size as 2,048. We use the QHAdam optimizer (Ma & Yarats 2019) with weight decay rate 10 v (0.7, 0.99), and 3 = (0.95, 0.998). We compare TABCAPS with several baseline models, including TabNet (Arik
al., 202T). DANet- 24 (Chen et al. (202)) Net-DNF (Abutbul et al., 2021). XGboost (Chen & Guestrin| 2016). Cat- Boost (Prokhorenkova et al. 2018), the Vector CapsNet (Sabour et al. 2017), and fully connected neural networks (FCNNs) (Nair & Hinton 2010) with lasso and mixup (Zhang al. 2018) (a 1.0) respectively. The Vector CapsNet (Sabour et al. (017)) was designed for images; we replace its convolution layer by fully connected layer for tabular data. Following PPopov et al. (019): Arik & Pfister 2020). the performances of all the methods are hyperparameter-tuned on the validation sets (e.g., by grid search). The hyperparameter search spaces and search algorithms are given in Appendix 1 We employ the "log-loss" metric to measure the model performances, which is considered as a more sensitive metric than accuracy, and is widely used (e.g., in Kaggle competitions). All the performances of the compared methods are obtained by running official codes.
& Phister 2020). NODE (Popov et al.] 2 FT-Transformer (Gorishnly
4.2 CLASSIFICATION PERFORMANCE COMPARISON
The classification performances are reported in Table One can see that our TABCAPS obtains the best or second best performances on 7 out of 8 datasets, and is especially superior on some datasets by clear margins. addition, we also report the computational complexity and efficiency on the Diabetes dataset, using the metrics of model size (# param.) and fps (the amount of records that model processes per second). Since XGboost and CatBoost are traditional machine learning models, we do not report their complexity. One can see that TABCAPS is much more efficient and greatly lighter (over 10x) than the other state-of-the-an art methods (c.g., FT-Transformer, Net-DNF, and 24N,t-24)), and significantly outperforms the models with similar model complexity (e.g., regularized FCNNs and Vector CapsNet). Also, the results compared with Vector CapsNet shows the superiority of TABCAPS, obtaining better performances with lower complexity on tabular data.
4.3 EXTREME GENERALIZATION TEST
To further examine the generalization capability of TABCAPS, we attempt to test it with train-test split that the training data and test data are with very dissimilar feature values. We conduct t-SNE projections (with default hypee--araaeters of the "sklearn" package ! on all the records of datasets,
hittps://scikee 1earn rn. weaar.oorg//e wwwgg//tale/////////ooo// http:///ttaaeeeeooooee//e/eerree
learn .man fold.
TSNK html

---

Published as a conference paper at ICLR 2023
Table 1: Classification Performances. The best and second best performances of deep learning approaches respectively marked in bold and underlined. Note that the reported log-loss values (the lower the better) are with x00x factor. The model size (# param.) and inference speed (fps) are on the Diabetes dataset. The
performances are reported as "mean
Method
Click
Diabetes
EEG
Gas
Heart
Hill
Higgs
Epsilon
|# param.
lps
XGioonst
14,338_0m
14.117
2.087-0m
32.371
69.049_. 1+
53.158 13.158_5mm
26. 26.748
Catboost
64.273a0ss
14.777
18,423
2.064
30.043auaa
69. 174
27.228
TabNet
-2.303 17.964-cu
+5.340
44.967 4.647-to
-7
54.668
26.743-auc
3.4M
73.1
Net-DNF
67. 67.633-mm =3
17.386+0a
1.229
55.371
15.787
53.417
27.122=0m
8.5M
175.2
NODE
63.206-uus
45.951 =oue
47.654
38.774
46,541 awss
69.222-cow
61.864=cue
27.838-124
13.4M
145.2
FT-Transformer
=0.487 12.382=co
7.446:00
2.258=0ox
27.547
20.084=0m
53.310=om
25.958a0as
9.3M
284.7
DANet-24
73.708+0z2
13.338
9.301 =ooM
2.171=0oc
49.643+10x
24.763-co
53.033
26.431 =am
5.5M
54.9
FCNN w/ mixup
203
9.572a00
2.083.g00
36.742ame
56.0052000
56,
78.7877 201
27.467
0.7M
594.3
FCNN w/ lasso
87.005 1017 41.071 Lor
4,141 31.8
44.881
69.302
132.102
32.282
0.7M
568.8
Vector CapsNet
g4.1 52.635_gm
53.587
161,547
58.516200s
51.391 =oo.
62.654
54,252gnm
0.4M
318.5
TABCAPS (Ours)
62.054 12.043 8.130 nor
2.013.
34.047 1uoz 14.301
53.776_cm
25.821
0.2M
501.1
Table 2: Extreme generalization performances. The best and second best performances of deep learning approaches are respectively marked in bold and underlined. Note that the reported log-loss values (the lower the better) are with a 100x factor. The Epsilon dataset not included due to its extremely high computation complexity in conducting t-SNE projection.
Method
Click
Diabetes
EEG
Gas
Heart
Hill
Higgs
Training-Test Test Split
XGboost
66.070=00z
65.886+am
70.654+aot
31. 31.504+am4
35.650=oor
69.657=am
54.557=004
Catboost
63.925 2004
68.819ium
68.799.mm
18.8641a0
35.207=mm
69.1622.mm
54.632-mm
TabNet
115.907
225.22.ius
79.666.um
158.618
44.967
14.114
55.763
Net-DNF
67.625 1002
58.792,1us
68.261 1004
15.
55.371
48.301
20m
55.738
NODE
2308
67.021 2004
68.357
2004
46.541
=0.0
69.771
61.870=om
Frannsssformer
78.431 +0.12
59.283 ano4
68.278
2007
6.416
26.132
0.05
66.972
53,970
DANet-24
74.401 1o0.
59.7362a0s
69.021 2000
10.395
49.643_mm
37.976-00
54.182-0..
FCNN mixup
66.052+aus
60.262
68.55 08.8
25.102+am
35.674-
62.126=
55.847
FCNN lasso
106.123
67.082++00
03.170
61.310+aaz
76.854-o.0
75.853+om
106.580=0om
Vector CapsNet
64.724.00s
66.009amm
67. 67.845.um 2000
163. (63.1932ssm)
60.848
644.73
62.791 =mm
TABCAPS (Ours)
63.355
58,409:a02
67.471 +0.00 +0.0
8.750
34.503
17.887 =004
54.707=am
and make the train-test split based on the t-SNE embedding (see the 2nd row in Table We use the records inside the black rectangles for training (containing ~70% data) and the rest for test. Notably, the rectangles are biasedly placed and the records for training and test do not overlap in the t-SNE embedding, so that they are clearly in different distributions. The performances reported in Table 2 show that our TABCAPS obtains best or second best performances among deep learning approaches on most of datasets. Such results imply that our method can learn robust features and generalize well. Besides, our TABCAPS exhibits considerably better generalization performances than the Vector CapsNet with very clear margins (e.g., TABCAPS obtains 8.750 x 10-2 on Gas dataset, while the Vector CapsNet obtains 163.193 x 10-2).
4.4 PERFORMANCE STABILITY DURING TRAINING
Many previous works required the early stop Yao et al.) 2007) to avoid over-fitting because they managed to exploit the potential of each tabular feature, and thus might be impacted by the uninfor- mative features. However, our TABCAPS does not seem to require such training trick since it learns the data-level semantics directly. To show this, we compare the real-time validation performances (on log-loss values) during traning, among several regularization strategies (e.g., FCNN with lasso, mixup (Zhang et al. 2018)), commonly used neural networks (c.g.. TabNet with sparse regularization, NODE and Net-DNF with ensemble learning strategies), and traditional boosting methods (XGboost and CatBoost), on the Click dataset. As shown in Fig. S most of these methods obtain their lowest log-loss values the validation set after few training steps. and then their models gradually collapse as the training step increases. Notably, during the training process, all the corresponding loss values on training data keeps on decreasing slightly fluctuating. One can that the lasso and mixup
7

---

Published as a conference paper at ICLR 2023
losslss Values
Table 3: Ablation study results. For the miss- ing value imputation (MVI) test, report the performances of a XGBoost model trained on training sets with filled values. The log-loss
Tab we decoder
TabCaps. w/ decoder
1.8
TabNet
NODE
Real- Real-Time Siml-TTmm Validation Log-loo
1.6
Nat-DNF
FCNN
values are 100x.
FCNN w mixup
FCNN w/ lesso
Method
Performances
XGBoost
Routing & projection
Classification
fps
CalBoost
1.2
Routing spanse projection
12,043-0000 04.50300000 13.16620.03
(21
dynamic routing spase projection
(3)
RoW Rouning simple projection
1683
Routing
Decoder
MVI (log loss (100x))
(3) HoW Routing
Ours
22.531=0.01
0.8
(4)
BoW Rouning
fully connection
26,338=0.07
(5)
Encoder classifier
25
Capanla layess
Classification
0.6
50
100
(6)
(7)
12.043-0.01
6.254
#. Training Steps
11,138-0000
Others
Chasification
Figure 3: Real-time validation performances (log- loss, the lower the better) on Click dataset. "step" (on the ær-aiis) indicates an "epoch" for neural networks and a "boosting step" for XGboost and CatBoost.
(9)
Tahular Festure Extraction
12.043-0.08
(10)
wio Tabular Feature Extraction
12.684-0.022
(1)
S.C. envemblle
12.043.0.00
(12)
single S.C.
12.515-0.05
regularization can alleviate this issues to some extent, but the validation log-loss values still increase slowly. Different from these methods, our TABCAPS with and without the decoder performs evidently stable, stabilizing the log-loss values at a low level (around 0.62). This finding reveals an advantage of using vectorial features as operating units and suggests that our method can facilitate the users to obtain a well-performed model without intensely monitoring the performances on the validation sets or agonizing over the use of training tricks (e.g., regularization methods or early stop).
4.5 EFFECTS OF BOW ROUTING
Here further inspect how our proposed BoW Routing determines the transformation paths for the vectorial features from primary capsules to senior capsules. We compute the averaged transformation weights (wyje in ())6) over the whole test set and visualize them in Fig. 4 One can that senior capsule features are yielded by merging limited vectorial feature votes from primary capsules. Besides, we found the weights are sparse and the values are similar to different samples (with the standard deviation values varying from 0.001 to 0.003), which suggests that primary capsules model some semantically meaningful data patterns and our BoW Routing succeeds in finding out clear decision paths for classification.
4.6 ABLATION STUDIES
We conduct ablation studies to examine the effects of the routing algorithms and the decoder (in missing value imputation test) on the Diabetes dataset. We compare our Bo W Routing with the
dynamic routing (Sabour et al. 2017) on our architecture. See rows (1), (2), and (3) in Table 3 It is obvious that TABCAPS with our proposed BoW Routing outperforms that with the dynamic routing (Sabour et al. 2017) on the Diabetes data classification task by around 2.5 x 10-2 on log-loss. We also compute the fps of the BoW Routing and dynamic routing, and find that BoW Routing is nearly 25% faster than the dynamic routing, suggesting the superiority of our straightforwardness design. Compared with the simple projection (without entmax), our sparse projection is witnessed to be beneficial and is compatible with the BoW Routing, which shows that the use of entmax benefits in finding informative features and excludes the useless ones.
To inspect the capability of our proposed decoder, we conduct test on missing value imputation, and employ an XGboost classifier to evaluate the effectiveness. We randomly mask out 20% of the feature values of the Diabetes training data, and simply fill the missing values with the means of feature values within the same classes. We train three models on the training set: (i) our TABCAPS with the proposed decoder (using the loss L in Eq. (8) and MSE), (ii) our TABCAPS with the fully- connection- based decoder of the Vector CapsNet (Sabour et al. 2017) (using L in Eq. K and MSE), and (iii) an auto-encoder with an auxiliary classifier that takes the hidden states as input (using MSE and cross-entropy loss). After training, we take the reconstructed feature values provided by these 3 decoders to refill the missing values respectively, and obtain different training sets. To
8

---

Published as a conference paper at ICLR 2023
Senlor Caps.
3.00
Semoor Caps.
010
Senior
0.4
0.12
Caps.
Caps. Diabetes
Primary
Click
EEG
Sentor Caps.
Senier Caps. ps.
-0395
035
0.0
-0015
005
Primary Caps.
Caps. Heart
Primary Caps.
Hill
Figure 4:
The averaged transformation weights determined by Bo W Routing. The x-axis
represents indices of primary capsules while the y-axis for senior capsules. The counts of primary capsules and senior capsules different datasets determined by the hyperparameter tuning.
fairly compare the qualities of the reconstructed values, train 3 XGboost models with identical hyper-parameters on these training sets, and report their performances on the original test set. A better performance indicates that the reconstructed values are more trusty.
As reported in Table 3 XGboost that is trained on the training set provided by our decoder is markedly superior to the other two methods, while the sully-coonnetion-baaeee decoder (Sabour et al. 2017) performs the worst. Notably, we train another XGboost on the training set in which the masked values remain missing, and find that this XGboost version still obtains 26.174 on the set. Thus, one can see that our proposed decoder is able to yield significant performance gain in filling the missing values, while the sully-comnection-bee based decoder predicts noisy values.
The effects of capsule layers were tested. See rows (6), (7), and (8). The 3- 3-layer and 4-layer TABCAPS obtain better performances on the Diabetes dataset compared with the default 2-layer model. On the Heart dataset, similar conclusion is attained that TABCAPS with 2, 3, 4 capsule layers obtained log-loss values of 0.34047, 0.33992, and 0.33673, respectively. In general, we consider TABCAPS with different capsule layers perform competitive, and 2-capsule-layer model is sufficient to attain good performance. We also inspected the effects of the "tabular feature extraction" " module at the bottom of TABCAPS (see Fig. [ and the effects of the ensemble of the senior capsules. Comparing row (9) with (10), and row (11) with (12), it evident that both of the tabular feature extraction and the core reasons of the superior performances of TABCAPS. We consider it is the novel architecture makes TABCAPS perform well.
the senior capsule ensemble are beneficial, but
5 CONCLUSIONS AND FUTURE WORK
This paper proposed novel and simple paradigm for supervised tabular learning, which encapsulated all features of a record into vectorial features as operating units and thus avoids interactions among heterogeneous tabular features. Consequently, our proposed new CapsNet (called TABCAPS), with new primary capsule synthesis approach, a novel sparse projection, and a novel BoW Routing, coped with vectorial features and obtained superior performances tabular data classification tasks at low computational costs. Specifically, the proposed BoW Routing performs straightforwardly without iterative clustering processes of previous CapsNets. This work is beneficial attempt in the research areas of capsule neural networks, routing algorithms, and supervised tabular learning. Following previous works, our proposed method only considered classification tasks, and it would be interesting to explore how to use TABCAPS for tabular data regression. Moreover, it is also interesting to design other approach for directly learning data-level semantics without feature- wise interactions.
Acknowledgements. This research was partially supported by the National Key R&D Program of China under grant No. 2018AAA0102102 and National Natural Science Foundation of China under grants No. 62132017.
Reproducibility Statement. To ensure the reproducibility, we include the settings of hyper- parameter tuning for various methods in Appendix A.I The classification performances measured by "acceurcy" and "ROC-AUC" are reported in Appendix A.2 reference. Besides, an additional discussion between our proposed "multivariate Gaussian embedding" and "linear feature embedding" approaches are given in Appendix A.3

---

Published as a conference paper at ICLR 2023
REFERENCES
Ami Abutbul, Gal Elidan, et al. Net-DNF: A neural architecture for tabular data. In ICLR, 2021.
Peter Martey Addo, Dominique Guegan, et al. Credit risk analysis using machine and deep learning models. Risks, 2018.
Karim Ahmed and Lorenzo Torresani. STAR-Caps: Capsule networks with straight-through attentive routing. In NeurIPS, 2019.
Sercan O Arik and Tomas Pfister. TabNet: Attentive interpretable tabular learning. In AAAI, 2020.
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high-energy physics with deep learning. Nature Communications, 2014.
Leo Breiman, Jerome H Friedman, et al. Classification and Regression Trees. CRC press, 1984.
PASCAL Challenge. Epsilon Dataset, 2008, https://www. cswe. ntu wtt. wttp//.dd...////./lnn
Jintai Chen, Hongyun Yu, et al. A receptor skeleton for capsule neural networks. In JCML, 2021.
Jintai Chen, Kuanlun Liao, et al. DANets: Deep abstract networks for tabular data classification and regression. In AAAI, 2022.
Tianqi Chen and Carlos Guestrin. XGBoost: A scalable tree boosting system. In KDD, 2016.
Davide Chicco and Giuseppe Jurman. Machine learning can predict survival of patients with heart failure from serum creatinine and ejection fraction alone. BMC Medical Informatics and Decision Making, 2020.
Jaewoong Choi, Hyun Seo, et al. Attention routing between capsules. CVPR Workshops, 2019.
KDD Cup. Predict the click-through rate of ads given the query and user information, 2012.
https://wwwt
http///wwwgglee.... kaggle www. http://c//kdddu2221112raa22222/
Dheeru Dua and Casey Graff. EEG eye state dataset, 2015.
orrg///
1471
Yury Gorishniy, Ivan Rubachev, et al. Revisiting deep learning models for tabular data. NeurIPS, 2021.
Leo Grinsztajn, Edouard Oyallon, and Gael Varoquaux. Why do tree-based models still outperform deep learning typical tabular data? In ICLRR, 2022.
Huifeng Guo, Ruiming Tang, al. DeepFM: factorization-machine based neural network for CTR prediction. In IJCAI, 2017.
Md Rafiul Hassan, Sadiq Al-Insaif, et al. A machine learning approach for prediction of pregnancy outcome following IVF treatment. Neural Computing and Applications, 2020.
Geoffrey Hinton, Sara Sabour, et al. Matrix capsules with EM routing. ICLR, 2018.
Guolin Ke, Qi Meng, et al. LightGBM: A highly efficient gradient boosting decision tree. In NeurIPS, 2017.
AR Kosiorek, S Sabour, al. Stacked capsule autoencoders. In NeurIPS, 2019.
Graham Lee and Oppacher Franz. Hill-valley data set, 2008. Iatt:://archive. uci
Chao Li, Zhiyuan Liu, Mengmeng Wu, et al. Multi-interest network with dynamic routing for recommendation at Tmall. In CIKM, 2019.
Jerry Ma and Denis Yarats, Quasi-hyperbolic momentum and Adam for deep learning. ICLR, 2019.
10

---

Published as a conference paper at ICLR 2023
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted Boltzmann machines, In ICML, 2010.
Andrew Ng. Feature selection, LI vs, L2 regularization, and rotational invariance. ICML, 2004.
Ben Peters, Vlad Niculae, et al. Sparse sequence-to-seqeenee models. In ACI,, 2019.
Gibran Fuentes Pineda, Hisashi Koga, al. Scalable object discovery: hash- hash-based approach to clustering co-occurring visual words. IEICE' Transactions on Information and Systems, 2011.
Sergei Popov, Stanislav Morozov, et al. Neural oblivious decision ensembles for deep learning on tabular data. In ICLR, 2019.
Liudmila Prokhorenkova, Gleb Gusev, et al. CatBoost: Unbiased boosting with categorical features. In NeurlPS, 2018.
J Ross Quinlan. Discovering rules by induction from large collections of examples. Expert Systems in the Micro Electronics Age, 1979.
J Ross Quinlan. C4.5: Programs for Machine Learning. Elsevier, 2014.
Jathushan Rajasegaran, Vinoj Jayasundara, al. DeepCaps: Going deeper with capsule networks. In CVPR, 2019.
DG Reina, SL Toral, et al. Improving discovery phase of reactive ad hoc routing protocols using Jaccard distance. The Journal of Supercomputing, 2014.
Fabio De Sousa Ribeiro, Georgios Leontidis, and Stefanos D Kollias. Introducing routing uncertainty in capsule networks. NewrlPS, 2020.
Fabio De Sousa Ribeiro, Kevin Duarte, Miles Everett, Georgios Leontidis, and Mubarak Shah. Learning with capsules: survey, arXiv preprint rrXiv:2206.02664 2022.
Sara Sabour, Nicholas Frosst, al. Dynamic routing between capsules. In NeurlPS, 2017.
Sara Sabour, Andrea Tagliasacchi, et al. Unsupervised part representation by flow capsules. In ICML, 2021.
Gerard Salton and Michael E Lesk. The SMART automatic document retrieval systems-An illustra- tion. Communications of the ACM, 1965.
Yao-Hung Hubert Tsai, Nitish Srivastava, et al. Capsules with inverted dot-product attention routing. In ICLR, 2019.
Vikas Ukani. Diabetes data set, 2020.
httpp::/www.kagg/ htt.:p/gp.....oomoo.ssssseeeee
diabetes-data-sett
Alexander Vergara, Shankar Vembu, et al. Chemical gas sensor drift compensation using classifier ensembles. Sensors and Actuators B: Chemical, 2012.
Ruoxi Wang, Bin Fu, et al. Deep & cross network for ad click predictions. In ADKDD, 2017.
Jiahuan Yan, Jintai Chen, Yixuan Wu, Danny Z Chen, and Jian Wu. T2G-Former: Organizing tabular features into relation graphs promotes heterogeneous feature interaction. In AAAI, 2023.
Yuan Yao, Lorenzo Rosasco, et al. On early stopping in gradient descent learning. Constructive Approximattin,, 2007.
Hongyi Zhang, Moustapha Cisse, et al. Mixup: Beyond empirical risk minimization. In ICLR, 2018.
Wei Zhao, Haiyun Peng, et al. Towards scalable and reliable capsule networks for challenging NLP applications. In ACL, 2019a.
Yongheng Zhao, Tolga Birdal, et al. 3D point capsule networks. In CVPR, 2019b.
Yongheng Zhao, Tolga Birdal, et al. Quaternion equivariant capsule networks for 3D point clouds. In ECCV, 2020.
11

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 HYPERPARAMETER TUNING SETTINGS
As mentioned in the main paper. some hyperparameters of methods were turned for datasets by grid search or the HyperOpt library with 50 steps of the Tree-structured Parzen Estimator (TPE), similar to the configurations in (Popov et 2019). The details are listed in the subsections below. As for XGBoost, CatBoost, FCNN, TabNet, DANets, Vector CapsNet, and our TABCAPS, we used the HyperOpt hyperparameter search library; for NODE and Net-DNF, we used grid search (following the settings in their papers). For FT-Transformer (Gorishniy et al. 2021), used their official open source.
The hyperparameter search spaces and search algorithms of XGBoost (Chen & Guestrin) 2016). CatBoost (Prokhorenkova et al., 2018), NODE (Popov et al.) 2019), and FCNN (Nair & Hinton 20T0) set identical in (Popov et al. 2019). We set the maximal number of the trees in XGBoost and CatBoost up to 4096 (instead of 2048 used in NODE (Popov et al. 2019), since we found that 2048 is not enough for XGBoost and CatBoost obtain good performances. We also set the maximal layer number in FCNN up to 9. Besides, the hyperparameter search settings Net-DNF (Abutbul) al. (021) following its original paper. The architectures FCNN with lasso regularization or mixup were constructed following the FCNN in (Popov et al. 2019). For both of DANet-24 (Chen al. 2022) and TabNet (Arik& Pfister (2020), we set the hyperparameter search spaces to contain all the possible values provided by the main papers.
A.1.1 XGBooST
For easy re-implementation, the hyperparameter variable names followed the sklearn package settings.
eta (learning rate): Log-Uniform distribution [e- 1]:
depth: Discrete uniform distribution [2, 10];
subsample: Uniform distribution [0.5, 1];
colsample bytree: Uniform distribution [0.5, 1];
colsample.bylevel: Uniform distribution [0.5, 1];
min.child.weightt Uniform distribution [e --166ee.
alpha: Uniform choice {0, Log-Uniform distribution [e-16, 16 c²1]:
lambda: Uniform choice (0, Log-Uniform distribution 16 e²1]:
gamma: Uniform choice {0, Log-Unite Log-Uniform distribution [e [e-16, e..
A.1.2 CATBOOST
For easy re-implementation, the hyperparameter variable names followed the sklearn package settings.
learning_rate Log-Uniform distribution [e-3, 1]:
random_strengee Discrete uniform distribution [1, 20];
one haeehoxmaxxxae Discrete uniform distribution [0, 25];
reg:: Log-Uniform distribution [1, 10];
bagging temperature: Uniform distribution [0, 1]:
regfeestination.ieeattioss Discrete uniform distribution [1, 10).
https://scikitt-eeem....
12

---

Published as a conference paper at ICLR 2023
A.1.3 NODE
number of layers: {2. 4, 8};
total number of trees: (1024, 2048);
depth: {6. 8):
output dimension of trees: {2, 3);
learning rale: 10-
A.1.4 NET-DNF
Following the original setting in (Abutbul et al. 2021). ) the learning rate is initialized as 0.05 and reduced by monitoring the training loss (check its official codes for details).
number of formulas: {64, 128, 256, 512, 1024, 2048, 3072];
feature selection beta: {1.6, 1.3, 1.0, 0.7, 0.4, 0.1}.
A.1.5 TABNET
sparse: (0.0, 10-6, 10-4, 10-3, 10-2,0220000 0.1):
7: {1.0, 1.2, 1,5, 2.0};
batch size B: {256, 512, 1024, 2048, 4096, 8192, 16384, 32768);
Na and Na: {8, 16, 24, 32, 64, 128}:
Nsteps: {3, 5, 6, 7, 8, 9, 10);
By: (256, 512, 1024, 2048, 4096);
mg: {0.6, 0.7, 0.8, 0.9, 0.95, 0.98):
learning rate: {0.002, 0.01, 0.02, 0.025).
A.1.6 FCNN
number of layers: Discrete uniform distribution [2, 9];
output dimensions of fullcommections::: Discrete uniform distribution (128, 256, 512, 1024):
learning rate: Uniform distribution
100=1;
dropout rate: Uniform distribution [O, 0.5];
lasso coefficient (optional): Discrete uniform distribution uniform [10-7, 10-4)
learning rate: Uniform distribution [10-4, 0.01].
A.1.7
FT-TRANSFORMER
the number of layers: Discrete uniform distribution [1, 4];
feature embedding size: Discrete uniform distribution [64, 512];
residual dropout: Uniform distribution [0, 0.2];
attention dropout: Uniform distribution [0, 0.5];
FFN dropout: Uniform distribution [0, 0.5];
FFN factor: Uniform distribution [3. $1
learning rate: Log- -niffrn distribution [10-5, 10-3];
weight decay: Log- Uniform distribution [10 6, 10-30 3];
the number of iterations: 100.
13

---

Published as a conference paper at ICLR 2023
A.1.8 DANET-24
The learning rate of DANet-24 is initially 0.008 and is decayed by 5% in every 20 epochs.
Ko: {5, 8):
do: {32, 48);
d1 {64, 96).
A.1.9 VECTOR CAPSNET
learning rate: Uniform distribution [0.02, 0.1]:
primary capsule size: Discrete uniform distribution [64, 128];
the number of primary capsules: Discrete uniform distribution [4, 32];
senior capsule size: Discrete uniform distribution [4, 32].
A.1.10 TABCAPS
learning rate: Uniform distribution [0.02, 0.1];
the number of senior capsules for class: Discrete uniform distribution [1, 5];
primary capsule size: Discrete uniform distribution [64, 128];
the number of primary capsules: Discrete uniform distribution [4, 32];
senior capsule size: Discrete uniform distribution [4, 32];
the number of learnable "words": Discrete uniform distribution [16, 64).
14

---

Published as a conference paper at ICLR 2023
A.2 CLASSIFICATION PERFORMANCES ON "ACCURACY" AND "ROC-AUC"
Here we report the model performances measured by the "accuracy" in Table and the binary classification performances measured by "ROC-AUC" in Table These classification performances are obtained in 5 runs with regard to random seeds. One can see that our TABCAPS obtains better or competitive performances in comparison with previous works.
Table 4: Classification Performances measured by "accuracy".
Method
Click Diabetes
EEG
Gas
Heart
Hill
Hig
Epsilon
XGboost
0.6666
0.9575
0.9342 0.9957 0.8833 0.5761 0.7262 0.8884
Catboost
0.6599
0.9250
0.9322 0.9935 0.8501 0.4856 0.7256
0.8878
TabNet
0.6684
0.9725
0.8815
0.9949 0.8166 0.9012 0.7190 0.8890
Net-DNF
0.6694
0.9775
0.9379 0.9942 0.8333 0.7819 0.7268
0.8922
NODE
0.6688
0.7903
0.7871
0.8606 0.8667 0.6214 0.7260 0.8952
FT-Transformer
0.6606
0.9725
0.9826
0.9953 0.9166 0.8971 0.7290 0.8976
DANet-24
0.6698
0.9687
0.9776 0.9949 0.8333 0.9095 0.7284 0.8962
FCNN mixup
0.6674
0.9375
0.9746 0.9956 0.7333 0.9136 0.7075
0.8934
FCNN lasso
0.6670
0.9403
0.9639
0.9863 0.6677 0.8949 0.6677 0.8958
Vector CapsNet
0.6678
0.9025
0.9726
0.9939 0.6833 0.6419 0.7102 0.8905
TABCAPS (Ours)
0.6685
0.9725
0.9776 0.9968 0.9023 0.9012 0.7300
0.8975
Table Binary classification Performances measured by "ROC-AUC". The Gas dataset is not included because it is not for binary classification.
Method
Click Diabetes
EEG
Heart
Hill
Hig
Epsilon
XGboost
0.7216
0.9859
0.9884 0.9447 0.5939 0.8054 0.9542
Catboost
0.7199
0.9878
0.9859 0.9330 0.4965 0.8048 0.9524
TabNet
0.7122
0.9853
0.9535
0.9095 0.9632 0.7911 0.9562
Net-DNF
0.7169
0.9932
0.9838 0.9412 0.8707 0.8018 0.9532
NODE
0.7146
0.8411
0.8173 0.9506 0.6626 0.7444 0.9497
FT-Transformer
0.7180
0.9883
0.9978 0.9764 0.9833 0.8078 0.9586
DANet-24
0.7189
0.9825
0.9981 0.8931 0.9710 0.8042 0.9582
FCNN mixup
0.7178
0.9653
0.9962 0.8978 0.9531 0.7844 0.9598
FCNN lasso
0.7128
0.9702
0.9958 0.8343 0,5809 0.7247 0.9323
Vector CapsNet
0.7168
0.9902
0.9965
0.9036 0.5724 0.7754 0.9545
TABCAPS (Ours)
0.7199
0.9907
0.9970 0.9436 0.9911 0.8081 0.9606
A.3 PERFORMANCES WITH OTHER EMBEDDING APPROACHIES
In this paper, we devise multivariate Gaussian kernels to synthesize vectorial features. Here we com- pare our proposed multivariate Gaussian embedding approach with the commonly used linear feature embedding approach (like the embedding approach in FTT-rranfformer). On the Diabetes dataset, TABCAPS with the linear embedding layer achieves 03122 on log-loss (4) while TABCAPS with multivariate Gaussian kernels achieves 0. 1044. The comparison results suggest that our multivariate Gaussian kernels is beneficial, which might be due to the non-line non-lincarity of the multivariate Gaussian embedding approach.
15


Reviews:
[{"review_id": "pOlG2SM9QE", "reviewer": "Reviewer_NQgm", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a bespoke CapsNet model called TabCaps that is tailored to tabular data processing. This seems to be the first effort to develop a bottom-up approach specifically for tabular data whereby all features are considered as a vector (per the properties of capsules) which are then transformed by a multivariate Gaussian kernel with some learnable parameters so that they can be adapted to the dataset in question. The higher-level capsules (called senior capsules) are tasked to learn the semantics of the target classes. This is achieved through a feature transformation by a sparse weight matrix into feature votes which are then fed into a newly proposed \"differentiable\" Bag of Words. The vector lengths of the senior capsules represent the existence probabilities of a specific semantic, whereby each class is represented by a number of senior capsules, akin to ensembles (authors use more than one senior capsule hence the ensemble, as opposed to previous work). The results are quite significant and impressive in that they are outperforming the defacto approaches that are based on XGBoost across several benchmark datasets.\n\n", "summary_of_the_review": "I personally believe that CapsNets are powerful learners that have also recently been shown to relate to the attention mechanism popularised by transformers (https://arxiv.org/pdf/2206.02664.pdf). Conceptually they are better grounded and more explainable. This paper contributes by presenting an alternative view of capsnets this time on tabular data that as a researcher myself would like to see published to support further fundamental research on capsnets. ", "strength_and_weaknesses": "Strengths: \nThere is a good empirical work presented in this paper.\na) It makes a decent step forward on capsnet research b) The experimental setup and datasets considered are adequate c) the method appears well-grounded and sound and the authors have provided an appendix with the hyperparameters.\n\nWeaknesses: a) I think the paper is positioned with respect to the original methods of Sabour and Hinton (dynamic routing and EM respectively) - that is not correct in my opinion as most of the works that followed up from these have gone on improving pretty much every facet of those algorithms, in terms of performance, number of parameters, iterative vs non-iterative, etc. some of these works have already been cited some others not, but the points made in the paper would be stronger had the authors focused on presenting the improvements with respect to the current state of the art (not in terms of results). b) I think some reflection on how the approach scales would have been helpful - for instance, traditional capsnet suffer from the collapse of learning making it hard to train, say on imagenet, or stuck several capsule layers.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dUyJWXGqoH", "reviewer": "Reviewer_NLd7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a capsule network method for tabular data classification. To tailor the capsule network for this task, the paper introduces several configurations with the focus on so-called BoW routing. ", "summary_of_the_review": "The paper proposes a capsule network for tabular data, some adaptations of which seem a bit ad-hoc. The performance improvement seems a bit marginal.", "strength_and_weaknesses": "Strength:\n\n1. I'm not an expert with capsule networks, but after some search, I didn't find the applications of capsule networks for tabular data. This paper might be the first one of applying it in this domain.\n\n2. The paper provides a relatively comprehensive experiments with several advanced methods for tabular data.\n\nWeaknesses:\n\n1. The main contribution of the paper is making capsule networks work for tabular data by introducing a few adaptations and modifications, such as sparse projection and BOW routing. However, many of the proposed adaptations are lack of clear motivations or intuitions. For example, why BOW routing works particularly with tabular data? The paper shows how but misses why, which make some of the adaptations more like heuristics instead of principled approaches.\n\n2. As the paper introduces several adaptations, although with the ablation study, I feel that more ablation study is needed to verify the performance of different combinations of the adaptations.\n\n3. The performance improvement of the proposed method seems to be marginal in many cases, shown in Table 1, given the error bars, not mentioning that improvement of classification accuracy is hardly observed in Table 4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "atslOOGpga", "reviewer": "Reviewer_Zv1Z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed TabCaps, to process the multiple features into one vectorized unit, avoiding the interaction among tabular features.  In TabCaps, a tabular instance is respectively encoded into several vectorial features by some optimizable multivariate Gaussian kernels in the primary capsule layer, where each vectorial feature represents a specific \"profile\" of the input instance and is transformed into senior capsule layer under the guidance of a novel straightforward routing algorithm. Rather than clustering, an efficient bag-of-words is used in routing.\n", "summary_of_the_review": "Overall, I lean to accept the paper. Although the experiment result is not very significant, I think this is a good paper, since it may inspire future research in neural networks for tabular data. ", "strength_and_weaknesses": "Strengths\n\n- The paper is well written. \n- The experiment setting is comprehensive.\n- The idea is interesting, and the experiment results demonstrate looks good.\n\n\nWeaknesses  & Questions\n\n1. Current method seems cannot apply to regression tasks.\n2. The experiment result seems not very significant, especially the accuracy comparison from Table 4.\n3. Can you explicitly show #classes of different tasks in the experiments? And I would like to see the AUC metric for the binary classification, rather than log-loss. Besides, how do you use XGBoost for multi-class (K > 2) tasks?\n4. Can we replace the \"Multivariate Gaussian Kernels\" with FFN? If it can, how about the performance? if not, why?\n5. In the widely used boosting tree based methods, there are many interactions among tabular features. It surprised me that non-feature-interaction is better in neural networks. Can authors explain more about that?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QeZvWbu1Vn", "reviewer": "Reviewer_sBG9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to encode the tabular features into vectorial space. To this end, the paper devises a capsule network to learn from the vectorial features by Gaussian kernels and routing method. Experiments on six real-world tabular datasets are conducted to validate the effectiveness of the proposed method named TabCaps. The results suggest that TabCaps outperforms or is competitive with the previous deep learning approaches.", "summary_of_the_review": "This work proposes a capsule-based framework to learn from tabular data. The idea is interesting, but it is unclear why the method can perform well and what the advantages are over the existing deep learning methods. The experiments are not convincing as well since there are only six datasets.", "strength_and_weaknesses": "Strengths:\n\n1. The idea of using capsules to encapsulate all tabular features of an instance into vectorial features is interesting. In this way, we do not have to deal with individual features so we may integrate heterogeneous features in learning.\n2. TabCaps achieves good performance with a small number of parameters in the six datasets.\n3. TabCaps also shows strong generalization performance.\n\nWeaknesses:\n\n1. It is counter-intuitive to me why deep learning methods have not encapsulated all tabular features. With several layers of networks and by embedding them into the hidden representations, deep nets could also learn how to integrate the information of individual features. That being said, it is unclear why TabCaps can achieve better performance.\n2. Only six tabular datasets are considered. Experiments on more datasets will enhance the results.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster