Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed TabCaps, to process the multiple features into one vectorized unit, avoiding the interaction among tabular features.  In TabCaps, a tabular instance is respectively encoded into several vectorial features by some optimizable multivariate Gaussian kernels in the primary capsule layer, where each vectorial feature represents a specific "profile" of the input instance and is transformed into senior capsule layer under the guidance of a novel straightforward routing algorithm. Rather than clustering, an efficient bag-of-words is used in routing.

strength_and_weaknesses: Strengths

- The paper is well written. 
- The experiment setting is comprehensive.
- The idea is interesting, and the experiment results demonstrate looks good.


Weaknesses  & Questions

1. Current method seems cannot apply to regression tasks.
2. The experiment result seems not very significant, especially the accuracy comparison from Table 4.
3. Can you explicitly show #classes of different tasks in the experiments? And I would like to see the AUC metric for the binary classification, rather than log-loss. Besides, how do you use XGBoost for multi-class (K > 2) tasks?
4. Can we replace the "Multivariate Gaussian Kernels" with FFN? If it can, how about the performance? if not, why?
5. In the widely used boosting tree based methods, there are many interactions among tabular features. It surprised me that non-feature-interaction is better in neural networks. Can authors explain more about that?
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is well-written and easy to follow.

Quality: The proposed method technologically sounds good, and the experiment design is solid.

Novelty: I think it is the first work to use Capsule Neural Network in tabular data.

Reproducibility: The details in the appendix help to reproduce, but it would be better if the authors can release the source code.
summary_of_the_review: Overall, I lean to accept the paper. Although the experiment result is not very significant, I think this is a good paper, since it may inspire future research in neural networks for tabular data.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a bespoke CapsNet model called TabCaps that is tailored to tabular data processing. This seems to be the first effort to develop a bottom-up approach specifically for tabular data whereby all features are considered as a vector (per the properties of capsules) which are then transformed by a multivariate Gaussian kernel with some learnable parameters so that they can be adapted to the dataset in question. The higher-level capsules (called senior capsules) are tasked to learn the semantics of the target classes. This is achieved through a feature transformation by a sparse weight matrix into feature votes which are then fed into a newly proposed "differentiable" Bag of Words. The vector lengths of the senior capsules represent the existence probabilities of a specific semantic, whereby each class is represented by a number of senior capsules, akin to ensembles (authors use more than one senior capsule hence the ensemble, as opposed to previous work). The results are quite significant and impressive in that they are outperforming the defacto approaches that are based on XGBoost across several benchmark datasets.


strength_and_weaknesses: Strengths: 
There is a good empirical work presented in this paper.
a) It makes a decent step forward on capsnet research b) The experimental setup and datasets considered are adequate c) the method appears well-grounded and sound and the authors have provided an appendix with the hyperparameters.

Weaknesses: a) I think the paper is positioned with respect to the original methods of Sabour and Hinton (dynamic routing and EM respectively) - that is not correct in my opinion as most of the works that followed up from these have gone on improving pretty much every facet of those algorithms, in terms of performance, number of parameters, iterative vs non-iterative, etc. some of these works have already been cited some others not, but the points made in the paper would be stronger had the authors focused on presenting the improvements with respect to the current state of the art (not in terms of results). b) I think some reflection on how the approach scales would have been helpful - for instance, traditional capsnet suffer from the collapse of learning making it hard to train, say on imagenet, or stuck several capsule layers.


clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow but assumes some prior knowledge of capsnets. There is some quality work presented here and the appendix helps to get an idea of hyperparameters used - releasing the code post-acceptance would be helpful so that people can reproduce it more easily.

A major limitation is the size of the dataset - It would have been nice to compare (or reflect upon) performance on larger datasets, which is an area CapsNet have been suffering from in general. Some additional visualisations on how the learning behaves from one layer to another would have been helpful - I know first-hand that this is not as straightforward as it is with CNNs, but it would be nice to show what the capsules learn, especially during routing/voting. It is more intuitive for images, but how does this actually work with tabular data?


summary_of_the_review: I personally believe that CapsNets are powerful learners that have also recently been shown to relate to the attention mechanism popularised by transformers (https://arxiv.org/pdf/2206.02664.pdf). Conceptually they are better grounded and more explainable. This paper contributes by presenting an alternative view of capsnets this time on tabular data that as a researcher myself would like to see published to support further fundamental research on capsnets.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a capsule network method for tabular data classification. To tailor the capsule network for this task, the paper introduces several configurations with the focus on so-called BoW routing. 
strength_and_weaknesses: Strength:

1. I'm not an expert with capsule networks, but after some search, I didn't find the applications of capsule networks for tabular data. This paper might be the first one of applying it in this domain.

2. The paper provides a relatively comprehensive experiments with several advanced methods for tabular data.

Weaknesses:

1. The main contribution of the paper is making capsule networks work for tabular data by introducing a few adaptations and modifications, such as sparse projection and BOW routing. However, many of the proposed adaptations are lack of clear motivations or intuitions. For example, why BOW routing works particularly with tabular data? The paper shows how but misses why, which make some of the adaptations more like heuristics instead of principled approaches.

2. As the paper introduces several adaptations, although with the ablation study, I feel that more ablation study is needed to verify the performance of different combinations of the adaptations.

3. The performance improvement of the proposed method seems to be marginal in many cases, shown in Table 1, given the error bars, not mentioning that improvement of classification accuracy is hardly observed in Table 4.
clarity,_quality,_novelty_and_reproducibility: Clarity:

1. As discussed before, more motivation, intuition, or analysis are needed for the proposed method.

2. Introduction to the background of capsule networks is needed.

Quality:

1. It seems that different recent methods for tabular data use different datasets and evaluation metrics (e.g. log-loss, RMSE, accuracy). I was wondering how these settings are selected in this paper. Due to this, I feel it's hard to cross compare different methods in unified settings.
summary_of_the_review: The paper proposes a capsule network for tabular data, some adaptations of which seem a bit ad-hoc. The performance improvement seems a bit marginal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method to encode the tabular features into vectorial space. To this end, the paper devises a capsule network to learn from the vectorial features by Gaussian kernels and routing method. Experiments on six real-world tabular datasets are conducted to validate the effectiveness of the proposed method named TabCaps. The results suggest that TabCaps outperforms or is competitive with the previous deep learning approaches.
strength_and_weaknesses: Strengths:

1. The idea of using capsules to encapsulate all tabular features of an instance into vectorial features is interesting. In this way, we do not have to deal with individual features so we may integrate heterogeneous features in learning.
2. TabCaps achieves good performance with a small number of parameters in the six datasets.
3. TabCaps also shows strong generalization performance.

Weaknesses:

1. It is counter-intuitive to me why deep learning methods have not encapsulated all tabular features. With several layers of networks and by embedding them into the hidden representations, deep nets could also learn how to integrate the information of individual features. That being said, it is unclear why TabCaps can achieve better performance.
2. Only six tabular datasets are considered. Experiments on more datasets will enhance the results.
clarity,_quality,_novelty_and_reproducibility: The clarity, quality novelty, and reproducibility are OK.
summary_of_the_review: This work proposes a capsule-based framework to learn from tabular data. The idea is interesting, but it is unclear why the method can perform well and what the advantages are over the existing deep learning methods. The experiments are not convincing as well since there are only six datasets.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a capsule neural network for tabular data classification.

The reviewers agreed the paper is well-written, with good experimental results and setup, and the method appears to be well-grounded in its motivation. The reviewers also agreed the paper is a nice step towards popularizing capsule networks in the context of tabular data.

Expanding the results to more datasets with clear gains would improve the significance of the paper.




justification_for_why_not_higher_score: More empirical significance would be helpful.
justification_for_why_not_lower_score: Good step towards making capsule nets work for tabular data.
Decision: Accept: poster