Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors propose a end-to-end deep learning model called Net-DNF to handle tabular data. The architecture of Net-DNF has four layers: the first layer is a dense layer (learnable weights) with tanh activation eq(1). The second layer (DNNF) is formed by binary conjunctions over literals eq(2). The third layer is an embedding formed by n DNNF blocks  eq(3). the last layer is a linear transformation of the embedding with a sigmoid activation eq(4). The authors also propose a feature selection method based on a trainable binarized selection with a modified L1 and L2 regularization. In the experimental analysis, Net-DNF outperforms fully connected networks. 

pros:

- A novel approach to handle tabular data using deep learning with an integrated feature selection
- The VC dimension bound gives theoretical motivation  for expressing decision trees as DNF formulas.

cons:

- Classical ML approaches are outperforming Net-DNF
- The experimental analyses are not convincing: The datasets are a bit limited and performance metrics are rather limited to log-loss which is difficult to interpret


remarks:

- It is unclear whether adding the Net-DNF layers, in the ablation study, gives improvement due to simply an increase of model capacity or the specific architecture of Net-DNF.
- In Exp 7, why there is OOM despite removing additional layer ?
- The authors did not discuss a similar recent work: TabNet (https://arxiv.org/abs/1908.07442) 
- The authors need to include additional datasets similar to TabNet experiments
- in $R(m_t, m_s)$ I think there should not be a division by 2 
review_title: Positioning w.r.t. TabNet is needed

final_score: rating: 7: Good paper, accept
review_content: The paper attempts to propose a neural network-based algorithm on tabular data, or non-distributed representation to address the unique challenges in tabular data, which are non-existing in the distributed data (images, language, etc). The primary motivation of this paper is the simulation of DNF, which simulates the Boolean formulas in decision making. The DNF form can better capture the non-distributional nature of tabular data and somehow simulates with the decision-tree algorithms in a more soft-threshold way. The general idea makes sense to me and sounds novel compared to the existing literature. However, I still have some doubts and questions about the paper:
1) In Sec 2.1, this soft approximation is a very critical part of the paper. Why do you decide to modify the constant from 1.0 to 1.5, you suggested in the Appendix that "the AND gate will be closer to 1", but the OR gate will be affected, right? Then OR(0,0,0,1) = tanh(0.5) = 0.4, which is pushed away from 1. I believe there are some trade-offs for this hyperparameter, how do you decide it should be 1.5 rather than other numbers, do you have supporting evidence for your decision? 
2) In your feature selection part, the $w_t$ is a learnable parameter to control the sparsity of the selected feature. You propose to achieve this by using the elastic net regularization. Is the trainable mask $w_t$ shared among all the DNNF blocks or is it specific to a certain DNNF block? 
3) In your experiment table 1, some results from FCN are shown to cause OOM. I kind of get the idea that DNNF uses a more compact parameterization, but not entirely sure how it quantitatively compare with FCN. What is the parameter complexity for DNNF? Could you give a more formal explanation about why it's more compact?
4) In table 3, some of the experimental results are better than XGBOOST and some are worse. I can understand that the current DNNF has not yet achieved the same performance as XGBOOST as XGBOOST has always been a go-to algorithm for tabular data. But could you explain why DNNF is much better than GAS Concentrations, is it because of some properties of this dataset? And do you have a more high-level suggestion as to "under what circumstances is DNNF likely to out-perform XGBOOST", what could be the reason for that? Is it because of the better generalization of the neural network?

Typo: 
2.4 Spacial Localization -> Spatial Localization

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a neural architecture that emulates the characteristics of decision-tree variants, in the hope of mirroring their successes on tabular data. The architecture consists of three components: DNNF blocks, feature-selection masks, and spacial-localization weightings of DNNF blocks. I find these components and their coherent combination into the Net-DNF structure to be novel.

However, their empirical evaluations do not place the performance of Net-DNF in the context of existing work. It is good that the authors have compared Net-DNF against the obvious baselines of XGBoost and FCN, but they have neglected to demonstrate where Net-DNF stands in relation to other tabular-inspired approaches (mentioned in the related work). For example, why did the authors not compare against Popov et al. (2019; whose code is open-sourced) and Shavitt & Segal (2018)?

I feel that the authors have left out a line of work that uses a deep learning approach for tabular data. The reference below is a representative publication. (NB: The VAE approach can be evaluated on prediction tasks like Net-DNF, and it works on multi-modal data that Net-DNF purports to handle.) Could the authors comment on where Net-DNF stands relative to this line of work (and include it in their related work section)?

Nazabal, Alfredo, et al. "Handling incomplete heterogeneous data using vaes." Pattern Recognition (2020)

The exposition of the paper is lucid throughout.

QUESTIONS

* Page 1, para 2, "Moreover,...multi-modal data,...is problematic":
How does Net-DNF handle multi-modal data (e.g., "medical records and images")? Would it simply encode an image as a vector of bits and stack it in input x?

* Page 1, para 2, "GBDTs...hard to scale":
How scalable is Net-DNF? What is its size complexity?

* Page 6, Table 1 and 2.
"# formulas" corresponds to the number of DNNFs in an Net-DNF, but what does it correspond to in an FCN? How many parameters are in each system? Wouldn't the number of parameters provide a fairer comparison of the capacities of the systems?

* Page 8, Figure 2.
The authors claim that space localization will improve the green line's performance (feature selection) on Syn4-6. Why don't the authors provide empirical results to show this? Seems like a straightforward data point to add.

NITS

* Page 2, first line: inherent GBDTs -> inherent in GBDTs
     
* Page 6, In is evident -> it is evident

review_title: Novel neural architecture emulating decision trees/forest, but with gaps in empirical evaluation.

Meta-review: The paper proposes an end-to-end architecture, Net-DNF,  for handling tabular data. This is a novel approach in the relatively under-explored domain of application of neural networks; the paper also presents justification of the design choices via ablation studies. The paper is clearly written, and empirical results are convincing.

Decision: Accept (Poster)