Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: In the paper, the authors proposed CrossGO, an algorithm for finding crossing features useful for prediction.
In CrossGO, one trains a neural network that captures feature crossing implicitly.
Then, possible crossing features are estimated using the gradient-based saliency.
The idea here is that, if a feature has a crossing with some other features, its contribution in the saliency can vary across different inputs.
Thus, by looking at the variation of the saliency, one can find candidates features for feature crossing.
CrossGO greedily selects candidate crossings based on the idea above.
In the last step, a simple logistic regression is trained using the candidate crossings, and the effective crossings are selected using a forward greedy feature selection.

I found the paper well-written and the idea is easy to follow.
My concern, however, is the lack of Factorization Machines (FM) in the experiments.
In Introduction, the authors mention to the deep version of FM and stated "(deep FMs are) not able to generate interpretable cross features".
But, as the authors are aware of, non-deep FMs are able to handle feature crossings in a interpretable way.
Thus, it would be essential to adopt non-deep FMs as the baseline in the experiments.
Because the important baseline is missing, I found the results are not convincing enough to claim the effectiveness of the proposed method.


### Updated after author response ###
The authors have partially addressed my concern by adding FM/HOFMs as the experiment baselines, which I greatly appreciate.
However, I found the current paper misses some other possible baselines for high-order interaction models [Ref1,2].
As the authors mentioned in the response, FMs find the feature crossing as a kind of embedded representations, which may not be suitable for modeling sparse interactions.
Thus, the sparse interaction models need to be taken into consideration as well.

[Ref1] Safe Feature Pruning for Sparse High-Order Interaction Models
[Ref2] Selective Inference for Sparse High-Order Interaction Models
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 
The paper presents a scheme to generate new features as cross-product
of binary features to improve the performance of linear models while
obtaining interpretable models. The candidate set of cross-features
can be exponential and is handled by the proposed scheme by utilizing
the gradient-based importances of the features in a (deep) neural
network. Features with large discrepancies in their local and global
interpretations are used as the seed set of candidate features for
generating new cross-features, and the final step performs a feature
selection to further reduce the final set of cross-features. The
empirical evaluation demonstrates the utility of the proposed scheme
on 8 datasets. 

While the proposed scheme does present a way to improve the accuracy
of interpretable models, I am currently recommending a reject for the
following reasons (given the higher standard recommended for papers
over 8 pages):

- While this paper does consider some baselines, it seems to be
  missing some crucial baselines that address the same (or very
  similar) problem. There are some papers [2,3] that learn boolean
  conjunctions (that can be seen as cross-features) to generate
  accurate interpretable models. Moreover, there are some search based
  feature generation schemes [1,4] that significantly improve upon the
  exhaustive feature generation scheme of Kanter & Veeramacheneni,
  2015. This technique can easily be applicable in learning boolean
  cross features with binary features. At the very least, it is important to 
  understand where this proposed scheme is positioned relative to the 
  aforementioned literature and why a comparison is not required.
- It is very unintuitive (at least to me) to tie the candidate
  generation scheme to a neural network especially given the
  sensitivity of neural network training to different initializations
  and other factors. For the same data and neural network, the local
  vs. global discrepancies can change significantly, thereby changing
  the candidate set of cross features. This can potentially make the
  proposed feature generation scheme somewhat unstable, and the
  interpretations from the subsequent models might not be as
  interpretable as they seem. It would be good to understand what I am
  missing here and why being tied to a neural network model is
  essential and not an issue here. 


Clarification:

- Lines 8-10 in Algorithm 1 is not clearly explained.
- The experiment to motivate Assumption 1 needs to be better
  explained.


Minor:

- The notation in equation (1) needs to be clarified better.


[1] Khurana, Udayan, et al. "Cognito: Automated feature engineering
for supervised learning." 2016 IEEE 16th International Conference on
Data Mining Workshops (ICDMW). IEEE, 2016. 
[2] Dash, Sanjeeb, Oktay Gunluk, and Dennis Wei. "Boolean decision
rules via column generation." Advances in Neural Information
Processing Systems. 2018. 
[3] Wei, Dennis, et al. "Generalized Linear Rule Models." Proceedings
of the 36th International Conference on Machine Learning. 2019.
[4] Khurana, Udayan, Horst Samulowitz, and Deepak Turaga. "Feature
engineering for predictive modeling using reinforcement learning."
Thirty-Second AAAI Conference on Artificial Intelligence. 2018. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper attempts to solve the cross feature generation problem efficiently. The state-of-the-art method AutoCross cannot control the size of the searching set of the candidates of cross feature fields. To narrow down this set, the authors provides a measurement called Interpretation Inconsistency. With an easy toy experiment, the authors conjecture that features with large Interpretation Inconsistency tend to interact with other features in the hidden layers of DNN. Therefore, based on this conjecture, the authors design an effective algorithm to discard those cross features with small Interpretation Inconsistency value, which finally narrows the set of cross features. With this narrowed set, the whole procedure can be accelerated largely. 

Pros:
This work can be regarded as an accelerated version of AutoCross. By incorporating Interpretation Inconsistency, the set of cross features can be effectively narrowed. Although this is an incremental work, this idea is relatively novel. 

Cons:
1.	The setting of the threshold for filtering feature fields is somewhat heuristic. The authors should provide some explanations on its setting. If not, we cannot trust it and doubt that it may cause some unexpected results and thus will not be robust.
2.	The experiments are somewhat not convincing. The main contribution is to accelerate AutoCross. Thus, I expect to see the time complexity comparison between them. However, I do not find it in this paper. Although the authors mention that on wide datasets, AutoCross simply cannot work, on narrow datasets, a time complexity comparison should be provided.
3.	In Table 5, only the numbers of cross feature fields of the proposed method are provided. A comparison with baseline methods on the number would be better to show the advantage of the proposed method.

Minor: In Section 2.1, the double quotation marks should be revised.
review_title: Official Blind Review #2

Meta-review: The authors propose a simple but effective method for feature crossing using interpretation inconsistency (as defined by the authors).

I think this is a good work and the authors as well as the reviewers participated well in the discussions. However, there is still disagreement about the positioning of the paper. In particular, all the reviewers  felt that additional baselines should be tried. While the authors have strongly rebutted the necessity of these baselines the reviewers are not convinced about it. Given the strong reservations of the all the 3 reviewers at this point I cannot recommend the acceptance of this paper. I strongly suggest that in subsequent submissions the authors should position their work better and perhaps compare with some of the related works recommended by the reviewers.
Decision: Reject