Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper is focused on identifying/discovering feature interactions in blackbox models (with a focus on recommendations). Specifically the technique works by first corrupting datapoints and then using this "local dataset" to find interacting features via lasso-regularized multi-level perceptron approach (NID from Tsang et al). Using an expanded feature space and repeated calls to the above steps this is then used to find the most commonly occurring patterns. 

The empirical section is quite interesting, with some nice mix of qualitative and quantitative results. The quantitative results in particular are quite intriguing -- across recommendation and non-recommendation tasks.

Overall the paper makes for an interesting read. On the whole I lean slightly positive -- largely due to the empirical section and the quantitative gains observed by adding the feature interactions as features to the model.

That said I had quite a few concerns about the work:

- In general the exposition was quite lacking in the methodological section. I had to re-read the paper a few times to be able to make out to fully understand the underlying methodology. I would make the overall pipeline very clear and explain out the MADEX pipeline very clearly. The feature expansion section (4.3) in particular was not very clear -- as to how it fit in with the rest of the pipeline and something that could do with more work.  Nomenclature like source and target model are used somewhat arbitrarily and again can be clarified.

- Another concern I had with the approach is how this would work if the underlying features itself were not known. Say for example you had a text understanding model -- the tokenization and vocabulary/OOV etc .. may all be components of the blackbox. Likewise for vision models, what features are being used may vary.

By assuming (effectively perfect) information about the features used, this is no longer really a "blackbox" model. I would have wanted to see some deeper discussion on this topic.

I also had some concerns about the scalability of dense features and their bucketization in this approach. I'm not entirely convinced the method would work as well and efficiently in such feature spaces. I would have wanted to see some more empirical evidence and understanding on this topic.

If possible it would have also been great to understand higher-order interactions and the ability of the model to find them empirically. 

- One thing I wasn't entirely certain of was the amount of novelty in the work since it leverages existing works like NID (Tsang et al) and LIME (Ribeiro et al) for some of the key aspects of the method. It would be nice to see some discussion on this front as well.

- I also would have liked to see significance testing being performed in general across the experiments.

- Another smaller concern stemmed from a lack of mention of the stopping criterion of the model. In particular I'm wondering how models were stopped further training / best checkpoint was picked. This would be good to elaborate on to make sure that the models were all compared fairly from a computational perspective.

- I also had some concerns with the scalability of the approach in large features spaces but am willing to give the authors the benefit of doubt on this one based on the 3 hour number they quoted from their experiments (which indicates something that is not prohibitively slow)

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes a method to detect which features in the input of recommender systems are interacted each other, i.e., combining them behaves useful information, and examines to feed extracted interactions directly into the recommender systems to measure effects on actual recommendation.

The interaction detector consists of 1. perturbing the input vectors,  2. training NNs to utilize its internal non-linear representation as a signal of interaction, and 3. aggregating detected interactions over training set. 1. and 2. are consisting of known methods so that the proposed method is an application of them. For 3. authors introduced a simple heuristic (Algorithm 1).

As long as I heard how the NID work to detect interaction from this paper, it is sensitive not only the true interaction between features but also set of features holding similar signals (e.g., if a hidden unit behaves as a feature A, it may be natural to aggregate all features that implies A by itself, regardless of the meaning of their interaction). This is not a desired case as long as the paper specified the interactions in section 3. Maybe it requires more detailed explanation about how good applying NID for this task is.

Experiments are conducted to show the behavior of the interaction detector and actual improvement of utilizing extracted interactions as additional features. Experiments look less informative for comparing the proposed method with other existing methods for similar motivations (e.g., some methods introduced in related work) since there is only a trivial baseline by the original LIME's method.

In the case-study of Figure 4(b), I thought that the proposed method is a bit biased for frequent but meaningless features, because it detected "I" or "a" that intuitively occur with any labels. It is maybe because the Algorithm 1. that simply aggregates all detected interactions regardless of their actual importance. For further improvement, it may be necessary to introduce some weighting strategy to detected interactions.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposed a model for extracting global feature interactions from the source model which was later being encoded in the target model to enhance its prediction performance.

Strong points:
1. The paper laid out the necessary background knowledge very clear even for the audiences outside this area.

2. The paper performed reasonable amount of experiments to compare the proposed model with various baseline models with various data sets, which are strong enough to support the claims made in this paper.

Comments:
1. The theoretical innovation of this paper is trivial. The local detection model for feature interactions, i.e. MADEX is simply employing the previous work, LIME and NID. Its global extension, i.e. GLIDER and the proposed encoding method are straightforward. 

2. The descriptions on the feature dimensions of the original data and the generated binary representation data x’ are rather confusing and inconsistent throughout the paper. If I understand it correctly, the data sample from the original feature space x \in R^p, and a  binary representation x’ \in R^d, where d <= p. However, when d first appears in the first paragraph of section 3, it is defined as the dimension of the original feature space and later in that paragraph the dimension became p in “f(.): R^p -> R”. There is no clarification on the differences between d and p till section 4.1 where it states “x \in R^p ”. However, the dimension of the data instance from the original feature space changed to d again in section 4.2 where it states “x =[x_1, x_2, …, x_d]”. 

3. “What’s more, the same approach to interpreting interactions can provide new insights into domains even beyond recommendation.” should be “What’s more, the same approach to interpret interactions can provide new insights into domains even beyond the recommendation.”

4. “An interaction, I, is a subset of all input features...”, but according to the following sections, I is the indices of a feature subset instead of the features themselves.
 
5. “...by requiring the same cross feature ID to occur more that T times in a batch of samples,...”, ‘that’ should be ‘than’.

Overall, although there is no significant theoretical innovation, it is a decent application paper.

review_title: Official Blind Review #3

Meta-review: The paper extracts feature interactions in recommender systems and studies the effect of these interactions on the recommendations. While the focus is on recommender systems the authors claim that the ideas can be generalised to other domains also. 

All reviewers found the empirical results and analysis thereof to be very interesting and useful. This paper saw a healthy discussion between the authors and reviewers and all reviewers agreed that this paper makes a useful contribution. I recommend that the authors address all the concerns of the reviewers in the final version of the paper. 
Decision: Accept (Poster)