Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose a graph approach that connects an item to classify and its features with relevant items via the common values. This graph is used to propagate feature and label information and then classify the item. Comparisons are made in multiple datasets for two tasks (recommendation and classification) with several state-of-the-art approaches. The results show that the proposed approach can improve the performance on the specific metrics.
strengths_and_weaknesses: This is an interesting approach and based on the results it seems to improve upon various state-of-the-art models. It is described quite well, even though I would like to see a bit more detailed description of the implementation. Some algorithm pseudo-code would help. Experiments are good with several models for comparison. The results are also good and not negligible I would say.

Something that is missing in the paper is the detailed description of how the baselines were tuned. For example, DeepFM is quite a strong approach but one need to tune properly from my experience. So, I am not sure if the difference is due to the approach that better captures and propagates the relevant information or just note well tuned approaches.

The authors could also have added graph approaches like for example GraphSage or other state-of-the-art ones.

Also, the retrieval of relevant instances is thoroughly studied. This has a big impact I imagine on the learning process. I would like to see here different scenarios and more analysis. 
questions: One question I have is about the relevant item extraction. This works well for binary or categorical features but did you try with real-valued features?



limitations: As I mentioned previously the retrieval process as defined seems to limit the applicability of the approach for the kind of features the authors consider.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose PET, an architecture that Propagates and Enhances the Tabular data representations for label prediction using hypergraphs.  Given tabular data, the novel end-to-end system consists of of several steps, beginning with hypergraph construction based on information retrieval, followed by partitioning of labels and feature into a bipartite graph, then several rounds of message passing to refine high-level interactions between nodes and edges of the bipartite graph, and finally target node embeddings are fed to an MLP for label prediction.  PET is shown to significantly outperform a wide variety of state-of-the-art label prediction methods for two tasks: click-through rate (CTR) prediction and top-n recommendations.  Furthermore, due to the complicated nature of the end-to-end system, an extensive ablation study is conducted to understand the affects different components of the model have on downstream performance.
strengths_and_weaknesses: Originality
Strengths: While PET borrows inspiration from previous methods (i.e, RIM also uses information retrieval to learn inter-row -column high-level interactions), the use of hypergraph neural networks for tabular data prediction is novel and extremely interesting.  
------------------
Quality:
Strengths: The approach, methodology, and experiments are all well done.  In particular, the various stages of PET make intuitive sense at drawing out and refining high-level interactions within the data to improve target node prediction.  Furthermore, the ablation study was thorough and illustrative to understand the impact of the hypergraph neural network on downstream performance, as well as visualizing the structured latent space learned in PET compared to the nearest SOTA competitor RIM.

Weaknesses: There are three large concerns given PET.  While the overall model has been shown to perform extremely well, model complexity is high.  While exact runtimes and the necessary compute required to put this model into action for large-scale data are not currently discussed, it does not seem like all researchers will have access to compute resources necessary to train and deploy such a model.  

The second concern is related to the first; how does runtime scale with the number of features considered?  While the datasets utilized in the paper contain large numbers of samples, they contain very small numbers of fields.  Are the various steps within PEP tractable when the number of features grows on the order of hundreds, as is commonly encountered in large-scale binary classification datasets?

Finally, it is not clear how computationally expensive hyperparameter optimization of PET is, nor how often this is necessary (each presented dataset has a different configuration of hyperparameters).

------------------
Clarity:
Strengths: The paper is very well written.

Weaknesses: A necessasry discussion of hyperparameter tuning for the various methods evaluated is currently lacking in the paper.  In particular, for the competitors in Table 2 and 3, the choice of hyperparameters is not discussed.

------------------
Significance:
Strengths: GNNs are an extremely active area of research right now.  Coupled with the impressive performance demonstrated for two important tasks, PET has the potential to significantly influence current work looking to similarly leverage GNNs to capture high-level latent interactions among features for different prediction tasks.
questions: What dataset are the T-SNE plots derived from?  Interestingly, PET is learning overarching structure inherent in the data, whereas RIM seems to just learn small clusters.  It would be interesting to see the influence of utilizing edge and node labels on the ability to learn this
structure, i.e., repeating study in Section 4.3.1 to regenerate these T-SNE plots and viewing whether PET (w/o node labels) learns less structure in the data (like RIM T-SNE plots) and the overarching structure becomes more apparent with each edge addition.

Can you comment on the process for choosing PET hyperparameters for the different datasets?  As the hyperparameters differ per dataset, how much computational time does hyperparameter optimization require per dataset?

PET is complicated, can the authors comment on the computational price incurred for the various steps, i.e., hypergraph construction/information retrieval, star expansion, message passing, embedding calculations, MLP training time, and hyperparameter tuning across all these various steps?  How does this compare to RIM?

Question from the previous section: Are the various steps within PEP tractable when the number of features grows on the order of dozens/hundreds, as is commonly encountered in large-scale binary classification datasets?
limitations: NA
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose a novel architecture for learning representations over “tabular data”. The problem setup is highly relevant as learning over tabular data has diverse practical implications in multiple fields. The proposed architecture, named “PET: Propagate and Enhance the Tabular data”, utilizes multiple components at its core - (i) relevance based retrieval to obtain $K$ instances similar to the target data instance;  (ii) message passing (over the star expansion of the generated hypergraph); (iii) label enhanced feature learning; (iv) higher-order feature interaction. To retrieve the relevant instances, the authors utilize the “relevance value” (proposed by Qin et al., 2021), which is akin to BM25 when the data instances are treated as documents and their feature values as the terms, as mentioned by the authors. The resulting set of the original instance along with the retrieved ones is processed as a hypergraph, where each distinct feature value in each individual field forms a node and each data instance constructs a hyperedge. This is further subjected to the star expansion to construct a bipartite graph. Subsequently, few steps of message passing are applied over the graph, with multiple operations involved. The messages are generated by the interaction of the edge and node representations - eq 6. The aggregation of the messages for each node is done by an attention mechanism, attributed to the widespread success of the basic attention module. The node embedding update is then performed via a transformation over the concatenation of previous layer representation with the aggregated messages. The edge embedding update follows a similar process. Finally, the target data node representation is passed through MLP for prediction and the model is learned end-to-end (cross-entropy is used for binary prediction tasks). As emphasized by the authors multiple times over the course of the paper, the proposed architecture aids the feature learning process by inducing the label information into the representations as well as allowing for higher order feature interactions with more rounds of propagation. The empirical analysis is done on 5 majorly used datasets and comparison has been performed against multiple baselines. 
strengths_and_weaknesses: 1. The results provided in tables 2, 3 and 4 “strongly” demonstrate the efficacy of the proposed model. The improvements in both - CTR and top-n recommendation tasks are significant.
2. Figure 4 provides interesting visualizations that demonstrate the efficacy of the proposed method PET and inline with the experimental results.
3. While the authors have provided some hyperparameter and sensitivity analyses for label usage (sec 4.3.1), retrieval schemes (sec 4.3.3) and $K$ (appendix A.3), it will be also interesting to see analyses of important attributes like the number of GNN layers and embedding size as well.
4. The authors utilize retrieved neighbors (eq 1 and 2) to obtain the representation of the target data instance under message passing framework. Furthermore, they mention explicitly in section 4.3.3 that the “relevance-based retrieval” can help achieve homophily, and thus improve performance. However, in the cases when relevance based retrieval fetches $K$ instances that are close in the input space, but have different labels, the method can give highly sub-par performance, particularly when the GNN layers are more than $2$. Such scenarios can occur when the output class label is heavily dependent on a few “fields” and the data instances have high similarity in the irrelevant fields (more in count). Can the authors provide some arguments and/or experiments under this setup?
5. The authors have mentioned prior works utilizing GNNs over tabular data in section 2 (lines 77-79), however the comparison in tables 2 and 3 doesn’t contain these baselines. Can the authors elaborate on this? It’s strongly recommended to provide the comparison for the completeness of the work.
6. Tables 6 and 7 of the appendix provide the corresponding standard deviations for the proposed model. I suggest the authors provide the same for the second best method, ie the best performing baseline, for completeness as well.

questions: 1. In section 4.3.1, the authors have performed two primary ablations by - (i) removing “edge labels” and (ii) removing “node labels”. Do the authors have some results where both “edge” and “node” labels are removed? Essentially, only the “feature interactions” are captured in this setup. This might help validate the efficacy of the structure of the proposed architecture.
2. The authors have utilized “directional” edges in the message passing mechanism. Do the authors have supporting arguments and/or experiments for the case where the graph obtained post the “star” expansion is undirected? In this case, the edge embeddings can be a composition of the two nodes. The composition function can be a simple operation like concatenation, which has been heavily utilized in this work, or something more sophisticated - MLP.
3. Can the authors also describe how they arrived at the hyperparameter values provided in table 8 of the appendix?
4. From figure 7 in the appendix, it is evident that $K=15$ or $K=20$ provide the best results. Is there a specific reason that the authors have used $K=10$ as shown in the hyperparameter table 8 of appendix? 

limitations: As mentioned in section A.1 of the appendix, the constructed graph can be dense when the number of fields is large. Moreover, the graph is created for **each** target instance, which can be computationally expensive. The problems of oversmoothing and oversquashing may occur in such cases. I commend the authors for explicitly stating this. I also believe that the assumption that relevance based retrieval can help realize the homophily may not always hold, as described previously. Although such scenarios can be rectified by utilizing the enhancements proposed in the recent literature focused on learning GNNs under heterophily (such as [1]), I would like to know the authors’ response to this argument. Lastly, I don’t see any serious negative societal impacts of the work. 

[1] ​​Graph Neural Networks with Heterophily - Jiong Zhu, Ryan A. Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K. Ahmed, Danai Koutra
review_title: null

Meta-review: metareview: This paper proposes (PET), an approach to classifying rows in tabular data using retrieval methods and hypergraph neural networks to make predictions. The key ideas are
- use information retrieval techniques to find similar rows to each row that needs to be labeled.
- connect the similar rows in a hypergraph structure.
- learn a representation over the hypergraph structure with graph neural networks.

Experiments show that PET can singificantly outperform multiple state of the art methods on two tasks. Ablations also validate the design and each component of PET. During the review process, the authors added additional experiments addressing many of the reviewers open concerns.

The reviewers agreed that the paper is very well written, presents a significantly useful method, and that while PET builds on pieces that have been developed separately, it combines them in an interesting way. Reviewers also felt that the work is likely to be of interest to the wider graph neural network community and has the potential to influence future work.
confidence: Less certain
Decision: Accept