Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
In this paper, authors propose a new message function searching method for Hyper Relational Knowledge Graphs. They also proposed a search space of message functions. Finally the authors proposed a one shot NAS algorithm for searching in the message function space and also over other GNN components for a given Hyper Relational Knowledge Graph.



main_review: I think the work is novel in including the message function in the search space for NAS. This introduced a novel way to learn the representation using flexibility of the knowledge from the message functions. The search algorithm is a variation of the NAS tailored for this framework. 

The experiments shows better results for the proposed method in comparison with other baselines. The other methods shows variable effectiveness over different datasets, but the proposed model beats all of them in all the presented results. 

The ablation study also looks good and shows effect of different message passing functions and algorithms for the framework. 

We have result on different arity (3 and 4) for the same dataset. It would be interesting to see the relation between arity and effectiveness of the method tabulated.

Authors have shown result in link prediction task and relation prediction tasks. It would be interesting to see the performance in more variety of tasks.

It may be outside of the scope of this paper, but one question can be how should the framework perform in non HKG problems? May be we can get good result for other one shot problems in graph.






summary_of_the_review: Overall I think the paper is technically strong and authors have shown effectiveness of their method. Although the scope may be somehow limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose to conduct neural architecture search (NAS) for hyper-relational knowledge graphs (HKGs). Compared with normal graphs, HKGs can better model the complex relationships between different entities. Specifically, a novel search space is proposed inspired by recent message-passing GNNs on HKGs such as G-MPNN, CompGCN, StarE, etc., and a differentiable search algorithm is utilized. Experiments demonstrate the effectiveness of the proposed method.
main_review: Pros:  
* Automatically searching message-passing functions for HKGs is an important yet underexplored problem.  
* The proposed search space seems to incorporate most state-of-the-art MPNNs for HKGs.  
* The experimental results are comprehensive, with ablation studies, examples, and various analyses provided in the appendix.  

Concerns and questions:  
* The writing of the paper could be improved, especially the methodology part where there are many symbols involved, which is hard to follow.   
* Some experimental results are not consistent with results presented in previous papers (e.g., StarE and G-MPNN). The authors need to explain where the difference comes from.  
* The code is not available yet, so there is currently no way to assess the proposed method experimentally. I believe the authors would make the code publicly available at publication time?  

summary_of_the_review: Overall, I think this paper makes non-trivial contributions in the search space design for HKGs with extensive experiments, and thus may be interesting to both KGs and NAS community.

===after rebuttal===
I have read the rebuttal and my score remains positive.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new neural architecture search framework over graph neural networks in hyper relational knowledge graphs (HKG). The main motivation is that the current GraphNAS system does not have an emphasis on edge representations and how message passing functions handle position-sensitive information on HKG. The paper proposes a new search space of these configurations and proposes a new HKG method that achieves state-of-the-art on several benchmarks.
main_review: Overall I feel the paper is nicely written and easy to follow. I am slightly confused by the main point of the paper. Is the main contribution of the paper proposing a new HKG method that achieves state-of-the-art in link prediction and relation prediction over HKGs, or the main contribution is a NAS platform with different configurations of GNNs in the search space? It seems that the authors claim the contribution to be the second but in the experiment, they just show that here’s a new method that achieves SOTA compared with prior methods. I am mostly curious about the performance of other prior baselines after doing the same hyperparameter search using the proposed framework. Besides, aren’t all the methods and details in Sec 3.2 already well established in the literature, including but not limited to how to frame the optimization as a search problem, gumbel softmax, etc.

- “But in scenarios of HKGs, it is important to know which relation (edge type) makes entities (nodes) adjacent.” What do you mean by this sentence?
- It should be “LP/RP” instead of “LR/RP” in the last row of Table 1.
- “firs” -> “first”, ‘cases studies” -> “case studies”
- The paper mentioned that they relax the discrete search space into a continuous and probabilistic space. How did you relax discrete parameters like different aggregation and activation functions? 
- What’s the difference between the position matrix and role embedding? I assume both are used to distinguish entities at different positions in the hyper edge.
- Many of the configurations in Sec. 3 seem to be also applicable in general graph neural networks, e.g, concatenation, composition, residual connection and so on. It would be helpful to clarify which ones are specific to HKG, which ones are not. Otherwise it’s confusing and it’s unclear how novel the proposed design choices are.

summary_of_the_review: I feel the paper is not currently clear in the contribution and there are some disjoint between the method section and the experiment section. Also I am not entirely convinced of the necessity of a new NAS framework specifically designed for GNNs on HKGs. It seems like a problem of limited scope, and many new configurations also apply to GNNs in other tasks.
review_title: null

Meta-review: The paper studies  neural architecture search for hyper-relational knowledge graphs (HKGs).  A  search space is put-forth, and  it is  searched with a differentiable search algorithm. The paper is technically strong. However,  there are some concerns about the narrow scope of the problem/solution, given that other more general formulations have also been studied.
Decision: Reject