Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Existing graph transformers have succeeded in different applications but require cumbersome architecture designs and tuning by experienced engineers. This paper proposed a novel search framework for graph transformers, automatically searching within the designed unified search space for optimal architecture designs. The proposed method is compatible with most of the existing state-of-art graph transformers designs without introducing a substantial amount of computation cost. The experimental results show proposed strategy results in better designs over hand-crafted baselines. 
strength_and_weaknesses: Strength 
1. Well-explained basic transformer and graph encoding strategy, which show excellent intuition of the search space design. 
2. The proposed framework is compatible with more existing graph encoding strategies, which allows a broader comparison between graph transformer designs. 
3. Tab5 shows an ablation study using a single supernet vs. additional supernets. The ablation study shows the improvement of performance by splitting a supernet. Such a design brings extra performance without extra parameters. 

Weakness:
1. Section 3 explained the proposed AutoGT well in text. However, neither Fig 1 nor Fig 2 are self-explanatory. I could not understand their relations with the proposed method without reading the main paragraphs. The small fonts in the figures made it more challenging.
2. Figure 2 was never mentioned in the text, which makes it tricky to understand which section it belongs to. 
3. Table 3 tries to show the importance of the proposed evolution search by comparing it to simple random selection and existing hand-crafted options. The results have large deviations. How can we ensure the proposed method results in consistently high-quality architecture designs? 
4. The proposed method aims to automatically provide graph transformer designs. However, AutoGT introduces extra hyper-parameters such as the number of supernet sizes and the search space's design. 

clarity,_quality,_novelty_and_reproducibility: Consistent quality

Table 3 tries to show the importance of the proposed evolution search by comparing it to simple random selection and existing hand-crafted options. Excellent experiment design! However, given the large deviation from the results, how can we ensure the proposed method results in consistently high-quality architecture designs? Without such a property, engineers will have concerns about deploying such a model to real-world applications. 


Using eight subnets

The ablation study shows the performance boost from splitting the supernet into eight subnets. I would like to know how this behavior scales. Can you have better/worse results by having 2,4,16,32,64 subnets? You don't have to run all these setups, and some intuitive explanation serves my curiosity. 


How to tune AutoGT

Since the proposed method aims to automatically provide graph transformer designs, AutoGT also has some hyper-parameters, for example, the number of supernet size and the search space. Can you justify why this makes the design process easier? How can you make sure the search space includes the right amount of options, not too many, not too few? 

summary_of_the_review: This paper proposed AutoGT, which can automatically search the design space, which is compatible with most of the existing state-of-art graph transformers designs, without introducing a substantial amount of computation cost. The experimental results show proposed strategy results in better designs over hand-crafted baselines. However, I have concerns about the ease of use since AutoGT introduces extra hyperparameters that need to be tuned. And the auto designs from AutoGT have less consistent performance. 

I recommend 6-marginally above the acceptance threshold. 


----- After hearing back from authors 11/16 -----

Thanks for the explanation and new results. They cleared all of my concerns. Changing to "8: accept, good paper "
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper investigates the interesting problem of automated graph transformer through neural architecture search. The authors first propose a unified graph Transformer formulation that can represent most of the state-of-the-art graph transformer architectures. Then, an encoding-aware architecture searching and supernet training strategy are proposed. Experimental results over several graph datasets including the large-scale OGB datasets show the effectiveness of the proposed method.
strength_and_weaknesses: ### Strengths:
+ The proposal of automatically searching the optimal graph transformer architecture is important in the community of machine learning.
+ Constructing a unified graph Transformer formulation by combining several typical transformer architectures and graph encoding strategies seems to be a novel contribution.
+ Taking graph-specific properties into the automated graph transformer architecture search process is helpful and can distinguish this paper from other approaches such as autoformer.
+ Experiments are conducted over several widely used and benchmark datasets.

### Weaknesses:
- The major focus in this work is the complex relations between transformer architectures and the graph encodings in transformer layer. The authors are suggested to further discuss why this problem is important and challenging, and how the authors address the problem specifically. 

clarity,_quality,_novelty_and_reproducibility: This paper is well written with comprehensive experimental evaluations over several well-used datasets. The idea proposed in this paper is novel. The paper is overall of good quality.
summary_of_the_review: This paper studies an interesting problem of automating the neural architecture design of graph transformer. The authors raise the challenges in the investigated problem, followed by the proposal of technical solid solution. Extensive experimental results validate the effectiveness of the proposed solution. I think this paper tends to be interesting to the research community and recommend acceptance for this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is the first work on automating the architecture design of graph Transformer, which is rarely studied in graph Transformer research. The contributions lie in solving the two challenges specifically related to graph-structured data during the design of neural architecture search.
strength_and_weaknesses: Strengths:
1. The novel idea of automating the design procedure of graph transformer.
2. Utilizing the neural architecture search for graph Transformers tends to become popular and practically important in graph representation learning community.
3. The proposed model is solid and the solution is appropriate for solving the challenges.
4. The paper is well written and the related work eases the understanding of neural architecture search and graph Transformer literature.

Weaknesses:
1. Fig. 1 is presented without explanation. Detailed explanations are expected to be provided. 
2. For the architecture search and training step, the authors are encouraged to further discuss the difference from the proposed main ideas from the previous works on neural architecture search.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, with good quality and novelty. Reproducibility will be guaranteed if the authors promise to release the code.

summary_of_the_review: This work proposes a neural architecture search based framework, Automated Graph Transformer (AutoGT), for automating the architecture design of graph Transformers. Tailored for non-euclidean graph data, the search space of AutoGT takes both the graph encoding strategy and Transformer architecture into considerations. Furthermore, the performance estimation strategy of AutoGT is designed to be encoding-aware through joint optimization of the encoding strategy and graph Transformer architectures. Experiments over several graph datasets with different sizes demonstrate the effectiveness of the proposed AutoGT. I think this paper investigates an interesting problem of good importance and I recommend to accept.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a new research problem, Automated Graph Transformer Architecture Search, and proposes a unified graph transformer formulation for existing graph transformer architectures. Then a novel architecture search method that can jointly optimize Transformer architecture and graph encoding strategies is proposed. The problem and the method are well motivated and formulated, witch extensive experimental supports. The paper is also well written. All the (minor) concerns are addressed after the rebuttal, and the reviewers unanimously suggest acceptance for this work. The AC also believes this work studies an important problem and is a solid contribution to the ICLR community. 
justification_for_why_not_lower_score: This paper receives three very positive ratings (8 8 8) and is among top 2.7% according to the conference reviewing statistics.
More importantly, this paper studies a unique and interesting research problem, automated graph transformer architecture search and propose a non-trivial and effective method for doing it.
Decision: Accept: notable-top-5%