Title: Learning to Flow from Generative Pretext Tasks for Neural Architecture Encoding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Graph representation learning

Neural architecture encoding

Self-supervised learning
Abstract: The performance of a deep learning model on a specific task and dataset depends heavily on its neural architecture, motivating considerable efforts to rapidly and accurately identify architectures suited to the target task and dataset. To achieve this, researchers use machine learning models—typically neural architecture encoders—to predict the performance of a neural architecture. Many state-of-the-art encoders aim to capture information flow within a neural architecture, which reflects how information moves through the forward pass and backpropagation, via a specialized model structure. However, due to their complicated structures, these flow-based encoders are significantly slower to process neural architectures compared to simpler encoders, presenting a notable practical challenge. To address this, we propose FGP, a novel pre-training method for neural architecture encoding that trains an encoder to capture the information flow without requiring specialized model structures. FGP trains an encoder to reconstruct a flow surrogate, our proposed representation of the neural architecture's information flow. Our experiments show that FGP boosts encoder performance by up to 106\% in Precision@1\%, compared to the same encoder trained solely with supervised learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose FGP, a pretraining strategy to train neural architecture encoders to perform NAS. The proposed strategy allows capturing flow based information to predict neural network performance while being computationally cheaper. The authors use a proxy flow based objective to achieve this and show improved downstream performance with their method on predicting neural network performance as well as finding better neural architectures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The proposed FGP method is able to predict neural network performance via a flow based objective which predicts NN performance by mimicking the forward and backward passes via a graph based step.\n2. The experiments are well structured and highlight the effectiveness of FGP in predicting neural network performance as well as finding better architectures. \n\nWeaknesses\n1. While a ResGatedGCN with FGP outperforms Flowerformer in Fig 8a, training with FGP increases the encoding time of the ResGatedGCN significantly (from $10^2$ to $10^3$) presumably due to the flow based objective, which suggests that the Flow based objective still introduces significant pretraining overhead compared to other non flow based methods. However in Fig 8b, FGP is faster than other pretraining methods. It is unclear which pretraining method has been compared for the ResGatedGCN in Fig 8a and how that is different from the baselines in Fig 8b. Overall, the time taken for pretraining with FGP seems to significantly increase over other pretraining methods. It would be nice to have a Pareto curve for performance vs time taken for each pretraining method to highlight the efficiency of FGP.\n2. While FGP is claimed to capture the flow based information implicitly, it is unclear how combining FGP with FlowerFormer is able to increase performance. It would suggest that the FGP objective adds information other than the flow to training, can the authors ablate this effect?\n\nThe empirical evaluations of the proposed method shows that the flow based objective can improve encoding performance, but it seems to come at additional cost. The exact tradeoffs between performance and efficiency, specifically due to the flow based objective can be made clearer by the authors.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nThe flow training objective seems to be limited to GNN based encoders, which can be computationally expensive. Perhaps as a more general direction, is there an extension to transformer based encoders which could be more efficient?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-22"}

comments: SUMMARY:
This paper aims to predict the performance of neural architectures effectively and efficiently. Specifically, the flow surrogate is proposed to simulate the ground-truth information flow, which helps to train the architecture encoder effectively. Then, the trained encoder can be used for the performance prediction of most of the neural architectures.

STRENGTHS AND WEAKNESSES:
Weaknesses:

1. There are some typos in this paper. Line 23: 'extensive' computational resources.....

2. The insight of this paper is not solid enough. The proof that the 'flow surrogate' can effectively represents the information flow is lacked. Moreover, after reviewed the details of the generation of 'flow surrogate', there is no constraint that ensures different information surrogates for two architectures with largely different performance, especially when the the inputs are random initialized.

3. There is no clear explanation that why the current flow-based methods are computationally heavy. Given the inference time of GNN, it is hard to understand why it takes such a long computation time.

4. Figure 3 is confusing. In the middle sub-figure of Process 1, why the 'In' information is directly used for the computation of the 'Out'? Even they are not directly connected. Is this means that all historical information will be used when computing the high-level information? Will this cause a requirement of large memory?

5. The sum pooling for multiple incoming information is not reasonable, since the summation can not distinguish the incoming source which is too rough for simulating the real information flow.

QUESTIONS:
Please refer to the Weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the problem of training neural architecture encoders, or models that can predict the downstream task performance given a proposed architecture description. Specifically, this work proposes an representation that captures the \"flow\" of a target model, where the flow refers to information flow w.r.t. the forward and backward passes. The proposed pre-training objective in this work allows an encoder to learn to predict the flow, which previously required specific architecture components to capture. \nThis is accomplished by first assigning a topological ordering over elements in the architecture, then tracing the forward and backward pass of the candidate architecture. This information is aggregated into an embedding that an encoder-decoder pair are trained on with a reconstruction objective. \n\nThe proposed method is evaluated for downstream performance prediction, as well as utility in predictor-aware-NAS. \nAn ablation study over the different constituent components is used to validate the efficacy of each.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The proposed method seems novel with respect to related work. Framing the pretraining objective on the flow-based surrogate itself seems like a more flexible design space compared to modeling it directly with the encoder\n* The paper is well-written and easy to follow\n* Experiments show a consistent (sometimes sizeable) increase in performance \n\nWeaknesses :\n* The values of \\lambda_1 and \\lambda_2 are not mentioned, only their search space. I'm concerned that the performance of the method might be tuned separately for each test setting, or that performance is not robust to the choice of \\lambda\n* It seems to me that a major part of this work is that no \"special components\" are needed to predict the flow, as that information is now captured in the embedding. That unlocks a large design space of less-constrained encoders. However, I don't see any mention of what other encoder designs / architectures were considered. \n\n\nOther comments:\nline 111: variation -> variational\n\nQUESTIONS:\nSee weaknesses section\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes FGP (Flow-based Generative Pre-training) which is a novel pre-training method for neural architecture encoding. The key idea is to train a neural architecture encoder to capture information flow within a neural architecture without requiring specialized and computationally expensive model structures. Specifically, FGP trains an encoder to reconstruct this flow surrogate, enabling it to learn information flow without complex model structures.

STRENGTHS AND WEAKNESSES:
Strengths:

- The paper argues that FGP can make simpler flow-aware encoders and provides clear training guidance. The claim is backed by the experimental results to show that FGP significantly boosts encoder performance with gains of up to 106% when compared to encoders trained solely with supervised learning. 
- FGP is substantially faster and consistently outperforms baseline pre-training methods in a majority of settings. 
 
Overall, the paper presents a well-argued and experimentally supported novel pre-training method for neural architecture encoding that addresses critical challenges of efficiency and effectiveness in the field.

Weaknesses:
- The paper acknowledges that FGP demonstrates empirical effectiveness across various benchmark, further theoretical investigation into their flow surrogates would make this work very strong.
- Even though the authors have touched on the issue that they have limited understanding of the types of information flows that FGP could capture, it would be good to see a thourough discussion around it. 
- Without the above two points, the claims are purely emperical and not strong.

QUESTIONS:
Address the points raised above under weaknesses

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-17