Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper presents an RL-based Neural Architecture Search method that uses elementary operations to construct novel building blocks for sequence datasets.  The paper focuses on searching for novel building blocks, rather than reusing widely used NN operations, such as the normalization layer. During the search step, the authors used synthetic datasets of increasing complexity to expedite the search procedure. This is a remarkable contribution.
overall_review: Strengths:

o	The manuscript is well-written and easy to follow. 

o	Proposing a NAS method that searches for elementary operations (e.g., add, multiply, etc.) to discover novel building blocks. 

o	The message of the paper is clear and well-supported.

Weaknesses:

o	Experiments of the paper are very limited to one sequential dataset. 

o	The novelty of the paper is marginal, similar problems have been previously addressed in [2].

o	Some parts of the paper need more clarification, such as the procedure of graph splitting. 

[2] Real, Esteban, et al. "Automl-zero: Evolving machine learning algorithms from scratch." International Conference on Machine Learning. PMLR, 2020.

review_summary: This paper emphasizes finding novel building blocks. However, the current version lacks diverse experimental results with deep analysis. Thus, I give this paper a weak rejection, but I will change my evaluation if the mentioned concerns are addressed.


actions_required_to_increase_overall_recommendation: As also mentioned in Section 5, I highly recommend authors consider evaluating the non-sequential datasets, such as CIFAR-10. This can significantly improve the paper's quality and impact. Please also consider addressing my concerns mentioned in the previous sections.
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: This paper proposes Elementary Neural Architecture Search (ELENAS), a method that learns to combine elementary mathematical operations to form new building blocks for deep neural networks. These building blocks are represented as computational graphs, which are processed by graph neural networks as part of a reinforcement learning system. In a set of experiments, the authors demonstrate that their method leads to efficient building blocks that achieve strong generalization and transfer well to real-world data. When stacked together, they approach and even outperform state-of-the-art neural networks at several prediction tasks. Using the proposed method, authors present the building blocks ELENA1 and ELENA2 for the domain of sequential data. 
overall_review: Overall the method proposed in this paper is sound to the reviewer.

strength:
- the paper is well-written and the details are discussed properly
- the motivation  and the overall presentation is clear

limitation:
- this paper lacks sufficient empirical results to support the proposed method, which is the most concerning part to the reviewer. Without extensive experiments, the audience does not have sufficient confidence in applying the proposed method. The reviewer would suggest authors conduct more experiments.

review_summary: Overall the method proposed in this paper is sound to the reviewer. However, the experiment evaluation is very limited, which is not convincing enough to tell whether the proposed method, the found building blocks are generally better than others. Given this limited evaluation, the reviewer leans to not support acceptance of this paper. 
actions_required_to_increase_overall_recommendation: I would suggest author conduct more empirical experiments, e.g. more datasets, compare with more blocks, etc.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The authors introduce ELENAS, a search method for constructing neural network building blocks based on RL + GNN search over computation graphs, consisting of nodes representing learnable parameters and edges representing elementary mathematical operations. Constructed computation graphs are then mapped back to architectures for performance estimation on synthetic datasets, serving as the reward function. Empirically, they show that ELENAS can design novel building blocks with strong predictive performance, in this case in the domain of sequential data.
overall_review: Strengths:
- Intuitive search process that is grounded in strong algorithmic foundations (PPO and GNN)
- Searched architectures, ELENA1 and ELENA2, demonstrate empirically strong performance on both synthetic and real-world datasets -- while being much smaller (in terms of # of parameters) than contemporary methods

Weaknesses:
- Limited experiment scope  
    - Only graphs with 32 nodes
    - Only consider sequential data domain
    - Not obvious how to extend synthetic dataset generation to e.g. image classification
- In results, no comparison with Transformer-based models

review_summary: The authors introduce ELENAS, a straightforward and reasonable search method to develop novel NN architecture blocks. They demonstrate strong results in the domain of sequential data. While experimental results hint at their method's usefulness, more comprehensive studies are lacking. Overall, an interesting idea with high impact potential, if more sufficiently studied.
actions_required_to_increase_overall_recommendation: More experiments -- see discussion above.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper presents ELENAS, a method for finding novel, performant neural architectural building blocks at the elementary level by combining mathematical operations. The building blocks are represented as computational graphs and processed by GNNs as part of a reinforcement learning system with the Proximal Policy Optimization (PPO) algorithm. Ultimately, ELENAS can design building blocks and is employed in the domain of sequential data. ELENA1 and ELENA2 building blocks are proposed for the sequential data domain with better performance compared to LSTM. The performance metric is based on the proposed reward function based on the average accuracy, a specific step, the factor by which the reward is weighted for a specific dataset, and reward shaping. The resulting building blocks are compared based on the accuracy of the test/validation set.
overall_review: Positive aspects: 
- Novel approach: The paper introduces an innovative idea in the field of Neural Architecture Search (NAS), focusing on searching for neural architectural building blocks using elementary mathematical operations. 
- Detailed methodology: The authors provide a comprehensive description of their approach, including the process of designing a building block, the utilization of the Proximal Policy Optimization (PPO) algorithm, the search for building blocks using a graph representation, and in the appendix, they have reported the values of all the hyperparameters.

Negative aspects of the paper:

- Reason for using PPO algorithm: The authors do not discuss the advantages of the PPO algorithm over other potential reinforcement learning algorithms.
- Impact of increasing the number of total nodes 𝑚: The authors mention that it's possible to increase 𝑚 to adapt to the search progress dynamically, but they do not provide discussion on the potential implications of increasing 𝑚.
- Incomplete results section: The results section seems to be incomplete, as it only reports outcomes related to one of the four datasets mentioned, and it only compares the proposed method to LSTM. Including results for all datasets and a more comprehensive set of building blocks would provide a better evaluation of the proposed method.
- Lack of validation set results: The results in Table 1 only include test set outcomes, and the validation set results are missing. Including validation set results would offer a more comprehensive understanding of the method's performance.
- Insufficient detail on splitting G into sub-graphs: The authors mention that splitting G into sub-graphs and predicting the edges accordingly improves the search for high-performing building blocks. However, they do not provide enough detail on the process of splitting G into sub-graphs or the criteria used for doing so.
review_summary: The paper presents an interesting idea in the field of NAS, proposing an innovative approach for searching neural architectural building blocks using elementary mathematical operations. The proposed idea has the potential to greatly affect how neural networks are designed and optimized. While the idea is intriguing, there are some important concerns that need to be addressed to achieve a more thorough understanding and assessment of the proposed method. The reported results seem to be incomplete, as Table 1 only compares the approaches to LSTM, and among the four datasets, results are only reported for one. Additionally, considering that in Table 2 there is only a small improvement for the fifth dataset compared to all other approaches and some partially good results for the fourth dataset, I decided to weakly reject the current state of the paper.
actions_required_to_increase_overall_recommendation: Address the minor issues: Ensure that minor issues such as the reference to the correct table and the explanation of the ROC-AUC performance metric are corrected and clarified, which would improve the overall readability and understanding of the paper.

Enhance the results in Table 1: As suggested earlier, either improve Table 1 by including results for all four datasets and comparisons with other building blocks besides LSTM, or provide a clear explanation for focusing on only one dataset in the experiments and the rationale behind comparing the method only with LSTM.
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: This paper proposes ELENAS, which is a method that searches for mathematical operations in a newly defined search space to form building blocks, i.e., combinations of operations applied on input features similar to one operation in a cell-based search space. This method forms these building blocks as computational graphs and utilizes Graph Neural Networks to learn node and graph embeddings. These embeddings are part of the search strategy, here proximal policy optimization. This paper shows that the proposed method is able to find efficient novel building blocks.
overall_review: Strengths:
The experiments are well done and explore an interesting idea of new building blocks, reducing the amount of human-expert knowledge. 
This paper is well-written and motivates the idea good.

Weaknesses:
There are some open questions (see Technical Quality And Correctness), which would when answered improve the overall assessment of the approach of finding novel and efficient building blocks.
My main concern is that there isn’t any standard benchmark from the NAS literature used and the stacking is not mentioned clearly.

Missing citation, which also introduces different building blocks.
Nicholas Roberts, Mikhail Khodak, Tri Dao, Liam Li, Christopher Ré, Ameet Talwalkar:
Rethinking Neural Operations for Diverse Tasks. NeurIPS 2021

review_summary: While the idea of searching for novel building blocks using GNNs with RL is novel,
I believe additional assessments and experiments are needed to improve this paper and clearly show its superiority over other search approaches.

actions_required_to_increase_overall_recommendation: In order to increase the overall recommendation, this paper would need to answer the open questions in Technical Quality And Correctness. 

**After Rebuttal**: I keep my score of 5.
review_title: null

Decision: Accept(Workshop)