Title: MathNAS: If Blocks Have a Role in Mathematical Architecture Design
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural Architecture Search
Abstract: Neural Architecture Search (NAS) has emerged as a favoured method for unearthing effective neural architectures. 
Recent development of large models has intensified the demand for faster search speeds and more accurate search results. 
However, designing large models by NAS is challenging due to the dramatical increase of search space and the associated huge performance evaluation cost. 
Consider a typical modular search space widely used in NAS, in which a neural architecture consists of $m$ block nodes and a block node has $n$ alternative blocks. 
Facing the space containing $n^m$ candidate networks, existing NAS methods attempt to find the best one by searching and evaluating candidate networks directly.
Different from the general strategy that takes architecture search as a whole problem, we propose a novel divide-and-conquer strategy by making use of the modular nature of the search space.
Here, we introduce MathNAS, a general NAS framework based on mathematical programming.  
In MathNAS, the performances of all possible building blocks in the search space are calculated first, and then the performance of a network is directly predicted based on the performances of its building blocks.
Although estimating block performances involves network training, just as what happens for network performance evaluation in existing NAS methods, predicting network performance is completely training-free and thus extremely fast. In contrast to the $n^m$ candidate networks to evaluate in existing NAS methods, which requires training and a formidable computational burden, there are only $m*n$ possible blocks to handle in MathNAS.
Therefore, our approach effectively reduces the complexity of network performance evaluation. 
The superiority of MathNAS is validated on multiple large-scale CV and NLP benchmark datasets. 
Notably on ImageNet-1k, MathNAS achieves 82.5\% top-1 accuracy, 1.2\% and 0.96\% higher than Swin-T and LeViT-256, respectively. 
In addition, when deployed on mobile device, MathNAS achieves real-time search and dynamic network switching within 1s (0.4s on TX2 GPU), surpassing baseline dynamic networks in on-device performance.

MathNAS: If Blocks Have a Role in Mathematical Architecture Design

Abstract

Neural Architecture Search (NAS) has emerged as a favoured method for unearthing effective neural architectures. Recent development of large models has intensified the demand for faster search speeds and more accurate search results. However, designing large models by NAS is challenging due to the dramatical increase of search space and the associated huge performance evaluation cost. Consider a typical modular search space widely used in NAS, in which a neural architecture consists of m block nodes and a block node has n alternative blocks. Facing the space containing n m candidate networks, existing NAS methods attempt to find the best one by searching and evaluating candidate networks directly. Different from the general strategy that takes architecture search as a whole problem, we propose a novel divide-and-conquer strategy by making use of the modular nature of the search space. Here, we introduce MathNAS, a general NAS framework based on mathematical programming. In MathNAS, the performances of the m ∗ n possible building blocks in the search space are calculated first, and then the performance of a network is directly predicted based on the performances of its building blocks. Although estimating block performances involves network training, just as what happens for network performance evaluation in existing NAS methods, predicting network performance is completely training-free and thus extremely fast. In contrast to the n m candidate networks to evaluate in existing NAS methods, which require training and a formidable computational burden, there are only m ∗ n possible blocks to handle in MathNAS. Therefore, our approach effectively reduces the complexity of network performance evaluation. The superiority of MathNAS is validated on multiple large-scale CV and NLP benchmark datasets. Notably on ImageNet-1k, MathNAS achieves 82.5% top-1 accuracy, 1.2% and 0.96% higher than Swin-T and LeViT-256, respectively. In addition, when deployed on mobile devices, MathNAS achieves real-time search and dynamic network switching within 1s (0.4s on TX2 GPU), surpassing baseline dynamic networks in on-device performance. Our code is available at https://github.com/wangqinsi1/MathNAS.

1 Introduction

Neural Architecture Search (NAS) has notably excelled in designing efficient models for Computer Vision (CV) [1, 2, 3, 4] and Natural Language Processing (NLP) [5, 6, 7] tasks. With the growing popularity of the Transformer architecture [8, 9], designers are increasingly drawn to using NAS to

develop powerful large-scale models. Many existing NAS studies focus on designing search spaces for large models and conducting searches within them [10, 11, 12].

However, designing large models by NAS is challenging due to the dramatical increase of search space and the associated huge performance evaluation cost [10, 13]. Consider a widely used modular search space, in which a neural architecture is treated as a topological organization of m different block nodes and each block node has n different block implementations. Obviously, the number of possible networks or neural architectures, n m , grows explosively with n and m . In addition, candidate networks of large models are larger and require more computation for performance evaluation. Therefore, in order to conduct an effective architecture search, a proper search strategy and a suitable performance evaluation method are extremely important.

It is noteworthy that to improve search strategy, recent researches [14, 15] convert NAS to mathematical programming (MP) problems, which substantially decrease the search cost. MP-NAS provides a promising direction for rapidly designing large models. However, current MP-NAS methods exhibit certain architectural constraints. For example, DeepMAD [14] is solely applicable for architecture design within CNN search spaces, and LayerNAS [15] is exclusively suitable for hierarchically ordered search spaces. These limitations impede the application of MP-NAS methods to advanced search spaces, such as SuperTransformer [13, 5]. Besides, alternative strategies for effective performance evaluation of candidate networks are also expected, despite the improvement brought by parameter sharing [16, 17], performance prediction based on learning curves [18, 19] and so on.

In this study, we introduce MathNAS, a novel MP-NAS framework for universal network architecture search. In contrast to previous studies which estimate the performance of a network by solving a whole problem, MathNAS adopts an alternative divide-and-conquer approach. In brief, MathNAS improves the performance evaluation of a candidate network by estimating the performance of each block of the network first, and then combining them to predict the overall performance of the network. Although estimating block performance involves network training, predicting network performance is completely training-free. Therefore, this approach reduces the complexity of network performance evaluation. MathNAS achieves further improvement of the search strategy by transforming NAS to a programming problem, reducing the search complexity to polynomial time.

MathNAS contains three key steps:

Block performance estimation: The performance of each block is estimated by the performance difference between networks having and having not that specific block.

Network performance prediction: The performance of a network is predicted based on the performances of its blocks.

NAS by ILP: Utilizing the block performances, NAS is solved as an Integer Linear Programming (ILP) problem.

We perform experiments on search spaces with various network architectures, including NAS-Bench201 (GNN) [20], MobileNetV3 (CNN) [21], SuperTransformer (Transformer) [5] and NASViT (CNN+Trans) [13]. Our experiments demonstrate that predicting network performance based on its blocks' performances is applicable to different network architectures. In particular, the Spearman coefficient between the actual and the predicted top-1 indices on four different search spaces achieve 0.97, 0.92, 0.93, and 0.95, respectively. At the same time, by using the merit of the divide-and-conquer strategy to transform NAS into an ILP problem, MathNAS can find models superior to state-of-the-art (SOTA) models across different search spaces and tasks. In CV tasks, MathNAS achieves 82.5% top-1 accuracy on ImageNet-1k with 1.5G FLOPs and 15M parameters, outperforming AutoFormer-small (81.7%) and LeViT-256 (81.6%). In NLP tasks, MathNAS reaches a Blue Score of 28.8, on par with Transformer (28.4), but requiring only 1/5 of the FLOPs. In summary, our contributions are as follows:

We propose a general framework for performance evaluation of candidate networks by estimating block performance first and then combining them to predict network performance, which greatly improves the evaluation efficiency.

By virtue of the established mapping between block performance and network performance, we transform NAS into an ILP problem, which reduces the search complexity to polynomial.

We demonstrate MathNAS by considering three key performance indices for network design, i.e. accuracy, latency and energy, and achieve results superior to SOTA models.

2 The Proposed Method

Search Space and Notations. In this paper, we consider a widely used modular search space S = {N 1 , ..., N k , ... } , in which network N k consists of m block nodes B i (1 ≤ i ≤ m ) . For a block node, there is n alternative blocks, i.e. B i = { b i, 1 , ..., b i,j , ..., b i,n } , where block b i,j (1 ≤ i ≤ m, 1 ≤ j ≤ n ) represents the j -th implementation of the i -th block node. A network can therefore be denoted as N = ( b 1 , J 1 , ..., b i, J i , ..., b m, J m ) , with its i -th block node implemented by block b i, J i . Totally, there are m n possible networks in the search space. Here, we focus on three key performance indices for network design, i.e. accuracy, latency and energy consumption. The accuracy, latency and energy consumption of network N are denoted as Acc ( N ) , Lat ( N ) , Eng ( N ) , respectively.

2.1 Problem Formulation: Reduce NAS Search Complexity from O ( n m ) to O ( m ∗ n ) .

The objective of hardware-aware NAS is to find the best neural architecture N ∗ with the highest accuracy under limited latency ˆ L and energy consumption ˆ E in the search space S :

In order to fulfill the above goal, NAS has to search a huge space, evaluate and compare the performance of candidate networks. Early NAS studies usually fully train the candidate networks to obtain their performance ranking, which is prohibitively time-consuming. Subsequent works introduce various acceleration methods. For instance, the candidate networks can avoid training from scratch by sharing weights [16, 17]. Also, the performance of a candidate network can be predicted based on the learning curve obtained from early termination of an incomplete training [18, 19]. Despite of these improvements, a candidate network has to be trained, no matter fully or partially, to obtain a reasonable performance evaluation. And the huge number of candidate networks in the search space poses a formidable efficiency challenge.

However, the modular nature of the search space may provide us with a novel possibility. Although there are n m candidate networks in the search space, they are all constructed by the n ∗ m blocks. If we can evaluate the performances of the blocks by training of a limited number of networks, and if we can combine these block performance indices to obtain a reasonable performance evaluation of networks, we can reduce the complexity of network performance evaluation from O ( n m ) to O ( n ∗ m ) .

Guided by this idea, we reformulate the search of N ∗ from S as a succession of sub-problems. Each sub-problem corresponds to the task of searching the block b i, J ∗ i with the highest accuracy within the block node B i . This approach notably simplifies the original problem:

where b A i,j , b L i,j , b E i,j represent the accuracy, latency, and energy of block b i,j respectively. ⊕ denotes the operation of adding a block to the network. With this approach, each block b i,j is searched only once and the complexity can be effectively reduced to O ( n ∗ m ) . However, due to the mutual influence between blocks, a unified understanding of the relationship between the performance of N and its constituent blocks remains elusive, posing a challenge to the application of this method.

2.2 Divide-and-Conquer: Network-Block-Network

Consider the switching process N ( i, 1) → ( i, J i ) , signifying the selection of the i -th B in network N as it switches from b i, 1 to b i, J i , with other selected blocks remaining unchanged. Thus, any network N = ( b 1 , J 1 , b 2 , J 2 , ..., b m, J m ) can be viewed as the outcome of the base network ˜ N = ( b 1 , 1 , b 2 , 1 , ..., b m, 1 ) undergoing m sequential switching processes. Guided by this idea, we explore two aspects:

E1: Can block performance be directly calculated as with network performance?

Considering the entire search space S , let us denote the collection of networks with the selected block b i, 1 as N Ω ( i, 1) , which comprises n m -1 networks. For any network N ( i, 1) in N Ω ( i, 1) , the switching process N ( i, 1) → ( i,j ) signifies the selection of the i -th B in network N as it switches from b i, 1

(a)

∆

Acc

(

N

(

i,

1)

→

(

i,j

)

)

vs

F

(

N

)

on NAS-Bench-201 (left) and MobileNetV3 (right).

(b)

∆

Lat

(

N

(

i,

1)

→

(

i,j

)

)

(left)/

∆

Eng

(

N

(

i,

1)

→

(

i,j

)

)

(right) vs

F

(

N

)

in NAS-Bench-201 on edgegpu.

1.0

68888868

0.4

7°

6

0.2

02

0.1

0.0

0.0

n.r.

50

75

FLOPs(M)

100

125

150

175

50

75

FLOPs(M)

100

125

150

175

50

75

FLOPs(M)

100

业

125

150

175

300

400500

FLOPs(M)

300

400500

FLOPS(M)

300400500

FLOPs(M)

Latency loss(ms)

Latency loss(ms)

Latency loss(ms)

Energy loss(mJ)

10.0

15 50 2

r

Energy loss(mJ)

.

Energy loss(mJ)

1

。

。

3

50

75

FLOPs(M)

100

175

2

-3

=4

50

75

FLOPs(M)

100

125150

175

12

=3

50

75

FLOPs(M)

100

125150

175

7.5

1.

50

75

FLOPs(M)

100125150

2

175

-4

6

50

75

FLOPs(M)

100

**

125

150

175

10

50

75

FLOPs(M)

100

125150

175

to b i,j , with other selected blocks remaining unchanged. In this switch, two changes occur. The first change sees the selected block in B i switching from b i, 1 to b i,j .The second change arises from an internal adjustment in B i , modifying its interactions with other block spaces in the network. These changes lead to a difference in performance between N ( i, 1) and N ( i,j ) , denote as ∆ Acc ( N ( i, 1) → ( i,j ) ) , ∆ Lat ( N ( i, 1) → ( i,j ) ) , ∆ Eng ( N ( i, 1) → ( i,j ) ) . By averaging performance difference obtained from all n m -1 switching processes, N Ω ( i, 1) → ( i,j ) , we can derive two key parameters:

∆ ϕ ( B ( i, 1) → ( i,j ) ) , the change in inherent capability of B i .

∆Φ( B ( i, 1) → ( i,j ) ) , the change in the interactive capability of B i within S .

Accordingly, we define the performances of b i,j as:

Similarly, b L i,j and b E i,j can be calculated employing the same methodology. The unbiased empirical validation and theoretical proof supporting this method can be found in the Appendix.

E2: How can we predict network performance using block performance?

To accurately derive the performance difference stemming from the switching process, we consider the performance difference that a particular block switch brings to different networks. We performed the identical process N ( i, 1) → ( i, J i ) over a range of networks within N Ω ( i, 1) . The outcome is illustrated in Figure 1, which depicts the relationship between the differences of three performances-latency, energy, and accuracy-and the FLOPs of the network. Our findings reveal that, within the same switching operation, latency and energy differences maintain consistency across different networks, while accuracy differences exhibit an inverse proportionality to the network's FLOPs, F ( N ) .

This finding is logical, as networks with smaller FLOPs have lower computational complexity, rendering them more susceptible to block alterations. Conversely, networks with larger FLOPs exhibit higher computational complexity, making them less sensitive to individual block switching. In the Appendix, we proceed to fit the inverse relationship between accuracy difference and network FLOPs using various formulas. The results suggest that the accuracy difference is approximately inversely related to the network's FLOPs. Consequently, we can posit ∆ Acc ( N ( i, 1) → ( i, J i ) ) = α ∗ 1 / F ( N ) , where α represents a specific block's coefficient.

Based on these observations, we can deduce that for any network N ( i, 1) within the set N Ω ( i, 1) , the performance difference resulting from the switching process can be approximated as follows:

(a)Lat on NB201

(b)Energy on NB201

(c)Acc on MBV3

(d)Acc on NB201

PredictLatency(ms)

8000

7500

8500

Predict Energy(ms)

35000

37500

32500

Predict topk1

79.0

79.5

Predicttopk1

50 4540 35 30 5 2 15

7000

30000

78.5

本

6500

6000

27500

25000

78.0

5500

22500

77.5

5000

4000

4500

4000

Actual Latency(ms)

5000

60007000

8000

150950000

20000

17500

200002500030000

Actual Energy(ms)

35000

77.0

77.077.578.078.5

Actual topk1

79.079.5

10

10

Actual topk1

20

30

40

50

0.11

1.3

14

0.055

1.2

0.10

13

3SW

0.050

3SW

1.1

0.09

MSE

12

0.045

11

1.0

0.040

0.07

10

0.035

0.9

0.06

9

0

20

Sample Ratio(%)

40

60

08

100

0

20

Sample Ratio(%)

40

60

80

100

0

20

Sample Ratio(%)

40

60

80

100

20

Sample Ratio(%)

40

60

80

100

By integrating Equation 3 and 4, we can estimate the performance of N = ( b 1 , J 1 , b 2 , J 2 , ..., b m, J m ) :

Proof-of-Concept Experiment. In our examination of the predictive efficacy of Equation 5 across diverse network types, as depicted in Figure 2, we observe that it accurately forecasts three performances for both sequentially connected networks and graph networks and both weight-independent and weight-shared networks, all without necessitating network training. To the best of our knowledge, our method is the first work to precisely estimate network performances using a linear formula, and notably, it is theoretically applicable to all architectures.

2.3 Simplification: Single-sampling Strategy Instead of Full-sampling

Despite the outstanding predictive performance displayed by Equation 5, its computation of loss averages proves to be costly. In this section, by employing a single-sample sampling strategy, we effectively reduce the time complexity from O ( n m ) to O ( n ∗ m ) , enhancing efficiency without compromising precision.

Partial Network Sampling Strategies. We

begin by investigating the sample size requirements for Equation 5. The second row of Figure 2 demonstrates rapid convergence of Equation 5 with a notably small sample count for performance prediction. Specifically, Figure 2(c)(d) reveals that a mere 5% random sampling of networks is sufficient for the prediction to converge towards optimal performance. This underscores the impressive efficacy of Equation 5, which exhibits rapid convergence even with a limited number of random samples.

Single Network Sampling Strategy. Building upon Equation 4, we can select an averageFLOPs network, denoted as N avg , in the search

Algorithm 1 Math Neural Architecture Search

Stage1: Determine the Average-FLOPs Network for i=1,2,...,m do

Calculate the average of the FLOPs of all blocks at the i -th note, F ( b i ) = ( F ( b i, 1 )+ ... + F ( b i,n )) /n ; Select b i, J i whose FLOPs is closest to F ( b i ) .

end

Define average-FLOPs net N avg = { b 1 , J 1 , .., b m, J m } .

Stage2: Calculate Block Performances

for

i=1,2,...,m

do

for

do

Switch the i -th block in N avg from b i, 1 to b i,j ; Calculate the performance difference of the net- work brought about by switching and use it as the performance of the block b A i,j , b L i,j , b E i,j .

end

end

Stage3: Prediction and Architecture Search

Calculate three characteristics of the base net ˜ N = { b 1 , 1 , . . . , b m, 1 } as Acc ( ˜ N ) , Lat ( ˜ N ) , Eng ( ˜ N ) ; For network N = { b 1 , J 1 , b 2 , J 2 , ..., b m, J m } , its accu- racy, latency and energy can be estimated by Equa. 5. Set required accuracy/latency/energy limit, and solve the corresponding ILP problem to obtain the architecture.

pensty

(a)Acc on MBv3

(b)Acc on NB201

(c)Acc on SuperTran

(d)Acc on SuperViT

(e)Lat on MBv3

(f)Lat on NB201

(g)Ener on NB201

(h)Lat on SuperTran

Spearman: 0.9714

Spearman: 0.9187

Spearman:0.9530

Predict topk1(%)

79.5

3.5

3.0

Predict topk1

0.008

0.014

0.012

0.010

Predict BLUE

27.0

26.5

26.0

Predict topk1(%)

11

82.25

82.50

3.0

2.5

7.5-

1.0

0.004

0.3

81.00

"4.

1.0

0.5

0.002

0.1

80.75

0.5

76.0

76

：

77

78

79

10

20

30

40

50

24.0

24

25

26

27

80.50

80.5

*#

81.0

81.5

82.0

82.5

Actual topk1(%)

Actual topk1

Actual BLUE

Actual topk1(%)

Predict Latency(ms)

8000

7000

Spearman:0.9842

Predict Latency(s)

Spearman:0.9552

0.8

37.5

Spearman:0.9256

0.035

Predict Latency(ms)

8000

7000

Spearman: 0.9901

0.030

6

Density

6000

5000

4000

5000

3000

0.2

0.010

4000

2

1

0.1

0.005

2000

2000

Actual Latency(ms)

4000

6000

8000

4

Actual Latency(s)

6

7

8

17.5

20

Actual Energy()

25

30

35

0000

Actual Latency(ms)

4000

6000

8000

space with FLOPs approximating the mean value, ensuring F ( N Ω ( i,j ) ) / F ( N avg ( i,j ) ) ≈ 1 . This leads to: b A i,j ≈ ∆ Acc ( N avg ( i, 1) → ( i,j ) ) , b L i,j ≈ ∆ Lat ( N avg ( i, 1) → ( i,j ) ) , b E i,j ≈ ∆ Eng ( N avg ( i, 1) → ( i,j ) ) By incorporating N avg into Equation 5, we gain the ability to calculate any network performance. Thus, we only need to verify the performance of N avg ( i, 1) and N avg ( i,j ) for b i,j , resulting in an O ( n ∗ m ) search time complexity. The complete MathNAS algorithm is presented in Algorithm 1, which, in theory, can achieve accurate prediction on any network structure with polynomial search complexity.

Validation experiment of single-sample strategy effectiveness. We assess the efficacy of Algorithm 1 across CNN, GCN, Transformer, and CNN+Transformer search spaces, with the outcomes displayed in Figure 3. It is evident that the Spearman correlation coefficient between predicted and actual values exceeds 0.9 for all networks. Remarkably, accuracies of 0.95 and 0.97 are attained on ViT and CNN architectures, respectively, emphasizing the algorithm's robustness.

2.4 MathNAS: Converting Architecture Search to ILP.

We denote b F i,j as the FLOPs of block b i,j , and b B i,j ∈ { 0 , 1 } as the indicator of whether block b i,j is used in a network. If block b i,j is selected as the implementation of block node B i in a network, b B i,j = 1 , otherwise 0. The problem that NAS needs to solve can be formulated as:

The objective is to obtain the maximum accuracy network under two constraints. First, the latency and energy cannot exceed the limit. Second, for any block node, only one block is used. As Equation 6 is a fractional objective function, it can be transformed into an ILP problem by variable substitution.

3 Performance Evaluation

Experiments were conducted at three levels to evaluate the efficacy of MathNAS. Firstly, we validated the effectiveness of MathNAS on CV tasks by conducting searches across three different search spaces. Then we employed MathNAS to design efficient architectures for NLP tasks, showcasing its remarkable generalization capabilities. Finally, we leveraged MathNAS to perform real-time searches on edge devices, considering hardware resources, and achieved exceptional on-device performance.

3.1 Experimental Setup

Search space. For CV tasks, we validate our method on three search spaces: (1) NAS-Bench-201 [20]: a search space encompasses 15,625 architectures in a DARTS-like configuration. (2) SuperViT [13]: a hybrid search space that combines ViT and CNN, containing approximately 4 × 10 10 network architectures. (3) MobileNetV3 [21]: a lightweight network search space comprising about 10 10 network architectures. For NLP tasks, we validate our approach on the SuperTransformer search space [5], which includes 10 15 networks within a lightweight Transformer framework.

Search and training settings. For NAS-Bench-201 and MobileNetV3, we adopt the training methodology employed in [16] and [22] to train the base net for 100 epochs. Subsequently, we conducted MathNAS search on the base net. As for SuperTransformer and SuperViT, we adhere to the training algorithm proposed by [13] to train the base net for 100 epochs before conducting MathNAS search. The settings of hyperparameters in the training are consistent with the original paper. We employ the Gurobipy solver to address the ILP problem. In the SuperViT and SuperTransformer search spaces, we impose a search time limit of 10 seconds to expedite the process. For the other search spaces, we do not enforce any time constraints.

The search cost of MathNAS consists of two stages: offline network pre-training that is conducted only once and online real-time search. During the offline network pre-training, MathNAS evaluates block performance once. During online searching, MathNAS is capable of multiple real-time searches based on the current hardware resource constraints. To negate the influence of variations in GPU models and versions on the pre-training time, and to facilitate comparisons by future researchers, we have adopted pre-trained networks provided by existing works. All mentions of search cost in the paper refer solely to the real-time search time on edge devices. We provide a detailed description of the search space, more experimental results, and visualizations of the searched architectures in the Appendix.

3.2 MathNAS for Designing Effective CV Networks

MathNAS for NAS-Bench-201. To assess the effectiveness of our method in striking a balance between accuracy and hardware efficiency, we compare networks searched by MathNAS under hardware efficiency constraints to those searched by BRP-NAS [30], which utilizes GNN predictors to estimate network performance. As illustrated in Figure 4, MathNAS consistently locates networks that approach Pareto optimality in the majority of cases, whereas the employment of GNN predictors leads to suboptimal model choices. An extensive comparison between the searched architectures and SOTA models is provided in the Appendix for further insight.

73

73

Accuracy(%)

72

70

Accuracy(%)

72

70

Accuracy(%)

72

71

70

Accuracy(%)

72

71

68

89

69

70

66

oracle

MathNAS

66

oracle

MathNAS

68

oracle

MathNAS

69

oracle

MathNAS

BRP-NAS

BRP-NAS

67

BRP-NAS

68

BRP-NAS

64

500

1000

1500

2000

64

400060008000 1000012000 14000 16000

66

5000

6000

7000

8000

LatencyonFPGA(ms)

EnergyonFPGA(mJ)

Latencyonedgegpu(ms)

Energyonedgegpu(mJ)

FLOPs(M)

Model

Top-1(%)

Top-5(%)

F(M)

Param(M)

ResNet-18[23]

77.7

93.2

1800

11.7

ReXNet[24]

77.9

93.9

400

4.8

MathNAS-T1

78.4

93.5

200

8.9

LeViT-128[25]

78.6

94.0

406

9.2

EfficientNet-B1[26]

79.1

94.4

700

7.8

ReXNet[24]

79.5

94.7

660

7.6

MathNAS-T2

79.6

94.3

325

9.3

ResNetY-4G[27]

80.0

94.8

4000

21

LeViT-192[25]

80.0

94.7

658

10.9

EfficientNet-B2[26]

80.1

94.9

1000

9.2

ReXNet[24]

80.3

95.2

900

9.7

ResNet-50[23]

80.6

95.1

4100

25.6

MathNAS-T3

81.3

95.1

671

13.6

Swin-T[28]

81.3

95.5

4500

29

EfficientNet-B3[26]

81.6

95.7

1800

12

LeViT-256[25]

81.6

95.4

1120

18.9

ReXNet[24]

81.6

95.7

1500

16

AutoFormer-small[29]

81.7

95.7

5100

22.9

MathNAS-T4

82.0

95.7

1101

14.4

AutoFormer-base[29]

82.4

95.7

11000

54

LeViT-384[25]

82.6

96.0

2353

39.1

MathNAS-T5

82.5

95.8

1476

14.8

8

82.5

82.0

Accuracy(%)

82

81.3

81

I-dol

79

LeViT

ImageNet 7

+

FBNetV3

78

BigNAS

EfficientNet

77

Mobile-Former

OnceForAll

★

ReXNet

76

DeepMAD

Ours(MathNAS-T)

75

0

250

500

750

1000

1250

1500

1750

2000

MathNAS for ViT. To assess the performance of MathNAS in designing larger models, we utilize it to create effective ViT models for ImageNet-1K classification. Figure 5 demonstrates that MathNAS surpasses or matches the performance of existing SOTA models. For instance, MathNAS-T5 achieves an accuracy of 82.5%, which is comparable to LeViT-384 [25] and Autoformer-base [29], while consuming only about 50% and 15% of their respective FLOPs. Similarly, MathNAS-T3 achieves comparable accuracy to RexNet [24] but with approximately half the FLOPs. MathNAS also exhibits exceptional performance in the realm of small networks. Particularly, MathNAS-T1 achieves a top-1 accuracy of 78.4%, surpassing ResNet-18 [23] and ReXNet [24] by 0.7% and 0.5% respectively.

MathNAS for Mobile CNNs. We employ MathNAS to design mobile CNN models for further investigation, conducting our search within the MobileNetV3 search space. As demonstrated in Table 1, MathNAS-MB4 achieves a top-1 accuracy of 79.2%, which is on par with EfficientNet-B1 (79.1%). It is important to note that EfficientNet-B1 is derived through a brute-force grid search, necessitating approximately 72,000 GPU hours [4]. Despite this, MathNAS-MB4 offers comparable performance to EfficientNet-B1 while only requiring 0.8 seconds to solve an ILP problem on the GPU and search for a suitable network. MathNAS also excels in the context of smaller networks. Notably, MathNAS-MB1 requires only 257M FLOPs to achieve a top-1 accuracy of 75.9%, surpassing the performance of FBNet-b [31], AtomNAS-A [32], and OFA [22], all of which demand higher computational resources.

3.3 MathNAS for Designing Effective NLP Networks

We perform a comparative evaluation of MathNAS against SOTA NLP models on the WMT'14 En-De task to gauge its effectiveness. Table 2 reveals that MathNAS surpasses all baseline models in terms of BLEU score while also achieving FLOPs reduction across three distinct devices. Specifically, on Intel Xeon CPUs, MathNAS with full precision attains a remarkable 74% reduction in FLOPs compared to Transformer [8] and a 23% reduction compared to HAT [5], while registering improved BLEU scores by 0.4 and 0.3, respectively. Additionally, MathNAS excels in designing lightweight

NLP models. On Nvidia TITAN Xp GPUs under latency constraints, MathNAS yields FLOPs comparable to HAT [5], but with a 0.3 higher BLEU score. A noteworthy aspect is that the network search process facilitated by MathNAS requires only 10 seconds, considerably reducing search time. As a result, employing MathNAS leads to a reduction of over 99% in CO 2 emissions compared to baseline models, underscoring its positive environmental impact.

3.4 MathNAS for Designing Dynamic Networks.

Deployed on edge devices (Raspberry Pi 4b, Jetson TX2 CPU, TX2 GPU), MathNAS allows dynamic network switching suited to device conditions. Within the MobileNetV3 search space, we account for memory limitations by calculating performance indices for each block, subsequently deploying five selected Pareto-optimal blocks balancing accuracy and latency in each block node. During runtime, latency is continuously monitored. Should it surpass a preset threshold, MathNAS immediately updates the blocks' latency. Then the device solves the ILP problem to identify the optimal network architecture, comparing and switching blocks with the searched network as required.

In comparison with SOTA dynamic network models, MathNAS demonstrates superior performance as outlined in Table 3 and Figure 6. Impressively, MathNAS solves the ILP problem on-device in a mere 0.4 seconds on the TX2 GPU, enabling real-time search. This notably enhances the number of executable networks on the device, outdoing SlimmableNet [39] and USlimmableNet [40] by factors of 781 and 116 respectively. Additionally, through a block-based approach, MathNAS enables efficient network alterations by replacing only essential blocks. When compared to Dynamic-OFA [41], which shares similar performance, MathNAS significantly reduces the switching time by 80%. The Appendix details the use of Pareto-optimal blocks and related network experiment results.

80

RaspberryPi

80

JetsonTX2CPU

80

JetsonTX2GPU

D-OFA

Accuracy(%)

Top-1 Accuracy(%)

Accuracy(%)

75

75

75

x4

70

70

70

AutoSlim

×7

Top-1

65

60

Slimmable MBv2

DynamicOFA

US-Slim MBv2

AutoSlim MnasNet

65

Slimmable MBv2

US-Slim MBv2

AutoSlim MnasNet

DynamicOFA

-d

65

Slimmable MBv2

US-Slim MBv2

AutoSlim MnasNet

DynamicOFA

sno

1.1x104

1.2×106

x3125

55

500

1000

Latency(ms)

1500

Ours(MathNAS-MB)

2000

2500

60

500

750

Latency(ms)

1000

Ours(MathNAS-MB)

1250

1500

60

50

75

Latency(ms)

100

Ours(MathNAS-MB)

125

150

Networks Search Time

(log(hours)

100

103

Subnets

Number

4 Conclusion

This paper introduces MathNAS, the first architecturally-general MP-NAS. By virtue of the modular nature of the search space, we introduce block performance and establish the mapping from block performance and network performance, which enables subsequent transformation of NAS to an ILP problem. This novel strategy reduces network search complexity from exponential to polynomial levels while maintaining excellent network performance. We propose three transformation formulas to depict this process and support them with theoretical proofs and extensive experiments. MathNAS achieves SOTA performance on various large-scale CV and NLP benchmark datasets. When deployed on mobile devices, MathNAS enables real-time search and dynamic networks, surpassing baseline dynamic networks in on-device performance. Capable of conducting rapid searches on any architecture, MathNAS offers an appealing strategy for expediting the design process of large models, providing a clearer and more effective solution.

5 Acknowledgment

We thank the anonymous reviewers for their constructive comments. This work was partially supported by National Key R&D Program of China (2022YFB2902302). Sihai Zhang gratefully acknowledges the support of Innovative Research (CXCCTEC2200001).

References

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition. In Proceedings of the IEEE/CVF international conference on computer vision , pages 12270-12280, 2021.

Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Bichen Wu, Zijian He, Zhen Wei, Kan Chen, Yuandong Tian, Matthew Yu, Peter Vajda, et al. Fbnetv3: Joint architecture-recipe search using predictor pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16276-16285, 2021.

Bichen Wu, Chaojian Li, Hang Zhang, Xiaoliang Dai, Peizhao Zhang, Matthew Yu, Jialiang Wang, Yingyan Lin, and Peter Vajda. Fbnetv5: Neural architecture search for multiple tasks in one run. arXiv preprint arXiv:2111.10007 , 2021.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. Hat: Hardware-aware transformers for efficient natural language processing. arXiv preprint arXiv:2005.14187 , 2020.

Nikita Klyuchnikov, Ilya Trofimov, Ekaterina Artemova, Mikhail Salnikov, Maxim Fedorov, Alexander Filippov, and Evgeny Burnaev. Nas-bench-nlp: neural architecture search benchmark for natural language processing. IEEE Access , 10:45736-45747, 2022.

Hung-yi Lee, Shang-Wen Li, and Ngoc Thang Vu. Meta learning for natural language processing: A survey. arXiv preprint arXiv:2205.01500 , 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR) , 54(10s):1-41, 2022.

Mojan Javaheripi, Gustavo de Rosa, Subhabrata Mukherjee, Shital Shah, Tomasz Religa, Caio Cesar Teodoro Mendes, Sebastien Bubeck, Farinaz Koushanfar, and Debadeepta Dey. Litetransformersearch: Training-free neural architecture search for efficient language models. Advances in Neural Information Processing Systems , 35:24254-24267, 2022.

Gresa Shala, Thomas Elsken, Frank Hutter, and Josif Grabocka. Transfer nas with metalearned bayesian surrogates. In Sixth Workshop on Meta-Learning at the Conference on Neural Information Processing Systems .

Zihao Sun, Yu Hu, Shun Lu, Longxing Yang, Jilin Mei, Yinhe Han, and Xiaowei Li. Agnas: Attention-guided micro and macro-architecture search. In International Conference on Machine Learning , pages 20777-20789. PMLR, 2022.

Chengyue Gong, Dilin Wang, Meng Li, Xinlei Chen, Zhicheng Yan, Yuandong Tian, Vikas Chandra, et al. Nasvit: Neural architecture search for efficient vision transformers with gradient conflict aware supernet training. In International Conference on Learning Representations , 2021.

Xuan Shen, Yaohua Wang, Ming Lin, Yilun Huang, Hao Tang, Xiuyu Sun, and Yanzhi Wang. Deepmad: Mathematical architecture design for deep convolutional neural network. arXiv preprint arXiv:2303.02165 , 2023.

Yicheng Fan, Dana Alon, Jingyue Shen, Daiyi Peng, Keshav Kumar, Yun Long, Xin Wang, Fotis Iliopoulos, Da-Cheng Juan, and Erik Vee. Layernas: Neural architecture search in polynomial complexity. arXiv preprint arXiv:2304.11517 , 2023.

Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean. Efficient neural architecture search via parameters sharing. In International conference on machine learning , pages 40954104. PMLR, 2018.

Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint arXiv:1708.05344 , 2017.

Bowen Baker, Otkrist Gupta, Ramesh Raskar, and Nikhil Naik. Accelerating neural architecture search using performance prediction. arXiv preprint arXiv:1705.10823 , 2017.

Boyang Deng, Junjie Yan, and Dahua Lin. Peephole: Predicting network performance before training. arXiv preprint arXiv:1712.03351 , 2017.

Xuanyi Dong and Yi Yang. Nas-bench-201: Extending the scope of reproducible neural architecture search. arXiv preprint arXiv:2001.00326 , 2020.

Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1314-1324, 2019.

Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791 , 2019.

Sasha Targ, Diogo Almeida, and Kevin Lyman. Resnet in resnet: Generalizing residual architectures. arXiv preprint arXiv:1603.08029 , 2016.

Dongyoon Han, Sangdoo Yun, Byeongho Heo, and YoungJoon Yoo. Rethinking channel dimensions for efficient model design. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition , pages 732-741, 2021.

Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1225912269, 2021.

Mingxing Tan and QuocV. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, May 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, Dec 2015.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , Mar 2022.

Minghao Chen, Houwen Peng, Jianlong Fu, and Haibin Ling. Autoformer: Searching transformers for visual recognition, Jul 2021.

Lukasz Dudziak, Thomas Chau, Mohamed Abdelfattah, Royson Lee, Hyeji Kim, and Nicholas Lane. Brp-nas: Prediction-based nas using gcns. Advances in Neural Information Processing Systems , 33:10480-10490, 2020.

Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10734-10742, 2019.

Jieru Mei, Yingwei Li, Xiaochen Lian, Xiaojie Jin, Linjie Yang, Alan Yuille, and Jianchao Yang. Atomnas: Fine-grained end-to-end neural architecture search. arXiv preprint arXiv:1912.09640 , 2019.

Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2820-2828, 2019.

Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.00332 , 2018.

Xiangxiang Chu, Bo Zhang, and Ruijun Xu. Fairnas: Rethinking evaluation fairness of weight sharing neural architecture search. In Proceedings of the IEEE/CVF International Conference on computer vision , pages 12239-12248, 2021.

Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16 , pages 544-560. Springer, 2020.

Jiahui Yu, Pengchong Jin, Hanxiao Liu, Gabriel Bender, Pieter-Jan Kindermans, Mingxing Tan, Thomas Huang, Xiaodan Song, Ruoming Pang, and Quoc Le. Bignas: Scaling up neural architecture search with big single-stage models. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VII 16 , pages 702-717. Springer, 2020.

David So, Quoc Le, and Chen Liang. The evolved transformer. In International conference on machine learning , pages 5877-5886. PMLR, 2019.

Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. Dynamic slimmable network. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 8607-8617, 2021.

Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1803-1811, 2019.

Wei Lou, Lei Xun, Amin Sabet, Jia Bi, Jonathon Hare, and Geoff V Merrett. Dynamic-ofa: Runtime dnn architecture switching for performance scaling on heterogeneous embedded platforms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3110-3118, 2021.

Jiahui Yu and Thomas Huang. Autoslim: Towards one-shot architecture search for channel numbers. arXiv preprint arXiv:1903.11728 , 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MathNAS, a novel approach to Neural Architecture Search (NAS) that utilizes mathematical programming techniques. MathNAS aims to improve the efficiency and accuracy of architecture search by dividing the search space into smaller building blocks and predicting network performance based on the performance of these blocks. The authors validate the effectiveness of MathNAS on computer vision (CV) and natural language processing (NLP) tasks, showcasing its superior performance compared to state-of-the-art models.\n\nSTRENGTHS:\n1. Efficient Performance Evaluation: MathNAS proposes a general framework for evaluating candidate networks by estimating block performance first and then combining them to predict network performance. This approach greatly improves evaluation efficiency .\n2. Reduced Search Complexity: By establishing a mapping between block performance and network performance, MathNAS transforms NAS into an Integer Linear Programming (ILP) problem, reducing the search complexity to polynomial time .\n3. Superior Results: The authors demonstrate MathNAS's capabilities by considering key performance indices such as accuracy, latency, and energy. The results show that MathNAS outperforms state-of-the-art models in terms of these metrics .\n\nWEAKNESSES:\n1. Lack of Detailed Implementation: The paper provides a high-level overview of MathNAS but lacks detailed implementation information. More specific details about the mathematical programming techniques used and the specific algorithms employed would have been beneficial.\n\n2. Section 2.2 of the paper presents some significant ambiguities that complicate understanding of the proposed method:\n\na. The paper does not clearly explain the definitions of \"inherent capability\" and \"interactive capability\". This makes it challenging to understand their intended meanings and how they function within the given neural network context.\n\nb. There is a lack of clarity on how the changes in these capabilities, represented as $\\Delta \\phi\\left(\\mathcal{B}{(i, 1) \\rightarrow(i, j)}\\right)$ and $\\Delta \\Phi\\left(\\mathcal{B}{(i, 1) \\rightarrow(i, j)}\\right)$, are evaluated. The methodology for these calculations needs to be explained more explicitly.\n\nc. Equation 3 lacks clarity due to the aforementioned uncertainties. Without clear definitions and calculation methods for the terms on the right-hand side, it is impossible to meaningfully interpret or apply this equation.\n\nd. Additionally, the processes of module swapping and internal adjustment are not well-defined. More details on how these operations are performed, both in terms of selecting which modules to swap and how the internal adjustments are made post-swap, are needed for a comprehensive understanding and replication of the method. Does every module have the sample input/output shape?\n\n\n3. Limited Discussion on Generalization: Although the paper mentions MathNAS's remarkable generalization capabilities in designing efficient architectures for NLP tasks, there is limited discussion on the factors contributing to this generalization and how it compares to other NAS methods.\n\nQUESTIONS:\n1. Can you clarify the approach or strategy employed to divide the networks into individual modules or blocks?\n\n2. Based on observations from Figure 1b, the Floating Point Operations Per Second (FLOPs) appear to have minimal impact on latency. Could you provide an explanation or insight into why this phenomenon occurs?\n\n3. Have you considered implementing a Graph Neural Network (GNN) model designed to predict performance variances that may arise as a result of interchanging different modules within the network? What outcomes or implications might this approach yield?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a way to perform neural architecture search (NAS) by solving an integer program to identify the constituents of a block-modular neural net, having pre-computed estimations of performance for the separate blocks by a novel method. The resulting \"MathNAS\" approach is demonstrated in various experiments to be able to yield competitive neural network designs compared to other state of the art NAS methods.\n\n-- update: I have read and acknowledged all other reviews and the authors rebuttals, see discussion. --\n\nSTRENGTHS:\nThe novel performance estimation method appears to be well-motivated and leads to the possibility of applying mixed-integer programming to compute an optimal architecture (w.r.t. the utilized performance estimates). The numerical results indicate that this works well in practice, giving somewhat better or comparable results with notable savings in training time. This new NAS pipeline also shows promise for future refinements, e.g., regarding the MIP problems used for the search once performance estimates have been pre-computed, given the inherent flexibility of MIP to model a variety of different aspects.\n\nWEAKNESSES:\nThe paper must be thoroughly proofread for English language and grammar/typos. The supplementary document is as long as the main paper, but provides mostly only some small additional information; some things I hoped to find in the supplement were however not included, e.g., an explanation for the MIP objective function and a description or at least literature reference for how to turn the fractional objective into a linear one. Math typesetting especially of optimization programs can be improved. Also, I got the impression that the authors are trying a bit too hard to \"sell\" their ideas; maybe tone down the language a bit (there's very many usages of words like \"impressive\" or \"remarkable\" and similar phrasings praising the paper's results). Finally, there are several parts that remained somewhat unclear to me (see \"Questions\"), so the presentation can also be improved in terms of clarity/rigor.\n\nQUESTIONS:\n- what exactly do you mean by \"polynomial-time search complexity\"? It is quite confusing to read something like \"...can be transformed into an integer programming problem, which further reduces the search complexity to polynomial time\" -- solving IPs is generally not possible in polynomial time!   I suspect the authors mean that they need to pre-compute polynomially many values (for their performance estimation scheme), but this does not really become clear.\n- In the same vein, it is mentioned that some neural network training is necessary, but it is never clearly stated what and when. Again, I guess training is required to pre-compute the block performance scores somehow, but the details of this are too vague; this certainly needs to be clarified.\n- Could you describe a bit more how flexible the block/modular architecture is? (i.e., mention earlier that blocks can be construed of essentially any (finite) number of sub-network; this is never clearly stated and only becomes more clearly apparent in the supplementary document where details on the search spaces in the different experiments are given).\n- You claim at several points that the integer programs were solved with Gurobi (that is the solver name; Gurobipy is merely its Python interface...) on a GPU.  I am quite sure that Gurobi does not run on GPUs. MIP solvers currently do not benefit from GPU vs. CPU computations; Gurobi even explain this on their website. So, please clarify and correct/elaborate in the paper! (Also, state the Gurobi version you used.)\n- Please clarify what exactly is being sampled (and to what purpose exactly) in Section 2.3\n- Please justify the IP objective function and clarify/give a reference for how the fractional IP can be turned into a linear IP.\n\nLIMITATIONS:\nLimitations have by and large been addressed sufficiently.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MathNAS, a mathematical programming NAS algorithm. MathNAS maps network performance to block performance, enabling the prediction of accuracy, latency, and energy consumption of large networks based on their constituent modules. This algorithm reduces the search space from n^m to m*n, allowing the NAS problem to be solved as an Integer Linear Programming problem.\n\nSTRENGTHS:\nOverall, this paper is well-written and organized, making it easy to follow along. The authors did a great job explaining their methods in detail, complete with loads of equations and clear explanations. Plus, the proposed algorithm has undergone numerous experiments under different models and application fields, which consistently showcase its effectiveness.\n\nWEAKNESSES:\n1)\tIn all the experiments, MathNAS takes significantly less \"search time\" than the other methods, especially some methods need thousands of hours while MathNAS only requires several seconds, which is very impressive. Is the time for all the other methods include the model training time (either training time of supernet or the subnet)? If yes, for the fair comparison, for the MathNAS, should the time that needed to train the base model be added to this time as well? Also, as algorithm 1 shows that there are three steps for MathNAS to determine the best architecture, should the time that calculate each block's performance also be added to the total \"search time\"?\n\n2)\tThe order of the tables and the description of the experimental results in the paper do not follow the same order. For example, the text follows GNN -> CNNs (Table 2) -> ViT -> NLP (Table 1) -> Dynamic Networks (Table 3), which leads is hard to follow.\n\n3)\tA typo in paragraph \"MathNAS for Mobile CNNs ...\", the top-1 accuracy of MathNAS-MB1 is 75.9% as shown in the table, not 76.4%.\n\nQUESTIONS:\nIn all the experiments, MathNAS takes significantly less \"search time\" than the other methods, especially some methods need thousands of hours while MathNAS only requires several seconds, which is very impressive. Is the time for all the other methods include the model training time (either training time of supernet or the subnet)? If yes, for the fair comparison, for the MathNAS, should the time that needed to train the base model be added to this time as well? Also, as algorithm 1 shows that there are three steps for MathNAS to determine the best architecture, should the time that calculate each block's performance also be added to the total \"search time\"?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MathNAS, a blockwise NAS framework. It begins by estimating the performance of the building blocks within the search space and then predicts the overall performance of a network based on the performance of its individual building blocks. This approach effectively reduces the complexity of network search from O(n^m) to O(nm). Additionally, a block performance evaluation scheme, utilizing average FLOPs, is proposed to further reduce the time complexity from O(n^m) to O(nm). Lastly, the network search process is formulated as an Integer Linear Programming (ILP) problem.\n\nSTRENGTHS:\n* Clarity: The paper is effectively written, employing appropriate mathematical notations throughout.\n\n* Quality: The experiments conducted in the paper are of high quality, as they include a comprehensive comparison with well-known networks in various domains, such as tabular search space, mobile CNN, ViT, and NLP.\n\n* Originality and significance: While the paper presents valuable contributions, there are some concerns that should be addressed. Please refer to the weaknesses section outlined below.\n\nWEAKNESSES:\nThe paper does not cite a few highly relevant papers on blockwise NAS.\n* LANA: Latency Aware Network Acceleration. https://arxiv.org/pdf/2107.10624.pdf\n* Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces. https://arxiv.org/pdf/2012.08859.pdf\n* BLOX: Macro Neural Architecture Search Benchmark and Algorithms. https://arxiv.org/pdf/2210.07271.pdf\n\nThe authors should thoroughly discuss the distinctions between their proposed approach and the previously mentioned works, while also considering the possibility of conducting experiments to compare them. Specifically, LANA adopts a constrained ILP approach that relies on block metrics derived from factors such as delta-accuracy and delta-latency. The proposed approach bears resemblance to LANA.\n\nWhile the authors have made an effort to explain the validity of their approach, as depicted in Figure 1 and Section C of the appendix, it remains unconvincing that Equation 2 holds true under all circumstances. Additionally, Equations 3 and 4 are similar to those presented in LANA.\n\nQUESTIONS:\n* Figure 1 in the appendix requires clarification. It appears to suggest the presence of three levels of blocks, with each level accommodating parallel blocks. Could you please clarify whether the blocks are connected sequentially or in parallel? Additionally, how many levels of blocks are there?\n\n* In Algorithm 1, there is an inconsistency with the existence of b_{i,0}, which does not align with the range specified for i (i={1, m}). Is this a typographical error?\n\n* Regarding the results on NB-201, it would be beneficial to know if multiple runs were performed to obtain averages using different seeds. Similarly, for the Mobile CNN, ViT, and NLP networks, were multiple runs conducted to calculate the means and variances?\n\nLIMITATIONS:\nThe paper does not explicitly mention the limitations and potential negative social impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a divide-and-conquer neural architecture search methodology that estimates DNN performance based on the individual performance of each block within a DNN. Accuracy is estimated based on the perturbed accuracy of replacing a block within the NN, and latency/energy are also used within a linear model to predict NN characteristics.\n\nSTRENGTHS:\nThe paper is generally well-written and the results are strong compared to the chosen baselines from both CNNs and transformer based vision models. The efficacy of the method on 4 search spaces is impressive.\n\nWEAKNESSES:\nThere are 3 main weeknesses / questions that I have regarding the paper:\n\n1. As a blockwise NAS approach, this paper largely ignores much of the prior literature on the topic, for example, [1,2,3]. These prior works also rely on blockwise layer statistics (sometimes obtained through distillation) then the layerwise performance is assembled either with a learned predictor [1,2] or ILP [3] similar to what is proposed in this work.\n2. A large part of the evaluation is done on NB201 which isn't a macro search space. I am a bit confused as to how that evaluation on NB201 demonstrates the presented blockwise methodology -- all blocks within NB201 are identical, just sized differently. Perhaps this is something the authors can clarify in the rebuttal? I understand that the plots in Fig. 1 are showing the accuracy delta when one block is replaced with another, bit this would switch out the entire network basically, right?\n3. Some conclusions (e.g. line 156) may be too specific to NB201. The inverse correlation of flops and accuracy is not a hard rule and many counterexamples can be presented. On NB201 specifically, it is widely known that this holds.\n4. Search cost is written as \"10 minutes\" does this include all the time to evaluate individual blocks and build up the model? I think this cost should be quantified clearly in the paper.\n\n\n[1] Distilling Optimal Neural Networks: Rapid Search in Diverse Spaces\n[2] Blockwisely Supervised Neural Architecture Search with Knowledge Distillation\n[3] LANA: Latency Aware Network Acceleration\n[4] BLOX: Macro Neural Architecture Search Benchmark and Algorithms\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-23"}