ResPerfNet: Deep Residual Learning for Regressional Performance Modeling of Deep Neural Networks
Authors: Chuan-Chi Wang
Ying-Chiao Liao
Chia-Heng Tu
Ming-Chang Kao
Wen-Yew Liang
Shih-Hao Hung
Venue:  (2021)

Abstract: The rapid advancements of computing technology facilitate the development of diverse deep learning applications. Unfortunately, the efficiency of parallel computing infrastructures varies widely with neural network models, which hinders the exploration of the design space to find high-performance neural network architectures on specific computing platforms for a given application. To address such a challenge, we propose a deep learning-based method, ResPerfNet, which trains a residual neural network with representative datasets obtained on the target platform to predict the performance for a deep neural network. Our experimental results show that ResPerfNet can accurately predict the execution time of individual neural network layers and full network models on a variety of platforms. In particular, ResPerfNet achieves 8.4% of mean absolute percentage error for LeNet, AlexNet and VGG16 on the NVIDIA GTX 1080Ti, which is substantially lower than the previously published works.

Under review as a conference paper at ICLR 2021
RESPERFNET: DEEP RESIDUAL LEARNING FOR RE- GRESSIONAL PERFORMANCE MODELING OF DEEP
NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The rapid advancements of computing technology facilitate the development of diverse deep learning applications. Unfortunately, the efficiency of parallel com- puting infrastructures varies widely with neural network models, which hinders the exploration of the design space to find high-performance neural network ar- chitectures on specific computing platforms for a given application. To address propose deep learning-basse based method, ResPerfNer, which
such challenge, trains a residual neural network with representative datasets obtained on the target platform to predict the performance for deep neural network. Our experimental results show that ResPerfNet can accurately predict the execution time of indi- vidual neural network layers and full network models on variety of platforms. In particular, ResPerfNet achieves 8.4% of mean absolute percentage error for LeNet, AlexNet and VGG16 on the NVIDIA GTX 1080Ti, which substantially lower than the previously published works.
INTRODUCTION
Deep learning (DL) has exploded successfully and is applied many application domains, such as image recognition and object detection Thus. lot of human experts design high-accurace neural network architectures for different applications. However, for Internet of Things (IoT) applications, large neural network models cannot fit into resnurce-constrained devices. On the other hand, system designer often tries find a proper computing platform or deep learning accelerator (DLA) to execute a DL. application with acceptable responsiveness. An exhaustive way to optimize the system design is to evaluate the cost and performance of desired DL models on all the available hardware/software options, but it is not only tedious but costly and lengthy in practice.
Since DL frameworks and accelerators are evolving rapidly, and even some slight changes could sig- nificantly impact the performance of DL applications, may be necessary update the performance models frequently. Therefore, we need a systematic and efficient approach to produce accurate per- formance models when changes occur. While several works (Qi et al.. Justus et al. (018); Wang et al.) have been proposed to estimate the delivered performance of a given DL model on a specific computing platform,
to rapidly evaluate design alternatives, the estimates from these efforts are not very accurate. For example, the mean absolute percentage error (MAPE) for estimating full neural network models such LeNet (LeCun al. (1988), AlexNet (Krizhevsky al. (2012) and VGG16 (Simonyan & Zisserman) on the NVIDIA GTX 1080Ti is as high as 24% in Wang et al., whose accuracy the best among the previous works, but still has room for improvement.
In this paper, we propose deep residual network architecture, called ResPerfNet, to efficiently and accurately model the performance of DL models running on wide range of DI. frameworks and DLAs. It is based on the residual function approach proposed by (He et al. (2016) and inspired by the prior works Liu & Yang 2): Tha et al. (2019) Wan et al. (2019)), which use residual neural networks to solve regression problems. The proposed model can be trained with performance data collected from many system configurations to establish a unified performance predictor which as- sists the users in selecting the DL model, the DL framework, and the DLA for their applications. Extensive experiments have been done to show that our unified approach not only provides more accurate performance estimates than the previous works, but also enables the users quickly pre-

---

Under review as a conference paper at ICLR 2021
dict the performance of their DL applications executed with various aaadlls--fammmooo---ceeeeera configurations. The contributions this paper are summarized as follows.
An unified DL-based approach for estimating the computing performance DL applica- tions variety of fra--ror---aeeeeco to explore the hardware/software design space quickly.
configurations, which enables the users
A novel deep residual neural architecture is proposed to deliver the most accurate perfor- aware of. Experimental results confirm that approach
mance predictions that yields lower prediction errors on across various platforms.
The remaining of this paper is organized as follows. Section tion describes the architecture of ResPerfNet. Section A shows the proposed systematic modeling method. Section | elaborates the dataset and training mechanism to train the ResPerfNet models within reasonable time span. Section evaluates the efficiency of our approach. Section 7 con- cludes the paper.
2 presents the related work. Sec-
BACKGROUND AND RELATED WORK
With the rapid evolving of both hardware accelerators and DL models, the performance mea- sure/cstimation of the DL models on the DLA platforms is an important task to evaluate the ef- fectiveness of the software/hardware solutions to the given problems. Different approaches have been proposed to serve the purposes.
Benchmarking approaches, such DAWNbench (Coleman et al. (017)) and MLPerf (Reddi al. (2020)) aim at the measurements of the training and inference performance of the machine-leariig (ML) models certain softwarc/hardwarc combinations. By offering set of standardized machine learning workloads and the instructions for performance benchmarking, these benchmarks are able to measure how fast system can perform the training and inference for ML models.
Analytical approach, as reported in PALEO (Qi al.). constructs the analytical performance model for DL systems. The execution time is decomposed into the total time for the computation and communication parts, which are derived from the utilization of the computing and communication resources on the target hardware, respectively. For instance, the computation time is estimated by dividing the total floating-pont -oint operations required by the DL model to the actual processing speed (i.e., the processed floating-poit point operations per second for the DL model) delivered by the computing hardware. The communication time is calculated by the similar approach. This approach highly relies on the accuracy of the benchmarking results (i.e., to provide the actual processing speed of the target model on the hardware), which requires its users to choose the benchmarks wisely to perfectly match the program characteristics of their target deep learning models, so as to give a proper estimate of the actual processing speed. However, the manual process (of the benchmarks selection) limit its widespread adoption.
DL-based approaches build the DNNs for estimating the DL models' performance by learning the relationships between the characteristics of the DL models and the specifications of the accelerating hardware. The following works focus TensorFlow-base a fully-connected multiple-layer perceptron (MLP) network for performance prediction. using the configurations of the DL model and the specification of the hardware accelerator, and the training data of the DL model as the input features to the MLP network. However, due to the simplified communication time estimation model, where the communications from GPU CPU for each of the DL layers are counted repeatedly for estimating the communication time, their model tends to provide over-estimated results. Waan al. use PerfNet (an ML.P network) to learn the relationships between the configurations and the execution time of the target DL. model. They further decompose the execution of a DL model into three phases, preprocessing, execution, and postprocessing, and train multiple PerfNet network instances, each of which learns the relationships between the model configurations and the model execution time for a specific phase. By aggregating the prediction results for the three phases, their proposed work is able to predict the total execution time of given DL model. Nevertheless, the MLP network has its own limitation, i.c., is hard to further enhance its performance since deeper MLP network will lead lower prediction accuracy.
DL models. Justus et al. (2018) use
2

---

Under review as a conference paper at ICLR 2021
ResPerfNet
Residual Blook
128
128
128
128
128
64
64
64
64
&
32
32
32
32
32
128
128
128
Conv
Conve
Cony
Conv 1D
Conv1D
Cony 1D
Conv1D
Cony
Cony
Conv1D
Conv 10
Conv
Conv1D
Conv
Conv1D
add
FC
FO
FC
dropout
Figure I: Network architecture of ResPerfNet for performance estimation, where the number above
each layer is the kernel number or the output neuron number for the corresponding layer.
In consideration of the limitations of the prior works listed above and the need of modeling the optimizing DL frameworks, our work uses the systematical approach to characterize the DL models built with various DL framework, and adopts the residual neural network to model their delivered performance on the DLAs.
3 RESPERFNET ARCHITECTURE
ResPerfNet adopts a ML-based approach for the performance estimation of different types of neural network layers. Furthermore, ResPerfNet is specially designed to prevent the degradation problem, which refers to the phenomenon that increasing the depth and/or the width of each layer for the DNN may not only necessarily improve the accuracy, but get saturated rapidly and then degrades sharply reported in (He & Sun (2015); Srivastava et al. 20155). In other words, it is more likely to lead to a higher training error on the neural network with a wider or deeper architecture. To solve the problem, the deep residual learning is proposed and applied to each group of the stacked NN layers (He et al. (2016)), where certain number of stacked layers are logically grouped together to form residual block. Hence, in this work, address the degradation problem, we adopt the deep residual learning to every few stacked layers (He et al. )2016)).
The residual block is defined Equation 1 where and y represent the input feature maps and the output vectors of the residual layer, respectively. The function F(x, (W.)) performs the residual operations to be learned. The operation F(x, (W:3) element- wise addition. Figure illustrates the network architecture of ResPerfNet. The second, third and fourth layers (i.e., two convolutional and one add layers) together form residual block, and there are total of six residual blocks in ResPerfNet.
is performed by a shorteut connection and
= =xx, (W,)) +x
(I)
As shown in Figure]) 1 the ResPerfNet consists of 26 layers, including 15 convolutional layers, 6 add layers, 4 fully-connected (FC) layers and dropout layer. Before FC layers, every 7 layers contain one head convolutional layer (e.g., Conv1D 3 representing the head convolutional layer for the first residual block) and two residual blocks, each of which consists of two convolutional layers with the same filters and an element-wise add residual function layer. The first head convolutional layer has 128 filters of kernel size 3 with stride length of 1. In order to reduce the complexity of ResPerfNet, the second head convolutional layer uses 64 filters of kernel size 3 with a stride length of 1. Moreover, the number of filters for the six residual blocks is decreasing from 128 filters in the first two blocks to 32 filters for the last two blocks. Three FC layers are attached the last residual block, where each of the FC layers has 128 neurons. The dropout layer with the ratio of 0.2 connected to the last FC layer, which uses a single neuron to perform the one-dimensional regression for predicting the elapsed time of the designated type of the layers.
Our proposed residual neural architecture, ResPerfNet, gets significant improvements in accuracy compared with traditional machine learning algorithms, such support vector regression, polyno- mial regression and XGBoost, and is even better than the MLP network. A series experiments has been done to show ResPerfNet is superior the previous works in Section
3

---

Under review as a conference paper at ICLR 2021
4 METHODOLOGY
This section presents the methodology of using ResPerfNet to relate the performance characteristics of a CNN layer the delivered performance of the given layer. We first define the target neural networks for the performance modeling in Section 4.I The three-phase based modeling of a given Lastly, the same modeling for a given NN layer further
CNN based presented in Section 4.2
described in Section 4.3
4.1 FORMALIZING THE NEURAL NETWORKS
A neural network can be represented by a directed acyclic graph, denoted as () con- sisting of an ordered sequence of k nodes, where each graph node u(() represents a layer of the neural network M, such convolutional, pooling, and fully-connected layers. The input and output feature maps of a graph node ( performing the operation f() are denoted as inpud(f()) and output(f f))), respectively. In this work. assume that given neural network will be run the host system with a single hardware accelerating device d.
4.2 THE WHREEE-HAASE PERFORMANCE MODELING
The execution time of given neural network model includes the computation time spent on the acceleration device d and the data communication time between the host system h and the device d. As most of the computations are performed by the accelerating device and the communications occur merely at the first and the last layers of the given model, the estimated execution time of given neural network model with & layers is formulated as follows, where the formulation assumes that all & layers within the given model are accelerated by the single device d.
T(N) = Prre (1)
ΣTeze (uee T Tpose( (1)
The above equation shows the three-phase performance modeling approach, where Tpre: Texes and T Tost represent the execution time for the preprocess, execution, and postprocess phases, respec- tively. Specifically, the communication time of bringing the input data from the host system the accelerating device at the first layer is denoted as T pree (v)),, where the i-th NN layer is represented
as (i) The summation of the execution time for all the NN layers is represented as Σ Teze ())) The communication time of transferring the inference results from the accelerating device to the host system is defined as (k))), Our prediction model delivers more accurate performance es- timates than previously proposed methods by modeling these three phases defined in the following subsection for DLA separately and adding the predicted results together Equation] 2
4.3 MODELING INDIVIDUAL NN LAYERS
The similar approach is used to model the performance of the i-th NN layer 0 In particular, for each layer u()), the execution times for the preprocess, execution, and postprocess phases are ()) ()), and T (post((()) respectively. The above time components constitute the estimated execution time of the layer u as defined in the equation below. The superscript index i is omitted simplify the looks of the equations by using the simpler form u.
T(u)= (u) + Texe(u) + (u)
The preprocess phase is for preparing the input data for the acceleration in d and involves with the four operations: 1) issuing the commands for copying input feature maps on and d asynchronously, 2) performing the memory copy of the input feature maps in 1, 3) issuing the commands for the operation f on d, and 4) performing the data reshaping operations for input feature maps. The data reshaping operations, which transform the input/output data to the more efficient format for the next operation on d, usually occur in data transmissions between h and d. The lengths of time for the four operations shown in Equation 4 the time consumed in the preprocess phase defined as the summation of the time required by the above four operations.
R( R(input(f f, h, d), M(input(f), Rff, d), and T (imputt(f) f), d), respectively. As
4

---

Under review as a conference paper at ICLR 2021
Tpre(2) = R(input(f))) d) + M(input(f),, + -R(f.d) + (input(ff).0
(4)
Intuitively, the time consumed for computation, which is C(f.d), d), in the execution phase would be identical the computation time of f on d. Unfortunately, the measured execution time layer from the micro-benchmarks includes the time consumed by the data reshaping operations in both di- rections, from h to d and from d to hi which are 'I (innpul(J): d) and ((oulpul(])) )), d), respectively. As the deployed NN layers collectively run on the acceleration device d, isolating the data reshaping time from the measured exccution time for the NN layer of each micro-benchman benchmark facilitates the execution time estimation of the deployed NN layers with the formula, Dim Teze ())) Regarding this situation, the time for the execution phase defined in Equation [
(u) C(J,d) T (inpull())
T (oulppll())
(5)
The postprocess phase is defined for dealing with the procedure of returning the inference compu- tation results back to the invoking application on the host system. That is, it is about reshaping the output vector into the format accepted by h, copying the output vector back to h from d, and moving the prediction result to the application level (i.c., the call site of the model inference) on the host system. The corresponding execution time for the above three operations are denoted as 'T ((oupput(f) d), M(output(J, d, h), and (ooupput(f); respectively.
Tpost (u) = T (outputt (f), + Moutppt((f) h) + V(output(f))h
(6)
5 TRAINING DATA AND LOSS FUNCTION
In this section, we present the details of the dataset used to build the proposed performance pre- diction models. In particular, the configurations of our developed benchmark tools for the training dataset is discussed in Section [.I] The tool collecting and extracting the data is described Sec- tion 5.2 and the data transformation techniques facilitate the training convergence is introduced in Section 5.3 The specially designed loss function to better deal with the unbalanced training data introduced in Section
5.1
DATA PREPARATION
The training data is the characteristics of the TensorFlow and TensorRT programs and the perfor- mance information of the programs rumning on the target computing hardware, where the proposed model helps correlate the characteristics and their runtimes during the training process. In order to better catch the characteristics of different TensorFlow and TensorRT configurations (i.e., the code patterns, which considered as the features during the model training process), have devel- oped a benchmark tool to generate a set of nhicro-bnncharrk, which are actually TensorFlow and TensorRT programs with different configurations for the three types of layers, including convolu- tion, pooling and dense layers. The generation of the benchmbenchmsrks are done by randomly selecting the configurations for each type of the layer, configurations. The possible configurations (or features) for all three layer types and their ranges are listed in Table 3 These configurations are actually the function parameters for the three types of lay- ers, which are extracted from TensorFlow I, 1,13 APIS, including tensorflow layers. conv2d, tensorfl layers. wwxpoo i in22 d, and tensorflow www.waayer.... dense, and their pos- sible combinations are 7.33 x 14 7.33 x 10 and 2.14 x 10", respectively. While each micro- benchmark takes at least seconds for the stable and accurate measurements, it is impossible cover the entire design space with brute force, which requires over 10 miero-benchmark runs.
collect the performance for different
5.2 DATA COLLECTION AND DATA EXTRACTION
The data preparation used to generate the TensorFlow- and TensorRT-based micro-benchmarks. The data collection takes about two weeks for running 100,000 different samples of the Tensor- Flow
on the DLAs to collect the performance data. On the other hand, for the TensorRT micro- benchmarnssmm,,s more than two weeks were spent to optimize and profile the 25,000 different configurations of the TensorRT programs. It interesting to note that the TensorRT exper- iments generate large optimized intermediate files, especially for the dense layer, where it requires
5

---

Under review as a conference paper at ICLR 2021
more than 5TB of storage space to keep its parameters. Due to the disk space limitation, select 16,000 out of 25,000 samples to run and profile their performance. For data extraction, our data processing tool filters out the outliers (data with extreme values) before feeding the profiled data for the model training. The total elapsed time of each layer is decomposed into the preprocessing time (Tpr)), the execution time (Teze), and the postprocessing time posst), as mentioned in the previous section. In order to test the accuracy of our trained model, the collected samples are split into 80% of the samples as training datasets and 20% testing datasets.
5.3 DATA TRANSFORMATION
given training dataset D, which is comprising observations and fea- tures of X and written as 'D = {ft, Til, Ti2,..., 2ipl. where t is a vector of observed values (i 1,..., m), and X could be seen a matrix of row-vectors x; (i 1, ..... m) of m- dimensional column-vectors X₃ 1, 1,...p.. The coefficients vector w keeps the weights of the model. The predicted value is denoted y(x, w), for any given model of weights and the dataset x. In order to improve the convergence efficiency and stability of the stochastic gradient descent (SGD) algorithm. the three types of data transformations are adopted in this work, including scalar multiplication. Z-scores transformation, and Box-Cox transformation. Scalar multiplication is used to provide fine-grained updates of the SGD procedure and scales each observed value 4,. Z-scores transformation puts each data feature X, from different sources into the same scale to eliminate the prejudicial bias of the features values. Box-Cox transformation converts the values of the features X, to standard normal random variables, which would further improve the effectiveness of Z-scores transformation. Details these data transformations are available Appendixes B and
Now, suppose
5.4 Loss FUNCTION
As the observed vector is with the positive-skew distribution and often contains some noises con- tributed by the measurement errors, we fine-tune the loss function as mean absolute percentage logarithmic error (MAPLE) for the prediction model (Wang et al.). as shown Equation 7 To deal with the situation of the skewed distribution, the logarithmic operations for the predicted values 1 + X,, w) and the observed values + 4, and the division operation on the observed values in MAPLE expected to enhance the accuracy of the small data, which occurs frequently. On the other head, the absolute value of MAPLE helps increase the resistance against outliers that may unexpectedly appear in the measured data. Moreover, to prevent over-fitting, L. regularization is added to the loss function, where A2 is a scaling factor for the regularization.
log(1 + y(x,, w)) log(1 + 4:) + A21lw|1
En(w) =
log(1 (1 + +t,)
EVALUATION
The layer-wise and model-wise performance results are evaluated to demonstrate the effectiveness of ResPerfNet in this section. In particular, we compare the layer-wise estimated execution time produced by ResPeriNet and the previous works to show that ResPeriNet is superior to other re- gression based approaches, such as polynomial regression, support vector regression and PerfNet. Three statistical metrics, including mean absolute percentage error (MAPE), root mean squared error (RMSE) and mean absolute error (MAE), are used to quantify the effectiveness for each tested per- formance modeling approach. In addition, to demonstrate the capability of ResPerfNet for the full model prediction, three popular CNNs considered in the model-wise experiments, e.g., LeNet, AlexNet, and VGG16. Note that three data transformations mentioned in Section 5.3 are applied in ResPerfNet by default unless specified otherwise. The details of our experimental environments are listed in Appendix
6.1 LAYER-WEL WISE EXECUTION TIME PREDICTION
Table I compares the MAPES of the execution time for the convolutional layers, estimated by Res- PerfNet and the prior works. While appropriate parameter adjustments are applied to obtain parame-

---

Under review as a conference paper at ICLR 2021
ters for better results, the MAPEs of polynomial regression, support vector regression, and XGBoost are over 29%, which means the error is quite large and indicates that the corresponding approaches not capable of doing good performance prediction for the real applications. On the contrary, the DL--based approaches, PerfNet and ResPerfNet, give more accurate estimations, which have less than 15% of the MAPES. In particular, ResPerfNet outperforms the other approaches and has 11.75% and 14.23% of the MAPEs for the TensorFlow and TensorRT models. The results suggest that ResPerINet correctly associates the program characteristies the performance model.
Table 1: Comparison of the prediction errors (MAPEs) for the convolutional layer execution time produced by different approaches.
Polynomial Regression
SVR RBF
ResPerfNet BoxCox
Framwork
Layer
Phases
XGBoost PerfNet
TensorFlow
convolutional execution
63.57 % 316.5 %
58.74 %
29.25 %
14.96 %
11.75 %
TensorRT
convolutional
execution
59.94 %
33.10 %
14.23 %
To further look into the effectiveness of PerfNet and ResPerfNet and the impact of the Box-Cox transformation on the predicted results, Figure plots the error curves of the TensorFlow convolu- tional layer using the PerfNet and ResPerfNet with and without performing the data transformation. Figure [a) shows that most of the MAPE of ResPerfNet on the testing dataset are below 15%, as de- picted by the red/black solid lines. Notably, ResPerfNet applying the Box-Cox data transformation reaches the lowest prediction error (11.7%). ). 2% less than ResPerfNet without the data transforming. Similar trends can be observed in Figure b) using the RMSE metrie, in which the black solid line also shows the best performance. The results presented in Figure 2 show that ResPeriNet with Box- Cox transformation has better convergence rate, given the same training epoch. The detailed training process is illustrated in Appendix Moreover, the R2 values for the predicted and measured execu- tion time of the convolutional, pooling and dense layers are all above 0.97, which demonstrate high prediction quality of ResPerfNet, as illustrated in Figure Of of Appendix
(a) MAPE
(b) RMSE
35
PerfNet
PeriNet
PerfNet with BoxCox
PerfNet with BoxCox
30
ResPerfNet
ResPerfNet
Percentage (%)
ResPerfNet BoxCox
188
ResPerfNet with BoxCox
Time (ms)
Epech
Epoch
Figure 2: Training and testing errors of PerfNet and ResPerfNet on their TensorFlow models, where the dashed lines represent the training errors, and the solid lines denote the testing errors, with respect to (a) MAPE and (b) RMSE.
The layer- performance results of the TensorFlow and TensorRT models delivered by Res- PerfNet are listed in Table 2 Overall, the MAPE for all phases are under 16%, which removes the concern of over-fitting. For RMSE, the value of the TensorFlow version convolutional layer is 0.84ms. It is better than the 0.98ms reported by PerfNet (Wang et al.), and is also better than the 2.55ms produced by the method in (Justus et al. (2018)). Detailed predicted results the three layers under different phases for TensorFlow (with 3 additional platforms) and for TensorRT are also presented in Appendixes and H respectively. From the tables, we can see that ResPerfNet has better predicted results for TensorFlow than TensorRT. That is because currently the TensorRT- based ResPerfNet is trained with less training data, as described in Section 5.2] We believe that the accuracy for TensorRT predictions can be further improved with sufficient data TensorFlow.
7

---

Under review as a conference paper at ICLR 2021
Table 2: ResperfNet: TensorFHow//eesorkk1 predicted inference results NVIDIA GTX 1080Ti.
Framwork
Layers
Phases
MAPE (%)
RMSE (ms)
MAE (ms)
convolutional
execution
11.75
0.840
0.336
TensorFlow
pooling
execution
6.221
0.367
0,125
dense
execution
4.515
0.069
0.036
convolutional
execution
14.23
0.649
0.263
TensorRT
pooling dense execution
execution
15.19
0.497
0.186
13.40
0.094
0.045
6.2 MODEL-WISE EXECUTION TIME PREDICTION
Figure plots the inference time estimated by PerfNet and ResPerfNet for the three popular DNNs, including LeNet, AlexNet, and VGGI6, using TensorFlow and TensorRT frameworks. Figure 3 a)- (c) shows that ResPerfNet has more accurate estimation than PerfNet since the averaged MAPE of the three models is 8.4% for all tested batch sizes, while PerfNet has the averaged MAPE of 24.04% % Figure digureBi)-(f) (f) illustrates the similar trend for TensorRT based DNNs. The averaged MAPE of these DNNs using ResPerfNet is 17%. The results show that our modeling and methodology are effective on the two popular frameworks.
(a) LeNet TeosorFlow
(b) AlexNet TensorFlow
VGG16 TensorFlow
15
Time Len5
(ms)
Time (ma))
60
0.5
10
0.1
16
32
32
Batch Size
Batch Size
Batch Size
Bi PerfNet a ResPerfNet m Actual Time
PerfNet mn ResPerfNet Actual Time
PerfNet ResPerfNet mI Actual Time
(d) LeNet TensorRT
(e) AlexNet TensorRT
VGG16 TensorRT
0.7
90
0.6
770
Time (ms))
0.4
Time
Time
0.2
0.1
32
32
Batch Size
Batch Sizze
Batch Size
ResPerffNet
Actual Time
ResPerfNet
Actual Time
ResPerfNet Actual Time
Figure 3: Inference Time Prediction NVDIA GTX 1080T:: Comparison with actual inference time for (a) LeNet on Tensorflow, (b) AlexNet LeNet on Tensorflow, (c) VGG16 on Tensorflow, (d) LeNet on TensorRT, (e) AlexNet on TensorRT, and (f) VGG16 TensorRT.
7 CONCLUSION
In this paper, we proposed deep residual network architecture, ResPerfNet, to model the per- formance of neural networks on the target DLAs by considering the interactions between the host and the GPU and decomposing a neural network operation into three phases. In addition, we ap- ply ResPerfNet predict the execution time of the optimized models, such as TensorRT, with the same performance characteristics as those used in unoptimized models. Our experimental results show that ResPerfNet is able to provide high-accuracy estimations on various DLAs, which helps facilitate the exploration of proper neural network architectures built with various DL frameworks.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Cody Coleman, Deepak Narayanan, Daniel Kang, Tian Zhao, Jian Zhang, Luigi Nardi, Peter Bailis, Kunle Olukotun, Chris Ré, and Matei Zaharia. DAWNBench: An End-to-End Deep Learning Benchmark and Competition. pp. 10, 2017.
Kaiming He and Jian Sun. Convolutional neural networks constrained time cost. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5353-5360, 2015.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Dipendra Jha, Logan Ward, Zijiang Yang. Christopher Wolverton, Ian Foster, Wei-keng Liao, Alok Choudhary, and Ankit Agrawal. IRNet: A General Purpose Deep Residual Regression Framework for Materials Discovery. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2385-2393, Anchorage AK USA, July 2019. ACM ISBN 378-1-4503-6201-6 doi: 10.1145/32925003330703
Daniel Justus, John Brennan, Stephen Bonner, and Andrew Stephen McGough. Predicting the Com- putational Cost of Deep Learning Models. 2018 IEEE International Conference on Big Data (Big Data), pp. 3873-3882, December 2018. doi: 10.1 10.1/09//gDDt 201.gaaaaaa.1.666611 8.8622396.
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convo- lutional Neural Networks. In F. Pereira, C. C. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 25, pp. 1097-1105. Curran Associates, Inc., 2012.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEFF, 8271222278--2222 1998.
Jen-Yu Yu Liu and Yi-Hsuan Yang. Denoising Auto-Encoder with Recurrent Skip Connections and Residual Regression for Music Source Separation. In 2018 17th IEEE International Conference Machine Learning and Applications (ICMLA), pp. 773-778, December 2018. doi: 10.1109/
RGMLA.2018.00122.
Hang Qi. Evan R. Sparks, and Ameet Talwalkar. Paleo: A Performance Model for Deep Neural Networks. In ICLR, 2017.
Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole- Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka. Cody Coleman, Sam Davis, Pan Deng, Greg Diamos, Jared Duke, Dave Fick, J. Scott Gardner, Itay Hubara, Sachin Idgunji, Thomas B. Jablin, Jeff Jiao, Tom St. John, Pankaj Kanwar, David Lee, Jeffery Liao, Anton Lokhmotov, Francisco Massa, Peng Meng, Paulius Micikevicius, Colin Oshorne, Gennady Pekhimenko, Arun Tejusve Raghunath Rajan, Dilip Sequeira, Ashish Sirasao, Fei Sun, Hanlin Tang, Michael Thomson, Frank Wei, Ephrem Wu, Lingjie Xu, Koichi Yamada, Bing Yu, George Yuan, Aaron Zhong, Peizhao Zhang, and Yuchen Zhou MLPerf In- ference Benchmark. In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), pp. 446 459, Valencia, Spain, May 2020. IEEE. ISBN 978-1-72814-661144 doi: 10.1109/SSCA44567.2200 202.000002200000455.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. In Interna- tional Conference on Machine Learning (ICML), 2015.
Jia Wan, Wenhan Luo. Baoyuan Wu, Antoni B. Chan, and Wei Liu. Residual Regression With Semantic Prior for Crowd Counting. In 2019 IEEE/VV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4031-4040, Long Beach, CA, USA, June 2019. IEEE ISBN 10.1..
978-1-72813-293-88 doi: 10.
Chuan-Cue Chi Wang. Ying-Chiar Liao, Ming-Chama Kao, Wen-Yer Yew Liang, and Shih-Hao Hung. Perfnet: Platform-awart performance modeling for deep neural networks. In Research in Adap- five and Convergent Systems (RACS), 2020.

---

Under review as a conference paper at ICLR 2021
A FEATURES OF TRAINING DATA
Table 3: Description of features.
Name
Description
Range
Scenario
Batch Size
The number of parallel processed in one iteration.
1-64 (I-512) (1-512)
D. P
Matrix Size
The dimensions the input data.
C,
Kernel Size
The size of the filter applied to the input data.
(1-7) (1-7)
C
Channel
The number of channels in the input data.
1-9999
C,
Channel Out
The number of channels in the output data.
1-9999
C
Strides
The amount of the window shifts for each dimen- sion of the input data with kernels.
(1-4) x (1-4)
C,
Padding
The number for preserving the original size of the image while the filter scans each pixel. O: Valid. I: Same.
0-1
C,
Activate Function
The number for representing what activation func- tion is used. O: Without an activate function. 1: ReLUU.
0-1
C.
Bias
The boolean number for utilizing additional in- tercept on training input data. The number of outputs from the previous layer.
0-1
C
Dimension
1-4096
D
Input
Dimension
The number of outputs of the layer.
1 1-4096
D
Output
Pooling Size
The windows size factors for scaling down the in- put data.
(1-7) (1-7)
P
In the scenario column, P. and D indicate which of the NN layer (i.e., Convolutional, Pooling, and Dense layer) the corresponding feature applies to.
B SCALAR MULTIPLICATION
Scalar multiplication is applied on the observed vector t as Equation g to magnify the prediction results since the original data are too small to provide accurate estimates. It is interesting note that the scalar multiplication would inefficient for some commonly used loss function, such as mean squared error MSE, based on experiences; nevertheless, works well with the MAPLE by making every gradient converging smoothly without frequently adjusting an appropriate learning rate in each epoch.
scalar maalaarmmltiplicatiion t = t X scaler
(8)
C Z-SCORES TRANSFORMATION
Z-scores transformation is performed on each n-dimensional column-vector X, as Equation where X, is the mean of the each column-vector X,, and o, is the standard deviation of each column-vector Xy. Z-scores transformation resales the values of the features ensure the mean be zero and the standard deviation to be one. The values of the features are rescaled within the range between zero and one, which is useful for gradient decent algorithms.
X;- X, X,
7-scores transformation
(9)
10

---

Under review as a conference paper at ICLR 2021
D BOx-CoX TRANSFORM
Box-Cox transformation transforms the input features Xj into a normal distribution for the best model accuracy. Box-Cox transformation shown in Equation I0] where λ is the best approxima- tion for the selected features, In our experiments, Box-Cox transformation is applied on Matrix Size and Kernel Size (See 3 for the convolutional layer data, and Matrix Size for the pooling layer data.
X A1 In In
Box-Coa trans Pransfoomaatin: x(Ax)
i
#0 =0
(10)
E
THE PROPOSED GRADIENT DESCENT ALGORITHM
Algorithm [ is the pseudo-code our proposed algorithm to train each phase of the layers. The required parameters are defined follows: optimizer: algorithm used update the attributes of neural network, [r tr.schduuer:: sets the learning rate of each parameter group to the initial Ir times given function, total eepochs: total epochs of the neural network algorithm, Ir: learning rate, bs: maximum of the batch size for each epoch, yy. period of learning rate decay, y. multiplicative factor of learning rate decay, and Az: multiplicative factor for the weight penalty.
Algorithm The stochastic gradient descent algorithm proposed by ResperfNet, where our default settings for the our DL regression problems are oplimizzr Adam, total epocces 200, Ir 0.1, 8s 128, 7) = 40, y = 0.5, X2 0.1, and scaler = 10.
Require: a: Multiplicative factor for the weight, n: Current batch size, T: Current iteration. scalar scalar-maltiplliatee multippicctiin (t, scalar)
Update t by Equation 8
2: x + Z-scores( (ooz-eooooooo (xxxCCo(xx)
Update x by Equation 10 and
3: for in total. epochs do
4: Ir --rrschheuuler(( 1/ 7)
>. Update the learning rate by scheduler.
5: for in (m//bs 1) do optimizer(Ir)
6:
Update the weight factor by optimizer.
7:
1 < x[b bs min ((b + 1) bs, m)]
Calculate n (current batch size). A Calculate E, by Equation
8:
V E. f- calculate gradient of E, on model w + w aVE, (x(E)
9:
W
10:
T +
11: end for
end for
F EXPERIMENTAL SETUP
The experiments are done on the Intel i7 processors with a variety of hardware accelerators listed in Table TensorFlow 1.13.1 and TensorRT 5.0.2.6 with Python 3.6 used to build the DL models, running on Ubuntu 18.04.4 LTS (kernel version 5.4 3.4.0-42-generic).
Table 4: Acceleration hardware specifications.
NVIDIA Device
Basic
CUDA Cores
Memory Clock
Memory Bandwith
Peak TFLOPS
Bus Standard
Clock
GTX 1080Ti
1481 MHz
3584
1376 MHz
484.4 GB/s
11.34
PCle
P1000
1266 MHz
640
1253 MHz
80.19 GB/s
1.894
PCIe
P2000
1076 MHz
1024
1752 MHz
140.2 GB/s
3.031
PCIe
P5000
1607 MHz
2560
1127 MHz
288.5 GB/s
8.873
PCIe
11

---

Under review as a conference paper at ICLR 2021
G
LAYER-WISE EXECUTION TIME PREDICTION FOR TENSORFLOW
Table TensorFlow predicted inference results on NVIDIA GTX 1080Ti.
Layers
Phases
MAPE (%)
RMSE (ms)
MAE (ms)
preprocess
1.603
0.215
0.126
convolutional
execution
11.75
0.84
0.336
postprocess
2.821
0.097
0.041
preprocess
T.177
0.150
0.094
pooling
execution
6.221
0.367
0.125
postprocess
1.967
0.091
0.031
preprocess
6.455
0.036
0.025
dense
execution
4.515
0.069
0.036
postprocess
13.86
0.011
0.009
Table TensorFlow predicted inference results on NVIDIA Quadro PI000.
Layers
Phases
MAPE (%)
RMSE (ms)
MAE (ms)
preprocess
1.943
0.478
0.271
convolutional
execution
12.74
5,022
2.046
postprocess
2.818
0.105
0.038
preprocess
I,511
0,206
0.122
pooling
execution
5,842
0.977
0.390
postprocess
2.860
0.091
0,035
preprocess
6.691
0.043
0.035
dense
execution
5.687
0.429
0.204
postprocess
13.61
0.018
0.008
Table 7: TensorFlow predicted inference results NVIDIA Quadro P2000.
Layers
Phases
MAPE (%)
RMSE (ms)
MAE (ms)
preprocess
1.535
0.536
0.308
convolutional
execution
12.347
4.209
1.671
postprocess
3.132
0.107
0.045
preprocess
1.321
0.407
0.185
pooling
execution
4.425
0.682
0.234
postprocess
2.728
0.098
0.034
preprocess
7.369
0.017
0,013
dense
execution
13.39
0.073
0.037
postprocess
14.03
0.011
0.008
Table 8: TensorFlow predicted inference results on NVIDIA Quadro P5000.
Layers
Phases
MAPE (%)
RMSE (ms)
MAE (ms)
preprocess
1.247
0.417
0.235
convolutional
execution
12.23
1.553
0.671
postprocess
4.072
0.148
0,065
preprocess
1.339
0.398
0.205
pooling
cxecution
6.444
0,527
0.174
postprocess
3.802
0.131
0.053
preprocess
5.126
0.045
0.035
dense
execution
5.298
0.121
0.121
postprocess
18.13
0.019
0.019
12

---

Under review as a conference paper at ICLR 2021
H
LAYER-WISE EXECUTION TIME PREDICTION FOR TENSORRT
Table 9: TensorRT predicted inference results on NVIDIA GTX 1080Ti.
Layers
Phases
MAPE (%)
RMSE (ms)
MAE (ms)
preprocess
2.283
0.646
0.356
convolutional
exccution
14.23
0.649
0.263
postprocess
4.334
0.274
0.089
preprocess
2.268
0.689
0.323
pooling
execution
15.19
0.497
0.186
postprocess
4.001
0.141
0.058
preprocess
5.612
0.035
0.027
dense
execution
13.40
0.094
0.045
postprocess
18.01
0.017
0.014
I PREDICTED VS. MEASURED TIME (TENSORFLOW)
(a) Convolutional
(b) Pooling
(c) Dense
Predieted Time (ms)
70
R 0.979
Prodicted (mss)
.50
R = 0.991
Predicted Time (ms)
R
0.986
60
50
40
2.5
2
20
10
10
Measured (ms)
Time (ms)
Time (ms)
Figure 4:
Predicted and measured times of execution phase of (a) Convolutional, (b) Pooling, and (c) Dense layer NVIDIA GTX 1080Ti.
J MODEL-WISE EXECUTION TIME PREDICTION FOR TENSORFLOW
(a) LeNet
2,00
1.40
Time (1s))
1.20
1.00
0.60
0.40
0.20
32
Batch Size
PI(H)) ResPeNe
P1000 Actual Time
P2000 ResPerfNet
P20NK) Actual Time
P5000 ResPerfNet
PS000 Actual Time 1080T: ResPerfNet
1080T: Actual Time
Figure 5: Predicted and actual inference time comparison for LeNet on TensorFlow.
13

---

Under review as a conference paper at ICLR 2021
(b) AlexNet
50.00
45.00
40.00
35.00
Time (ms))
30.00
25.00
20.00
15.00
5.00
0.00
16
32
Batch Size
P1000 ResPerfNNt
P1000 Actual Time
P2000 ResPerfNet
P2000 Actual Time
P5000 ResPerfNet
P5000 Actual Time 1080TI ResPerfNet
1080Ti Actual Time
Figure 6: Predicted and actual inference time comparison for AlexNet on TensorFlow.
(c) VGG16
500.00
450.00
400.00
350.00
Time (1S8
300.00
250.00
200.00
150.00
100.00
0.00
Batch Size
P1000 ResPerlNNt
P1000 Actual Time P2000 ResPerfNet
P2000 Actual Time
P5000 ResPerfNet
P5000 Actual Time 1080TI ResPerfNNt
1080Ti Actual Time
Figure 7: Predicted and actual inference time comparison for VGG16 on TensorFlow.
14


Reviews:
[{"review_id": "y5iurqPbA82", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Topic\n\nUsing a residual-based network to the performance of another DL-based network.\n\nContributions:\n\nUsing a residual-based network for estimating the computing performance of DL applications on a variety of models-framework-accelerator configurations, which enables the users to explore the hardware/software design space.\nUsing three-phase performance modeling to estimate computation time.\n\nWeakness\n\n1, The main problem is the lack of Novelty and technical contributions. Using a DL-based model to predict the performance is not novel. Many NAS methods using the same trick to estimate the performance of one architecture ahead of running it directly on the hardware.\n\n2, Using a DL-base regression is better than normal regression when the sample size is large. The experiment results are not surprising.\n\n3, Box-cox transformation is a common technique in regression. That is not new.\n\n4, Need to prepare the dataset using a lot of samples, i.e. 100000, which is computationally heavy. \nRequire huge storage space to keep the samples for even one platform.\nThus the whole method is not efficient. It is hard to generalize it to other hardware/platform.\n\n5, Besides, the method only considers some common operations such as conv, pooling, and FC-layer. However, more operations should be considered for example ROI Pooling, NMS, Spatial-to-depth. Those operations are also commonly used in tasks such as detection and SR. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "eSE5YlO8nOL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The authors design a specific ResNet for predicting the model execution time on different platforms.\n\nPros\n- conduct extensive experiments, particularly collect a large scale dataset for measuring different architectures, which can be helpful for further works if it can be released publicly\n\nCons\n- The idea is not novel. The main idea is to utilize a ResNet to perform regression on network latency data, which can only be considered as a normal application of ResNet. \n- The motivation is questionable. In my opinion, making model execution time prediction more accurate should not be the ultimate end. The proposed ResPerfNet should be applied in network evaluation and search stages in Neural Architecture Search (NAS) area, and validate that a more accurate model performance predictor is helpful for architecture search. But I didn't see any supporting experimental results in this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3oPQdCd5N2H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method, called ResPerfNet, to predict the performance of deep neural networks. The method relies on a residual neural network that is trained on a large number of different network architectures and performance measures on real hardware. \n\nThe paper evaluates the proposed method on three networks, LeNet, AlexNet, and VGG16, in two different frameworks, i.e., TensorFlow and TensorRT. The results are promising, but comparison with other approaches is weak. For example, the proposed method, ResPerfNet, is only compared to one other approach, PerfNet (from a paper that don't seem to be published yet, at least I couldn't find it), using TensorFlow (but not using TensorRT).\n\nThe paper has potential to have impact, but it needs to be improved before publication. For example, the following issues need to be addressed:\n* Motivation for the selected structure / architecture of ResPerfNet.\n* Some confusion about kernels, filters, etc. in the description of the ResPerfNet architecture (Section. 3 + Fig. 1)\n* I'm a bit surprised that the dropout layer is very close to the end of the network. Why? Why 0.2 dropout (and not 0.1 or 0.4)?\n* It's a bit confusing (and inconsistent) that the index I is left out sometimes and sometimes not, e.g., Eq (2) vs. Eq (3) vs. how it is written in the text flow. \n* C(f,d) in Eq (5) is never defined.\n* Platform for sample selection / data collection should be mentioned in Section 5.2\n* Section 5.4. Although using defining the loss function as MAPLE does reduce the problem with a skewed distribution, it doe not solve it so \"cope with it\" is a bit strong formulation.\n* It is disturbing that one of the main references [Wang 2020] can't be found, despite extensive searching. This is problematic since the only other solution ResPerfNet is compared to is PerfNet, which is published in [Wang 2020]. This limits the possibility to compare this work with previous work. The conference where the [Wang 2020] paper was published took place in mid October 2020, which is after the deadline for ICLR.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new method to predict the performance of deep neural networks. It evaluates the method on three different networks: LeNet, AlexNet, and VGG16 under two different frameworks, TensorFlow and TensorRT.

Reviewer 2 thought that the results were promising but comparison with other approaches was weak (PerfNet being the only baseline). They also asked for motivation for the selected architecture as well as raised a number of points for clarification. R2 was also concerned that the single baseline appeared to have not yet been published. The authors clarified this in their response (it was published in ACM RACS, obtaining results directly from those authors).

Reviewer 1 said that the experiments were extensive, but did not find the approach novel (“a normal application of ResNet”). They suggested NAS as a motivating application rather than stopping at predicting execution time. The authors agreed with the importance of predicting execution time in NAS.

Reviewer 3 agreed with Reviewer 1’s assessment of lacking novelty and technical contribution. They also pointed towards NAS, where many methods are already using neural networks to predict execution time. They were also disappointed by the reduced set of architectural elements considered. The authors responded to R3’s comments, but R3 was still not convinced of novelty.

This looks like a fairly straightforward rejection on the basis of not enough technical merit. The authors are encouraged to explore their approach in the context of NAS as per R1’s suggestion.
Decision: Reject