Question: Is it true that baselines are omitted from the experiment?
Evidence: Task setup. We cast the NN performance prediction as a classification task. Specifically, the generated time-evolving graphs are labeled as high and low accuracy based on the performance of their corresponding NNs; Table lists the threshold value chosen for low and high accuracy labels based on the final accuracy range of trained NNs, as well as the early stopping epochs for each architecture. Five-fold cross validation is used to predict the label of the test graphs in a binary classification task using SVM and MLP, where the input is the set of temporal signatures {s t tr1 , s t tr2 , ..., s t trn }. We report the classification accuracy. Since the sample of NNs (App. A.1) is randomly selected with balanced high/low accuracy instances, the accuracy of a random classifier as the baseline is 50% (omitted from the charts to avoid clutter). Additionally, to show that our proposed graph representation and signatures are general and can be useful across different NN architectures, we consider the following setup: we train the classifier on a small set of NN models (i.e., different architectures-such as LeNet, AlexNet, and VGG-and hyperparameters), and predict the performance on unseen architectures (e.g., ResNet). We describe these experiments in App. A.4.
Results. Figure illustrates the performance of SVM and MLP classifiers operating on weighted degree-based signatures of the rolled and unrolled graph representations of the LeNet and AlexNet architectures, trained on the CIFAR-10 dataset. We omit the results on VGG and ResNet, as well as image classification on ImageNet, because the unrolled graph generation process is prohibitively expensive, both in terms of time and space. Overall, the rolled and unrolled graph representations show similar trends in classifying NNs by effectively capturing their early training dynamics: the structural changes in the training NN architectures during the first 6-15 iterations are sufficient to classify the performance of NN instances with over 90% accuracy. However, as we discuss in § 4.3, our proposed rolled representation is significantly more efficient than the unrolled representation, and can generalize to deeper and larger NNs. We provide more details for these experiments in App. A.2. OBSERVATION 1 Both the rolled and unrolled time-evolving graph representations of NNs are effective in capturing the changes in the NN dynamics during the training phase, and can be used to predict the accuracy of an NN instance after observing only a few training epochs. Our proposed rolled representation is also space-and time-efficient, unlike the unrolled representation.
In the remainder of this analysis, we focus on the rolled representation, which is more efficient for larger NN models and datasets. We present the classification results for both types of signatures for the temporal graphs corresponding to the training dynamics of VGG and ResNet-44 (CIFAR-10 dataset) in Fig.(e)-(h) and LeNet, AlexNet and ResNet-50 (ImageNet datatset) in Fig.. In addition to the results discussed above for LeNet and AlexNet, we provide the NN classification accuracy for the eigenvector-based signatures in Fig. in the appendix. In all the cases, classification accuracy of 80-95% is achieved in less than 10 training epochs. For degree-based signatures, SVM tends to outperform MLP, while the trend is reversed for eigenvector-based signatures. For example, for both VGG and AlexNet for the CIFAR-10 image classification task, MLP can predict the performance with accuracy ∼95% using the eigenvector-based signatures from the first 6 training epochs; the same trend is observed on ImageNet for the LeNet and AlexNet architectures. For ResNet on the ImageNet classification task, SVM tends to perform well for both degree and eigenvector centrality signature vectors; while MLP tends to perform poorly in this case, the MLP variant operating on the exponential average of signature vectors outperforms the original MLP and all SVM variants. In all the cases, both classifiers reach performance over 80%-90% significantly before the early stopping point for all the architectures. In general, for ImageNet, we observe that the signatures based on eigenvector centrality are more effective in the NN performance prediction task compared to the weighted degree-based signatures-irrespective of NN architecture.
OBSERVATION 2 For the CIFAR-10 image classification task, the rolled graph representation for all NN architectures and both signature types achieve accuracy of >90%. For ImageNet, the eigenvector centrality-based signatures tend to yield higher performance compared to the weighted degree-based signatures, though both achieve accuracy of 80-90%.
Results. Figure 4 illustrates the performance of SVM and MLP classifiers operating on weighted
degree-based signatures of the rolled and unrolled graph representations of the LeNet and AlexNet
architectures, trained on the CIFAR-10 dataset. We omit the results on VGG and ResNet, as well
as image classification on ImageNet, because the unrolled graph generation process is prohibitively
expensive, both in terms of time and space. Overall, the rolled and unrolled graph representations
show similar trends in classifying NNs by effectively capturing their early training dynamics: the
structural changes in the training NN architectures during the first 6-15 iterations are sufficient to
classify the performance of NN instances with over 90% accuracy. H
Answer: To the best of our knowledge, there is no other method that aims to solve the problem that we introduced in this paper, i.e., the prediction of the accuracy of NNs. Nevertheless, we reported results on a previously proposed graph representation for convolutional layers (i.e., the unrolled graph representation introduced by Rieck et al. 2019) in order to compare the efficiency and accuracy of our proposed rolled representation. We reported results on this “baseline” representation in Fig 4 (c) and (d). Additionally, we considered different options for other steps of our proposed methodology beyond the graph construction (e.g., different signature constructions, classification vs. regression) and discussed our observations. Finally, we compared the efficiency of our approach to the “early stopping method”.