Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Instead of using validation accuracy to determine the efficacy of a network,  this paper recommends to use Sum over Training Losses (SOTL). SOTL-E is a variant where the sum of training losses begins to be computed after the first E epochs. 
They also designed an early stopping mechanism based on Baker et al's SVR where they extrapolate SOTL instead of validation accuracy.

Questions:
1. How can training loss be used to identify a good network? It should theoretically lead to overfitting and poor generalization. Going by this argument, if we apply any kind of regularization such as dropout or weight decay, the training loss would not be low  while the test accuracy might still improve.
It is surprising that SOTL-E is able to rank the networks better than TestL at 200 for cifar10 and cifar100. Why do you think this is the case? 

2. DARTS Experiment: 
   (a) In  Figure 3(a), how is DARTS search replicated using just 100 random architectures? As it uses a SuperNet, it requires all possible architectures possible with the chosen operations. 
   (b) In  Figure 3(b) and 3(c), the final architecture needs to be trained for 600 epochs. So it is natural that the rank correlation of SOVL, SOVL-E etc is poor for the first 100 epochs. 

3. What would be more interesting is,  DARTS currently they perform a bi-level optimization. So instead of the architecture parameters $\alpha$ trying to minimize the validation loss, can they also minimize the training loss (theoretically this should not generalize well)? If not this, can you devise a way to plug in SOTL in the bi-level optimization and choosing the best architecture? If SOTL performs well even in that case, then you could a stronger case.

4.  Training very deep networks is not easy and takes more than 100 epochs to obtain good accuracy. So your observation might be a side effect of that too.  As SOTL could be applied to any deep learning networks, can you also repeat the experiment by training 100 networks sampled from a smaller search space, such as mobile search space (mobilenet, squeezenet, shufflenet etc), that takes less than 80 epochs to finish training, to see if it still holds true? Then use this SOTL and SOTL-E to determine the best network. Also compare with the baselines.

5. In Figure 4, how do SOVL, SOVL-E and Validation accuracy fare? Please include those too in the plots. 

6. What is the difference between the setup of 1 and 2 (a) to (c) apart from the fact that SOVL-E and TestL are not included in 2?

7. If SOTL-E is the average training loss of the final epoch, why not call it that? (I understand that it is still the sum of losses for all the batches but as it not across epochs it is misleading).

As this is paper is proposing something that is fundamentally opposite to what has been studied widely thus far, it requires a lot more scrutiny. I do not think we can accept it with just empirical results and the theoretical motivation currently provided.

_____________________________
_____________________________
Post Rebuttal:

Thank you for replying to all of my questions.. 

Plugging in your new metric to DARTS seems to be promising, especially if it alleviates the DARTS collapse problem. Given that the community is more interested in one-shot NAS algorithms, this might be worthwhile pursuing

From the new plot in Figure 4 and NAS experiment in Figure 5, it is evident that the sum of training loss is able to rank the networks more effectively in the first 50 epochs. So one could use SOTL-E for early stopping rather than validation accuracy. This would also be effective in hyperband where the architectures are discarded after training them for very few epochs.



review_title: Official Blind Review #3

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a simple model-free method to estimate the generalization performance of deep neural architectures based on their early training losses. The proposed method uses the sum of training losses during training to estimate the performance and is motivated by recent empirical and theoretical results. The experimental results show that the proposed estimator outperforms the existing methods that predict the performance ranking among architectures.

Pros
- The proposed approach is simple yet shows better performance than the existing estimator.
- This paper is discussed from both theoretical and empirical perspectives.

Cons
- I am wondering how the proposed estimator can be used in recent gradient-based NAS methods. For instance, DARTS optimizes architecture parameters during the search phase and pick up the top two operations that have the highest values after optimization. Is it possible to use the proposed estimator for DARTS optimization (i.e. recent gradient-based methods) and to speed up the optimization? Also, although One-shot NAS based on random sampling (e.g [1]) achieves good performance, can we apply the proposed method to such methods? I would like to know the scope of the application of the proposed method.
- In the first experiment, this paper randomly samples 100 architectures from the DARTS search space and evaluate the proposed estimator based on them. I would like to see the behavior of the proposed method with more samples to get a more accurate understanding because the search space of DARTS is much large.
- Regarding Figure 5, in the case of the regularized evolution (RE), I would like to know how much the speed of the optimization is improved. For instance, it would be nice to provide how much the proposed method can speed up the optimization to achieve the same performance reported in their paper. I am also interested in how fast it is compared with recent gradient-based NAS methods.
[1] G. Bender+, Understanding and Simplifying One-Shot Architecture Search, ICML, 2018

Overall, this paper proposes a simple yet effective method to estimate the generalization performance among deep neural networks and it is motivated by both empirical and theoretical aspects. However, there are some unclear points to be clarified for the publication as described above.
review_title: Review of Revisiting the Train Loss: an Efficient Performance Estimator for Neural Architecture Search

final_score: rating: 3: Clear rejection
review_content: ###############################################################

Summary:

This paper provides a new method for estimating the generalization performance of neural architectures. This method used the sum of training loss as a criterion. The paper gave some intuitions about the method from the perspective of Bayeian model selection. 

###############################################################

Reason for Score:

Using training loss to improve generalization performance is unreasonable. And the paper didn't give some convincing reason to this method. The method has no theoretical guarantee, and its analogy with Bayesian model selection seems problematic.

###############################################################

cons:

1, The method in this paper purely used training loss as a criterion for generalization performance. This is unreasonable. And the paper didn't give some convincing reason to this method.

2, The analogy with the Bayesian model selection is problematic. In Bayesian model selection, the parameter in the following step is the posterior estimator based on previous data. This paper think of the SGD optimizer as a way to find the posterior estimator. This is problematic.
review_title: Using training loss to improve generalization performance is unreasonable

Meta-review: The paper proposes to use the sum of training losses during training, or a variant where the sum of training losses begins to be computed after the first E epochs, to estimate the generalization performance of the corresponding network. Although the results seem promising for query-based NAS strategies, the reviewers agree that as the paper proposes something that is fundamentally opposite to the common practice, it requires more careful and thorough analysis. Besides, while the connection made by authors to the Bayesian marginal likelihood is interesting, it's not a rigorous argument that convinces the audience about the applicability of the proposed method. I strongly encourage the authors to add more analysis and discussion to the revised version to strengthen their claim and clarify its scope.


Decision: Reject