Relying solely on training-free evaluation methods in Neural Architecture Search (NAS) may have some drawbacks or limitations:
Limited Accuracy: Training-free methods provide quick estimates of architecture performance but may lack precision compared to full training-based evaluations. This limitation can lead to suboptimal architectural choices based on incomplete information.
Generalization Challenges: Training-free evaluations may not capture all nuances and complexities present in real-world datasets, limiting their ability to generalize well across different scenarios.
Overfitting Risk: Without actual training and validation cycles, there is a risk that architectures selected based on training-free evaluations alone may overfit certain aspects of the dataset or fail to perform optimally under varied conditions.
To mitigate these limitations, it is essential to combine both training-free and training-based evaluation methods in an integrated approach like ATLAS presented in the context above. By leveraging the strengths of each method strategically within an anytime NAS framework, researchers can achieve more robust and accurate architectural selections.