Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks
Year: 2024

Hyperparameter tuning, the process of searching for suitable hyperparameters, becomes more difficult as the computing resources required to train neural networks continue to grow.This topic continues to receive little attention and discussion-much of it hearsaydespite its obvious importance.We attempt to formalize hyperparameter sensitivity using two metrics: similarity-based sensitivity and performance-based sensitivity.We then use these metrics to quantify two such claims: (1) transformers are