The ℓ∞ norm constraint on the optimization process with AdamW has significant implications for the generalization performance of trained models. By constraining the ℓ∞ norm of the parameters, AdamW implicitly regularizes the model towards solutions that have smaller values in the ℓ∞ norm. This regularization can lead to improved generalization performance by preventing the model from overfitting to the training data.
The analysis shows that AdamW converges to KKT points of the constrained optimization problem, where the ℓ∞ norm of the parameters is bounded by the weight decay factor. This constraint ensures that the model does not have excessively large parameter values, which can help prevent overfitting and improve the model's ability to generalize to unseen data. By implicitly biasing the optimization process towards solutions with smaller ℓ∞ norms, AdamW can lead to models that generalize better and are less prone to overfitting.