Title: A Non-vacuous Test Error Guarantee for Deep Learning without Altering the Model
Primary area: learning theory
Keywords: Deep Learning, generalization bounds, Error bounds, Model-dependent bounds, non-vacuous

Abstract: Deep neural networks (NN) with millions or billions of parameters can perform really well on unseen data, after being trained from a finite training set. Various prior theories have been developed to explain such excellent ability of NNs, but do not provide a meaningful bound on the test error. Some recent theories, based on PAC-Bayes and mutual information, are non-vacuous and hence promising. However, they often require a stringent assumption and extensive modification (e.g. compression, quantization) to the trained model of interest. Therefore, those prior theories provide a guarantee for the modified versions only. In this paper, we propose two novel bounds on the test error of a model. One of our bounds can be exactly computable from the training set only, without modification to the model, and hence provides a theoretical guarantee for a trained model. Those bounds are verified on a large class of modern NNs, pretrained by Pytorch on the ImageNet dataset. The exactly computable bound is found to be non-vacuous. To the best of our knowledge, this is the first non-vacuous  bound at this large scale, without any modification to the pretrained models.