Scalable Diverse Model Selection for Accessible Transfer Learning

1 Introduction . Deep Neural Networks ( DNNs ) have shown to be very capable of solving a wide variety of visual tasks . However , these networks often require large amounts of data and training time to perform well , limiting the accessibility of deep learning for computer vision . One approach to alleviate this problem is to employ transfer learning , commonly by fine-tuning an off-the-shelf model on the desired task . With the increasing number of off-the-shelf models available spanning different tasks , datasets , training methods , and architectures , choosing the best model from which to transfer is a challenging endeavor . A common heuristic in computer vision has been to use models pretrained on ImageNet [ 7 ] ( specifically the ILSVRC challenge data ) , but more recent work is starting to expose weaknesses in the generalization performance of ImageNet features [ 39 , 28 , 40 ] , and many of these works find that other pretraining datasets may perform better for some target tasks . Aside from source dataset , which architecture to select isn ’ t clear ( when comparing similarly fast models ) . One could theoretically train several transfers from a diverse set of pretrained weights and keep the best performing model , but this isn ’ t feasible in most practical applications where compute is limited . This motivates the need for a model selection method that could query a large bank of existing pretrained models ( of which several already exist , e.g. , [ 36 , 56 , 18 ] ) with a small subset of the target data and return a set of weights which performs well when fine-tuned on all target data . Because of the sheer quantity of pretrained weights available to download off the internet today , such a model bank has the potential to be massive and cover a wide variety of tasks , datasets , and architectures . Any ∗Equal Contribution †https : //dbolya.github.io/parc/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . selection method that intends to operate on such a massive library of weights would thus need two important properties : it would need to be scalable in order to accommodate 100s to 1000s of source models , and it would need to perform well on a diverse set of input weights due to the wide variety of models available . In this paper , we formalize this setting as Scalable Diverse Model Selection . Currently , there are two main lines of work that could address this scenario . The first is model selection [ 11 , 46 , 10 ] , which attempts to select a viable transfer from a suite of arbitrary pretrained models . However , existing approaches require an initial model trained on the target data to suggest the transfer ( which limits accessibility ) and are not very successful when comparing across architectures ( see Tab . 1 ) . The second line of work is in transferability estimation [ 4 , 52 , 34 ] , which attempts to predict how well a source model will transfer to a target task . While this might seem similar to model selection , the two tasks are subtly different in evaluation . Transferability estimation methods usually fix one source model and vary the target task ( i.e. , “ For my source model , which task would it transfer to the best ? ” ) , while model selection methods do the opposite : fix a target task and vary the source models ( i.e. , “ For my target task , which source model will transfer the best ? ” ) . While it might seem the same methods could work for either case , this turns out to not be the case ( see Tab . 2 ) . Contributions . We formalize the task of Scalable Diverse Model Selection , which intends to make deep learning for computer vision more accessible . While other papers might have explored aspects of this space already , we standardize it by introducing several tools and benchmarks for evaluating model selection methods in this setting . First , we provide a controlled environment that includes exhaustively trained transfers from 8 source datasets to 6 target datasets across 4 different commonly used architectures for a total of 168 ground truth transfers for analysis ( Sec . 3 ) . We show that current state-of-the-art transferability and model selection methods fail to beat simple baselines in this new setting ( Tab . 1 ) . We then analyze why this is the case and provide techniques to improve performance ( Sec . 4 ) . Using insights from this analysis , we develop PARC , a method that outperforms other methods on this benchmark ( Sec . 5 ) . Finally , we show that these results generalize to a larger experiment with an extra dataset and 33 additional off-the-shelf pretrained models downloaded from the internet ( for a total of 65 source models and 423 transfers ) and by extending PARC to object detection ( Sec . 6 ) . We have released all benchmarks and evaluation code at https : //dbolya.github.io/parc/ in hopes to further the development of this promising area of research . 2 Related Work . In this paper , we introduce a new task for Scalable Diverse Model Selection , which attempts to make transfer learning more accessible by selecting the best pretrained model for a downstream task from a massive bank of off-the-shelf models . This is adjacent to several fields such as accessible transfer learning , transferability estimation , and Taskonomy model selection . Transfer Learning . Transfer learning is the act of using a model trained in one setting to boost the performance or speed up the training of a model in another setting , and is an extensively studied field in deep learning [ 49 ] . Several approaches exist for transfer learning from a source model to a target dataset , the most popular of which include fine-tuning a new head to the target domain [ 9 , 43 , 60 , 39 ] , and fine-tuning the entire network [ 2 , 17 ] . While there are more sophisticated methods for fine-tuning ( e.g. , [ 19 , 37 ] ) , we study fine-tuning the entire network , as it ’ s simple and widely adopted . Accessible Transfer Leaning . A few works have tried to make transfer learning more accessible . Neural Data Server [ 57 ] allows users to augment their own data with similar data indexed from several massive image datasets . While helpful in a limited data environment , this requires the user to have additional compute in order to train with the extra images . Iterating on this idea , Scalable Transfer Learning with Expert Models [ 38 ] suggests pretrained models to the user instead , relaxing the compute requirement . However , the focus of the paper is primarily on creating these pretrained “ expert ” models and not in selecting between them . We argue that there are several “ expert ” models already widely available and trained with a large amount of data ( e.g. , [ 47 , 33 , 39 ] ) , and thus focus on the model selection aspect of this problem . Both of these works use simple baselines to guide their model selection : Neural Data Server [ 57 ] uses the accuracy of a logistic classifier on the source features fit to the target task , while Scalable Transfer Learning [ 38 ] uses nearest neighbors with hold-one-out cross-validation . We benchmark both baseline techniques in our setting . Transferability Estimation . There are several works that attempt to predict how well a model will transfer to new tasks , ranging from methods that attempt to assess a models capacity for transfer [ 23 , 28 , 40 ] to those that attempt to predict transfer accuracy [ 12 , 3 , 45 ] . Others attempt to predict the gap in generalization between training and test time [ 24 , 53 , 25 ] . There has also been a recent line of work that directly attempts to estimate transfer learning accuracy given a source model and target dataset [ 4 , 52 , 34 ] . This line of work is the most applicable to our setting because the only assumption they make on the source model is that it was trained on classification . While not ideal , this allows us to benchmark these methods ( H-Score [ 4 ] , NCE [ 52 ] , and LEEP [ 34 ] ) in our setting . Otherwise , Deshpande et al . [ 8 ] propose perhaps what is most directly applicable to our work . Their setting is very similar ( and fairly concurrent ) , though not as diverse and with no restrictions on evaluation speed . In addition , LogMe [ 58 ] is concurrent work that also focuses on practical transferability estimation , but they don ’ t release their benchmark and their setting is more narrow . Taskonomy Model Selection . The Taskonomy [ 59 ] models and dataset has been used to benchmark a previous line of model selection algorithms [ 11 , 46 , 10 ] . Taskonomy attempts to model similarities between tasks by how well models transferring from one task to another perform after fine-tuning . This makes it a natural test-bed for model selection evaluation , as it contains a large number of pretrained transfers to use as a benchmark . However , using Taskonomy as a benchmark is incomplete . All Taskonomy models are trained on the same data ( with different labels ) and follow roughly the same architecture , only varying the source and target task . This means that the benchmark doesn ’ t test robustness to source model diversity , which is a core tenant of this work . Furthermore , typical model selection works on Taskonomy require a network trained on the target data to suggest transfers , which should be avoided to make transfer learning more accessible . We benchmark the performance of RSA [ 11 ] and DDS [ 10 ] , as they are the best performing methods on Taskonomy model selection . There are other works in this space such as [ 6 ] and [ 1 ] , but the former scales poorly with the number of sources , and the latter can not compare across architectures by design , so we don ’ t include either in our experiments . 3 Scalable Diverse Model Selection . In this work , we address the problem of model selection for transfer learning but through an expressly practical lens . The goal of model selection from a practitioner ’ s point of view is to find a pretrained off-the-shelf model that will perform well after fine-tuning on their data . In order for a model selection method to perform well in this setting , the models it selects from need to be diverse ( i.e. , cover a wide variety of source datasets , architectures , and pretraining tasks ) , and the selection method needs to be scalable ( since the number of off-the-shelf models available today is massive and growing ) . In order to provide a realistic transfer learning model selection scenario , we propose to select a model from a large source model bank ( over 100 models spanning several datasets and architectures ) that transfers well to a target training set after full model fine-tuning . The naive approach to this problem would be to simply fully fine-tune a transfer from each source model to the target training set , but this would of course be computationally infeasible . We ’ d like to work in the practical setting where we don ’ t train any extra models , so we ’ d like a computationally efficient transferability estimation method to predict which source models would transfer well . For the same reason , it ’ s also infeasible to use the entire target training set for this estimate , since extracting all the features for 100 different source models is akin to training a new model for 100 epochs ( though without the backward pass ) . Thus , we restrict the method to only use a small subset of the target training data for its estimate . In this scenario , the target dataset is fixed while the source models can vary in dataset , architecture , and task . Previous work in model selection and transferability estimation typically only vary one of these aspects when evaluating their method ( i.e. , just task [ 59 , 4 , 11 , 10 ] , just dataset [ 34 , 52 ] , or just architecture [ 34 ] , but not all three at once ) . It ’ s unclear how well , if at all , these methods benchmarked by varying only one factor will perform when all of these factors can change . Thus , we ’ ve set out to test these methods in this much more challenging setting .

Summary: The paper studies methods to select source models from a diverse set, considering both the quality of the proxy for downstream performance, and the speed of computing the proxy. The paper introduces a new benchmark for evaluating the ability to choose from diverse source models, studies a number of existing methods, and highlights some shortcomings. It then proposes some tweaks to improve these. Finally, the paper proposes a new proxy PARC, which performs well on their benchmark (also using the aforementioned tricks), and also on an extended evaluation on crowd-sourced models.