Scalable Diverse Model Selection for Accessible Transfer Learning

1 Introduction . Deep Neural Networks ( DNNs ) have shown to be very capable of solving a wide variety of visual tasks . However , these networks often require large amounts of data and training time to perform well , limiting the accessibility of deep learning for computer vision . One approach to alleviate this problem is to employ transfer learning , commonly by fine-tuning an off-the-shelf model on the desired task . With the increasing number of off-the-shelf models available spanning different tasks , datasets , training methods , and architectures , choosing the best model from which to transfer is a challenging endeavor . A common heuristic in computer vision has been to use models pretrained on ImageNet [ 7 ] ( specifically the ILSVRC challenge data ) , but more recent work is starting to expose weaknesses in the generalization performance of ImageNet features [ 39 , 28 , 40 ] , and many of these works find that other pretraining datasets may perform better for some target tasks . Aside from source dataset , which architecture to select isn ’ t clear ( when comparing similarly fast models ) . One could theoretically train several transfers from a diverse set of pretrained weights and keep the best performing model , but this isn ’ t feasible in most practical applications where compute is limited . This motivates the need for a model selection method that could query a large bank of existing pretrained models ( of which several already exist , e.g. , [ 36 , 56 , 18 ] ) with a small subset of the target data and return a set of weights which performs well when fine-tuned on all target data . Because of the sheer quantity of pretrained weights available to download off the internet today , such a model bank has the potential to be massive and cover a wide variety of tasks , datasets , and architectures . Any ∗Equal Contribution †https : //dbolya.github.io/parc/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . selection method that intends to operate on such a massive library of weights would thus need two important properties : it would need to be scalable in order to accommodate 100s to 1000s of source models , and it would need to perform well on a diverse set of input weights due to the wide variety of models available . In this paper , we formalize this setting as Scalable Diverse Model Selection . Currently , there are two main lines of work that could address this scenario . The first is model selection [ 11 , 46 , 10 ] , which attempts to select a viable transfer from a suite of arbitrary pretrained models . However , existing approaches require an initial model trained on the target data to suggest the transfer ( which limits accessibility ) and are not very successful when comparing across architectures ( see Tab . 1 ) . The second line of work is in transferability estimation [ 4 , 52 , 34 ] , which attempts to predict how well a source model will transfer to a target task . While this might seem similar to model selection , the two tasks are subtly different in evaluation . Transferability estimation methods usually fix one source model and vary the target task ( i.e. , “ For my source model , which task would it transfer to the best ? ” ) , while model selection methods do the opposite : fix a target task and vary the source models ( i.e. , “ For my target task , which source model will transfer the best ? ” ) . While it might seem the same methods could work for either case , this turns out to not be the case ( see Tab . 2 ) . Contributions . We formalize the task of Scalable Diverse Model Selection , which intends to make deep learning for computer vision more accessible . While other papers might have explored aspects of this space already , we standardize it by introducing several tools and benchmarks for evaluating model selection methods in this setting . First , we provide a controlled environment that includes exhaustively trained transfers from 8 source datasets to 6 target datasets across 4 different commonly used architectures for a total of 168 ground truth transfers for analysis ( Sec . 3 ) . We show that current state-of-the-art transferability and model selection methods fail to beat simple baselines in this new setting ( Tab . 1 ) . We then analyze why this is the case and provide techniques to improve performance ( Sec . 4 ) . Using insights from this analysis , we develop PARC , a method that outperforms other methods on this benchmark ( Sec . 5 ) . Finally , we show that these results generalize to a larger experiment with an extra dataset and 33 additional off-the-shelf pretrained models downloaded from the internet ( for a total of 65 source models and 423 transfers ) and by extending PARC to object detection ( Sec . 6 ) . We have released all benchmarks and evaluation code at https : //dbolya.github.io/parc/ in hopes to further the development of this promising area of research . 2 Related Work . In this paper , we introduce a new task for Scalable Diverse Model Selection , which attempts to make transfer learning more accessible by selecting the best pretrained model for a downstream task from a massive bank of off-the-shelf models . This is adjacent to several fields such as accessible transfer learning , transferability estimation , and Taskonomy model selection . Transfer Learning . Transfer learning is the act of using a model trained in one setting to boost the performance or speed up the training of a model in another setting , and is an extensively studied field in deep learning [ 49 ] . Several approaches exist for transfer learning from a source model to a target dataset , the most popular of which include fine-tuning a new head to the target domain [ 9 , 43 , 60 , 39 ] , and fine-tuning the entire network [ 2 , 17 ] . While there are more sophisticated methods for fine-tuning ( e.g. , [ 19 , 37 ] ) , we study fine-tuning the entire network , as it ’ s simple and widely adopted . Accessible Transfer Leaning . A few works have tried to make transfer learning more accessible . Neural Data Server [ 57 ] allows users to augment their own data with similar data indexed from several massive image datasets . While helpful in a limited data environment , this requires the user to have additional compute in order to train with the extra images . Iterating on this idea , Scalable Transfer Learning with Expert Models [ 38 ] suggests pretrained models to the user instead , relaxing the compute requirement . However , the focus of the paper is primarily on creating these pretrained “ expert ” models and not in selecting between them . We argue that there are several “ expert ” models already widely available and trained with a large amount of data ( e.g. , [ 47 , 33 , 39 ] ) , and thus focus on the model selection aspect of this problem . Both of these works use simple baselines to guide their model selection : Neural Data Server [ 57 ] uses the accuracy of a logistic classifier on the source features fit to the target task , while Scalable Transfer Learning [ 38 ] uses nearest neighbors with hold-one-out cross-validation . We benchmark both baseline techniques in our setting . Transferability Estimation . There are several works that attempt to predict how well a model will transfer to new tasks , ranging from methods that attempt to assess a models capacity for transfer [ 23 , 28 , 40 ] to those that attempt to predict transfer accuracy [ 12 , 3 , 45 ] . Others attempt to predict the gap in generalization between training and test time [ 24 , 53 , 25 ] . There has also been a recent line of work that directly attempts to estimate transfer learning accuracy given a source model and target dataset [ 4 , 52 , 34 ] . This line of work is the most applicable to our setting because the only assumption they make on the source model is that it was trained on classification . While not ideal , this allows us to benchmark these methods ( H-Score [ 4 ] , NCE [ 52 ] , and LEEP [ 34 ] ) in our setting . Otherwise , Deshpande et al . [ 8 ] propose perhaps what is most directly applicable to our work . Their setting is very similar ( and fairly concurrent ) , though not as diverse and with no restrictions on evaluation speed . In addition , LogMe [ 58 ] is concurrent work that also focuses on practical transferability estimation , but they don ’ t release their benchmark and their setting is more narrow . Taskonomy Model Selection . The Taskonomy [ 59 ] models and dataset has been used to benchmark a previous line of model selection algorithms [ 11 , 46 , 10 ] . Taskonomy attempts to model similarities between tasks by how well models transferring from one task to another perform after fine-tuning . This makes it a natural test-bed for model selection evaluation , as it contains a large number of pretrained transfers to use as a benchmark . However , using Taskonomy as a benchmark is incomplete . All Taskonomy models are trained on the same data ( with different labels ) and follow roughly the same architecture , only varying the source and target task . This means that the benchmark doesn ’ t test robustness to source model diversity , which is a core tenant of this work . Furthermore , typical model selection works on Taskonomy require a network trained on the target data to suggest transfers , which should be avoided to make transfer learning more accessible . We benchmark the performance of RSA [ 11 ] and DDS [ 10 ] , as they are the best performing methods on Taskonomy model selection . There are other works in this space such as [ 6 ] and [ 1 ] , but the former scales poorly with the number of sources , and the latter can not compare across architectures by design , so we don ’ t include either in our experiments . 3 Scalable Diverse Model Selection . In this work , we address the problem of model selection for transfer learning but through an expressly practical lens . The goal of model selection from a practitioner ’ s point of view is to find a pretrained off-the-shelf model that will perform well after fine-tuning on their data . In order for a model selection method to perform well in this setting , the models it selects from need to be diverse ( i.e. , cover a wide variety of source datasets , architectures , and pretraining tasks ) , and the selection method needs to be scalable ( since the number of off-the-shelf models available today is massive and growing ) . In order to provide a realistic transfer learning model selection scenario , we propose to select a model from a large source model bank ( over 100 models spanning several datasets and architectures ) that transfers well to a target training set after full model fine-tuning . The naive approach to this problem would be to simply fully fine-tune a transfer from each source model to the target training set , but this would of course be computationally infeasible . We ’ d like to work in the practical setting where we don ’ t train any extra models , so we ’ d like a computationally efficient transferability estimation method to predict which source models would transfer well . For the same reason , it ’ s also infeasible to use the entire target training set for this estimate , since extracting all the features for 100 different source models is akin to training a new model for 100 epochs ( though without the backward pass ) . Thus , we restrict the method to only use a small subset of the target training data for its estimate . In this scenario , the target dataset is fixed while the source models can vary in dataset , architecture , and task . Previous work in model selection and transferability estimation typically only vary one of these aspects when evaluating their method ( i.e. , just task [ 59 , 4 , 11 , 10 ] , just dataset [ 34 , 52 ] , or just architecture [ 34 ] , but not all three at once ) . It ’ s unclear how well , if at all , these methods benchmarked by varying only one factor will perform when all of these factors can change . Thus , we ’ ve set out to test these methods in this much more challenging setting .

Summary: The paper studies model selection for transfer learning: given a number of source or upstream models, and a new downstream or target data, how can we choose the best model to finetune? The paper offers two main contributions. First, it proposes a specific benchmark with source models, target tasks, and an evaluation metric to compare different approaches. Second, after testing on a number of methods from the recent literature, it identifies some algorithmic aspects that seem to hurt these methods and proposes an alternative approach called Pairwise Annotation Representation Comparison (PARC) which addresses those. Experiments suggest PARC outperforms the original algorithms.