Package: ensemble-transformers
Ensemble TransformersEnsembling Hugging Face Transformers made easy!Why Ensemble Transformers?Ensembling is a simple yet powerful way of combining predictions from different models to increase performance. Since multiple models are used to derive a prediction, ensembling offers a way of decreasing variance and increasing robustness. Ensemble Transformers  provides an intuitive interface for ensembling pretrained models available in Hugging Facetransformers.InstallationEnsemble Transformers is available onPyPIand can easily be installed with thepippackage manager.pip install -U pip wheel
pip install ensemble-transformersTo try out the latest features, clone this repository and install from source.git clone https://github.com/jaketae/ensemble-transformers.git
cd ensemble-transformers
pip install -e .QuickstartImport an ensemble model class according to your use case, specify the list of backbone models to use, and run training or inference right away.>>>fromensemble_transformersimportEnsembleModelForSequenceClassification>>>ensemble=EnsembleModelForSequenceClassification.from_multiple_pretrained("bert-base-uncased","distilroberta-base","xlnet-base-cased")>>>batch=["This is a test sentence","This is another test sentence."]>>>output=ensemble(batch)>>>outputEnsembleModelOutput(logits:[tensor([[0.2430,-0.0581],[0.2145,-0.0541]],grad_fn=<AddmmBackward0>),tensor([[-0.0094,-0.0117],[-0.0118,-0.0046]],grad_fn=<AddmmBackward0>),tensor([[-0.0962,-1.1581],[-0.2195,-0.7422]],grad_fn=<AddmmBackward0>)],)>>>stacked_output=ensemble(batch,mean_pool=True)>>>stacked_outputEnsembleModelOutput(logits:tensor([[0.0458,-0.4093],[-0.0056,-0.2670]],grad_fn=<SumBackward1>),)UsageEnsembling with ConfigurationTo declare an ensemble, first create a configuration object specifying the Hugging Face transformers auto class, as well as the list of models to use to create the ensemble.fromensemble_transformersimportEnsembleConfig,EnsembleModelForSequenceClassificationconfig=EnsembleConfig("AutoModelForSequenceClassification",model_names=["bert-base-uncased","distilroberta-base","xlnet-base-cased"])The ensemble model can then be declared viaensemble=EnsembleModelForSequenceClassification(config)Ensembling withfrom_multiple_pretrainedA more convenient way of declaring an ensemble is viafrom_multiple_pretrained, a method similar tofrom_pretrainedin Hugging Face transformers. For instance, to perform text classification, we can use theEnsembleModelForSequenceClassificationclass.fromensemble_transformersimportEnsembleModelForSequenceClassificationensemble=EnsembleModelForSequenceClassification.from_multiple_pretrained("bert-base-uncased","distilroberta-base","xlnet-base-cased")Unlike Hugging Face transformers, which requires users to explicitly declare and initialize a preprocessor (e.g.tokenizer,feature_extractor, orprocessor) separate from the model, Ensemble Transformers automatically detects the preprocessor class and holds it within theEnsembleModelForXclass as an internal attribute. Therefore, you do not have to declare a preprocessor yourself; Ensemble Transformers will do it for you.In the example below, we see that theensembleobject correctly holds 3 tokenizers for each model.>>>len(ensemble.preprocessors)3>>>ensemble.preprocessors[PreTrainedTokenizerFast(name_or_path='bert-base-uncased',vocab_size=30522,model_max_len=512,is_fast=True,padding_side='right',truncation_side='right',special_tokens={'unk_token':'[UNK]','sep_token':'[SEP]','pad_token':'[PAD]','cls_token':'[CLS]','mask_token':'[MASK]'}),PreTrainedTokenizerFast(name_or_path='distilroberta-base',vocab_size=50265,model_max_len=512,is_fast=True,padding_side='right',truncation_side='right',special_tokens={'bos_token':'
','eos_token':'
','unk_token':'<unk>','sep_token':'
','pad_token':'<pad>','cls_token':'
','mask_token':AddedToken("<mask>",rstrip=False,lstrip=True,single_word=False,normalized=False)}),PreTrainedTokenizerFast(name_or_path='xlnet-base-cased',vocab_size=32000,model_max_len=1000000000000000019884624838656,is_fast=True,padding_side='left',truncation_side='right',special_tokens={'bos_token':'
','eos_token':'
','unk_token':'<unk>','sep_token':'<sep>','pad_token':'<pad>','cls_token':'<cls>','mask_token':AddedToken("<mask>",rstrip=False,lstrip=True,single_word=False,normalized=False),'additional_special_tokens':['<eop>','<eod>']})]Heterogenous ModalityFor the majority of use cases, it does not make sense to ensemble models from different modalities, e.g., a language model and an image model. As mentioned, Ensemble Transformers will auto-detect the modality of each model and prevent unintended mixing of models.>>>fromensemble_transformersimportEnsembleConfig>>>config=EnsembleConfig("AutoModelForSequenceClassification",model_names=["bert-base-uncased","google/vit-base-patch16-224-in21k"])Traceback(mostrecentcalllast):File"<stdin>",line1,in<module>File"/Users/jaketae/Documents/Dev/github/ensemble-transformers/ensemble_transformers/config.py",line37,in__init__raiseValueError("Cannot ensemble models of different modalities.")ValueError:Cannotensemblemodelsofdifferentmodalities.Loading Across DevicesBecause ensembling involves multiple models, it is often impossible to load all models onto a single device. To alleviate memory requirements, Ensemble Transformers offers a way of distributing models across different devices. For instance, say you have access to multiple GPU cards and want to load each model onto different GPUs. This can easily be achieved by the following line.ensemble.to_multiple(["cuda:0","cuda:1","cuda:2"])The familiarto(device)method is also supported, and it loads all models onto the same device.ensemble.to("cuda")Forward PropagationTo run forward propagation, simply pass a batch of raw input to the ensemble. In the case of language models, this is just a batch of text.>>>batch=["This is a test sentence","This is another test sentence."]>>>output=ensemble(batch)>>>outputEnsembleModelOutput(logits:[tensor([[0.2430,-0.0581],[0.2145,-0.0541]],grad_fn=<AddmmBackward0>),tensor([[-0.0094,-0.0117],[-0.0118,-0.0046]],grad_fn=<AddmmBackward0>),tensor([[-0.0962,-1.1581],[-0.2195,-0.7422]],grad_fn=<AddmmBackward0>)])>>>output.outputs[SequenceClassifierOutput(loss=None,logits=tensor([[0.1681,-0.3470],[0.1573,-0.1571]],grad_fn=<AddmmBackward0>),hidden_states=None,attentions=None),SequenceClassifierOutput(loss=None,logits=tensor([[0.1388,-0.0711],[0.1429,-0.0841]],grad_fn=<AddmmBackward0>),hidden_states=None,attentions=None),XLNetForSequenceClassificationOutput(loss=None,logits=tensor([[0.5506,0.1506],[0.4308,0.1397]],grad_fn=<AddmmBackward0>),mems=(tensor([[[0.0344,0.0202,0.0261,...,-0.0175,-0.0343,0.0252],[-0.0281,-0.0198,-0.0387,...,-0.0420,-0.0160,-0.0253]],...,[[0.2468,-0.4007,-1.0839,...,-0.2943,-0.3944,0.0605],[0.1970,0.2106,-0.1448,...,-0.6331,-0.0655,0.7427]]])),hidden_states=None,attentions=None)]By default, the ensemble returns aEnsembleModelOutputinstance, which contains all the outputs from each model. The raw outputs from each model is accessible via the.outputsfield. TheEnsembleModelOutputclass also scans across each of the raw output and collects common keys. In the example above, all model outputs contained a.logitsfield, which is why it appears as a field in theoutputinstance.We can also stack or mean-pool the output of each model by togglingmean_pool=Truein the forward call.>>>stacked_output=ensemble(batch,mean_pool=True)>>>stacked_outputEnsembleModelOutput(logits:tensor([[0.0458,-0.4093],[-0.0056,-0.2670]],grad_fn=<SumBackward1>),)If the models are spread across different devices, the result is collected inmain_device, which defaults to the CPU.Preprocessor ArgumentsPreprocessors accept a number of optional arguments. For instance, for simple batching,padding=Trueis used. Moreover, PyTorch models requirereturn_tensors="pt". Ensemble Transformers already ships with minimal, sensible defaults so that it works out-of-the-box. However, for more custom behavior, you can modify thepreprocessor_kwargsargument. The example below demonstrates how to use TensorFlow language models without padding.ensemble(batch,preprocessor_kwargs={"return_tensors":"tf","padding":False})ContributingThis repository is under active development. Any and all issues and pull requests are welcome. If you would prefer, feel free to reach out to me atjaesungtae@gmail.com.LicenseReleased under theMIT License.