Package: adapters
Note: This repository holds the codebase of theAdapterslibrary, which has replacedadapter-transformers. For the legacy codebase, go to:https://github.com/adapter-hub/adapter-transformers-legacy.AdaptersA Unified Library for Parameter-Efficient and Modular Transfer Learningadaptersis an add-on toHuggingFace's Transformerslibrary, integrating adapters into state-of-the-art language models by incorporatingAdapterHub, a central repository for pre-trained adapter modules.Installationadapterscurrently supportsPython 3.8+andPyTorch 1.10+.
Afterinstalling PyTorch, you can installadaptersfrom PyPI ...pip install -U adapters... or from source by cloning the repository:git clone https://github.com/adapter-hub/adapters.git
cd adapters
pip install .Quick TourLoad pre-trained adapters:fromadaptersimportAutoAdapterModelfromtransformersimportAutoTokenizermodel=AutoAdapterModel.from_pretrained("roberta-base")tokenizer=AutoTokenizer.from_pretrained("roberta-base")model.load_adapter("AdapterHub/roberta-base-pf-imdb",source="hf",set_active=True)print(model(**tokenizer("This works great!",return_tensors="pt")).logits)Learn MoreAdapt existing model setups:importadaptersfromtransformersimportAutoModelForSequenceClassificationmodel=AutoModelForSequenceClassification.from_pretrained("t5-base")adapters.init(model)model.add_adapter("my_lora_adapter",config="lora")model.train_adapter("my_lora_adapter")# Your regular training loop...Learn MoreFlexibly configure adapters:fromadaptersimportConfigUnion,PrefixTuningConfig,ParBnConfig,AutoAdapterModelmodel=AutoAdapterModel.from_pretrained("microsoft/deberta-v3-base")adapter_config=ConfigUnion(PrefixTuningConfig(prefix_length=20),ParBnConfig(reduction_factor=4),)model.add_adapter("my_adapter",config=adapter_config,set_active=True)Learn MoreEasily compose adapters in a single model:fromadaptersimportAdapterSetup,AutoAdapterModelimportadapters.compositionasacmodel=AutoAdapterModel.from_pretrained("roberta-base")qc=model.load_adapter("AdapterHub/roberta-base-pf-trec")sent=model.load_adapter("AdapterHub/roberta-base-pf-imdb")withAdapterSetup(ac.Parallel(qc,sent)):print(model(**tokenizer("What is AdapterHub?",return_tensors="pt")))Learn MoreUseful ResourcesHuggingFace's great documentation on getting started withTransformerscan be foundhere.adaptersis fully compatible withTransformers.To get started with adapters, refer to these locations:Colab notebook tutorials, a series notebooks providing an introduction to all the main concepts of (adapter-)transformers and AdapterHubhttps://docs.adapterhub.ml, our documentation on training and using adapters withadaptershttps://adapterhub.mlto explore available pre-trained adapter modules and share your own adaptersExamples folderof this repository containing HuggingFace's example training scripts, many adapted for training adaptersImplemented MethodsCurrently, adapters integrates all architectures and methods listed below:MethodPaper(s)Quick LinksBottleneck adaptersHoulsby et al. (2019)Bapna and Firat (2019)Quickstart,NotebookAdapterFusionPfeiffer et al. (2021)Docs: Training,NotebookMAD-X,Invertible adaptersPfeiffer et al. (2020)NotebookAdapterDropRücklé et al. (2021)NotebookMAD-X 2.0,Embedding trainingPfeiffer et al. (2021)Docs: Embeddings,NotebookPrefix TuningLi and Liang (2021)DocsParallel adapters,Mix-and-Match adaptersHe et al. (2021)DocsCompacterMahabadi et al. (2021)DocsLoRAHu et al. (2021)Docs(IA)^3Liu et al. (2022)DocsUniPELTMao et al. (2022)DocsPrompt TuningLester et al. (2021)DocsSupported ModelsWe currently support the PyTorch versions of all models listed on theModel Overviewpagein our documentation.Developing & ContributingTo get started with developing onAdaptersyourself and learn more about ways to contribute, please seehttps://docs.adapterhub.ml/contributing.html.CitationIf you useAdaptersin your work, please consider citing our library paper:Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning@inproceedings{poth-etal-2023-adapters,
    title = "Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning",
    author = {Poth, Clifton  and
      Sterz, Hannah  and
      Paul, Indraneil  and
      Purkayastha, Sukannya  and
      Engl{\"a}nder, Leon  and
      Imhof, Timo  and
      Vuli{\'c}, Ivan  and
      Ruder, Sebastian  and
      Gurevych, Iryna  and
      Pfeiffer, Jonas},
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-demo.13",
    pages = "149--160",
}Alternatively, for the predecessoradapter-transformers, the Hub infrastructure and adapters uploaded by the AdapterHub team, please consider citing our initial paper:AdapterHub: A Framework for Adapting Transformers@inproceedings{pfeiffer2020AdapterHub,
    title={AdapterHub: A Framework for Adapting Transformers},
    author={Pfeiffer, Jonas and
            R{\"u}ckl{\'e}, Andreas and
            Poth, Clifton and
            Kamath, Aishwarya and
            Vuli{\'c}, Ivan and
            Ruder, Sebastian and
            Cho, Kyunghyun and
            Gurevych, Iryna},
    booktitle={Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
    pages={46--54},
    year={2020}
}