Package: opendelta
An Open-Source Framework for Paramter-Efficient Tuning (Delta Tuning).Overview•Installation•Basic Usage•Docs•Performance•OverviewOpenDelta is a toolkit for parameter-efficient tuning methods (we dub it asdelta tuning), by which users could flexibly assign (or add) a small amount parameters to update while keeping the most paramters frozen. By using OpenDelta, users could easily implement prefix-tuning, adapters, Lora, or any other types of delta tuning with preferred PTMs.The latest version of OpenDelta is tested on Python==3.8.13, PyTorch==1.12.1, transformers==4.22.2. Other versions are likely to be supported as well. If you encounter bugs when using your own package versions, please raise an issue, we will look into it as soon as possible.A demo of using Opendelta to modify the PLM (E.g., BART).News2022.10.25Release v0.3.2. SupportBMTrain! Improve docs. Add inspect utilities.2022.10.14Release v0.3.0. We make the usage of default configurations of each delta tuning methods (i.e., the position they are attached) more friendly! If a custom model has our supported models as submodules inside, the default configuration is also available. Other key changes can be seen inUpdate Log2022.10.10Merge a long-developed branch v0.2.4 into the master branch. Key updates are (1) the an example unifying the delta tuning paradigm and the prompt-tuning paradigm; (2) and support forDelta Center, whose webpage is still under construction. Details can be seen inUpdate Log2022.03.24We notice several bugs in Soft Prompt Tuning and Prefix Tuning, mainly due to their need to customize attention ids, token_type_ids, we are fixing it! Currently, please use the other methods since they are stabler and better in performance.2022.03.20Add acolab exampleto illustrate efficient training and space-saving multitask-serving.2022.03.20A new pip version released.2022.02.16Supportregular expressionin named-based addressing.Installationcreate a virtualenv (optional)condacreate-nopendelta_envpython=3.8
condaactivateopendelta_env2 install the lastest versionpipinstallgit+https://github.com/thunlp/OpenDelta.gitorinstall the lastest pip version (more stable)pipinstallopendeltaorbuild from sourcegitclonegit@github.com:thunlp/OpenDelta.gitcdOpenDelta
pythonsetup.pyinstall# python setup.py develop # if you want to do some modifications on the code for your research:Must TryThe following codes and comments walk you through the key functionality of OpenDelta. It is also inmust_try.pyandmust_try.ipynb in colab.# use tranformers as usual.fromtransformersimportAutoModelForSeq2SeqLM,AutoTokenizert5=AutoModelForSeq2SeqLM.from_pretrained("t5-large")t5_tokenizer=AutoTokenizer.from_pretrained("t5-large")# A running exampleinputs_ids=t5_tokenizer.encode("Is Harry Poter wrtten by JKrowling",return_tensors="pt")t5_tokenizer.decode(t5.generate(inputs_ids)[0])# >>> '<pad><extra_id_0>? Is it Harry Potter?
'# use existing delta modelsfromopendeltaimportAutoDeltaModel,AutoDeltaConfig# use existing delta models from DeltaCenterdelta=AutoDeltaModel.from_finetuned("thunlp/Spelling_Correction_T5_LRAdapter_demo",backbone_model=t5)# freeze the whole backbone model except the delta models.delta.freeze_module()# visualize the changedelta.log()t5_tokenizer.decode(t5.generate(inputs_ids)[0])# >>> <pad> Is Harry Potter written by JK Rowling?
# Now save merely the delta models, not the whole backbone model, to tmp/delta.save_finetuned(".tmp")importos;os.listdir(".tmp")# >>>  The state dict size is 1.443 MB# >>>  We encourage users to push their final and public models to delta center to share them with the community!# reload the model from local url and add it to pre-trained T5.t5=AutoModelForSeq2SeqLM.from_pretrained("t5-large")delta1=AutoDeltaModel.from_finetuned(".tmp",backbone_model=t5)importshutil;shutil.rmtree(".tmp")# don't forget to remove the tmp files.t5_tokenizer.decode(t5.generate(inputs_ids)[0])# >>> <pad> Is Harry Potter written by JK Rowling?
# detach the delta models, the model returns to the unmodified status.delta1.detach()t5_tokenizer.decode(t5.generate(inputs_ids)[0])# >>> '<pad><extra_id_0>? Is it Harry Potter?
'# use default configuration for cunstomized wrapped models which have PLMs inside. This is a common need for users.importtorch.nnasnnclassWrappedModel(nn.Module):def__init__(self,inner_model):super().__init__()self.inner=inner_modeldefforward(self,*args,**kwargs):returnself.inner(*args,**kwargs)wrapped_model=WrappedModel(WrappedModel(t5))# say we use LoRAdelta_config=AutoDeltaConfig.from_dict({"delta_type":"lora"})delta2=AutoDeltaModel.from_config(delta_config,backbone_model=wrapped_model)delta2.log()# >>> root#       -- inner#          -- inner#             ...#             ... lora_A:[8,1024], lora_B:[1024,8]delta2.detach()# use a not default configuration# say we add lora to the last four layer of the decoder of t5, with lora rank=5delta_config3=AutoDeltaConfig.from_dict({"delta_type":"lora","modified_modules":["[r]decoder.*((20)|(21)|(22)|(23)).*DenseReluDense\.wi"],"lora_r":5})delta3=AutoDeltaModel.from_config(delta_config3,backbone_model=wrapped_model)delta3.log()Verified Default ConfigurationsYou can try to use OpenDelta onanybackbone models based on PyTorch.However, with small chances that the interface of the submodules of the backbone model is not supported. Therefore we verified some commonly
used models that OpenDelta are sure to support.We will keep testing more and more emerging models.Pull requests are welcomed when you successfully apply OpenDelta on your own backbone model.