Package: grouphug
grouphugGroupHug is a library with extensions to 🤗 transformers for multitask language modelling.
In addition, it contains utilities that ease data preparation, training, and inference.OverviewThe package is optimized for training a single language model to make quick and robust predictions for a wide variety of related tasks at once,
as well as to investigate the regularizing effect of training a language modelling task at the same time.You can train on multiple datasets, with each dataset containing an arbitrary subset of your tasks. Supported tasks include:A single language modelling task (Masked language modelling, Masked token detection, Causal language modelling).The default collator included handles most preprocessing for these heads automatically.Any number of classification tasks, including single- and multi-label classification and regressionA utility function that automatically creates a classification head from your data.Additional options such as hidden layer size, additional input variables, and class weights.You can also define your own model heads.Quick StartThe project is based on Python 3.8+ and PyTorch 1.10+. To install it, simply use:pip install grouphugDocumentationDocumentation can be generated from docstrings usingmake htmlin thedocsdirectory, but this is not yet on a hosted site.Example usageimportpandasaspdfromdatasetsimportload_datasetfromtransformersimportAutoTokenizerfromgrouphugimportAutoMultiTaskModel,ClassificationHeadConfig,DatasetFormatter,LMHeadConfig,MultiTaskTrainer# load some data. 'label' gets renamed in huggingface, so is better avoided as a feature name.task_one=load_dataset("tweet_eval",'emoji').rename_column("label","tweet_label")both_tasks=pd.DataFrame({"text":["yay :)","booo!"],"sentiment":["pos","neg"],"tweet_label":[0,14]})# create a tokenizerbase_model="prajjwal1/bert-tiny"tokenizer=AutoTokenizer.from_pretrained(base_model)# preprocess your data: tokenization, preparing class variablesformatter=DatasetFormatter().tokenize().encode("sentiment")# data converted to a DatasetCollection: essentially a dict of DatasetDictdata=formatter.apply({"one":task_one,"both":both_tasks},tokenizer=tokenizer,test_size=0.05)# define which model heads you would likehead_configs=[LMHeadConfig(weight=0.1),# default is BERT-style masked language modellingClassificationHeadConfig.from_data(data,"sentiment"),# detects dimensions and typeClassificationHeadConfig.from_data(data,"tweet_label"),# detects dimensions and type]# create the model, optionally saving the tokenizer and formatter along with itmodel=AutoMultiTaskModel.from_pretrained(base_model,head_configs,formatter=formatter,tokenizer=tokenizer)# create the trainertrainer=MultiTaskTrainer(model=model,tokenizer=tokenizer,train_data=data[:,"train"],eval_data=data[["one"],"test"],eval_heads={"one":["tweet_label"]},# limit evaluation to one classification task)trainer.train()TutorialsSeeexamplesfor a few notebooks that demonstrate the key features.Supported ModelsThe package has support for the following base models:Bert, DistilBert, Roberta/DistilRoberta, XLM-RobertaDeberta/DebertaV2ElectraGPT2, GPT-J, GPT-NeoX, OPTExtending it to support other models is possible by simply inheriting from_BaseMultiTaskModel, although language modelling head weights may not always load.LimitationsThe package only supports PyTorch, and will not work with other frameworks. There are no plans to change this.Grouphug was developed and tested with 🤗 transformers 4.19-4.22. We will aim to test and keep compatibility with the latest version, but it is still recommended to lock the latest working versions.See thecontributing pageif you are interested in contributing.Licensegrouphug was initially developed atChatdeskand is licensed under the Apache 2license.