Package: hyped
:boom: hypedA collection of data pipelines to ease the training of transformer modelsinstallationThe package can be installed from PyPI:pipinstall-UhypedAlternatively you can also install the package from source by cloning this repositorygitclonehttps://github.com/ndoll1998/hyped.gitcdhyped
pipinstall-e.dependencieshypedcombines the 🤗 huggingface libraries to implement pipelines that go from data preparation to model training and finally evaluation. The following table shows which library is used as backend to which pipeline stage.stagebackendversiondata preparationdatasets>=2.12.0model trainingadapter-transformers>=3.2.1model evaluationevaluate>=0.4.0usageThe purpose ofhypedis to train complex transformer models with minimal effort. The typical ML pipeline consists of three main stages, namely adata preparation, amodel trainingand amodel evaluationstage. Each one of these stages can be executed by thehypedcommand line interface.The following demonstates how to train a text classification model on theimdbdataset by going through all three stages of the ML pipeline (fromexamples/run.sh).# data preparation stage# runs the data preparation pipeline specified in the# configuration file and saves the prepared dataset# to the specified locationpython-mhyped.stages.prepare\-cexamples/text_cls/imdb/distilbert_data.json\-ooutput/distilbert_data# model training stage# trains a model on the prepared data generated by the# previous stagepython-mhyped.stages.train\-cexamples/text_cls/imdb/distilbert_run.json\-doutput/distilbert_data\-ooutput/model# model evaluation stage# evaluates the trained model on the test split of the# prepared dataset generated earlierpython-mhyped.stages.evaluate\-cexamples/text_cls/imdb/distilbert_run.json\-doutput/distilbert_data\-moutput/model/best-modelfeatureshypedcurrently implements all components required to train and evaluate models for the following NLP tasks:Text ClassificationNamed Entity RecognitionMulti-label ClassificationCausal Language Modelingadaptershypedis build uponadapter-transformers, which allows the configuration of a variety of model architectures. See the following excerpt for how to configure an adapter model in the run configuration (fromexamples/cls/imdb/distilbert_run.json):{"model":{"pretrained_ckpt":"distilbert-base-uncased","adapter_name":"imdb","adapter":{"train_adapter":true,"adapter_config":"pfeiffer"},"heads":{"cls":{"head_type":"hyped-cls-head","label_column":"labels"}}}}Theadapterfield in the above configuration mirrors theAdapterArgumentsclass and can specify all it's values (e.g.load_adapterfor loading pretrained adapters). Furthermore, by settingtrain_adapterto True, only the paramters of the adapter and head are trained, while the pretrained encoder parameters are frozen.Alternatively, the adapter configuration can also be omitted, in which case no adapter is used and the encoder output is directly passed to the prediction heads.dvchypedis designed to be easily integratable into existing data pipelines. Thus, it fits well withdvcpipelines. Seeexamples/dvcfor an example setup.deepspeedhypedsupports distributed training allowing for both pre-training and fine-tuning of very large language models. Seeexamples/lm/run.jsonfor an example configuration. To lauch a distributed training you need to usehyped's command line interface:deepspeed--no_pythonhypedtrain-cconfig.json-ddata/-omodel/roadmaphypedis currently in its early stages of development. Over time, the goal is to expand its applicability to a diverse range of tasks and setups. The following features are planned for future development:support adapter trainingsupport transformers modelssupport more tasksMasked Language ModelingCausal Language Modelingnested Named Entity RecognitionQuestion Answeringsupport multi-modal encodersLayoutLMLiLTsupport distributed training/inferencedeepspeedpytorch DDP and FSDP