Package: t2t-tuner
t2t-tunerConvenient Text-to-Text Training for Transformerspipinstallt2t-tunerRequires PyTorch: either followPyTorch installation instructionsoruse a PyTorch container.FeaturesEasy training for text-to-text (and text generation) tasksTraining methods/features:Supervised fine-tuningGradient checkpointingModel parallelismSoft prompt tuning (based on this paper)Freeze encoder/decoder/embeddingsMove embeddings to CPUPrint model summaryDeepSpeedBased on the wonderfulHuggingFace Transformerslibrary. Tested on T5 and GPT type of models. In theory, it should work with other models that supportAutoModelForSeq2SeqLMorAutoModelForCausalLMas well.The Trainer in this library here is a higher level interface to work based on HuggingFace'srun_translation.pyscript for text-to-text generation tasks. I decided I want a more more convenient interface for training and inferencing, along with access to things like gradient checkpointing and model parallel to fit larger models - these are already in the HuggingFace library but not exposed in the script. I also added in some features that I wanted (prompt tuning, model summary), integrated it with autoregressive LM training and wrapped it as a single library that can be pip installed.ExamplesTraining Modelsimportt2ttrainer_arguments=t2t.TrainerArguments(model_name_or_path="t5-small",train_file=YOUR_DATASET)trainer=t2t.Trainer(arguments=trainer_arguments)# train without validationtrainer.train(valid=False)For more concrete examples, check out the notebooks linked below:Simple exampleSimple example on ColabSoft Prompt TuningGradient checkpointingModel parallelismData FormatSeq2Seq Training{"translation":{"s":"TEXT","t":"LABEL"}}The format of data is json-lines, following HuggingFace original script. Each example is one line.Define the source and target IDs inTrainingArguments.source_idandTrainingArguments.target_id(defaults tosandt).Include the prefix in the data file, or define the prefix to prepend to the text inTrainingArguments.prefix.Example notebook for data preprocessing from CSV fileAutoregressive LM TrainingAny text file will workTraining Large ModelsThis section will outline how to train large language models (> 1 bil parameters) on relatively simple setups.Some notes for the configurations reported below:GradCheckpoint: Gradient checkpointing to reduce VRAM usage, but increase computation (setTrainerArguments.gradient_checkpointing).FreezeEmbeds: Freeze (do not train) embedding layer to reduce VRAM usage and computation (settrainer.freeze(embeddings=True)).Adafactor uses less VRAM than Adam, but is slightly slower and can converge slightly differently.You can use gradient accumulation (TrainingArguments.gradient_accumulation_steps) to make up to a larger batch size if needed. The batch sizes reported arewithoutgradient accumulation.Moving embeddings to CPU seems to have almost no impact on both VRAM usage and performance, therefore is not used.GPT ModelsSome GPT configurations that were tested to able to train on a single RTX 3090 (24GB) card (without DeepSpeed):ModelParamsPrecisionOptimizerInputLenBatchSizeOthergpt21.5bFP16Adafactor1284Nonegpt21.5bFP16Adafactor5121Nonegpt21.5bFP16Adafactor10244GradCheckpointgpt-neo1.3bFP16Adafactor10241Nonegpt-neo1.3bFP16Adafactor20484GradCheckpointgpt-neo2.7bFP16Adafactor20484GradCheckpoint,FreezeEmbedsT5 ModelsSome T5 configurations that were tested to able to train on a single RTX 3090 (24GB) card (without DeepSpeed):ModelParamsPrecisionOptimizerSeq2SeqLenBatchSizeOthert53bFP32Adafactor128->1281FreezeEmbedst53bFP32Adafactor128->1281GradCheckpointt53bFP32Adafactor128->128128GradCheckpoint,FreezeEmbedst53bFP32Adafactor512->51232GradCheckpoint,FreezeEmbedsModel Parallelism for T5-11b modelsUsing this library, you also can fine-tune thet5-11b checkpointsquite easily (single node) with the following settings (without Deepspeed):Suggested checkpoint:t5-11bBatch size 1 + gradient accumulation to make up to whatever batch size you need.Batch size of 8 is possible with gradient checkpointing, but doesn't improve the speed.Model parallel across multiple GPUs:At least ~90 GB of VRAMExamples: 8x 16GB or 4x 32GB GPU (V100), or 2x 48GB (RTX8000/A6000)FP32 (no need for mixed precision/FP16)FP16 would actually be better, but the pretrained T5 checkpoints don't play well with FP16.On Ampere cards (RTX30XX, A100, A6000), TF32 is used, which is faster than FP32 and doesn't suffer from the same issues as FP16.Likely reason: the existing activations are too large (github issue tracking,some more info)Note that depending on your system, the loading time for the checkpoint (46GB) can be very long. You'll need ample CPU RAM (at least ~90GB) to load it successfully.ONNX RTONNX RT works with some models (not T5, yet) and can provide a small boost in speed.Install ORT, then setTrainingArguments.torch_ort=Truepipinstalltorch-ort-fhttps://onnxruntimepackages.z14.web.core.windows.net/onnxruntime_stable_torch190.cu111.html

python-mtorch_ort.configureDevelopmentBuilding Packagepython3-mpipinstall--upgradebuildtwine
python3-mbuild
python3-mtwineuploaddist/*DisclaimersThis library as developed as a personal project for my own use. Please feel free to fork or use it for your own purposes as well. I will not take responsibility for any mishaps that occur as a result of this library's usage.Note for 3090 FE cards, if your fans hit 100%, it means your VRAM temps are high (>100 deg C). Training for long hours at these temperatures in theory should be fine, but if you want a peace of mind (like me), you can lower the power limit incur minor impact on training speeds. As long as your fans never hit 100%, your VRAM temperatures should be good. For example, to lower power limit to 300W (from 350W):sudonvidia-smi-pl300