OuteAI

/

Lite-Oute-1-65M

like 8

Hi,
You're correct this model, like many others, can be fine-tuned.
However, providing a specific code example for fine-tuning isn't straightforward, as the process can vary depending on your specific use case, dataset, and desired outcome.

I'd recommend reviewing:
https://pytorch.org/tutorials/beginner/introyt/trainingyt.html
https://huggingface.co/docs/trl/main/en/sft_trainer
https://huggingface.co/docs/transformers/perf_train_gpu_one
https://github.com/hiyouga/LLaMA-Factory

This code has been known to work, I've tried it.

It can be scaled to 10 epochs or more with no major time extension. I don't know what's the epoch-to-data ratio before overfitting occurs.

from transformers import TextDataset, DataCollatorForLanguageModeling
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers import Trainer, TrainingArguments
import logging
import torch

logging.basicConfig(level=logging.INFO)


def load_dataset(file_path, tokenizer, block_size=128):
    return TextDataset(
        tokenizer=tokenizer,
        file_path=file_path,
        block_size=block_size,
    )


def load_data_collator(tokenizer, mlm=False):
    return DataCollatorForLanguageModeling(
        tokenizer=tokenizer,
        mlm=mlm,
    )


def train():
    logging.info("Initializing...")
    tokenizer = AutoTokenizer.from_pretrained("Lite-Oute-1-65M")

    train_dataset = load_dataset("dataset.txt", tokenizer)
    data_collator = load_data_collator(tokenizer)

    tokenizer.save_pretrained("result")

    model = AutoModelForCausalLM.from_pretrained("Lite-Oute-1-65M")

    model.save_pretrained("result")

    training_args = TrainingArguments(
        output_dir="result",
        num_train_epochs=5,
        learning_rate=0.001,
        per_device_train_batch_size=16,
        gradient_accumulation_steps=16,
        fp16=True,
        warmup_ratio=0.1,
        fp16_opt_level="O3",
        logging_strategy="epoch",
        save_steps=250,
        save_total_limit=2,
        report_to="none",
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=train_dataset,
    )

    if torch.cuda.is_available():
        logging.info("Training using GPU")
    else:
        logging.info("Training using CPU")

    trainer.train()

    trainer.save_model()
    tokenizer.save_pretrained("result")


if __name__ == "__main__":
    train()