PodAI/dr-gemma · Hugging Face

Model Details

A Gemma2B-it model finetuned on Medical-Data

Model Description

This is a fine-tuned version of the Gemma2B-it model, a large language model developed by Google. The fine-tuning process has been performed on a medical reasoning dataset to adapt the model for answering medical-related questions.

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

Developed by: Ananya S Kaligal, Kashish Varma, Ajay S Patil
Mentored by: Dr.Shabbeer Basha S H
Supported by: RV University
Model type: Instruct
Language(s) (NLP): english-en
License: [MIT]
**Finetuned from model: google/gemma2b-it

Model Sources : Huggingface

**Repository: google/gemma2b-it

How to Get Started with the Model

Install Required Libraries: Begin by installing the necessary Python libraries and their specific versions. In this case, ensure you have the required versions of bitsandbytes, peft, trl, accelerate, datasets, and transformers. You can use the following commands to install them:

!pip3 install -q -U bitsandbytes==0.42.0
!pip3 install -q -U peft==0.8.2
!pip3 install -q -U trl==0.7.10
!pip3 install -q -U accelerate==0.27.1
!pip3 install -q -U datasets==2.17.0
!pip3 install -q -U transformers==4.38.0

Library Imports: Import the required libraries in your Python script or Jupyter Notebook. These include torch, transformers, datasets, bitsandbytes, peft, and trl. These libraries provide functionalities for model training, optimization, and evaluation.
Model Setup: Initialize the language model for causal language modeling (LM) using the AutoModelForCausalLM class from the transformers library. Load the pre-trained model from the Hugging Face Model Hub using its identifier, such as "google/gemma-2b-it". Configure any additional settings or optimizations required for your task.
Data Loading: Load your training dataset using the load_dataset function from the datasets library. Specify the dataset identifier or path to load the desired dataset. This dataset will be used to fine-tune the language model for your specific task.
Prompt Generation: Generate prompts for each data point in your training dataset. These prompts should include instructions for the task and any associated input data required for inference.
Tokenization: Tokenize the generated prompts using the AutoTokenizer from the transformers library. Tokenization converts the text data into numerical inputs suitable for the language model.
Dataset Splitting: Split your dataset into training and testing subsets using the train_test_split method. This allows you to evaluate the model's performance on unseen data during training.
Model Configuration: Configure the language model for training by setting up various parameters and optimizations. This may include gradient checkpointing, low-rank adaptation, and other techniques to improve training efficiency and performance.
Training: Train the language model using the SFTTrainer class from the trl library. Specify the training arguments such as batch size, learning rate, and optimization strategy. Monitor the training progress and evaluate the model's performance on the testing subset.
Model Saving: Save the trained model and tokenizer locally using the save_pretrained method. This allows you to reload the model for future use without retraining.
Model Deployment: Optionally, push the trained model and tokenizer to the Hugging Face Model Hub for sharing and deployment. This makes your model accessible to the wider community for use in various applications.

Training Data

[ruslnmv] : ["https://huggingface.co/api/datasets/ruslanmv/ai-medical-chatbot/parquet/default/train/0.parquet"]
[prsdm] : ["prsdm/MedQuad-phi2-1k"]

Training Procedure

Data Loading: The training data is loaded using the load_dataset function from the datasets library. In this case, the dataset used is "mamachang/medical-reasoning".
Model Setup: The language model for causal language modeling (LM) is initialized using the AutoModelForCausalLM class from the transformers library. The model is loaded from the "google/gemma-2b-it" pre-trained model. Additionally, a BitsAndBytesConfig is used to configure low-precision quantization for the model.
Prompt Generation: The training data is preprocessed by generating prompts for each data point. These prompts include instructions for a task and any associated input data.
Tokenization: The generated prompts are tokenized using the AutoTokenizer from the transformers library. Tokenization is necessary to convert the text data into numerical inputs suitable for the model.
Dataset Splitting: The dataset is split into training and testing subsets using the train_test_split method. This allows for evaluating the model's performance on unseen data during training.
Model Configuration: Various model configurations, such as gradient checkpointing and low-rank adaptation (LORA), are set up to optimize training performance and efficiency.
Training: The model is trained using the SFTTrainer class from the trl library. Training is performed using the specified training arguments, including batch size, learning rate, and optimization strategy.
Model Saving: After training, the fine-tuned model is saved locally using the save_pretrained method. Additionally, the tokenizer's configuration is saved for future use.
Merging Base and Fine-tuned Models: The fine-tuned model is merged with the base model to create a new model that retains the optimizations and adaptations applied during training.
Push to Model Hub: The merged model and tokenizer are pushed to the Hugging Face Model Hub for sharing and deployment.
Model Inference: Finally, the trained model is used for text completion tasks by providing a prompt/question, encoding it, and generating a response using the generate method.

Preprocessing [optional]

dataset = load_dataset("mamachang/medical-reasoning")

This line loads the dataset "mamachang/medical-reasoning" using the load_dataset function from the datasets library. It fetches the dataset from the Hugging Face dataset repository.

def generate_prompt(data_point):
    # Generate prompt
    prefix_text = 'Below is an instruction that describes a task. Write a response that ' \
               'appropriately completes the request.\n\n'
    # Samples with additional context into.
    if data_point['input']:
        text = f"""<start_of_turn>user {prefix_text} {data_point["instruction"]} here are the inputs {data_point["input"]} <end_of_turn>\n<start_of_turn>model{data_point["output"]} <end_of_turn>"""
    # Without
    else:
        text = f"""<start_of_turn>user {prefix_text} {data_point["instruction"]} <end_of_turn>\n<start_of_turn>model{data_point["output"]} <end_of_turn>"""
    return text

text_column = [generate_prompt(data_point) for data_point in dataset["train"]]
dataset = dataset["train"].add_column("prompt", text_column)

Here, you define a function generate_prompt to create prompts for each data point in the dataset. The prompts include instructions for the task, along with any input data required. These prompts are then added as a new column named "prompt" to the training subset of the dataset.

dataset = dataset.shuffle(seed=1234)  # Shuffle dataset here
dataset = dataset.map(lambda samples: tokenizer(samples["prompt"]), batched=True)
dataset = dataset.train_test_split(test_size=0.1)
train_data = dataset["train"]
test_data = dataset["test"]

The dataset is shuffled to randomize the order of data samples using the shuffle method. This helps prevent any inherent ordering bias in the dataset.
Next, the map function is used to apply tokenization to the generated prompts using the tokenizer instantiated earlier. The prompts are tokenized in batches for efficiency.
The preprocessed dataset is split into training and testing subsets using the train_test_split method. This allows for evaluating the model's performance on unseen data during training.

These parts of your code handle various aspects of dataset preprocessing, including data loading, prompt generation, tokenization, shuffling, and dataset splitting, to prepare the dataset for training the language model.

Training Hyperparameters

BitsAndBytes Configuration:
- load_in_4bit=True: Model weights are loaded in 4-bit precision.
- bnb_4bit_use_double_quant=True: Double quantization is used for 4-bit quantization.
- bnb_4bit_quant_type="nf4": Normalization-Free 4-bit quantization is used.
- bnb_4bit_compute_dtype=torch.bfloat16: Computations are performed in bfloat16 precision.
LoRA Configuration:
- r=64: Rank of the LoRA matrices.
- lora_alpha=32: Alpha value for LoRA scaling.
- target_modules: Determined dynamically based on the model's linear layers.
- lora_dropout=0.05: Dropout rate for LoRA matrices.
- bias="none": No bias is used for LoRA.
- task_type="CAUSAL_LM": Task type is set to casual language modeling.
Training Arguments:
- per_device_train_batch_size=1: Batch size for training.
- gradient_accumulation_steps=4: Gradient accumulation steps.
- warmup_steps=0.03: Number of warmup steps for the learning rate scheduler.
- max_steps=100: Maximum number of training steps.
- learning_rate=2e-4: Learning rate for the optimizer.
- logging_steps=1: Logging interval (in steps).
- output_dir="outputs": Output directory for saving checkpoints and logs.
- optim="paged_adamw_8bit": Optimizer is 8-bit Paged AdamW.
- save_strategy="epoch": Save strategy is set to save checkpoints after each epoch.
Data Collator:
- transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False): Data collator is set for language modeling without masked language modeling.
Other Configurations:
- max_seq_length=2500: Maximum sequence length for input data.
- model.config.use_cache=False: Caching is disabled for the model.

These hyperparameters control various aspects of the training process, such as quantization, LoRA configuration, batch size, gradient accumulation, learning rate, and optimizer settings. Adjusting these hyperparameters can significantly impact the training performance, memory efficiency, and model accuracy.

Evaluation

trainer = SFTTrainer(
    model=model,
    train_dataset=train_data,
    eval_dataset=test_data,
    dataset_text_field="prompt",
    peft_config=lora_config,
    max_seq_length=2500,
    args=transformers.TrainingArguments(
        per_device_train_batch_size=1,
        gradient_accumulation_steps=4,
        warmup_steps=0.03,
        max_steps=100,
        learning_rate=2e-4,
        logging_steps=1,
        output_dir="outputs",
        optim="paged_adamw_8bit",
        save_strategy="epoch",
    ),
    data_collator=transformers.DataCollatorForLanguageModeling(tokenizer, mlm=False),
)

The SFTTrainer class from the trl (Text Rewriting for Language Understanding) library is used for training and evaluation. It is initialized with parameters including the model, training dataset (train_data), evaluation dataset (test_data), dataset text field (which specifies the input field to use for the dataset), peft_config (configuration for the low-rank adaptation technique), and various training arguments specified in transformers.TrainingArguments.
Evaluation is performed on the evaluation dataset (test_data). During evaluation, the model is evaluated on unseen data to assess its performance and generalization capabilities.

trainer.train()

After initializing the trainer, the train method is called to start the training process. During training, the model is fine-tuned on the training dataset (train_data) using the specified training arguments and configurations.

result = get_completion(query=query, model=merged_model, tokenizer=tokenizer)
print(result)

Lastly, a function named get_completion is called to generate completions for a given query using the trained model (merged_model). This completion is printed to the console for inspection or further analysis.

Summary

The code implements fine-tuning and evaluation of a language model using Hugging Face's Transformers library. It loads a pre-trained model, prepares the dataset, trains the model, and evaluates its performance. However, explicit evaluation metrics are not defined. The trained model is saved and can be used for generating text completions.

Compute Infrastructure

Utilised 20GB of NVIDIA A100

Hardware

NVIDIA DGX-A100

[More Information Needed]

Software

Python 3.10.12 Transformers 4.38.0 Torch 2.3.0 BitsAndBytes 0.42.0 TRL 0.7.10 PEFT 0.8.2

Glossary [optional]

BitsAndBytes: A Python library that provides functionality for low-precision quantization and optimization of neural network models, allowing for efficient inference on resource-constrained devices by using 4-bit and 8-bit data types.
Quantization: The process of converting floating-point model parameters and activations into lower precision formats, such as 4-bit or 8-bit integers, to reduce memory storage requirements and improve computational efficiency during inference.
Low-Rank Adaptation (LORA): A compression technique used to reduce the computational complexity of neural network models by approximating weight matrices with low-rank matrices, thereby decreasing the number of parameters and improving inference speed.
PeftModel: A model variant optimized for low-precision training and inference, capable of handling data types such as 4-bit and 8-bit integers efficiently, typically used in scenarios where computational resources are limited.
Gradient Checkpointing: A memory-saving technique used during training that trades off computational overhead for reduced memory usage by recomputing intermediate activations during backpropagation, rather than storing them in memory.
Data Collator: A component responsible for batch processing and data preparation during training, including tasks such as padding sequences to a uniform length, shuffling batches, and converting data into tensor format suitable for input to the model.
Hardware Type: The specific hardware infrastructure used for model training and inference, including details such as GPU (Graphics Processing Unit), CPU (Central Processing Unit), or specialized hardware accelerators like TPUs (Tensor Processing Units) or FPGAs (Field-Programmable Gate Arrays).
Paged AdamW Optimizer: An optimization algorithm derived from the AdamW optimizer, which updates model parameters based on gradient descent principles while incorporating weight decay regularization to prevent overfitting.
Sequence Length: The maximum number of tokens or words allowed in input sequences processed by the model, which impacts memory consumption and computation time during training and inference.
Cloud Provider: The service provider responsible for hosting the computational resources used for training and deploying the model, including cloud computing platforms such as Amazon Web Services (AWS), Google Cloud Platform (GCP), or Microsoft Azure.
Carbon Emissions: The total amount of greenhouse gas emissions, typically measured in grams of CO2 equivalents, generated as a result of running the model on computational hardware, taking into account factors such as energy consumption and infrastructure efficiency.
Throughput: The rate at which the model processes input data or performs inference, typically measured in examples or tokens per second, and influenced by factors such as batch size, model complexity, and hardware specifications.