PhantomPal/gemma2-2b-it-openstax-finetuned

Model Card for Gemma2-2B-IT Fine-Tuned on OpenStax Textbooks

This model is a fine-tuned version of google/gemma2-2b-it, specifically adapted for summarizing and answering questions related to OpenStax textbooks in economics and mathematics. It has been trained on text extracted from various OpenStax textbooks, making it suitable for educational applications such as content summarization, question answering, and educational content generation.

Model Details

Model Description

This model leverages the capabilities of the gemma2-2b-it language model and has been fine-tuned using a knowledge distillation approach on the OpenStax open-source textbook crumb/openstax-text dataset. The fine-tuning process involved training the model to better understand and generate responses related to textbook content, particularly in the fields of economics and mathematics.

• Developed by: Seungjoo Kim, Seonghoon Yi, Intaek Hong
• Model type: Causal Language Model (Decoder-only Transformer)
• Language(s): English
• Fine-tuned from model: google/gemma2-2b-it

Model Sources

• Repository: Gemma2-2B-IT OpenStax Model
• Dataset: crumb/openstax-text

Uses

Direct Use

The model can be directly used for:

• Summarizing textbook chapters and sections in desired depth of the user.
• Generating quizzes and practice questions based on textbook content. (Multiple choice, short answer, application, essay, etc.)
• Answering questions related to economics and mathematics topics covered in OpenStax textbooks.

Downstream Use

The model can be integrated into educational platforms and applications to enhance learning experiences, provide personalized study assistance, and develop educational content.

Out-of-Scope Use

• Generating content outside the scope of economics and mathematics as covered in OpenStax textbooks.
• Creating inappropriate, offensive, or harmful content.
• Any use that violates ethical guidelines or legal restrictions.

Bias, Risks, and Limitations

• Biases: The model may reflect the biases present in the training data, which is limited to OpenStax textbooks and may not encompass all perspectives.
• Knowledge Cutoff: Limited to the information available in the OpenStax textbooks; may not include the most recent developments.
• Accuracy: May produce incorrect or nonsensical answers; verification is recommended.

Recommendations

Users should:

• Verify generated content with authoritative sources.
• Exercise caution when using the model for critical applications.
• Be aware of the model's limitations in scope and accuracy.

How to Get Started with the Model

Use the code below to get started with the model:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Replace 'your-username' with your actual Hugging Face username
model_repo = "your-username/gemma2-2b-it-openstax"

# Load the tokenizer
tokenizer = AutoTokenizer.from_pretrained(f"{model_repo}/fine_tuned_tokenizer")

# Load the base model
base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma2-2b-it",
    device_map="auto",
    torch_dtype=torch.float16
)

# Load the adapter weights
model = PeftModel.from_pretrained(
    base_model,
    f"{model_repo}/fine_tuned_model"
)

# Prepare the input text
input_text = "What is the law of demand?"

# Tokenize the input
inputs = tokenizer(input_text, return_tensors="pt")
inputs = {key: value.to(model.device) for key, value in inputs.items()}

# Generate a response
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,
        temperature=0.7,
        top_p=0.9
    )

# Decode and print the response
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Training Details

Training Data

The model was fine-tuned on the crumb/openstax-text dataset, which contains text extracted from various OpenStax textbooks covering economics and mathematics. The data was preprocessed to extract relevant sections and formatted into question-answer pairs.

Training Procedure

The model was fine-tuned using parameter-efficient fine-tuning techniques, specifically using adapters via the PEFT library.

Preprocessing

• Data Cleaning: Removed irrelevant content and standardized text.
• Formatting: Converted text into a suitable format for training, including tokenization.
• Splitting: Divided data into training and validation sets.

Training Hyperparameters

• Training regime: Mixed precision (fp16)
• Batch size: 1
• Learning rate: 1e-4
• Number of epochs: 10
• Optimizer: paged_adamw_8bit

Evaluation

Testing Data, Factors

Testing Data

• Validation Split: Used a portion of the crumb/openstax-text dataset not seen during training.

Factors

• Domains: Economics and mathematics topics as covered in OpenStax textbooks.

Summary

The model demonstrates strong capabilities in generating accurate and contextually relevant responses within its trained domain. However, performance may decline on topics outside the training scope.

Technical Specifications

Model Architecture and Objective

• Architecture: Decoder-only Transformer model with approximately 2 billion parameters.
• Objective: Causal language modeling—predicting the next token in a sequence.

Compute Infrastructure

Hardware

• Training Hardware: [e.g., 4 x NVIDIA V100 GPUs]

Software

• Frameworks: PyTorch, Transformers, PEFT
• Operating System: [Specify if known]
• CUDA/cuDNN version: [Specify if known]

Citation

BibTeX:

@misc{gemma2-openstax,
  author = {Seungjoo Kim, Seonghoon Yi, Intaek Hong} 
  title = {Gemma2-2B-IT Fine-Tuned on OpenStax Textbooks},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/PhantomPal/gemma2-2b-it-openstax}},
}

APA:

Seungjoo Kim, Seonghoon Yi, Intaek Hong. (2024). Gemma2-2B-IT Fine-Tuned on OpenStax Textbooks. Hugging Face. Retrieved from https://huggingface.co/your-username/gemma2-2b-it-openstax

Glossary

• Causal Language Modeling: A task where the model predicts the next word in a sequence.
• Adapters: Lightweight modules added to a pre-trained model to enable fine-tuning on new tasks without retraining the entire model.

Model Card Authors

• Seungjoo Kim
• Seonghoon Yi
• Intaek Hong

Model Card Contact

• Email:
• phantompal32@gmail.com (Seonghoon Yi)
• anthony.i.hong@gmail,com (Intaek Hong)