Model Card for Angora-1600

This model has been created to help computer science students at St. Olaf College (Northfield, MN) answer questions about fundamental CS principles as well as questions about the specific technical stacks and procedures St. Olaf Computer Science uses.

Angora-1600 Details

This model is built off of Google's Gemma 7b-it model. It was fine tuned with a dataset created with the purpose of addressing St. Olaf specific Computer Science questions. Some of these questions reference the specific instance of git the institution uses or address steps to declare the computer science major. This model was fine-tuned using MLX on an Apple M3 Max Chip. This model was trained for 1600 iterations using LoRA as the method for finetuning.

Developed by: Ben Anderson & Keegan Murray
Funded by: St. Olaf College MSCS Department
Model type: Generative
License: MIT
Finetuned from model: google/gemma-7b-it

Repository: See the GitHub repository here
Paper: Coming soon...
Demo: A video demo is available here.

Uses

This is intended to be used by Computer Science students at St. Olaf College. While it can be used broadly for general computer science questions, it has been finetuned to answer questions specific to the St. Olaf Computer Science program.

How to Get Started with the Model

Use the code below to get started with the model.

Direct Use With Transformers Library

Use a pipeline as a high-level helper

from transformers import pipeline

pipe = pipeline("text-generation", model="band2001/stolaf-angora-1600")

Load model directly

from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("band2001/stolaf-angora-1600")
model = AutoModelForCausalLM.from_pretrained("band2001/stolaf-angora-1600", device_map="auto")

input_ids = tokenizer("YOUR PROMPT HERE", return_tensors="pt").to("YOUR DEVICE IF USING GPU ACCELERATION")

outputs = model.generate(**input_ids, max_new_tokens=256)
decoded_output =  tokenizer.decode(outputs[0])

Direct Use With MLX Library

Note MLX can only be used with Apple Silicon Macs. It is also recommended to use one of their Max series chips or higher.

from mlx_lm import load, generate

def format_prompt(prompt, system_prompt = "YOUR SYSTEM PROMPT"):

    return """<bos><start_of_turn>user
## Instructions
{}
## User
{}<end_of_turn>
<start_of_turn>model
""".format(system_prompt, prompt)

model, tokenizer = load("band2001/stolaf-angora-1600")

prompt = format_prompt("YOUR PROMPT HERE")

decoded_output = generate(
    model,
    tokenizer,
    prompt=prompt,
    verbose=True,
    temp=0.0,
    max_tokens=256,
)

Out-of-Scope Use

Outside of using this model to ask questions about computer science topics (generally and specific to St. Olaf College), this model should not be used for other inference. Asking questions about other topics will likely yield answers; however, they have not been fine-tuned and will most likely contain errors and/or could potentially include offensive content.

Bias, Risks, and Limitations

As we created the fine-tuning dataset from scratch, it is relatively limited compared to the overall size of the model. Our dataset has about 2000 observations, while the model has roughly 8.5B parameters. So while our dataset had a noticeable effect on the tuning of this model, it still will fall back on other knowledge occasionally and provide partially incorrect answers for St. Olaf specific questions.

Also note the limitations present in the google/gemma-7b-it model and assume they are present in this model as well.

Training Details

Training Data

The training data can be found in the St. Olaf Angora Dataset (band2001/stolaf-angora).

Training Procedure

To train the model, the data needs to be in the following format. Note the data in band2001/stolaf-angora already is.

<bos><start_of_turn>user
## Instructions
  system prompt goes here

## User
  prompt/query goes here<end_of_turn>

<start_of_turn>model
  model response here (put a response here for tuning purposes)<end_of_turn><eos>

Once the data is in the correct format, QLoRA is recommended. The model can be fine-tuned either using mlx-lm and mps (to tune on an Apple Silicon machine) or a bitsandbytes configuration and cuda (to tune on a machine with Nvidia GPUs).

Preprocessing

To preprocess your data to be in the correct format outlined above, you can use the following helper function:

def generate_prompt(entry, system_prompt = SYSTEM_PROMPT):
    '''
    This function formats a question/answer pair to gemma's chat template.

    :param: entry - a dictionary with an instruction and a response
    :param: system_prompt: the system prompt to be used

    :return: the formated string for gemma's chat template
    '''

    return """<bos><start_of_turn>user
## Instructions
{}
## User
{}<end_of_turn>
<start_of_turn>model
{}<end_of_turn><eos>""".format(system_prompt, entry["instruction"], entry["response"])

When trying to use inference with this model, you can format the user's query using this helper function:

def format_prompt(prompt, system_prompt = SYSTEM_PROMPT):
    '''
    This function formats a question to gemma's chat template.

    :param: prompt - a string with the user's query
    :param: system_prompt: the system prompt to be used

    :return: the formated string for gemma's chat template
    '''

    return """<bos><start_of_turn>user
## Instructions
{}
## User
{}<end_of_turn>
<start_of_turn>model
""".format(system_prompt, prompt)

Training Process

The MLX LoRA fine-tuning approach was used. You can learn more about MLX LoRA here. The Gemma-7b-it was loaded in without any conversion. The default batch_size = 16 was chosen and to reach a 1600 iteration fine-tuned model the model was tuned with 800 iterations two times. Once the fine-tuned weights were created, the model was fused using MLX's fuse functionality. You can learn more about fusing with MLX here. One important change made when fusing with MLX was to change some of the MLX package code to include "format":"pt" in the metadata so this model can be used with the transformers library. To do that, the following was done: you can tweak the library code like below in /mlx_lm/utils.py by replacing mx.save_safetensors(str(shard_path), shard, metadata={"format":"mlx"}) with mx.save_safetensors(str(shard_path), shard, metadata={"format":"pt"}) to output fused weights with the metadata attribute. Special thanks to Alexweberk's guide on GitHub to help solve this issue. Finally, the fused model was uploaded to this HuggingFace repo!

If you look at the GitHub repo for this project, mlx_lora.sh includes the command used for the LoRA fine-tuning, mlx_fuse.sh includes the command for the model fusing, and mlx_upload.sh includes the upload command. There is additionally an optional mlx_convert.sh for converting the Google Gemma 7b-it model before fine-tuning if desired.

Evaluation

Testing loss and perplexity were the two metrics used to evaluate the Angora models. A summary of the results for all the different iteration models is included below.

Results

Number of iterations	Testing Loss	Perplexity
800	0.569	1.766
1600	0.302	1.352
2400	0.225	1.252
3200	0.185	1.203
4000	0.170	1.185

Testing Data

The testing data is available here.

Model Card Contact

Ben Anderson - ander6@stolaf.edu

Keegan Murray - murray7@stolaf.edu

band2001
/

stolaf-angora-1600