README.md

---
license: mit
datasets:
- band2001/stolaf-angora
---

# Model Card for Angora-1600

<!-- Provide a quick summary of what the model is/does. -->

This model has been created to help computer science students at St. Olaf College (Northfield, MN) answer questions about fundamental CS principles as well as questions about the specific technical stacks and procedures St. Olaf Computer Science uses. 

## Angora-1600 Details

This model is built off of [Google's Gemma 7b-it](https://huggingface.co/google/gemma-7b-it) model. It was fine tuned with a dataset created with the purpose of addressing St. Olaf specific Computer Science questions. Some of these questions reference the specific instance of git the institution uses or address steps to declare the computer science major. This model was fine-tuned using MLX on an Apple M3 Max Chip. This model was trained for 1600 iterations using LoRA as the method for finetuning.

- **Developed by:** Ben Anderson & Keegan Murray
- **Funded by:** St. Olaf College MSCS Department
- **Model type:** Generative
- **License:** MIT
- **Finetuned from model:** [google/gemma-7b-it](https://huggingface.co/google/gemma-7b-it)

<!-- Provide the basic links for the model. -->

- **Repository:** See the GitHub repository [here](https://github.com/band2001/stolaf-angora)
- **Paper:** Coming soon...
- **Demo:** A video demo is available [here](https://drive.google.com/file/d/1iwThVj88FTgLNANZdv2NineRcBXAqtZp/view?usp=sharing).

## Uses

This is intended to be used by Computer Science students at St. Olaf College. While it can be used broadly for general computer science questions, it has been finetuned to answer questions specific to the St. Olaf Computer Science program.

## How to Get Started with the Model

Use the code below to get started with the model.

### Direct Use With Transformers Library


#### Use a pipeline as a high-level helper
```python
from transformers import pipeline

pipe = pipeline("text-generation", model="band2001/stolaf-angora-1600")
```

#### Load model directly
```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("band2001/stolaf-angora-1600")
model = AutoModelForCausalLM.from_pretrained("band2001/stolaf-angora-1600", device_map="auto")

input_ids = tokenizer("YOUR PROMPT HERE", return_tensors="pt").to("YOUR DEVICE IF USING GPU ACCELERATION")

outputs = model.generate(**input_ids, max_new_tokens=256)
decoded_output =  tokenizer.decode(outputs[0])
```

### Direct Use With MLX Library

Note MLX can only be used with Apple Silicon Macs. It is also recommended to use one of their Max series chips or higher. 

```python
from mlx_lm import load, generate

def format_prompt(prompt, system_prompt = "YOUR SYSTEM PROMPT"):

    return """<bos><start_of_turn>user
## Instructions
{}
## User
{}<end_of_turn>
<start_of_turn>model
""".format(system_prompt, prompt)

model, tokenizer = load("band2001/stolaf-angora-1600")

prompt = format_prompt("YOUR PROMPT HERE")

decoded_output = generate(
    model,
    tokenizer,
    prompt=prompt,
    verbose=True,
    temp=0.0,
    max_tokens=256,
)
```

### Out-of-Scope Use

Outside of using this model to ask questions about computer science topics (generally and specific to St. Olaf College), this model should not be used for other inference. Asking questions about other topics will likely yield answers; however, they have not been fine-tuned and will most likely contain errors and/or could potentially include offensive content.

## Bias, Risks, and Limitations

As we created the fine-tuning dataset from scratch, it is relatively limited compared to the overall size of the model. Our dataset has about 2000 observations, while the model has roughly 8.5B parameters. So while our dataset had a noticeable effect on the tuning of this model, it still will fall back on other knowledge occasionally and provide partially incorrect answers for St. Olaf specific questions. 

Also note the limitations present in the [google/gemma-7b-it](https://huggingface.co/google/gemma-7b-it) model and assume they are present in this model as well.

## Training Details

### Training Data

The training data can be found in the St. Olaf Angora Dataset ([band2001/stolaf-angora](https://huggingface.co/datasets/band2001/stolaf-angora)).


### Training Procedure

To train the model, the data needs to be in the following format. Note the data in [band2001/stolaf-angora](https://huggingface.co/datasets/band2001/stolaf-angora) already is.

```
<bos><start_of_turn>user
## Instructions
  system prompt goes here

## User
  prompt/query goes here<end_of_turn>

<start_of_turn>model
  model response here (put a response here for tuning purposes)<end_of_turn><eos>
```

Once the data is in the correct format, QLoRA is recommended. The model can be fine-tuned either using mlx-lm and mps (to tune on an Apple Silicon machine) or a bitsandbytes configuration and cuda (to tune on a machine with Nvidia GPUs).

#### Preprocessing

To preprocess your data to be in the correct format outlined above, you can use the following helper function:

```python
def generate_prompt(entry, system_prompt = SYSTEM_PROMPT):
    '''
    This function formats a question/answer pair to gemma's chat template.

    :param: entry - a dictionary with an instruction and a response
    :param: system_prompt: the system prompt to be used

    :return: the formated string for gemma's chat template
    '''

    return """<bos><start_of_turn>user
## Instructions
{}
## User
{}<end_of_turn>
<start_of_turn>model
{}<end_of_turn><eos>""".format(system_prompt, entry["instruction"], entry["response"])
```

When trying to use inference with this model, you can format the user's query using this helper function:

```python
def format_prompt(prompt, system_prompt = SYSTEM_PROMPT):
    '''
    This function formats a question to gemma's chat template.

    :param: prompt - a string with the user's query
    :param: system_prompt: the system prompt to be used

    :return: the formated string for gemma's chat template
    '''

    return """<bos><start_of_turn>user
## Instructions
{}
## User
{}<end_of_turn>
<start_of_turn>model
""".format(system_prompt, prompt)
```

#### Training Process

The MLX LoRA fine-tuning approach was used. You can learn more about [MLX LoRA here](https://github.com/ml-explore/mlx-examples/blob/main/lora/README.md). The Gemma-7b-it was loaded in without any conversion. The default `batch_size = 16` was chosen and to reach a 1600 iteration fine-tuned model the model was tuned with 800 iterations two times. Once the fine-tuned weights were created, the model was fused using MLX's fuse functionality. You can learn more about [fusing with MLX here](https://github.com/ml-explore/mlx-examples/blob/main/lora/README.md#Fuse-and-Upload). One important change made when fusing with MLX was to change some of the MLX package code to include `"format":"pt"` in the metadata so this model can be used with the transformers library. To do that, the following was done: you can tweak the library code like below in <path_to_your_site-packages>/mlx_lm/utils.py by replacing `mx.save_safetensors(str(shard_path), shard, metadata={"format":"mlx"})` with `mx.save_safetensors(str(shard_path), shard, metadata={"format":"pt"})` to output fused weights with the metadata attribute. Special thanks to [Alexweberk's guide on GitHub](https://gist.github.com/alexweberk/635431b5c5773efd6d1755801020429f) to help solve this issue. Finally, the fused model was uploaded to this HuggingFace repo!

If you look at the GitHub repo for this project, mlx_lora.sh includes the command used for the LoRA fine-tuning, mlx_fuse.sh includes the command for the model fusing, and mlx_upload.sh includes the upload command. There is additionally an optional mlx_convert.sh for converting the Google Gemma 7b-it model before fine-tuning if desired.

## Evaluation

Testing loss and perplexity were the two metrics used to evaluate the Angora models. A summary of the results for all the different iteration models is included below.

### Results

| Number of iterations | Testing Loss | Perplexity |
|:----------|:----------|:---------|
|800 | 0.569 | 1.766 |
| 1600 | 0.302 | 1.352 |
| 2400 | 0.225 | 1.252 |
| 3200 | 0.185 | 1.203 |
| 4000 | 0.170 | 1.185 |

### Testing Data

The testing data is available [here](https://huggingface.co/datasets/band2001/stolaf-angora/viewer/default/test).

## Model Card Contact

Ben Anderson - [ander6@stolaf.edu](mailto:ander6@stolaf.edu)

Keegan Murray - [murray7@stolaf.edu](mailto:murray7@stolaf.edu)