|
--- |
|
license: other |
|
commercial: false |
|
datasets: |
|
- aisquared/databricks-dolly-15k |
|
language: |
|
- en |
|
library_name: transformers |
|
--- |
|
|
|
|
|
|
|
# Model Card for `chopt-1_3b` |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
AI Squared's `chopt-1_3b` is a large language model which is derived from Meta AI's Open Pre-trained Transformer language modelsand fine-tuned on a corpus of 15k records ([Databricks' "Dolly 15k" Dataset](https://huggingface.co/datasets/aisquared/databricks-dolly-15k)) to help it exhibit chat-based capabilities. Despite the permissive license of the Dolly 15k dataset, due to this model being a derivative of OPT it is restricted to use for **non-commercial research purposes**. The ChOPT family of models from AI Squared are licensed under the OPT-175B license, Copyright (c) Meta Platforms, Inc. All Rights Reserved. |
|
|
|
While `chopt-1_3b` is **not a state-of-the-art model**, we believe that the level of interactivity that can be achieved on such a small model that is trained so cheaply is important to showcase, as it continues to demonstrate that creating powerful AI capabilities may be much more accessible than previously thought. |
|
|
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
- **Developed by:** AI Squared, Inc. |
|
- **Shared by:** AI Squared, Inc. |
|
- **Model type:** Large Language Model |
|
- **Language(s) (NLP):** EN |
|
- **License:** other |
|
- **Finetuned from model:** OPT |
|
|
|
|
|
## Bias, Risks, and Limitations |
|
|
|
<!-- This section is meant to convey both technical and sociotechnical limitations. --> |
|
|
|
**`chopt-1_3b` is not a state-of-the-art language model.** `chopt-1_3b` is an experimental technology and is not designed for use in any |
|
environment other than for research purposes. Furthermore, the model can sometimes exhibit undesired behaviors. Some of these behaviors include, |
|
but are not limited to: factual inaccuracies, biases, offensive responses, toxicity, and hallucinations. |
|
Just as with any other LLM, we advise users of this technology to exercise good judgment when applying this technology. |
|
|
|
|
|
## Usage |
|
|
|
To use the model with the `transformers` library on a machine with GPUs, first make sure you have the `transformers` and `accelerate` libraries installed. |
|
From your terminal, run: |
|
|
|
```python |
|
pip install "accelerate>=0.16.0,<1" "transformers[torch]>=4.28.1,<5" "torch>=1.13.1,<2" |
|
``` |
|
|
|
The instruction following pipeline can be loaded using the `pipeline` function as shown below. This loads a custom `InstructionTextGenerationPipeline` |
|
found in the model repo [here](https://huggingface.co/aisquared/chopt-1_3b/blob/main/instruct_pipeline.py), which is why `trust_remote_code=True` is required. |
|
Including `torch_dtype=torch.bfloat16` is generally recommended if this type is supported in order to reduce memory usage. It does not appear to impact output quality. |
|
It is also fine to remove it if there is sufficient memory. |
|
|
|
```python |
|
from transformers import pipeline |
|
import torch |
|
|
|
generate_text = pipeline(model="aisquared/chopt-1_3b", torch_dtype=torch.bfloat16, trust_remote_code=True, device_map="auto") |
|
``` |
|
|
|
You can then use the pipeline to answer instructions: |
|
|
|
```python |
|
res = generate_text("Who was George Washington?") |
|
print(res) |
|
``` |
|
|
|
Alternatively, if you prefer to not use `trust_remote_code=True` you can download [instruct_pipeline.py](https://huggingface.co/aisquared/chopt-1_3b/blob/main/instruct_pipeline.py), |
|
store it alongside your notebook, and construct the pipeline yourself from the loaded model and tokenizer: |
|
|
|
```python |
|
from instruct_pipeline import InstructionTextGenerationPipeline |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
import torch |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("aisquared/chopt-1_3b", padding_side="left") |
|
model = AutoModelForCausalLM.from_pretrained("aisquared/chopt-1_3b", device_map="auto", torch_dtype=torch.bfloat16) |
|
|
|
generate_text = InstructionTextGenerationPipeline(model=model, tokenizer=tokenizer) |
|
``` |
|
|
|
### Model Performance Metrics |
|
|
|
We present the results from various model benchmarks on the EleutherAI LLM Evaluation Harness for all models in the ChOPT family. |
|
Model results are sorted by mean score, ascending, to provide an ordering. These metrics serve to further show that none of the DLite models are |
|
state of the art, but rather further show that chat-like behaviors in LLMs can be trained almost independent of model size. |
|
|
|
| Model | openbookqa | arc_easy | winogrande | hellaswag | arc_challenge | piqa | boolq | |
|
|:--------------------|-------------:|-----------:|-------------:|------------:|----------------:|---------:|---------:| |
|
| chopt-125m | 0.178 | 0.443182 | 0.501973 | 0.294165 | 0.197099 | 0.630577 | 0.476758 | |
|
| chopt-research-125m | 0.17 | 0.436027 | 0.503552 | 0.294762 | 0.205631 | 0.62568 | 0.48685 | |
|
| opt-125m | 0.166 | 0.435606 | 0.501973 | 0.291775 | 0.190273 | 0.6284 | 0.554434 | |
|
| chopt-350m | 0.178 | 0.450758 | 0.508287 | 0.325334 | 0.21843 | 0.650707 | 0.559633 | |
|
| opt_350m | 0.176 | 0.441077 | 0.52644 | 0.320056 | 0.207338 | 0.645267 | 0.57737 | |
|
| chopt-research-350m | 0.172 | 0.462542 | 0.514601 | 0.327524 | 0.235495 | 0.643634 | 0.589908 | |
|
| opt-1.3b | 0.234 | 0.569865 | 0.596685 | 0.414957 | 0.232935 | 0.718172 | 0.577676 | |
|
| chopt-research-1_3b | 0.232 | 0.564815 | 0.59116 | 0.424716 | 0.276451 | 0.713275 | 0.634557 | |
|
| chopt-1_3b | 0.236 | 0.569444 | 0.584057 | 0.42621 | 0.268771 | 0.723069 | 0.658104 | |
|
| opt-2.7b | 0.25 | 0.608165 | 0.608524 | 0.458176 | 0.267918 | 0.738303 | 0.603058 | |
|
| chopt-2_7b | 0.276 | 0.616582 | 0.601421 | 0.472615 | 0.288396 | 0.75136 | 0.552294 | |
|
| chopt-research-2_7b | 0.262 | 0.610269 | 0.625099 | 0.458176 | 0.295222 | 0.742111 | 0.636697 | |
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) |
|
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_aisquared__chopt-1_3b) |
|
|
|
| Metric | Value | |
|
|-----------------------|---------------------------| |
|
| Avg. | 30.94 | |
|
| ARC (25-shot) | 31.48 | |
|
| HellaSwag (10-shot) | 56.63 | |
|
| MMLU (5-shot) | 25.35 | |
|
| TruthfulQA (0-shot) | 40.19 | |
|
| Winogrande (5-shot) | 58.25 | |
|
| GSM8K (5-shot) | 0.0 | |
|
| DROP (3-shot) | 4.67 | |
|
|
