File size: 5,258 Bytes
3e632c4 012ebea 3e632c4 a270932 3e632c4 1cbabf3 31c1102 3e632c4 a270932 012ebea |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 |
---
license: apache-2.0
model-index:
- name: openchat-3.5-0106_Rebased_Mistral-7B-v0.2
results:
- task:
type: text-generation
name: Text Generation
dataset:
name: IFEval (0-Shot)
type: HuggingFaceH4/ifeval
args:
num_few_shot: 0
metrics:
- type: inst_level_strict_acc and prompt_level_strict_acc
value: 37.06
name: strict accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: BBH (3-Shot)
type: BBH
args:
num_few_shot: 3
metrics:
- type: acc_norm
value: 10.91
name: normalized accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MATH Lvl 5 (4-Shot)
type: hendrycks/competition_math
args:
num_few_shot: 4
metrics:
- type: exact_match
value: 3.85
name: exact match
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: GPQA (0-shot)
type: Idavidrein/gpqa
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 2.91
name: acc_norm
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MuSR (0-shot)
type: TAUR-Lab/MuSR
args:
num_few_shot: 0
metrics:
- type: acc_norm
value: 20.57
name: acc_norm
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
name: Open LLM Leaderboard
- task:
type: text-generation
name: Text Generation
dataset:
name: MMLU-PRO (5-shot)
type: TIGER-Lab/MMLU-Pro
config: main
split: test
args:
num_few_shot: 5
metrics:
- type: acc
value: 20.33
name: accuracy
source:
url: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard?query=Pretergeek/openchat-3.5-0106_Rebased_Mistral-7B-v0.2
name: Open LLM Leaderboard
---
This model was created as an experiment on using LoRA extraction to replicate [Openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106) using [Mistral-7B-v0.2](https://huggingface.co/mistral-community/Mistral-7B-v0.2) as a base model instead of the original [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1).
Openchat-3.5-0106 is an excellent model but was based on Mistral-7B-v0.1 which has a context window of 8192 tokens. Mistral-7B-v0.2 has a context window of 32768 tokens. I could have extended OpenChat-3.5 context myself with RoPE and/or YaRN but that has been done. There are many models on HF that have done exactly that. Instead I decided to try and replicate OpenChat-3.5-0106 using the LoRA extraction method available in mergekit. These are the steps I followed:
- Extract a LoRA with rank 512 from OpenChat-3.5-0106 using [One](https://huggingface.co/imone)'s [Mistral_7B_with_EOT_token](https://huggingface.co/imone/Mistral_7B_with_EOT_token) as the base model.
- Replicate imone's work by adding the EOT token to Mistral-7B-v0.2, creating [Mistral-7B-v0.2_EOT](https://huggingface.co/Pretergeek/Mistral-7B-v0.2_EOT).
- Merge the LoRA's weights to the Mistral-7B-v0.2_EOT model.
This is the result. This model is not meant for use, it was created to test if this method is viable for replacing the base model of fine-tuned models (when tokenizer and weights have not been changed too much). I am uploading here for evaluation. I don't expect this model to match the original OpenChat-3.5-0106 since I used a LoRA with rank 512, so it won't be equivalent to a full fine-tuning. I have been able to extract LoRAs with higher rank, but currently I don't have the resources to merge them with the model as the memory requirements exceed what I have at my disposal.
If you would like to help my work, check my Ko-Fi and/or Patreon:
* https://ko-fi.com/pretergeek
* https://patreon.com/Pretergeek
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Pretergeek__openchat-3.5-0106_Rebased_Mistral-7B-v0.2)
| Metric |Value|
|-------------------|----:|
|Avg. |15.94|
|IFEval (0-Shot) |37.06|
|BBH (3-Shot) |10.91|
|MATH Lvl 5 (4-Shot)| 3.85|
|GPQA (0-shot) | 2.91|
|MuSR (0-shot) |20.57|
|MMLU-PRO (5-shot) |20.33|
|