Mubin1917 commited on
Commit
e08207b
1 Parent(s): 1681022

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +21 -1
README.md CHANGED
@@ -14,7 +14,7 @@ tags:
14
 
15
  ## Overview
16
 
17
- The **Fhi-3.5-mini-instruct** is a fine-tuned version of the [unsloth/Phi-3.5-mini-instruct](https://huggingface.co/unsloth/Phi-3.5-mini-instruct) model, optimized for function-calling capability. This model provides fast, accurate, and structured responses based on input queries and available APIs. It supports enhanced function-calling features on top of its existing Phi-3.5-mini-instruct's capabilities.
18
 
19
  ### Usage
20
 
@@ -64,5 +64,25 @@ The result will look like this:
64
  ## Testing and Benchmarking
65
  This model is still undergoing testing and evaluation. Use it at your own risk until further validation is complete. Performance on benchmarks like MMLU and MMLU-Pro will be updated soon.
66
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
67
  ## Credits
68
  Will be updated soon
 
14
 
15
  ## Overview
16
 
17
+ The **Fhi-3.5-mini-instruct** is a fine-tuned version of the [unsloth/Phi-3.5-mini-instruct](https://huggingface.co/unsloth/Phi-3.5-mini-instruct) model, optimized for function-calling.
18
 
19
  ### Usage
20
 
 
64
  ## Testing and Benchmarking
65
  This model is still undergoing testing and evaluation. Use it at your own risk until further validation is complete. Performance on benchmarks like MMLU and MMLU-Pro will be updated soon.
66
 
67
+ | Benchmark | Fhi-3.5 Mini-Ins | Phi-3.5 Mini-Ins | Mistral-7B-Instruct-v0.3 | Mistral-Nemo-12B-Ins-2407 | Llama-3.1-8B-Ins | Gemma-2-9B-Ins | Gemini 1.5 Flash | GPT-4o-mini-2024-07-18 (Chat) |
68
+ |----------------------------|------------------|------------------|--------------------------|---------------------------|------------------|----------------|------------------|-------------------------------|
69
+ | Multilingual MMLU | ____ | 55.4 | 47.4 | 58.9 | 56.2 | 63.8 | 77.2 | 72.9 |
70
+ <!-- | Multilingual MMLU-Pro | 30.9 | 30.21 | 15.0 | 34.0 | 21.4 | 43.0 | 57.9 | 53.2 |
71
+ | **Average** | **55.2** | **52.3** | **47.9** | **55.3** | **47.5** | **59.6** | **64.3** | **76.6** |
72
+ -->
73
+ <!-- The table below shows Multilingual MMLU scores in some of the supported languages.
74
+
75
+ | Benchmark | Phi-3.5 Mini-Ins | Phi-3.0-Mini-128k-Instruct (June2024) | Mistral-7B-Instruct-v0.3 | Mistral-Nemo-12B-Ins-2407 | Llama-3.1-8B-Ins | Gemma-2-9B-Ins | Gemini 1.5 Flash | GPT-4o-mini-2024-07-18 (Chat) |
76
+ |-----------|------------------|-----------------------|--------------------------|---------------------------|------------------|----------------|------------------|-------------------------------|
77
+ | Arabic | 44.2 | 35.4 | 33.7 | 45.3 | 49.1 | 56.3 | 73.6 | 67.1 |
78
+ | Chinese | 52.6 | 46.9 | 45.9 | 58.2 | 54.4 | 62.7 | 66.7 | 70.8 |
79
+ | Dutch | 57.7 | 48.0 | 51.3 | 60.1 | 55.9 | 66.7 | 80.6 | 74.2 |
80
+ | French | 61.1 | 61.7 | 53.0 | 63.8 | 62.8 | 67.0 | 82.9 | 75.6 |
81
+ | German | 62.4 | 61.3 | 50.1 | 64.5 | 59.9 | 65.7 | 79.5 | 74.3 |
82
+ | Italian | 62.8 | 63.1 | 52.5 | 64.1 | 55.9 | 65.7 | 82.6 | 75.9 |
83
+ | Russian | 50.4 | 45.3 | 48.9 | 59.0 | 57.4 | 63.2 | 78.7 | 72.6 |
84
+ | Spanish | 62.6 | 61.3 | 53.9 | 64.3 | 62.6 | 66.0 | 80.0 | 75.5 |
85
+ | Ukrainian | 45.2 | 36.7 | 46.9 | 56.6 | 52.9 | 62.0 | 77.4 | 72.6 |
86
+ -->
87
  ## Credits
88
  Will be updated soon