README.md

---
license: other
library_name: transformers
base_model: meta-llama/Meta-Llama-3-8B
datasets:
- mlabonne/orpo-dpo-mix-40k
- Open-Orca/SlimOrca-Dedup
- jondurbin/airoboros-3.2
- microsoft/orca-math-word-problems-200k
- m-a-p/Code-Feedback
- MaziyarPanahi/WizardLM_evol_instruct_V2_196k
model-index:
- name: llama-3-neural-chat-v1-8b
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
    - type: acc_norm
      value: 60.84
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag (10-Shot)
      type: hellaswag
      split: validation
      args:
        num_few_shot: 10
    metrics:
    - type: acc_norm
      value: 84.13
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU (5-Shot)
      type: cais/mmlu
      config: all
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 64.69
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: TruthfulQA (0-shot)
      type: truthful_qa
      config: multiple_choice
      split: validation
      args:
        num_few_shot: 0
    metrics:
    - type: mc2
      value: 56.34
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: Winogrande (5-shot)
      type: winogrande
      config: winogrande_xl
      split: validation
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 78.22
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GSM8k (5-shot)
      type: gsm8k
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 54.81
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Locutusque/llama-3-neural-chat-v1-8b
      name: Open LLM Leaderboard
---

# llama-3-neural-chat-v1-8b

<!-- Provide a quick summary of what the model is/does. -->


![image/png](https://cdn-uploads.huggingface.co/production/uploads/6437292ecd93f4c9a34b0d47/6XQuhjWNr6C4RbU9f1k99.png)



## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

I fine-tuned llama-3 8B on an approach similar to Intel's neural chat language model. I have slightly modified the data sources so it is stronger in coding, math, and writing. I use both SFT and DPO.

- **Developed by:** Locutusque
- **Model type:** Built with Meta Llama 3
- **Language(s) (NLP):** Many?
- **License:** Llama 3 license https://huggingface.co/meta-llama/Meta-Llama-3-8B/blob/main/LICENSE

## Quants

### EXL2 [@bartowski](https://huggingface.co/bartowski/)

- https://huggingface.co/bartowski/llama-3-neural-chat-v1-8b-exl2

### GGUF [@bartowski](https://huggingface.co/bartowski/)

- https://huggingface.co/bartowski/llama-3-neural-chat-v1-8b-GGUF

## Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

This model has great performance in writing and coding.

## Training Data
- Open-Orca/SlimOrca-Dedup
- jondurbin/airoboros-3.2
- microsoft/orca-math-word-problems-200k
- m-a-p/Code-Feedback
- MaziyarPanahi/WizardLM_evol_instruct_V2_196k
- mlabonne/orpo-dpo-mix-40k

### Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

Conversational AI.

## Evaluations

|              Tasks              |Version|     Filter     |n-shot|  Metric   |Value |   |Stderr|
|---------------------------------|-------|----------------|-----:|-----------|-----:|---|-----:|
|truthfulqa_mc2                   |      2|none            |     0|acc        |0.5627|±  |0.0154|
|gsm8k                            |      3|strict-match    |     5|exact_match|0.5481|±  |0.0137|
|                                 |       |flexible-extract|     5|exact_match|0.5557|±  |0.0137|
|agieval_nous                     |N/A    |none            |     0|acc        |0.3763|±  |0.0093|
|                                 |       |none            |     0|acc_norm   |0.3665|±  |0.0093|
| - agieval_aqua_rat              |      1|none            |     0|acc        |0.2087|±  |0.0255|
|                                 |       |none            |     0|acc_norm   |0.2047|±  |0.0254|
| - agieval_logiqa_en             |      1|none            |     0|acc        |0.3456|±  |0.0187|
|                                 |       |none            |     0|acc_norm   |0.3594|±  |0.0188|
| - agieval_lsat_ar               |      1|none            |     0|acc        |0.1826|±  |0.0255|
|                                 |       |none            |     0|acc_norm   |0.1783|±  |0.0253|
| - agieval_lsat_lr               |      1|none            |     0|acc        |0.3549|±  |0.0212|
|                                 |       |none            |     0|acc_norm   |0.3451|±  |0.0211|
| - agieval_lsat_rc               |      1|none            |     0|acc        |0.5242|±  |0.0305|
|                                 |       |none            |     0|acc_norm   |0.5130|±  |0.0305|
| - agieval_sat_en                |      1|none            |     0|acc        |0.6650|±  |0.0330|
|                                 |       |none            |     0|acc_norm   |0.6505|±  |0.0333|
| - agieval_sat_en_without_passage|      1|none            |     0|acc        |0.4175|±  |0.0344|
|                                 |       |none            |     0|acc_norm   |0.3738|±  |0.0338|
| - agieval_sat_math              |      1|none            |     0|acc        |0.4227|±  |0.0334|
|                                 |       |none            |     0|acc_norm   |0.3682|±  |0.0326|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Locutusque__llama-3-neural-chat-v1-8b)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |66.50|
|AI2 Reasoning Challenge (25-Shot)|60.84|
|HellaSwag (10-Shot)              |84.13|
|MMLU (5-Shot)                    |64.69|
|TruthfulQA (0-shot)              |56.34|
|Winogrande (5-shot)              |78.22|
|GSM8k (5-shot)                   |54.81|