---
license: apache-2.0
tags:
- merge
- mergekit
base_model:
- failspy/Llama-3-8B-Instruct-abliterated
model-index:
- name: Aura-Llama-Abliterated
  results:
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: AI2 Reasoning Challenge (25-Shot)
      type: ai2_arc
      config: ARC-Challenge
      split: test
      args:
        num_few_shot: 25
    metrics:
    - type: acc_norm
      value: 49.23
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: HellaSwag (10-Shot)
      type: hellaswag
      split: validation
      args:
        num_few_shot: 10
    metrics:
    - type: acc_norm
      value: 72.27
      name: normalized accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: MMLU (5-Shot)
      type: cais/mmlu
      config: all
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 55.71
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: TruthfulQA (0-shot)
      type: truthful_qa
      config: multiple_choice
      split: validation
      args:
        num_few_shot: 0
    metrics:
    - type: mc2
      value: 46.63
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: Winogrande (5-shot)
      type: winogrande
      config: winogrande_xl
      split: validation
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 69.3
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
      name: Open LLM Leaderboard
  - task:
      type: text-generation
      name: Text Generation
    dataset:
      name: GSM8k (5-shot)
      type: gsm8k
      config: main
      split: test
      args:
        num_few_shot: 5
    metrics:
    - type: acc
      value: 27.6
      name: accuracy
    source:
      url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=TheSkullery/Aura-Llama-Abliterated
      name: Open LLM Leaderboard
---
**Exllamav2** quant (**exl2** / **5.0 bpw**) made with ExLlamaV2 v0.0.21

Other EXL2 quants:
| **Quant** | **Model Size** | **lm_head** |
| ----- | ---------- | ------- |
|<center>**[2.2](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-2_2bpw_exl2)**</center> | <center>3944 MB</center> | <center>6</center> |
|<center>**[2.5](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-2_5bpw_exl2)**</center> | <center>4258 MB</center> | <center>6</center> |
|<center>**[3.0](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-3_0bpw_exl2)**</center> | <center>4829 MB</center> | <center>6</center> |
|<center>**[3.5](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-3_5bpw_exl2)**</center> | <center>5403 MB</center> | <center>6</center> |
|<center>**[3.75](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-3_75bpw_exl2)**</center> | <center>5688 MB</center> | <center>6</center> |
|<center>**[4.0](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-4_0bpw_exl2)**</center> | <center>5975 MB</center> | <center>6</center> |
|<center>**[4.25](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-4_25bpw_exl2)**</center> | <center>6260 MB</center> | <center>6</center> |
|<center>**[5.0](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-5_0bpw_exl2)**</center> | <center>7115 MB</center> | <center>6</center> |
|<center>**[6.0](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-6_0bpw_exl2)**</center> | <center>8369 MB</center> | <center>8</center> |
|<center>**[6.5](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-6_5bpw_exl2)**</center> | <center>8934 MB</center> | <center>8</center> |
|<center>**[8.0](https://huggingface.co/Zoyd/TheSkullery_Aura-Llama-Abliterated-8_0bpw_exl2)**</center> | <center>10593 MB</center> | <center>8</center> |


<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0"> 
  <title>Aura-llama-3 Data Card</title> 
  <link href="https://fonts.googleapis.com/css2?family=Quicksand:wght@400;500;600&display=swap" rel="stylesheet"> 
  <style> body { font-family: 'Quicksand', sans-serif; background: linear-gradient(135deg, #2E3440 0%, #1A202C 100%); color: #D8DEE9; margin: 0; padding: 0; font-size: 16px; } 
    .container { width: 100%; max-width: 1440px; margin: 20px auto; background-color: rgba(255, 255, 255, 0.02); padding: 20px; border-radius: 12px; box-shadow: 0 4px 10px rgba(0, 0, 0, 0.2); backdrop-filter: blur(10px); border: 1px solid rgba(255, 255, 255, 0.1); } 
    .header h1 { font-size: 28px; color: #ECEFF4; margin: 0 0 20px 0; text-shadow: 2px 2px 4px rgba(0, 0, 0, 0.3); } 
    .update-section { margin-top: 30px; } .update-section h2 { font-size: 24px; color: #88C0D0; } 
    .update-section p { font-size: 16px; line-height: 1.6; color: #ECEFF4; } 
    .info img { width: 100%; border-radius: 10px; margin-bottom: 15px; } 
    a { color: #88C0D0; text-decoration: none; } 
    a:hover { color: #A3BE8C; } 
    pre { background-color: rgba(255, 255, 255, 0.05); padding: 10px; border-radius: 5px; overflow-x: auto; } 
    code { font-family: 'Courier New', monospace; color: #A3BE8C; } </style> </head> <body> <div class="container"> 
      <div class="header"> 
        <h1>Aura-llama-3-Abliterated</h1> </div> <div class="info"> 
          <img src="https://cdn-uploads.huggingface.co/production/uploads/64545af5ec40bbbd01242ca6/AwLNDVB-GIY7k0wnVV_TX.png" alt="Aura-llama-Abliterated Image"> 
          <p>Now that the cute anime girl has your attention.</p> 
          <p>UPDATE: Model is now using the abliterated version of meta llama 3 8b </p>
          <p>Aura-llama is using the methodology presented by SOLAR for scaling LLMs called depth up-scaling (DUS), which encompasses architectural modifications with continued pretraining. Using the solar paper as a base, I integrated Llama-3 weights into the upscaled layers, and In the future plan to continue training the model.</p> 
          <p>Aura-llama is a merge of the following models to create a base model to work from:</p> 
          <ul> 
            <li><a href="https://huggingface.co/failspy/Llama-3-8B-Instruct-abliterated">failspy/Llama-3-8B-Instruct-abliterated</a></li> 
            <li><a href="https://huggingface.co/failspy/Llama-3-8B-Instruct-abliterated">failspy/Llama-3-8B-Instruct-abliterated</a></li> 
          </ul> 
        </div> 
      <div class="update-section"> 
        <h2>Abliterated Merged Evals (Has Not Been Finetuned):</h2> 
        <p>Aura-llama-Abliterated</p> 
        <ul> 
          <li>Avg: ?</li> 
          <li>ARC: ?</li> 
          <li>HellaSwag: ?</li> 
          <li>MMLU: ?</li> 
          <li>T-QA: ?</li> 
          <li>Winogrande: ?</li> 
          <li>GSM8K: ?</li> 
        </ul> 
        <h2>Non Abliterated Merged Evals (Has Not Been Finetuned):</h2> 
        <p>Aura-llama-Original</p> 
        <ul> 
          <li>Avg: 63.13</li> 
          <li>ARC: 58.02</li> 
          <li>HellaSwag: 77.82</li> 
          <li>MMLU: 65.61</li> 
          <li>T-QA: 51.94</li> 
          <li>Winogrande: 73.40</li> 
          <li>GSM8K: 52.01</li> 
        </ul> 
      </div> 
      <div class="update-section"> 
        <h2>🧩 Configuration</h2> 
        <pre><code>
dtype: bfloat16
merge_method: passthrough
slices:
- sources:
  - layer_range: [0, 12]
    model: failspy/Llama-3-8B-Instruct-abliterated
- sources:
  - layer_range: [8, 20]
    model: failspy/Llama-3-8B-Instruct-abliterated
- sources:
  - layer_range: [16, 28]
    model: failspy/Llama-3-8B-Instruct-abliterated
- sources:
  - layer_range: [24, 32]
    model: failspy/Llama-3-8B-Instruct-abliterated
        </code></pre> 
      </div> 
    </div> 
    </body> 
    </html>

# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_TheSkullery__Aura-Llama-Abliterated)

|             Metric              |Value|
|---------------------------------|----:|
|Avg.                             |53.46|
|AI2 Reasoning Challenge (25-Shot)|49.23|
|HellaSwag (10-Shot)              |72.27|
|MMLU (5-Shot)                    |55.71|
|TruthfulQA (0-shot)              |46.63|
|Winogrande (5-shot)              |69.30|
|GSM8k (5-shot)                   |27.60|