TheSkullery
/

Aura-llama

@@ -2,26 +2,52 @@
 tags:
 - merge
 - mergekit
-- lazymergekit
 - NousResearch/Meta-Llama-3-8B-Instruct
 base_model:
 - NousResearch/Meta-Llama-3-8B-Instruct
-- NousResearch/Meta-Llama-3-8B-Instruct
-- NousResearch/Meta-Llama-3-8B-Instruct
-- NousResearch/Meta-Llama-3-8B-Instruct
 ---
-# Aura-llama
-Aura-llama is a merge of the following models using [LazyMergekit](https://colab.research.google.com/drive/1obulZ1ROXHjYLn6PPZJwRR6GzgQogxxb?usp=sharing):
-* [NousResearch/Meta-Llama-3-8B-Instruct](https://huggingface.co/NousResearch/Meta-Llama-3-8B-Instruct)
-* [NousResearch/Meta-Llama-3-8B-Instruct](https://huggingface.co/NousResearch/Meta-Llama-3-8B-Instruct)
-* [NousResearch/Meta-Llama-3-8B-Instruct](https://huggingface.co/NousResearch/Meta-Llama-3-8B-Instruct)
-* [NousResearch/Meta-Llama-3-8B-Instruct](https://huggingface.co/NousResearch/Meta-Llama-3-8B-Instruct)
-## 🧩 Configuration
-```yaml
 dtype: float16
 merge_method: passthrough
 slices:
@@ -37,29 +63,8 @@ slices:
 - sources:
   - layer_range: [24, 32]
     model: NousResearch/Meta-Llama-3-8B-Instruct
-```
-## 💻 Usage
-```python
-!pip install -qU transformers accelerate
-from transformers import AutoTokenizer
-import transformers
-import torch
-model = "TheSkullery/Aura-llama"
-messages = [{"role": "user", "content": "What is a large language model?"}]
-tokenizer = AutoTokenizer.from_pretrained(model)
-prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
-pipeline = transformers.pipeline(
-    "text-generation",
-    model=model,
-    torch_dtype=torch.float16,
-    device_map="auto",
-)
-outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
-print(outputs[0]["generated_text"])
-```

 tags:
 - merge
 - mergekit
 - NousResearch/Meta-Llama-3-8B-Instruct
 base_model:
 - NousResearch/Meta-Llama-3-8B-Instruct
 ---
+<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <meta name="viewport" content="width=device-width, initial-scale=1.0">
+  <title>Aura-llama Data Card</title>
+  <link href="https://fonts.googleapis.com/css2?family=Quicksand:wght@400;500;600&display=swap" rel="stylesheet">
+  <style> body { font-family: 'Quicksand', sans-serif; background: linear-gradient(135deg, #2E3440 0%, #1A202C 100%); color: #D8DEE9; margin: 0; padding: 0; font-size: 16px; }
+    .container { width: 80%; max-width: 800px; margin: 20px auto; background-color: rgba(255, 255, 255, 0.02); padding: 20px; border-radius: 12px; box-shadow: 0 4px 10px rgba(0, 0, 0, 0.2); backdrop-filter: blur(10px); border: 1px solid rgba(255, 255, 255, 0.1); }
+    .header h1 { font-size: 28px; color: #ECEFF4; margin: 0 0 20px 0; text-shadow: 2px 2px 4px rgba(0, 0, 0, 0.3); }
+    .update-section { margin-top: 30px; } .update-section h2 { font-size: 24px; color: #88C0D0; }
+    .update-section p { font-size: 16px; line-height: 1.6; color: #ECEFF4; }
+    .info img { width: 100%; border-radius: 10px; margin-bottom: 15px; }
+    a { color: #88C0D0; text-decoration: none; }
+    a:hover { color: #A3BE8C; }
+    pre { background-color: rgba(255, 255, 255, 0.05); padding: 10px; border-radius: 5px; overflow-x: auto; }
+    code { font-family: 'Courier New', monospace; color: #A3BE8C; } </style> </head> <body> <div class="container">
+      <div class="header">
+        <h1>Aura-llama</h1> </div> <div class="info">
+          <img src="https://cdn-uploads.huggingface.co/production/uploads/64545af5ec40bbbd01242ca6/QYpWMEXTe0_X3A7HyeBm0.webp" alt="Aura-llama image">
+          <p>Now that the cute anime girl has your attention.</p>
+          <p>UPDATE: Model has been fixed</p>
+          <p>Aura-llama is using the methodology presented by SOLAR for scaling LLMs called depth up-scaling (DUS), which encompasses architectural modifications with continued pretraining. Using the solar paper as a base, I integrated Llama-3 weights into the upscaled layers, and In the future plan to continue training the model.</p>
+          <p>Aura-llama is a merge of the following models to create a base model to work from:</p>
+          <ul>
+            <li><a href="https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct">meta-llama/Meta-Llama-3-8B-Instruct</a></li>
+            <li><a href="https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct">meta-llama/Meta-Llama-3-8B-Instruct</a></li>
+          </ul>
+        </div>
+      <div class="update-section">
+        <h2>Merged Evals (Has Not Been Finetuned):</h2>
+        <p>Aura-llama</p>
+        <ul>
+          <li>Avg: ?</li>
+          <li>ARC: ?</li>
+          <li>HellaSwag: ?</li>
+          <li>MMLU: ?</li>
+          <li>T-QA: ?</li>
+          <li>Winogrande: ?</li>
+          <li>GSM8K: ?</li>
+        </ul>
+      </div>
+      <div class="update-section">
+        <h2>🧩 Configuration</h2>
+        <pre><code>
 dtype: float16
 merge_method: passthrough
 slices:
 - sources:
   - layer_range: [24, 32]
     model: NousResearch/Meta-Llama-3-8B-Instruct
+        </code></pre>
+      </div>
+    </div>
+    </body>
+    </html>