Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,12 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+Medorca-2x7b-Q3_K_L.gguf filter=lfs diff=lfs merge=lfs -text
+Medorca-2x7b-Q3_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Medorca-2x7b-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Medorca-2x7b-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Medorca-2x7b-Q4_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Medorca-2x7b-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+Medorca-2x7b-Q5_K_S.gguf filter=lfs diff=lfs merge=lfs -text
+Medorca-2x7b-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
+Medorca-2x7b-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text

Medorca-2x7b-Q3_K_L.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5adc47244d850ff743298ed809993e90ff870f9cf4aa2e54dcb31e68d5049605
+size 5829553280

Medorca-2x7b-Q3_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d751c74a73b27bf305dc8f66ff087e3ff57dadc7ed453b60f51cd5cd78045ca
+size 5361364096

Medorca-2x7b-Q3_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0c1a914d7e1f420fe2a1686e365b5403cf09904742eab4d76bdb45fbb5653e7c
+size 4808764544

Medorca-2x7b-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e71e2a3fcb1b7e78cdab94e19ed185f07a01c8a15e3504a89d781efc3697ea8f
+size 6702338784

Medorca-2x7b-Q4_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:694c6ebd3ffc158703cc63047cdeeade03b6cef8a1e120a5ebff81b0592d66c8
+size 6314627808

Medorca-2x7b-Q5_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cb7d6d67c4de9b5795f437017ea65ecdfc86870a79ee3322307d7cebda40d039
+size 7855380704

Medorca-2x7b-Q5_K_S.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b2af7f642471e460dcd68b04a0baba0a3b693856b23eb60e957c67d8b8461aec
+size 7628101856

Medorca-2x7b-Q6_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:893c13d7289e29840c670111f116d02ef228615fafcb81d2df5c4ed4589ec0ed
+size 9080487744

Medorca-2x7b-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5772b4c3970f3c07e4707ad5a8889fe40d970e0e011516b26793ad87fbdbd4b4
+size 11760703104

README.md

@@ -0,0 +1,94 @@
+---
+license: apache-2.0
+tags:
+- moe
+- merge
+- epfl-llm/meditron-7b
+- microsoft/Orca-2-7b
+---
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/63486df1f8f01fcc4b23e97d/MVYcLAR1Inm5dY-XHiAhe.png)
+
+# Medorca-2x7b
+
+Medorca-2x7b is a Mixure of Experts (MoE) made with the following models:
+* [epfl-llm/meditron-7b](https://huggingface.co/epfl-llm/meditron-7b)
+* [microsoft/Orca-2-7b](https://huggingface.co/microsoft/Orca-2-7b)
+
+## Evaluations
+
+| Benchmark | Medorca-2x7b | Orca-2-7b | llama-2-7b | meditron-7b | meditron-70b |
+| --- | --- | --- | --- | --- | --- |
+| MedMCQA |  |  |  |  |  |
+| ClosedPubMedQA |  |  |  |  |  |
+| PubMedQA |  |  |  |  |  |
+| MedQA |  |  |  |  |  |
+| MedQA4 |  |  |  |  |  |
+| MedicationQA |  |  |  |  |  |
+| MMLU Medical |  |  |  |  |  |
+| MMLU | 53.3 | **56.37** |  |  |  |
+| TruthfulQA | 48.04 | **52.45** |  |  |  |
+| GSM8K | **20.64** | 14.71 |  |  |  |
+| ARC | 54.1 | 54.1 |  |  |  |
+| HellaSwag | 76.04 | **76.19** |  |  |  |
+| Winogrande | **74.51** | 73.48 |  |  |  |
+
+More details on the Open LLM Leaderboard evaluation results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_Technoculture__Medorca-2x7b.)
+
+## 🧩 Configuration
+
+```yaml
+base_model: microsoft/Orca-2-7b
+gate_mode: hidden
+dtype: bfloat16
+experts:
+  - source_model: epfl-llm/meditron-7b
+    positive_prompts: 
+      - "How does sleep affect cardiovascular health?"
+      - "Could a plant-based diet improve arthritis symptoms?"
+      - "A patient comes in with symptoms of dizziness and nausea..."
+      - "When discussing diabetes management, the key factors to consider are..."
+      - "The differential diagnosis for a headache with visual aura could include..."
+    negative_prompts:
+      - "Recommend a good recipe for a vegetarian lasagna."
+      - "Give an overview of the French Revolution."
+      - "Explain how a digital camera captures an image."
+      - "What are the environmental impacts of deforestation?"
+      - "The recent advancements in artificial intelligence have led to developments in..."
+      - "The fundamental concepts in economics include ideas like supply and demand, which explain..."
+  - source_model: microsoft/Orca-2-7b
+    positive_prompts:
+      - "Here is a funny joke for you -"
+      - "When considering the ethical implications of artificial intelligence, one must take into account..."
+      - "In strategic planning, a company must analyze its strengths and weaknesses, which involves..."
+      - "Understanding consumer behavior in marketing requires considering factors like..."
+      - "The debate on climate change solutions hinges on arguments that..."
+    negative_prompts:
+      - "In discussing dietary adjustments for managing hypertension, it's crucial to emphasize..."
+      - "For early detection of melanoma, dermatologists recommend that patients regularly check their skin for..."
+      - "Explaining the importance of vaccination, a healthcare professional should highlight..."
+```
+
+## 💻 Usage
+
+```python
+!pip install -qU transformers bitsandbytes accelerate
+
+from transformers import AutoTokenizer
+import transformers
+import torch
+
+model = "Technoculture/Medorca-2x7b"
+
+tokenizer = AutoTokenizer.from_pretrained(model)
+pipeline = transformers.pipeline(
+    "text-generation",
+    model=model,
+    model_kwargs={"torch_dtype": torch.float16},
+)
+
+messages = [{"role": "user", "content": "Why am i feeling so tired this month?"}]
+prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
+print(outputs[0]["generated_text"])
+```

huggingface-metadata.txt

@@ -0,0 +1,10 @@
+url: https://huggingface.co/Technoculture/Medorca-2x7b
+branch: main
+download date: 2024-01-22 10:17:21
+sha256sum:
+    b7312033024738baff24be5c914396422b027380cc3cbc44cb6c0eb62b74cf94 model-00001-of-00002.safetensors
+    f1c840e1ed0c60c73ab21953cf450dee5362a2412db3411fdae858d649d08261 model-00001-of-00003.safetensors
+    89ae0a8853bb7c1d3029f8a3bb295d8ba70835180159f5edd840fd28d24d947a model-00002-of-00002.safetensors
+    ae2432a8a64091cc18c79124794cdebdef9b2d0dfe6a6d466398f53e4d77e57e model-00002-of-00003.safetensors
+    24f8c33dd495ac7f6bb3210049edf02737817427591f78ce81257a78abad30bc model-00003-of-00003.safetensors
+    9e556afd44213b6bd1be2b850ebbbd98f5481437a8021afaf58ee7fb1818d347 tokenizer.model

mergekit_config.yml

@@ -0,0 +1,17 @@
+
+slices:
+  - sources:
+      - model: epfl-llm/meditron-7b
+        layer_range: [0, 32]
+      - model: microsoft/Orca-2-7b
+        layer_range: [0, 32]
+merge_method: slerp
+base_model: epfl-llm/meditron-7b
+parameters:
+  t:
+    - filter: self_attn
+      value: [0, 0.5, 0.3, 0.7, 1]
+    - filter: mlp
+      value: [1, 0.5, 0.7, 0.3, 0]
+    - value: 0.5 # fallback for rest of tensors
+dtype: float16

mergekit_moe_config.yml

@@ -0,0 +1,30 @@
+
+base_model: microsoft/Orca-2-7b
+gate_mode: hidden
+dtype: bfloat16
+experts:
+  - source_model: epfl-llm/meditron-7b
+    positive_prompts: 
+      - "How does sleep affect cardiovascular health?"
+      - "Could a plant-based diet improve arthritis symptoms?"
+      - "A patient comes in with symptoms of dizziness and nausea..."
+      - "When discussing diabetes management, the key factors to consider are..."
+      - "The differential diagnosis for a headache with visual aura could include..."
+    negative_prompts:
+      - "Recommend a good recipe for a vegetarian lasagna."
+      - "Give an overview of the French Revolution."
+      - "Explain how a digital camera captures an image."
+      - "What are the environmental impacts of deforestation?"
+      - "The recent advancements in artificial intelligence have led to developments in..."
+      - "The fundamental concepts in economics include ideas like supply and demand, which explain..."
+  - source_model: microsoft/Orca-2-7b
+    positive_prompts:
+      - "Here is a funny joke for you -"
+      - "When considering the ethical implications of artificial intelligence, one must take into account..."
+      - "In strategic planning, a company must analyze its strengths and weaknesses, which involves..."
+      - "Understanding consumer behavior in marketing requires considering factors like..."
+      - "The debate on climate change solutions hinges on arguments that..."
+    negative_prompts:
+      - "In discussing dietary adjustments for managing hypertension, it's crucial to emphasize..."
+      - "For early detection of melanoma, dermatologists recommend that patients regularly check their skin for..."
+      - "Explaining the importance of vaccination, a healthcare professional should highlight..."