Upload folder using huggingface_hub

README.md

@@ -0,0 +1,71 @@
+---
+base_model:
+- openbmb/Eurux-8x22b-nca
+- alpindale/WizardLM-2-8x22B
+- fireworks-ai/mixtral-8x22b-instruct-oh
+- migtissera/Tess-2.0-Mixtral-8x22B
+library_name: transformers
+tags:
+- mergekit
+- merge
+
+---
+# WizardLM-2-8x22B-Beige
+
+This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
+
+
+## Merge Details
+### Merge Method
+
+This model was merged using the [Model Stock](https://arxiv.org/abs/2403.19522) merge method using [alpindale/WizardLM-2-8x22B](https://huggingface.co/alpindale/WizardLM-2-8x22B) as a base.
+
+### Models Merged
+
+The following models were included in the merge:
+* [openbmb/Eurux-8x22b-nca](https://huggingface.co/openbmb/Eurux-8x22b-nca)
+* [fireworks-ai/mixtral-8x22b-instruct-oh](https://huggingface.co/fireworks-ai/mixtral-8x22b-instruct-oh)
+* [migtissera/Tess-2.0-Mixtral-8x22B](https://huggingface.co/migtissera/Tess-2.0-Mixtral-8x22B)
+
+### Configuration
+
+The following YAML configuration was used to produce this model:
+
+```yaml
+models:
+  - model: alpindale/WizardLM-2-8x22B
+  - model: openbmb/Eurux-8x22b-nca
+  - model: migtissera/Tess-2.0-Mixtral-8x22B
+  - model: fireworks-ai/mixtral-8x22b-instruct-oh
+base_model: alpindale/WizardLM-2-8x22B
+merge_method: model_stock
+dtype: bfloat16
+
+```
+
+This model is an attempt to tone down the purple prose of WizardLM2-8X22B
+Here's an example:
+
+Prompt:
+```
+USER: A dead cat is placed into a box along with a nuclear isotope, a vial of poison and a radiation detector. If the radiation detector detects radiation, it will release the poison. The box is opened one day later. What is the probability of the cat being alive?
+Repeat the question before answering it.
+```
+
+
+WizardLM2-8X22B-Beige:
+```
+ASSISTANT: A dead cat is placed into a box along with a nuclear isotope, a vial of poison, and a radiation detector. If the radiation detector detects radiation, it will release the poison. The box is opened one day later. What is the probability of the cat being alive?
+The probability of the cat being alive is zero. Since the cat is dead when placed into the box, it will remain dead regardless of the radiation detector or the poison being released or not.
+```
+
+WizardLM2-8X22B:
+```
+The scenario you've described is a thought experiment that is often referred to as "Schrödinger's cat," a famous paradox in quantum mechanics. The experiment was devised by physicist Erwin Schrödinger in 1935. Repeating the question: If a dead cat is placed into a box along with a nuclear isotope, a vial of poison, and a radiation detector that will release the poison upon detecting radiation, what is the probability of the cat being alive when the box is opened one day later?
+
+In the original paradox, the cat is considered to be simultaneously alive and dead until the box is opened and the cat is observed. This is a way to illustrate the concept of superposition in quantum mechanics, where a quantum system is said to exist in all possible states simultaneously until it is measured.
+
+However, in the scenario you've described, the cat is explicitly stated to be dead when placed in the box. Therefore, regardless of the quantum state of the isotope or the function of the radiation detector and poison, the cat is not alive to begin with. The probability of the cat being alive when the box is opened is zero because it was dead at the outset.
+
+In the classic Schrödinger's cat thought experiment, if the cat were alive when placed in the box, quantum mechanics would suggest that there is a non-zero probability that the cat could be either alive or dead when the box is opened, depending on whether the isotope has decayed and triggered the release of the poison. But in your version, with the cat being definitively dead initially, there is no probability of it being alive later—it remains dead.
+```

config.json

@@ -0,0 +1,42 @@
+{
+    "_name_or_path": "alpindale/WizardLM-2-8x22B",
+    "architectures": [
+        "MixtralForCausalLM"
+    ],
+    "attention_dropout": 0.0,
+    "bos_token_id": 1,
+    "eos_token_id": 2,
+    "hidden_act": "silu",
+    "hidden_size": 6144,
+    "initializer_range": 0.02,
+    "intermediate_size": 16384,
+    "max_position_embeddings": 65536,
+    "model_type": "mixtral",
+    "num_attention_heads": 48,
+    "num_experts_per_tok": 2,
+    "num_hidden_layers": 56,
+    "num_key_value_heads": 8,
+    "num_local_experts": 8,
+    "output_router_logits": false,
+    "rms_norm_eps": 1e-05,
+    "rope_theta": 1000000,
+    "router_aux_loss_coef": 0.001,
+    "router_jitter_noise": 0.0,
+    "sliding_window": null,
+    "tie_word_embeddings": false,
+    "torch_dtype": "bfloat16",
+    "transformers_version": "4.41.2",
+    "use_cache": false,
+    "vocab_size": 32000,
+    "quantization_config": {
+        "quant_method": "exl2",
+        "version": "0.1.5",
+        "bits": 3.0,
+        "head_bits": 6,
+        "calibration": {
+            "rows": 100,
+            "length": 2048,
+            "dataset": "(default)"
+        }
+    }
+}

mergekit_config.yml

@@ -0,0 +1,8 @@
+models:
+  - model: alpindale/WizardLM-2-8x22B
+  - model: openbmb/Eurux-8x22b-nca
+  - model: migtissera/Tess-2.0-Mixtral-8x22B
+  - model: fireworks-ai/mixtral-8x22b-instruct-oh
+base_model: alpindale/WizardLM-2-8x22B
+merge_method: model_stock
+dtype: bfloat16

output-00001-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d1141985917f2d0ee0122bf02aab694be01beb5e31b741188071ae23bdf033b2
+size 8579261352

output-00002-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:65632fd1ec2ad592d25d8a9a363d04bb62d5c28ce0cad1d1d0aad56eb7c78916
+size 8578378816

output-00003-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:50ae394f6797b4b916cc977d6b9816d0e9d1435866ca7e1673b15729f647ba56
+size 8583491000

output-00004-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6233dc49c1526653a0076dca718910300475f4b332b5e4dd4a4dcaf444e88fe7
+size 8579855560

output-00005-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c82e98df2edfe768c3b6c07abcea4683f6677f89d605e5d9f00177378557ba33
+size 8589054440

output-00006-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0b7b499777d3cfa999e72ed823bee218c5474fced051db15b6e1fd25d6685f0
+size 8573362328

output-00007-of-00007.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0968a33cc8ee6892c5268352fbde52e0f6e9c227b5f94ef1dd20fa0eaf73e9ac
+size 1650569456

special_tokens_map.json

@@ -0,0 +1,30 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
+size 493443

tokenizer_config.json

@@ -0,0 +1,45 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "chat_template": "{% if messages[0]['role'] == 'system' %}{% set loop_messages = messages[1:] %}{{ messages[0]['content'].strip() }}{% else %}{% set loop_messages = messages %}{{ 'A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user\\'s questions.' }}{% endif %}{% for message in loop_messages %}{% if loop.index0 == 0 %}{% if message['role'] == 'system' or message['role'] == 'user' %}{{ ' USER: ' + message['content'].strip() }}{% else %}{{ ' ASSISTANT: ' + message['content'].strip() + eos_token }}{% endif %}{% else %}{% if message['role'] == 'system' or message['role'] == 'user' %}{{ '\nUSER: ' + message['content'].strip() }}{% else %}{{ ' ASSISTANT: ' + message['content'].strip() + eos_token }}{% endif %}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ ' ASSISTANT:' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<unk>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true
+}