Hugging Face's logo

Technoculture
/
Medmerge-tulu-70b

Text Generation
Transformers
Safetensors
llama
Merge
mergekit
epfl-llm/meditron-70b
allenai/tulu-2-dpo-70b
text-generation-inference
Inference Endpoints
Model card Files Community
2
satyamt commited on
Commit
9f9c8e0
•
1 Parent(s): 2afc638

Upload folder using huggingface_hub

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. README.md +64 -0
  2. config.json +29 -0
  3. mergekit_config.yml +20 -0
  4. model-00001-of-00080.safetensors +3 -0
  5. model-00002-of-00080.safetensors +3 -0
  6. model-00003-of-00080.safetensors +3 -0
  7. model-00004-of-00080.safetensors +3 -0
  8. model-00005-of-00080.safetensors +3 -0
  9. model-00006-of-00080.safetensors +3 -0
  10. model-00007-of-00080.safetensors +3 -0
  11. model-00008-of-00080.safetensors +3 -0
  12. model-00009-of-00080.safetensors +3 -0
  13. model-00010-of-00080.safetensors +3 -0
  14. model-00011-of-00080.safetensors +3 -0
  15. model-00012-of-00080.safetensors +3 -0
  16. model-00013-of-00080.safetensors +3 -0
  17. model-00014-of-00080.safetensors +3 -0
  18. model-00015-of-00080.safetensors +3 -0
  19. model-00016-of-00080.safetensors +3 -0
  20. model-00017-of-00080.safetensors +3 -0
  21. model-00018-of-00080.safetensors +3 -0
  22. model-00019-of-00080.safetensors +3 -0
  23. model-00020-of-00080.safetensors +3 -0
  24. model-00021-of-00080.safetensors +3 -0
  25. model-00022-of-00080.safetensors +3 -0
  26. model-00023-of-00080.safetensors +3 -0
  27. model-00024-of-00080.safetensors +3 -0
  28. model-00025-of-00080.safetensors +3 -0
  29. model-00026-of-00080.safetensors +3 -0
  30. model-00027-of-00080.safetensors +3 -0
  31. model-00028-of-00080.safetensors +3 -0
  32. model-00029-of-00080.safetensors +3 -0
  33. model-00030-of-00080.safetensors +3 -0
  34. model-00031-of-00080.safetensors +3 -0
  35. model-00032-of-00080.safetensors +3 -0
  36. model-00033-of-00080.safetensors +3 -0
  37. model-00034-of-00080.safetensors +3 -0
  38. model-00035-of-00080.safetensors +3 -0
  39. model-00036-of-00080.safetensors +3 -0
  40. model-00037-of-00080.safetensors +3 -0
  41. model-00038-of-00080.safetensors +3 -0
  42. model-00039-of-00080.safetensors +3 -0
  43. model-00040-of-00080.safetensors +3 -0
  44. model-00041-of-00080.safetensors +3 -0
  45. model-00042-of-00080.safetensors +3 -0
  46. model-00043-of-00080.safetensors +3 -0
  47. model-00044-of-00080.safetensors +3 -0
  48. model-00045-of-00080.safetensors +3 -0
  49. model-00046-of-00080.safetensors +3 -0
  50. model-00047-of-00080.safetensors +3 -0
README.md ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ tags:
4
+ - merge
5
+ - mergekit
6
+ - wanglab/ClinicalCamel-70B
7
+ - epfl-llm/meditron-70b
8
+ - allenai/tulu-2-dpo-70b
9
+ ---
10
+
11
+ # Medmerge-3-70b
12
+
13
+ Medmerge-3-70b is a merge of the following models:
14
+ * [wanglab/ClinicalCamel-70B](https://huggingface.co/wanglab/ClinicalCamel-70B)
15
+ * [epfl-llm/meditron-70b](https://huggingface.co/epfl-llm/meditron-70b)
16
+ * [allenai/tulu-2-dpo-70b](https://huggingface.co/allenai/tulu-2-dpo-70b)
17
+
18
+ ## 🧩 Configuration
19
+
20
+ ```yamlmodels:
21
+ - model: NousResearch/Llama-2-70b-hf
22
+ # no parameters necessary for base model
23
+ - model: wanglab/ClinicalCamel-70B
24
+ parameters:
25
+ weight: 0.08
26
+ density: 0.45
27
+ - model: epfl-llm/meditron-70b
28
+ parameters:
29
+ weight: 0.08
30
+ density: 0.45
31
+ - model: allenai/tulu-2-dpo-70b
32
+ parameters:
33
+ weight: 0.08
34
+ density: 0.45
35
+ merge_method: dare_ties
36
+ base_model: NousResearch/Llama-2-70b-hf
37
+ parameters:
38
+ int8_mask: true
39
+ dtype: bfloat16```
40
+
41
+ ## 💻 Usage
42
+
43
+ ```python
44
+ !pip install -qU transformers accelerate
45
+
46
+ from transformers import AutoTokenizer
47
+ import transformers
48
+ import torch
49
+
50
+ model = "Technoculture/Medmerge-3-70b"
51
+ messages = [{"role": "user", "content": "I am feeling sleepy these days"}]
52
+
53
+ tokenizer = AutoTokenizer.from_pretrained(model)
54
+ prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
55
+ pipeline = transformers.pipeline(
56
+ "text-generation",
57
+ model=model,
58
+ torch_dtype=torch.float16,
59
+ device_map="auto",
60
+ )
61
+
62
+ outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
63
+ print(outputs[0]["generated_text"])
64
+ ```
config.json ADDED
@@ -0,0 +1,29 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "NousResearch/Llama-2-70b-hf",
3
+ "architectures": [
4
+ "LlamaForCausalLM"
5
+ ],
6
+ "attention_bias": false,
7
+ "attention_dropout": 0.0,
8
+ "bos_token_id": 1,
9
+ "eos_token_id": 2,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 8192,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 28672,
14
+ "max_position_embeddings": 4096,
15
+ "model_type": "llama",
16
+ "num_attention_heads": 64,
17
+ "num_hidden_layers": 80,
18
+ "num_key_value_heads": 8,
19
+ "pad_token_id": 0,
20
+ "pretraining_tp": 1,
21
+ "rms_norm_eps": 1e-05,
22
+ "rope_scaling": null,
23
+ "rope_theta": 10000.0,
24
+ "tie_word_embeddings": false,
25
+ "torch_dtype": "bfloat16",
26
+ "transformers_version": "4.36.2",
27
+ "use_cache": true,
28
+ "vocab_size": 32000
29
+ }
mergekit_config.yml ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ models:
2
+ - model: NousResearch/Llama-2-70b-hf
3
+ # no parameters necessary for base model
4
+ - model: wanglab/ClinicalCamel-70B
5
+ parameters:
6
+ weight: 0.08
7
+ density: 0.45
8
+ - model: epfl-llm/meditron-70b
9
+ parameters:
10
+ weight: 0.08
11
+ density: 0.45
12
+ - model: allenai/tulu-2-dpo-70b
13
+ parameters:
14
+ weight: 0.08
15
+ density: 0.45
16
+ merge_method: dare_ties
17
+ base_model: NousResearch/Llama-2-70b-hf
18
+ parameters:
19
+ int8_mask: true
20
+ dtype: bfloat16
model-00001-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d3ced2089eea73ea0457686a8e2e82f3ecca97e1884752e8ac87199b6f13bd91
3
+ size 1950384952
model-00002-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c0e67a151aa27b13d7ed9ce2247afe7ab6568e62b72ccaa4d2fa4e6fdcf6231e
3
+ size 1996539384
model-00003-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b4ec37a51a402e653e9c38f2dd79508f1a9708b4de71b7a5ed98b0d6a987d1e
3
+ size 1711309872
model-00004-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:600b705448bda34e08cf31038328d9b752e4bf45e75c103e3c41e424b1f1f23d
3
+ size 1711309872
model-00005-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:115d5f0a7e3b80bacd7de00bc7fa8d677758ee93b6350c457c61a5e831529774
3
+ size 1711293376
model-00006-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:15a4c1f6dde186a7b9db33f791642dc501262710891627549beaf6171679db7e
3
+ size 1711293360
model-00007-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:13fa636e45935157da9edf60648df439ccb27d619b4b9926eedf0c152927415a
3
+ size 1543538064
model-00008-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4eff7d593cd0076c238c2248a0e0456782789b8a9d3b69bfdbed8ac46a162610
3
+ size 1895875512
model-00009-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:12aa3faf0b3a5e9ff51f2e8192292d994094ca1972b8d0037f7cc51e8a6c5ee7
3
+ size 1996539384
model-00010-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:60f3a4574b8f09fdf8299f08a0a2ed6ef2ffabfdbd5c62f7e3137ef22b447037
3
+ size 1711309872
model-00011-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fc3a2f6254e2fd7911eee3f9b77fdcd622d602e5314cda185504e645dc6cfb8f
3
+ size 1711309864
model-00012-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f11cce4ee6b359a0b79c1bdd07b28b22b30b95802187dc63892fe1c6257b4f5
3
+ size 1879081688
model-00013-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e7f95538aae37bffd80a108d314ed86ef18fab06d34f4673b1dafe07e7b59f3
3
+ size 1711309880
model-00014-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bdb918ff8b2c8bae24a316f29891cb4c88103926e47e4db50d0f76f5467bcbc9
3
+ size 1711309880
model-00015-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ac0f5c76ddbd3ebe222d5088b04918cbc435af4d09e8bb1aea8ffae12ab35acc
3
+ size 1711309880
model-00016-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2741dd6a40de61f322fe9c9742c66a39746f57d37a2269d43562d1ec68d4451f
3
+ size 1711309880
model-00017-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c7f08ba6131483cf988f29140a1eab5188f91f98fc073eef7ae417b07234439c
3
+ size 1711309880
model-00018-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:90bbe18a95c6628087747506c313364e32280de929315fc869e6a932719ff154
3
+ size 1711309880
model-00019-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:210ca719e8bcd3e27c408d8f7c439d9411ab3c97b0923dc88b8fbb72cfc1107e
3
+ size 1711309880
model-00020-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e486c4972dee90a104ef85d0128ba41601b5a87be1ba3e58bd10458b5812ea3
3
+ size 1711309880
model-00021-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:41c6d30e72ac006e0d0483a78a9edfa7c05b7cf91352c9dff4121e6e9ffbe7e9
3
+ size 1711309880
model-00022-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b8b54ded2ed324ce2c7eaf2e8bb3cdbb4e6ac07ca7a24e6fa769d13b75fe101
3
+ size 1996506408
model-00023-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:308b1779919d280d3c52ef642048ee23ea4fbb38fb268c2c17b3e1cb5167007c
3
+ size 1895875496
model-00024-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1f10b280aa7e8ec45236220af3faaea7be29beb6c026394263af0b10e686fada
3
+ size 1711309872
model-00025-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b8a9b7cdd1cde71ccb43f39a2bc1df1b72314a8ded1183dafe320eaa471c0ecf
3
+ size 1711309872
model-00026-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7501a651c30efe81b14d61de66ed84dbfb785d70d75255b308b693311ce7814f
3
+ size 1711309872
model-00027-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:680e9c40673bda686944121c3ad867fd9b58aa8dbebcc2eb3f0e7a9ca89b4314
3
+ size 1711309872
model-00028-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c66b2e927f4ee718b43ccd69b62dac2c6b0454d5fe68bed885793cc814eb8537
3
+ size 1543538064
model-00029-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3e589f832e16f4f14ecb29eb6537dfb9130ebbd0fa96374bf9667b9f20d643fd
3
+ size 1879081680
model-00030-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:466e1500fff02089b73fe679f1274d49c77e9fabfe4ba657a52052d24e34807b
3
+ size 1711309872
model-00031-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d33872aaeb8abd4ffa149e208a0a43fd3a503b43b44ec55ecce498b00ee377c7
3
+ size 1711309872
model-00032-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:31c1dd88600d2633e6e8d278fc15885237e77282effc7d620a8738a25094f91e
3
+ size 1711309872
model-00033-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bc6d1e1e734ca38916b9fdc039f949a000480aa63e7518212183b089c8f36316
3
+ size 1711309872
model-00034-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9a4a75cd5afb5ff81b672096536fd9a49a7ff631ff612554c333362ae58b22e1
3
+ size 1543538064
model-00035-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ace076638c43b79ddc3be4ff33c4355dead2af76bbc22931affcca9720d8d089
3
+ size 1879081680
model-00036-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fa0c2f931bfc35d31ca75ad72d2db3eaefdaa4ceae5457abfe94641095c5dd90
3
+ size 1711309872
model-00037-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:877994977bbd4f5ed10f6b0cdac53179f6f34611921023e2f793876ee0ac25f6
3
+ size 1711309872
model-00038-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:535b00127982e2080315ba7d06dc383257842596c47b705e14d03399a060bec4
3
+ size 1711309872
model-00039-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6ac898d4c16fbcda41e787f65d2e156ecf12ff4efcfe4566cfa3f733461cbf61
3
+ size 1711309872
model-00040-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:73d2596c44f852e4e49a6a7a805d4c2c42a134850eb1b9cdccd16dcb7f014eeb
3
+ size 1711309872
model-00041-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:497f130334c4e6a5f6f5f0e041842439f1268dfba5eb48105e3e929b9bdfdd41
3
+ size 1711309872
model-00042-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c45b3e779607f2cea6d9f2368838b617b4df9281c9eaa24f2a770729d0bdf205
3
+ size 1711309872
model-00043-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9aa8425018f7efc17b7d2961d9b535fc1928a87360485514299094ffad0c032e
3
+ size 1711309872
model-00044-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f0eef686a0798e9a93e5245e6dc80d5590b56663c68396f3f1cc7f448562c96c
3
+ size 1711309872
model-00045-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:534a46bf0aabcfee8708770254f66247131bbecc66ebe92b4d624ffcbb444706
3
+ size 1543538064
model-00046-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fb9b0039c856a677251fc0aae53f8e33a8a1e5ffed014fa40c8c398bb616d063
3
+ size 1879081680
model-00047-of-00080.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6746c63b555528e0e1d782f19f0e215909d560994f4a0ed539e84e5c992d5b92
3
+ size 1711309872