Nitral-AI commited on
Commit
486a9a5
1 Parent(s): 530b774

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +40 -45
README.md CHANGED
@@ -1,45 +1,40 @@
1
- ---
2
- base_model: []
3
- library_name: transformers
4
- tags:
5
- - mergekit
6
- - merge
7
-
8
- ---
9
- # Domain-Fusion-L3-8B
10
-
11
- This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
12
-
13
- ## Merge Details
14
- ### Merge Method
15
-
16
- This model was merged using the SLERP merge method.
17
-
18
- ### Models Merged
19
-
20
- The following models were included in the merge:
21
- * ./Hathor_Variant-X
22
- * ./T-900xBioLLM
23
-
24
- ### Configuration
25
-
26
- The following YAML configuration was used to produce this model:
27
-
28
- ```yaml
29
- slices:
30
- - sources:
31
- - model: ./Hathor_Variant-X
32
- layer_range: [0, 32]
33
- - model: ./T-900xBioLLM
34
- layer_range: [0, 32]
35
- merge_method: slerp
36
- base_model: ./Hathor_Variant-X
37
- parameters:
38
- t:
39
- - filter: self_attn
40
- value: [0, 0.5, 0.3, 0.7, 1]
41
- - filter: mlp
42
- value: [1, 0.5, 0.7, 0.3, 0]
43
- - value: 0.5
44
- dtype: bfloat16
45
- ```
 
1
+ ---
2
+ base_model: []
3
+ library_name: transformers
4
+ tags:
5
+ - mergekit
6
+ - merge
7
+
8
+ ---
9
+ # Domain-Fusion-L3-8B
10
+
11
+ (Meme model in testing)
12
+
13
+ ### Models Merged
14
+
15
+ The following models were included in the merge:
16
+ * ./Hathor_Variant-X
17
+ * ./T-900xBioLLM
18
+
19
+ ### Configuration
20
+
21
+ The following YAML configuration was used to produce this model:
22
+
23
+ ```yaml
24
+ slices:
25
+ - sources:
26
+ - model: ./Hathor_Variant-X
27
+ layer_range: [0, 32]
28
+ - model: ./T-900xBioLLM
29
+ layer_range: [0, 32]
30
+ merge_method: slerp
31
+ base_model: ./Hathor_Variant-X
32
+ parameters:
33
+ t:
34
+ - filter: self_attn
35
+ value: [0, 0.5, 0.3, 0.7, 1]
36
+ - filter: mlp
37
+ value: [1, 0.5, 0.7, 0.3, 0]
38
+ - value: 0.5
39
+ dtype: bfloat16
40
+ ```