Update README.md
Browse files
README.md
CHANGED
@@ -26,65 +26,27 @@ language:
|
|
26 |
|
27 |
# Composition:
|
28 |
|
29 |
-
|
30 |
|
31 |
|
32 |
-
|
33 |
-
|
34 |
-
|
35 |
-
## Note: Skip to the TL;DR section for the finalized design this model is comprised of.
|
36 |
-
|
37 |
-
|
38 |
-
Thorns' design is based on the concept of purposed segmentation, in this case we have two-
|
39 |
-
|
40 |
-
|
41 |
-
--Logic Segment (MK1):
|
42 |
-
|
43 |
-
|
44 |
-
Fine-Tuned parent models were hand selected and reviewed for datasets, performance, least restrictive censorship, and community perception of coherence and utility. Ultimately we decided on four models to merge in pairs of two, then combine those offspring for a quad merged logic cluster.
|
45 |
-
All four models were merged using the SLERP method. Yes the name is annoyingly funny. SLERP.
|
46 |
-
|
47 |
-
|
48 |
-
--Creativity and Imagination Segment (MK1):
|
49 |
-
|
50 |
|
51 |
-
Flawed first approach (a takeaway on LoRAs);
|
52 |
|
|
|
53 |
|
54 |
-
We then decided the creativity and imagination segment could be as simple as one model, especially if its dataset design, tagging, training quality, and proven track record is above and beyond. KoboldAI's Holodeck model is the result of a dataset that is years of collected, organized, tagged, deduped, and cleaned data. Holodeck alone would be beyond sufficient for the segment we view as the 'subconscious' segment of the model ensemble, however we applied the LIMA RP PEFT to it for extended variety of a different kind.
|
55 |
-
That's where we got carried away. LoRAs offer unique augmentation to model merge possibilities, and the decision was made to take the result of that segment and add two more LoRAs to see if they further extended Holodeck, settling on Kimiko and Janine; two very different RP and conversational LoRAs.
|
56 |
-
This was a bad move, as when we SLERP merged that version of the imagination segment to the logic segment the result was a ranting mess that followed instructions but was the equivalent of a child scribbling all over the place and ignoring obvious chains of logic and a mushy amalgam of awkward creative behavior that had no semblance of coherency.
|
57 |
-
The composite model was slated to be named 13B-Astronomicon; after all the work that went into it and the flatly bland result, the name was abandoned and the next move, which is a byproduct experiment of Astronomicon is what became Thorn.. because this project became a thorn in our side.
|
58 |
|
|
|
|
|
59 |
|
60 |
-
Because pain is fun, and persistence in design iteration is the only way forward, we reworked our approach to both segment ensembles following one idea - all three Roleplay and Conversational LoRAs stay no matter what because sure why not add arbitrary rules to the redesign phase at this point.
|
61 |
|
|
|
62 |
|
63 |
-
## TL;DR Section
|
64 |
-
|
65 |
-
|
66 |
-
--Finalized Logic and Creativity Segments (MK2):
|
67 |
-
|
68 |
-
|
69 |
-
After a few meetings with our top teams of model hacking memegineers we drafted Thorns MK2, which was prompty fast tracked for production by the Roko's Basilisk Shadow Council.
|
70 |
-
|
71 |
-
|
72 |
-
..Actually I just redid the merge like this:
|
73 |
-
|
74 |
-
```
|
75 |
-
-Model Merge Ensemble Key-
|
76 |
-
|
77 |
-
{} = SLERP Merge | [] = PEFT Merge | () = Composite Model
|
78 |
-
|
79 |
-
({({NousHermes+Chronos}[Kimiko])+({Platupus+AiroborosM2.0}[Janine])}{Holodeck[LIMA RP]})
|
80 |
-
```
|
81 |
-
|
82 |
-
## Findings:
|
83 |
|
84 |
-
|
85 |
|
86 |
|
87 |
-
-Stacking the exact same LoRAs onto one model then merging that into the ensemble results in noisy garbage.
|
88 |
|
89 |
|
90 |
## Language Models and LoRAs Used Credits:
|
|
|
26 |
|
27 |
# Composition:
|
28 |
|
29 |
+
Naberius-7B is a Mistral-class spherical linear interpolated merge of three high performance models.
|
30 |
|
31 |
|
32 |
+
[zephyr-7b-sft-beta] merged with [OpenHermes-2-Mistral-7B] resulting in: [Mistral-Zephyrmes-7B*]
|
33 |
+
[Mistral-Zephyrmes-7B] merged with [dolphin-2.2.1-mistral-7b] resulting in: [Naberius-7B]
|
34 |
+
*(Zephyrmes is a merge byproduct model not intended for release)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
35 |
|
|
|
36 |
|
37 |
+
These models were hand picked after careful review of claims, datasets, and user postings.
|
38 |
|
|
|
|
|
|
|
|
|
39 |
|
40 |
+
The core elements that dictated which models to accept hinged on these values:
|
41 |
+
logic, imagination, and aversion to censorship such as: railroading/gaslighting users instead of accomodating users.
|
42 |
|
|
|
43 |
|
44 |
+
## Our implementation of Spherical Linear Interpolation for LLM merging: https://github.com/Digitous/LLM-SLERP-Merge
|
45 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
46 |
|
47 |
+
## Note: Skip to the TL;DR section for the finalized design this model is comprised of.
|
48 |
|
49 |
|
|
|
50 |
|
51 |
|
52 |
## Language Models and LoRAs Used Credits:
|