athirdpath's picture
Update README.md
b76c439
---
license: cc-by-nc-4.0
---
### Description
After I put down the joint and [RTFM](https://arxiv.org/pdf/2311.03099.pdf), I have a better idea exactly what's going on. I considered doing something similar with WANDA or SparseGPT a while back, but stopped when I ran into issues. Thus, I'm fascinated by this new method's execution.
### Hypothesis
By lowering the density, I hit closer to the sweet-spot shown in the paper. Also, I'm using my fixed base model, so hopefully that helps too. Weights are adjusted to make the later layers more aligned with ORCA 2.
### Results
I'm quite happy with this model for what it is, a personable and effective assistant. It does infodump a bit, but what genius doesn't? It writes okay erotica and general fiction, it just has an "artifical" tone.
### Recipe
merge_method: dare_ties
- base_model: athirdpath/BigLlama-20b
- model: athirdpath/CleverGirl-20b
weight: 0.60 / density: 0.35
- model: athirdpath/CleverGirl-20b-Inverted
weight: 0.40 / density: 0.30
int8_mask: true
dtype: bfloat16