README.md

---
license: apache-2.0
datasets:
- anthracite-org/kalo-opus-instruct-22k-no-refusal
- Nopm/Opus_WritingStruct
- Gryphe/Sonnet3.5-SlimOrcaDedupCleaned
- Gryphe/Sonnet3.5-Charcard-Roleplay
- Gryphe/ChatGPT-4o-Writing-Prompts
- Epiculous/Synthstruct-Gens-v1.1-Filtered-n-Cleaned
- Epiculous/SynthRP-Gens-v1.1-Filtered-n-Cleaned
- nothingiisreal/Reddit-Dirty-And-WritingPrompts
- allura-org/Celeste-1.x-data-mixture
- allura-org/shortstories_synthlabels
base_model: EVA-UNIT-01/EVA-Qwen2.5-7B-v0.1
tags:
- llama-cpp
- gguf-my-repo
---

# Triangle104/EVA-Qwen2.5-7B-v0.1-Q5_K_M-GGUF
This model was converted to GGUF format from [`EVA-UNIT-01/EVA-Qwen2.5-7B-v0.1`](https://huggingface.co/EVA-UNIT-01/EVA-Qwen2.5-7B-v0.1) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space.
Refer to the [original model card](https://huggingface.co/EVA-UNIT-01/EVA-Qwen2.5-7B-v0.1) for more details on the model.

---
Model details:
-
A RP/storywriting
specialist model, full-parameter finetune of Qwen2.5-7B on mixture of
synthetic and natural data.

It uses Celeste 70B
0.1 data mixture, greatly expanding it to improve 



versatility,
creativity and "flavor" of the resulting model.








Version 0.1 notes:


Dataset was deduped
and cleaned from 



version 0.0, and
learning rate was adjusted. Resulting model seems to be


stabler, and 0.0
problems with handling short inputs and min_p sampling


seem to be mostly
gone.








Will be retrained
once more, because this run crashed around e1.2 (out


of 3) (thanks,
DeepSpeed, really appreciate it), and it's still 



somewhat
undertrained as a result.








Prompt format is
ChatML.








Recommended sampler
values:








Temperature: 0.87


Top-P: 0.81


Repetition Penalty:
1.03








Model appears to
prefer lower temperatures (at least 0.9 and lower). Min-P seems to
work now, as well.








Recommended
SillyTavern presets (via CalamitousFelicitousness):








Context








Instruct and System
Prompt








Training data:








Celeste 70B 0.1 data
mixture minus Opus Instruct subset. See that model's card for
details.


Kalomaze's
Opus_Instruct_25k dataset, filtered for refusals.


A subset (1k rows)
of ChatGPT-4o-WritingPrompts by Gryphe


A subset (2k rows)
of Sonnet3.5-Charcards-Roleplay by Gryphe


A cleaned subset
(~3k rows) of shortstories_synthlabels by Auri


Synthstruct and
SynthRP datasets by Epiculous








Training time and
hardware:








2 days on 4x3090Ti
(locally)








Model was trained by
Kearm and Auri.


Special thanks:


to Gryphe, Lemmy,
Kalomaze, Nopm and Epiculous for the data


to Alpindale for
helping with FFT config for Qwen2.5


and to
InfermaticAI's community for their continued support for our
endeavors








---
## Use with llama.cpp
Install llama.cpp through brew (works on Mac and Linux)

```bash
brew install llama.cpp

```
Invoke the llama.cpp server or the CLI.

### CLI:
```bash
llama-cli --hf-repo Triangle104/EVA-Qwen2.5-7B-v0.1-Q5_K_M-GGUF --hf-file eva-qwen2.5-7b-v0.1-q5_k_m.gguf -p "The meaning to life and the universe is"
```

### Server:
```bash
llama-server --hf-repo Triangle104/EVA-Qwen2.5-7B-v0.1-Q5_K_M-GGUF --hf-file eva-qwen2.5-7b-v0.1-q5_k_m.gguf -c 2048
```

Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.
```
git clone https://github.com/ggerganov/llama.cpp
```

Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).
```
cd llama.cpp && LLAMA_CURL=1 make
```

Step 3: Run inference through the main binary.
```
./llama-cli --hf-repo Triangle104/EVA-Qwen2.5-7B-v0.1-Q5_K_M-GGUF --hf-file eva-qwen2.5-7b-v0.1-q5_k_m.gguf -p "The meaning to life and the universe is"
```
or 
```
./llama-server --hf-repo Triangle104/EVA-Qwen2.5-7B-v0.1-Q5_K_M-GGUF --hf-file eva-qwen2.5-7b-v0.1-q5_k_m.gguf -c 2048
```