Instructions to use DarkArtsForge/Magistaroth-24B-v1.1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use DarkArtsForge/Magistaroth-24B-v1.1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="DarkArtsForge/Magistaroth-24B-v1.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("DarkArtsForge/Magistaroth-24B-v1.1")
model = AutoModelForMultimodalLM.from_pretrained("DarkArtsForge/Magistaroth-24B-v1.1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use DarkArtsForge/Magistaroth-24B-v1.1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "DarkArtsForge/Magistaroth-24B-v1.1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DarkArtsForge/Magistaroth-24B-v1.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/DarkArtsForge/Magistaroth-24B-v1.1

SGLang

How to use DarkArtsForge/Magistaroth-24B-v1.1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "DarkArtsForge/Magistaroth-24B-v1.1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DarkArtsForge/Magistaroth-24B-v1.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "DarkArtsForge/Magistaroth-24B-v1.1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "DarkArtsForge/Magistaroth-24B-v1.1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use DarkArtsForge/Magistaroth-24B-v1.1 with Docker Model Runner:
```
docker model run hf.co/DarkArtsForge/Magistaroth-24B-v1.1
```

Comparison

by redaihf - opened Apr 21

Discussion

redaihf

Apr 21

•

edited Apr 21

The 1.1 version is unfortunately inferior to its predecessor. While it lacks refusals its ability to follow prompts is poor and it also exhibits some early terminations. It is possible that these problems result from either covert noncompliance or merge issues.

Naphula

DarkArtsForge org Apr 22

Likely this is due to the PDQ/QHE formula which already operates at a 10% threshold (any further breaks the model entirely).

Naphula

DarkArtsForge org Apr 23

•

edited Apr 23

New versions (v1.2) of Magistaroth and Slimaki are being released, each uses the della method. Both should be fully uncensored out the box no ablations or jailbreaks needed. No normalize: false components were used for these afaik so there should only be 200% norm inflation at most. C72-C78 were either censored or less intelligent than C79 and C80. C81 was the smartest of them all but had some refusals.

The updated Magistaroth v1.2 may provide a good middle ground for instruction following + lack of censorship. Instead of using all unablated components + ablating after the merge, I tested various configs to see where the "refusal threshold" was for swapping out individual donors with ablated versions. It appears that only PF v4.1 and Tiamat required ablated versions (heretic paper witch), Della's magprune with normalize: false and total weights at 2.0 handled the rest. This achieves full uncensorship while at least some components (Magidonia, Precog, PF v3) remain fully unablated. More testing is needed to see if this is better than the standard merge → ablate pipeline.

There's also Mergedonia-Karcher being uploaded, this has some refusals but seems quite smart.

The new prometheus method includes some features like QHE's novelty-injection, but version M was kind of lobotomized so I'm hoping version P is higher quality. A quick test variant is merging now at 100 iterations for Mergedonia (Magidonia, Cydonia, Precog, Rivermind, Fallen Mistral). If this works then maybe a prometheus version of Magistaroth or Slimaki can be released too.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment