Instructions to use PocketDoc/Dans-PersonalityEngine-V1.3.0-12b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use PocketDoc/Dans-PersonalityEngine-V1.3.0-12b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="PocketDoc/Dans-PersonalityEngine-V1.3.0-12b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("PocketDoc/Dans-PersonalityEngine-V1.3.0-12b")
model = AutoModelForCausalLM.from_pretrained("PocketDoc/Dans-PersonalityEngine-V1.3.0-12b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use PocketDoc/Dans-PersonalityEngine-V1.3.0-12b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/PocketDoc/Dans-PersonalityEngine-V1.3.0-12b

SGLang

How to use PocketDoc/Dans-PersonalityEngine-V1.3.0-12b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PocketDoc/Dans-PersonalityEngine-V1.3.0-12b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use PocketDoc/Dans-PersonalityEngine-V1.3.0-12b with Docker Model Runner:
```
docker model run hf.co/PocketDoc/Dans-PersonalityEngine-V1.3.0-12b
```

The King of 12B

by iliasfromilios - opened Aug 9

Discussion

iliasfromilios

Aug 9

•

edited Aug 9

I can't praise this model highly enough. It puts 12B models to dust. I've tried every model I could think of, from the supposed prose champion in UGI Famino, to Irix, Krix, Nemo merges, mythomax, and whatnot. Everything I could download and try.

This blows everything out of the water. All I had to do was make to Lorebook that works around some of it's little quirks (https://chub.ai/lorebooks/DarkHorizon/a-narrative-compendium-20230809cc12). Now I'm in 20k context chats and it's just one shotting everything and looks and feels like Deepseek level prose and context awareness.

Very well done, seriously. There's nothing else out there I can think of that beats this model in both immersive roleplay and character card creation.

iliasfromilios

about 1 month ago

Just one thing worth noting. Presence Pentalty at -2 seems to work better.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment