Instructions to use Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M

Use Docker

docker model run hf.co/Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M

LM Studio
Jan

vLLM

How to use Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M

Ollama
How to use Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1 with Ollama:
```
ollama run hf.co/Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1 with Docker Model Runner:
```
docker model run hf.co/Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M
```

Lemonade

How to use Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1:Q4_K_M

Run and chat with the model

lemonade run user.gemma-3-4b-it-roleplay-tuned-v1-Q4_K_M

List all available models

lemonade list

Atomic Chat

What is it?

This model is my attempt at further improving the results of LFM2.5 1.2b instruct finetune, since 1.2b is very little, I decided to work on a heavier, but still somewhat tiny architecture. So why not qwen 3.5 4b? I honestly tested, hell, even made a similar finetune on it, but roleplay wise, gemma 3 felt better to me.

But generally, this is me trying to create a portable model that anybody can run, I am doing my best to make those smaller models do roleplay.

A word on how good is it:

Advantages:

Better writing
less slop
small size
improved punctuation

Disadvantages:

Logic is massively improved over LFM 2.5 1.2b tune, but is not ideal
Due to being a rather small model, it will tend to sometimes repeat itself, altough I did my best to prevent this from happening

A word on the quants for this model:

BF16- Mostly overkill, however, highest quality
Q8_0- Amazing quality, near lossless
Q6_K- High quality, fast
Q5_K_M- Mid to high quality, small and fast
Q4_K_M- Mid quality, very small and very fast

Quants are located in the repo, along with safetensors

V2:

v2 is on the way, I am training it on way larger dataset(17M unique tokens of roleplay data). It will improve current results, and I will make sure it will do so without overfitting. I will continue making small roleplay tunes in order to make local roleplay possible for anyone, no matter the hardware.

Downloads last month: 367

Safetensors

Model size

4B params

Tensor type

BF16

Model tree for Indexnusrefather/gemma-3-4b-it-roleplay-tuned-v1

Base model

google/gemma-3-4b-pt

Finetuned

google/gemma-3-4b-it

Adapter

(478)

this model

Adapters

2 models

Indexnusrefather
/

gemma-3-4b-it-roleplay-tuned-v1