Instructions to use Guilherme34/Firefly-v4-BetterGGUF-iMatrix with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Guilherme34/Firefly-v4-BetterGGUF-iMatrix with Transformers:

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Guilherme34/Firefly-v4-BetterGGUF-iMatrix", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use Guilherme34/Firefly-v4-BetterGGUF-iMatrix with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M

Use Docker

docker model run hf.co/Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M

LM Studio
Jan
Ollama
How to use Guilherme34/Firefly-v4-BetterGGUF-iMatrix with Ollama:
```
ollama run hf.co/Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M
```

Unsloth Studio

How to use Guilherme34/Firefly-v4-BetterGGUF-iMatrix with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Guilherme34/Firefly-v4-BetterGGUF-iMatrix to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Guilherme34/Firefly-v4-BetterGGUF-iMatrix to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Guilherme34/Firefly-v4-BetterGGUF-iMatrix to start chatting

How to use Guilherme34/Firefly-v4-BetterGGUF-iMatrix with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use Guilherme34/Firefly-v4-BetterGGUF-iMatrix with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use Guilherme34/Firefly-v4-BetterGGUF-iMatrix with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use Guilherme34/Firefly-v4-BetterGGUF-iMatrix with Docker Model Runner:
```
docker model run hf.co/Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M
```

Lemonade

How to use Guilherme34/Firefly-v4-BetterGGUF-iMatrix with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Guilherme34/Firefly-v4-BetterGGUF-iMatrix:Q4_K_M

Run and chat with the model

lemonade run user.Firefly-v4-BetterGGUF-iMatrix-Q4_K_M

List all available models

lemonade list

Firefly-v4 i1 GGUF

Weighted imatrix GGUF quantizations of Firefly-v4, made for smaller local inference while preserving as much of the model's character as possible.

GGUF Imatrix Weighted Vision Model Audio Roleplay Local Inference

Format

GGUF

Quant Type

i1 / imatrix

Focus

Creative

Mode

Multimodal

About

This repository contains weighted / imatrix GGUF quantizations of Firefly-v4. Imatrix quantization uses calibration data to guide compression, making it especially useful for low-bit quants where every bit has to earn its rent.

These files are intended for local inference with GGUF-compatible software such as:

llama.cpp LM Studio KoboldCpp Jan Text Generation WebUI Ollama

Original model: https://huggingface.co/Guilherme34/Firefly-v4

Static GGUF repository: https://huggingface.co/mradermacher/Firefly-v4-GGUF

Recommended Start

Use Q4_K_M if you want the safest default for quality, speed, and memory.

Small Systems

Try IQ3_S, IQ3_M, or Q3_K_M for lower VRAM machines.

Efficient Sweet Spot

IQ4_XS is a strong compact option when you want good quality without going heavy.

Higher Quality

Use Q5_K_M or Q6_K when you can spare more memory.

Provided Files

Sorted by file size, not necessarily quality. IQ quants are often preferable over similarly sized non-IQ quants.

Download	Type	Size	Notes
GGUF	imatrix	0.1 GB	Imatrix calibration file for creating your own quants. Not an inference model.
GGUF	i1-Q2_K	3.1 GB	Very small. IQ3_XXS is probably better when available.
GGUF	i1-Q3_K_S	3.2 GB	Low memory. IQ3_XS is probably better when available.
GGUF	i1-IQ3_S	3.2 GB	Strong low-bit option. Often beats Q3_K variants.
GGUF	i1-IQ3_M	3.2 GB	Compact and balanced.
GGUF	i1-Q3_K_M	3.3 GB	Balanced Q3 option. IQ3_S is probably better.
GGUF	i1-Q3_K_L	3.4 GB	Higher Q3 quality. IQ3_M is probably better.
GGUF	i1-IQ4_XS	3.4 GB	Excellent efficiency pick.
GGUF	i1-IQ4_NL	3.5 GB	Good, but IQ4_XS is usually preferred.
GGUF	i1-Q4_0	3.5 GB	Fast legacy-style quant, lower quality than K-quants.
GGUF	i1-Q4_K_S	3.5 GB	Fast, efficient, and practical.
GGUF	i1-Q4_K_M	3.5 GB	Recommended for most users.
GGUF	i1-Q4_1	3.6 GB	Alternative Q4 format.
GGUF	i1-Q5_K_S	3.7 GB	Higher quality while staying fairly light.
GGUF	i1-Q5_K_M	3.7 GB	Great quality if you have the memory.
GGUF	i1-Q6_K	3.9 GB	Excellent quality. Very close to static Q6_K behavior.

Which Quant Should I Download?

Use Case	Recommended Quant
Best default	Q4_K_M
Fast and efficient	Q4_K_S
Compact but smart	IQ4_XS
Very low memory	IQ3_S or IQ3_M
Higher quality	Q5_K_M
Best quality in this repo	Q6_K
Making your own quants	Firefly-v4.imatrix.gguf

Usage

Download one of the .gguf files and load it in your preferred GGUF-compatible backend.

Example with llama.cpp:

llama-cli -m Firefly-v4.i1-Q4_K_M.gguf -p "Write a cinematic opening scene about a lost signal in deep space."

For multimodal usage, make sure your backend supports the model architecture and has the required projection files from the static GGUF repository when needed.

If you are new to GGUF files, you can also look at community GGUF README examples for setup notes, backend flags, and multi-part file handling.

Attribution

Original Model: Firefly-v4
Author: Guilherme34

Original model page: https://huggingface.co/Guilherme34/Firefly-v4

Quantization repository: https://huggingface.co/mradermacher/Firefly-v4-i1-GGUF

This repository contains GGUF quantizations. All credit for the model training, fine-tuning, datasets, and original work belongs to the original author.

License

This repository follows the Firefly-v4 Attribution License.

Commercial use, quantizations, merges, and derivative works are permitted under the terms described in the included LICENSE file.

Please preserve the required attribution when redistributing or building upon this model.

Disclaimer

This model is uncensored and may generate content without built-in refusals. It is intended for creative fiction and roleplay between consenting adults. The creator is not responsible for how the model is used. Do not use it to produce content that is illegal in your jurisdiction.