Instructions to use IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0
# Run inference directly in the terminal:
llama cli -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0
# Run inference directly in the terminal:
llama cli -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0
# Run inference directly in the terminal:
./llama-cli -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

Use Docker

docker model run hf.co/IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

LM Studio
Jan

vLLM

How to use IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

Ollama
How to use IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF with Ollama:
```
ollama run hf.co/IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0
```
Unsloth Desktop

How to use IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF with Docker Model Runner:
```
docker model run hf.co/IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0
```

Lemonade

How to use IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

Run and chat with the model

lemonade run user.Occamy-1.0-APEX-I-MiniPlus-V2-GGUF-Q8_0

List all available models

lemonade list

Hermes Agent

How to use IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "IsValorum/Occamy-1.0-APEX-I-MiniPlus-V2-GGUF:Q8_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

This size is not suitable for adding mmproj.

by jian2023 - opened 1 day ago

Discussion

jian2023

1 day ago

Bundling it into mmproj is pointless. If you have enough VRAM, why not use a higher-precision quantized version? The greatest value lies in removing MTP and mmproj so that the model can run smoothly on 16GB of VRAM, since the closest matching graphics card specification is 16GB.

IsValorum

Owner 1 day ago

•

edited 1 day ago

Hugging Face CLI

A couple of practical points to clarify here:

mmproj is a separate, optional file. In llama.cpp, vision isn't baked into the main GGUF. If you don't pass --mmproj, it uses 0 MB of VRAM. If you have a 16GB card and only care about text/code, just load the main model and you get plenty of room for context. Removing the projector from the repo makes no sense because it would just break vision for anyone who wants it.
Occamy doesn't have MTP. It's a 35B hybrid model with 30 linear attention layers and native vision; Iris and Apodex are the ones with MTP heads.
Why this over Q4 on 24GB VRAM:
A standard Q4_K_M weighs ~19.5 GB. On a 24GB card (RTX 3090/4090), that leaves barely ~4 GB for KV cache and CUDA buffers. Try running a 64k or 256k context on that and you'll OOM immediately.

The entire purpose of this build is keeping the weights at 13.6 GB without the quality collapse of flat automated 3-bit quants. The router gates (ffn_gate_inp) are kept in uncompressed F32 so the 256 micro-experts don't drift, output.weight is isolated in Q6_K so vocabulary classification doesn't break syntax, and the shared expert uses IQ4_NL. It matches the reasoning and output quality of a much heavier Q4, but leaves enough VRAM headroom to actually run large contexts.

Also, don't confuse APEX-I-MiniPlus-V2 with a generic baseline APEX-I-Mini. Traditional APEX-I-Mini drops core experts aggressively to 2-bit IQ2_S and leaves output.weight at 3-bit Q3_K_M, which creates a noticeable perplexity hit on complex reasoning tasks. V2 was specifically re-engineered to avoid that quality floor (keeping core experts at calibrated IQ3_XXS, output in Q6_K, and routers in F32).

To put the numbers in perspective: this cuts nearly 2 GB off a flat 3-bit quant (~~15.6 GB), and weighs only about ~1 GB more than a generic APEX-I-Mini (~~12.5 GB). For that single extra gigabyte of VRAM, you get a massive jump in reasoning and syntactic stability.

Take a look at the tensor-by-tensor comparison table in the model card above if you want to check out the exact architectural differences and why this specific allocation is optimal.

That's specifically what this was built for.

jian2023

1 day ago

Thank you for your help. It loaded completely and performed well on my 4060ti 16G graphics card, currently only using 64K of context.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment