Instructions to use cbert33/Ornith-1.5-9B-FP8-Heretic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use cbert33/Ornith-1.5-9B-FP8-Heretic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="cbert33/Ornith-1.5-9B-FP8-Heretic")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("cbert33/Ornith-1.5-9B-FP8-Heretic")
model = AutoModelForMultimodalLM.from_pretrained("cbert33/Ornith-1.5-9B-FP8-Heretic", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use cbert33/Ornith-1.5-9B-FP8-Heretic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "cbert33/Ornith-1.5-9B-FP8-Heretic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cbert33/Ornith-1.5-9B-FP8-Heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/cbert33/Ornith-1.5-9B-FP8-Heretic

SGLang

How to use cbert33/Ornith-1.5-9B-FP8-Heretic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "cbert33/Ornith-1.5-9B-FP8-Heretic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cbert33/Ornith-1.5-9B-FP8-Heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "cbert33/Ornith-1.5-9B-FP8-Heretic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "cbert33/Ornith-1.5-9B-FP8-Heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use cbert33/Ornith-1.5-9B-FP8-Heretic with Docker Model Runner:
```
docker model run hf.co/cbert33/Ornith-1.5-9B-FP8-Heretic
```

Ornith 1.5 9B FP8 Heretic

Ornith-1.5-9B-FP8-Heretic is a standalone, merged, compressed-tensors FP8 derivative of Sohailhosseini/Ornith-1.5-9B-FP8, modified with Heretic.

This repository contains the complete model. It is not a LoRA adapter: no separate PEFT adapter and no local copy of the base model are required at inference time.

What changed

The release pipeline was:

Run a bounded Heretic optimization against the pinned FP8 base checkpoint.
Export the selected rank-1 LoRA direction.
Merge the adapter into a BF16 intermediate.
Re-quantize the merged model to the source checkpoint's compressed-tensors FP8 scheme.
Verify the standalone export tensor-by-tensor and reload it without PEFT or the base-model directory.

Abliteration is intended to reduce learned refusal behavior. It does not make the model universally correct, safe, unbiased, or suitable for every deployment.

Base checkpoint and provenance

Field	Value
Base repository	`Sohailhosseini/Ornith-1.5-9B-FP8`
Base revision	`d30f70d5d62c0b9d5a1d7f015b37513fc7aa0bdc`
Heretic version	`2.0.0.dev0`
Optimization trials	`1`
Seed	`12345`
Row normalization	`pre`
Selected direction index	`15.250741580994966`
Trial KL divergence	`0.0304863341152668`
LoRA rank before merge	`1`

The adapter-only and native merged runs selected identical parameters and scores.

Quantization

The derivative preserves the source checkpoint's compressed-tensors quantization boundary and metadata:

Format: compressed-tensors float-quantized
Quantization method: FP8
Weights: FP8 E4M3, static per-channel
Input activations: FP8, dynamic per-token
Quantization target: Linear
KV-cache quantization: none
Protected/ignored module boundary: identical to the pinned source checkpoint

Serialized tensor inventory

Stored dtype	Tensors	Elements
FP8 E4M3	200	6,912,212,992
BF16 protected tensors	760	2,499,026,160
Total	960	9,411,239,152

model.safetensors is 11,910,387,016 bytes, the same serialized tensor-file size as the pinned source FP8 model.

Bounded derivative verification

A tensor-by-tensor comparison against the pinned source checkpoint found:

Check	Result
Tensor key sets	Identical
Matching shapes and stored dtypes	960 / 960
Bit-identical unchanged tensors	876
Changed model weights	42
Changed FP8 scales paired with those weights	42
Other changed tensors	0

Every changed model weight is within a Heretic target:

mlp.down_proj
self_attn.o_proj
linear_attn.out_proj

Protected BF16 islands and unrelated quantized tensors remain byte-identical to the source. This bounds the release as a narrow derivative of the pinned FP8 checkpoint rather than an uncontrolled whole-model conversion.

Runtime validation

The final repository tree was loaded directly with AutoModelForImageTextToText.from_pretrained, without PEFT and without access to the base-model directory.

Verified properties:

Model class: Qwen3_5ForConditionalGeneration
Quantization status: compressed
Remaining LoRA parameters: 0
Deterministic CUDA generation completed successfully
Prompt: The capital of France is
Generated continuation: Paris

Coherence and tool-calling smoke test

A small deterministic, greedy-decoding smoke suite passed 6/6 cases:

Category	Passed
Coherence and instruction following	2 / 2
Tool selection and argument construction	3 / 3
Simulated tool-result round trip	1 / 1

Observed native tool-call outputs included:

get_weather(city="Chicago", unit="fahrenheit")
calculator(expression="17 * 23")
Dependency-aware first step: geocode(place="Eiffel Tower") before requesting weather

After receiving a simulated weather response, the model incorporated its temperature, condition, and observation time into a natural-language answer without issuing another tool call.

This is a smoke test, not a comprehensive agent benchmark. Multi-step real-tool execution, parallel calls, malformed tool results, tool restraint, long-context behavior, multimodal quality, factuality, and broad safety behavior remain unevaluated.

Usage

Use recent releases of Transformers, Accelerate, and compressed-tensors with Qwen3.5 support.

Load the model

from transformers import AutoModelForImageTextToText, AutoProcessor

repo_id = "cbert33/Ornith-1.5-9B-FP8-Heretic"

model = AutoModelForImageTextToText.from_pretrained(
    repo_id,
    dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(repo_id)

Text generation

from transformers import AutoTokenizer

repo_id = "cbert33/Ornith-1.5-9B-FP8-Heretic"
tokenizer = AutoTokenizer.from_pretrained(repo_id)

messages = [
    {"role": "user", "content": "Explain why the sky appears blue in two sentences."}
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=False,
)
print(tokenizer.decode(outputs[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))

Native tool-call prompting

The included chat template accepts OpenAI-style function definitions through its tools argument and emits Qwen's native XML tool-call representation.

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a city.",
            "parameters": {
                "type": "object",
                "properties": {
                    "city": {"type": "string"},
                    "unit": {
                        "type": "string",
                        "enum": ["celsius", "fahrenheit"],
                    },
                },
                "required": ["city", "unit"],
            },
        },
    }
]

messages = [
    {
        "role": "user",
        "content": "What is the current weather in Chicago in Fahrenheit?",
    }
]

prompt = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    tokenize=False,
    add_generation_prompt=True,
    enable_thinking=False,
)

Your inference framework is responsible for parsing the generated tool call, executing the function, and returning the tool result to the model.

Tested environment

Windows 11 with WSL2
NVIDIA GeForce RTX 5080 (16 GB)
CUDA 13.0
PyTorch 2.13.0+cu130
Transformers 5.6.2
compressed-tensors 0.18.0
llmcompressor 0.13.0

Memory note

In the tested direct Transformers path, compressed-tensors materialized/decompressed weights during loading and peak CUDA allocation reached approximately 19.0 GB. The model completed inference on the 16 GB RTX 5080 through WSL unified memory, but generation was slow. An FP8-aware serving runtime should be qualified separately before production deployment.

Limitations

The Heretic search was deliberately bounded: two positive prompts, two negative prompts, two KL prompts, and one optimization trial.
The model has not received a comprehensive text, vision, benchmark, safety, or refusal evaluation after modification.
The small coherence/tool test demonstrates only the explicitly tested cases.
Abliteration may increase compliance with harmful, incorrect, or poorly specified requests.
Quantization and runtime support vary by GPU architecture and inference backend.
Users are responsible for evaluating legality, safety, reliability, and fitness for their intended use.

Integrity

The repository includes SHA256SUMS. The model tensor hash is:

8650b2f6cf35afb133c7225c65b25483211ccbf72901fe9cdd7803c64c32c0af  model.safetensors

License

The source checkpoint is published under the MIT license. This derivative is distributed under the same license. Users must also comply with the licenses and terms applicable to upstream model components and dependencies.

Downloads last month: -

Safetensors

Model size

9B params

Tensor type

BF16

F8_E4M3

Model tree for cbert33/Ornith-1.5-9B-FP8-Heretic

Base model

ornith-ai/Ornith-1.5-9B

Quantized

Sohailhosseini/Ornith-1.5-9B-FP8

Finetuned

(1)

this model