TULLUS/marin-8b-instruct-GGUF

Marin 8B Instruct — GGUF

This repository contains my Gold Standard BF16 GGUF conversion of marin-community/marin-8b-instruct, alongside the verified Q6_K and Q4_K_M quantizations derived from it.

TULLUS // Did not train or fine-tune the original Marin model. This repository distributes three GGUF artifacts for local inference with compatible llama.cpp builds. The results below document specific static inspections and local experiments; they are not claims of broad benchmark performance.

Provenance and conversion record

• Upstream base model: marin-community/marin-8b-base
• Recorded original source revision: 0378f9c932bd64ca189d2f2ca5bd2fb076cba61d
• Recorded llama.cpp conversion revision: 27c7a769da1671c97b2e3f6cb88704a6e1e0d0cd
• Runtime used for the recorded local experiments: llama.cpp 0.6.0-dev, build 11719, commit 5aeac37a35b44fde4bdffaec9159f2fcb253b622

Quantization notes

• BF16: the largest of these three artifacts, retaining BF16 precision where represented in the file.
• Q6_K: a lower-size llama.cpp K-quantization variant.
• Q4_K_M: a mixed-precision K-quantization variant. _M does not mean every tensor is stored in Q4_K; tensor storage types may vary.

File size alone does not establish comparative output quality, speed, or total memory use. Those depend on the runtime, hardware, context allocation, and workload. No perplexity study or broad comparative quantization benchmark is claimed here.

Architecture and configuration

The following are recorded source-model configuration values unless explicitly identified as GGUF metadata.

Property	Recorded value
Architecture	LlamaForCausalLM
Approximate parameter count	8 billion
Vocabulary size	128,256
Hidden size	4,096
Intermediate size	14,336
num_hidden_layers	32
Transformer layers	32
Attention heads	32
Key/value heads	8
Source max_position_embeddings	4,096
RoPE theta	500,000
Source RoPE style	Llama 3-style llama3
Source RoPE scaling factor	8
Low-frequency factor	1
High-frequency factor	4
original_max_position_embeddings	8,192
BOS token ID	128000
EOS token ID	128009
tie_word_embeddings	false

Recorded GGUF metadata includes:

llama.context_length = 4096
llama.rope.freq_base = 500000
llama.rope.dimension_count = 128

The recorded GGUF context metadata is 4,096 tokens. The source configuration's original_max_position_embeddings = 8192 is part of the source RoPE configuration; it does not by itself establish reliable native 8,192-token inference. Source configuration, stored GGUF metadata, static tensor contents, runtime overrides, and measured retrieval results are distinct types of evidence.

======================================================================
TULLUS — MARIN 8B BF16 TENSOR AUDIT
======================================================================

BF16 file: /tmp/marin-community/marin-8b-instruct/output/marin-8b-instruct-BF16.gguf
BF16 size: 14.965 GiB
Original indexed tensors: 291

GGUF tensor count: 292
GGUF metadata field count: 60

• Why are there 292 tensors instead of 291? The converter adds rope_freqs.weight, a tensor associated with Llama 3 RoPE scaling. This explains the additional tensor rather than indicating a missing or duplicated source weight. llama.cpp can represent Llama 3 scaling through these frequency values rather than the usual generic RoPE scaling metadata fields. The 66 F32 tensors are also consistent with the structure we expect: the layer-normalization weights, output normalization weights, and the additional RoPE frequency tensor. The principal model matrices are BF16.

Rope sanity check

--- BF16 ---
/tmp/marin-community/marin-8b-instruct/output/marin-8b-instruct-BF16.gguf
PASS  BF16 file exists
PASS  BF16: llama.context_length matches source
      GGUF=4096; expected=4096
PASS  BF16: llama.rope.freq_base matches source
      GGUF=500000; expected=500000
PASS  BF16: llama.rope.dimension_count matches source
      GGUF=128; expected=128
PASS  BF16: exactly one rope_freqs.weight
      Found 1
Storage type: F32
Tensor shape: (np.uint64(64),)
PASS  BF16: RoPE storage is F32
PASS  BF16: RoPE contains 64 values
PASS  BF16: stored factors match conversion formula
      Maximum absolute error: 0

THATS THE ROPE RESULT NEEDED, ALL PASS WITH ZERO NUMERICAL ERRORS!

The decisive lines:

BF16:   Maximum absolute error: 0
Q6_K:   Maximum absolute error: 0
Q4_K_M: Maximum absolute error: 0

And the cross-quantization checks:

PASS  BF16 vs Q6_K: RoPE values identical
PASS  BF16 vs Q4_K_M: RoPE values identical

What was established

1. The RoPE conversion is mathematically correct. All three GGUFs contain the exact 64 frequency-scaling factors calculated from the original configuration using the pinned llama.cpp conversion formula.

2. Quantization preserved the RoPE tensor. BF16, Q6_K, and Q4_K_M contain identical RoPE values. Neither quantization changed these factors.

3. The essential metadata agrees with the source. All three artifacts retain rope_theta = 500000, a RoPE dimension count of 128, and the declared context length of 4096.

4. The longer BF16 inference also succeeded. also separately verified prompt evaluation of 3,910 tokens with a 4,096-token context allocation. The model generated the expected response and exited successfully.

And what about 8192?

The static audit confirms that original_max_position_embeddings = 8192 was correctly incorporated into the RoPE calculation. It does not independently demonstrate reliable inference at an 8,192-token context.

TULLUS — Marin 8B Instruct

Structural validation: PASS. BF16, Q6_K, and Q4_K_M inference: PASS. RoPE formula validation: PASS, zero error across all three artifacts. Cross-quantization RoPE preservation: PASS. Original weights and GGUF artifacts: unchanged.

I now have the evidence i were looking for. The RoPE factors are exact, the quantizations preserve them, and all three artifacts can generate text.

Now i move forward with the process ☕

Verify the Ropes with LLama.cpp

=== GGUF metadata and static RoPE tensor audit ===
Selected stored metadata:
  general.architecture = llama
  general.file_type = 18
  llama.attention.head_count = 32
  llama.attention.head_count_kv = 8
  llama.block_count = 32
  llama.context_length = 4096
  llama.embedding_length = 4096
  llama.rope.dimension_count = 128
  llama.rope.freq_base = 500000.0
  llama.vocab_size = 128256
  tokenizer.ggml.add_bos_token = True
  tokenizer.ggml.add_eos_token = False
  tokenizer.ggml.bos_token_id = 128000
  tokenizer.ggml.eos_token_id = 128009

GGUF tensor count: 292 (reported for the record; no Q6_K full structural audit is implied here)
Tokenizer vocabulary entries: 128256 (expected 128256): PASS

Recorded metadata checks:
  PASS — general.architecture: got 'llama'; expected 'llama'
  PASS — general.file_type: got 18; expected 18
  PASS — llama.context_length: got 4096; expected 4096
  PASS — llama.embedding_length: got 4096; expected 4096
  PASS — llama.feed_forward_length: got 14336; expected 14336
  PASS — llama.block_count: got 32; expected 32
  PASS — llama.attention.head_count: got 32; expected 32
  PASS — llama.attention.head_count_kv: got 8; expected 8
  PASS — llama.rope.dimension_count: got 128; expected 128
  PASS — llama.rope.freq_base: got 500000.0; expected 500000.0
  PASS — tokenizer.ggml.bos_token_id: got 128000; expected 128000
  PASS — tokenizer.ggml.eos_token_id: got 128009; expected 128009

• ADDING --override-kv llama.context_length=int:8192 also -n 512 to not go over 8192 ctx.

/llama.cpp/build/bin/llama-server \
  -m "/TULLUS/marin-8b-instruct-GGUF/marin-8b-instruct-Q6_K.gguf" \
  -c 8192 \
  -np 1 \
  -n 512 \
  --override-kv llama.context_length=int:8192

Marin 8K test: PASS!

Thats exactly the result i was hoping for. The 512-token generation limit worked, the server retained its 8,192-token context slot, and generation finished without hitting the context ceiling.

The most important evidence is right here:

n_slots = 1, n_ctx_slot = 8192, kv_unified = 'false'

And at the end of generation:

prompt eval time =    3237.23 ms / 7449 tokens
eval time        =    7128.72 ms / 512 tokens
total time       =   10365.95 ms / 7961 tokens
stop processing: n_tokens = 7960, truncated = 0

Test results

Measurement	Result	Status
Configured context	8,192 tokens	PASS
Prompt processed	7,449 tokens	PASS
Tokens generated	512	PASS
Total reported tokens	7,961	PASS
Remaining context headroom	231 tokens	PASS
Context truncation	truncated = 0	PASS
Prompt processing speed	2,301 tokens/s	Excellent
Total elapsed time	10.37 seconds	PASS

And also

Final measured prompt: 7175 tokens
Filler blocks per gap: 64
Submitting retrieval questions to Marin ...

================================================================
MARIN 8K LONG-CONTEXT QUALITY TEST
================================================================
Reported prompt tokens: 7176
Reported output tokens: 200
Finish reason: length

• Test complete. One successful run is an initial quality check, not proof of reliable recall across every possible 8K prompt.

Model was showing more accurate outputs when I started using --yarn for prompts over 5k

Cool, the first 5500 ctx run passed so i left server running and used the WEB GUI AND IT PASSED THE 8K TEST !

DAY TWO - MORE TESTING AND I SAID HEY, LETS USE YARN AND GOT A HIT!

/llama.cpp/build/bin/llama-server \
  -m "/TULLUS/marin-8b-instruct-GGUF/marin-8b-instruct-Q6_K.gguf" \
  -c 8192 \
  -np 1 \
  -n 512 \
  --override-kv llama.context_length=int:8192 \
  --rope-scaling yarn \
  --rope-scale 2 \
  --yarn-orig-ctx 4096 \
  --yarn-ext-factor 1.0

7,047 tokens
3.2s
2,197.72 tokens/s  <- ingest t/s

Answer: Yes, I found all three strings in the dossier.

The exact primary archive calibration code from the opening control record is MARIN-ORCHID-7391.
The exact navigation cross-check phrase from the mid-dossier control record is CEDAR-ROUTE-2846.
The exact closing review marker from the final control note is CLOUD-HARBOR-6173.

marin-8B-Q6_K-instruct
91 tokens 1.3s

and the 8k test

====================================================================
MARIN 8K RECALL VALIDATION — THREE NEW IDENTIFIERS
====================================================================

TEST 1 — SHORT-PROMPT CONTROL
Answer:
OPENING_CODE: FROST-LANTERN-5827
MIDDLE_PHRASE: AMBER-CASCADE-3169
CLOSING_MARKER: INDIGO-VAULT-8402

Identifier retrieval:
  OPENING_CODE: PASS
  MIDDLE_PHRASE: PASS
  CLOSING_MARKER: PASS
Exact format: PASS

Exploratory coding evaluation

These small, custom coding probes were designed to establish an executable local evaluation methodology. They are not HumanEval, MBPP, or another standard public benchmark, and must not be used to characterize the model's overall coding ability.

JSON-record normalization task

The function under test was normalize_records(records). The specification required it to:

• inspect dictionary inputs and ignore arbitrary non-dictionary items;
• require id, name, and score;
• accept only integer IDs greater than or equal to zero, explicitly rejecting booleans;
• require a non-empty string name after stripping whitespace;
• accept integer scores, explicitly rejecting booleans, while allowing negative scores;
• ignore invalid records and drop extra fields;
• avoid mutating the input list and input dictionaries;
• return a new list of new dictionaries;
• deduplicate by ID, keeping the highest score, and keep the first valid occurrence on tied scores;
• sort by descending score and then ascending ID; and
• return [] for empty or wholly invalid input.

The harness had eight explicit cases × three checks (24 checks), plus three property sets × four checks (12 checks), for 36 checks total. The same baseline prompt definition and expected test cases were reused for the final greedy/default-sampling comparison.

JSON-record greedy baseline

Configuration: temperature 0.0, fixed seed 20261020.

Measurement	Recorded result
Rendered prompt	657 tokens
Generated output	136 tokens
Static screen	PASS

Across these custom tasks, the reported scores are task-specific and should remain separate. No aggregate coding score is calculated. The observed short-response generation speed of approximately 80 tokens/second is a local run observation, not a universal or sustained-throughput guarantee.

Running with llama.cpp

Use a compatible llama.cpp build and substitute the path to the artifact you downloaded. The standard examples below use a 4,096-token context, consistent with the recorded GGUF metadata.

Standard-context CLI

llama-cli \
  -m ./marin-8b-instruct-Q6_K.gguf \
  -c 4096 \
  -n 256 \
  -p "Explain the difference between an observation and an inference."

Replace the filename with marin-8b-instruct-BF16.gguf or marin-8b-instruct-Q4_K_M.gguf to use another release. This is a basic example, not a claim of optimal settings for every system.

Standard-context server

llama-server \
  -m ./marin-8b-instruct-Q6_K.gguf \
  -c 4096 \
  --host 127.0.0.1 \
  --port 8080

Experimental extended-context server — Q6_K + YaRN only

This is the recorded experimental configuration, shown for reproducibility. It is not the default configuration and does not rewrite the GGUF's stored context metadata.

/llama.cpp/build/bin/llama-server \
  -m "/TULLUS/marin-8b-instruct-GGUF/marin-8b-instruct-Q6_K.gguf" \
  -c 8192 \
  -np 1 \
  -n 512 \
  --override-kv llama.context_length=int:8192 \
  --rope-scaling yarn \
  --rope-scale 2 \
  --yarn-orig-ctx 4096 \
  --yarn-ext-factor 1.0

This command uses a local path from the recorded workstation; update it for your own file location. Use a llama.cpp build that supports these options. The recorded successful experiment used the runtime build and hardware listed above. Option availability, memory requirements, behavior, and results can vary by build and system. Leave room for both the prompt and generated completion within the allocated context, and validate your own workload. The command does not make all prompts up to 8,192 tokens reliable and does not establish a full-context test.

Download and verify an artifact

Using the Hugging Face CLI, for example, download the Q6_K file:

hf download TULLUS/marin-8b-instruct-GGUF \
  marin-8b-instruct-Q6_K.gguf \
  --local-dir ./marin-gguf

Check the exact file size and calculate the checksum locally:

stat -c '%n %s bytes' ./marin-gguf/marin-8b-instruct-Q6_K.gguf
sha256sum ./marin-gguf/marin-8b-instruct-Q6_K.gguf

Release downloads

Quantization	Download	Exact size in bytes	SHA-256
BF16	marin-8b-instruct-BF16.gguf	16,068,895,360	bccdb95ce05c69ebdaa6b6b9c4491424d0651f0378447aed86aa083a870f0b25
Q6_K	marin-8b-instruct-Q6_K.gguf	6,596,010,624	9c74e537c88a91a2c2d47bd08e9e8544ed409a4159fb149de8333245f001b60c
Q4_K_M	marin-8b-instruct-Q4_K_M.gguf	4,920,738,432	aa23593b3e66379e24990996bab2f32140087f1626b2fdb97c6432b9d5a4b373

The exact byte counts and SHA-256 values above are from the release record. The checksums were verified against the local artifacts before upload. After upload, remote file presence and reported sizes were checked against expectations.

Limitations and evidence boundaries

• Static RoPE validation: all three quantizations passed the recorded static RoPE audit. The rope_freqs.weight tensor is F32, contains the expected 64 factors, and matches the independently calculated factors exactly, with maximum absolute error of zero. The audited RoPE tensors are identical across BF16, Q6_K, and Q4_K_M. This establishes static artifact consistency; it does not independently prove runtime correctness at every context length.

• Tensor-structure audit: the broader 292-tensor structural audit recorded 32 layers, zero tensor-name or shape mismatches, exact preservation of all 66 audited original F32 payloads, no unexpected BF16 tensors, and an unchanged RoPE tensor.

• Quantization and validation: All three quantizations were produced on Kaggle. Smoke tests and validation checks were performed during the conversion workflow to verify tensor integrity, tokenizer consistency, and successful model loading and generation. The recorded checks passed within the scope of the tests performed.

• Short runtime smoke tests: all three quantizations passed short CPU-only load-and-generation checks and produced seemingly coherent output. These checks establish basic loadability and generation for the tested prompts, not broad model quality or parity across quantizations.

• Near-limit context test: BF16 completed the recorded test with an approximately 3,910-token prompt at a configured context size of 4,096, generated CONTEXT TEST PASSED, and exited successfully. This result is specific to that test; it is not a general long-context guarantee.

• These next tests were done locally on my 9070xt

• Experimental extended-context retrieval: Models recovered the expected identifiers in several specific tests beyond 4,096 prompt tokens under the local experimental YaRN configuration, including tests with approximately 5,471, 6,978, and 7,407 prompt tokens. In the 7,407-token test, all three expected identifiers and the required three-line output format were correct. These are useful positive retrieval results under the tested configuration, not proof that every prompt will work at extended context. A prompt filling the complete 8,192-token context was not validated, and these results do not establish equivalent extended-context behavior for BF16 or Q4_K_M. They also do not isolate YaRN as the sole cause of the observed results.

License, attribution, and dataset metadata

The upstream instruct model is identified as Apache-2.0 licensed. Consult the upstream instruct model card, the upstream base model, and license files distributed with the upstream source for applicable terms and attribution. This community GGUF release does not claim ownership of the original architecture, training process, or learned weights.

The upstream model-card dataset metadata is preserved here:

• TIGER-Lab/AceCode-87K
• bespokelabs/Bespoke-Stratos-17k
• cognitivecomputations/dolphin-r1
• tuenguyen/dolphin_r1_reasoning
• facebook/natural_reasoning
• open-r1/OpenThoughts-114k-math
• HuggingFaceTB/smoltalk

Language metadata: English (en). Recorded base model: marin-community/marin-8b-base.

This model card reports only the evidence recorded for the released files and the specified local tests. Each PASS is limited to its stated scope; experimental observations should not be read as guarantees beyond the tested conditions. '''

Bias, Risks, and Limitations

Like any base language model or fine-tuned model without safety filtering, these models can easily be prompted by users to generate harmful and sensitive content. Such content may also be produced unintentionally, especially in cases involving bias, so we recommend that users consider the risks when applying this technology. Additionally, many statements from Marin or any LLM are often inaccurate, so responses should be verified.

Marin 8B has not undergone any safety tuning or evaluation. We strongly recommend that users use this model with caution and consider the risks when applying this technology. In particular, this model is not intended for fully autonomous use.

TULLUS BAKES 🍪
Fresh From The AI Ovens, It's The Goods..