ConeML/coneml-1.2b-experimental

ConeML 1.2B Experimental

FP32 safetensors and Q8_0 GGUF for an experimental research model. The benchmark tables below report FP32 results; Q8 export checks are described separately.

On the shuffled-fact relational endpoint panel, ConeML — approximately 348M active / 1.174B stored parameters — scores higher than the locally tested Qwen2.5-1.5B-Instruct, SmolLM2-1.7B-Instruct, Llama-3.2-1B-Instruct and Gemma-3-1B-IT at the tested depths 3, 5, 8, 12 and 24.

This experimental English instruction model identifies the highest, lowest, oldest, youngest, first or last entity from supplied relational facts. The comparison measures this specific skill under the disclosed native-chat protocol.

On a fixed native-chat comparison at depth 5 with shuffled facts, ConeML scores 438/512, compared with 273/512 for Qwen2.5-1.5B-Instruct and 155/512 for SmolLM2-1.7B-Instruct under first-choice scoring. Strict answer-only scores are 436, 273 and 154, respectively. Only two of ConeML's 438 credited answers are lost under the stricter rule, so the depth-5 advantage remains when explanatory prose receives no credit. Coding and ordinary conversation remain substantially weaker than these binding results.

Why this model exists

ConeML explores how much useful language and reasoning behavior can be expressed with a small active parameter footprint. This release provides a research model for studying supplied-fact reasoning, sensitivity to wording and fact order, and the gap between structured task success and conversational reliability.

The candidate accounts for approximately 11.42 billion processed input-token positions. This includes repeated positions and special tokens; it is not a unique-data count, total project cost, or a causal sample-efficiency measurement. For scale context, SmolLM2-1.7B reports 11 trillion pretraining tokens, while the Qwen2.5 family report describes up to 18 trillion. Qwen's figure is a family-level upper figure, not an audited consumed-token total for this exact 1.5B model. The peer figures exclude complete post-training accounting, whereas ConeML's figure includes its instruction-stage input positions; tokenizers and data also differ. These totals do not establish a matched-token efficiency advantage. SmolLM2 report, Qwen2.5 report.

Active parameter count likewise does not establish a FLOP or latency advantage.

Evaluations

All comparator scores below come from local evaluations. Different panels use different protocols, described with their results. Scores apply to these measured tasks and should not be combined into an overall model ranking.

Relational endpoint binding with shuffled facts

Each cell contains 512 questions across four families: height, alternative height-query wording, older/younger relations and coloured-card order. Depth is the number of links in the supplied chain. Facts are shuffled while preserving their meaning. Both endpoints are queried.

Numbers are correct first-choice entity answers, out of 512.

Depth	ConeML	Qwen2.5-1.5B-Instruct	SmolLM2-1.7B-Instruct	Llama-3.2-1B-Instruct	Gemma-3-1B-IT
1	487	481	426	267	343
3	471	410	221	197	224
5	438	273	155	136	138
8	343	202	102	73	100
12	240	105	58	47	79
24	74	47	21	13	19

ConeML is higher than these four comparators at depths 3–24 on this shuffled endpoint panel. Its absolute success still declines with depth: 438/512 at depth 5, 240/512 at depth 12 and 74/512 at depth 24. Ordered facts are easier: ConeML scores 490/512, 460/512 and 328/512 at those same depths.

First-choice scoring credits the first recognized entity and can credit prose that contains an incorrect explanation. The stricter results below require only the correct entity, allowing surrounding whitespace. They combine the six reported depths and both fact orders. Interior questions ask about two specified people inside a longer chain, rather than its global endpoint.

Model	Endpoints, strict answer-only	Interior comparisons, strict answer-only
ConeML	4,745/6,144	212/1,536
Qwen2.5-1.5B-Instruct	3,840/6,144	687/1,536
SmolLM2-1.7B-Instruct	1,966/6,144	693/1,536
Llama-3.2-1B-Instruct	1,403/6,144	553/1,536
Gemma-3-1B-IT	2,592/6,144	660/1,536

The endpoint advantage does not extend to interior comparisons. ConeML scores 13.8% under strict scoring; the peers score 36.0–45.1%. All fall below the 50% reference obtained by choosing uniformly between the two requested people. The models generate unconstrained text and can name someone outside that pair, so 50% is a specified reference strategy, not their output-space chance rate or a statistical significance claim. ConeML often returns a global extreme instead of a requested person. The peers perform better, but none reliably solves this panel.

Protocol: identical human prompts and fact sets, each model's native chat template, GPU FP32 inference, greedy decoding, repetition penalty 1.0, a 16-token short-answer budget and structured delimiter/EOS stopping. No prompt truncation was used. Template-specific defaults remain part of each model's native interface.

This is an exploratory suite, not a sealed or certified training-disjoint evaluation. Reciprocal questions, ordered/shuffled pairs and related task families are correlated. Each family/depth/order cell contains 64 chains with two questions each, rather than 128 independent trials. These results demonstrate endpoint selection under the stated protocol; they do not by themselves establish chain traversal or a particular internal mechanism.

ConeML was also evaluated through depth 128. At depths 64 and 128, shuffled endpoint scores were 5/512 and 6/512 under first-choice scoring. The model therefore does not maintain reliable shuffled binding at those depths. No peer comparison beyond depth 24 is claimed here. Results from earlier dense ConeML releases use different suites and protocols; no direct cross-release score improvement is claimed.

Code and wording robustness

Panel	ConeML	Qwen2.5-1.5B-Instruct	SmolLM2-1.7B-Instruct
Fixed Python execution panel, native chat	41/129	63/129	55/129
Python wording variants, native chat	15/20	20/20	20/20
HumanEval, native chat, original tests	12/164 (7.32%)	94/164 (57.32%)	58/164 (35.37%)

The fixed Python panels are repeatedly inspected development diagnostics with disclosed training-template overlap, particularly the context-manager interface. Passing their fixtures does not establish compliance with every prose requirement. ConeML's main-panel prompts additionally contain Task/Solution markers absent from the historical peer prompts; the 20 wording prompts match exactly. Saved ConeML fixed-panel inference used CPU FP32 and peers used GPU FP32. All use greedy decoding and repetition penalty 1.0.

HumanEval uses the same natural-language instruction and original 164 problems/tests for all three models, each model's native chat format, CPU FP32, greedy decoding, repetition penalty 1.0 and a 4,096-token response budget. Generation stops only at native EOS or that budget. Scoring extracts the first complete Python/py/unlabelled fenced block, or the entire response when unfenced, preserves supplied imports/helpers and runs the unchanged tests in isolation. There is no code repair or repetition-loop truncation in the primary score. ConeML reaches the token budget on 44/164 problems. Inspected capped responses include repetition loops: repeated function definitions, repeated example calls and extended repeated literals. A cap hit therefore does not necessarily represent 4,096 tokens of useful code. The 44 is a cap-hit count, not a count of independently classified loops. Training overlap has not been comprehensively excluded; this is not certified unseen transfer.

Context binding and public-task subsets

The separate 48-item context-binding panel scores 32/48 for ConeML, 34/48 for Qwen and 20/48 for SmolLM2. It uses bare completion for all three, rather than the native-chat depth protocol above. It is a repeatedly inspected development panel.

Task	Correct / examples	Accuracy
BLiMP	407/536	75.93%
LAMBADA	52/128	40.62%
PIQA	80/128	62.50%
SciQ, with support passage	78/128	60.94%
ARC Easy	58/128	45.31%

These are fixed CPU-FP32 subsets, not complete benchmark test sets. SciQ includes the support passage. The same model's checkpoint immediately before instruction tuning scored 59/128 on the same LAMBADA subset; this candidate scores 52/128 and retains a seven-item deficit. The independent sealed binding confirmation panel has not been consumed for this candidate.

Ordinary two-turn conversation

ConeML fully satisfied none of the six follow-up requests in this sample (0/6).

A small, manually reviewed sample tested six two-turn conversations covering code, supplied facts, explanation, configuration, message revision and storytelling. Each follow-up used the model's own actual preceding answer.

Model	Fully successful answers / 12	Partial / 12	Failed / 12	Fully successful follow-ups / 6
ConeML	1	1	10	0
Qwen2.5-1.5B-Instruct	8	1	3	4
SmolLM2-1.7B-Instruct	7	3	2	3

The same human prompts and frozen criteria were used, with native templates, GPU FP32, one greedy response, repetition penalty 1.0 and a 4,096-token budget. Grading concerns fulfillment of the request, not stylistic polish. The sample is small and purposively chosen; these counts are not a population accuracy estimate. They do show that strong structured binding does not currently translate into dependable everyday assistance. No tuned-decoding experiment or retry is included.

Intended format

Use the supplied native chat format for assistant interactions and code requests. Bare completion is a separate interface. The tokenizer encodes role markers and message bodies separately; concatenating unformatted text can change the input tokens.

The supplied chat template reproduces this encoding for ordinary text messages, including normal generated replies. Empty content or content beginning with multiple spaces, a tab or a newline requires the exact segmented encoder below. The template raises an error for these cases rather than trimming the message. This model is text-only and does not implement tool calling.

Intended uses include controlled research on relational endpoint selection, wording sensitivity and small sparse-model behavior, with outputs checked against task requirements.

Loading

FP32 safetensors

Tested with Transformers 5.12.1, PyTorch 2.12.1, and tokenizers 0.22.2. Loading requires no remote model code. The weights occupy approximately 4.70 GB; runtime memory is higher.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ConeML/coneml-1.2b-experimental"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.float32, attn_implementation="sdpa"
).eval()  # CPU; move to a suitable GPU if desired.

# Encode each segment independently, preserving all message whitespace.
def encode_messages(messages):
    markers = {"system": "System:\n", "user": "User:\n",
               "assistant": "Assistant:\n", "tool": "Tool:\n"}
    ids = []
    for message in messages:
        content = message["content"]
        if not isinstance(content, str):
            raise TypeError("Pass message content as text")
        ids += tokenizer.encode(markers[message["role"]], add_special_tokens=False)
        suffix = tokenizer.eos_token + "\n" if message["role"] == "assistant" else "\n"
        ids += tokenizer.encode(content + suffix, add_special_tokens=False)
    ids += tokenizer.encode("Assistant:\n", add_special_tokens=False)
    return ids

messages = [{"role": "user", "content":
    "Jonathan is taller than June. Of all of them, the tallest is who? Return only the name."}]
ids = torch.tensor([encode_messages(messages)], device=model.device)
with torch.inference_mode():
    output = model.generate(
        input_ids=ids, attention_mask=torch.ones_like(ids),
        do_sample=False, repetition_penalty=1.0, max_new_tokens=64,
        eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(output[0, ids.shape[1]:], skip_special_tokens=True))

For ordinary messages, tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_dict=False) is also supported. The segmented encoder remains the exact interface for whitespace-sensitive inputs. Do not add a second BOS token or a generic instruction template.

Q8_0 GGUF

coneml-1.2b-experimental-Q8_0.gguf is approximately 1.25 GB, with additional runtime memory needed. It contains the native chat template and tokenizer. Validated on CPU with llama.cpp binary revision 9ebfc3a8cf4c1c6983258c4d603274b2b3d3dd65; conversion used source revision 169e4a7ff201e666c2325bcd35afb64573e39d09. Other runtimes need support for this Mixtral configuration: nine experts per layer, two selected per token.

For example, after downloading the GGUF:

llama-server -m coneml-1.2b-experimental-Q8_0.gguf \
  --host 127.0.0.1 --port 8080 -ngl 0 -c 16384 \
  --jinja --skip-chat-parsing

Use the embedded template, greedy decoding and repeat_penalty: 1.0 when comparing with the reported protocol. --skip-chat-parsing keeps generated text in the response content rather than trying to interpret tool or reasoning tags. Runtime defaults may otherwise differ. For whitespace-sensitive messages, send the exact token IDs from the segmented encoder to llama.cpp's /completion endpoint.

The release check used 28 fixed existing cases: 24 relational endpoint questions, two code turns and two conversational turns. FP32 GGUF matched the FP32 safetensors answer text on 28/28; Q8 matched on 27/28, including all binding and code answers. The changed conversational response failed the request in both versions. All 28 rendered prompts matched the reference token IDs in both runtimes. This is a small conversion regression check, not a rerun of the benchmark tables or a guarantee of identical Q8 behavior on other prompts.

Architecture

Property	Value
Stored parameters	1,174,219,776
Active parameters per token, conventional count	Approximately 348,466,176
Transformer layers	30
Hidden width	1,024
Attention / KV heads	8 / 2
Experts per layer / selected per token	9 / 2
Configured context	16,384 tokens
Vocabulary	32,768

Mixtral-style decoder with a tied embedding/output table. Comparator sizes below were counted from the pinned weights used in the local evaluations, with tied weights stored once.

Model	Stored parameters, rounded	Architecture
ConeML	1.174B	Sparse MoE; approximately 0.348B active
Qwen2.5-1.5B-Instruct	1.544B	Dense
SmolLM2-1.7B-Instruct	1.711B	Dense
Llama-3.2-1B-Instruct	1.236B	Dense
Gemma-3-1B-IT	1.000B	Dense

ConeML has fewer stored parameters than Qwen, SmolLM2 and Llama, and more than Gemma. Active parameter participation is not measured inference cost. The configured context length does not establish reliable behavior throughout that context.

Strengths

• Shuffled relational endpoints: 438/512 at depth 5 and 240/512 at depth 12, versus Qwen's 273/512 and 105/512 on the same questions; the strict depth-5 result is 436/512.
• Non-name entities: on the expanded coloured-card suite at depth 5, 106/128 ordered and 85/128 shuffled, all correct under strict answer-only scoring.
• Executable Python: 41/129 on the fixed execution panel, 15/20 on wording variants and 12/164 on the stated HumanEval protocol. These establish specific successes alongside the larger coding gap reported above.

Known limitations

Interior pairwise comparisons, multi-turn corrections and preservation of earlier constraints are unreliable. Code can omit requirements, fail tests or disagree with its explanation. Wording changes can break otherwise successful responses. Narrative answers can stop early or repeat, and long-form code can exhaust the response budget. Factual coverage and ordinary conversational reliability remain limited.

This is an experimental research release. Generated code and factual answers require independent checking. The measurements do not establish suitability for autonomous execution or high-stakes decisions, or broad assistant parity with the comparators.

License

Creative Commons Attribution–NonCommercial 4.0 International (CC BY-NC 4.0). Third-party rights remain applicable.