Juniper 1 Base (juniper1-base-v1)

Cinqic's first original base language model, trained from scratch: 142,771,456 parameters, one epoch over 3.41 billion tokens of the j1-base-v0.2.2 corpus, released under Apache-2.0 for the Cinqic-owned weights, tokenizer and code. Code, evidence and reproduction guide: https://github.com/Cinqic-Research/Juniper-1-Base.

This is a base next-token model. It has no chat template, instruction tuning, tool use, reasoning mode or safety tuning. Cinqic's post-training attempts on it did not qualify and are recorded as closed negative results (see "Post-training" below). Do not deploy it as an assistant.

Files

File	SHA-256	Size
model.safetensors (float32, exact tensors of the frozen checkpoint)	9dcf74b5db154951d1edbeea6801c04193facb8efc4d5b8b2796e6e675301952	571,107,560 B
config.json	see MANIFEST.json
tokenizer.json (cand5-j1tok-24k)	176a89e2197c24d3f024fbb60ce6f346635e7cba912b3579d93b37d486c9adf1	680,825 B
MANIFEST.json	lists the hashes above and the frozen identity
full-state/	the unmodified original checkpoint directory ckpt-000052034-336e5967 (model.pt, optim.pt, trainer.pt, manifest.json, COMPLETE) for resumable-training research; PyTorch pickles, loadable with weights_only=True through training/j1bench

Tensor-tree digest of the weights (reproduced by the loader): 1fd07c0d0c6c6b355460ab82ebe02530047b5682a4e6a41b9e4f0146323076e4. Source model.pt SHA-256: 71773e46db2511332af97891b6485b04bf414a448dab9c4b050988e0b532649c.

How to use

The model is a custom architecture (per-head QK-Norm, GQA, SwiGLU, tied embeddings). It is not loadable with AutoModelForCausalLM, and no hosted inference or widget is provided. Use the native loader:

pip install "juniper1[hub] @ git+https://github.com/Cinqic-Research/Juniper-1-Base@v1.0.0"
juniper1 complete "The Pythagorean theorem states that" --max-new-tokens 48

from juniper1 import load_model, complete, GenerationSettings
model, tok, info = load_model("CinqicX/juniper1-base")   # verifies hashes, config and tensor digest
print(complete(model, tok, "Photosynthesis is the process by which", GenerationSettings(max_new_tokens=32))["completion"])

Recorded greedy output (CPU float32) for the first prompt: "... the square of the length of the hypotenuse is equal to the sum of the squares of the lengths of the other two sides. In this case, the length of the hypotenuse is 10 and the length of the other two sides is". CPU float32 greedy decoding is deterministic; CUDA float32 logits differ by at most about 1e-4; bfloat16/float16 are derived variants that change logits.

Tokenizer: load tokenizer.json only through juniper1.Juniper1Tokenizer (or set encode_special_tokens = True yourself). The JSON does not persist that flag, and a bare Tokenizer.from_file turns a literal <|user|> in text into privileged control ID 3. Control IDs 1-63 were never trained.

Architecture

Type	dense decoder-only causal Transformer, j1-wide-20x768
Layers / width / FFN	20 / 768 / SwiGLU 1,920
Attention	grouped-query, 12 query heads, 6 KV heads, head_dim 64; learned per-head RMSNorm on q and k before RoPE
Positions	RoPE, θ = 100,000, rotate-half convention; 2,048-token context qualified (4,096 unqualified)
Norms	pre-RMSNorm (eps 1e-5) computed in fp32; final RMSNorm
Embeddings	tied input/output; vocabulary 24,576; no biases anywhere
Parameters	142,771,456 total; 123,897,088 non-embedding
Init	normal std 0.02; output projections std/√(2·layers)

Training

Data	j1-base-v0.2.2 train split: 2,954,191 documents, 3,410,114,054 stream tokens (EOS-separated); 52,034 updates × 65,536 targets = 3,410,100,224 targets, exactly one epoch, no repeats; windows of 2,048 tokens over the EOS-separated stream, cross-document attention allowed, every position a target
Optimizer	AdamW (fused), β = (0.9, 0.95), eps 1e-8, weight decay 0.1 on 2-D projection weights only (embeddings and norms excluded), gradient clipping 1.0
Schedule	peak LR 1.6971e-3, linear warmup 500 updates, cosine decay to 10 % of peak at update 52,034
Batch	32 sequences × 2,048 tokens per update (micro-batch 16, accumulation 2)
Precision	BF16 autocast, eager PyTorch (compiled execution failed an equivalence check and was not used); RMSNorm and losses in fp32
Hardware	one NVIDIA A100-SXM4-80GB (RunPod, US-MD-1); torch 2.8.0+cu128, CUDA 12.8, Python 3.12
Time	52,025 s of active training (about 14.5 h), with one restart after a power loss at update 8,585 (replayed updates verified against identical batch fingerprints)
Seed	1 (window permutation seeded by (seed, epoch))
Code identity	Juniper-1-Train j1bench at commit 828edd19, package SHA-256 81a3a277…204e99; published under training/

Training and validation curves: evidence/training_curve_j1-base-r2.json and the CSVs beside it in the GitHub repository.

Evaluation

Held-out language modeling on the full j1-base-v0.2.2 validation split (1,572 windows of 2,048, 3,218,204 targets, 12,311,466 scored bytes), re-measured independently on 2026-10-12 from the frozen bytes:

Checkpoint	bpb fp32	loss (nats)	perplexity	logged (A100 bf16)
50,000	0.917827	2.433787	11.402	0.917911
52,000	0.916198	2.429468	11.353	0.916280
52,034 (released)	0.916381	2.429953	11.358	0.916455

By source at 52,034 (logged, bpb): stackv2-edu 0.635, libretexts 0.861, finemath 0.885, arxiv 0.888, oercommons 0.912, pressbooks 0.942, fineweb-edu 0.949, wikimedia 1.001, python-peps 1.043, stackexchange 1.088.

The validation split was observed during training as a monitoring set (never trained on). It supports the checkpoint choice; it is not an unbiased capability estimate. The test split (2,604 documents) was never used. 52,000 is lower than 52,034 by 0.020 % relative, inside the pre-registered 0.1 % practical-tie threshold; 52,034 was selected as the run's terminal state with the stronger recoverability evidence. The selection protocol was registered after the training run's logged values were known; this is disclosed in training/docs/audit_2026-10-12/base_checkpoint_selection_report.md.

Small descriptive probes (greedy, n ≤ 20 each; not benchmarks): likelihood multiple choice on simple facts 17/20; executable Python with hidden asserts 3/8; few-shot two-digit addition 1/20; JSON prefix to valid JSON 1/6. Full prompts and outputs: training/docs/audit_2026-10-12/results/behavior_cpu.json.

Training data and rights

Ten sources, all pinned to exact upstream revisions and documented in data/docs/data_card.md and data/configs/sources.toml: FineWeb-Edu (SmolLM-Corpus dedup) 31.6 % of tokens, Stack v2 Edu (Common Pile, permissive licenses) 17.2 %, Wikimedia (Common Pile) 13.9 %, Stack Exchange (Common Pile) 13.0 %, FineMath 4+ 12.4 %, arXiv papers (Common Pile) 6.2 %, Pressbooks 3.1 %, LibreTexts 2.3 %, OER Commons 0.25 %, Python PEPs 0.08 %. English-first; multilingual content is incidental. Contamination controls: a layered benchmark-exclusion index (0 hits on the accepted corpus) and the F9 adjudication that removed 7,008 documents matching HumanEval+/MBPP+/MBPP test material; internal evaluation sets have low matchable fractions and remain a documented coverage limit.

The owner approved these sources as training input on 2026-10-08 and authorized this public release on 2026-10-16. The model weights are licensed under Apache-2.0 by Cinqic. That license covers the weights, not the third-party texts they were trained on: FineWeb-Edu and FineMath are ODC-By 1.0 databases over Common Crawl content (notice under ODC-By §4.3 in NOTICE); Wikimedia and Stack Exchange text is CC BY-SA; the textbook and arXiv sources carry per-document CC BY / CC BY-SA / CC0 / public-domain licenses; Stack v2 Edu code carries permissive open-source licenses. Whether weights trained on ShareAlike text are adaptations of it is not settled law; Cinqic releases the weights under Apache-2.0 and discloses the sources rather than asserting a conclusion. The dataset release (CinqicX/juniper1-base-corpus-v0.2.2) carries each payload under its original license and publishes reconstruction manifests, not text, for sources Cinqic does not redistribute.

Post-training (closed, not released)

Stage	Outcome
Instruction SFT pilot j1-sft-v1.2 (protocol v3) and two replication seeds	PILOT_CANDIDATE_QUALIFIED as a research candidate; explicitly non-production
SFT v2 on dataset v1.3	FAILED: generalization improved, but arithmetic, JSON, injection resistance and instruction-hierarchy regressed
SFT v2.1, v2.2 diagnostics	STOP_FAILED (hierarchy and JSON regressions against the registered non-inferiority gates)
Distillation v2 from a teacher model (three arms + control)	NO_IMPROVEMENT_DETECTED

All candidates, protocols and decision records are preserved (private Cinqic archives; decision files under training/docs/posttrain/results/). None of them is this release. Juniper 1 R&D was closed on 2026-10-16.

Limitations and intended use

Intended for research on small from-scratch language models, reproducibility studies, tokenizer and training infrastructure work, and as a base for others' experiments. Not intended for end-user applications without further training and evaluation. It confabulates facts, cannot do arithmetic reliably, repeats itself under greedy decoding, has no safety mitigations, and reflects the biases of web, forum, encyclopedia, textbook, paper and code sources. Context beyond 2,048 tokens is unqualified.

Citation

@misc{cinqic2026juniper1base,
  title  = {Juniper 1 Base: a 143M-parameter base language model trained from scratch},
  author = {Cinqic},
  year   = {2026},
  url    = {https://github.com/Cinqic-Research/Juniper-1-Base}
}