CloudSurf-1B
CloudSurf-1B is a 0.89-billion-parameter base language model trained from random weights by CloudSurf Software LLC. CloudSurf built every part of it: the tokenizer, the trainer, the corpus pipeline and the weights. It is the on-device tier of Surfy, CloudSurf's assistant. It runs on a phone or a laptop with no network and no per-token cost. Its jobs there are routing requests, writing SurfDoc, and answering when the device is offline. It hands harder work to larger models.
This release is the base checkpoint. It is not instruction-tuned and has no tool-calling tune. Those come in the fine-tune that follows, which has its own pre-registered bars.
Inside CloudSurf this project was called TinySurf, which is why the GGUF files' embedded general.name reads Tinysurf 1b. The bytes are the ones we measured, so we did not re-stamp them. The released model is CloudSurf-1B.
Official GGUF quantizations (f16, Q8_0, Q4_K_M) are in cloudsurf-software/CloudSurf-1B-GGUF.
Run it
The model was trained with <|bos|> as the first token, and add_bos_token is true in the tokenizer. Prompts must start with it; llama.cpp adds it from the GGUF metadata, transformers adds it from the tokenizer config.
llama.cpp (the Q4_K_M quant, 0.55 GB):
llama-completion -hf cloudsurf-software/CloudSurf-1B-GGUF:Q4_K_M \
-p $'---\ntitle: The Colorado River\ntype: article\n---\n\n# The Colorado River\n\nThe Colorado River is' \
-n 120 --temp 0.7 --top-k 50 --repeat-penalty 1.1 -no-cnv
transformers (fp32 safetensors, 3.56 GB):
from transformers import AutoTokenizer, AutoModelForCausalLM
tok = AutoTokenizer.from_pretrained("cloudsurf-software/CloudSurf-1B")
model = AutoModelForCausalLM.from_pretrained("cloudsurf-software/CloudSurf-1B")
prompt = "---\ntitle: The Colorado River\ntype: article\n---\n\n# The Colorado River\n\nThe Colorado River is"
ids = tok(prompt, return_tensors="pt") # add_bos_token=True prepends <|bos|>
out = model.generate(**ids, max_new_tokens=120, do_sample=True, temperature=0.7, top_k=50, repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))
It is a base model: give it the start of a document and it continues the document. A SurfDoc front matter block followed by a heading is the prompt shape it saw most.
1. At a glance
| Field | Value |
|---|---|
| Parameters | 889,260,032 (counted from the built model, seed 1337) |
| Shape | decoder-only transformer · 16 layers · model width 2,048 · 16 heads (head dim 128) · SwiGLU MLP hidden 5,632 · RMSNorm (eps 1e-5) · rotary positions (theta 10,000) · no biases · tied input/output embedding |
| Context | 2,048 tokens |
| Tokenizer | tokenizer-v3: byte-level BPE, 32,768 ids, trained in house on a 2.0B-token sample of the corpus; specials <|bos|> = 1, <|eos|> = 2, pad = 0, ids 0..16 reserved (tokenizer.json sha256 begins 552fa13d) |
| Training tokens | 18,159,239,168 (one pass over corpus-v4, 17,318 steps × 1,048,576 tokens) |
| Compute | 8 × NVIDIA H100 80 GB SXM, 2026-10-23 21:07 UTC to 2026-10-24 05:36 UTC (about 8.5 hours), bf16 autocast, torch.compile, DDP |
| Formats | Hugging Face safetensors (fp32) · GGUF f16 · Q8_0 · Q4_K_M |
| Weights license | Apache-2.0 (section 9) |
2. Intended use
- For: routing requests to the CloudSurf tool registry and continuing or scaffolding SurfDoc documents. These are the uses the fine-tune targets. Also a small, fully disclosed base model for research on structured-document pretraining.
- Not for: chat or assistant use as released (it is a base model), factual question answering, high-stakes decisions, or languages other than English.
3. Training
The trainer is our own (ts_train.py, version 2). It uses DDP across 8 GPUs, fused AdamW, bf16 autocast, torch.compile, and a sharded uint16 loader. The recipe is standard for this model class:
- Peak learning rate 6e-4. Linear warmup over 1,000 steps, then cosine decay to 6e-5.
- Weight decay 0.1, betas (0.9, 0.95), gradient clip 1.0.
- Global batch of 1,048,576 tokens: 8 GPUs × micro-batch 8 × accumulation 8 × 2,048.
- Seed 1337.
The data mix has two phases. Phase 1 (steps 0–15,585) draws web 71.4 % · Wikipedia-as-SurfDoc 20.4 % · synthetic SurfDoc 5.1 % · code 3.1 %. Phase 2 (the last 10 %) draws web 43.6 % · Wikipedia-as-SurfDoc 32.7 % · synthetic 16.3 % · code 5.4 % · CloudSurf tool-routing examples 2.0 %.
Every bar was stamped before the first training token, at 2026-10-23T21:06:39Z, and none was changed during the run. The final validation loss was 2.3928 at step 17,000.
4. Data — sources, licenses, attribution
Corpus version 4 (manifest sha256 begins cea5e986); train 18,158,542,796 tokens / 15,752,206 documents.
| Source | Train tokens | License | Attribution |
|---|---|---|---|
FineWeb-Edu sample-100BT (Hugging Face, from Common Crawl) |
14,000,000,117 | ODC-By 1.0 | Common Crawl; HuggingFaceTB/fineweb-edu |
Wikipedia, converted to SurfDoc by CloudSurf's importer (medium-and-above tiers; 389,164 docs, 100 % surf validate clean) |
2,558,515,908 | CC BY-SA 4.0 | Wikipedia contributors |
| Synthetic SurfDoc (CloudSurf template generator, corpus-v1 + ts-gen-2) | 1,000,000,624 | owned | CloudSurf Software LLC |
| Source code, RedPajama-1T GitHub subset, permissive licenses only (MIT 955,610 · Apache-2.0 760,868 · BSD-3 201,420 · BSD-2 41,936 files before dedup) | 600,000,412 | per file MIT / Apache-2.0 / BSD | the original repositories (repo name kept per row) |
| CloudSurf tool-routing examples (phase 2 only) | 25,735 | owned | CloudSurf Software LLC |
Never in the corpus: output from Claude or any other agent, CloudSurf workspace or client documents, and SurfBench or BFCL fixtures (the corpus is decontaminated against these, never trained on them). We had planned a slice of text rewritten by a larger model and dropped it, so no text written by a language model is in the corpus. A case-insensitive grep for the deprecated product name finds 0 hits in every row set CloudSurf generated or transformed. It finds 3 lines in third-party FineWeb-Edu text, which were left as written.
Decontamination. Each document is checked against the evaluation sets for shared 13-grams. Cases shorter than 13 words are indexed whole, down to 6 words. Text is normalized first: lowercase, punctuation stripped. Any match drops the document. A planted-collision test passes.
Documents dropped, by eval set (a document can match more than one set):
| Slice | Dropped | ARC-C | ARC-E | HellaSwag | PIQA | MMLU | BFCL |
|---|---|---|---|---|---|---|---|
| FineWeb-Edu (of 23,316,000) | 32,798 | 128 | 942 | 4,489 | 126 | 24,715 | 2,486 |
| Wikipedia-as-SurfDoc (of 392,652) | 2,487 | — | 8 | 39 | 1 | 555 | 1,885 |
| Code (of 1,959,834) | 242 | — | — | — | — | 3 | 239 |
SurfBench (1,301 cases) is in the same index. The sealed validation sets (synthetic SurfDoc, Wikipedia-as-SurfDoc, and 18,558 FineWeb-Edu web documents) were never trained on; their hashes are in the corpus manifest. A self-check confirms none of the 18,558 val-web documents begins the way a train document does.
5. Evaluation
Bits per byte on sealed validation sets (n = 200 docs, ctx 2,048; lower is better). Bars were stamped before training.
| Validation set | CloudSurf-1B | Pre-registered bar | Random-init control | Result |
|---|---|---|---|---|
| corpus-v1 (synthetic SurfDoc) | 0.5193 | ≤ 0.5297 | 4.4026 | PASS |
| corpus-v3 (Wikipedia-as-SurfDoc) | 0.7789 | < 1.1484 | 4.3657 | PASS |
| FineWeb-Edu val-web | 0.7969 | < 1.30 (stated assumption) | 3.4038 | PASS |
General benchmarks — lm-eval-harness 0.4.9.1, 0-shot (MMLU 5-shot), bf16. Context only, no bar. † published numbers (TinyLlama EVAL.md), older harness version, so the margins are approximate.
| Task | CloudSurf-1B (18B tok) | Random-init control | Pythia-1.0B † (300B tok) | TinyLlama-1.1B † (3T tok) |
|---|---|---|---|---|
| HellaSwag (acc_norm) | 44.08 | 26.03 | 47.16 | 59.20 |
| ARC-Easy (acc_norm) | 54.04 | 25.63 | 48.99 | 55.25 |
| ARC-Challenge (acc_norm) | 29.44 | 25.00 | 27.05 | 30.12 |
| PIQA (acc_norm) | 67.36 | 51.80 | 69.21 | 73.29 |
| MMLU 5-shot (acc) | 24.80 | 23.87 | — | — |
MMLU is at chance, which is expected for a 1B base model at this token count.
6. Verdict against the pre-registration
Verdict of record: FAIL. The bits-per-byte clauses pass, the random-init control is vacuous, and the structural clause fails as registered.
Structural clause (b) — 500 real-document prefixes (Wikipedia-as-SurfDoc, sealed), continuation up to 1,536 tokens, temperature 0.7, top-k 50, checked by surf gate 0956e91c:
| Measure | CloudSurf-1B | Bar | Earlier 32M model (T1-B), same set and gate |
|---|---|---|---|
| Rows removed as repetitive (8-gram ≥ 4×, before scoring) | 315 / 500 | — | 312 / 500 |
| Parse-valid (of kept rows) | 88.1 % (163/185) | ≥ 97 | 46.8 % |
| Every block closed | 88.1 % | ≥ 99 | 47.3 % |
| Legal block names and attributes | 97.8 % | ≥ 99 | 60.1 % |
| Front matter valid | 100 % | ≥ 99 | 100 % |
| Rows that hit the 1,536-token cap | 94.8 % (474/500) | ≤ 10 % | 72.2 % (361/500) |
Case-read of every failure:
- No genuine syntax errors. All 22 failures among the kept rows are a table or figure cut off by the length cap.
- The repetition is real. The same detector removes only 19 of the 500 real continuations, against 315 of the model's. It also misses some loops, such as a list that increments a number on every line.
- The bars were taken from a different, easier set. They were copied from the 32M model's score on a short synthetic set. On this real-document set, the 32M model scores far lower than CloudSurf-1B. Any other reading of the clause is a second, separately registered read; the verdict above is the one of record.
A second read, registered before sampling (2026-10-24 22:36 UTC) and read 2026-10-25 01:06 UTC: PASS on every clause. The same weights were sampled on the set the structural bars were derived from (500 short synthetic SurfDoc prefixes, held out from training), with the identical sampler and gate. The verdict of record is unchanged.
| Measure | CloudSurf-1B | Bar | Earlier 32M model (T1-B), same set and gate |
|---|---|---|---|
| Rows removed as repetitive | 0 / 500 | ≤ 10 % | 0 / 500 |
| Parse-valid | 98.8 % (494/500) | ≥ 97 | 99.2 % |
| Every block closed | 100 % | ≥ 99 | 100 % |
| Legal block names and attributes | 100 % | ≥ 99 | 100 % |
| Front matter valid | 100 % | ≥ 99 | 100 % |
| Rows that hit the 1,536-token cap | 0 % (0/500) | ≤ 10 % | 0.2 % (1/500) |
| Rows with at least one block | 99.8 % | ≥ 95 | 99.6 % |
All six parse failures are one class: an empty ::code block, which the validator flags as a warning; the 32M model's four failures are the same class. A scan for loops the repetition detector misses found none. Read together with the verdict of record: the model writes closed, legal SurfDoc; on long real documents it still repeats, and that is the measured target of the fine-tune that follows.
A fine-tune of this base (not released; verdict 2026-10-29: FAIL on one bar of five). We fine-tuned this base on CloudSurf's own task data, a tool-routing set over the app's 140 tools (16 % of requests fit no tool) and 6,020 synthetic SurfDoc edit examples, for 1,000 steps of 32,768 tokens. It routes 97.7 % of held-out asks to the right tool (96.7 % on the subset with no near-twin in training), refuses 100 % of the asks no tool fits, writes SurfDoc edits that validate 96.25 % of the time, and moves held-out web-text bits/byte by only +0.021. It fails the fifth bar, no regression on long real documents: parse-valid 50.0 % against the base's 88.1 %, every block closed 73.75 % against 88.1 %, legal block names and attributes 55.0 % against 97.8 %, front matter 100 %; 420 of 500 continuations loop, against the base's 315. Read one by one, 30 of the 41 failing documents carry a real syntax error and 11 are only cut off by the length cap. The main one is measured: real documents write a table as format=table, the edit set writes format="table", and the tuned model blends the two into an unclosed quote. Ai2's OLMo 2 1B (Apache-2.0) given the identical data and recipe is the comparison arm: 72.2 % routed (71.7 % non-twin), 68.8 % refused, 95.8 % valid edits, +0.096 bits/byte — 1 bar of 3. CloudSurf-1B clears more bars, so it stays the base. Two disclosures go with the edit numbers. The continue task validates the prompt document plus the output, so a model that opens no block scores 100 % there; untrained models score 0 on the other four edit tasks. Randomly initialized models trained with the same recipe validate 76.7 % and 83.3 % of their edits, so the pre-registered control clause did not hold as written; the verdict above is read under a second clause registered before those controls were sampled, and under the original clause the read is "no verdict". Neither tuned checkpoint is released; the numbers are reported whichever way they land.
7. Limitations
- It is a base model: it continues text and does not follow instructions.
- It invents facts with confidence, including false and damaging claims about real people. Do not publish its output about a person unchecked. MMLU is at chance.
- It loops on long generations. At temperature 0.7 and top-k 50, 63 % of 1,536-token continuations of real documents repeat an 8-word span 4 or more times (real text: 3.8 %). Use a repetition penalty or short generations; the fine-tune that follows targets this.
- It is English-centric, and its code ability is minor (code is 3 % of the mix).
- The synthetic SurfDoc slice uses the August 2026 block set. The 18 block kinds added since (surf-parse v0.42.0) appear only through the Wikipedia importer, or not at all.
- Single seed and single run, with no ablations.
8. Files
Release files. The sha256 column shows the first eight characters; the full hashes are in SHA256SUMS in the GGUF repo.
| File | Where | Size | sha256 |
|---|---|---|---|
model.safetensors (fp32) |
this repo | 3.56 GB | 7064e999 |
CloudSurf-1B-f16.gguf |
CloudSurf-1B-GGUF | 1.78 GB | ef59f441 |
CloudSurf-1B-Q8_0.gguf |
CloudSurf-1B-GGUF | 0.95 GB | 8238324f |
CloudSurf-1B-Q4_K_M.gguf |
CloudSurf-1B-GGUF | 0.55 GB | 5a69a1dd |
Technical note for runtimes. The tensors are exported in the standard Llama decoder layout, so LlamaForCausalLM in transformers and the llama architecture in llama.cpp load it. The export matches the trained model to a maximum logit difference of 5.6e-05 in fp32. The GGUFs were converted with llama.cpp v0.7.0. The tokenizer's pre-tokenizer is the GPT-2 byte-level regex, and add_bos_token is true: the model was trained with <|bos|> first, so prompts must start with it. llama.cpp tokenization equals the Hugging Face tokenization on 20 of 20 validation docs. Perplexity on 40 val-web docs: f16 11.534 · Q8_0 11.536 · Q4_K_M 11.708.
9. License — Apache-2.0
The weights and the tokenizer are released under the Apache License 2.0, the same license as CloudSurf-4B-FC. The trainer (tinysurf/ts_train.py) carries the same license inside CloudSurf's training repository, which is not public at this release; the recipe above is complete enough to reproduce the run. Apache-2.0 grants no trademark rights: "CloudSurf" and the CloudSurf marks stay with CloudSurf Software LLC.
The training sources and their licenses are listed in section 4 and travel with the weights. Attribution is provided for every source: Wikipedia contributors (CC BY-SA 4.0, 14 % of training tokens), Common Crawl and HuggingFaceTB/fineweb-edu (ODC-By 1.0), and the original repositories of the code slice (MIT, Apache-2.0, BSD, kept per row in the manifest). CloudSurf's position is that the trained weights are not an adaptation of any individual training document, which is the position of the field's comparable from-scratch open models (OLMo 2, Pythia, TinyLlama, SmolLM2, all Apache-2.0 and all trained on Wikipedia and web text); the share-alike clause of CC BY-SA 4.0 is therefore not applied to the weights, and the attribution clause is honoured in full.
Never released under any license: the private evaluation holdout and the shadow-replay transcripts.
10. Citation
CloudSurf-1B, CloudSurf Software LLC, 2026. https://cloudsurf.com
@misc{cloudsurf1b2026,
title = {CloudSurf-1B: a 0.89B base model trained from scratch on 18B tokens of web, Wikipedia-as-SurfDoc, synthetic SurfDoc and code},
author = {CloudSurf Software LLC},
year = {2026},
url = {https://huggingface.co/cloudsurf-software/CloudSurf-1B}
}
- Downloads last month
- 438