Crowd-v1

CrowdGPT's first community-distributed language model architecture.

• Model: Crowd-v1
• Organization: CrowdGPT
• Parameters: ~657M
• Architecture: SotaGPT
• License: MIT
• Status: Randomly initialized / training-ready
• Repository: https://github.com/Vxtzq/CrowdGPT
• CrowdGPT: https://crowdgpt.net

Overview

Crowd-v1 is the first official model architecture released for CrowdGPT, a community-driven distributed AI project.

Unlike a conventional pretrained model release, Crowd-v1 is distributed with randomly initialized weights. The purpose of this release is to provide a common model definition and weight format that CrowdGPT clients can download and collectively train.

The model is designed to be consumed by the CrowdGPT distributed training infrastructure, where individual participants contribute compute toward training a shared model.

Important: Crowd-v1 is not pretrained. Its initial weights contain no useful language knowledge and the model should not be expected to generate coherent text before training.

Architecture

Crowd-v1 contains approximately 502 million parameters.

Component	Configuration
Architecture	SotaGPT
Parameters	657,787,392
Vocabulary size	151,669
Hidden dimension	1,536
Layers	24
Attention heads	16
KV heads	4
Head dimension	96
MLP hidden dimension	2,560
Maximum sequence length	2048
Attention	Grouped-Query Attention
Position encoding	Rotary Position Embeddings (RoPE)
MLP	SwiGLU
Weight tying	Yes
Attention	Causal / chunked
Initialization	Normal distribution, σ = 0.02

Key features

Grouped-Query Attention (GQA) Crowd-v1 uses 16 query heads but only 4 key/value heads, reducing KV-cache and attention-related memory requirements.

Rotary Position Embeddings (RoPE) Rotary embeddings are applied to query and key representations.

SwiGLU The feed-forward network uses a SwiGLU-style gated MLP with a hidden dimension of 2,560.

Weight tying The token embedding matrix and language-model head share weights, reducing the total parameter count.

Causal attention Each token can only attend to preceding tokens, making the architecture suitable for autoregressive language modeling.

Initialization

The released weights are generated deterministically using seed 42.

Non-normalization parameters are initialized from:

N(0, 0.02)

LayerNorm weights are initialized to 1.0 and LayerNorm biases to 0.0.

This means that different copies of the initial model can be verified against the published SHA-256 checksums.

Weight formats

The repository contains flat binary weight files designed specifically for the CrowdGPT client.

BF16

weights_bf16.bin

Approximately 1.0 GB.

This is the recommended file for normal CrowdGPT client usage.

FP32

weights_fp32.bin

Approximately 2.0 GB.

The FP32 weights are provided for environments or experiments that require full-precision initialization.

The binary weights are stored as a single flattened parameter array. The tensor ordering is identical to the parameter ordering used by the CrowdGPT SotaGPT implementation, allowing the weights to be loaded directly without converting them into a framework-specific checkpoint format.

Configuration

The model configuration is provided in config.json.

{
  "architecture": "SotaGPT",
  "vocabSize": 151,669,
  "dim": 1536,
  "nLayers": 24,
  "nHeads": 16,
  "nKvHeads": 4,
  "headDim": 96,
  "maxSeqLen": 2048,
  "mlpHidden": 2560,
  "weightTying": true
}

Intended use

Crowd-v1 is primarily intended for:

• Distributed training through CrowdGPT
• Research into community-driven AI training
• Experiments with distributed language-model training
• Reproducible model initialization
• Development of CrowdGPT-compatible clients
• Educational experimentation with transformer architectures

Out-of-scope use

Because the released weights are randomly initialized, Crowd-v1 should not be used as:

• A production language model
• A chatbot
• A text-generation model
• A pretrained foundation model
• A source of factual knowledge

The model must first undergo training before it becomes useful for language generation.

Tokenizer

The current release uses a Qwen3-compatible vocabulary of 151,669 tokens.

The tokenizer is an architectural dependency: changing the tokenizer changes the vocabulary size and therefore the dimensions of the tied embedding / language-model head.

Future CrowdGPT releases may use a custom tokenizer with a different vocabulary.

If the tokenizer is changed, vocabSize in config.json must be updated accordingly and the model must be reinitialized with the new vocabulary dimensions.

Reproducibility

Crowd-v1 is generated from a fixed architecture and initialization seed.

The model-generation implementation is available in the CrowdGPT repository:

https://github.com/Vxtzq/CrowdGPT

The repository also contains the CrowdGPT client implementation responsible for consuming the flat weight format.

Verification

Each distributed weight file is accompanied by its SHA-256 checksum in config.json.

This allows clients and researchers to verify that downloaded weights have not been corrupted or modified.

License

Crowd-v1 is released under the MIT License.

See the repository for the complete license text.

Citation

If you use Crowd-v1 or CrowdGPT in your research or project, please reference the CrowdGPT project:

@misc{crowdgpt,
  title        = {CrowdGPT},
  author       = {Vxtzq},
  year         = {2026},
  url          = {https://github.com/Vxtzq/CrowdGPT}
}

About CrowdGPT

CrowdGPT is a community-driven approach to training AI.

Instead of concentrating all training compute in a single data center, CrowdGPT is designed around participants contributing their own compute to a shared training process.

The goal is simple:

Community compute → Shared training → Open model

Learn more at https://crowdgpt.net