GSN — Gated Sparse Network (Open Weights)

Acid Research (ACRS)

This repository contains trained weights for GSN, a language model architecture designed around one principle: do not spend compute you do not need.

Unlike transformers — which run the full model on every input regardless of complexity — GSN gates compute dynamically. Simple inputs take a shallow path. Complex inputs go deeper. The network decides, not the configuration.

For the full architecture, training code, and inference CLI, see AcidAI/Acid-GSN-Architecture.

Architecture

GSN is built from three core ideas stacked together:

1. Gated Depth A lightweight gate network evaluates each input and decides how many layers to activate. Layers that are not needed do not run. Their compute cost is exactly zero.

2. Sparse Activation Within each active layer, only the top-k neurons fire. The rest are masked to zero. A 4-layer GSN running at 25% sparsity uses a fraction of the compute a dense model would.

3. Recurrent Encoder Input tokens are processed sequentially through a GRU encoder before the gate sees anything. This replaces mean pooling — token order matters, context accumulates, and the gate receives a hidden state that actually encodes sequence structure.

4. Compute Penalty in Training The loss function penalizes wasted compute. The model is trained to be cheap, not just accurate. Over time the gate learns: if I can answer correctly with one layer, using two is a mistake.

No attention. No transformers. O(T) encoding. O(1) per sparse layer.

Files

File	Description
`gsn_fw_weights.npz`	Model weights (~19.7 MB)
`gsn_fw_opt.npz`	Adam optimizer state — m1/m2 moments and global step for resuming training (~39.4 MB)
`LICENSE.md`	APRIL license

Usage

Requirements

pip install numpy

No PyTorch. No CUDA. No framework dependency. Pure NumPy.

Loading the Weights

import numpy as np

# Load weights
weights = np.load("gsn_fw_weights.npz")
print(list(weights.keys()))  # inspect layer names

# Load optimizer state (for resuming training)
opt_state = np.load("gsn_fw_opt.npz")

For training and inference code, clone the architecture repository:

# Clone the architecture repo alongside these weights
# python infer.py "your prompt here" --max_new 100

See AcidAI/Acid-GSN-Architecture for the full usage guide.

Training Details

Corpus: FineWeb-Edu (high-quality educational web text) Vocabulary: 1024 BPE tokens Hardware: 2 physical CPU cores (Debian Linux) Training time: ~25 minutes Best loss: ~3.72 Average gate depth: ~2 / 4 layers

The gate settled at adaptive depth 2, confirming the compute penalty is functioning as intended. Depth variation is expected to increase on more diverse corpora.

Limitations

Undertrained checkpoint. These weights are a research-stage snapshot, not a production model. Coherent generation requires significantly more training on a larger corpus.
Small vocabulary. 1024 tokens is minimal. Real deployments should use 8k–32k.
Gate behavior is corpus-dependent. A model trained on FineWeb-Edu will gate differently than one trained on code or conversational text.
Single-sequence inference only. Batched inference is not yet implemented.

License

Licensed under the Acid Research Protected Interests License (APRIL) v1.0.

Free for personal, academic, and non-commercial use.
Derivatives must be open sourced under APRIL.
Commercial use requires written permission from ACRS.
Attribution to Acid Research (ACRS) is required in all derivatives.

See LICENSE.md for full terms.

Citation

If you use GSN in research or build on this architecture, please cite:

@misc{gsn2025,
  title  = {GSN: Gated Sparse Network},
  author = {Acid Research (ACRS)},
  year   = {2025},
  url    = {https://huggingface.co/AcidAI/GSN-Open-Weights}
}

About Acid Research

Acid Research (ACRS) is an independent AI research organization building CPU-native, economically viable alternatives to transformer-based architectures.

Current architecture portfolio:

HAM — Hebbian Architecture Model
RSM — Recurrent State Machine
RDM — Recurrent Depth Machine
IMA — Intent Machine Architecture
GSN — Gated Sparse Network

"Make it linear, or else your cost ain't going to be."

Downloads last month: -; Downloads are not tracked for this model. How to track

AcidAI
/

GSN-Open-Weights