Alethic 151M --- SinGatedAttention Language Model

Alethic is an experimental ~151M-parameter decoder-only language model pretrained from scratch on FineWeb-Edu.

The project investigates SinGatedAttention (SGA): an additional attention-conditioned nonlinear transformation inserted between standard causal self-attention and the feed-forward network in every Transformer block.

The central research question is simple:

Can an attention-derived signal act as a useful nonlinear computational gate, rather than being used only as a contextual representation?

Status: Active research / experimental pretraining. The current checkpoint is not presented as a finished or compute-optimal language model.

Current Training Run

Property Value

Parameters ~151M Dataset FineWeb-Edu Tokenized corpus 424,051,527 tokens Training split 419,811,011 tokens Validation split 4,240,516 tokens Tokenizer SentencePiece Unigram Vocabulary 20,000 Context length 256 Batch size 8 Tokens / optimization step 2,048 Current training step 37,500+ Recent training loss ~4.1 Recent validation loss ~4.5 Training hardware NVIDIA Tesla T4 16 GB Precision BF16 Observed throughput ~2,446 tokens/sec

At step 37,521, the run had sampled approximately:

37,521 × 8 × 256 = 76,843,008 training tokens

or about 76.8M sampled training tokens.

Because training windows are sampled randomly from the memory-mapped corpus, this is a count of tokens used in optimization, not a claim that 76.8M unique corpus tokens have been visited.

Relative to a ~151M parameter model, the current run is still heavily undertrained:

76.8M / 151M ≈ 0.51 sampled training tokens per parameter

The current validation loss of roughly 4.5 should therefore be interpreted as an intermediate training measurement, not a final model result.

Architecture

Each Alethic Transformer block contains three residual transformations:

x₁ = x  + MHA(LN(x))
x₂ = x₁ + SGA(LN(x₁))
x₃ = x₂ + FFN(LN(x₂))

A conventional block would normally contain the attention and feed-forward pathways. Alethic adds SinGatedAttention as a separate residual computation between them.

Model Configuration

Component Configuration

Layers 12 Hidden dimension 768 Standard attention heads 12 Attention head dimension 64 FFN expansion 4× FFN activation GELU Normalization LayerNorm Position representation Learned positional embeddings Vocabulary 20,000 Context window 256 Objective Autoregressive next-token prediction

SinGatedAttention

SinGatedAttention first computes a causal attention signal:

A = Attention(x)

It then applies a learned linear transformation to the input and modulates that representation using a sinusoidal transformation of the attention signal:

SGA(x) = (W x + b) ⊙ [α sin(A)]

where:

W and b are learned parameters;
A is produced by causal self-attention;
α is a learnable scalar;
⊙ denotes element-wise multiplication.

The core implementation is equivalent to:

A = Attention(x)

Wx = Linear(x)
gate = alpha * sin(A)

output = Wx * gate

The attention result therefore acts as a gate over a learned representation instead of being directly used as the output representation.

Intuition

Conceptually, SGA separates representation transformation from context-dependent modulation:

             ┌── Linear(x) ──────────────┐
x ───────────┤                            × ── output
             └── Attention(x) → sin(.) ──┘

The linear branch determines the transformed representation available to the layer, while the attention branch controls its modulation.

Since sine is bounded,

-α ≤ α sin(A) ≤ α

the gate remains bounded by the learned scale α.

This is an architectural motivation, not evidence by itself that SGA improves language-model performance.

Full Model

Token Embedding
      +
Learned Position Embedding
      │
      ▼
┌──────────────────────────────┐
│ Transformer Block × 12       │
│                              │
│ LN → Causal MHA → Residual   │
│ LN → SGA        → Residual   │
│ LN → GELU FFN   → Residual   │
└──────────────────────────────┘
      │
      ▼
Final LayerNorm
      │
      ▼
LM Head
      │
      ▼
Next-token logits

The standard causal attention pathway uses PyTorch scaled dot-product attention. SGA contains its own causal multi-head attention operation followed by SinGatedLinear.

Dataset

Alethic is pretrained using FineWeb-Edu:

HuggingFaceFW/fineweb-edu

The current prepared dataset contains:

Total tokenized corpus     424,051,527
Training tokens            419,811,011
Validation tokens            4,240,516

The prepared token stream uses an approximately 99% / 1% train-validation split.

Documents are encoded and separated with <eos>. During training, contiguous 256-token windows are randomly sampled from the memory-mapped training partition.

The current script selects the first 5 Parquet shards when preprocessing a new dataset cache.

Tokenizer

The run uses a custom SentencePiece Unigram tokenizer rather than a tokenizer inherited from another pretrained language model.

Property Value

Vocabulary size 20,000 Algorithm SentencePiece Unigram Character coverage 0.9995 Tokenizer sample Up to 500,000 documents Sampling strategy Reservoir sampling

Special tokens:

<pad>
<unk>
<bos>
<eos>

Reservoir sampling is used so tokenizer-training examples can be selected across the processed corpus rather than simply taking the first documents encountered.

Training

Alethic uses autoregressive next-token prediction with cross-entropy loss.

For tokens

t₁, t₂, ..., tₙ

the model learns to predict:

P(tᵢ₊₁ | t₁ ... tᵢ)

Optimization Configuration

Setting Value

Optimizer AdamW Learning rate 3e-4 Batch size 8 Sequence length 256 Tokens / step 2,048 Evaluation interval 500 steps Validation batches / evaluation 20 Script target validation loss 3.5

On CUDA, automatic mixed precision is enabled. The script uses BF16 when supported and falls back to FP16 otherwise.

A recent Tesla T4 session reported approximately:

0.60 optimization steps/sec
~2,446 tokens/sec

Throughput can vary between environments and sessions.

Checkpointing

The training script maintains:

best_fineweb_edu.pt
latest_fineweb_edu.pt

The best checkpoint tracks the lowest observed validation loss. The latest checkpoint stores resumable training state.

Checkpoint state includes:

model parameters;
optimizer state;
AMP scaler state;
current training step;
best validation loss;
vocabulary size;
model configuration.

The script is designed to tolerate limited-duration compute sessions. At startup it benchmarks throughput, estimates how many steps fit into the configured session budget, periodically evaluates the model, and saves progress.

Try It Yourself

The current train.py is a research script rather than a packaged trainer. Before running it, review the configuration and path sections for your environment.

1. Install Dependencies

A minimal environment needs Python and PyTorch plus the preprocessing dependencies used by the script:

pip install torch sentencepiece pyarrow huggingface_hub numpy

A CUDA-capable PyTorch installation is recommended for training.

2. Configure Storage Paths

The script expects persistent locations for:

downloaded FineWeb-Edu shards;
the SentencePiece tokenizer;
the encoded .bin token cache;
checkpoints.

If you are not using the original Google Drive/Colab layout, change the path definitions near the top of train.py to directories that exist on your machine.

For example:

from pathlib import Path

DRIVE_ROOT = Path("./alethic_data")
RAW_DATA = DRIVE_ROOT / "fineweb_edu"
TOKENIZER_PREFIX = DRIVE_ROOT / "tokenizer_fineweb_edu_sp_20k"
BIN_CACHE = DRIVE_ROOT / "dataset_fineweb_edu_sp.bin"
CHECKPOINT_DIR = DRIVE_ROOT / "checkpoints"

DRIVE_ROOT.mkdir(parents=True, exist_ok=True)
RAW_DATA.mkdir(parents=True, exist_ok=True)
CHECKPOINT_DIR.mkdir(parents=True, exist_ok=True)

Keep the tokenizer and .bin cache together as a matched preprocessing set. A binary cache encoded with a different tokenizer must not be reused as if it belonged to the new tokenizer.

3. Start Smaller

The repository configuration targets the full research model. If you only want to verify that the architecture and training loop work, reduce the configuration first.

For example:

@dataclass
class GPTConfig:
    batch_size = 4
    block_size = 128

    n_dims = 256
    vocab_size = 20_000

    n_head = 4
    head_dims = 64

    n_layer = 4

The important constraint is:

n_dims = n_head × head_dims

For the original model:

768 = 12 × 64

After confirming that preprocessing, forward/backward passes, evaluation, and checkpointing work, increase the configuration according to your available compute.

4. Change the Dataset Amount

The number of FineWeb-Edu Parquet shards selected for preprocessing is controlled by:

NUM_SHARDS = 5

For a quick test:

NUM_SHARDS = 1

For the current Alethic corpus:

NUM_SHARDS = 5

Increasing this value increases preprocessing time, disk usage, and the available token corpus.

Important: if you change the dataset or tokenizer configuration, create new tokenizer/cache filenames or remove the old prepared artifacts intentionally. Otherwise the script may detect the old cache and skip preprocessing.

5. Change the Tokenizer

The default vocabulary is:

vocab_size = 20_000

Tokenizer training currently samples up to:

max_lines = 500_000

and uses:

model_type="unigram"
character_coverage=0.9995

If you change the vocabulary, tokenizer algorithm, source corpus, or special-token layout, regenerate the tokenized .bin cache as well.

6. Adjust Batch Size and Context Length

The main memory controls are:

batch_size = 8
block_size = 256

If you run out of GPU memory, reduce batch_size first:

batch_size = 4

or:

batch_size = 2

Reducing block_size also lowers memory requirements, but it changes the model's training context and therefore changes the experiment more substantially.

Tokens per optimization step are:

batch_size × block_size

For the original run:

8 × 256 = 2,048 tokens/step

7. Adjust the Training Session

The current script uses a time-budgeted session:

SESSION_HOURS = 4.0

Change it to match the compute environment:

SESSION_HOURS = 1.0

or:

SESSION_HOURS = 8.0

The script benchmarks recent training throughput and estimates how many optimization steps fit inside the usable session time.

The stopping target is currently:

TARGET_LOSS = 3.5

This is a script stopping condition, not a claim that 3.5 is theoretically optimal or guaranteed to be reached.

8. Evaluation Frequency

Validation is currently configured as:

EVAL_INTERVAL = 500
EVAL_BATCHES = 20

More validation batches produce a more stable estimate but consume additional compute.

For a quick development run you might use:

EVAL_INTERVAL = 250
EVAL_BATCHES = 5

Do not compare research runs using substantially different evaluation procedures without accounting for that difference.

9. Checkpoints and Resuming

If latest_fineweb_edu.pt exists, the script automatically attempts to resume from it.

If you intentionally change the architecture, do not load an incompatible old checkpoint. Use a new checkpoint directory or rename/remove the previous checkpoint files.

For separate experiments, a clean structure is useful:

experiments/
├── baseline/
│   └── checkpoints/
├── sga/
│   └── checkpoints/
└── ablation_tanh/
    └── checkpoints/

This also reduces the chance of accidentally mixing checkpoints between architectures.

10. Run Training

Once the paths and configuration are ready:

python train.py

The script will:

prepare/load tokenizer
        ↓
prepare/load token cache
        ↓
initialize model
        ↓
resume checkpoint if available
        ↓
benchmark throughput
        ↓
train
        ↓
periodically evaluate
        ↓
save checkpoints
        ↓
open interactive generation

A CUDA GPU is required by the current main execution path; the script exits when CUDA is unavailable.

Reproducing the Research

Simply getting train.py to run is different from testing whether SinGatedAttention works better.

A meaningful SGA comparison should control as many variables as possible:

Variable Control

Dataset Same Tokenizer Same Training tokens Same Sequence length Same Optimizer Same Learning rate Same Seeds Multiple Parameter count Matched or explicitly accounted for Training FLOPs Matched/accounted for Runtime Reported Evaluation procedure Same

A useful first comparison is:

Standard Transformer
        vs.
Transformer + SinGatedAttention

Additional ablations can investigate:

sin(A)
tanh(A)
linear/non-periodic gate
fixed α
learnable α
SGA without the standard MHA pathway

The current SGA block adds another attention operation and additional learned parameters. Therefore a same-depth vanilla Transformer is not automatically a parameter- or compute-matched baseline.

Research Questions

This project is intended to investigate:

Does SinGatedAttention improve learning relative to a controlled Transformer baseline?
If an improvement exists, does it come from the sinusoidal gating mechanism or from additional parameters/compute?
Does the effect persist across random seeds?
How does SGA behave as model scale increases?
How sensitive is the architecture to the gating function?
Can attention-conditioned multiplicative modulation provide useful computation beyond conventional attention + FFN pathways?

These remain experimental questions until supported by controlled results.

Generation

After training, the current script opens a simple interactive generation loop:

ALETHIC INTERACTIVE GENERATION
Type /exit to stop.

Alethic >> The future of artificial intelligence

Generation is autoregressive and samples from the softmax distribution. When the generated context exceeds the configured context window, only the most recent tokens are provided to the model.

Project Status

Experimental / active research.

The current ~151M run demonstrates the architecture at a substantially larger scale than the initial small-model experiments, but it should not be interpreted as a finished language model or as proof that SinGatedAttention outperforms conventional architectures.

Results, checkpoints, controlled baselines, ablations, and scaling experiments are the evidence that matters.

Author

Bekhruz Suleyman

Independent AI architecture research and language-model pretraining.

License

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Downloads last month: -; Downloads are not tracked for this model. How to track

bekhruzsuleyman
/

alethic-151m