ash3241/Amplitude-based-tokenizer

Continuous Amplitude Tokenization (CAT) - Toy Model

This repository contains the trained weights and vocabulary for a proof-of-concept toy model utilizing the novel Continuous Amplitude Tokenization (CAT) architecture.

Model Details

Traditional LLMs rely on discrete BPE tokenization, forcing them to treat intensity modifiers ("very", "extremely", "slightly") as independent, arbitrary tokens. The CAT architecture splits language into two streams:

1. Concept ID (Discrete): e.g., "bad", "large".
2. Amplitude (Continuous): e.g., 1.0, 2.0, 5.0.

This toy model processes sentences like "The food was incredibly bad" as a single compressed token mapping: [Concept: "bad", Amplitude: 4.0], completely eliminating the need for combinatorial adjective tokens and physically shortening sequence lengths.

• Architecture: Custom Dual-Stream Causal Transformer (CAT)
• Key Components: Fourier-FiLM Embeddings, AP-RMSNorm, 4-way Bilinear Attention, Spatio-Amplitude RoPE.
• Parameters: ~4.6 Million
• Vocabulary Size: 15,000 words (Modifiers deleted and replaced by continuous scalars)
• Dataset: 15 Million tokens (Yelp Polarity Reviews)
• Training Iterations: 1,000 (Early-stage viability test)

Empirical Benefits

During benchmarking against a standard NanoGPT model on the exact same text, this CAT model achieved:

• 20% KV Cache Memory Savings (due to reduced sequence lengths).
• Vocab Compression (by removing standard modifiers from the dictionary).
• Zero-Shot Controllability (ability to manually inject Amplitude: 10.0 to force intensity scaling during generation).

Usage

To use these weights, you must use the custom CAT Model code from the official GitHub repository.

1. Clone the architecture:

git clone https://github.com/Yash943/CAT-Concept-Amplitude-Tokenization-.git
cd CAT-Concept-Amplitude-Tokenization-
pip install -e .

2. Load the model (PyTorch):

import torch
from cat.model import CATModel, CATConfig
from huggingface_hub import hf_hub_download

# Initialize the architecture
config = CATConfig(
    vocab_size=15000, 
    hidden_size=128, 
    num_hidden_layers=4, 
    num_attention_heads=4,
    intermediate_size=512,
    max_position_embeddings=64
)
model = CATModel(config)

# Load the weights
weights_path = hf_hub_download(repo_id="your-repo-id/here", filename="cat_toy_model.pt")
model.load_state_dict(torch.load(weights_path))
print("CAT Model loaded successfully!")

Intended Use

This is an experimental "toy" model designed strictly to prove the mathematical viability of Fourier-FiLM embeddings and AP-RMSNorm at reducing token fertility. It is not intended for production chat or generation tasks.

GitHub Repository

For full mathematical breakdowns, empirical verification scripts, and architectural source code, visit the official GitHub repository.