BrodyMakezAI

/

gpt-5horts

Like 0

Hi Brody,

"Times it brought up Roblox unprompted: 2" and "Vibes: Immaculate" is easily the most honest benchmark suite on Hugging Face this month. Training a pure 10.7M model to exclusively simulate the linguistic entropy of YouTube Shorts comments is fantastic chaotic engineering.

Looking at the underlying mechanics of a 6-layer, 384-dim character-level configuration:

1. The zero vocabulary tax advantage:
   While modern sub-50M models often waste 20% to 35% of their parameter budget on massive 32k or 49k BPE embedding lookup tables, a character-level model has a microscopic vocabulary. Virtually 100% of your 10.7M parameters are allocated directly into active math and attention layers.

2. The character-level sequence inflation tax:
   The trade-off with character-level causal GPTs is token fertility: spelling out typical Shorts chaos ("AHHH MOST POPULAR SHORT OF APPIL?????????!") eats 50+ tokens in seconds.
   Because standard quadratic attention scales as O(L^2), parsing or continuing across longer comment threads quickly saturates the attention window, while single characters carry very low individual semantic salience.

3. Constant O(1) state updates for character streams:
   In an open architecture project called Maba v2 (reference release: https://huggingface.co/AndrewThompson1233/maba-v2-architecture), we explore sub-quadratic scaling using a 3:1 macro-topology (DGDA linear recurrence paired with sparse latent attention):
   Character-level modeling is historically the natural habitat for recurrent state transitions. Instead of storing an expanding quadratic attention cache for every single letter, space, and emoji, linear recurrence continuously folds incoming characters into an associative memory matrix in constant O(1) space.
   This allows multi-thousand character rollouts with strictly flat decode latency and near-zero memory footprint.

Did you train with a fixed byte-level vocabulary (e.g. 256 bytes) to capture any raw unicode/emoji directly, or did you prune to a custom character set?

Best,
Andrew