StarAtNyte1/beam-boot

Beam Boot: experiments toward a computer in an attention model

The goal is to run a tiny CPU and bootable monitor through model forward passes, investigate whether the idea can be adapted to Beam's architecture, and let Beam interact with it. This is research in progress: there is no general-purpose operating system or modified pretrained Beam checkpoint yet.

Run

Requires Python 3.10+ and PyTorch. For a small CPU-only installation:

uv venv --python 3.12 .venv
uv pip install --python .venv/bin/python torch --index-url https://download.pytorch.org/whl/cpu
uv pip install --python .venv/bin/python "transformers>=5.20.1,<6"
OMP_NUM_THREADS=1 .venv/bin/python -m unittest discover -s tests -v
.venv/bin/python -m beamboot.demo

Run the experimental CPU monitor:

OMP_NUM_THREADS=1 .venv/bin/python -m beamboot.os

It boots to BEAM>; try help, version, add 7 8, and quit. Addition is modulo 16. This is a single-task ROM monitor, not a general-purpose OS. The host maps terminal commands to memory-mapped device tokens, renders output, and feeds generated state tokens back to the model. The original scalar sketch is available via python -m beamboot.demo, and the earlier 32-instruction monitor via python -m beamboot.monitor.

What exists

Tensor attention memory (beamboot/tensor_memory.py). Address keys and byte values are fed to PyTorch's scaled_dot_product_attention. A bounded recency signal makes the latest matching write dominate, and initial zero entries allow reading unwritten addresses. Host Python creates the queries and the cache, rounds the answer, and enforces a finite write limit. It is not a transformer layer and does not use Beam weights. Tests compare 256 seeded updates against a dictionary-like reference. A sliding window can evict old memory and return the wrong value. Softmax and finite precision mean these tests do not prove exactness for all configurations. The scheme needs an address-score range proportional to the write budget and must be tested with actual Beam attention, positional encoding, cache implementation, and hardware.

Compiled neural CPUs (beamboot/cpu.py, beamboot/machine.py). Two small feed-forward networks have their weights constructed from program-specific truth tables; neither is trained. The second has a 4-bit accumulator, 4-bit program counter, one 4-bit RAM cell, branch instructions, and an output flag. Each forward pass advances one instruction. The host encodes the current state into bits, thresholds the resulting bits, feeds them back, and handles output. A test compares all 8,192 non-halted states against the reference interpreter for one program, including all instructions used in that program. This is a useful exact finite-state experiment, not an attention CPU: memory is encoded in the state, ROM is compiled into a ReLU lookup circuit, and the machine has no general-purpose memory or operating system. The construction scales exponentially with state size.

Scalar attention sketch (beamboot/memory.py). The original dependency-free experiment remains for comparison and exposes a host-side missing-key check.

Attention-backed CPU (beamboot/attention_cpu.py, beamboot/monitor.py). This model connects a ROM-specific ReLU CPU circuit to SDPA-addressed RAM: each forward pass computes a query from the program counter, reads the selected address through attention, and computes CPU control signals. The host appends emitted writes to a KV cache and provides memory-mapped input. It executes no instruction semantics. It has 16 four-bit RAM locations, a 32-instruction fixed ROM, a four-bit accumulator, and a finite write budget. The CPU is a truth-table circuit, with exponential scaling; the host constructs cache keys and signals. The module is not a stock Transformer checkpoint. It runs its little monitor but not a general-purpose OS.

Causal event CPU and ROM monitor (beamboot/sequence_cpu.py, beamboot/large_cpu.py, beamboot/os.py). The current monitor runs in a causal attention/MLP circuit with 256 ROM instructions, a four-bit accumulator, and sixteen four-bit RAM locations. An event sequence stores model-emitted write events and host-provided device inputs. The model projects keys and values from those events, reads RAM with causal attention, computes the next instruction state, and emits write/output signals. The host serializes events and executes I/O, not the instruction semantics. Full-sequence causal and single-query inference are both supported, but the runtime recomputes keys and values from the event history on each step instead of using an optimized transformer KV cache. ROM is still compiled into a large truth-table MLP. Its firmware polls an input device, dispatches help/version/add commands, and performs four-bit addition through the neural circuit. This is a bespoke attention module with structured event vectors, not a standard tokenizer, a Hugging Face Transformers checkpoint, or Beam's own architecture. Finite precision and the finite event budget limit it.

Actual Beam attention backend (beamboot/hf_beam_attention.py, beamboot/hf_beam_cpu.py). The earlier monitor backend uses Hugging Face Transformers' real BeamAttention and DynamicCache implementations, with one six-query-head / one-KV-head group of 128-dimensional heads. It uses the full-attention, NoPE path, actual Q/K RMS normalization, position-dependent query scaling, and sigmoid attention gating. Its weights are hand-constructed, not pretrained. The surrounding CPU state machine is still a bespoke MLP and host-serialized event protocol, not the Beam decoder or checkpoint. The runtime incrementally updates the actual KV cache. The monitor boots and runs its commands with this backend; a separate test compares instructions to an independent Python interpreter. The Q normalization gain is constructed specifically for this toy machine and is not taken from Beam. Sliding-window layers and trained model behavior are not tested by this backend.

Two stock decoder layers (beamboot/two_layer_cpu.py). The default monitor backend now runs its CPU through two hand-constructed Hugging Face BeamDecoderLayer modules and a DynamicCache. Layer 0 compiles the ROM's PC-to-address mapping into its SwiGLU weights; layer 1 uses attention to read RAM and its SwiGLU weights to compute the next state and output flags. A test removes the Python ROM after compilation and the machine still executes a load/store program. The host still serializes binary state and write events, extracts model output bits, tracks a write counter, and handles I/O. Weights are custom constructed and the transition MLP is an exponential truth table; this is not a normal language-model token stream, a complete BeamForCausalLM checkpoint, or pretrained Beam. The memory uses explicit host-supplied time fields, full attention, float32 and a finite 128-write budget. It has not been shown to work in sliding-window layers, BF16, or the full checkpoint.

Token-based BeamForCausalLM checkpoint (beamboot/causal_lm.py). The default monitor backend packages the constructed layers in a stock Transformers BeamForCausalLM and uses input_ids plus the standard KV cache. Each generated machine token encodes the next program counter, accumulator, selected address and control flags; the host feeds it back and injects write/device tokens. It does not inspect hidden states or interpret CPU instructions. A checkpoint round-trip test loads the saved weights with the stock Transformers loader and compares execution. This is a two-layer, 128-wide model with a custom machine-token vocabulary and hand-constructed weights, not the pretrained 501B Beam model. Host I/O includes a write counter embedded in write tokens. The model does not autonomously emit memory-write tokens and cannot accept natural-language prompts. It uses full attention only and has a 128-write limit. The monitor is single-task and has 16 four-bit RAM cells; it is not a general-purpose OS.

A prebuilt local checkpoint is in artifacts/beamboot-monitor (about 483 MiB). It is ignored by Git. Rebuild it with OMP_NUM_THREADS=1 PYTHONPATH=. .venv/bin/python scripts/export_checkpoint.py artifacts/beamboot-monitor, or run it without recompiling via OMP_NUM_THREADS=1 .venv/bin/python -m beamboot.os --checkpoint artifacts/beamboot-monitor.

Unified monitor + tiny LM in one model (beamboot/unified.py, tests/test_unified.py). Run OMP_NUM_THREADS=1 PYTHONPATH=. .venv/bin/python -m beamboot.os --backend unified and try lm a 5. One stock BeamForCausalLM (3 layers, hidden 512, 17 heads / 1 KV head, union vocabulary of about 302k tokens, hand-constructed weights) holds the monitor's two CPU layers and the tiny LM's layer. The CPU's 128 residual dims are scattered into dims the tiny LM does not use, norm gains are halved to preserve RMSNorm, and attention gates plus lm_head penalties keep the two token families from interfering. The net's weights are tokens in the same KV cache as the CPU's RAM. The ROM (211 instructions with the extra command) parses lm <char a-h> <n 1-10> in model weights, then the model emits its own 42-token-per-character trace (embed, hidden, logits, attention argmax); the printed characters are those tokens and match the integer reference. The monitor keeps working afterwards. Limits: trace tokens are keyed by iteration and lane rather than time, so a second trace in the same context would read the first one's activations (observed: a wrong second answer); the host therefore crops the KV cache back to its pre-trace length after each lm run. Without the crop, a first trace followed by add still worked in a prototype run, but that was one session only. the host still serializes device input, including the start character token, steps the model token by token and stops after counting n output tokens; it does not do the net's arithmetic. The model does not itself switch modes: the host chooses to feed the start token after the firmware reports LM:. The net is the 8-symbol, int4 cycle network (n is at most 10, the trace vocabulary's length), not a trained language model. Roughly 0.4 s per token on CPU (about 17 s per printed character), about 2.4 GB of float32 weights, float32 and eager attention only, no checkpoint export, and tested only on short sessions.

Beam-style attention probe (beamboot/beam_probe.py). Beam's released configuration has 128-dimensional heads, six query heads per KV head, per-head Q/K RMS normalization, and full-attention layers with position-dependent query scaling. The probe simulates these features for a single GQA group, with a constructed normalization gain and an explicit write-counter key. It is not BeamAttention, doesn't load model weights, and does not prove compatibility. With a gain of 1, even a two-write latest-value check fails; a calibrated gain passes a 128-write float32 seeded test. Full-attention NoPE does not supply our write-counter encoding automatically. Beam also has 512-token sliding attention, so the global-memory design cannot be assumed to work there.

None of the components has been run inside the pretrained 501B Beam model.

Next milestones

1. Remove the host-generated write/clock protocol and make the decoder generate its own complete execution trace. Investigate how to combine the circuit with the actual pretrained Beam checkpoint without changing its language behavior.
2. Test positional encoding and selection at a declared cache length and across precisions, including sliding-window eviction; compare every memory access and instruction against a reference emulator.
3. Replace the exponential truth table with a compositional instruction circuit and enlarge memory and the instruction set enough to run a meaningful operating system or monitor.
4. Integrate a hosted Beam agent as the operator, recording tool calls and preserving the distinction between Beam and the machine. Investigate whether the circuit can actually be embedded in Beam's model architecture and weights; do not claim this based on a standalone model.

Beam has gated grouped-query attention and sliding-window attention, rather than Qwenputer's Gated DeltaNet state. Architectural compatibility, inference cost, and full-checkpoint integration remain open research questions.

Live Beam operator

The demo agent sends commands to the local monitor (using its constructed token-based BeamForCausalLM backend) through hosted Beam tool calls. Its HTTP requests run in a small Modal function using the existing huggingface-secret secret; the token stays in the remote environment. Install Modal into the environment (uv pip install --python .venv/bin/python modal) and run:

OMP_NUM_THREADS=1 .venv/bin/modal run scripts/modal_beam_operator.py

The Baseten provider through Hugging Face Inference Providers is used, and provider/Modal charges may apply. The agent is limited to five chat requests; the monitor has a finite write budget. The actual Beam model operates the constructed machine as an external tool; the machine does not run in the Beam checkpoint.

A live test produced the tool calls version → BEAMBOOT 0.1 and add 7 8 → 15, which Beam reported correctly. The script preserves reasoning_content in multi-turn messages if the provider supplies it. It does not upload the monitor or the result to the Hub.

Reproduce the selected CPU precision sweep with:

OMP_NUM_THREADS=1 PYTHONPATH=. .venv/bin/python scripts/probe_precision.py

For seed 417, the probe produced zero immediate-read errors in float32 and float16 at horizons 128, 512, and 1024. BF16 had 0, 7, and 49 errors, respectively. This is a single seeded test on PyTorch CPU, not a general error rate or a claim about Beam's actual kernels. Values were checked immediately after writing; long-term retention and adversarial patterns need separate tests.

Precision and attention backends

The default checkpoint runs in float32 with eager attention. An experimental bfloat16/SDPA mode loads the same checkpoint at lower precision:

OMP_NUM_THREADS=1 .venv/bin/python -m beamboot.os --checkpoint artifacts/beamboot-monitor --precision bfloat16 --attention sdpa

The runtime rejects bfloat16 with eager attention: it produced an incorrect memory read in a short six-instruction program on the tested CPU. In the same environment, bfloat16 with SDPA produced the correct trace for that program, a separate 51-step seeded program, and the monitor's version and addition commands. This is not proof of correctness for all programs, CPUs, GPUs or cache lengths.

A separate constructed attention probe compares a scalar write counter with a bipolar binary counter. On PyTorch CPU with seed 417 and 1,024 sequential writes, scalar bfloat16/SDPA produced 49 immediate-read errors and binary produced zero. In the actual Transformers BeamAttention eager path, the binary clock still produced 11 errors over 128 writes, while SDPA produced zero for the same seed. The binary-clock probe has not been integrated into the monitor checkpoint. The backend difference is an observed result, not proof of a particular kernel-level cause.

Reproduce the small backend comparison with OMP_NUM_THREADS=1 PYTHONPATH=. .venv/bin/python scripts/precision_report.py; its output is stored in results/precision_smoke.json. Compare the independent clock encodings with OMP_NUM_THREADS=1 PYTHONPATH=. .venv/bin/python scripts/compare_clock_encodings.py. Both are CPU smoke tests of constructed weights, not evaluations of the pretrained Beam model.