Ehraim/mixture-of-tiny-sliding-experts

Mixture of Tiny Sliding Experts

One frozen public checkpoint is loaded at a time on a 24 GB GPU. A hash router picks it. Optional chains pass text from one expert to the next. A bag of those chains votes on the final string. A separate 128-dimensional cache, shared only by the routers, can slide along the chain.

This repository is an assembly of published methods. It does not introduce a new routing, memory, or decoding algorithm. The bibliography at the bottom is the full set of ideas used in the code. Upstream weight files keep their own licenses and are not redistributed here.

What the system does

A request is hashed into a bag of word ids. A small classifier scores 138 expert ids. Confidence under 0.45 selects the int4 14B coder. An image cue loads a diffusion checkpoint and writes a PNG. Otherwise the chosen expert prefills a text memory into its own private cache and decodes greedily.

chain runs up to three hops. The hop router depends on the previous kind: after a diffusion, vLLM, LLM, or classifier step. Named use cases fix the kinds: diagram, label-then-explain, draft-then-illustrate, and match-then-redraw. A cascade repeats image-then-caption. A bag draws experts from the router distribution and majority-votes exact terminal strings.

The public checkpoints stay frozen. Training covers the router, the cache slot policy, and the slide.

Why this exists

A 24 GB laptop can run one 7B checkpoint. It cannot keep dozens of specialists resident at once. The useful split is which model to load, not a new layer inside one model. The router makes that choice from the request text. Chains exist because some requests need two kinds of model in order: draw, then read the drawing; label, then explain the label. Each hop writes text (or a PNG path) into a shared note the next expert can prefill. That note is ordinary text. Heterogeneous checkpoints do not share a native key-value cache, so the slide cache is a separate 128-wide ring used only by the routers.

The slide stores where a layer sits on the chain, how far it moved, and what it wrote there. A later hop can land on an earlier model's slot or shift one step further down the chain. The loss is the router's expert label, and it flows back through that shift. PPO uses the same reward as REINFORCE, with a learned value and a clipped policy ratio (Schulman et al., 2017). Neither update changes the frozen expert. The expert still does the generation.

python -m mote.train_router
python -m mote.train_chain_routers
python -m mote --map
python -m mote "fix the pytest" --root .
python -m pytest tests -q

Four pools

The downloadable specialists are nested: 6, then 18, then 24, then 36. Each larger pool contains the smaller ones. Every pool has text, an image generator, and a classifier. From 18 upward, each pool also has speech and OCR. The list and the card text for each cut are in mote/variants.py. A card for the 6, 18, or 24 cut still names the checkpoints that appear only in the full 36, because the router and the stacked cache are the same system.

Where each idea is used

Piece in this repo	Published idea
Router over specialist checkpoints	Mixture of experts (Jacobs et al., 1991; Shazeer et al., 2017; Fedus et al., 2022)
Word-hash features into the router	Feature hashing (Weinberger et al., 2009)
Text handoff from one model to the next, including image then language	Socratic Models (Zeng et al., 2022); HuggingGPT (Shen et al., 2023)
Cascade that picks a later model from an earlier result	FrugalGPT (Chen et al., 2024)
Several drafts, then a vote on the terminal string	Bagging (Breiman, 1996); self-consistency (Wang et al., 2023); Mixture-of-Agents (Wang et al., 2024)
vLLM as a serving runtime, not a second weight file	PagedAttention (Kwon et al., 2023)
External memory the experts do not share as native key-value state	Memory Networks (Weston et al., 2015); Neural Turing Machines (Graves et al., 2014)
Stacked reads, each layer with its own weights, trained by backpropagation	End-to-end memory networks (Sukhbaatar et al., 2015)
Residual write into a memory slot	Residual networks (He et al., 2016); Neural Turing Machines (Graves et al., 2014)
Attention mix over the stacked reads	Transformers (Vaswani et al., 2017)
REINFORCE chooses the slot; backpropagation writes the content	REINFORCE (Williams, 1992); RL-NTM (Zaremba and Sutskever, 2015)
Clipped slot-policy update with a learned value	Proximal policy optimization (Schulman et al., 2017)
Learned gate across stacked memory hops	Gated end-to-end memory networks (Liu and Perez, 2017)
A learned shift that moves one layer's cache onto another chain position	Deformable convolution (Dai et al., 2017); shifted windows (Liu et al., 2021); soft expert slots (Puigcerver et al., 2024)

Two papers also use the name Chain-of-Experts and are not this system. Xiao et al. (2024) assign roles to separate LLM agents for operations-research problems. Wang et al. (2025) pass a token through experts inside one MoE layer. This repo chains separate checkpoints by text, outside the layer.

Finetuned weights

Training in this project produced six checkpoints. The public specialist models were not finetuned here, and their weight files are not in this repo. At decode time the loader fetches the chosen specialist from its original Hub id.

File	Trained object
checkpoints/router.pt	Entry router
checkpoints/routers/after-diffusion.pt	Hop router after an image
checkpoints/routers/after-vllm.pt	Hop router after a vLLM step
checkpoints/routers/after-llm.pt	Hop router after a text model
checkpoints/routers/after-classifier.pt	Hop router after a classifier
checkpoints/exkv.pt	Stacked external cache: slot policy, value head, and slide

Decoder

Decoding is greedy token generation from the one resident expert, or a diffusion sampler when the hop is an image model. The shared slide cache is added to the router. It is not the expert's past_key_values. The resident expert builds that private cache by prefilling the rendered text memory.

Clone this repo so checkpoints/ sits next to the mote package. Route only, with no expert download:

python -m mote "fix the pytest" --root .

Route, load the one selected checkpoint, and decode:

python -m mote "fix the pytest" --root . --generate

FileAgent loads checkpoints/router.pt, checkpoints/exkv.pt, and checkpoints/routers/after-*.pt on startup. Step-by-step notes are in docs/DECODER.md.

Tests

python -m pytest tests -q

tests/test_exkv.py checks residual cache writes, router injection, REINFORCE, stack gradients, and the slide from model A onto B or C. tests/test_chain.py, tests/test_bag.py, tests/test_cascade.py, and tests/test_usecases.py check the text handoff, the vote, the image cascade, and the named use cases. tests/test_experts.py checks that cue strings still route to the intended expert id. These tests do not download the 7B checkpoints.

Bibliography

Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87, 1991.

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations, 2017. arXiv:1701.06538.

Fedus, W., Zoph, B., and Shazeer, N. Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1–39, 2022. arXiv:2101.03961.

Weinberger, K., Dasgupta, A., Langford, J., Smola, A., and Attenberg, J. Feature hashing for large scale multitask learning. In International Conference on Machine Learning, 2009.

Zeng, A., Attarian, M., Ichter, B., Choromanski, K., Wong, A., Welker, S., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., and Florence, P. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv:2204.00598, 2022.

Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. HuggingGPT: Solving AI tasks with ChatGPT and its friends in Hugging Face. In Advances in Neural Information Processing Systems, 2023. arXiv:2303.17580.

Chen, L., Zaharia, M., and Zou, J. FrugalGPT: How to use large language models while reducing cost and improving performance. Transactions on Machine Learning Research, 2024. arXiv:2305.05176.

Breiman, L. Bagging predictors. Machine Learning, 24(2):123–140, 1996.

Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In International Conference on Learning Representations, 2023. arXiv:2203.11171.

Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., and Zou, J. Mixture-of-Agents enhances large language model capabilities. arXiv:2406.04692, 2024.

Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Symposium on Operating Systems Principles, 2023.

Weston, J., Chopra, S., and Bordes, A. Memory networks. In International Conference on Learning Representations, 2015. arXiv:1410.3916.

Graves, A., Wayne, G., and Danihelka, I. Neural Turing machines. arXiv:1410.5401, 2014.

Sukhbaatar, S., Szlam, A., Weston, J., and Fergus, R. End-to-end memory networks. In Advances in Neural Information Processing Systems, 2015. arXiv:1503.08895.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.

Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256, 1992.

Zaremba, W. and Sutskever, I. Reinforcement learning neural Turing machines. arXiv:1505.00521, 2015.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv:1707.06347, 2017.

Liu, F. and Perez, J. Gated end-to-end memory networks. In European Chapter of the Association for Computational Linguistics, 2017.

Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y. Deformable convolutional networks. In International Conference on Computer Vision, 2017.

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin Transformer: Hierarchical vision transformer using shifted windows. In International Conference on Computer Vision, 2021.

Puigcerver, J., Riquelme, C., Mustafa, B., and Houlsby, N. From sparse to soft mixtures of experts. In International Conference on Learning Representations, 2024. arXiv:2308.00951.

Xiao, Z., Zhang, D., Wu, Y., Xu, L., Wang, Y., Han, X., Fu, X., Zhong, T., Zeng, J., Song, M., and Chen, G. Chain-of-Experts: When LLMs meet complex operations research problems. In International Conference on Learning Representations, 2024.

Wang, Z., Pan, R., Yao, J., Csordás, R., Li, L., Yin, L., Wu, J., Zhang, T., Li, M., and Liu, S. Chain-of-Experts: Unlocking the communication power of mixture-of-experts models. arXiv:2506.18945, 2025.

BibTeX for the same list is in CITATIONS.bib.