Model Card for Model ID

Model Details

MoM: Mixture of Mixture

This Model is a first test to combine Jamba architecture with 1.58 bits linear layers, mixture of attention head and mixture of depth.

The goal is to developpe and test if this kind of architectures have not too much quality loss for a fast inference.

Model type: Mixture of attention head mixture of depth and mixture of expert 1.58bit linear layers
License: Apache licence 2.0

If you want to test this model please look at this repo at this commit

We use the first 100k data of Locutusque/UltraTextbooks to train this model

We use adam-8 bits with default betas and epsilon values

The data fit the model max length i.e. 512 tokens

Please look at this file to see the hyperparameters