README.md

---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- moe
- moah
- mod
datasets:
- Locutusque/UltraTextbooks
---

# Model Card for Model ID

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

MoM: Mixture of Mixture

This Model is a first test to combine [Jamba](https://huggingface.co/ai21labs/Jamba-v0.1) architecture with bf16 bits linear layers, mixture of attention head and mixture of depth.

The goal is to developpe and test if this kind of architectures have not too much quality loss for a fast inference.


- **Model type:** Mixture of attention head mixture of depth and mixture of expert bf16 linear layers 
- **License:** Apache licence 2.0

### Model Sources [optional]


- **Repository:** https://github.com/ostix360/optimized-LLM


## How to Get Started with the Model

This model has a generation problem because of a softmax application in the mod process


If you want to test  this model please look at this repo at this [commit](https://github.com/ostix360/optimized-LLM/tree/796cfe43cf16461b92102cf0f41e8960cd91340b)


## Training Details

  - **wandb**: [training detail](https://wandb.ai/ostix360/Mixture%20of%20mixture%20(mod,%20moah%20moe)/runs/6mpcy0ck)

### Training Data

We use the first ~0.5B tokens of Locutusque/UltraTextbooks to train this model

### Training Procedure

We use adam-8 bits with default betas and epsilon values

#### Preprocessing [optional]


The data fit the model max length i.e. 512 tokens


#### Training Hyperparameters

Please look at the wandb metadata to see the hyperparameters or the train.py file in the repo


## Technical Specifications 

### Compute Infrastructure

#### Hardware

- one 4070 ti GPU 

#### Software

- pytorch, transformers etc