README.md

---
language:
- en
license: apache-2.0
library_name: transformers
tags:
- moe
- moah
- mod
- mh-moe
datasets:
- Locutusque/UltraTextbooks
---

# Model Card for Model ID

## Model Details

### Model Description

<!-- Provide a longer summary of what this model is. -->

MoM: Mixture of Mixture

This Model is a first test to combine [Jamba](https://huggingface.co/ai21labs/Jamba-v0.1) architecture with bf16 bits linear layers, mixture of attention head and **multi head** mixture of depth.

The goal is to developpe and test if this kind of architectures have not too much quality loss for a fast inference.


- **Model type:** Mixture of attention head mixture of depth and mixture of expert bf16 linear layers 
- **License:** Apache licence 2.0

### Model Sources [optional]


- **Repository:** https://github.com/ostix360/optimized-LLM


## How to Get Started with the Model

This model has a generation problem because of a softmax application in the mod process


If you want to test  this model please look at this repo at this [commit](https://github.com/ostix360/optimized-LLM/tree/1f937b3c35074c9eb48ccde52677bb0439f71960)


## Training Details

  - **wandb**: [training detail](https://wandb.ai/ostix360/Mixture%20of%20mixture%20(mod,%20moah%20moe)/runs/ygwwa30r)

### Training Data

We use the first ~0.5B tokens of Locutusque/UltraTextbooks to train this model

### Training Procedure

We use adam-8 bits with default betas and epsilon values

#### Preprocessing [optional]


The data fit the model max length i.e. 512 tokens


#### Training Hyperparameters

Please look at the wandb metadata to see the hyperparameters or the train.py file in the repo


## Technical Specifications 

### Compute Infrastructure

#### Hardware

- one 4070 ti GPU 

#### Software

- pytorch, transformers etc