---
datasets:
- c-s-ale/alpaca-gpt4-data
- Open-Orca/OpenOrca
- Intel/orca_dpo_pairs
- allenai/ultrafeedback_binarized_cleaned
- HuggingFaceH4/no_robots
license: cc-by-nc-4.0
language:
  - en
library_name: ExLlamaV2
pipeline_tag: text-generation
tags:
  - Mistral
  - SOLAR
  - Quantized Model
  - exl2
base_model:
  - rishiraj/meow
---

# exl2 quants for meow

This repository includes the quantized models for the [meow](https://huggingface.co/rishiraj/meow) model by [Rishiraj Acharya](https://huggingface.co/rishiraj). meow is a fine-tune of [SOLAR-10.7B-Instruct-v1.0](https://huggingface.co/upstage/SOLAR-10.7B-Instruct-v1.0) with the [no_robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots) dataset.

## Current models

| exl2 BPW | Model Branch | Model Size | Minimum VRAM (4096 Context, fp16 cache) |
|-|-|-|-|
| 2-Bit | main | 3.28 GB | 6GB GPU |
| 4-Bit | 4bit | 5.61 GB | 8GB GPU |
| 5-Bit | 5bit | 6.92 GB | 10GB GPU, 8GB with swap |
| 6-Bit | 6bit | 8.23 GB | 10GB GPU |
| 8-Bit | 8bit | 10.84 GB | 12GB GPU |

### Note

Using a 12GB Nvidia GeForce RTX 3060 I got on average around 20 tokens per second on the 8-bit quant in full 4096 context.

## Where to use

There are a couple places you can use an exl2 model, here are a few:

- [tabbyAPI](https://github.com/theroyallab/tabbyAPI)
- [Aphrodite Engine](https://github.com/PygmalionAI/aphrodite-engine)
- [ExUI](https://github.com/turboderp/exui)
- [oobabooga's Text Gen Webui](https://github.com/oobabooga/text-generation-webui)
  - When using the downloader, make sure to format like this: Anthonyg5005/rishiraj-meow-10.7B-exl2**\:QuantBranch**
- [KoboldAI](https://github.com/henk717/KoboldAI) (Clone repo, don't use snapshot)

# How to download:

### oobabooga's downloader

use something like [download-model.py](https://github.com/oobabooga/text-generation-webui/blob/main/download-model.py) to download with python requests.\
Install requirements:

```shell
pip install requests tqdm
```

Example for downloading 5bpw:

```shell
python download-model.py Anthonyg5005/rishiraj-meow-10.7B-exl2:5bit
```

### huggingface-cli

You may also use huggingface-cli\
To install it, install python hf-hub

```shell
pip install huggingface-hub
```

Example for 5bpw:

```shell
huggingface-cli download Anthonyg5005/rishiraj-meow-10.7B-exl2 --local-dir rishiraj-meow-10.7B-exl2-5bpw --revision 5bit
```
### Git LFS (not recommended)

I would recommend the http downloaders over using git, they can resume downloads if failed and are much easier to work with.\
Make sure to have git and git LFS installed.\
Example for 5bpw download with git:

Have LFS file skip disabled
```shell
# windows
set GIT_LFS_SKIP_SMUDGE=0
# linux
export GIT_LFS_SKIP_SMUDGE=0
```

Clone repo branch
```shell
git clone https://huggingface.co/Anthonyg5005/rishiraj-meow-10.7B-exl2 -b 5bit
```