Text Generation
Transformers
Safetensors
PyTorch
mistral
Safetensors
text-generation-inference
Merge
7b
mistralai/Mistral-7B-Instruct-v0.2
HuggingFaceH4/zephyr-7b-beta
Generated from Trainer
en
dataset:HuggingFaceH4/ultrachat_200k
dataset:HuggingFaceH4/ultrafeedback_binarized
arxiv:2305.18290
arxiv:2310.16944
Eval Results
Inference Endpoints
has_space
conversational
| license: apache-2.0 | |
| tags: | |
| - Safetensors | |
| - text-generation-inference | |
| - merge | |
| - mistral | |
| - 7b | |
| - mistralai/Mistral-7B-Instruct-v0.2 | |
| - HuggingFaceH4/zephyr-7b-beta | |
| - transformers | |
| - pytorch | |
| - safetensors | |
| - mistral | |
| - text-generation | |
| - generated_from_trainer | |
| - en | |
| - dataset:HuggingFaceH4/ultrachat_200k | |
| - dataset:HuggingFaceH4/ultrafeedback_binarized | |
| - arxiv:2305.18290 | |
| - arxiv:2310.16944 | |
| - base_model:mistralai/Mistral-7B-v0.1 | |
| - license:mit | |
| - model-index | |
| - autotrain_compatible | |
| - endpoints_compatible | |
| - has_space | |
| - text-generation-inference | |
| - region:us | |
| # zephyr-7b-beta-Mistral-7B-Instruct-v0.2 | |
| zephyr-7b-beta-Mistral-7B-Instruct-v0.2 is a merge of the following models: | |
| * [mistralai/Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) | |
| * [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) | |
| ## Repositories available | |
| * [GPTQ models for GPU inference, with multiple quantisation parameter options.](https://huggingface.co/MaziyarPanahi/zephyr-7b-beta-Mistral-7B-Instruct-v0.2-GPTQ) | |
| * [2, 3, 4, 5, 6 and 8-bit GGUF models for CPU+GPU inference](https://huggingface.co/MaziyarPanahi/zephyr-7b-beta-Mistral-7B-Instruct-v0.2-GGUF) | |
| ## 🧩 Configuration | |
| ```yaml | |
| slices: | |
| - sources: | |
| - model: mistralai/Mistral-7B-Instruct-v0.2 | |
| layer_range: [0, 32] | |
| - model: HuggingFaceH4/zephyr-7b-beta | |
| layer_range: [0, 32] | |
| merge_method: slerp | |
| base_model: mistralai/Mistral-7B-Instruct-v0.2 | |
| parameters: | |
| t: | |
| - filter: self_attn | |
| value: [0, 0.5, 0.3, 0.7, 1] | |
| - filter: mlp | |
| value: [1, 0.5, 0.7, 0.3, 0] | |
| - value: 0.5 | |
| dtype: bfloat16 | |
| ``` | |
| ## 💻 Usage | |
| ```python | |
| !pip install -qU transformers accelerate | |
| from transformers import AutoTokenizer | |
| import transformers | |
| import torch | |
| model = "MaziyarPanahi/zephyr-7b-beta-Mistral-7B-Instruct-v0.2" | |
| messages = [{"role": "user", "content": "What is a large language model?"}] | |
| tokenizer = AutoTokenizer.from_pretrained(model) | |
| prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) | |
| pipeline = transformers.pipeline( | |
| "text-generation", | |
| model=model, | |
| torch_dtype=torch.float16, | |
| device_map="auto", | |
| ) | |
| outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95) | |
| print(outputs[0]["generated_text"]) | |
| ``` |