Noorhan/Chatbot_Education_2

🌟 Mistral-7B-Instruct GGUF - Quantized Model

This repository hosts a quantized version of Mistralai/Mistral-7B-Instruct-v0.3 in GGUF format, optimized for efficient inference using llama.cpp or compatible libraries.

📌 Model Details

• Base Model: Mistral-7B-Instruct-v0.3
• Format: GGUF (mistral-7b-instruct-v0.3.gguf)
• Quantization: Q8_0
• Size: ~7.7GB

🚀 How to Use

You can use this model with llama.cpp, ctransformers, or text-generation-webui.

Using llama-cpp-python

from llama_cpp import Llama

model_path = "mistral-7b-instruct-v0.3.gguf"  # Path to downloaded model
llm = Llama(model_path=model_path, n_ctx=2048)

response = llm("What is Retrieval-Augmented Generation?")
print(response)

Using ctransformers

from ctransformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Noorhan/Chatbot_Education_2",
    model_file="mistral-7b-instruct-v0.3.gguf",
    model_type="mistral"
)

response = model("Explain the importance of RAG for AI-powered chatbots.")
print(response)

🛠 Running in a Hugging Face Space

To deploy this model in a Hugging Face Space, ensure your app.py script loads the GGUF model using llama-cpp and provides an API for inference.

Example Space setup:

• requirements.txt

llama-cpp-python
streamlit

• app.py

import streamlit as st
from llama_cpp import Llama

st.title("🔹 Mistral-7B-Instruct Chatbot")

model_path = "mistral-7b-instruct-v0.3.gguf"
llm = Llama(model_path=model_path, n_ctx=2048)

query = st.text_input("Ask a question:")
if query:
    response = llm(query)
    st.write(response["choices"][0]["text"])

📜 License & Usage

This model is a quantized version of Mistralai/Mistral-7B-Instruct-v0.3. Make sure to follow Mistral AI's license when using this model for deployment or research.

🔗 Links

• Mistral-7B-Instruct Base Model
• Hugging Face Spaces
• llama.cpp GitHub
• ctransformers Docs

---

✅ If you find this model useful, give it a ⭐ on Hugging Face!