Hugging Face's logo

RichardErkhov
/
pankajmathur_-_orca_mini_3b-gguf

GGUF
Inference Endpoints
Model card Files Community
Edit model card
YAML Metadata Warning: empty or missing yaml metadata in repo card (https://huggingface.co/docs/hub/model-cards#model-card-metadata)

Quantization made by Richard Erkhov.

Github

Discord

Request more models

orca_mini_3b - GGUF

Name Quant method Size
orca_mini_3b.Q2_K.gguf Q2_K 1.84GB
orca_mini_3b.IQ3_XS.gguf IQ3_XS 1.84GB
orca_mini_3b.IQ3_S.gguf IQ3_S 1.84GB
orca_mini_3b.Q3_K_S.gguf Q3_K_S 1.84GB
orca_mini_3b.IQ3_M.gguf IQ3_M 1.92GB
orca_mini_3b.Q3_K.gguf Q3_K 1.99GB
orca_mini_3b.Q3_K_M.gguf Q3_K_M 1.99GB
orca_mini_3b.Q3_K_L.gguf Q3_K_L 2.06GB
orca_mini_3b.IQ4_XS.gguf IQ4_XS 1.86GB
orca_mini_3b.Q4_0.gguf Q4_0 1.84GB
orca_mini_3b.IQ4_NL.gguf IQ4_NL 1.86GB
orca_mini_3b.Q4_K_S.gguf Q4_K_S 2.24GB
orca_mini_3b.Q4_K.gguf Q4_K 2.4GB
orca_mini_3b.Q4_K_M.gguf Q4_K_M 2.4GB
orca_mini_3b.Q4_1.gguf Q4_1 2.04GB
orca_mini_3b.Q5_0.gguf Q5_0 2.23GB
orca_mini_3b.Q5_K_S.gguf Q5_K_S 2.42GB
orca_mini_3b.Q5_K.gguf Q5_K 2.57GB
orca_mini_3b.Q5_K_M.gguf Q5_K_M 2.57GB
orca_mini_3b.Q5_1.gguf Q5_1 2.42GB
orca_mini_3b.Q6_K.gguf Q6_K 3.39GB
orca_mini_3b.Q8_0.gguf Q8_0 3.39GB

Original model description:

language: - en license: cc-by-nc-sa-4.0 library_name: transformers datasets: - psmathur/alpaca_orca - psmathur/dolly-v2_orca - psmathur/WizardLM_Orca pipeline_tag: text-generation model-index: - name: orca_mini_3b results: - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 41.55 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/orca_mini_3b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 61.52 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/orca_mini_3b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 26.79 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/orca_mini_3b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 42.42 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/orca_mini_3b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 61.8 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/orca_mini_3b name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 0.08 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=psmathur/orca_mini_3b name: Open LLM Leaderboard

orca_mini_3b

Passionate about Generative AI? I help companies to privately train and deploy custom LLM/MLLM affordably. For startups, I can even assist with securing GPU grants to get you started. Let's chat!

https://www.linkedin.com/in/pankajam Looking forward to connecting!


Use orca-mini-3b for Free on Google Colab with T4 GPU :)

Open In Colab

An OpenLLaMa-3B model model trained on explain tuned datasets, created using Instructions and Input from WizardLM, Alpaca & Dolly-V2 datasets and applying Orca Research Paper dataset construction approaches.

Dataset

We build explain tuned WizardLM dataset ~70K, Alpaca dataset ~52K & Dolly-V2 dataset ~15K created using approaches from Orca Research Paper.

We leverage all of the 15 system instructions provided in Orca Research Paper. to generate custom datasets, in contrast to vanilla instruction tuning approaches used by original datasets.

This helps student model aka this model to learn thought process from teacher model, which is ChatGPT (gpt-3.5-turbo-0301 version).

Please see below example usage how the System prompt is added before each instruction.

Training

The training configurations are provided in the table below.

The training takes on 8x A100(80G) GPUs and lasts for around 4 Hours for cost of $48 using Lambda Labs

We used DeepSpeed with fully sharded data parallelism, also know as ZeRO stage 3 by writing our own fine tunning scripts plus leveraging some of the model training code provided by amazing OpenAlpaca repo

Here are some of params used during training:

batch_size 64
train_micro_batch_size_per_gpu 4
gradient_accumulation_steps 2
Learning rate 2e-5
Max length 1024
Epochs 3
Optimizer AdamW

Example Usage

Below shows an example on how to use this model

import torch
from transformers import LlamaForCausalLM, LlamaTokenizer

# Hugging Face model_path
model_path = 'psmathur/orca_mini_3b'
tokenizer = LlamaTokenizer.from_pretrained(model_path)
model = LlamaForCausalLM.from_pretrained(
    model_path, torch_dtype=torch.float16, device_map='auto',
)


#generate text function
def generate_text(system, instruction, input=None):
    
    if input:
        prompt = f"### System:\n{system}\n\n### User:\n{instruction}\n\n### Input:\n{input}\n\n### Response:\n"
    else:
        prompt = f"### System:\n{system}\n\n### User:\n{instruction}\n\n### Response:\n"
    
    tokens = tokenizer.encode(prompt)
    tokens = torch.LongTensor(tokens).unsqueeze(0)
    tokens = tokens.to('cuda')

    instance = {'input_ids': tokens,'top_p': 1.0, 'temperature':0.7, 'generate_len': 1024, 'top_k': 50}

    length = len(tokens[0])
    with torch.no_grad():
        rest = model.generate(
            input_ids=tokens, 
            max_length=length+instance['generate_len'], 
            use_cache=True, 
            do_sample=True, 
            top_p=instance['top_p'],
            temperature=instance['temperature'],
            top_k=instance['top_k']
        )    
    output = rest[0][length:]
    string = tokenizer.decode(output, skip_special_tokens=True)
    return f'[!] Response: {string}'

# Sample Test Instruction Used by Youtuber Sam Witteveen https://www.youtube.com/@samwitteveenai
system = 'You are an AI assistant that follows instruction extremely well. Help as much as you can.'
instruction = 'Write a letter to Sam Altman, CEO of OpenAI, requesting him to convert GPT4 a private model by OpenAI to an open source project'
print(generate_text(system, instruction))


[!] Response:
Dear Sam Altman,

I am writing to request that you convert the GPT4 private model developed by OpenAI to an open source project. As a user of OpenAI, I have been waiting for the day when I can use the advanced natural language processing capabilities of GPT4 in a more open and accessible way.

While OpenAI has made significant progress in developing AI applications, it has primarily focused on building private models that are not accessible to the general public. However, with the recent release of GPT-3, there is a growing demand for more open and accessible AI tools.

Converting GPT4 to an open source project would allow for greater transparency, collaboration, and innovation. It would also help to build trust in the technology and ensure that it is used ethically and responsibly.

I urge you to consider converting GPT4 to an open source project. This would be a significant contribution to the AI community and would help to create a more open and accessible future.

Thank you for your consideration.

Sincerely,

[Your Name]

Next Goals:

  1. Try more data like actually using FLAN-v2, just like Orka Research Paper (I am open for suggestions)
  2. Provide more options for Text generation UI. (may be https://github.com/oobabooga/text-generation-webui)
  3. Provide 4bit GGML/GPTQ quantized model (may be TheBloke can help here)

Limitations & Biases:

This model can produce factually incorrect output, and should not be relied on to produce factually accurate information. This model was trained on various public datasets. While great efforts have been taken to clean the pretraining data, it is possible that this model could generate lewd, biased or otherwise offensive outputs.

Disclaimer:

The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please cosult an attorney before using this model for commercial purposes.

Citiation:

If you found wizardlm_alpaca_dolly_orca_open_llama_3b useful in your research or applications, please kindly cite using the following BibTeX:

@misc{orca_mini_3b,
  author = {Pankaj Mathur},
  title = {wizardlm_alpaca_dolly_orca_open_llama_3b: An explain tuned OpenLLaMA-3b model on custom wizardlm, alpaca, & dolly datasets},
  year = {2023},
  publisher = {GitHub, HuggingFace},
  journal = {GitHub repository, HuggingFace repository},
  howpublished = {\url{https://github.com/pankajarm/wizardlm_alpaca_dolly_orca_open_llama_3b}, \url{https://https://huggingface.co/psmathur/wizardlm_alpaca_dolly_orca_open_llama_3b}},
}

@misc{mukherjee2023orca,
      title={Orca: Progressive Learning from Complex Explanation Traces of GPT-4}, 
      author={Subhabrata Mukherjee and Arindam Mitra and Ganesh Jawahar and Sahaj Agarwal and Hamid Palangi and Ahmed Awadallah},
      year={2023},
      eprint={2306.02707},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

@software{openlm2023openllama,
  author = {Xinyang Geng and Hao Liu},
  title = {OpenLLaMA: An Open Reproduction of LLaMA},
  month = May,
  year = 2023,
  url = {https://github.com/openlm-research/open_llama}
}

@misc{openalpaca,
  author = {Yixuan Su and Tian Lan and Deng Cai},
  title = {OpenAlpaca: A Fully Open-Source Instruction-Following Model Based On OpenLLaMA},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/yxuansu/OpenAlpaca}},
}

@misc{alpaca,
  author = {Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {Stanford Alpaca: An Instruction-following LLaMA model},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/stanford_alpaca}},
}

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 35.5
ARC (25-shot) 41.55
HellaSwag (10-shot) 61.52
MMLU (5-shot) 26.79
TruthfulQA (0-shot) 42.42
Winogrande (5-shot) 61.8
GSM8K (5-shot) 0.08
DROP (3-shot) 14.33

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric Value
Avg. 39.03
AI2 Reasoning Challenge (25-Shot) 41.55
HellaSwag (10-Shot) 61.52
MMLU (5-Shot) 26.79
TruthfulQA (0-shot) 42.42
Winogrande (5-shot) 61.80
GSM8k (5-shot) 0.08
Downloads last month
108
GGUF
Model size
3.43B params
Architecture
llama

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

View +2 files
Inference API
Unable to determine this model's library. Check the docs .