LlaMa 2 13b 4-bit Chain of Thought Reasoning π©βπ»
LlaMa-2 13b fine-tuned on the kaist-ai/CoT-Collection dataset by using the method QLoRA in 4-bit with PEFT library.
Pretrained description
Meta developed and publicly released the Llama 2 family of large language models (LLMs), a collection of pretrained and fine-tuned generative text models ranging in scale from 7 billion to 70 billion parameters.
Model Architecture Llama 2 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align to human preferences for helpfulness and safety
Training data
Qunatization Configuration
The following bitsandbytes
quantization config was used during training:
- bits: 4
- group_size: 128
- dataset: "c4"
- desc_act: False
- tokenizer:tokeniaer
- device_map: "auto"
Framework versions
- PEFT 0.4.0
Training
Downloading (β¦)okenizer_config.json: 100%
725/725 [00:00<00:00, 118kB/s]
Downloading (β¦)/main/tokenizer.json: 100%
1.84M/1.84M [00:01<00:00, 1.45MB/s]
Downloading (β¦)cial_tokens_map.json: 100%
437/437 [00:00<00:00, 35.9kB/s]
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py:472: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.
warnings.warn(
Downloading (β¦)lve/main/config.json: 100%
631/631 [00:00<00:00, 53.8kB/s]
Downloading (β¦)/adapter_config.json: 100%
452/452 [00:00<00:00, 39.1kB/s]
Downloading (β¦)lve/main/config.json: 100%
587/587 [00:00<00:00, 50.1kB/s]
Downloading (β¦)fetensors.index.json: 100%
33.4k/33.4k [00:00<00:00, 2.97MB/s]
Downloading shards: 100%
3/3 [28:42<00:00, 546.82s/it]
Downloading (β¦)of-00003.safetensors: 100%
9.95G/9.95G [10:35<00:00, 15.3MB/s]
Downloading (β¦)of-00003.safetensors: 100%
9.90G/9.90G [11:04<00:00, 15.9MB/s]
Downloading (β¦)of-00003.safetensors: 100%
6.18G/6.18G [06:56<00:00, 14.5MB/s]
Loading checkpoint shards: 100%
3/3 [00:03<00:00, 1.01s/it]
/usr/local/lib/python3.10/dist-packages/transformers/utils/hub.py:374: FutureWarning: The `use_auth_token` argument is deprecated and will be removed in v5 of Transformers.
warnings.warn(
Downloading (β¦)neration_config.json: 100%
188/188 [00:00<00:00, 16.5kB/s]
Downloading readme: 100%
2.38k/2.38k [00:00<00:00, 162kB/s]
Repo card metadata block was not found. Setting CardData to empty.
Downloading data files: 100%
1/1 [00:42<00:00, 42.25s/it]
Downloading data: 100%
319M/319M [00:42<00:00, 6.57MB/s]
Extracting data files: 100%
1/1 [00:04<00:00, 4.02s/it]
Generating train split:
356317/0 [00:01<00:00, 224199.18 examples/s]
Quantizing model.layers blocks : 100%
40/40 [32:31<00:00, 50.21s/it]
CUDA extension not installed.
Downloading adapter_model.bin: 100%
26.3M/26.3M [00:03<00:00, 8.64MB/s]
Example of usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "aiplanet/effi-13b-int4-GPTQ"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")
tst = """Read the Instruction below and provide an answer the question asked.Stick to to theinstruction .Do not repeat the answers.
### INSTRUCTION:
Virgin Australia, the trading name of Virgin Australia Airlines Pty Ltd, is an Australian-based airline. It is the largest airline by fleet size to use the Virgin brand. It commenced services on 31 August 2000 as Virgin Blue, with two aircraft on a single route. It suddenly found itself as a major airline in Australia's domestic market after the collapse of Ansett Australia in September 2001. The airline has since grown to directly serve 32 cities in Australia, from hubs in Brisbane, Melbourne and Sydney.Is Virgin Australia and Virgin Blue the same airlines?
"""
#
prompt = f"[INST] <<SYS>>\n{system_message}\n<</SYS>>\n\n{tst}. [/INST]"
#
# Tokenize the input
input_ids = tokenizer(prompt, return_tensors="pt", truncation=True).input_ids.cuda()
# Run the model to infere an output
outputs = model.generate(input_ids=input_ids, max_new_tokens=100, do_sample=True, top_p=0.9,temperature=0.1)
# Print the result
print(f"Prompt:\n{prompt}\n")
print(f"Generated instruction:\n{tokenizer.batch_decode(outputs.detach().cpu().numpy(), skip_special_tokens=True)[0][len(prompt):].split(' [/INST]')[0]}")
Citation
@misc {Plaban81,
author = { {Plaban Nayak} },
title = { Quantized version of effi-13b by AI Planet},
year = 2023,
url = { https://huggingface.co/aiplanet/effi-13b },
publisher = { Hugging Face }
}
- Downloads last month
- 18
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.