anon8231489123
/

gpt4-x-alpaca-13b-native-4bit-128g

Text Generation

text-generation-inference

Model card Files Files and versions

Instructions to use anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g")
model = AutoModelForCausalLM.from_pretrained("anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g")

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g

How to use anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g with Docker Model Runner:
```
docker model run hf.co/anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g
```

Resources

View closed (4)

OSError: anon8231489123/gpt4-x-alpaca-13b-native-4bit-128g does not appear to have a file named pytorch_model-00001-of-00006.bin.

#55 opened about 2 years ago by

Model cant be loaded

#54 opened over 2 years ago by

"RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!"

#53 opened almost 3 years ago by

Fine tuning "gpt4-x-alpaca-13b-native-4bit-128g".

#52 opened almost 3 years ago by

Setup on MBP

#51 opened almost 3 years ago by

IndexError: list index out of range

#50 opened almost 3 years ago by

The model weights are not tied and json-files are different from the original LlamaTokenizer file?

#49 opened almost 3 years ago by deleted

error loading model: unexpectedly reached end of file

#48 opened almost 3 years ago by

Error: failed to load model 'ggml-model-q4_1.bin'

#47 opened almost 3 years ago by

Any Constructive help is always welcome. :/

#46 opened about 3 years ago by

Requantize to support latest code on llama.cpp

#45 opened about 3 years ago by

你好

#44 opened about 3 years ago by

Where is the info on how to format the prompt?

#43 opened about 3 years ago by

Out of memory? somehow

#42 opened about 3 years ago by

error, says this model is not found?

#41 opened about 3 years ago by

Running out of memory with 12GB of VRAM on 3080TI

#39 opened about 3 years ago by

Issues running the model in python

#38 opened about 3 years ago by

How I got this to run with oobabooga/ text-generation-webui

#37 opened about 3 years ago by

Colab for finetuning

#36 opened about 3 years ago by

RuntimeError: [enforce fail at C:\cb\pytorch_1000000000000\work\c10\core\impl\alloc_cpu.cpp:72] data. DefaultCPUAllocator: not enough memory: you tried to allocate 1048576 bytes.

#35 opened about 3 years ago by

gradio error?

#34 opened about 3 years ago by

Gpt4-x-alpaca gives gibberish numbers instead of words

#33 opened about 3 years ago by

RTX 3070, only getting about 0,38 tokens/minute

#32 opened about 3 years ago by

No response from Abaca when using GPU version

#31 opened about 3 years ago by

For those who complain about that its Censored

#30 opened about 3 years ago by

Running an rtx3060 with 12GBvram - managed to get this model working on method in link in description

#28 opened about 3 years ago by

RuntimeError: Internal: D:\a\sentencepiece\sentencepiece\src\sentencepiece_processor.cc(1102) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

#27 opened about 3 years ago by deleted

for people with low Mid pc specs

#26 opened about 3 years ago by

Request for Colab Version

#25 opened about 3 years ago by

Emoji overload

#24 opened about 3 years ago by

ValueError: Tokenizer class LlamaTokenizer does not exist or is not currently imported. (while using Transformers)

#23 opened about 3 years ago by

Temporal fix for "DefaultCPUAIIocator: not enough memory: you tried to allocate 13107200 bytes" error

#22 opened about 3 years ago by

Issue with tokenizor using Ooga Booga?

#21 opened about 3 years ago by

Stuck at "Filtering content: 40% (2/5)" when cloning repository

#20 opened about 3 years ago by

I'm trying to run this using oobabooga but I'm getting 0.17 tokens/second.

#18 opened about 3 years ago by

LLAMA running slow with this model.

#17 opened about 3 years ago by

Error loading gpt4-x-alpaca-13b-native-4bit-128g on Alienware M15 Ryzen Edition R5 laptop

#16 opened about 3 years ago by

out of memory error when launching from oobabooga web ui

#15 opened about 3 years ago by

torch.cuda.OutOfMemoryError

#14 opened about 3 years ago by

Only small amount of tokens generated in oobabooga

#13 opened about 3 years ago by

Is Vicuna involved by any means?

#12 opened about 3 years ago by

Does this model able to run on 3060 12g?

#11 opened about 3 years ago by

CUDA out of memory

#10 opened about 3 years ago by

Cuda error with ooba-booga WebUI

#9 opened about 3 years ago by

Hmmm.... Problem with another language...

#8 opened about 3 years ago by

Safetensors version?

#7 opened about 3 years ago by

Error using ooba-gooba

#6 opened about 3 years ago by

7B model thanks?

#4 opened about 3 years ago by

How can i use this model with GPTQ-for-LLaMa?

#3 opened about 3 years ago by

Error for LlamaForCausalLM.from_pretrained in HuggingFace

#2 opened about 3 years ago by