Instructions to use PhilLovesCats/LLaMA-1-65B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use PhilLovesCats/LLaMA-1-65B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use PhilLovesCats/LLaMA-1-65B-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PhilLovesCats/LLaMA-1-65B-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PhilLovesCats/LLaMA-1-65B-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M

Ollama
How to use PhilLovesCats/LLaMA-1-65B-GGUF with Ollama:
```
ollama run hf.co/PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use PhilLovesCats/LLaMA-1-65B-GGUF with Docker Model Runner:
```
docker model run hf.co/PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M
```

Lemonade

How to use PhilLovesCats/LLaMA-1-65B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull PhilLovesCats/LLaMA-1-65B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.LLaMA-1-65B-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

LLaMA 1 65B - GGUF

Model creator: Meta Llama
The 'Provided files'-section layout, as well as the metadata layout are heavily inspired from TheBloke
The 'License'-section, aswell as the 'LICENSE'-file are burrowed by huggyllama

Please tell me when i have violated your copyright, so that i can remove the belonging section!

Description

Due the fact that the old LLaMA (1) models are in a deprecated format and really hard to use with modern tools, i converted them using the transformers library's convert_llama_weights_to_hf.py script to convert them into a PyTorch model.

Further i used the convert_hf_to_gguf.py script from llama.cpp and to bring them into the well-known GGUF file format. Afterwards, i've build llama.cpp and ran llama-quantize.exe to downscale the model files.

Models > 50 GB were split with the gguf-split tool from llama.cpp and using this guide: https://github.com/ggml-org/llama.cpp/discussions/6404

Provided files

The following table only lists the quantized (.gguf) files (without the PyTorch (.bin) ones).

Name	Quant method	Bits	Size	MD5-Hash	Use case
llama-1-65b.TQ1_0.gguf	TQ1_0	1	13.70 GB	14AFF138168B0C07E218661253669162	smallest, significant quality loss - not recommended for most purposes
llama-1-65b.TQ2_0.gguf	TQ2_0	2	16.50 GB	21029F9B17EF252BFD12060450575434	very small, significant quality loss - not recommended for most purposes
llama-1-65b.Q2_K.gguf	Q2_K	2	22.40 GB	24ECE55B5157469C9F343430D03C6B14	very small, significant quality loss - not recommended for most purposes
llama-1-65b.Q3_K_S.gguf	Q3_K_S	3	26.20 GB	182A25B6477F4B9E08A20DF7C53EA99C	very small, high quality loss
llama-1-65b.Q3_K_M.gguf	Q3_K_M	3	29.40 GB	CF1AC34185D7EB309EA7F0887405E760	very small, high quality loss
llama-1-65b.Q3_K_L.gguf	Q3_K_L	3	32.20 GB	C824C9BFE1B52C6A70C00326172E1D0C	small, substantial quality loss
llama-1-65b.Q4_0.gguf	Q4_0	4	34.20 GB	FDB69C9A0E4CA4413FDBD2EEA5C419D8	legacy; small, very high quality loss - prefer using Q3_K_M
llama-1-65b.Q4_K_S.gguf	Q4_K_S	4	34.50 GB	1BC80164AF921D33174590D216800F69	small, greater quality loss
llama-1-65b.Q4_K_M.gguf	Q4_K_M	4	36.60 GB	1C94842178FEE7062EDA5A91D76E8B91	medium, balanced quality - recommended
llama-1-65b.Q5_0.gguf	Q5_0	5	41.80 GB	2166618D0CA3FB8DBFA6E494D54A345F	legacy; medium, balanced quality - prefer using Q4_K_M
llama-1-65b.Q5_K_S.gguf	Q5_K_S	5	41.80 GB	3402780368E0C82A7D9B385ED95462C5	large, low quality loss - recommended
llama-1-65b.Q5_K_M.gguf	Q5_K_M	5	43.00 GB	CF486B3FD3CFEF6F4817456B51D8915A	large, very low quality loss - recommended
llama-1-65b.Q6_K.gguf	Q6_K	6	49.80 GB	D5EF35C61A7FEA2518A963202BA04D02	very large, extremely low quality loss
llama-1-65b.Q8_0.gguf	Q8_0	8	64.60 GB	C9E1EB00A0831F857B19D416A8B2D63D	very large, extremely low quality loss - not recommended
llama-1-65b.F16.gguf	F16	16	121.00 GB	A0AE5F69C985BA6CF48527E87CDC2AFA	original, no quality loss - not recommended
llama-1-65b.BF16.gguf	BF16	16	121.00 GB	8C6E8AAFAD5B0853F083A1DA871CE40E	original, no quality loss - not recommended

Notice: Hashes were calculated with Get-FileHash .\FILENAME -Algorithm MD5 on Microsoft Windows 11 64-bit (x86_x64).

License

This model is under a non-commercial license (see the LICENSE file). You should only use this repository if you have been granted access to the model by filling out this form but either lost your copy of the weights or got some trouble converting them to the Transformers format.

Downloads last month: 270

GGUF

Model size

65B params

Architecture

llama

Hardware compatibility

1-bit

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit