Instructions to use daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1", trust_remote_code=True)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1", trust_remote_code=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1

SGLang

How to use daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1 with Docker Model Runner:
```
docker model run hf.co/daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1
```

Llama-3.1-7B — 20% Compressed from Llama-3.1-8B (English)

Built with Llama. This repository is part of the Efficient and Robust AI System (E-AI) Project by Vincent-Daniel Yun. This model is a compressed edition of meta-llama/Llama-3.1-8B with 6 of 32 transformer layers removed (26 layers remain, ≈6.72B parameters), recovered training-free. It is a base (non–chat-tuned) model.

🔗 Project: https://www.worldwidedaniel.com/eai-project 📅 Release: 2026-06-28 · Version: V1

⚖️ License: governed by the Llama 3.1 Community License Agreement (https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/LICENSE), and subject to the Llama 3.1 Acceptable Use Policy (https://www.llama.com/llama3_1/use-policy). By using this model you agree to those terms. "Llama" is a trademark of Meta Platforms, Inc. Built with Llama.

⚠️ Language: English-focused. Use as a discrimination / classification engine (classification, reading comprehension, domain QA, judging, safety screening) — open-ended long-form generation is degraded by compression (see PPL). For factual questions use retrieval (RAG).

About E-AI

Modern AI is powerful but heavy. State-of-the-art models are enormous and their inference is slow. The E-AI (Efficient-AI) project builds compact yet capable AI — making every model lightweight and fast, and keeping multi-agent teams reliable even when some members fail — so AI can assist people in urgent, high-stakes moments.

Method

The pruning method and the training-free recovery method are proprietary, undisclosed methods created by Vincent-Daniel Yun and are not released. Only the resulting model is shared.

Results (measured, full test sets)

PPL on 2048-token context (↓ better); downstream & MMLU are 0-shot via lm-eval-harness (↑ better).

Metric	Llama-3.1-8B (base)	This model (20%)
PPL · WikiText2 ↓	6.24	15.06
PPL · C4 ↓	8.68	18.95
ARC-c ↑	0.5350	0.4420
ARC-e ↑	0.8114	0.7256
BoolQ ↑	0.8208	0.6538
COPA ↑	0.8700	0.8300
HellaSwag ↑	0.7889	0.6968
OpenBookQA ↑	0.4480	0.4040
RACE ↑	0.3914	0.3847
RTE ↑	0.6931	0.7004
WinoGrande ↑	0.7356	0.7182
Avg. downstream (9) ↑	0.6771	0.6173
MMLU ↑	0.6360	0.6254

Task-suitability (vs base Llama-3.1-8B)

Where this compressed model stays close to the base on discrimination tasks (full test sets):

Task	Llama-3.1-8B (base)	This model (20%)
Topic classification (AG News)	0.564	0.772
LLM-as-judge (RewardBench)	0.660	0.665
SafetyBench (MCQ)	0.743	0.657
MultiRC	0.572	0.572
WiC	0.511	0.500
MRPC	0.674	0.684
CB (NLI)	0.643	0.786
SST-2	0.775	0.803
MedQA	0.598	0.600
MedMCQA	0.564	0.563
PubMedQA	0.758	0.630
Belebele-en	0.788	0.791
Belebele-ko	0.662	0.654
XNLI-zh	0.395	0.400
ToxiGen	0.432	0.432
TruthfulQA	0.452	0.456

(Discrimination is largely preserved; generation/PPL degrades with compression. Yes/No safety-F1 benchmarks are omitted as the non-chat base is poorly calibrated for them.)

Efficiency (measured, fp16, single GPU)

	Llama-3.1-8B (base)	This model (20%)
Layers	32	26
Parameters	8.03B	6.72B
Peak inference memory (fp16)	19.31 GB	16.49 GB (−15%)
Forward latency (fp16)	1293 ms	1114 ms (−14%)

Usage — Transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1", trust_remote_code=True, dtype=torch.float16, device_map="cuda")
tok = AutoTokenizer.from_pretrained("daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1", trust_remote_code=True)
ids = tok("The capital of France is", return_tensors="pt").to("cuda")
print(tok.decode(m.generate(**ids, max_new_tokens=20)[0]))

trust_remote_code=True is required (custom decoder layer in modeling_llama_recovered.py).

License & Acknowledgements

This model is a derivative of Llama 3.1 and is distributed under the Llama 3.1 Community License. Built with Llama. Thanks to Prof. Sai Praneeth Karimireddy (USC) and Prof. Sunwoo Lee (Inha University) for guidance, and to Meta for releasing Llama 3.1.

Downloads last month: 19

Safetensors

Model size

7B params

Tensor type

F16

Model tree for daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1

Base model

meta-llama/Llama-3.1-8B

Finetuned

(1436)

this model

Collection including daniel-eai/Llama-3.1-7B-20pct-Compressed-8B-EN-V1

Compressed Models V1

Collection

Compressed LLMs (Qwen3, Llama families). V1 (First Version) • 16 items • Updated 1 day ago • 1