Instructions to use PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone")
model = AutoModelForCausalLM.from_pretrained("PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone

SGLang

How to use PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone with Docker Model Runner:
```
docker model run hf.co/PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone
```

Model Details

The quantized MobileLLM models is introduced in: "ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization"

Model Developer: Meta

Model Architecture: ParetoQ is the first unified framework that facilitates rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings. By optimizing training schemes and refining quantization functions, ParetoQ surpasses all previous methods tailored to specific bit widths. Specifically, the 1.58-bit ParetoQ LLaMA-3 8B model reduces the performance gap to full precision by relatively 37.8% compared to the 1-bit Era’s 1.58-bit LLaMA-3 8B model, while using only 30% of the training tokens.

With the SoTA points obtained through ParetoQ, we are able to improve the scaling law analysis. Figure (a) (b) demonstrates that sub-4-bit quantization, including binary, ternary, 2-bit, and 3-bit, often outperform 4-bit quantization. Notably, 2-bit and ternary models reside on the Pareto frontier. When considering hardware-friendliness and real-time speed, we generally recommend exploring 2-bit quantization for on-device applications.

How to use

We are providing 2 ways to run the model:

HuggingFace

ParetoQ codebase

HuggingFace

To load the pretrained model for further finetuning or evaluation:

from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("facebook/MobileLLM-ParetoQ-125M-BF16", use_fast=False)
model = AutoModelForCausalLM.from_pretrained("facebook/MobileLLM-ParetoQ-125M-BF16", trust_remote_code=True)

Note that the default tokenizer does not contain special tokens. For example you can use:

tokenizer.add_special_tokens(
    {
        "eos_token": "</s>",
        "bos_token": "<s>",
        "unk_token": "<unk>",
    }
)

ParetoQ codebase

We provide the pretraining code in https://github.com/facebookresearch/ParetoQ

> git clone https://github.com/facebookresearch/ParetoQ
> pip3 install torch
> pip install -r requirement.txt

# Specify the data path and the pre-trained full-precision model path in run_train.sh file
> Run `bash 1_run_train.sh $w_bit` E.g. `bash 1_run_train.sh 2` for 2-bit weight quantization.

Evaluation

We evaluate ParetoQ models using zero-shot commonsense reasoning task accuracy and WikiText-2 perplexity.

Method	Arc-e	Arc-c	Boolq	Piqa	Siqa	HellaSwag	Obqa	WinoGrande	Avg.	Wiki
MobileLLM-ParetoQ-125M
MobileLLM-ParetoQ-125M-BF16	56	34.5	56.3	65.5	42	40.1	42.2	51.3	48.5	15.1
MobileLLM-ParetoQ-125M-1-bit	43.9	29.1	61.2	59.2	39.8	29.8	33.7	52.7	43.7	25.8
MobileLLM-ParetoQ-125M-1.58-bit	49.3	30.9	61	62.1	41	34.3	40.4	52.9	46.5	20.0
MobileLLM-ParetoQ-125M-2-bit	50.7	32.7	59.8	63.3	41	36.3	40.6	52.7	47.1	18.2
MobileLLM-ParetoQ-125M-3-bit	53.5	33.7	56.1	65.6	41.7	40	41.2	51.3	47.9	15.0
MobileLLM-ParetoQ-125M-4-bit	55.4	35.2	54.1	66.2	41.7	40.8	44	52.1	48.7	14.1
MobileLLM-ParetoQ-350M
MobileLLM-ParetoQ-350M-BF16	65.5	42.3	57.4	71	43.5	53.3	47.3	58.3	54.8	10.5
MobileLLM-ParetoQ-350M-1-bit	52.7	31.3	61.6	63.9	40.9	38.3	39.5	52.9	47.6	17.0
MobileLLM-ParetoQ-350M-1.58-bit	56.8	36.3	62.2	67.1	43.5	44	46.3	55.2	51.4	14.5
MobileLLM-ParetoQ-350M-2-bit	59	39.4	63.5	68.8	43.1	47.3	44.1	57.5	52.8	12.5
MobileLLM-ParetoQ-350M-3-bit	63.9	40.5	61.4	70.6	43.2	51.4	50	56.6	54.7	10.9
MobileLLM-ParetoQ-350M-4-bit	64.9	41.6	57.8	71.3	44.4	53.5	48.2	57.9	55.0	10.3
MobileLLM-ParetoQ-600M
MobileLLM-ParetoQ-600M-BF16	68.5	47.6	60.5	72.5	44.4	59.5	51.4	61.4	58.2	9.1
MobileLLM-ParetoQ-600M-1-bit	58.9	36.0	60.5	65.2	43.1	44.2	40.7	53.9	50.3	14.0
MobileLLM-ParetoQ-600M-1.58-bit	65.5	43.8	62.3	70.6	44.7	51.3	47.1	58.8	55.5	11.5
MobileLLM-ParetoQ-600M-2-bit	67.7	43.3	63	72.1	44.8	53.9	49.8	58.4	56.6	10.5
MobileLLM-ParetoQ-600M-3-bit	68.2	47.4	64.2	73.1	44.2	58.1	50.2	62.4	58.5	9.4
MobileLLM-ParetoQ-600M-4-bit	69.3	48.9	64.8	73.2	44.2	59.5	51.2	62.1	59.2	8.9
MobileLLM-ParetoQ-1B
MobileLLM-ParetoQ-1B-BF16	73.4	50.8	67.6	74.1	46.7	64.7	56.6	62.7	62.1	8.0
MobileLLM-ParetoQ-1B-1-bit	62.6	40.2	62.1	69.5	42.8	49.5	48.8	54.9	53.8	12.8
MobileLLM-ParetoQ-1B-1.58-bit	68.5	47.6	62.8	72.1	45.3	57.4	52.9	61.3	58.5	10.0
MobileLLM-ParetoQ-1B-2-bit	73.3	49.3	65.7	74.2	45.9	60.3	57.4	61.6	61.0	9.2
MobileLLM-ParetoQ-1B-3-bit	72.3	51.4	67	74.5	45.7	63.4	53.7	62.1	61.3	8.4
MobileLLM-ParetoQ-1B-4-bit	74.7	52.1	67.9	74.8	46.9	64.8	56.2	62.1	62.5	7.9
MobileLLM-ParetoQ-1.5B
MobileLLM-ParetoQ-1.5B-BF16	73.9	51.4	70	74.8	46.6	66.4	55.1	63.2	62.7	7.9
MobileLLM-ParetoQ-1.5B-1-bit	67.9	42.4	63.4	70.2	44.5	54.2	47.4	57.6	55.9	11.0
MobileLLM-ParetoQ-1.5B-1.58-bit	70.2	48	65.8	73.4	47.3	61.8	55.3	62.4	60.5	9.0
MobileLLM-ParetoQ-1.5B-2-bit	73.3	47.5	70.1	74.1	46.8	64.6	55.5	62.5	61.8	8.3
MobileLLM-ParetoQ-1.5B-3-bit	72.6	49.9	70.6	75.7	47.7	66	56.2	64.5	62.9	8.0
MobileLLM-ParetoQ-1.5B-4-bit	74.4	51.7	71.8	75.3	47.3	67.2	57.6	63	63.6	7.6

Acknowledgement

This code is partially based on HuggingFace Transformers repo under Apache License.

Citation

If you find our code useful for your research, please consider citing:

@article{liu2025paretoq,
  title={ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization},
  author={Liu, Zechun and Zhao, Changsheng and Huang, Hanxian and Chen, Sijia and Zhang, Jing and Zhao, Jiawei and Roy, Scott and Jin, Lisa and Xiong, Yunyang and Shi, Yangyang and others},
  journal={arXiv preprint arXiv:2502.02631},
  year={2025}
}

License

ParetoQ is FAIR NC licensed as of now

Downloads last month: 29

Paper for PeytonT/mobilellm-paretoq-350m-modelstack-nf4-iphone

ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization

Paper • 2502.02631 • Published Feb 4, 2025 • 4