GLM-Edge-4B-Chat-mlx-4bit

This repository contains a 4-bit quantized MLX format compilation of Zhipu AI's GLM-Edge-4B-Chat model. It is optimized for hardware-accelerated local inference on Apple Silicon using the Apple MLX framework.

Model Details

• Architecture: GLM-Edge-4B-Chat
• Format: MLX (Apple Silicon native GPU format)
• Quantization: 4-bit (Average 4.501 bits per weight)
• Memory Footprint: Under 3 GB (ideal for Apple hardware with 8GB unified memory)

Local Execution with MLX

You can run this model directly using the mlx_lm package.

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("SirSahOl/glm-edge-4b-chat-mlx-4bit")
response = generate(model, tokenizer, prompt="Write a Python script to fetch stock prices.", verbose=True)

LM Studio Setup Guide

To prevent runway loops and ensure correct conversational turn-taking, you must configure the custom stop strings in your LM Studio preset or inference settings.

Custom Stop Strings Configuration

Ensure that the following sequences are bound to the inference settings:

1. <|user|>
2. <|observation|>
3. <|endoftext|>

Steps to Import to LM Studio

1. Verify the model symlink matches your local publisher path ~/.cache/huggingface/hub/local_models/SirSahOl/.
2. Launch LM Studio and select the model from your dropdown.
3. Go to the My Models > Inference > Settings side menu.
4. Navigate to Stop Strings and manually add <|user|>, <|observation|>, and <|endoftext|> as strict stop boundaries.