abenzerps/Hy-Memory-30B-A3B-Quantized

Hy-Memory-30B-A3B Quantized

GGUF, Safetensors, and MLX quantized formats of tencent/Hy-Memory-30B-A3B for local memory operations and agent execution.

Tencent's Hy-Memory-30B-A3B is a specialized Mixture-of-Experts (MoE) model (30B total parameters, 3B active per token) trained specifically as a high-performance memory writer and agent-memory operator. It excels at session summarization, context compaction, entity memory extraction, and agent memory management.

Benchmark

Quantized Files

Quantization	File	Size
FP8	hy-memory-30b-a3b-fp8.safetensors	30.58 GB
NVFP4	hy-memory-30b-a3b-NVFP4.safetensors	17.65 GB
MLX 4-bit	hy-memory-30b-a3b-MLX-4bit.safetensors	17.65 GB
MLX 6-bit	hy-memory-30b-a3b-MLX-6bit.safetensors	25.04 GB
MLX 8-bit	hy-memory-30b-a3b-MLX-8bit.safetensors	32.43 GB
Q8_0	Hy-Memory-30B-A3B-Q8_0.gguf	31.99 GB
Q6_K	Hy-Memory-30B-A3B-Q6_K.gguf	24.71 GB
Q5_K_M	Hy-Memory-30B-A3B-Q5_K_M.gguf	21.37 GB
Q4_K_M	Hy-Memory-30B-A3B-Q4_K_M.gguf	18.24 GB
Q4_0	Hy-Memory-30B-A3B-Q4_0.gguf	17.02 GB

Q4_K_M is recommended for the best balance of size and quality for GGUF. NVFP4 and FP8 offer high performance on supported NVIDIA GPUs, while MLX formats are optimized for Apple Silicon.

Usage

1. llama.cpp / llama-server (GGUF)

Interactive Chat via CLI

Run multi-turn interactive conversation directly in your terminal:

./llama-cli -m Hy-Memory-30B-A3B-Q4_K_M.gguf \
    -cnv \
    -c 8192 \
    -n 512 \
    --temp 0.6 \
    -t 16

OpenAI-Compatible API Server

Start llama-server to serve OpenAI-compatible /v1/chat/completions endpoints:

./llama-server -m Hy-Memory-30B-A3B-Q4_K_M.gguf \
    -c 16384 \
    --port 8080

Query via curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [
      {
        "role": "user",
        "content": "Extract key entities and memory facts from this text: Alice is a software architect living in Berlin who prefers dark mode and Python."
      }
    ],
    "temperature": 0.6
  }'

2. Ollama (GGUF)

Create a Modelfile:

FROM ./Hy-Memory-30B-A3B-Q4_K_M.gguf

TEMPLATE """{{- if .System }}<｜hy_User｜>{{ .System }}
{{ end }}{{- range .Messages }}<｜hy_{{ if eq .Role "user" }}User{{ else }}Assistant{{ end }}｜>{{ .Content }}{{ if eq .Role "assistant" }}<eos:6124c78e>{{ end }}{{- end }}{{- if .Prompt }}<｜hy_User｜>{{ .Prompt }}<｜hy_Assistant｜>{{- end }}"""

PARAMETER stop "<eos:6124c78e>"
PARAMETER stop "<｜hy_User｜>"
PARAMETER stop "<｜hy_Assistant｜>"
PARAMETER stop "<｜hy_begin▁of▁sentence｜>"
PARAMETER temperature 0.6

Build and run:

ollama create hy-memory -f Modelfile
ollama run hy-memory

3. Apple Silicon (MLX Safetensors)

For Apple Silicon Macs, load MLX quantized weights (4-bit, 6-bit, or 8-bit):

import mlx.core as mx
from safetensors.numpy import load_file

# Load MLX-quantized weights
weights = load_file("hy-memory-30b-a3b-MLX-4bit.safetensors")

# Convert to native MLX array dictionary
mlx_model_weights = {k: mx.array(v) for k, v in weights.items()}
print(f"Loaded {len(mlx_model_weights)} MLX tensor parameters successfully.")

4. NVIDIA FP8 & NVFP4 (Safetensors)

Load the quantized weights directly via safetensors for TensorCore-accelerated pipelines (such as comfy-kitchen, vLLM, or custom inference):

import safetensors.torch
import torch

# Load FP8 (torch.float8_e4m3fn with per-tensor scales)
fp8_weights = safetensors.torch.load_file("hy-memory-30b-a3b-fp8.safetensors")

# Load NVFP4 (packed 4-bit with block-wise scales)
nvfp4_weights = safetensors.torch.load_file("hy-memory-30b-a3b-NVFP4.safetensors")

Source and Build

• Source model: tencent/Hy-Memory-30B-A3B
• Architecture: HYV3ForCausalLM (MoE: 128 experts, 8 active per token, 30B total / 3B active)
• Conversion & Quantization: llama.cpp & comfy-kitchen
• License: Apache 2.0
• Checksums: SHA256SUMS