Instructions to use FesarovLab/ai-mentor-nanbeige4.2-3b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use FesarovLab/ai-mentor-nanbeige4.2-3b with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

Use Docker

docker model run hf.co/FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

LM Studio
Jan

vLLM

How to use FesarovLab/ai-mentor-nanbeige4.2-3b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "FesarovLab/ai-mentor-nanbeige4.2-3b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "FesarovLab/ai-mentor-nanbeige4.2-3b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

Ollama
How to use FesarovLab/ai-mentor-nanbeige4.2-3b with Ollama:
```
ollama run hf.co/FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M
```
Unsloth Desktop

How to use FesarovLab/ai-mentor-nanbeige4.2-3b with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use FesarovLab/ai-mentor-nanbeige4.2-3b with Docker Model Runner:
```
docker model run hf.co/FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M
```

Lemonade

How to use FesarovLab/ai-mentor-nanbeige4.2-3b with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

Run and chat with the model

lemonade run user.ai-mentor-nanbeige4.2-3b-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use FesarovLab/ai-mentor-nanbeige4.2-3b with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use FesarovLab/ai-mentor-nanbeige4.2-3b with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "FesarovLab/ai-mentor-nanbeige4.2-3b:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

AI Mentor Nanbeige4.2-3B (Abliterated)

An abliterated version of Nanbeige/Nanbeige4.2-3B, built exclusively for use as the built-in model of the AI Mentor application — a local AI mentor for people getting started with LLM engineering.

About this model

Base model: Nanbeige4.2-3B (looped transformer, 22 layers × 2 loop passes). Abliteration was performed with a custom tool based on multi-directional refusal-direction search (SVD over the difference matrix of harmful/harmless prompt activations, applied globally across 44 modules — o_proj/down_proj of each layer).

This abliteration was not released as a standalone open-source artifact — the model is distributed solely as a component of the AI Mentor application and is tuned for that purpose, not as a general-purpose model.

Fine-tuning

The model was fine-tuned (SFT) on a custom dataset generated for local LLM engineering topics: vLLM/llama.cpp setup, GGUF quantization, abliteration, LoRA/PEFT, interpretability, and related subjects. The dataset includes RAG-context dialogues (the model was trained to rely explicitly on the provided context and to admit when an answer isn't in the context, rather than making one up).

Files

File	Quantization	Size	Notes
`nanbeige4.2-3b-abliterated.Q4_K_M-imat.gguf`	Q4_K_M + imatrix, attention/embedding/output tensors upgraded to Q6_K	~2.8GB	Recommended default

Quantization was done with imatrix calibration on a custom in-domain corpus (the AI Mentor knowledge-base documentation), with manual per-tensor precision overrides for attention K/V, embeddings, and the output layer (upgraded to Q6_K), while the bulk of the FFN weights remain Q4_K_M. This is not Unsloth's official Dynamic Quants method — it's an independent implementation of the same idea.

Usage

This model is intended to be used through the AI Mentor application, which downloads and configures it automatically. Manual usage via llama.cpp:

llama-cli -m nanbeige4.2-3b-abliterated.Q4_K_M-imat.gguf \
  --chat-template chatml -ngl 99

⚠️ Warning

Abliteration removes the model's built-in refusal mechanisms. This model may generate content the base model would have declined, including content that could be harmful, inaccurate, or otherwise inappropriate. Use at your own risk and in accordance with applicable law.

License

Licensed under Apache 2.0, same as the base model Nanbeige/Nanbeige4.2-3B. As required by Apache 2.0: this is a modified version of the original work (abliteration + SFT fine-tuning applied); the original copyright notice and license terms are preserved by reference to the base model.

Downloads last month: 6

GGUF

Model size

4B params

Architecture

nanbeige

Hardware compatibility

4-bit

Model tree for FesarovLab/ai-mentor-nanbeige4.2-3b

Base model

Nanbeige/Nanbeige4.2-3B-Base

Finetuned

Nanbeige/Nanbeige4.2-3B

Quantized

(46)

this model