Instructions to use Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF", dtype="auto")

llama-cpp-python

How to use Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF",
	filename="SARC-Taigi-12b-Q3_K_L.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M

SGLang

How to use Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF with Ollama:
```
ollama run hf.co/Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M
```

Unsloth Studio new

How to use Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF to start chatting

Docker Model Runner
How to use Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF with Docker Model Runner:
```
docker model run hf.co/Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M
```

Lemonade

How to use Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.SARC-Taigi-LLM-12b-GGUF-Q4_K_M

List all available models

lemonade list

SARC-Taigi-LLM-12b-GGUF (Taiwanese LLM)

This repository contains GGUF format model files for SARC-Taigi-LLM-12b. This model is a specialized version of google/gemma-3-12b-it, fine-tuned by the Speech AI Research Center (SARC) using IMA's 'Taiwan Tongues' Taigi Datasets and QLoRA.

The GGUF versions are optimized for inference on consumer-grade hardware (CPUs or GPUs with limited VRAM) via llama.cpp, Ollama, or other compatible backends.

1. Main Capabilities

Taigi Dialogue and Consultation: Understands and responds to inquiries in Taigi using both Taiwanese Chinese characters (Hàn-jī) and Romanization (Tâi-lô).
Linguistic Knowledge Retrieval: Supports queries regarding the meaning, usage, and cultural context of Taigi vocabulary.
Logical Reasoning: Capable of complex reasoning and problem-solving within a Taiwanese linguistic and cultural framework.

2. Quantization Versions (GGUF)

For a 12b model, we recommend Q4_K_M for the best balance between speed and linguistic precision.

File Name	Method	RAM/VRAM Required	Description
`SARC-Taigi-LLM-12b-Q8_0.gguf`	Q8_0	~14-16 GB	Near-lossless quality. Recommended for academic research and high-precision tasks.
`SARC-Taigi-LLM-12b-Q6_K.gguf`	Q6_K	~12-14 GB	High precision for specialized linguistic research.
`SARC-Taigi-LLM-12b-Q4_K_M.gguf`	Q4_K_M	~9-11 GB	Highly Recommended. Optimal quality/performance ratio for 12GB GPUs.
`SARC-Taigi-LLM-12b-Q3_K_L.gguf`	Q3_K_L	~7-9 GB	Lightweight version for memory-constrained or 8GB VRAM devices.

3. Main Capabilities

Taigi Dialogue and Consultation: Capable of understanding and responding to daily and professional inquiries in Taigi (using Taiwanese Chinese characters (Tâi-bûn Hàn-jī) or Romanization (Tâi-lô)).
Linguistic Knowledge Retrieval: Supports queries regarding the meaning, usage, and cultural background of Taigi vocabulary.
Logical Reasoning: Performs logical judgment and problem-solving specifically within a Taigi linguistic context.

4. Demostration

https://llm.ivoice.tw:64441/
QA Examples

5. Training Pipeline

The model underwent a two-stage training process designed to build a robust linguistic foundation, followed by instruction alignment:

Phase 1: Continual Pre-Training (CPT)
- Ministry of Education Dictionary of Frequently-Used Taiwanese Taigi.
- Taigi Literature Collection (taigi-literature): A diverse corpus of classical and modern Taigi literary works.
Phase 2: Supervised Fine-Tuning (SFT)
- Taigi-version Alpaca Dataset: Instruction-following data optimized for Taigi dialogue.
- Grand Challenge Training Set: Multiple-choice questions from the training text of the 1st "Grand Challenge" (科技大擂台) competition.

6. Evaluation on << 2020 Grand Challenge, Talk to AI >> Final-Test Dataset

We evaluated the models using the << 2020 Grand Challenge, Talk to AI (科技大擂台，與AI對話)>> Final-Test Dataset, which consists of 1,000 multiple-choice reading comprehension questions. This serves as a benchmark for Taigi language understanding:

Question Example

Experimental Results

	Model	Accuracy	Note
Stage	Gemma-3-12b-it	Gemma-3-27b-it
Original	0.80320	0.86214	Baseline performance
After CPT	0.88312	0.92296	Knowledge internalization
After SFT	0.89610	0.92582	Instruction alignment

7. Model Usage

via Ollama

# Ensure Ollama is installed and your Ollama SSH public key (typically found at ~/.ollama/id_ed25519.pub) is registered in your Hugging Face account.

ollama pull huggingface.co/Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M

ollama run huggingface.co/Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M

ollama ps

ollama run huggingface.co/Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M

via llama.cpp

# Ensure you are in the llama.cpp directory
cd /path/to/llama.cpp

# Run the Taigi model with optional optimization flags
./build/bin/llama-cli \
  -hf Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF:Q4_K_M \
  -p "<start_of_turn>user\n用台語共我紹介一下你自己。<end_of_turn>\n<start_of_turn>model\n" \
  -n 512 \
  -ngl 99 \
  --temp 0.7

Optional Arguments:
- -n, --n-predict (Default: 128)
  - Specifies the maximum number of tokens to generate. For a 12b model, we recommend 512 or higher to ensure the Taigi responses are not cut off mid-sentence.
- -ngl, --n-gpu-layers (Default: 0)
  - Crucial for Performance: Determines how many model layers are offloaded to the GPU.
  - Setting it to 99 (or any number higher than the actual layers) forces the entire model into VRAM for maximum speed.
  - Note: If your VRAM is insufficient (e.g., less than 20GB for Q4_K_M), decrease this number to perform "partial offloading" to the CPU. If omitted, the model runs entirely on the CPU, which will be significantly slower.
- --temp (Default: 0.8)
  - Adjusts the randomness of the output.
  - 0.7 provides a good balance between creativity and coherence for Taigi dialogue. Use a lower value (e.g., 0.2) for factual tasks.

8. Roadmap: Beyond SFT

While the current release is the result of CPT and SFT, this is only the beginning. Our multi-stage alignment strategy includes:

Phase I (CPT): Building linguistic foundation (Completed).
Phase II (SFT): Instruction and dialogue alignment (Current Release).
Phase III (GRPO): Future reinforcement learning using Group Relative Policy Optimization (GRPO) to further enhance self-correction and complex reasoning chains.

9. Training Resources

Learn how to perform this multi-stage fine-tuning (CPT + SFT) with our custom callbacks for Loss minimization and Gap stability on GitHub:

[GitHub: SARC-Taigi-LLM Training Pipeline]

Citation

If you find this project useful, please cite the IMA's Taiwan Tongues resource page and the Speech AI Research Center organization pages on Hugging Face and GitHub.

@misc{ima_taiwan_2026,
  title        = {IMA-Taiwan},
  author       = {Information Management Association of R.O.C. (IMA)},
  year         = {2026},
  howpublished = {https://huggingface.co/IMA-Taiwan},
  note         = {Hugging Face organization page for Taiwan Tongues resources}
}

@misc{sarc_hf_2026,
  title        = {Speech-AI-Research-Center},
  author       = {Speech AI Research Center (SARC)},
  year         = {2026},
  howpublished = {https://huggingface.co/Speech-AI-Research-Center},
  note         = {Hugging Face organization page for released Taigi model adapters}
}

@misc{sarctaigillm_repo_2026,
  title        = {Speech-AI-Research-Center},
  author       = {Speech AI Research Center (SARC)},
  year         = {2026},
  howpublished = {https://github.com/Speech-AI-Research-Center},
  note         = {GitHub organization page for released Taigi-LLM training project}
}

License

This model is subject to the Gemma Terms of Use. By using this model, you agree to comply with Google’s licensing requirements.

Downloads last month: 113

GGUF

Model size

12B params

Architecture

gemma3

Hardware compatibility

3-bit

4-bit

6-bit

8-bit

Model tree for Speech-AI-Research-Center/SARC-Taigi-LLM-12b-GGUF

Base model

google/gemma-3-12b-pt

Finetuned

google/gemma-3-12b-it

Quantized

(149)

this model

Speech-AI-Research-Center
/

SARC-Taigi-LLM-12b-GGUF