Instructions to use AlexHung29629/gemma-4-E2B-sft-text with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AlexHung29629/gemma-4-E2B-sft-text with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AlexHung29629/gemma-4-E2B-sft-text")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("AlexHung29629/gemma-4-E2B-sft-text")
model = AutoModelForMultimodalLM.from_pretrained("AlexHung29629/gemma-4-E2B-sft-text", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AlexHung29629/gemma-4-E2B-sft-text with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AlexHung29629/gemma-4-E2B-sft-text"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/gemma-4-E2B-sft-text",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/AlexHung29629/gemma-4-E2B-sft-text

SGLang

How to use AlexHung29629/gemma-4-E2B-sft-text with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AlexHung29629/gemma-4-E2B-sft-text" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/gemma-4-E2B-sft-text",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AlexHung29629/gemma-4-E2B-sft-text" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AlexHung29629/gemma-4-E2B-sft-text",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use AlexHung29629/gemma-4-E2B-sft-text with Docker Model Runner:
```
docker model run hf.co/AlexHung29629/gemma-4-E2B-sft-text
```

See axolotl config

axolotl version: 0.18.0

# Gemma-4-E2B 純文字 SFT（單機 8x H100，DDP)
#
# 跟 gemma4-e2b-sft.yaml（多模態,含圖片)是分開的一份設定：這份完全不吃圖片,
# 資料全部是用 prepare_pi_traces_input_output.py / fetch_nemotron_agentic.py
# 轉成的 axolotl `input_output` 格式（segments: [{label, text}, ...]),已經
# 套用過 Gemma4 chat_template 並標好哪些 special token 要訓練（每一則
# assistant 訊息在某一筆樣本裡都是「最後一則」,含它自己的 <|turn>model、
# <|channel>thought、<|tool_call>、<turn|> 等 token；tool 執行結果永遠落在
# 下一筆樣本的 label:false 前綴,不會被訓練)。
#
# 因為完全不會餵圖片，不需要 gemma4-e2b-sft.yaml 裡那些多模態專用設定
# （processor_type/skip_prepare_dataset/remove_unused_columns/image_size 等)——
# 這份走 axolotl 標準的 ChatTemplateStrategy .map() 前處理路徑，用一般
# AutoTokenizer 即可，也因此會依 sequence_len 自動丟掉過長樣本（不像
# skip_prepare_dataset: true 那條路完全不會過濾)。
#
# vision_tower/audio_tower/embed_vision/embed_audio 依然凍結
# (freeze_mm_modules)——文字資料不會用到這些參數，凍結純粹是為了省
# DDP 下每張卡的 optimizer state，不是因為訓練圖片。

# base_model 接續 stage1（gemma4-e2b-sft-stage1-embed.yaml，embedding-only）訓練完的權重
base_model: /mnt/shared/p01/alex/E2B/treadon/gemma4-E2B-it-Abliterated-AND-Disinhibited-USE-THIS
hub_model_id: AlexHung29629/gemma-4-E2B-sft-text

freeze_mm_modules: true

# --- Plugins（跟 gemma4-e2b-sft.yaml 一致，理由同上，見該檔案註解)---
plugins:
  - axolotl.integrations.liger.LigerPlugin
  - axolotl.integrations.cut_cross_entropy.CutCrossEntropyPlugin

cut_cross_entropy: true

liger_rms_norm: true
liger_glu_activation: false
liger_rope: true
liger_layer_norm: false
liger_fused_linear_cross_entropy: false

strict: false

# --- Dataset ---
# 六份資料都是 input_output 格式（見各自的 prepare 腳本)：
#   - pi_traces：agentic trace（GSM8K 類數學題 + bash/工具使用),含思考與
#     不含思考兩個版本都放進來 —— 同一批對話的最終答案文字會被訓練兩次
#     （一次連 <|channel>thought 推理過程、一次不含),故意讓這批資料的
#     權重是其他資料的 2 倍。
#   - nemotron_instruction_following：純聊天/指令遵循（無 tool),1000 筆。
#   - nemotron_agentic 三個子集（interactive_agent/search/tool_calling)：
#     tool-use agent 對話，各 1000 筆原始對話展開後的所有 assistant turn。
# 五份非 pi_traces 資料都是 enable_thinking=False + 拿掉 reasoning_content
# 轉出來的，完全不含思考過程文字。
datasets:
  - path: data/pi_traces_sft_input_output.jsonl
    type: input_output
  - path: data/pi_traces_sft_input_output_no_thinking.jsonl
    type: input_output
  - path: data/nemotron_instruction_following_input_output.jsonl
    type: input_output
  - path: data/nemotron_agentic_interactive_agent_input_output.jsonl
    type: input_output
  - path: data/nemotron_agentic_search_input_output.jsonl
    type: input_output
  - path: data/nemotron_agentic_tool_calling_input_output.jsonl
    type: input_output

val_set_size: 200
dataset_num_proc: 8

# --- Sequence length ---
# 六份資料合計 26493 筆，token 長度分布：nemotron_agentic 的 search 子集
# （多輪工具搜尋)尾部很長，p99 約 37k、最長 72k token；其餘資料 p99 都在
# 1.2 萬 token 內。sequence_len=16384 跟 gemma4-e2b-sft.yaml 一致（已用
# Cut Cross Entropy 驗證過在這個環境下不會 OOM),會篩掉整體 13.4%
# （約 3561 筆，幾乎都是 search 子集偏長的樣本)——這份走標準
# ChatTemplateStrategy 前處理，過長樣本會被 axolotl 自動丟掉，不需要像
# pi_traces 那樣另外寫腳本手動過濾。
sequence_len: 16384
pad_to_sequence_len: false

attn_implementation: flash_attention_2
gemma4_hybrid_attn_impl: true

# --- DDP（放棄 FSDP2，理由同 gemma4-e2b-sft.yaml)---
gradient_checkpointing: true

# --- Training ---
num_epochs: 4
micro_batch_size: 1
gradient_accumulation_steps: 8
# effective batch = 1 x 8 GPUs x 8 accum = 64 samples/step

learning_rate: 1.0e-5
lr_scheduler: constant_with_warmup
warmup_ratio: 0.03
weight_decay: 0.01
adam_beta1: 0.9
adam_beta2: 0.95
adam_epsilon: 1.0e-8
max_grad_norm: 1.0
optimizer: adamw_torch_8bit

bf16: true

# --- Logging / Saving / Eval ---
eval_strategy: epoch
save_strategy: best
save_total_limit: 1
load_best_model_at_end: true
metric_for_best_model: eval_loss
greater_is_better: false
logging_steps: 1

use_wandb: true
wandb_project: gemma4-e2b-sft-text
use_tensorboard: true

output_dir: ./outputs/gemma4-e2b-sft-text

seed: 42
dataloader_num_workers: 8

gemma-4-E2B-sft-text

This model was trained from scratch on the data/pi_traces_sft_input_output.jsonl, the data/pi_traces_sft_input_output_no_thinking.jsonl, the data/nemotron_instruction_following_input_output.jsonl, the data/nemotron_agentic_interactive_agent_input_output.jsonl, the data/nemotron_agentic_search_input_output.jsonl and the data/nemotron_agentic_tool_calling_input_output.jsonl datasets. It achieves the following results on the evaluation set:

Loss: 0.7972
Ppl: 2.2193
Memory/max Active (gib): 28.81
Memory/max Allocated (gib): 28.81
Memory/device Reserved (gib): 57.59

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-05
train_batch_size: 1
eval_batch_size: 1
seed: 42
distributed_type: multi-GPU
num_devices: 8
gradient_accumulation_steps: 8
total_train_batch_size: 64
total_eval_batch_size: 8
optimizer: Use OptimizerNames.ADAMW_TORCH_8BIT with betas=(0.9,0.95) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
lr_scheduler_type: constant_with_warmup
lr_scheduler_warmup_steps: 42
training_steps: 1421

Training results

Training Loss	Epoch	Step	Validation Loss	Ppl	Active (gib)	Allocated (gib)	Reserved (gib)
No log	0	0	3.9103	49.9152	20.02	20.02	20.56
0.8613	1.0	356	0.9344	2.5457	28.81	28.81	57.73
0.5877	2.0	712	0.8594	2.3617	28.81	28.81	56.76
0.7885	3.0	1068	0.8226	2.2764	28.81	28.81	56.62
0.7696	3.9937	1421	0.7972	2.2193	28.81	28.81	57.59

Framework versions

Transformers 5.14.1
Pytorch 2.11.0
Datasets 4.8.4
Tokenizers 0.22.2

Downloads last month: -

Safetensors

Model size

5B params

Tensor type

BF16