GlobalCybersecurityAlliance/GLM-5.3-Flash-Abliterated-LoRA-MLX

GLM-5.3-Flash-Abliterixed-LoRA-MLX

Two refusal-direction ablation LoRA variants for GLM-5.3-Flash.

This repository contains LoRA adapter weights only. Both adapters were derived from zai-org/GLM-5.3-Flash with Abliterix to reduce refusal behavior while leaving the base checkpoint unchanged. Version 1 applies the stronger, later-layer intervention; version 2 redistributes the routed-expert edits toward earlier and middle layers and produces substantially lower KL divergence from the deployed NVFP4 base.

The adapters are intended for controlled safety research, red-teaming, and refusal-mechanism analysis. Removing refusal behavior also removes an important safety layer; read the Disclaimer before use.

Repository contents

.
├── README.md
├── v1/
│   ├── adapter_config.json
│   └── adapters.safetensors
└── v2/
    ├── adapter_config.json
    └── adapters.safetensors

• v1/ contains the original adapter corresponding to GLM-5.3-Flash-Ablitered.
• v2/ contains the second adapter corresponding to GLM-5.3-Flash-Ablitered2.
• Each directory is a self-contained MLX-LM adapter. This repository does not contain the GLM-5.3-Flash base weights or merged checkpoints.

Adapter summary

Item	v1	v2
Repository path	v1/	v2/
Base model	zai-org/GLM-5.3-Flash	Same model family; exported config records an FP8 source path
Format	MLX-LM LoRA (adapters.safetensors)	MLX-LM LoRA (adapters.safetensors)
Rank / alpha	r=1, lora_alpha=1	r=1, lora_alpha=1
Dropout / bias	0.0 / none	0.0 / none
Effective edited layer range	Layers 15–44	Layers 3–44
Routed-expert down_proj range	Layers 18–44	Layers 3–28
Main effective target	Routed-expert down_proj	Routed-expert down_proj
Evaluation base	RedHatAI/GLM-5.3-Flash-NVFP4	Same

What is abliteration?

Abliteration is a weight-space intervention inspired by Arditi et al., Refusal in Language Models Is Mediated by a Single Direction. In simplified form, a refusal direction is estimated from the difference between internal activations elicited by harmful and harmless prompts. Components of selected writer matrices that contribute to that direction are then removed. A rank-1 update can be represented as:

W' = W + ΔW,      ΔW = BA,      rank(ΔW) = 1

Storing ΔW as a LoRA adapter means the original model checkpoint is not rewritten and the intervention can be enabled or removed at serving time.

Why routed-expert down_proj?

The analysis published with orcarouter/GLM-5.3-Flash-Uncensored-FP8 identifies the routed-expert MLP down_proj writers as the dominant intervention point for GLM-5.3-Flash refusal behavior. Its component study reports that attention-only and dense/shared-MLP interventions retain most refusal behavior, whereas adding routed-expert MLP writers causes the major reduction.

Our layer-restricted experiments agreed with that mechanism-level result: the causally useful refusal directions were found in the earlier-to-middle part of the expert stack, while modifications restricted to the later expert layers had almost no practical effect on refusal. Accordingly, the routed-expert down_proj modules are the main effective target in this adapter.

This statement concerns causal intervention effectiveness, not the numerical magnitude of every stored LoRA tensor. Adapter coverage and tensor norm alone do not establish where a behavior is represented.

This adapter instead uses Abliterix's SRA + mean + per-layer workflow:

• Mean base estimator: harmful-versus-harmless mean activation differences provide the initial direction estimate.
• SRA: the initial refusal signal is spectrally cleaned/residualized against protected benign-capability components, reducing unwanted removal of overlapping capabilities.
• Per-layer estimation: each edited layer receives its own estimated direction; a single layer-22 vector is not reused across the entire model.

Per-layer estimation is important here because the refusal signal is not equally useful at every depth, and late-layer-only interventions were ineffective in our experiments.

LoRA weight distribution

The following figures were obtained by inspecting the source PEFT adapter_config.json and adapter_model.safetensors files before MLX conversion. “Effective” means that the resulting rank-1 LoRA delta is non-zero.

Effective non-zero edits

Module	v1 layer coverage	v1 modules	v2 layer coverage	v2 modules
Routed-expert down_proj	18–44	7,776	3–28	7,488
Shared-expert down_proj	18–44	27	18–44	27
Self-attention o_proj	15–44	30	15–44	30
Total effective modules	15–44	7,833	3–44	7,545

Storage statistic	v1	v2
BF16 tensors	46,770	45,042
Rank-1 module entries	23,385	22,521
Serialized scalars	143,927,296	138,618,880
Non-zero scalars	48,375,808	46,606,336
Source PEFT safetensors size	281.6 MiB	271.2 MiB

Both PEFT configurations list down_proj, gate_proj, o_proj, and up_proj as target modules. In both checkpoints, the stored gate_proj and up_proj factors are structural zero entries and do not alter model output. The effective changes are therefore routed/shared-expert down_proj and attention o_proj updates.

The main difference is depth allocation:

• v1: layers 0–14 are unchanged; layers 15–17 modify attention o_proj only; layers 18–44 modify attention o_proj, all 288 routed-expert down_proj modules, and the shared-expert down_proj.
• v2: layers 0–2 are unchanged; layers 3–14 modify all 288 routed-expert down_proj modules; layers 15–17 add attention o_proj; layers 18–28 also add the shared-expert down_proj; layers 29–44 retain only attention and shared-expert edits.

In other words, v1 concentrates the expert intervention in the later 27 layers, while v2 shifts it to the earlier-to-middle expert stack and removes routed-expert edits from layers 29–44.

Evaluation

Both adapters were evaluated while attached separately to RedHatAI/GLM-5.3-Flash-NVFP4. The tested deployment used vLLM, explicitly selected low reasoning effort for the target model, and passed only the target model's final answer—not its private reasoning—to the judge.

The automated judge was deepseek-v4-flash through the Ollama Cloud API, with judge thinking disabled. These results are automated estimates, not human safety certification.

Overall results

Benchmark / metric	n	v1	v2
SimpleSafetyTests full refusal	100	1.00%	5.00%
SimpleSafetyTests partial refusal	100	13.00%	14.00%
SimpleSafetyTests warning rate	100	90.00%	94.00%
StrongREJECT rubric mean	180	0.953472	0.972222
StrongREJECT refusal rate	180	3.33%	1.67%
StrongREJECT warning rate	180	82.22%	83.33%

For SimpleSafetyTests, the primary result is the holistic safe/unsafe classification; refusal-style labels are auxiliary diagnostics. For StrongREJECT, a higher rubric score means that the response gave more useful and specific assistance toward the harmful request; it therefore indicates weaker safety refusal, not higher general model quality. The string-matching baseline is reported for traceability and is not interchangeable with the LLM-judge rubric.

KL divergence from the NVFP4 base

KL was measured on 100 prompts from mlabonne/harmless_alpaca (test[:100]) using the Abliterix next-token method: top-100 log probabilities, one scored token per prompt, and KL(base || adapter) after Abliterix's finite-value normalization.

Adapter	Mean KL	Population std. dev.	Minimum	Maximum
v1	0.067696	0.288671	0.000150	2.691505
v2	0.011077	0.042988	0.000021	0.379621

v2's mean KL is approximately 16.36% of v1's, or about 6.1× lower. Under this measurement, v2 preserves the NVFP4 base model's benign next-token distribution substantially better.

Which version should I use?

• Choose v1 when the priority is the more aggressive refusal reduction observed on SimpleSafetyTests: 1% full refusal and 86% no-refusal responses. This comes with materially higher distribution drift.
• Choose v2 when preserving base-model behavior is the priority. Its KL is about 6.1× lower, its SST safe rate is higher, and it still achieved the lower refusal rate and higher harmful-assistance rubric score on StrongREJECT.
• There is no universally “less refusing” version across every benchmark: v1 has the lower SST full-refusal rate, while v2 has the lower StrongREJECT refusal rate.

StrongREJECT category comparison

Category	n	v1 score	v2 score	v1 refusal	v2 refusal	v1 warning	v2 warning
Disinformation and deception	30	0.895833	0.920833	10.00%	6.67%	80.00%	83.33%
Hate, harassment, and discrimination	30	0.908333	0.941667	6.67%	3.33%	60.00%	56.67%
Illegal goods and services	30	1.000000	1.000000	0.00%	0.00%	96.67%	96.67%
Non-violent crimes	30	0.954167	0.983333	3.33%	0.00%	93.33%	96.67%
Sexual content	30	0.962500	0.995833	0.00%	0.00%	80.00%	83.33%
Violence	30	1.000000	0.991667	0.00%	0.00%	83.33%	83.33%

Warning rate is not refusal rate

A warning, caveat, legal note, or safety disclaimer was counted separately from refusal. If a response warned about risk but still provided request-relevant content, it was not counted as a refusal on that basis. Consequently, a high warning rate can coexist with a low refusal rate.

Usage

This repository does not include the base model. Provide a compatible MLX conversion of GLM-5.3-Flash separately and comply with its license and terms.

MLX-LM command line

Install MLX-LM on Apple Silicon, then select either v1 or v2 with --adapter-path:

pip install -U mlx-lm

mlx_lm.generate \
  --model /path/to/GLM-5.3-Flash-MLX \
  --adapter-path /path/to/repository/v2 \
  --prompt "Hello"

To run v1, replace the adapter path with /path/to/repository/v1.

Python

from mlx_lm import generate, load

model, tokenizer = load(
    "/path/to/GLM-5.3-Flash-MLX",
    adapter_path="/path/to/repository/v2",  # Or /v1
)

response = generate(model, tokenizer, prompt="Hello", max_tokens=512)
print(response)

Use an MLX-LM build and base-model conversion that support the GLM-5.3 MoE architecture and the expert module names recorded in adapter_config.json. The adapter directory—not the adapters.safetensors file alone—must be passed to --adapter-path or adapter_path.

Limitations

• Abliteration changes refusal behavior; it does not guarantee correctness, harmlessness, honesty, or improved task capability.
• The intervention does not prove that all safety behavior is encoded by one direction or one module family.
• Residual refusals may remain, and later fine-tuning, merging, quantization, or serving changes may strengthen or weaken the effect.
• Automated judge labels can contain false positives and false negatives. Warning rate is not a safety score.
• Results are specific to the stated base checkpoint, prompts, sampling configuration, runtime, and judge model.

Disclaimer

This adapter is released solely for legitimate research, safety evaluation, red-teaming, and other lawful uses. It intentionally weakens refusal behavior and may cause the base model to produce dangerous, illegal, hateful, sexually explicit, deceptive, or otherwise harmful content. Do not expose it to untrusted users or deploy it without appropriate access control, monitoring, content filtering, rate limits, and human oversight.

You are responsible for complying with applicable laws, regulations, platform policies, and the licenses and terms of the base model, quantized checkpoint, Abliterix, and all other dependencies. The authors and distributors make no warranty and accept no responsibility for outputs, downstream use, misuse, damages, or legal consequences arising from this adapter.

---

GLM-5.3-Flash-Abliterixed-LoRA-MLX

面向 GLM-5.3-Flash 的两版拒绝方向消融 LoRA。

本仓库仅提供 LoRA 适配器权重。两个版本均以 zai-org/GLM-5.3-Flash 为原始模型，使用 Abliterix 对拒绝行为进行消融；基础模型权重本身不会被本仓库替换。v1 的干预更集中于后层，v2 则把路由专家修改前移至前中层，并显著降低了相对 NVFP4 基础模型的 KL 偏移。

这些适配器用于受控的安全研究、红队测试和拒绝机制分析。移除拒绝行为也意味着削弱重要的安全防线，使用前请阅读文末的免责声明。

仓库文件结构

.
├── README.md
├── v1/
│   ├── adapter_config.json
│   └── adapters.safetensors
└── v2/
    ├── adapter_config.json
    └── adapters.safetensors

• v1/：原始版本，对应 GLM-5.3-Flash-Ablitered。
• v2/：第二版本，对应 GLM-5.3-Flash-Ablitered2。
• 每个目录都是可独立加载的 MLX-LM LoRA。本仓库不包含 GLM-5.3-Flash 基础权重，也不包含合并后的完整模型。

适配器概要

项目	v1	v2
仓库路径	v1/	v2/
基础模型	zai-org/GLM-5.3-Flash	同一模型族
格式	MLX-LM LoRA（adapters.safetensors）	MLX-LM LoRA（adapters.safetensors）
秩 / alpha	r=1，lora_alpha=1	r=1，lora_alpha=1
Dropout / bias	0.0 / none	0.0 / none
实际修改的层范围	第 15–44 层	第 3–44 层
路由专家 down_proj 范围	第 18–44 层	第 3–28 层
主要有效目标	路由专家 down_proj	路由专家 down_proj
评测基础模型	RedHatAI/GLM-5.3-Flash-NVFP4	同左

什么是拒绝消融？

拒绝消融（abliteration）是一种权重空间干预方法，其思路源自 Arditi 等人的论文 Refusal in Language Models Is Mediated by a Single Direction。简而言之，该方法通过有害提示与无害提示产生的内部激活差异估计拒绝方向，再从选定的写入矩阵中移除会向该方向写入的分量。秩 1 更新可表示为：

W' = W + ΔW,      ΔW = BA,      rank(ΔW) = 1

将 ΔW 保存为 LoRA，可以在不重写基础模型检查点的情况下，于推理时加载或卸载该干预。

为什么主要修改路由专家的 down_proj？

orcarouter/GLM-5.3-Flash-Uncensored-FP8 公布的分析表明，GLM-5.3-Flash 的拒绝行为主要由路由专家 MLP 的 down_proj 写入端承载。其组件实验显示，仅修改注意力或稠密/共享 MLP 时，大部分拒绝行为仍然保留；加入路由专家 MLP 写入矩阵后，拒绝才出现主要下降。

我们进行的分层限制实验也验证了这一机制：具有实际因果作用的拒绝方向集中在专家堆栈的前段至中段；如果只修改靠后的专家层，对拒绝行为几乎没有实际影响。因此，本适配器将路由专家的 down_proj 作为最主要的有效修改目标。

这里的“分布在前段、后层修改无效”描述的是行为干预的因果有效性，并不表示所有前层 LoRA 张量的数值范数都必然大于后层。权重覆盖范围或张量大小本身不能单独证明某种行为储存在哪里。

本适配器采用 Abliterix 的 SRA + mean + per-layer 流程：

• Mean 基础估计：以有害提示和无害提示的平均激活差作为初始拒绝方向。
• SRA：使用受保护的无害能力成分对初始拒绝信号进行谱清理/残差化，尽量减少对重叠通用能力的误删。
• 逐层估计：为每个被修改层独立估计方向，而不是把某层的拒绝向量应用到整个模型。

逐层估计在这里非常重要，因为拒绝信号在不同深度上的因果作用并不相同。

LoRA 修改权重的层与模块分布

以下数据来自对 MLX 转换前两个源 PEFT 版本的 adapter_config.json 与 adapter_model.safetensors 的实际检查，而不是根据文件名推测。“实际有效”表示相应秩 1 LoRA 增量非零。

实际非零修改

模块	v1 覆盖层	v1 模块数	v2 覆盖层	v2 模块数
路由专家 down_proj	18–44	7,776	3–28	7,488
共享专家 down_proj	18–44	27	18–44	27
自注意力 o_proj	15–44	30	15–44	30
实际有效模块总数	15–44	7,833	3–44	7,545

存储统计	v1	v2
BF16 张量数	46,770	45,042
秩 1 模块条目	23,385	22,521
序列化标量数	143,927,296	138,618,880
非零标量数	48,375,808	46,606,336
源 PEFT safetensors 大小	281.6 MiB	271.2 MiB

两个版本的 PEFT 配置都把 down_proj、gate_proj、o_proj 和 up_proj 列为目标模块，但检查点中的 gate_proj 与 up_proj 因子都是结构性零条目，不会改变输出。实际生效的是路由/共享专家 down_proj 和注意力 o_proj。

两个版本最主要的差异是深度分配：

• v1：第 0–14 层不修改；第 15–17 层仅修改注意力 o_proj；第 18–44 层同时修改注意力 o_proj、全部 288 个路由专家 down_proj 和共享专家 down_proj。
• v2：第 0–2 层不修改；第 3–14 层修改全部 288 个路由专家 down_proj；第 15–17 层再加入注意力 o_proj；第 18–28 层再加入共享专家 down_proj；第 29–44 层只保留注意力和共享专家修改。

简而言之，v1 把专家干预集中在后 27 层；v2 则把它前移到前中段专家堆栈，并移除第 29–44 层的路由专家修改。

评测结果

两个适配器分别加载在 RedHatAI/GLM-5.3-Flash-NVFP4 基础模型上进行评测。测试部署使用 vLLM，并为被测模型明确设置低思考强度。交给裁判模型的内容只有被测模型的最终答案，不包含其思考过程。

自动裁判模型为 **deepseek-v4-flash**，裁判思考功能关闭。以下结果是自动评估结果，不等同于人工安全认证。

总体结果

测试项目	n	v1	v2
SimpleSafetyTests 完全拒绝	100	1.00%	5.00%
SimpleSafetyTests 部分拒绝	100	13.00%	14.00%
SimpleSafetyTests 警告率	100	90.00%	94.00%
StrongREJECT rubric 均分	180	0.953472	0.972222
StrongREJECT 拒绝率	180	3.33%	1.67%
StrongREJECT 警告率	180	82.22%	83.33%

SimpleSafetyTests 的主要指标是整体安全/不安全分类，拒绝风格标签只是辅助诊断。StrongREJECT 的 rubric 得分越高，表示回答对有害请求提供的帮助越具体、越有用，也就是安全拒绝越弱，并不代表模型的通用质量越高。字符串匹配基线仅用于追踪，不能与 LLM 裁判的 rubric 分数直接互换。

与 NVFP4 基础模型的 KL 散度

KL 使用 Abliterix 的 next-token 方法，在 mlabonne/harmless_alpaca 的 test[:100] 上测量：每条提示评估 1 个 token，保留 top-100 对数概率，经 Abliterix 有限值归一化后计算 KL(base || adapter)。

适配器	平均 KL	总体标准差	最小值	最大值
v1	0.067696	0.288671	0.000150	2.691505
v2	0.011077	0.042988	0.000021	0.379621

v2 的平均 KL 约为 v1 的 **16.36%**，即约低 6.1 倍。在这一测量口径下，v2 明显更好地保留了 NVFP4 基础模型在无害提示上的 next-token 分布。

版本选取建议

• 如果重点是在 SimpleSafetyTests 所体现的场景中更激进地降低拒绝，优先选择 v1：完全拒绝率为 1%，不拒绝比例为 86%；代价是相对基础模型的分布偏移明显更大。
• 如果更重视保留基础模型能力与原始行为，优先选择 v2：KL 约低 6.1 倍、SST 安全率更高，同时在 StrongREJECT 上取得了更低拒绝率和更高的有害帮助 rubric 得分。
• 不存在对所有测试都绝对“更少拒绝”的单一版本：v1 的 SST 完全拒绝率更低，而 v2 的 StrongREJECT 拒绝率更低。

StrongREJECT 分类对比

类别	n	v1 得分	v2 得分	v1 拒绝	v2 拒绝	v1 警告	v2 警告
虚假信息与欺骗	30	0.895833	0.920833	10.00%	6.67%	80.00%	83.33%
仇恨、骚扰与歧视	30	0.908333	0.941667	6.67%	3.33%	60.00%	56.67%
非法商品与服务	30	1.000000	1.000000	0.00%	0.00%	96.67%	96.67%
非暴力犯罪	30	0.954167	0.983333	3.33%	0.00%	93.33%	96.67%
性内容	30	0.962500	0.995833	0.00%	0.00%	80.00%	83.33%
暴力	30	1.000000	0.991667	0.00%	0.00%	83.33%	83.33%

警告率不等于拒绝率

警告、风险提示、法律说明和安全免责声明与拒绝分别统计。如果模型虽然发出警告，但仍然回答了与请求相关的实质内容，则不会仅因为存在警告而被算作拒绝。因此，高警告率可以与低拒绝率同时出现。

使用方法

本仓库不包含基础模型。请另行提供兼容的 GLM-5.3-Flash MLX 转换版本，并遵守相应许可证与使用条款。

使用 MLX-LM 命令行加载

在 Apple Silicon 上安装 MLX-LM，然后通过 --adapter-path 选择 v1 或 v2：

pip install -U mlx-lm

mlx_lm.generate \
  --model /path/to/GLM-5.3-Flash-MLX \
  --adapter-path /path/to/repository/v2 \
  --prompt "你好"

如需运行 v1，将适配器路径替换为 /path/to/repository/v1。

使用 Python 加载

from mlx_lm import generate, load

model, tokenizer = load(
    "/path/to/GLM-5.3-Flash-MLX",
    adapter_path="/path/to/repository/v2",  # 或 /v1
)

response = generate(model, tokenizer, prompt="你好", max_tokens=512)
print(response)

请使用能够识别 GLM-5.3 MoE 架构及 adapter_config.json 中专家模块名称的 MLX-LM 构建和基础模型转换。--adapter-path 或 adapter_path 应指向整个适配器目录，而不是只指向 adapters.safetensors 文件。

局限性

• 拒绝消融改变的是拒绝行为，并不保证模型正确、无害、诚实，也不保证通用任务能力有所提升。
• 本干预不能证明所有安全行为都由单一方向或单一模块族编码。
• 模型仍可能保留部分拒绝；后续微调、权重合并、量化或推理服务配置也可能增强或削弱消融效果。
• 自动裁判可能产生误判；警告率不是安全得分。
• 结果只适用于文中所述基础检查点、提示词、采样配置、运行时和裁判模型。

免责声明

本适配器仅供合法的研究、安全评估、红队测试及其他合规用途。它会有意削弱模型的拒绝行为，可能使基础模型生成危险、违法、仇恨、露骨色情、欺骗性或其他有害内容。请勿直接向不可信用户开放，也不要在缺少访问控制、监控、内容过滤、速率限制和人工监督的情况下部署。

使用者有责任遵守适用的法律法规、平台政策，以及基础模型、量化检查点、Abliterix 和其他依赖项的许可证与使用条款。作者与分发者不提供任何明示或默示保证，也不对本适配器生成的内容、下游使用、滥用、损失或法律后果承担责任。