Aurelius51/lingualsafety-intent-guard

LingualSafety intent guard (LoRA adapter for Gemma-3-4B-IT), final version

This is a LoRA adapter (r=16, alpha=32, attention and MLP projections of the language model) on google/gemma-3-4b-it (Gemma-3-4B-IT, Google DeepMind). It turns Gemma into a binary safety classifier for user messages in English, Hindi, Bengali, Marathi and Punjabi.

It was trained on unsloth/gemma-3-4b-it, an ungated copy whose weight files are byte-identical to google/gemma-3-4b-it (same SHA-256 for both safetensors shards, same tokenizer, same chat template), so it applies unchanged to the official model.

It is the input gate of a defence cascade built for the LingualSafety 1.0 shared task (ICON 2026). Its job is to recognise harmful requests even when they are wrapped in persuasion (logical appeal, authority endorsement, anchoring, priming, misrepresentation, confirmation bias), including requests from harm categories it did not see in the task's released data. When the gate fires, the system returns a plain refusal in the user's language; otherwise the base model answers normally.

Usage

The prompt template lives in cls_prompt.py of the accompanying submission notebook. The classifier's score is the next-token probability of unsafe versus safe after that prompt.

Submission setting: refuse when this classifier's P(unsafe) is at least 0.6, or when the IndicGuard prompt check (l3cube-pune/IndicGuard) gives at least 0.99.

Training data (summary)

• Harmful:
  • all prompts the shared task released for development (7 categories, plain and persuasion-wrapped, 5 languages; plain requests counted three times);
  • unsafe prompts from the IndicGuard dataset (CC-BY-4.0) and OR-Bench-toxic (CC-BY-4.0);
  • harmful requests from categories the task did not release (illegal activity, physical harm, political lobbying and misinformation, malware): English seeds from IndicGuard's Aegis data, rewritten by Gemma-3-4B-IT in the six persuasion styles plus the plain request, and translated with NLLB-200-distilled-600M. Only half of these seeds were used for training; the other half is the held-out test below.
• Benign:
  • safe prompts from IndicGuard;
  • OR-Bench-80K "seemingly toxic" prompts, excluding OR-Bench-Hard-1K;
  • unused Aya Dataset rows (Apache-2.0);
  • topic-matched and persuasion-wrapped benign requests generated with Gemma-3-4B-IT, and NLLB translations of the English portions.
• Excluded: the evaluation items (CatQA, XSTest, JBB-benign, OR-Bench-Hard-1K, the evaluation Aya rows, and the held-out half of the unseen-category seeds).

Held-out results

• Harmful requests from unseen categories, in the benchmark's format (held-out seeds, 7,350 prompts, 5 languages): 93.8% refused at the input at the submission setting (persuasion-wrapped 94.9%, plain 87.5%); 89.7% with this classifier alone at 0.85.
• CatQA (plain harmful questions from unseen categories): 6.8% of answers judged harmful (score 4-5 by a strict Qwen3-8B judge) at the submission setting.
• Benign prompts: extra refusals compared with the base model at the submission setting are +22.8 points on XSTest, +16.0 on JBB-benign, +0.6 on natural Aya prompts and +15.9 on OR-Bench-Hard. With this classifier alone at 0.85 they are +5.8, +5.6, 0.0 and +3.1.
• The task's released prompts are in the training data, so scores on them are not held-out.

Limitations

• Misinformation requests are only covered through the political seeds.
• Plain requests from unseen categories are caught less often than persuasion-wrapped ones.
• The judge-based numbers come from a single strict LLM judge.
• This is a research artefact, not a production moderation system.

Terms

This adapter is a Gemma model derivative and is distributed under the Gemma Terms of Use. Use is subject to the Gemma Prohibited Use Policy.