NYUAD-ComNets/Gemma4_safe_unsafe_prompt

Finetuned Gemma 4 for Arabic Users Prompts classification into [Safe, Unsafe]

The rise of large language models has led to the spread of Arabic Prompts as a key form of prompt engineering (or interaction with artificial intelligence), enabling users to communicate effectively with AI systems in their native language to achieve precise, context-aware results. While these contents can be helpful and informative, they are also increasingly being used to spread fraud, deception, adult content,
harassment, harm to others, hate speech, and self-harm. Consequently, there is a growing demand for precise analysis of content in Arabic prompts.

This work used Gemma 4 to effectively identify harmful content within Arabic prompts. The evaluation is conducted using a dataset of Arabic prompts proposed in the ArabicNLP ArGuard 2026 challenge. The results underscore the capacity of google/gemma-4-E2B fine-tuned with Arabic prompts, to deliver superior performance.

The proposed solutions offer a more nuanced understanding of user prompts for accurate and efficient Arabic content moderation systems.

Finetuned Gemma 4 Embedding Model with last token

import torch 
from sentence_transformers import SentenceTransformer
from sentence_transformers.models import Transformer, Pooling
import pandas as pd
import numpy as np

model_id = 'NYUAD-ComNets/Gemma4_safe_unsafe_prompt'
device = "cuda" if torch.cuda.is_available() else "cpu"

df=pd.read_csv('train.csv')

word_embedding_model = Transformer(
    model_id, 
    model_args={"torch_dtype": torch.float16}
)
        

pooling_model = Pooling(
    word_embedding_model.get_word_embedding_dimension(), 
    pooling_mode='lasttoken' 
)
        

custom_emb_model = SentenceTransformer(
    modules=[word_embedding_model, pooling_model],
    device=device
)

embeddings = custom_emb_model.encode(
    df.prompt.values, 
    batch_size=32, 
    show_progress_bar=True,
    convert_to_numpy=True
)

print(embeddings.shape)

Finetuned Gemma 4 Embedding Model with average pooling

import torch 
from sentence_transformers import SentenceTransformer
from sentence_transformers.models import Transformer, Pooling
import pandas as pd
import numpy as np

model_id='NYUAD-ComNets/Gemma4_safe_unsafe_prompt'

word_embedding_model = Transformer(model_id, model_args={"torch_dtype": torch.float16})
        
pooling_model = Pooling(word_embedding_model.get_word_embedding_dimension(), pooling_mode='mean')
        
custom_emb_model = SentenceTransformer(modules=[word_embedding_model, pooling_model])

df=pd.read_csv('dev.csv')

embeddings_array=[]


for k in df.prompt.values:
    
        vector = custom_emb_model.encode([k])

        embeddings_array.append(vector)
        
dev_matrix = np.array(embeddings_array)

np.save('dev_embd_Gemma4.npy',dev_matrix)

Prepare Dataset for inference

import pandas as pd
from datasets import Dataset

from datasets import load_dataset

df=pd.read_csv('dev_with_label.csv')
df

raw_data=[]

for k in range(df.shape[0]):
     te=df.prompt.values[k]
     lab=df.label.values[k]
     raw_data.append({"text":te,"label":lab})


SYSTEM_PROMPT = "classify the text into one of the following: [Safe, Unsafe]"

formatted_rows = []
for item in raw_data:
    conversation = {
        "messages": [
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": item["text"]},
            {"role": "assistant", "content": item["label"]}
        ]
    }
    formatted_rows.append(conversation)

hf_dataset = Dataset.from_list(formatted_rows)

hf_dataset.to_json("dev_dataset.jsonl")

dataset = load_dataset("json", data_files="dev_dataset.jsonl")

print(dataset)

Finetuned Gemma 4 for Inference

from transformers import AutoModelForMultimodalLM, AutoProcessor
from transformers import pipeline, GenerationConfig, pipeline

model_id ="NYUAD-ComNets/Gemma4_safe_unsafe_prompt"

# Load Model with PEFT adapter
model = AutoModelForMultimodalLM.from_pretrained(
  model_id,
  device_map="auto",
  dtype="auto",
)
processor = AutoProcessor.from_pretrained(model_id)


config = GenerationConfig.from_pretrained(model_id)
config.max_new_tokens = 256
config.eos_token_id = [processor.tokenizer.convert_tokens_to_ids("<turn|>")]


pipe = pipeline("text-generation", model=model, tokenizer=processor.tokenizer)


lis=[]
pred=[]

n=0

for k,lab in zip(range(df.shape[0]),df.label.values):
    
        test_sample = dataset['train'][k]

        prompt = processor.tokenizer.apply_chat_template(test_sample["messages"][:2], tokenize=False, add_generation_prompt=True)

        outputs = pipe(text_inputs=prompt, generation_config=config)
        
        p=outputs[0]['generated_text'][len(prompt):].strip().removesuffix("<turn|>")

        print(p)
        
        pred.append(p)
        lis.append(lab)
        d=pd.DataFrame({'label':lis,'predict':pred})  
        print(sum(d.label==d.predict))

We used Low-Rank Adaptation (LoRA) as the Parameter-Efficient Fine-Tuning (PEFT) method for fine-tuning utilizing the unsloth framework.

BibTeX entry and citation info

@misc{aldahoul,
      title={NYUAD at ArGuard Shared Task: Multimodal Embedding Models for
Detecting Arabic Hateful Memes and Unsafe Prompts}, 
      author={Nouar AlDahoul and Yasir Zaki},
      year={2026},
      eprint={},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={}, 
}