OmniFake

Paper & Project Page: Omni-Fake: Benchmarking Unified Multimodal Social Media Deepfake Detection

OmniFake is a multimodal media-forensics model fine-tuned from Qwen2.5-Omni-7B. This repository holds the full merged weights. Given one image, audio clip, video, or audio-visual talking-head clip together with a task prompt, it writes a short analysis inside <think> </think> and a final verdict inside <answer> </answer>.

Input	Verdict	Localization (TAMPERED only)
Image	REAL / FULL_SYNTHETIC / TAMPERED	<box>(x1,y1),(x2,y2)</box>, coordinates normalized to 0-999
Audio	REAL / FULL_SYNTHETIC / TAMPERED	<interval>start,end</interval>, in seconds
Video	REAL / FULL_SYNTHETIC / TAMPERED	<box>(x1,y1),(x2,y2)</box>, coordinates normalized to 0-999
Audio-visual clip	REAL / FULL_SYNTHETIC	-

Loading

The architecture and loading code are the same as Qwen2.5-Omni-7B: Qwen2_5OmniForConditionalGeneration and Qwen2_5OmniProcessor in transformers (tested with transformers==4.57.1). Only text output is used; the speech head can be disabled.

Prompts

Use the system prompt You are a helpful assistant. (the chat template default) and one of the user prompts below verbatim. <image>, <audio> and <video> mark where the media goes.

Image:

<image>
Classify this image as REAL, FULL_SYNTHETIC, or TAMPERED.
- REAL: authentic, no AI generation or manipulation.
- FULL_SYNTHETIC: entirely AI-generated.
- TAMPERED: only part of the image is manipulated/edited.
First reason step by step about any inconsistencies or signs of forgery in <think> </think>, then give the final answer in <answer> </answer>.
If TAMPERED, include one bounding box inside the answer as <box>(x1,y1),(x2,y2)</box> (coordinates normalized to 0-999).
i.e., <think> reasoning </think>
<answer>{Answer}</answer>

Audio:

<audio>
Classify this audio as REAL, FULL_SYNTHETIC, or TAMPERED.
- REAL: authentic recording.
- FULL_SYNTHETIC: entirely AI-generated/synthesized.
- TAMPERED: only part of the audio is manipulated/spliced.
First reason step by step about any inconsistencies or signs of forgery in <think> </think>, then give the final answer in <answer> </answer>.
If TAMPERED, include the manipulated time span inside the answer as <interval>start,end</interval> (seconds).
i.e., <think> reasoning </think>
<answer>{Answer}</answer>

Video:

<video>
Classify this video as REAL, FULL_SYNTHETIC, or TAMPERED.
- REAL: authentic, no AI generation or manipulation.
- FULL_SYNTHETIC: entirely AI-generated.
- TAMPERED: only part of the video is manipulated/edited.
First reason step by step about any inconsistencies or signs of forgery in <think> </think>, then give the final answer in <answer> </answer>.
If TAMPERED, include one bounding box inside the answer as <box>(x1,y1),(x2,y2)</box> (coordinates normalized to 0-999).
i.e., <think> reasoning </think>
<answer>{Answer}</answer>

Audio-visual clip:

<video>
This is an audio-visual clip. Classify it as REAL or FULL_SYNTHETIC by checking whether the talking-head and its audio are authentic or AI-generated.
- REAL: authentic audio-visual recording.
- FULL_SYNTHETIC: AI-generated talking-head and/or synthesized speech.
First reason step by step about lip-sync, facial, and audio consistency in <think> </think>, then give the final answer in <answer> </answer>.
i.e., <think> reasoning </think>
<answer>{Answer}</answer>

Input settings

Results depend on matching these settings:

• Image: resized to 364 x 364.
• Video and audio-visual clips: sampled at 2 fps, at most 24 frames (clips up to 12 s), each frame resized to 100,352 pixels (128 x 28 x 28), with the soundtrack fed together with the frames (use_audio_in_video=True). Clips without sound should carry a silent audio track.
• Audio: 16 kHz mono, up to 12 s.
• Decoding: greedy (temperature 0), max_new_tokens=320.

Quick start with ms-swift (all modalities)

pip install "ms-swift==4.5.2" "transformers==4.57.1" qwen-omni-utils soundfile av
export MAX_PIXELS=132496 MIN_PIXELS=132496
export VIDEO_MAX_PIXELS=100352 VIDEO_MIN_PIXELS=100352 FPS=2 FPS_MAX_FRAMES=24
export USE_AUDIO_IN_VIDEO=true SWIFT_AUDIO_LOAD_BACKEND=soundfile_pyav
swift infer --model Zlatan-Ye/OmniFake --use_hf true --model_type qwen2_5_omni \
  --torch_dtype bfloat16 --infer_backend transformers \
  --val_dataset samples.jsonl --split_dataset_ratio 0 \
  --max_new_tokens 320 --temperature 0 --result_path pred.jsonl

samples.jsonl holds one JSON object per line. Put the prompt (including its <image> / <audio> / <video> tag) in the user message and the media path in images, audios or videos:

{"messages": [{"role": "user", "content": "<image>\nClassify this image as REAL, FULL_SYNTHETIC, or TAMPERED.\n..."}], "images": ["example.jpg"]}

Quick start with transformers (image)

import torch
from PIL import Image
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor

repo = "Zlatan-Ye/OmniFake"
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
model.disable_talker()
processor = Qwen2_5OmniProcessor.from_pretrained(repo)

IMAGE_PROMPT = (
    "Classify this image as REAL, FULL_SYNTHETIC, or TAMPERED.\n"
    "- REAL: authentic, no AI generation or manipulation.\n"
    "- FULL_SYNTHETIC: entirely AI-generated.\n"
    "- TAMPERED: only part of the image is manipulated/edited.\n"
    "First reason step by step about any inconsistencies or signs of forgery in <think> </think>, "
    "then give the final answer in <answer> </answer>.\n"
    "If TAMPERED, include one bounding box inside the answer as <box>(x1,y1),(x2,y2)</box> "
    "(coordinates normalized to 0-999).\n"
    "i.e., <think> reasoning </think>\n<answer>{Answer}</answer>"
)

image = Image.open("example.jpg").convert("RGB").resize((364, 364), Image.BICUBIC)
text = (
    "<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n"
    "<|im_start|>user\n<|vision_bos|><|IMAGE|><|vision_eos|>\n" + IMAGE_PROMPT + "<|im_end|>\n"
    "<|im_start|>assistant\n"
)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=320, do_sample=False, return_audio=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])

Parsing the output

import re

def parse(text):
    m = re.search(r"<answer>(.*?)</answer>", text, re.S)
    ans = m.group(1) if m else ""
    label = next((c for c in ("FULL_SYNTHETIC", "TAMPERED", "REAL") if c in ans.upper()), None)
    box = re.search(r"<box>\((\d+),(\d+)\),\((\d+),(\d+)\)</box>", ans)
    interval = re.search(r"<interval>([\d.]+),([\d.]+)</interval>", ans)
    return {
        "label": label,
        "box": [int(v) for v in box.groups()] if box else None,
        "interval": [float(v) for v in interval.groups()] if interval else None,
    }

Intended use and limitations

For research on detecting AI-generated and manipulated media. Verdicts are model predictions, not proof. Accuracy drops on generators and editing methods unlike those it has seen, and on heavily re-compressed or resized media. Do not use it as the sole basis for decisions about people.

License

Apache-2.0, the same as the base model Qwen2.5-Omni-7B (see LICENSE).