Sik0213/shield-v-rtfs-faceswap-baseline

SHIELD-V RTFS face-swap baseline

This is the single 9,884,507-parameter model selected from the 2026-11-06 RTFS-only paired-original experiment. It outputs one score for a ten-second video-only clip: higher means more evidence of a face swap. The score is a sigmoid output, not a calibrated probability. The operating threshold is 0.26435837, selected on Validation under a maximum 10% Real false-positive-rate policy.

This is a research baseline for face-swap detection. Its training set contains no explicit AI-background-only or wholly AI-generated examples. Its face-centered crop may include some surrounding background pixels, but background manipulation detection is unvalidated. It is not the original SHIELD design's full-frame detector.

Files

File	Purpose
best.pt	Original verified checkpoint: model weights and training state
model_config.json	Input contract, labels, threshold, architecture, hashes
shield_v/	Model, training, preprocessing, and data-validation source
video_preprocess.py	Standalone inference crop matching the frozen native-frame recipe
prepare_dataset.py	Build portable face-crop frame caches from the selected source videos
inference.py	One-video command-line inference
api.py	Local REST API: GET /health, GET /model, POST /predictions
verify_package.py, verification_results.json	Public-video smoke test and the observed GPU verification result
data/selected.jsonl	Source selection, labels, original groups, windows, and Train/Validation/Test assignments; no video data
requirements.txt	Python dependencies, including pinned SISA and FLA Git revisions

Checkpoint SHA-256: 8a9fcbf2208893e689818c3317a369c9c7b57256426b20f7e4f76fb5ab49d76d. Load only this checkpoint for the reported results. torch.load(..., weights_only=True) is used in the provided inference code.

Dataset, labels, and split

Source: Kontur.AI RTFS-10K, revision 83063ae33220e8da968587d4365bd986c7005835. The dataset card lists CC BY-SA 4.0 and identifies the original video links. Source videos are not included here; the selection file retains the source links and file IDs. The dataset's original_videos files map to Real (0); successful inswapper and uniface face-swap files map to Fake (1).

The frozen data/selected.jsonl SHA-256 is f65547442715e993ed0b2225e7f839d94ff9b80905f9ef5d1232621076a4213b. It contains only source metadata, not decoded frames or original videos. The selection file includes metadata derived from RTFS-10K and retains source attribution; redistribute that metadata under the source dataset's CC BY-SA 4.0 terms. This statement does not assign a license to the model weights.

Selection required at least ten seconds, at least 2 FPS, and the source's indoor tag; Fake entries also required the dataset's successful-generation flag. One central ten-second window was selected from each of 6,340 eligible videos. The split was assigned before frame extraction using recording/source-photo grouping. A Real source and its Fake derivatives were kept in the same split. The included manifest has 1,338 source groups and zero groups crossing splits. Identity independence across different recordings was not fully verified.

Split	Real	Fake	Total
Train	1,050	4,450	5,500
Validation	211	216	427
Test	232	181	413

Training used paired-original sampling: 1,049 eligible original groups, with one unpaired original omitted from training batches. Each epoch exposed 2,098 Real and 2,098 Fake clips by rotating the two Fake methods. The source files themselves and the fixed Validation/Test splits were not changed.

Video preprocessing

1. Decode a complete central ten-second window; take the first source frame at or after each 0.5-second target, up to 20 distinct frames. Reject missing frames or gaps over 0.25 second. Audio is ignored.
2. Letterbox each RGB frame to 320×180 for face localization. Detect the largest face with OpenCV's frontal-face Haar cascade at frame indices 0, 5, 10, 15, and 19; interpolate centers across frames. Use a crop height of 1.6 times the median detected face height, bounded to 80–180 pixels, at 16:9. If every detection fails, use the whole frame.
3. Decode the same source PTS frames at native quality, letterbox to 1280×720, apply the corresponding crop, then resize to 320×180 with OpenCV area interpolation. This adds no lossy re-encode. The training frame cache was lossless RGB uint8 NPZ.
4. At model input, scale RGB to [0,1] and normalize with ImageNet mean [0.485,0.456,0.406] and std [0.229,0.224,0.225]. Tensor shape is [batch,20,3,180,320].

The inference implementation is in video_preprocess.py; prepare_dataset.py applies it to every frozen source row. Source selection and RGB letterbox sampling code are under shield_v/prepare_rtfs.py and shield_v/preprocess.py. Real and Fake source files may retain different codec/compression characteristics; this remains a possible dataset shortcut.

To recreate portable face-crop caches from the pinned source files, use the same extracted RTFS directory layout referenced by data/selected.jsonl:

python prepare_dataset.py --raw-root /path/to/rtfs-10k --output /path/to/prepared-facecrop

This checks the frozen selection, file lengths, and split groups, then writes a lossless frame cache and manifest. Cache compression and manifest hashes can differ from the historical run without changing the decoded pixels. It does not download or redistribute source videos.

Model and training

MobileNetV3-Large features (ImageNet1K V2 initialization) produce 4×4 spatial tokens per frame. A 384-wide sequence of 320 video tokens plus one last readout token passes through GDN-A, the same GDN-B mixer twice, and SISA1 once. The final readout yields one logit. The trained model has 9,884,507 parameters.

Training used unweighted BCEWithLogitsLoss, AdamW, CNN learning rate 3e-5, new-module learning rate 3e-4, weight decay 0.01 with bias/norm/SISA-lambda exclusions, 5% warmup then cosine decay, gradient clipping 1.0, microbatch 4 × accumulation 8 (effective batch 32), at most 30 epochs, patience 5, and the highest Validation AUROC checkpoint. Seed was 42; the selected checkpoint came from the twelfth epoch. Train-only conference augmentation included resolution reduction, limited blur, and noise.

The corresponding training command is:

python -m shield_v.train --manifest /path/to/prepared-facecrop/manifest.jsonl --run-dir /path/to/new-run --epochs 30 --batch-size 4 --effective-batch 32 --workers 8 --seed 42 --patience 5 --target-fpr 0.1 --allow-prototype --class-balance none --augmentation conference --paired-original-batches --fused-adamw --execution-mode channels-last

This reproduces the training recipe. Hardware, library versions, and stochastic operations can prevent a bit-for-bit identical retrain; use the included checkpoint for the reported metrics.

Evaluation

RTFS split	AUROC	Fake recall at threshold	Real specificity at threshold
Validation	0.97586449	206/216 = 95.37%	190/211 = 90.05%
Test	0.99048628	180/181 = 99.45%	210/232 = 90.52%

The 0.99048628 is the result on the 413-video RTFS Test split. This Test split had already been inspected during model development, so the number is an exploratory within-dataset result, not an untouched final generalization estimate. An additional Fake personal video was misclassified as Real. The model has not been validated for AI-background manipulation, whole-video generation, or general video-call deployment. No personal video is included in this repository.

On 2026-11-10, this package was transferred to a Runpod NVIDIA L4 and checked with two public RTFS Test videos. The package-produced 20-frame arrays were pixel-identical to the historical face-crop caches. On that L4, package inference and inference from those caches returned identical scores, and REST matched the CLI. The L4 scores differed from the earlier stored evaluation scores: Real 0.13117322 versus 0.15507847, Fake 0.46125942 versus 0.42393497; both classifications stayed correct. The cause of this cross-run numerical difference remains unconfirmed. The full 413-video AUROC was not recomputed on the L4. Details are in verification_results.json.

The historical Test evaluator processed batches of four, while the CLI and REST API process one video per call. Batch shape is a possible source of the score difference, but it was not isolated in this check. Treat the published threshold and AUROC as values from the historical evaluation configuration until a full deployment-mode validation is run.

Relation to the SHIELD-V design PDF

Reference: SHIELD_V_Recurrent_GDN_SISA_GitHub_설계도.pdf, especially pages 2-8. This package is a face-swap research baseline, not a claim that the entire proposed video-security service has been implemented or validated.

Design requirement	This checkpoint and package	Status
One participant's whole camera frame at RGB 320×180, without face detection or crop (pp. 2-3)	Dynamically localizes a face, takes a face-centered 16:9 crop with some surrounding context, then resizes to 320×180.	Changed input field of view; background and objects outside the crop are not analyzed.
Real versus any AI-generated or AI-manipulated video, including faces, backgrounds, and fully generated video (pp. 2, 7)	Trained only on RTFS-10K Real videos and inswapper/uniface face swaps. No explicit AI-background-only or fully generated training examples.	Narrower task; performance on background or whole-video generation is unknown.
Recent ten seconds, 2 FPS, 20 frames, 320×180, MobileNetV3-Large features, 4×4 tokens per frame, 321 causal tokens, GDN-A → shared GDN-B twice → one SISA1, ≤10M parameters (pp. 2-6)	Uses these dimensions and block order after the face-centered crop; measured parameter count is 9,884,507.	Architecture and clip shape retained, subject to the input-field change above.
Initial optimizer, loss, batch, scheduler, clipping, early stopping, and Validation-AUROC checkpoint recipe (p. 7)	Uses unweighted BCEWithLogitsLoss, AdamW, the specified learning rates and decay, 5% warmup plus cosine decay, microbatch 4 × accumulation 8, clipping 1.0, up to 30 epochs, patience 5, and best Validation AUROC.	Initial training recipe retained; paired-original batch sampling and conference augmentation were added as training choices.
Source-group split before extracting overlapping windows; original and Fake derivatives stay together (p. 8)	Groups recordings and source photos before the selected ten-second window; no recorded source groups cross splits. Person identity across unrelated recordings was not independently verified.	Partially verified; identity separation is not established.
Choose the warning threshold using Validation; report AUROC, fixed-FPR recall, warning false alarms and latency, including concurrent p95 (p. 8)	Threshold 0.26435837 was chosen on Validation with an added 10% maximum Real-FPR policy. RTFS Validation/Test AUROC and class recalls were recorded. Test was repeatedly inspected; deployment-mode full-Test metrics, concurrent p95 and detection delay were not measured.	Partial evaluation; 0.99048628 is exploratory within-dataset Test AUROC.
Re-evaluate a rolling ten-second window every two seconds, cache per-participant CNN features, and return analyzing/unavailable status when appropriate (pp. 6-8)	CLI and local REST API score one complete video/window at a time and recompute its features. They do not implement a live ring buffer, two-second update scheduler, participant ID, or the complete service status/metadata contract.	Service integration remains to be built and timed.

The source Real and Fake files may also differ in codec and compression; this dataset shortcut remains unresolved. No cross-dataset or independent deployment claim follows from the RTFS Test result.

Run locally

Use Linux, Python 3.10–3.12, and an NVIDIA CUDA GPU with BF16 support. The recorded environment used PyTorch 2.10.0, torchvision 0.25.0, and CUDA 12.8 wheels. Install those wheels first, then the remaining dependencies:

python -m pip install torch==2.10.0 torchvision==0.25.0 --index-url https://download.pytorch.org/whl/cu128
python -m pip install -r requirements.txt
python inference.py /path/to/video.mp4
# Optional reproducibility check: downloads two public RTFS Test videos
python verify_package.py

For a local REST service:

uvicorn api:app --host 127.0.0.1 --port 8000
curl -F "file=@/path/to/video.mp4" http://127.0.0.1:8000/predictions

The endpoint accepts one MP4 file of at most 200 MiB and returns the selected central ten-second window's score and decision. It removes the temporary upload after each request. This example API has no authentication and is bound to localhost; deployment behind a public endpoint requires an authentication layer. A Hugging Face model repository stores these files but does not automatically run this FastAPI service.

Provenance and limits

The checkpoint's embedded manifest SHA-256 is f24261c808e382a8c1873499d7002cb6de2988ddb7cd9d9ec2d234571f93f2f5; the inference loader checks it, the checkpoint hash, parameter count, and threshold. The original source dataset's labels were used, but person-level separation and all-frame manipulation status were not independently verified. The checkpoint contains optimizer state in addition to model weights; inference.py loads only the model state.

Please cite and follow the RTFS-10K dataset card when using the source material. No independent license for this model's weights is declared in this package.