RelayFormer: A Unified Local-Global Attention Framework for Scalable Image and Video Manipulation Localization

## Abstract

Abstract Visual manipulation localization (VML) aims to identify tampered regions in images and videos, a task that has become increasingly challenging with the rise of advanced editing tools. Existing methods face two main issues: resolution diversity, where resizing or padding distorts forensic traces and reduces efficiency, and the modality gap, as images and videos often require separate models. To address these challenges, we propose RelayFormer, a unified framework that adapts to varying resolutions and modalities. RelayFormer partitions inputs into fixed-size sub-images and introduces Global-Local Relay (GLR) tokens, which propagate structured context through a global-local relay attention (GLRA) mechanism. This enables efficient exchange of global cues, such as semantic or temporal consistency, while preserving fine-grained manipulation artifacts. Unlike prior methods that rely on uniform resizing or sparse attention, RelayFormer naturally scales to arbitrary resolutions and video sequences without excessive overhead. Experiments across diverse benchmarks demonstrate that RelayFormer achieves state-of-the-art performance with notable efficiency, combining resolution adaptivity without interpolation or excessive padding, unified modeling for both images and videos, and a strong balance between accuracy and computational cost. Code is
available at: https://github.com/WenOOI/RelayFormer.

## 1 Introduction

Visual manipulation localization (VML), encompassing both image and video modalities, is a fundamental task in digital forensics. Its goal is to precisely identify tampered regions within visual content. With the rapid proliferation of advanced editing tools, detecting and localizing such manipulations has become increasingly challenging (see Fig. [1](https://arxiv.org/html/2508.09459v2#S1.F1)(a)).

While recent studies have proposed specialized approaches for either images or videos, existing methods still face two key limitations that hinder their applicability in real-world scenarios. First, resolution diversity poses a significant challenge. In-the-wild content ranges from low-resolution (e.g., 256×256) to 4K, and unlike in standard vision tasks, interpolation can destroy the subtle low-level traces crucial for forensic analysis (Guillaro et al., 2023; Ma et al., 2023). Prior works rely on fixed-resolution training, forcing a trade-off: either down-sampling inputs to a uniform size (e.g., 512×512), which risks losing manipulation artifacts (Guillaro et al., 2023; Su et al., 2025), or padding smaller inputs to a large canvas (e.g., 1024×1024), which incurs substantial computational redundancy (Ma et al., 2023). Furthermore, uniform resizing disproportionately distorts content with non-standard aspect ratios (e.g., 9:19.5 in modern smartphones), further compromising forensic reliability.
Second, the modality gap limits deployment efficiency. Algorithms are often specialized for either images or videos. Image-specific models fail to leverage temporal dependencies, while video-oriented models struggle to generalize to single images. This dichotomy necessitates maintaining two distinct models in real-world systems, increasing both computational cost and complexity.

Manipulation localization in images and videos demands a delicate balance between fine-grained sensitivity and global semantic reasoning. Manipulated regions are typically small and visually subtle, yet their reliable detection often hinges on scene-level consistency cues, such as illumination patterns, object semantics, or temporal coherence across frames. While dense global attention can, in principle, capture such dependencies, it is computationally prohibitive for high-resolution content. As illustrated in Fig. [1](https://arxiv.org/html/2508.09459v2#S1.F1)(b), the global cues essential for manipulation detection are relatively coarse, reflecting scene-level regularities rather than exhaustive pixel-level correspondence. For example, in the splicing case (top-right), inconsistencies often manifest as illumination mismatches across the scene, while in the copy-move case (bottom-right), beyond local artifacts, detection relies on structural redundancy between the duplicated region and its source. These characteristics suggest that sparse yet effective global information propagation is both sufficient and desirable.

Building on this insight, we propose RelayFormer, a unified, efficient, and flexible architecture for VML. The key idea is to leverage structured global-local interactions without incurring the prohibitive cost of dense attention. RelayFormer dynamically partitions inputs into fixed-size sub-images according to resolution, and introduces Global-Local Relay (GLR) tokens, which mediate information exchange through a global-local relay attention (GLRA) mechanism. Acting as information bottlenecks, these tokens iteratively absorb the scene-level consistency cues, transmit compressed semantics across the entire sample, and reinject enriched context back into their respective regions. Unlike prior approaches  (Yang et al., 2021; Su et al., 2025) that reduce computation primarily via sparse attention, RelayFormer is specifically tailored for VML: it dynamically allocates computation according to input resolution while enabling task-oriented global information propagation. This design ensures scalability to arbitrary resolutions and seamless extension to videos.

To comprehensively validate the effectiveness of our framework, we conduct extensive experiments on a wide range of widely used benchmarks, covering both image and video modalities. We further provide detailed quantitative and qualitative analyses to demonstrate how our method consistently achieves superior performance while maintaining efficiency across diverse settings.

This design provides several advantages:

- •
Resolution adaptivity – allocating appropriate computation to each input without interpolation or excessive padding.
- •
Unified modality handling – naturally extending from spatial to spatio-temporal modeling, enabling a single model to process both images and videos.
- •
Efficiency–accuracy trade-off – avoiding costly full-resolution global attention while effectively capturing structured forensic cues.

Overall, our framework bridges the gap between flexibility and efficiency in VML. It offers a scalable solution that supports diverse resolutions and multi-modalities with reduced redundancy, paving the way for practical and real-time forensic applications.

Figure: Figure 1: Illustration of several common types of visual manipulation, including splicing, copy-move, and inpainting. (a) Examples of manipulated regions and their corresponding boundaries generated by these methods. (b) A schematic illustration highlighting the need for both local and global information to accurately localize manipulated regions.
Refer to caption: x1.png

## 2 Related Work

### 2.1 Image Manipulation Localization

Early works (Wu et al., 2022; Wu & Zhou, 2021) focus on extracting visual artifacts across multiple levels. FOCAL (Wu et al., 2023) leverages contrastive learning, while Mesorch (Zhu et al., 2025) combines CNNs and Transformers to capture fine-grained traces. NCL (Zhou et al., 2023) proposes a contrastive framework for boundary-aware learning without requiring pretraining. IML-ViT (Ma et al., 2023) enhances ViT performance on small datasets using high-resolution inputs, multi-scale features, and edge supervision.

Other methods address modality fusion and robustness. TruFor (Guillaro et al., 2023) fuses RGB and noise fingerprints via a Transformer for anomaly detection. CAT-Net (Kwon et al., 2022) adopts dual-stream RGB-DCT learning with JPEG-based pretraining and multi-resolution inputs to improve localization.

### 2.2 Video Manipulation Localization

Video-based methods extend spatial analysis to the temporal domain. VideoFACT (Nguyen et al., 2024) combines forensic and contextual embeddings with deep self-attention to detect various video forgeries. ViLocal (Lou et al., 2025) uses a 3D Uniformer and supervised contrastive learning to uncover subtle inconsistencies from video inpainting. UVL2 (Pei, 2023) integrates local (e.g., edge artifacts) and global (e.g., pixel distribution) features through a CNN-ViT pipeline for robust generalization. VIDNet (Zhou et al., 2021) employs a dual-stream encoder (RGB + ELA), a ConvLSTM decoder, and directional attention to enhance temporal consistency and cross-type robustness.

Existing methods often face a trade-off between efficiency and performance—either compromising manipulation fidelity due to interpolation or incurring high computational costs from full-resolution global modeling. Furthermore, most are modality-specific, supporting either image or video inputs, but not both. These limitations motivate the need for a unified, efficient, and scalable solution to visual manipulation localization across diverse input conditions.

Figure: Figure 2: Overview of our proposed framework, which consists of three main components. First, the input image or video is partitioned into unified local sub-images without interpolation, preserving fine-grained spatial details. Second, we propose the GLRA module to achieve efficient global information propagation. Finally, a carefully designed lightweight mask decoder efficiently produces the prediction masks. For clarity, the positional encoding components are omitted from the figure.
Refer to caption: x2.png

## 3 Method

We present RelayFormer, a unified and modular framework for Visual Manipulation Localization (VML) that scales to arbitrary image resolutions and temporal lengths. The framework is composed of three main components: *Input Unification*, *Global-Local Relay Attention*, and a *Query-based Mask Decoder*. These components together enable efficient spatial-temporal reasoning by balancing global consistency with local expressivity, while ensuring computational scalability.

### 3.1 Input Unification

To unify image and video inputs into a common representation suitable for parallel computation, we decompose all inputs into slightly overlapping local sub-images, which serve as the atomic processing elements in our framework.

##### Image inputs.

Given an image
$x\in\mathbb{R}^{C\times H_{\text{img}}\times W_{\text{img}}},$
we partition it into slightly overlapping sub-images of spatial size
$H_{p}\times W_{p}.$
Let the sliding strides along height and width be $S_{h}$ and $S_{w}$. Padding is applied if the remaining region is smaller than a full sub-image. The number of sub-images along each spatial dimension is

$$ $N_{h}=\left\lceil\frac{H_{\text{img}}-H_{p}}{S_{h}}\right\rceil+1,\quad N_{w}=\left\lceil\frac{W_{\text{img}}-W_{p}}{S_{w}}\right\rceil+1,$ $$

so the total number of sub-images for the image is
$N_{\text{img}}=N_{h}\times N_{w}.$
The resulting tensor has shape
$(N_{\text{img}},C,H_{p},W_{p}).$

##### Video inputs.

For a video
$x\in\mathbb{R}^{T\times C\times H_{\text{vid}}\times W_{\text{vid}}},$
we first merge the batch and temporal dimensions, treating the video as
$(T,C,H_{\text{vid}},W_{\text{vid}}).$
Each frame is partitioned in the same way as images, producing
$N_{\text{vid}}=N_{h}\times N_{w}$
sub-images per frame. The resulting tensor has shape
$(T\cdot N_{\text{vid}},C,H_{p},W_{p}).$

##### Unified representation.

Finally, all sub-images from images and videos are concatenated into a batch of shape

$$ $(B_{\text{total}},C,H_{p},W_{p}),$ $$

where
$B_{\text{total}}=\sum_{\text{images}}N_{\text{img}}+\sum_{\text{videos}}T\cdot N_{\text{vid}}.$
Each sub-image is treated as an independent sample in the subsequent local modeling stage, enabling large-batch parallel computation without explicitly distinguishing between image and video inputs. We provide pseudocode in the Appendix [A.2.1](https://arxiv.org/html/2508.09459v2#A1.SS2.SSS1).

### 3.2 Global-Local Relay Attention (GLRA)

To balance efficiency and expressiveness, we propose Global-Local Relay Attention (GLRA), which enables efficient propagation of global context through a small set of learnable tokens, while retaining fine-grained local modeling. Fig. [3](https://arxiv.org/html/2508.09459v2#S3.F3) shows the detailed structure of GLRA.

##### Local-aware Attention.

For each sub-image $U_{i}$, we apply a ViT patch embedding to obtain patch tokens
$X_{i}\in\mathbb{R}^{P\times d},$
where $P$ is the number of tokens and $d$ is the feature dimension.
We append a small set of learnable Global-Local Relay [GLR] tokens $T_{i}\in\mathbb{R}^{m\times d}$ to each sub-image:

$$ $[T_{i}^{(l)},X_{i}^{(l)}]=\mathrm{SelfAttn}_{\text{local}}([T_{i}^{(l-1)};X_{i}^{(l-1)}]),$ (1) $$

where $l=1,\dots,L$ represents the layer. In this stage, the [GLR] tokens both relay global information obtained from previous layers and absorb localized details from their corresponding sub-images.

##### Relay-based Global Attention.

To enable global information exchange, we aggregate [GLR] tokens from all sub-images:

$$ $T_{\text{flat}}=\mathrm{Concat}_{j=1}^{N_{i}}T_{j}\in\mathbb{R}^{(N_{i}\cdot m)\times d},$ (2) $$

where $N_{i}$ denotes the number of sub-images in the sample. Each [GLR] token is encoded with temporal index, spatial location, and token identity using 4D Rotary Positional Embeddings (RoPE) (Su et al., 2024; Wang et al., 2024). The global attention step is then:

$$ $T_{\text{updated}}=\mathrm{SelfAttn}_{\text{global}}(\mathrm{RoPE}_{4D}(T_{\text{flat}})).$ (3) $$

After global attention, the updated [GLR] tokens are injected back into their corresponding sub-images, enabling iterative information relay:
1) in the local attention stage, [GLR] tokens transmit global context into local sub-images while gathering new local evidence;
2) in the global attention stage, they exchange these enriched representations with [GLR] tokens of other sub-images.

##### Parameter-efficient strategy.

Using shared parameters for local sub-module and global sub-module would reduce performance because they have conflicting goal. Shared weights lead to poor performance in both. While conceptually separating local and global attention into two distinct Transformer layers is straightforward, this naive approach doubles the parameter count overhead for each such block.

Our core motivation stems from the hypothesis that the computational processes for local and global attention, while functionally distinct, share a substantial underlying structure. To capitalize on this insight, we propose a parameter-efficient strategy. We maintain a single, shared Transformer backbone layer for both the local and global attention computations. To induce the necessary functional specialization, we introduce two distinct adaptation modules (e.g., LoRA (Hu et al., 2022) or Adapters (Poth et al., 2023)), one for each attention mechanism. Specifically, the shared backbone layer learns the common, foundational features of the attention mechanism. The adaptation module for local attention learns the specific residual transformation required to specialize the shared function for processing fine-grained patterns, while the module for global attention learns the residual required for long-range, contextual reasoning. This approach allows us to achieve the expressive power and performance nearly identical to a two-layer model, but with only a marginal increase in parameters over a single-layer baseline, thereby achieving a superior trade-off between performance and efficiency. We provide a more detailed discussion and implementation details of this in the Appendix [A.2.2](https://arxiv.org/html/2508.09459v2#A1.SS2.SSS2).

Figure: Figure 3: Detailed architecture of the proposed Global-Local Relay Attention (GLRA) module.
Refer to caption: x3.png

##### 4D RoPE Formulation and Extrapolation.

We decompose the hidden dimension of each token into five groups: temporal ($T$), token index ($id$), vertical ($H$), horizontal ($W$), and the remaining. Specifically, for a token vector:

$$ $x=[x_{T},x_{id},x_{H},x_{W},x_{rem}],$ $$

where $x_{T}\in\mathbb{R}^{d_{T}},x_{id}\in\mathbb{R}^{d_{id}},x_{H}\in\mathbb{R}^{d_{S}},x_{W}\in\mathbb{R}^{d_{S}},x_{rem}\in\mathbb{R}^{d_{rem}}$, with $d_{T}+d_{id}+2d_{S}+d_{rem}=d$.

For each group, a standard 1D RoPE (Su et al., 2024) is applied independently with the corresponding positional index (temporal id, token id, height id, width id).
Formally, for a sub-vector $x_{g}\in\mathbb{R}^{d_{g}}$ and index $p_{g}$, we apply:

$$ $\mathrm{RoPE}(x_{g}^{(2i)},x_{g}^{(2i+1)})=\begin{bmatrix}x_{g}^{(2i)}\cos(p_{g}\theta_{i})-x_{g}^{(2i+1)}\sin(p_{g}\theta_{i})\\ x_{g}^{(2i)}\sin(p_{g}\theta_{i})+x_{g}^{(2i+1)}\cos(p_{g}\theta_{i})\end{bmatrix},$ $$

where $\theta_{i}=10000^{-2i/d_{g}}$.

The final rotated embedding is:

$$ $\mathrm{RoPE}_{4D}(x)=[\mathrm{RoPE}(x_{T}),\mathrm{RoPE}(x_{id}),\mathrm{RoPE}(x_{H}),\mathrm{RoPE}(x_{W}),x_{rem}].$ $$

This formulation applies independent rotary encodings across temporal, token index, and spatial dimensions, equipping our model with strong extrapolation capabilities to arbitrary resolutions.

### 3.3 Query-based Mask Decoder

To avoid decoding becoming a computational bottleneck, we design a lightweight query-based Transformer decoder, inspired by Mask2Former (Cheng et al., 2022). Given the reassembled feature map $F\in\mathbb{R}^{H_{f}\times W_{f}\times d}$, we first project it into a lower-dimensional space $\tilde{F}\in\mathbb{R}^{H_{f}\times W_{f}\times d_{low}}$. A small set of learnable queries $Q\in\mathbb{R}^{M_{f}\times d}$ then interacts with the projected feature map.

The decoder is composed of $K$ stacked layers. At the $k$-th layer ($k=1,\dots,K$), query features are updated via a cross-attention followed by a self-attention operation:

$$ $Q^{(k)^{\prime}}=\mathrm{CrossAttn}(Q^{(k-1)},\tilde{F}),$ (4) $$

$$ $Q^{(k)}=\mathrm{SelfAttn}(\mathrm{RoPE}(Q^{(k)^{\prime}})).$ (5) $$

Finally, a gating MLP assigns weights to each query, modulating its contribution to the predicted manipulation masks.

### 3.4 Loss Function

Following previous methods (Ma et al., 2023), we adopt a combination of binary cross-entropy (BCE) loss and edge loss.
The overall loss is defined as:

$$ $\mathcal{L}=\mathcal{L}_{\text{BCE}}(P,M)+\lambda\cdot\mathcal{L}_{\text{Edge}}(P\odot M_{e},M\odot M_{e})$ (6) $$

where $P$ is the predicted mask, $M$ is the ground truth, and $M_{e}$ is the edge mask.

The edge loss applies BCE on the edge regions to emphasize boundary accuracy:

$$ $\mathcal{L}_{\text{Edge}}(P\odot M_{e},M\odot M_{e})=\mathcal{L}_{\text{BCE}}(P\odot M_{e},M\odot M_{e})$ (7) $$

Here, $\lambda$ is a weighting factor balancing the two loss terms.

## 4 Experiments

##### Datasets.

In our experiments, we conducted comprehensive evaluations using a diverse set of benchmark datasets, including CASIA v1.0 (Dong et al., 2013), CASIA v2.0 (Dong et al., 2013), Columbia (Hsu & Chang, 2006), Coverage (Wen et al., 2016), NIST16 (Guan et al., 2019), IMD2020 (Novozamsky et al., 2020), DAVIS2016 (Perazzi et al., 2016), and MOSE (Ding et al., 2023). Following widely accepted and fair evaluation protocols, we adhered to the evaluation guidelines recommended by IMDLBench (Ma et al., 2024), ensuring consistency and comparability with prior studies.

##### Implementation Details.

To ensure fair comparisons and consistent experimental conditions, all experiments were conducted using the IMDLBench (Ma et al., 2024) framework. We conduct experiments using ViT and SegFormer as backbones, referred to as Relay-ViT and Relay-Seg, respectively. We set the number of [GLR] tokens to $n=2$, sub-image size to $512\times 512$.
For video, we set the sub-image size to $224\times 224$ and the clip length to 4.
We trained our models for 200 epochs using the AdamW optimizer (Loshchilov & Hutter, 2019) with a base learning rate of 1e-4, scheduled by a cosine decay policy (Loshchilov & Hutter, 2017). For more details, see the Appendix [A.4](https://arxiv.org/html/2508.09459v2#A1.SS4).

##### Evaluation Metrics.

We evaluate the performance of the predicted masks using two commonly adopted metrics: F1 score (with a fixed threshold of 0.5) and Intersection over Union (IoU).

**Table 1: Pixel-level comparison on the image manipulation localization task. Following the MVSS-Net protocol, we train the model on CASIAv2 and evaluate on five standard datasets. Scores indicate the F1 scores with a fixed threshold of 0.5.**
| Method | COVERAGE | Columbia | NIST16 | CASIAv1 | IMD2020 | Average |
| --- | --- | --- | --- | --- | --- | --- |
| Mantra-Net (Wu et al., 2019) | 0.090 | 0.243 | 0.104 | 0.125 | 0.055 | 0.123 |
| MVSS-Net (Chen et al., 2021) | 0.259 | 0.386 | 0.246 | 0.534 | 0.279 | 0.341 |
| CAT-Net (Kwon et al., 2022) | 0.296 | 0.584 | 0.267 | 0.594 | 0.268 | 0.402 |
| ObjectFormer (Wang et al., 2022) | 0.294 | 0.336 | 0.173 | 0.429 | 0.172 | 0.281 |
| PSCC-Net (Liu et al., 2022) | 0.231 | 0.605 | 0.200 | 0.378 | 0.233 | 0.329 |
| NCL-IML (Zhou et al., 2023) | 0.225 | 0.446 | 0.260 | 0.502 | 0.237 | 0.334 |
| Trufor (Guillaro et al., 2023) | 0.419 | 0.865 | 0.311 | 0.721 | 0.317 | 0.527 |
| IML-ViT (Ma et al., 2023) | 0.438 | 0.747 | 0.269 | 0.718 | 0.328 | 0.500 |
| Mesorch (Zhu et al., 2025) | 0.276 | 0.623 | 0.283 | 0.743 | 0.256 | 0.436 |
| SparseViT (Su et al., 2025) | 0.287 | 0.781 | 0.245 | 0.646 | 0.230 | 0.438 |
| Relay-ViT | 0.551 | 0.762 | 0.335 | 0.740 | 0.381 | 0.554 |
| Relay-Seg | 0.569 | 0.756 | 0.273 | 0.760 | 0.357 | 0.543 |

**Table 2: Quantitative comparison on the video manipulation localization task on three different video inpainting methods. For studies without open-source implementations, we report the results as presented in their original papers to ensure a fair comparison.**
| Methods | MOSE |  |  |
| --- | --- | --- | --- |
| E2FGVI | FuseFormer | STTN |  |
| (IoU / F1) | (IoU / F1) | (IoU / F1) |  |
| Mantra-Net (Wu et al., 2019) | 0.378/0.524 | 0.385/0.531 | 0.356/0.505 |
| MVSS-Net (Chen et al., 2021) | 0.038/0.057 | 0.051/0.074 | 0.094/0.133 |
| TruFor (Guillaro et al., 2023) | 0.311/0.414 | 0.285/0.388 | 0.260/0.353 |
| FOCAL (Yang et al., 2021) | 0.098/0.150 | 0.138/0.206 | 0.152/0.226 |
| Videofact (Nguyen et al., 2024) | 0.085/0.187 | 0.076/0.177 | 0.088/0.191 |
| TruVIL (Lou et al., 2024) | 0.521/0.674 | 0.557/0.699 | 0.462/0.612 |
| ViLocal (Lou et al., 2025) | 0.485/0.620 | 0.597/0.721 | 0.393/0.524 |
| Relay-ViT | 0.552/0.689 | 0.561/0.695 | 0.549/0.684 |
| Relay-Seg | 0.561/0.698 | 0.554/0.692 | 0.534/0.674 |

### 4.1 Compare with SoTA Methods

##### Image Manipulation Localization.

Following Protocol-MVSS (Chen et al., 2021), we train on CASIAv2 and test on COVERAGE, Columbia, NIST16, CASIAv1, and IMD2020. As shown in Table [1](https://arxiv.org/html/2508.09459v2#S4.T1), Relay-ViT and Relay-Seg achieve superior or competitive results across all datasets. Our framework reaches the highest average score (0.554), surpassing prior methods such as Trufor and IML-ViT. Despite being lightweight, Relay-Seg matches Relay-ViT overall and outperforms it on COVERAGE and CASIAv1, highlighting strong generalization.

##### Video Manipulation Localization.

Following TruVIL (Lou et al., 2024) and ViLocal, we train on OP (Oh et al., 2019) and VI (Kim et al., 2019) edited DAVIS2016 (Perazzi et al., 2016), and test on E2FGVI (Li et al., 2022), FuseFormer (Liu et al., 2021), and STTN (Zeng et al., 2020) edited MOSE. Table [2](https://arxiv.org/html/2508.09459v2#S4.T2) shows that both models achieve state-of-the-art results: Relay-Seg leads on E2FGVI, while Relay-ViT performs best on STTN, confirming robustness across different inpainting models.

As shown in Fig. [4](https://arxiv.org/html/2508.09459v2#S4.F4) and in the Appendix [A.9](https://arxiv.org/html/2508.09459v2#A1.SS9), our method also demonstrates superior performance in visual results.

**Table 3: Model complexity comparison: parameter counts (M) and computational cost (GFLOPs). The bolded part in our models indicates additional parameters. Multiple GFLOPs values correspond to different sub-image counts $N=1,2,4$.**
| Model | Parameters (M) | GFLOPs | Note |
| --- | --- | --- | --- |
| MVSS | 150.53 | 171.01 | Input: 512×512 |
| PSCC | 3.67 | 376.83 | Input: 256×256 |
| CAT-Net | 116.74 | 137.22 | Input: 512×512 |
| TruFor | 68.70 | 236.54 | Input: 512×512 |
| Mesorch | 85.75 | 124.93 | Input: 512×512 |
| IML-ViT | 91.78 | 576.78 | Input: 1024×1024 |
| Relay-ViT | 89.55+2.36 | 119.18 / 238.20 / 476.12 | $N=1,2,4$ |
| Relay-Seg | 45.90+2.39 | 52.71 / 105.41 / 210.83 | $N=1,2,4$ |

Figure: Figure 4: Visual qualitative results for image and video scenarios.
Refer to caption: x4.png

**Table 4: Ablation study on manipulation detection (F1 scores) across five benchmarks. We vary the number of [GLR] tokens ($n=0,1,2,3$), ($n=0$) means without the GLRA module, and evaluate the performance of our mask decoder.**
| [GLR] ($n$) | Decoder | COV. | Col. | NIST16 | CASIAv1 | IMD2020 | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| 0 | - | 0.486 | 0.596 | 0.248 | 0.691 | 0.248 | 0.454 |
| 1 | - | 0.548 | 0.718 | 0.289 | 0.751 | 0.301 | 0.521 |
| 1 | ✓ | 0.559 | 0.696 | 0.292 | 0.757 | 0.355 | 0.532 |
| 2 | ✓ | 0.551 | 0.762 | 0.335 | 0.740 | 0.381 | 0.554 |
| 3 | ✓ | 0.556 | 0.714 | 0.260 | 0.761 | 0.327 | 0.524 |

### 4.2 FLOPs and Parameters

As shown in Table [3](https://arxiv.org/html/2508.09459v2#S4.T3), our framework adapts dynamically to varying input resolutions, reducing redundant computation with minimal parameter overhead. See the Appendix [A.8](https://arxiv.org/html/2508.09459v2#A1.SS8) for a more detailed analysis of time complexity and parallelism.

**Table 5: Ablation of GLRA along spatial and temporal dimensions (MOSE).**
| Spatial | Temporal | F1 | IoU |
| --- | --- | --- | --- |
| – | – | 0.6124 | 0.4828 |
| ✓ | – | 0.6745 | 0.5391 |
| ✓ | ✓ | 0.6877 | 0.5524 |

### 4.3 Ablation Study

We conduct ablation experiments to assess the contribution of each component from three perspectives: (1) the number of [GLR] tokens and the role of the GLRA module ($n{=}0$), (2) the Query-based Mask Decoder, and (3) spatial-temporal cues and interpolation strategies. We provide a further analysis of the behavior of the [GLR] token along with additional visualizations in the Appendix [A.5](https://arxiv.org/html/2508.09459v2#A1.SS5).

Table [4](https://arxiv.org/html/2508.09459v2#S4.T4) reports results on five benchmarks.
Adding a single [GLR] token ($n{=}1$) improves results, and substituting the MLP with our decoder further boosts performance (0.532). The best performance occurs at $n{=}2$, while $n{=}3$ slightly degrades results due to redundancy (Fig. [6](https://arxiv.org/html/2508.09459v2#A1.F6)).

##### Effectiveness of GLRA along Temporal Dimensions

We further investigate the effectiveness of applying GLRA solely along the spatial dimension versus jointly across both spatial and temporal dimensions in video detection, in order to verify that our method can indeed extend to capturing temporal information in videos. The corresponding results are presented in Table [6](https://arxiv.org/html/2508.09459v2#S4.T6).

##### Effect of Input Resolution

As shown in Table [6](https://arxiv.org/html/2508.09459v2#S4.T6), resizing inputs to $1024{\times}1024$ reduces cost but introduces artifacts and loses high-frequency cues. Operating at raw resolution yields much better results (0.453 vs. 0.350), simultaneously demonstrating our model’s ability to generalize to arbitrary resolutions.

Figure: Figure 5: Robustness analysis results of the model under common perturbations.
Refer to caption: x5.png

### 4.4 Robustness Evaluation

We assess the robustness of different methods under common corruptions: Gaussian Blur, Gaussian Noise, and JPEG Compression. As shown in Fig. [5](https://arxiv.org/html/2508.09459v2#S4.F5), RelayFormer consistently outperforms prior methods across all distortion types and levels. It maintains higher F1 scores under increasing blur, noise, and compression, demonstrating strong generalization to real-world degradations.

## 5 Conclusion

In this work, we introduced RelayFormer, a unified framework for visual manipulation localization that addresses two long-standing challenges: resolution diversity and the image–video modality gap. By decomposing inputs into fixed-size sub-images and employing Global-Local Relay [GLR] tokens, our Global-Local Relay Attention (GLRA) mechanism enables efficient propagation of global context while preserving fine-grained local evidence. This design allows RelayFormer to adapt seamlessly to arbitrary input resolutions and video sequences without relying on costly resizing or modality-specific models. Extensive experiments across diverse benchmarks demonstrate that RelayFormer not only achieves state-of-the-art performance but also offers a favorable trade-off between accuracy and computational efficiency. These results highlight RelayFormer as a practical and scalable solution for robust manipulation localization in both images and videos.

## References

- Chen et al. (2021)
Xinru Chen, Chengbo Dong, Jiaqi Ji, Juan Cao, and Xirong Li.
Image manipulation detection by multi-view multi-scale supervision.
In *2021 IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 14165–14173, Montreal, QC, Canada, Oct 2021. IEEE.
ISBN 978-1-66542-812-5.
doi: 10.1109/ICCV48922.2021.01392.
URL [https://ieeexplore.ieee.org/document/9710015/](https://ieeexplore.ieee.org/document/9710015/).
- Cheng et al. (2022)
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar.
Masked-attention mask transformer for universal image segmentation.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 1290–1299, 2022.
- Ding et al. (2023)
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai.
Mose: A new dataset for video object segmentation in complex scenes.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 20224–20234, 2023.
- Dong et al. (2013)
Jing Dong, Wei Wang, and Tieniu Tan.
Casia image tampering detection evaluation database.
In *2013 IEEE China Summit and International Conference on Signal and Information Processing*, pp. 422–426, Beijing, China, Jul 2013. IEEE.
ISBN 978-1-4799-1043-4.
doi: 10.1109/ChinaSIP.2013.6625374.
URL [http://ieeexplore.ieee.org/document/6625374/](http://ieeexplore.ieee.org/document/6625374/).
- Guan et al. (2019)
Haiying Guan, Mark Kozak, Eric Robertson, Yooyoung Lee, Amy N. Yates, Andrew Delgado, Daniel Zhou, Timothee Kheyrkhah, Jeff Smith, and Jonathan Fiscus.
Mfc datasets: Large-scale benchmark datasets for media forensic challenge evaluation.
In *2019 IEEE Winter Applications of Computer Vision Workshops (WACVW)*, pp. 63–72, Waikoloa Village, HI, USA, Jan 2019. IEEE.
ISBN 978-1-72811-392-0.
doi: 10.1109/WACVW.2019.00018.
URL [https://ieeexplore.ieee.org/document/8638296/](https://ieeexplore.ieee.org/document/8638296/).
- Guillaro et al. (2023)
Fabrizio Guillaro, Davide Cozzolino, Avneesh Sud, Nicholas Dufour, and Luisa Verdoliva.
Trufor: Leveraging all-round clues for trustworthy image forgery detection and localization.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 20606–20615, 2023.
- Hsu & Chang (2006)
Yu-feng Hsu and Shih-fu Chang.
Detecting image splicing using geometry invariants and camera characteristics consistency.
In *2006 IEEE International Conference on Multimedia and Expo*, pp. 549–552, Toronto, ON, Canada, Jul 2006. IEEE.
ISBN 978-1-4244-0367-7.
doi: 10.1109/ICME.2006.262447.
URL [http://ieeexplore.ieee.org/document/4036658/](http://ieeexplore.ieee.org/document/4036658/).
- Hu et al. (2022)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al.
Lora: Low-rank adaptation of large language models.
*ICLR*, 1(2):3, 2022.
- Kim et al. (2019)
Dahun Kim, Sanghyun Woo, Joon-Young Lee, and In So Kweon.
Recurrent temporal aggregation framework for deep video inpainting.
*IEEE transactions on pattern analysis and machine intelligence*, 42(5):1038–1052, 2019.
- Kwon et al. (2022)
Myung-Joon Kwon, Seung-Hun Nam, In-Jae Yu, Heung-Kyu Lee, and Changick Kim.
Learning jpeg compression artifacts for image manipulation detection and localization.
*International Journal of Computer Vision*, 130(8):1875–1895, 2022.
- Li et al. (2022)
Zhen Li, Cheng-Ze Lu, Jianhua Qin, Chun-Le Guo, and Ming-Ming Cheng.
Towards an end-to-end framework for flow-guided video inpainting.
In *Proceedings of the IEEE/CVF conference on computer vision and pattern recognition*, pp. 17562–17571, 2022.
- Liu et al. (2021)
Rui Liu, Hanming Deng, Yangyi Huang, Xiaoyu Shi, Lewei Lu, Wenxiu Sun, Xiaogang Wang, Jifeng Dai, and Hongsheng Li.
Fuseformer: Fusing fine-grained information in transformers for video inpainting.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 14040–14049, 2021.
- Liu et al. (2022)
Xiaohong Liu, Yaojie Liu, Jun Chen, and Xiaoming Liu.
Pscc-net: Progressive spatio-channel correlation network for image manipulation detection and localization.
*IEEE Transactions on Circuits and Systems for Video Technology*, 32(11):7505–7517, 2022.
- Loshchilov & Hutter (2017)
Ilya Loshchilov and Frank Hutter.
Sgdr: Stochastic gradient descent with warm restarts.
(arXiv:1608.03983), May 2017.
URL [http://arxiv.org/abs/1608.03983](http://arxiv.org/abs/1608.03983).
arXiv:1608.03983 [cs, math].
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization.
(arXiv:1711.05101), Jan 2019.
URL [http://arxiv.org/abs/1711.05101](http://arxiv.org/abs/1711.05101).
arXiv:1711.05101 [cs, math].
- Lou et al. (2024)
Zijie Lou, Gang Cao, and Man Lin.
Trusted video inpainting localization via deep attentive noise learning.
*arXiv preprint arXiv:2406.13576*, 2024.
- Lou et al. (2025)
Zijie Lou, Gang Cao, and Man Lin.
Video inpainting localization with contrastive learning.
*IEEE Signal Processing Letters*, 2025.
- Ma et al. (2023)
Xiaochen Ma, Bo Du, Xianggen Liu, Ahmed Y Al Hammadi, and Jizhe Zhou.
Iml-vit: Image manipulation localization by vision transformer, 2023.
- Ma et al. (2024)
Xiaochen Ma, Xuekang Zhu, Lei Su, Bo Du, Zhuohang Jiang, Bingkui Tong, Zeyu Lei, Xinyu Yang, Chi-Man Pun, Jiancheng Lv, et al.
Imdl-benco: A comprehensive benchmark and codebase for image manipulation detection & localization, 2024.
- Nguyen et al. (2024)
Tai D Nguyen, Shengbang Fang, and Matthew C Stamm.
Videofact: detecting video forgeries using attention, scene context, and forensic traces.
In *Proceedings of the IEEE/CVF winter conference on applications of computer vision*, pp. 8563–8573, 2024.
- Novozamsky et al. (2020)
Adam Novozamsky, Babak Mahdian, and Stanislav Saic.
Imd2020: A large-scale annotated dataset tailored for detecting manipulated images.
In *2020 IEEE Winter Applications of Computer Vision Workshops (WACVW)*, pp. 71–80, Snowmass Village, CO, USA, March 2020. IEEE.
ISBN 978-1-72817-162-3.
doi: 10.1109/WACVW50321.2020.9096940.
URL [https://ieeexplore.ieee.org/document/9096940/](https://ieeexplore.ieee.org/document/9096940/).
- Oh et al. (2019)
Seoung Wug Oh, Sungho Lee, Joon-Young Lee, and Seon Joo Kim.
Onion-peel networks for deep video completion.
In *Proceedings of the IEEE/CVF international conference on computer vision*, pp. 4403–4412, 2019.
- Pei (2023)
Pengfei Pei.
Uvl2: A unified framework for video tampering localization.
*arXiv preprint arXiv:2309.16126*, 2023.
- Perazzi et al. (2016)
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung.
A benchmark dataset and evaluation methodology for video object segmentation.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 724–732, 2016.
- Poth et al. (2023)
Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer.
Adapters: A unified library for parameter-efficient and modular transfer learning.
In *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*, pp. 149–160, Singapore, December 2023. Association for Computational Linguistics.
URL [https://aclanthology.org/2023.emnlp-demo.13](https://aclanthology.org/2023.emnlp-demo.13).
- Su et al. (2024)
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.
Roformer: Enhanced transformer with rotary position embedding.
*Neurocomputing*, 568:127063, 2024.
- Su et al. (2025)
Lei Su, Xiaochen Ma, Xuekang Zhu, Chaoqun Niu, Zeyu Lei, and Ji-Zhe Zhou.
Can we get rid of handcrafted feature extractors? sparsevit: Nonsemantics-centered, parameter-efficient image manipulation localization through spare-coding transformer.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pp. 7024–7032, 2025.
- Wang et al. (2022)
Junke Wang, Zuxuan Wu, Jingjing Chen, Xintong Han, Abhinav Shrivastava, Ser-Nam Lim, and Yu-Gang Jiang.
Objectformer for image manipulation detection and localization.
In *2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 2354–2363, New Orleans, LA, USA, Jun 2022. IEEE.
ISBN 978-1-66546-946-3.
doi: 10.1109/CVPR52688.2022.00240.
URL [https://ieeexplore.ieee.org/document/9880322/](https://ieeexplore.ieee.org/document/9880322/).
- Wang et al. (2024)
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al.
Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.
*arXiv preprint arXiv:2409.12191*, 2024.
- Wen et al. (2016)
Bihan Wen, Ye Zhu, Ramanathan Subramanian, Tian-Tsong Ng, Xuanjing Shen, and Stefan Winkler.
Coverage — a novel database for copy-move forgery detection.
In *2016 IEEE International Conference on Image Processing (ICIP)*, pp. 161–165, Phoenix, AZ, USA, Sep 2016. IEEE.
ISBN 978-1-4673-9961-6.
doi: 10.1109/ICIP.2016.7532339.
URL [http://ieeexplore.ieee.org/document/7532339/](http://ieeexplore.ieee.org/document/7532339/).
- Wu & Zhou (2021)
Haiwei Wu and Jiantao Zhou.
Iid-net: Image inpainting detection network via neural architecture search and attention.
*IEEE Transactions on Circuits and Systems for Video Technology*, 32(3):1172–1185, 2021.
- Wu et al. (2022)
Haiwei Wu, Jiantao Zhou, Jinyu Tian, Jun Liu, and Yu Qiao.
Robust image forgery detection against transmission over online social networks.
*IEEE Transactions on Information Forensics and Security*, 17:443–456, 2022.
- Wu et al. (2023)
Haiwei Wu, Yiming Chen, and Jiantao Zhou.
Rethinking image forgery detection via contrastive learning and unsupervised clustering.
*arXiv preprint arXiv:2308.09307*, 2023.
- Wu et al. (2019)
Yue Wu et al.
Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features.
In *2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 9535–9544, Long Beach, CA, USA, Jun 2019. IEEE.
ISBN 978-1-72813-293-8.
doi: 10.1109/CVPR.2019.00977.
URL [https://ieeexplore.ieee.org/document/8953774/](https://ieeexplore.ieee.org/document/8953774/).
- Yang et al. (2021)
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao.
Focal self-attention for local-global interactions in vision transformers.
*arXiv preprint arXiv:2107.00641*, 2021.
- Zeng et al. (2020)
Yanhong Zeng, Jianlong Fu, and Hongyang Chao.
Learning joint spatial-temporal transformations for video inpainting.
In *European conference on computer vision*, pp. 528–543. Springer, 2020.
- Zhou et al. (2023)
Jizhe Zhou, Xiaochen Ma, Xia Du, Ahmed Y Alhammadi, and Wentao Feng.
Pre-training-free image manipulation localization through non-mutually exclusive contrastive learning.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 22346–22356, 2023.
- Zhou et al. (2021)
Peng Zhou, Ning Yu, Zuxuan Wu, Larry S Davis, Abhinav Shrivastava, and Ser-Nam Lim.
Deep video inpainting detection.
*arXiv preprint arXiv:2101.11080*, 2021.
- Zhu et al. (2025)
Xuekang Zhu, Xiaochen Ma, Lei Su, Zhuohang Jiang, Bo Du, Xiwen Wang, Zeyu Lei, Wentao Feng, Chi-Man Pun, and Ji-Zhe Zhou.
Mesoscopic insights: orchestrating multi-scale & hybrid architecture for image manipulation localization.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, pp. 11022–11030, 2025.

## Appendix A Appendix

### A.1 Limitations

##### Limitations on global context modeling.

While our partition-and-relay design achieves a favorable trade-off between accuracy and efficiency, it inevitably introduces a limitation compared to full global attention. Specifically, when manipulations span across multiple sub-images, the Global-Local Relay Attention (GLRA) propagates contextual information through relay tokens rather than establishing exhaustive pairwise interactions. This relay mechanism is computationally more efficient, yet it cannot capture cross-partition dependencies as precisely as a global attention scheme if computational constraints are disregarded.

### A.2 Detailed Description of the Method

#### A.2.1 Input unification

To more clearly demonstrate how we preprocess videos and images into a unified form, we provide relevant pseudocode [1](https://arxiv.org/html/2508.09459v2#alg1).

Figure: Algorithm 1 Sub-images Extraction

#### A.2.2 Parameter-efficient strategy

Using shared parameters for both local and global attention severely degrades performance because the two modules serve fundamentally different purposes and operate on distinct feature spaces. Local attention works on dense, low-level patch tokens ($X_{i}$), focusing on fine-grained textures, edges, and object parts. Global attention, by contrast, processes sparse, high-level [GLR] tokens ($T_{\text{flat}}$), which summarize sub-images and model long-range dependencies. A single set of projection weights cannot simultaneously specialize in local detail extraction and global structural reasoning, leading to suboptimal representations in both tasks. Therefore, separate parameterization is essential to preserve both local fidelity and global coherence.

As shown in Fig. [3](https://arxiv.org/html/2508.09459v2#S3.F3), our solution introduces functional specialization via a dynamic parameter-sharing scheme based on Low-Rank Adaptation (LoRA). Each Transformer block maintains a shared set of backbone projection matrices ($W_{Q}$, $W_{K}$, $W_{V}$), which are fully trainable. On top of this backbone, we add two task-specific sets of LoRA parameters: $\{A_{\text{local}},B_{\text{local}}\}$ for $\text{SelfAttn}_{\text{local}}$ and $\{A_{\text{global}},B_{\text{global}}\}$ for $\text{SelfAttn}_{\text{global}}$. During the forward pass, the effective weight is dynamically constructed. For example, in local attention:

$$ $W^{\prime}_{Q}=W_{Q}+B_{Q,\text{local}}A_{Q,\text{local}},$ $$

while in global attention:

$$ $W^{\prime\prime}_{Q}=W_{Q}+B_{Q,\text{global}}A_{Q,\text{global}}.$ $$

Here, $W_{Q}$ provides a shared backbone, and LoRA contributes lightweight, context-specific adjustments.

Unlike conventional LoRA fine-tuning, our backbone remains trainable, and the LoRA parameters are never merged into it. This design is crucial: rather than adapting a frozen model to a single task, we enable two co-existing functional modes that can be switched dynamically. The result is efficient parameter sharing that preserves specialization for both local and global reasoning.

### A.3 Datasets

#### A.3.1 Image Datasets

We use the following publicly available datasets for the detection of spliced and copy-moved images, following previous settings (Ma et al., 2023; 2024), we didn’t use real images in all datasets:

To provide a detailed overview of these datasets, Table [7](https://arxiv.org/html/2508.09459v2#A1.T7) summarizes key attributes, including the number of images or videos, forgery types, and other relevant characteristics. All details are sourced from official or authoritative descriptions to ensure reliability.

**Table 7: Overview of benchmark datasets for image forgery detection. Forgery types include splicing (S), copy-move (C), removal/inpainting (R), enhancement (E), and others (O).**
| Dataset | Year | # Authentic | # Forged | Forgery Types |
| --- | --- | --- | --- | --- |
| Image Forgery Datasets |  |  |  |  |
| CASIA v1.0 | 2013 | 800 | 921 | S, C, R |
| CASIA v2.0 | 2013 | 7,491 | 5,123 | S, C, R |
| Columbia | 2004 | 183 | 180 | S |
| Coverage | 2016 | 100 | 100 | C |
| NIST16 | 2016 | 560 | 564 | S, C, R |
| IMD2020 | 2020 | 414 | 2010 | S, C, O |
| Defacto | 2019 | Variable | 190k | S, C, R, O |
| AI-Generated Forgery Datasets |  |  |  |  |
| AutoSplice | 2023 | 2,273 | 3,621 | S, O |
| CocoGlide | 2023 | Variable | Variable | S, O |

Key characteristics of the datasets: CASIA datasets provide ground truth masks and include post-processing artifacts; Columbia focuses on uncompressed splicing evaluation; Coverage contains copy-move forgeries with similar genuine objects; NIST16 offers high sensor diversity from the Nimble Challenge; IMD2020 covers real-life manipulations from diverse camera models; and AI-generated datasets (AutoSplice, CocoGlide) feature semantically meaningful manipulations with mask annotations.

#### A.3.2 Video Datasets

For video inpainting experiments, we use the following datasets:

**Table 8: Video datasets used in our workflow.**
| Dataset | Clips | #Frames/Masks | Resolution | Use |
| --- | --- | --- | --- | --- |
| DAVIS 2016 (Perazzi et al., 2016) | 50 | 3,455 | 1080p | Generate training via OP / VI |
| MOSE (Ding et al., 2023) | 2,149 | 431,725 | 1080p–4K | Cross-dataset test |

Details of usage. DAVIS 2016 contains 50 short videos (3–4s) with 3,455 densely annotated frames at 1080p resolution, split into 30 training and 20 validation clips (Perazzi et al., 2016). We use two video inpainting models—OP (Oh et al., 2019) and VI (Kim et al., 2019)—to generate corrupted–reconstructed frame pairs for training.

The MOSE dataset includes videos and object masks across 36 categories with complex scenarios, such as occlusions and dense crowds (Ding et al., 2023). We use the validation split of 100 clips as a test set for evaluating, using three methods: E2FGVI (Li et al., 2022), FuseFormer (Liu et al., 2021), and STTN (Zeng et al., 2020) models to create validation datasets.

#### A.3.3 Data Split Summary

- •
Image tasks: CASIA v2.0 is used for training. Other image datasets (CASIA v1.0, Columbia, Coverage, NIST16, IMD2020) are used for cross-dataset testing.
- •
Video tasks: DAVIS 2016 is used to generate training data via OP and VI models. MOSE validation split is used for testing with E2FGVI, FuseFormer, and STTN.

This setup allows evaluation of both in-domain performance and cross-domain generalization for image forgery detection and video inpainting.

### A.4 Implementation Details

To ensure fair comparisons and consistent experimental conditions, all experiments were conducted using the IMDLBench (Ma et al., 2024) framework. We adopted the same set of training hyperparameters across all backbone models, strictly following the configuration used in IML-ViT. We conduct experiments using ViT and SegFormer as backbones, referred to as Relay-ViT and Relay-Seg, respectively. We set the number of [GLR] tokens to $n=2$. The sub-image size was set to $528\times 528$ pixels with an overlap of 16 pixels.

In our implementation, we replace all Transformer blocks in the backbone with GLRA modules. The layer $K$ of the mask decoder is set to 3, and the number of learnable queries is 8. During the first epoch, we freeze the pre-trained parameters and fine-tune only the newly introduced parameters, allowing them to adapt to the task independently. In the loss function, the edge loss is weighted with a coefficient $\lambda=20$.

For images larger than $1024\times 1024$, we first resize them by scaling the longer side to 1024 pixels while preserving the original aspect ratio. After resizing, all images are zero-padded to a uniform resolution of $1024\times 1024$.
For video data, we use a sub-image size of $224\times 224$ and a temporal clip length of 4 frames.

In the GLRA module, we apply LoRA (Hu et al., 2022) to both the query–key–value projections and the feed-forward networks, using a rank of 8 and a scaling factor of 2.

During training, we apply the standard data augmentation strategies introduced in IML-ViT (Ma et al., 2023), including re-scaling, horizontal flipping, Gaussian blurring, random rotation, and manipulations such as copy-move and inpainting of rectangular regions within the same image.

All experiments are conducted on 4 NVIDIA RTX 3090 GPUs using PyTorch’s distributed training via ‘torchrun‘ and mixed precision (AMP). We use a per-GPU batch size of 4 and apply gradient accumulation with 4 steps, resulting in an effective batch size of 64. Models are trained for 200 epochs using the AdamW optimizer (Loshchilov & Hutter, 2019) with a base learning rate of $1\times 10^{-4}$, cosine decay schedule (Loshchilov & Hutter, 2017), and weight decay of 0.05. The learning rate is linearly warmed up for the first 2 epochs, and the minimum learning rate is set to $5\times 10^{-7}$. A fixed random seed (42) is used during training, and all results are reported as the median of three runs with different seeds to reduce the impact of random variation. The best-performing checkpoint on the validation set is used for final evaluation. No test-time augmentation or post-processing is applied.

Figure: Figure 6: Attention map visualization of [GLR] tokens for analyzing their behavioral patterns.
Refer to caption: x6.png

Figure: Figure 7: Qualitative results illustrating the role of GLRA.
Refer to caption: x7.png

### A.5 Understanding GLRA Behavior

To further investigate the effect of GLRA, we visualize intermediate attention maps and feature activation patterns in Fig. [6](https://arxiv.org/html/2508.09459v2#A1.F6) and Fig. [7](https://arxiv.org/html/2508.09459v2#A1.F7). Without GLRA, the representations of different spatial sub-images tend to diverge, as local self-attention lacks a mechanism for sufficient global interaction. This leads to fragmented feature distributions that fail to capture cross-region dependencies. In contrast, GLRA introduces an explicit relay pathway for long-range communication, enforcing semantic consistency across sub-images and strengthening the alignment of manipulated and pristine regions. Furthermore, we observe that using two [GLR] tokens consistently provides broader and more informative coverage compared to a single token, while introducing a third token mainly leads to redundant overlapping attention. These observations support our empirical finding that setting the number of relay tokens to $n{=}2$ is sufficient to achieve a favorable balance between modeling capacity and computational efficiency.

### A.6 Additional Cross-Dataset Evaluation

We further evaluate our models on three additional datasets, with results summarized in Table [9](https://arxiv.org/html/2508.09459v2#A1.T9). Overall, our Relay-based methods consistently achieve leading performance under broad testing conditions. In particular, Relay-Seg attains the highest average F1 score (0.372), surpassing all competing baselines. This demonstrates that the proposed GLRA mechanism generalizes well across diverse forgery types and input distributions.

A noteworthy observation is the impact of backbone choice when facing AI-generated forgery datasets. For example, both Relay-ViT and IML-ViT, which are based on Vision Transformer architectures, achieve a similar level of performance. In contrast, Relay-Seg outperforms other methods, while SparseViT also exhibits competitive results. We attribute this advantage to the architectural design: Relay-Seg and SparseViT both adopt hierarchical Transformer encoders that produce high-resolution coarse features and low-resolution fine features, while incorporating more convolutional operations. Such hybrid designs appear to be particularly effective in capturing the subtle artifacts present in AI-generated forgeries.

These findings not only validate the robustness of our relay-based formulation but also suggest that backbone-level inductive biases play a significant role in detecting AI-generated content. Importantly, our approach benefits from these architectural strengths while introducing only minimal overhead, thereby offering both efficiency and adaptability.

**Table 9: Performance comparison of different methods on three datasets (metric: F1 score, higher is better)**
| Method | AutoSplice | CocoGlide | Defacto-12k | Average |
| --- | --- | --- | --- | --- |
| IML-ViT (Ma et al., 2023) | 0.221 | 0.210 | 0.367 | 0.266 |
| SparseViT (Su et al., 2025) | 0.386 | 0.142 | 0.242 | 0.257 |
| Mesorch (Zhu et al., 2025) | 0.216 | 0.120 | 0.292 | 0.209 |
| Relay-ViT | 0.289 | 0.203 | 0.416 | 0.303 |
| Relay-Seg | 0.379 | 0.328 | 0.409 | 0.372 |

Figure: Figure 8: Impact of GLRA layer replacement strategies on CASIA v1.
Refer to caption: figures/n_layer.png

### A.7 More Ablation Studies

To evaluate the effectiveness of GLRA when integrated into different parts of the backbone, we conducted experiments with three replacement strategies on the CASIA v1 dataset: (1) inserting GLRA into a sparse set of layers across the transformer encoder $[0,3,7,11]$, (2) replacing all layers in the latter half of the encoder $[6\text{--}11]$, and (3) replacing all layers in the encoder (i.e., full replacement). As shown in Figure [8](https://arxiv.org/html/2508.09459v2#A1.F8), progressively increasing the number of GLRA-applied layers leads to consistent improvements in classification accuracy: 73.24%, 74.21%, and 75.50%, respectively. These results indicate that GLRA contributes more significantly when applied to deeper layers and that full-layer integration yields the best performance. This suggests that GLRA is both effective and scalable when applied throughout the model architecture.

### A.8 Complexity and Parallelism Analysis

As shown in Table [3](https://arxiv.org/html/2508.09459v2#S4.T3), both Relay-ViT and Relay-Seg introduce only a negligible number of additional parameters ($\sim 2.4$M) compared to their respective backbones, demonstrating that GLRA incurs minimal memory overhead. Despite this, our methods substantially reduce computational cost relative to prior transformer-based baselines. For example, Relay-ViT achieves a lower GFLOPs budget than IML-ViT even when operating with $N{=}4$ sub-images at $1024\times 1024$ resolution. Moreover, the scalability of our design is evident: the GFLOPs grow linearly with the number of sub-images, while the parameter count remains nearly constant. This highlights the efficiency of our relay-based formulation, which decouples global reasoning capacity from the quadratic growth in input resolution. Overall, Relay-ViT and Relay-Seg strike a favorable balance between model size, computational efficiency, and representational power, validating the practical advantage of the proposed GLRA mechanism.

##### Time complexity.

In the local attention stage, each sub-image $U_{i}$ contains $P$ patch tokens and $m$ [GLR] tokens, yielding a total of $P+m$ tokens per sub-image. The self-attention operation in Eq. equation [1](https://arxiv.org/html/2508.09459v2#S3.E1) thus requires
$\mathcal{O}((P+m)^{2}d)$
computations per layer, where $d$ is the hidden dimension. Since $N$ typically dominates $m$, the asymptotic cost is comparable to standard sub-image self-attention. Across all sub-images in the batch, the total complexity scales linearly with
$B_{\text{total}}$,
the unified number of sub-images.

In the global attention stage, the complexity depends only on the number of [GLR] tokens. For a sample with $N_{i}$ sub-images, the concatenated sequence length is $N_{i}\cdot m$, leading to a cost of
$\mathcal{O}((N_{i}m)^{2}d)$
per layer in Eq. equation [3](https://arxiv.org/html/2508.09459v2#S3.E3). Compared to local attention, this is relatively lightweight, since $m\ll P$ and $N_{i}$ is typically small.

##### Parallelism.

The unified representation in Sec. [3.1](https://arxiv.org/html/2508.09459v2#S3.SS1) enables straightforward parallelization across all sub-images. However, the number of sub-images $N_{i}$ may vary across samples due to differing input resolutions, resulting in variable numbers of [GLR] tokens. To enable efficient batched computation, we pad the sequence of [GLR] tokens in each sample to the maximum length within the batch. This ensures that global attention can be executed in parallel without irregular memory access patterns. Since the number of sub-images per sample is usually small, the overhead introduced by such padding is negligible in practice, while the benefit of full parallelization is substantial.

### A.9 More Visualization Results

In this section, we provide additional qualitative comparisons to further demonstrate the effectiveness of our method. Figure [9](https://arxiv.org/html/2508.09459v2#A1.F9) showcases a variety of manipulated images and videos, along with their corresponding ground truth masks and the predicted results from different baseline methods, including CAT-Net, PSCC-Net, Trufor, and Mesorch. Our method, RelayFormer, consistently generates more accurate and fine-grained manipulation masks, with better localization and fewer false positives compared to previous approaches.

For manipulated video sequences, our model not only detects spatial tampering more precisely but also captures temporal consistency across frames, which is essential for robust manipulation detection in videos.

#### A.9.1 Supplementary material

##### Video Visualizations Results.

To better visualize the temporal performance of our method on manipulated videos, we provide a rich set of video demonstrations in the supplementary material. These visualizations clearly illustrate the robustness and temporal coherence of our predictions.

##### Code.

To ensure reproducibility, we submit the code in the supplementary material.

Figure: Figure 9: Qualitative comparisons on manipulated images and videos. Our method (RelayFormer) shows superior performance in both spatial and temporal prediction accuracy compared to prior methods.
Refer to caption: x8.png

## Appendix B Statement on the Use of Large Language Models (LLMs)

In the preparation of this manuscript, a Large Language Model (LLM) was used solely for the purpose of language polishing, including minor grammar correction and stylistic refinement of the authors’ original text. The LLM did not contribute to the conceptualization of the research, the design of experiments, the analysis of results, or the interpretation of findings. All research ideas, methods, and conclusions presented in this paper are entirely the work of the authors.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Will the code be publicly available?
soundness: 3
strengths: - The insight of bridging image and video manipulation localization holds novelty. It will offer a benchmark case for future multi-modal VML tasks.
- The proposed method’s resolution strategy offers a significant computational efficiency advantage within the current field of manipulation detection.
- The experiments in this paper are comprehensive and well-organized, effectively demonstrating the proposed claims.
summary: This paper proposes RelayFormer, a unified framework for visual manipulation localization across both images and videos. It addresses two major challenges: resolution diversity and the image–video modality gap. The key innovation is the Global-Local Relay Attention (GLRA), where Global-Local Relay (GLR) tokens propagate global context efficiently between local sub-images, enabling scalability without interpolation or dense attention. A lightweight query-based decoder produces precise masks. Extensive experiments show state-of-the-art accuracy, strong robustness, and excellent efficiency on multiple benchmarks.
weaknesses: - The proposed method shares some similarities with Visual Prompt Tuning[A], as it introduces additional trainable tokens to transmit information and assist decision-making. It is recommended that the authors discuss this connection in an appropriate section of the paper.
- To the best of my knowledge, RoPE is introduced for the first time in a manipulation localization model. The authors may consider analyzing the advantages of this positional embedding for a pure computer vision task like manipulation detection, ideally supported by additional experiments.
- We are currently in an era that values scaling up as an important perspective. Although the authors claim that the proposed structure bridges image and video manipulation localization, the experiments are conducted separately for the two modalities. It would be interesting to explore whether joint training on both images and videos could lead to further improvements or provide additional insights into scaling behavior.

## Reference
- [A] Jia, Menglin, et al. "Visual prompt tuning." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: Please see the weaknesses.
soundness: 3
strengths: A new framework for unified image and video forensics.

Achieves top average F1 on image benchmarks, and consistently high IoU/F1 across video inpainting methods on MOSE.

Robustness curves for Gaussian blur, noise, and JPEG compression.
summary: The paper tackles visual manipulation localization for both images and videos, addressing resolution diversity and modality gaps. It proposes RelayFormer, which partitions inputs into fixed-size sub-images and introduces Global-Local Relay (GLR) tokens to propagate global context via a Global-Local Relay Attention (GLRA) mechanism. The method includes a parameter-efficient strategy using shared backbone layers with task-specific adapters, a 4D RoPE positional encoding, and a lightweight query-based mask decoder. Experiments on multiple image and video benchmarks show improved F1/IoU performance against prior work, with analyses of FLOPs/parameters and robustness.
weaknesses: The proposed approach relies on fusing local and global information, a well-established technique in computer vision whose effectiveness is often assumed.

While results from training on both are presented, there is no clear explanation or investigation into how these modalities mutually influence each other. An intuitive hypothesis is that images can be considered single frames of videos, and training on image forgery could enhance video forgery detection (and vice-versa). To truly demonstrate the benefit of a unified approach, it is suggested to include experiments evaluating model performance when trained exclusively on one modality (image or video) and then tested on both. This would shed light on the synergy between the two data types.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: Refer to the weaknessses.
soundness: 3
strengths: 1.The research problem is well-motivated. The paper correctly identifies that manipulation detection requires simultaneous consideration of both fine-grained local artifacts and coarse-grained global consistency cues (such as illumination mismatches and structural redundancies). This provides a sound foundation for proposing an efficient global information propagation mechanism and offers valuable insights for the forensics community.

2.The input unification strategy (Section 3.1) is practical. By partitioning variable-resolution images and videos into fixed-size overlapping sub-images, the method avoids interpolation operations that could destroy forensic traces. This strategy is both simple and effective, supports parallel processing, and demonstrates practical engineering value.

3.The experimental validation is comprehensive. The paper evaluates the approach on 8 datasets covering both image and video scenarios. Ablation studies systematically analyze GLR token quantity (Table 4), temporal dimension (Table 5), and interpolation effects (Table 6). Robustness evaluation (Figure 5) tests against common perturbations, with generally sound experimental design.
summary: This paper proposes the RelayFormer framework for unified visual manipulation localization in both images and videos. The authors claim their method addresses two primary challenges: resolution diversity and the image-video "modality gap." The core innovation is the Global-Local Relay Attention (GLRA) mechanism, which propagates global context across fixed-size sub-images through learnable Global-Local Relay (GLR) tokens. The paper divides inputs into overlapping sub-images to avoid interpolation, introduces GLR tokens as an information bottleneck for efficient global-local interaction, and employs a LoRA-based parameter-efficient strategy. Experimental validation is conducted on multiple image and video manipulation datasets.
weaknesses: 1. The authors emphasize that images and videos belong to different modalities, which is imprecise. Multimodality typically refers to different data types (e.g., vision and text, vision and audio). Both images and videos belong to the visual modality, differing only in the presence or absence of temporal dimension. This conceptual confusion weakens the theoretical positioning of the paper. It would be more appropriate to frame this as a "temporal dimension extension problem" rather than a "modality gap."

2. The technical novelty is limited. The technical contributions of this work are relatively incremental. The use of fixed-size sub-images and Global-Local Relay (GLR) tokens represents fairly standard techniques. Fixed-size sub-images are a common operation in image and video processing (standard practice in ViT and similar architectures). The proposed GLR tokens show no significant distinction from existing prompt tuning methods [1][2][3][4], as they essentially all perform information aggregation through learnable tokens. The paper fails to adequately differentiate GLR tokens from prior work such as Dgl [3] and Visual Prompt Tuning [4]. The innovative contributions are primarily at the application level rather than the methodological level.
[1] Liu, Xiao, et al. "P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks." arXiv preprint arXiv:2110.07602 (2021).
[2]Zhang, Ji, et al. "Dept: Decoupled prompt tuning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[3] Yang, Xiangpeng, et al. "Dgl: Dynamic global-local prompt tuning for text-video retrieval." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 7. 2024.
[4] Jia, Menglin, et al. "Visual prompt tuning." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

3. The readability needs improvement and the related work section is weak. The writing quality requires enhancement. For example, lines 103-104 and 152-153 employ excessive use of dashes. The Related Work section (Section 2) is overly brief, appearing more as a literature listing than a critical analysis. It lacks in-depth discussion of why existing methods fail. The writing should be more direct and professional to enhance clarity.

4. The 4D RoPE design lacks sufficient justification. The 4D RoPE positional encoding design (Section 3.2) lacks adequate theoretical foundation. How are the dimension allocations (d_T, d_id, d_H, d_W, d_rem) determined (lines 261)? The paper only provides formulas without sensitivity analysis. More importantly, the paper claims "strong extrapolation capability" (line 270) but provides no experimental verification. For instance, what is the performance on 2K or 4K images after training at 512×512 resolution? This claim lacks empirical support.

5. Hyperparameter selection lacks rigor. The choice of n=2 for GLR token quantity (line 311) is determined only through ablation studies, but why does n=3 lead to performance degradation? The paper merely speculates "due to redundancy" without deeper analysis. Similarly, the choices of sub-image sizes (512×512 for images, 224×224 for videos) and overlap size (16 pixels) lack theoretical or experimental justification. Additional experimental analysis of hyperparameter selection should be provided.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Quantifying Relay Compression Loss:
How much performance degradation, if any, occurs when comparing GLRA against a full global-attention model (even at a reduced resolution)?
Such an experiment could empirically validate the trade-off between efficiency and precision.

2. Evaluation under CAT Protocol:
Given that the paper currently reports results under the MVSS protocol, could the authors also train and evaluate a CAT-protocol version of RelayFormer?

3. Temporal Sparsity in Manipulation Frames
In real-world video forgeries, only a few frames may be manipulated while the majority remain authentic.
Given that RelayFormer aggregates temporal information through relay tokens across frames, how effectively can the model detect and localize sparse manipulations that occur in only a small subset of frames?
soundness: 3
strengths: 1. The core GLRA mechanism introduces a small set of learnable [GLR] (Global-Local Relay) tokens to act as information proxies. This design reduces the computational complexity of global attention from quadratic in the total number of patches to quadratic in the number of sub-images, achieving an exponential reduction in computational cost. The framework employs an elegant information fusion strategy, iteratively executing a two-step process of "local perception" and "global relay" within each Transformer layer. 

2. The paper ingeniously employs a strategy that combines a shared Transformer backbone with two independent, lightweight LoRA adapters. By dynamically switching between these adapters, the design achieves functional specialization for the distinct local and global attention tasks at a minimal additional parameter cost.
summary: This paper introduces RelayFormer, a unified attention framework designed to address two core challenges in Visual Manipulation Localization (VML): resolution diversity and the modality gap across images and videos. The core of RelayFormer is a novel Global-Local Relay Attention (GLRA) mechanism. This mechanism partitions inputs of arbitrary resolution into fixed-size sub-images and introduces a small set of learnable "relay" tokens to absorb fine-grained manipulation artifacts locally before engaging in a sparse information exchange at the global level to propagate context. Furthermore, the framework incorporates a parameter-efficient design, leveraging a shared Transformer backbone with task-specific LoRA adapters to optimize efficiency with minimal impact on performance. The authors validate the framework's efficiency and performance through extensive experiments on multiple image and video benchmarks.
weaknesses: 1. Information Bottleneck of Relay Tokens:
Since global information exchange is mediated through a small set of relay tokens, this mechanism may introduce mild information loss when manipulations span multiple sub-images with complex cross-region dependencies (e.g., fine-grained copy-move forgeries).
Although this trade-off is computationally justified, it would be useful to quantify how much contextual precision is sacrificed compared to a full-attention baseline.


2. Missing CAT Protocol Comparison:
The image manipulation localization experiments primarily follow the MVSS protocol (training on CASIA-v2 and testing on standard benchmarks).
However, stronger and more comprehensive evaluation settings, such as the CAT protocol,should also be considered.


Decision: Accept (Poster)