Adapting Vision Transformers to Ultra-High Resolution Semantic Segmentation with Relay Tokens
Authors: Yohann PERRON
Vladyslav Sydorov
Christophe Pottier
Loic Landrieu
Venue: Accepted by TMLR ()

Abstract: Current approaches for segmenting ultra-high-resolution images either slide a window, thereby discarding global context, or downsample and lose fine detail. We propose a simple yet effective method that brings explicit multi-scale reasoning to vision transformers, simultaneously preserving local details and global awareness. Concretely, we process each image in parallel at a local scale (high-resolution, small crops) and a global scale (low-resolution, large crops), and aggregate and propagate features between the two branches with a small set of learnable relay tokens. The design plugs directly into standard transformer backbones (e.g. ViT and Swin) and adds fewer than 2 % parameters. Extensive experiments on three ultra-high-resolution segmentation benchmarks, Archaeoscape, URUR, and Gleason, and on the conventional Cityscapes dataset show consistent gains, with up to 15 % relative mIoU improvement. Code and pretrained models are available at https://archaeoscape.ai/work/relay-tokens/.

Published in Transactions on Machine Learning Research (01/2026)
Adapting Vision Transformers to Ultra-High Resolution Se- mantic Segmentation with Relay Tokens
Yohann Perron
pohann.perron@enpe.ff
Ecole française d' E'Ettrême- Orient (EFBO)
École Nationale des Ponts et Chaussées, IP Paris, Unio Gustave Eiffel, CNRS
Vladyslav Sydorov
aladiplaa..sloorooleeeio...
École française d' E'Eatrême-Orient (EFEO)
Christophe Pottier
christopheepottteereeeeeo
École française d "Estrême-Orienl (EFEO)
Loic Landricu
loie.landrieu@eep...f
Ecole Nationale des Ponts et Chaussées, IP Paris, Univ Gustave Eiffel, CNRS
Reviewed on OpenReview: https: openreview. net/ forum9 id= tadYprMlsg
Abstract
Current approaches for segmenting ultra-high-resolttie images either slide window, thereby discarding global context, or downsample and lose fine detail. We propose simple yet effective method that brings explicit multi-scale reasoning to vision transformers, simultaneously preserving local details and global awareness. Concretely, we process each image in parallel at local scale (high-resolution, small crops) and global scale (low-rosolution, large crops), and aggregate and propagate features between the two branches with small set of learnable relay tokens. The design plugs directly into standard transformer backbones (e.g. VIT and Swin) and adds fewer than 2 % paramcters. Extensive experiments on three altra-high-rcsolution segmentation benchmarks, Archaeoscape, URUR, and Gleason, and on the conventional Cityscapes dataset show consistent gains, with up to 15 % relative mToU improvement. Code and pretrained models are available at http:///rccaaaeaaaaaaaaaa///raas
1 Introduction
Most vision benchmarks operate on images of 224 224 pixels. In contrast, domains such as Earth observation and medical imaging routinely involve scenes spanning hundreds of megapixels regime we refer
ultra-high resolution (UHR) (Sun et al. 2024 Ji et al. 2023 Guo et al. 2022 Shen et al. 2022 Chen et al. 2019). Segmenting such images requires both long-range reasoning to place objects within global context and fino-grained spatial resolution, as targets of interest may occupy only a few pixels. This dual requirement makes it necessary to process extremely large images while preserving detailed local information. The quadratic token token interactions in standard transformer layers (Dosovitskiy et al. 2021 Liu et al.) 2021 2022 Oquab al. 2024 make off-tho-shelf ViT-style architoctures impractical for ultra-high-rcsohun seranttic segmentation.
dliding-windww To circumvent this limitation, existing approaches either (i) apply sliding window inference, which ignores long-range context, or (ii) downsample the input, which foregoes finc structures (sce Fig. T Dedicated multi-scale architectures partly alleviate these issues, but at the cost of bespoke designs and full retrain-
ing CChen et al.) 2019; Wang et al. 2024 Guo et. al. 2022 Ji et al.) (023). Recent linear-complexity transformers (Gupta et al. 2024 Zhu et al. 2024a) can process large images, yet still struggle to reconcile detail and context.

---

Published in Transactions on Machine Learning Research (01/2026)
(a) Archaeoscape
(b) URUR
(c) Gleason
(d) Cityscapes
Figure 1: Context and Details. Examples of scenes where both the high-resolution details (top-left) and
the low-rcsolutinn context (bottom) are necessary to correctly predict the truc label maps (top right).
Relay Tokens. We introduce plug-and-play mechanism that endows any ViT-style backbone with explicit multi-scale reasoning without altering its structure or discarding pretrained weights. Inspired by the Perceiver's latent arrays (Jaegle et al.} 2021), we add handful of learnable relay tokens that are shared between local branch (high-resolution, small crops) and global branch (low-resolution, large crops). During self-attention, these tokens collect features from one scale and propagate them into the other, effectively transmitting information across resolutions (Fig. 2.. The resulting network increases parameter count by less than 2% (only 0.0005% when sharing patch embedding across scales) and incurs negligible runtime overhead compared to processing both local and global windows separately.
Empirical Validation. We evaluate our approach three UHR segmentation benchmarks: aerial LiDAR (Archacoscape), satellite photography (URUR), and histology slides (Gleason), well conventional vision dataset (Cityscapes) We consider two widely used backbone families, ViT (Dosovitskiy et al. 2021
and Swin [Lu et al. 2021 2022), as well as the specialised GLAM architccture (Themyr et al. 2023) and the
linear-complxxity Flatten-Swin (Han et al. 2023 Relay tokens consistently improve mloU, by up to 15 %, while decreasing peak GPU memory by an order of magnitude relative to full-resolution processing.
Our contributions are:
A simple, flexible, and drop-in modification of vision transformers that preserves pretrained weights and adds % parameters.
Extensive experiments on UHR and classic vision benchmarks demonstrating substantial accuracy gains and memory savings compared to other scaling approaches.
2
Related Work
Classical Semantic Segmentation. Fully Convolutional Networks (Long et al. 2015) and U-Nets Ron
neberger et al., 2015) pionccred pixel-wise classification with deop noural networks. Subsoquent architectures improved segmentation quality primarily through enhanced receptive fields, leveraging dilated convolu- tions (Chen et al.) 2017a 2018), pyramid pooling modules (Zhao al., 22017), and boundary-foussed feature extraction techniques (Ding et al.) 2019; Takikawa et al. 2019
Chen al.) 2016). More recently, Vision Transformers (ViTs) (Dosovitskiy et al. 2021 Jain et al. 2023 Zhang al. 2023 Kirillov et al.
2023) have demonstrated superior performance and versatility, shifting the segmentation landscape towards transformer-basee architectures.
Hierarchical Image Representation. Segmenting high-resolution images poses substantial computational and memory challenges. Common approaches inchude sliding window techniques, which introduce boundary artefacts and lose global context, or downsampling preprocessing steps, which blur critical details (Tokunaga) et al.] 2019 Ho et al. 2021). To mitigate these limitations, multiple approaches have been proposed to
2

---

Published in Transactions on Machine Learning Research (01/2026)
integrate local, high-resolution patches with broader low-rcsolution contexts at various network depths, ranging from late fusion (Tokunaga et al.
2019 Wang et al.
2024), intermediate bottleneck fusion (Schmitz)
ett al. 2021), to progressive fusion schemes
(Gu et al. 2018 Chen et al. 2019 Ho et al. 2021). Alternatively,
spatial pyramids (Burt & Adelson 1983
Lazebnik et al 2006 explicitly generate multi-rcsolution feature
2022) or incrementally representations, integrating them either at the final prediction stage (Guo et al.;
throughout the network (Zhu et al., 2024b). Coarse-to-fiee refinement methods iteratively improve segmenta- tion predictions through a hierarchy of using increasingly detailed feature resolutions (Cheng et al.) 2020 Shen et al., 2022). Our proposed relay token approach shares conceptual similarities with these hierarchical methods; however, unlike prior techniques, we retain pretrained transformer backbones unchanged and enable multi-scale reasoning through a small set of shared tokens.
Efficient Architecture Backbone. Another line of research rely on efficient architectures to directly handle largo-scale inputs without explicit multi-scalc fusion. Early solutions leverage the hicrarchical structure inherent to convolutional networks (Chen al.) 2017a). More recent transformer variants reduce
computational complexity with hierarchical window attention (Liu et al.] 2021 Xie et al. 2021 Wang et al. 2021), clustering tokens spatially (Sun et al. 2024), linear-complexity state-space models (Fu et al. 2024 Zhu et al. 2024a Gupta et al., 2024). However, these methods typically require substantial architectural redesign and training from scratch or discard pretrained model parameters. In contrast, relay tokens require minimal modifications, retain pretrained weights, and seamlessly enable multi-scale reasoning capabilities into existing ViT- architectures with minimal overhcad.
Token-Based Cross-Resolution Embeddings. Several works combine multi-resolution information using latent vectors, referred to as global tokens (Themyr et al., 2023 Zhang et al. 2021 memory tokens Thhemyr et al., 2022). Unlike our relay tokens, these methods process the full-resolution, full-context image at every scale, limiting the effective context.. Longformer (Zhang et al. 2021) l and GLAM
(heemyr et al.)
2023) additionally introduce custom multi-resolution blocks on top of Swin (Liu et al.,
2021 whereas relay tokens are a backbone-aguostic drop-in compatible with any ViT-style encoder. Moreover, their global tokens are local to each transformer block and are discarded afterwards; relay tokens persist throughout the network, propagating information across stages and input resolutions. FINE (Themyr et al., persistent memory tokens and is not lied to particular backbone, but its code has not been released, and its evaluation is limited to medical images.
2022) also employs
3 Method
We propose a straightforward extension to Vision Transformers for efficient multi-scale image segmentation; see Fig. _ Specifically, we segment large images (e.g., 5000 x 5000) by simultaneously processing two windows: small, high-resolution local window (2.ocal e.g. 256 x 256) and larger, downsampled global window (zolohal e-g. 1024 1024 downsampled to 256 256). These paired windows capture both high-resolution details and broader context. In Sec. [.1 we describe the minimal changes required to adapt existing ViT models, and in we detail our training losses.
Sec. 3.2
3.1 Architecture
We first briefly review standard ViTs and then describe how to add relay tokens.
Background: Vision Transformers. A standard ViT (Dosovitskiy et al. 2021 segments images by first dividing them into fixed-size patches (typically 16 x 16 pixels). Each patch is then embedded to a D-dimensional vector by projector 6ami, augmented with positional encodings, and processed through B transformer blocks øbllcck ..... Φ blookk A segmentation head geeg maps the resulting embeddings to pixel-level predictions. Formally, given the patches plocal of the local window loccal the segmentation predictions glocal are defined as:
z glccll 20eR dB
block o... @block (prol(p)ocal) + (po(())caa)),
(1)
where pos denotes the positional encoding function.
3

---

Published in Transactions on Machine Learning Research (01/2026)
Crop, downsample and patchify
xB
Alobal
piproj
b
block
Calobal
z:llbbll
cooa
@
block
glsng
Cloesi
Crop and patchify local
local
local
Figure 2: VIT with Cross-Resolution Relay Tokens. We simultaneously process a small high-resolution window gloca and a larger low-resolution window zalobal with a shared network augmented with relay tokens D. In cach of the consocutive transformer blocks, relay tokens are first addod the patch tokens of the global window and processed by the transformer. The updated relay tokens then processed with the tokens of the local window and passed to the next block. The model applies supervision at each scale using losses Lalobal and Clocal, while enforcing consistency between resolutions via a cross-resolution loss geons, We denole shared weights with @ and independent parameters with &
Shared Weights. We extend this framework by jointly processing both local (zlocul) and global (aslobal) windows using the same transformer, with weights shared across all modules except the projection layers. We first embed the local and global patches poeal and pelobal separately:
flocal proj (pt local) pos
(2)
gglobal d
global (p global pos,
(3)
Relay Tokens. We add R relay tokens frelay E R learned as free parameters of the model. Each transformer block b = ,,....,, B sequentially updates tokens in two steps:
Algorithm 1: Vision Transformer With Relay Tokens. PyTorch code to add relay tokens to a ViT, here !..' stands for standard VIT code. See Appendix for more details.
[ paloball = ob
block (Ist : (global))
(4) embedding dimension
number of relays
[sndley plornl] block (1mrlay, flocal))
(5)
class VIT withwith_ae relaye(nn) (na. Module)) def _.intt__ (self, D, R, ...))
where [.-]] concatenates tokens, and b + 1 indicates the intermediate update of the relay tokens after pro- cessing the global tokens only. In Eqs. 4 and (5) the output of @block is split such that the first term corresponds to the R updated relay lokens, and the second one the image token embeddings. After the final block, the relay tokens are discarded. We then apply the same segmentation head to the final local and global features:
self relay nn. Parameter (torch randr (R, D))
def (erlf, ing_loc, img.glo):
anc_ityeetdpppeeeeeeeeeee 1_loc patchify_ and_l__nd____________ project (ing loc)
f_glo seccciff_and__roooeee proj_gt(imm_glo (img_glo) f_r = repeat relay, batch_size)
for block in
f_r_glo block(torch. cat(((_r, f_glo),
f_glo, f_r f_r_glol; R:], f_r_glol; :R]
f_r_loc block(torch.cat f_loc) ,)) f_loc, f_r f_r_loc[; R:], f_r_loc[;, :R]
local
00B (f
(6)
2 globall
( palobal
(7) return out
Algorithm 1 shows the code adding relays to ViT.
4

---

Published in Transactions on Machine Learning Research (01/2026)
Beyond ViTs. Adding relay tokens to vanilla VIT requires only few extra lines of code (see Algorithm Extending the same principle to hierarchical models like SwinV2 (Liu et al. 2022 or even more complex approaches such as GLAM (Themyr et al.) 2023) is similarly simple: we copy the relay tokens for each window-attention block, then average their updated values across windows before applying the feod-forward layer.
3.2 Training
Our training procedure employs three complementary loss terms to encourage precise segmentation at both local and global scales, while enforcing consistency between them. All supervision leverages ground truth labels y /occl associated solcly with the high-resolution local window.
Local Supervision. We supervise the local prediction glocal using standard cross-entropy loss:
Clocal
(2 (olaall, local)
(8)
where H(z,y) = Ex ye log(zz).
Global Supervision. We encourage the global- scale embeddings to focus primarily on context rel- evant to local-scale segmentation accuracy. We first crop the global predictions zglobal to the spatial ex- tent of the local window, denoted as crop(>#lobal) Next, we apply average pooling (avg) to aggregate the high-resolution local labels glocul into reser-resolttion distributions. The global loss is:
gglobal
crop( crop(28loha) 281obal
local
Galobal (crop( loca))
(9)
Figure 3: Predictions at Different Scales.
Consistency Loss. Similarly to SGNet (Wang et al. 2024 align predictions across scales with consistency loss. Specifically, we match the cropped global predictions to the spatially averaged local predictions:
coons__ H (crop( avg((=loca))
(10)
This term encourages effective cross-rosolution commuication, which may reduce boundary discontinuities and enhance the segmentation consistency.
4
Experiments
We first present the datasets and baselines in Sec.
4.1
We then discuss our main experimental results
(Sec. 4.2 and provide an extended analysis (Sec. 4.3 followed by an ablation study (Sec. 4.4
4.1
Datasets and Baselines
We evaluate our method on four datasets that collectively cover LiDAR archacology, satellite image analysis, histopathology, and urban-scene imagery. Performance is measured by mean Intersection-over-Unne (mloU).
Datascts. We evaluate all models on three ultra-high resolution image sogmentation datasots and one computer vision dataset, illustrated in Fig. 4 (see Appendix for additional details):
Archaeoscape (Perron et al., 2024) is LiDAR RGB dataset spanning 888 km2 at 0.5m ground-sampling distance. It compriscs 23 parcols of up to 720 x 106 pixels, densely annotated with four classes (water features, earthen mounds, temples, background). We report mloU over the three foreground classes on the test split.
5

---

Published in Transactions on Machine Learning Research (01/2026)
Cityscapes
Archacoscape
URUR
Gleason
Imput
GT
Sliding win.
Relays
Figure 4: Qualitative Results. We visualise a 2048 1024 region from each dataset, comparing our multi-scale approach the baseline sliding window. We highlight the extents of the global and local images and mark points of interest with O.
URUR (Ji et al., 2023) contains 3,008 satellite images of 5012 x 5012 pixels with eight land-cover classes. We follow the standard split (2,157 280 / 571 images for train val test).
Gleason (Karimi et al., 2020) features 244 histology micro-array tiles (5120 x 5120 pixels) across seven slides, labelled as benign or one of three Gleason grades. Official servers are no longer available
lo evaluate on the lest set and recent works did not disclose their test split (Wang el al. 2024). We use slide as our test set, train on the remaining slides, and reserve images with core number under 15 for validation.
Cityscapes (Cordts et al., 2016) offers 2,975 training and 500 validation street-view images at 1024 x 2048 resolution with 19-class dense annotations. While not UHR, this dataset provides a familiar benchmark to evaluate our design in a classic computer vision setting. We train on the train split and report mloU on val.
Baselines and Competing Methods.
We benchmark our approach against several approaches:
Sliding Window. We evaluate three backbone ViT models
VIT (Dosovitskiy et. al., 2021
SwinV2 (Liu et al.
2022). and GLAM (Themyr al. 2023) in sliding window setting, with and
without relay tokens.
Scalable Vision Transformers. We evaluate Flatten Swin - Swin configuration of Flatten Trans-
2023). which employs linear-attention modules inside a Transformer architecture. We also evaluate two ViT models able to scale to large windows thanks to their lincar-complesity transformer replacement: xT (Gupta et al.] 2024) and Vision Mamba (Zhu et al. 2024a
former (Han et al.
Multi-Scale Approaches. We evaluate three models that also process large images with a dual sliding window approach: GLNet (Chen et al. 2019h, SGNet (Wang et al. 2024 and ISDNet (Gu) et al.) 2018). We use the window sizes recommended in the articles.
Classic Models. We also evaluate classic models such DeepLabv3 (Chen el al. 2017b U-Net Ronneberger et al.) 2015), and PVTv2 (Wang et al. 2022) to provide context in terms of performance. To reflect the lesser memory usage compared to transformers, we use a larger sliding window size of 512 for CNN-based models.

---

Published in Transactions on Machine Learning Research (01/2026)
Table 1: Quantitative Evaluation. Performance (mloU) for different scaling strategies on three UHR and one vision dataset. We report the (+% mIOU) improvement of relay tokens over the sliding-window window bascline.
SW sliding window
4 subsampling
performance on the test set performance on an undisclosed test set
MS multi-scale
full entire input image
Archaeoscape
URUR
Gleason
Cityscapes (val)
Backbone
Scaling strategy
up 30K 40K
5012
5012
5120 x 5120
1024 x 2048
VIT
SW: 256
46.5
36.3
33.2
53.0
+ relays (ours)
MS: 256 + 1024 4 4
46.7 (+0.2)
40.4 (+4.1)
49.1 (+15.9)
61.0 (+8.0)
SwinV2
SW: 256
51.9
41.0
48.1
68.2
+ relays (ours)
MS: 256 + 1024 J
57.8 (+-5.9)
46.4 (+5.4)
55.5 (+7.4)
75.1 (+6.9)
Flatten Swin
SW: 256
53.4
43.0
50.7
70.2
+ relays (ours)
MS: 256 + 1024 J
55.2 (+1.8)
45.8 (+2.8)
57.7 (+7.0)
77.5 (+7.3)
GLAM
SW: 256
52.5
43.9
47.7
72.9
+ relays (ours)
MS: 256 + 1024 J
53.8 (+1.3)
44.5 (+0.6)
54.1 (+6.4)
76.9 (+4..0)
X
SW: 1024
40.6
44.0
46.0
68.6
Vision Mamba 6
SW: 1024
45.9
31.6
22.1
44.8
GLNet
MS: 512 full +
41.2
71.2
SGNet 8
MS: 512 + 1024
61.2
70.4
ISDnet
MS: full + full +
43.6
45.8
60.0
76.0
DeepLabv3 10
SW: 512
49.7
41.7
46.3
56.6
U-Net 11
SW: 512
49.9
43.1
53.5
67.8
PvTv2 12
SW: 256
52.1
41.2
48.6
66.1
SOTA
52.1
46.9
14
87.4
Dosovilskiy al. 2021 2 Liu 2022
8 HHan al. (2023
Themyy el
2023)
Gupta et
2004
Zhu al.
2024a;
Chen
(019)
Wang al.
2024)
Guo al.
2022
Chen al. 2017b)
Ronneberger al. 2015
Wang ct al. (0222)
Perrou et
2024
Ji ct
2023
Erisen (2044
Notations. We use the downsampling factor notation n. k to indicate that an n X n image is downsampled k times; e.g. 1024| 4 represents a 1024 x 1024 image downsampled to 256 x 256. The '+' operator denotes models combining multiple scales. For example, 256 + 1024| 4 refers to a model using a 256 x 256 local window and a 4x downsampled 1024 x 1024 crop as global image. We refer to the entire input image as 'full'
Dataloader. For each training step, we randomly select one pixel as the centre of both local and global crops. If more than 80% of the local window lies outside the main image, we skip that sample. We pad the out-of-bounds parts of the local and global windows with the image mean. We apply random scaling in [0.5, 2] and random rotations in [-90-, ] each with 0.5 probability. We do not use any test time augmentations.
Implementation Details. All models utilise the S (small) variant of each backbone, initialised from ImageNet (Ridnik et al. 2021 Russakovsky et al. 2015), and four relay tokens. For all datasets, we train with the combined loss [ Local + 0.1 gglobal + 0.1 C geons using AdamW (Kingmaa & Ba, 2015 Loshchilov & Hutter, 2019) with an initial learning rate of 10-4, no weight decay, a linear warmup for 5% of training, and a ReduceLROnnllten (PyTorch 2.7p schedule based on validation accuracy. For URUR and Gleason with ViT and SwinV2 backbones, the patch embedding networks of both scales share parameters. In all other cases, including Cityscapes, Archacoscape, and experiments with Flatten Swin and GLAM backbones, these networks are trained independently.

---

Published in Transactions on Machine Learning Research (01/2026)
4.2 Results
We compare relay lokens against standard scaling baselines and recent state-of-the-arl (SOTA) models in terms of precision and efficiency.
Quantitative Analysis. Table | demonstrates that relay tokens consistently outperforms the sliding- window (SW) baseline across all four evaluated datasets and four backbone architectures, achieving substantial improvements: up to +5.9 on Archacoscape, +5.4 on URUR, and +15.9 on Gleason, and +8.0 mIoU on Cityscapes. Au exception to this observation the case of vanilla ViT on Archaeoscape, which have already been observed to perform poorly (Perron et al. 2024). and relay tokens do not appear to significantly improve this performance.
Despite being a simple extension of ViT-based architectures, relay Lokens match or surpass specialised multi-scale methods such as ISDNet, GLNet, and SGNet across all UHR datasets. Transformer variants with lincar attention, such as Flatten Swin, alrcady perform competitively but are consistently improved by the addition of relay tokens. complexcy Mamba models (Vision-Mambee and xT), while capable of processing full images in single pass, generally underperform in our setting. CNN-based baselines remain computationally efficient but lag behind in accuracy.
Comparison to SOTA. Our main objective in this paper is not to chase incremental SOTA improvements against highly specialised approaches but to demonstrate how a simple, practical strategy can significantly enhance standard ViT-based architectures. Nevertheless, a SwinV2 backbone equipped with relay tokens achieves a clear SOTA on Archaeoscape by over 5 points, and trails by only 0.5 mIoU points behind the highly specialised remote sensing model WSDNet (Ji et al., (023)) on URUR without any structural modifications. On Gleason, public test servers are no longer available and recent works rely on private splits (Wang et al., 2024], which complicates fair comparison; evaluated on our clean train/Lest split, Flatten + lokens outperform all other approaches.
Although Cityscapes uses higher-resolutie images than most vision benchmarks, it is not an UHR dataset. We therefore do not expect relay tokens to reach state-of the--rrttt---rtttee performance against highly specialised architoctures such as Normer (Erigen 2024), which reports m7.4 Instead, our goal to demonstrate that relay tokens provide a lightweight, efficient, and easily integrable improvement to standard vision transformer pipelines. Accordingly,
do not rely on practices commonly used to maximise performance Cityscapes (Chen et al.) 2018 Tao et al. 2020 Xie et al., 2021), such as pretraining on the 20K coarse annotations or on Mapillary Vistas (Ncuhold et al. 2017), nor do we apply 10-scale test-time augmentation. Under these conditions, Flatten Swin and GLAM with relay tokens report the highest performance among all evaluated methods.
Qualitative Analysis. Figure 4 illustrates how relay tokens provide more coherent and contextually aware predictions across datasets. On Cityscapes, the local-only model mislabels partial objects (e-g., identifying cyclist's torso a pedestrian), whereas including global context resolves such ambiguities. Likewise, Archaeoscape, global windows help identify large water feat (blue) and earth structures (yellow). On URUR, fields (light green) are better delineated against barelands (grey) and woodlands (dark green). For Gleason, the sliding window approach is not able to consistently classify the cell as malignant (bluc), and predicts either normal tissue (white) or another cancer class (green).
Comparison to Other Scaling Approaches. We benchmark four common altornatives:
Sliding Window. We independently segment local windows local and stitch predictions, with overlap ratio of 50% for Archaeoscape and 0% elsewhere.
Sliding Window + Registers. We add relay tokens but process only the local window. This amounts to adding registers (Darcet et al.) 2024).
Downsampling. We process only the global window at a reduced resolution. the image remains too large, we slide the global window.
Decision Fusion. We process local and global windows in parallel, upsample the global prediction to the local resolution, and average the final logits.
8

---

Published in Transactions on Machine Learning Research (01/2026)
Table 3: Scaling Baselines. We evaluate several scaling strategies with a SwinV2 backbone. We report the improvement with respect to the sliding window baseline with a red-lo-green colour scale
Scaling strategy
Archaeoscape
URUR
Gleason
Cityscapes (val)
Sliding window
SW:
51.9
41.0
48.1
Registers
SW: 256
53.2
+1.3
41.4
+0.4
47.8
-0.3
68.0
-0.2
Downsampling
SW: 1024/4
47.1
-4.8
45.3
+-1.3
55.1
+7.0
61.8
-6.4
Decision fusion
MS: 2524 4
56.5
+-4.6
43.8
4.2.8
55.1
+7.0
72.3
+4111
Relay tokens (ours)
MS: 256+1024] 4
57.8
+5.9
45.6
+4.6
57.0
+8.9
75.1
6.9
256 1024 4
1024
75
512
Performance (mIOU)
256 + 1024|
512 + all
70
10244
256
1024
512
65
1024 4
Notations
Scaling strategy
60
256
+ 1024
Relay tokens
Decision fusion
256 X
local
global window
downsample factor
Sliding window
GLNet
window
Downsampling
X xT
55
25
50
100
250
500
1000
A6000
A100/H100
Memory Requirement (in GB)
Figure 5: Performance vs. Memory. We plot the training GPU memory requirement and performance of a SwinV2 on Cityscapes for several scaling strategics. Relay tokens offer a better trado-off between memory and precision. In particular, we can train a SwinV2 model with relay tokens on a GPU with 48GB of RAM while outperforming models that require 8 80GB GPUs.
Table 3 shows that decision fusion is the strongest baseline, yet relay tokens outperform it by exchanging multi-scale features at every block rather than only at the output. Registers alone yield small, inconsistent gains, confirming that performance stems from genuine cross-scale interaction.
Complexity Analysis. Adding relay tokens to existing ViT or Swin model introduces only a small number of additional learnable parameters: the relay tokens themselves and an additional copy of the projec- for. practice, this results in less than 1.7% additional parameters for ViTs and 0.7% for Swin variants. Adding one relay token to ViT-S adds only 384 parameters (0.002% of the model) and 96 for SwinV2-S (0.00014%); the remaining additional parameters come from the resolution-specific projectors.
Table 4: Complexity Analysis. For a 256 px imput, without sharing the patch encoder.
Model
#Param, [M] GFLOPs
ViT
23.9
6.6
VIT + relay tokens
24.3
(+1.6%)
13.6 (+106%)
SwinV2
64.7
12.6
SwinV2 + relay tokens 65.1
(+0.9%%)
26.0 (+106%)
GLAM
132
(+1.6%)
75.4
In contrast, methods such as ISDNet (Guo et al. 2022 and GLNet (Chen et. al., (2019) substantially increase their parameter count by using separate backbones for local and global processing, in addition to other auxil- iary modules. Tab. compares parameter counts and GFLOPs of different models.
GLAM relay tokens
133
(+0.77)
152 (+102%)
DeepLabv3
39.6
41.0
xT
166
20.8

---

Published in Transactions on Machine Learning Research (01/2026)
Performance us. Memory Trade-off. Although process- ing dual-scale (local and global) windows approximately dou- bles the memory and computational costs compared to single- scale sliding-window baselines (see Tabs. and 5/. relay tokens remain significantly more efficient than approaches operating at high resolution. Figure illustrates this cfficiency-accuracy trade-off for Cityscapes: using relay tokens and a single 48 GB GPU, we achieve competitive performance to a SwinV2 applied with 1024 1024 window, which requires eight 80 GB GPUs to train. Decision Fusion also presents comparable efficiency trade-off but still falls short in accuracy.
Table 5: Computational Costs. Training time and throughput for a 256 px input.
Epoch time
Throughput (full img/s)
Backbone
(minutes) Train
Train
Test
VIT
32
9.3
31.1
ViT + relays
78
4.5
14.5
SwinV2
33
3.1
9.9
SwinV2 + relays
75
1.4
4.7
Test Set Performance for Cityscapes. Due to limited Cityscapes evaluation server submissions, we test only two SwinV2-based configurations. The dlding-windoon baseline reaches 69.1% mloU, while adding relay tokens improves it substantially to 75.0% (+5.9 points). These lest set results closely aligu with our validation set outcomes, confirming the robustness of our experimental protocol.
4.3 Analysis
Despite its conceptual simplicity, our approach reveals valuable insights into how context and resolution interplay in modern segmentation architectures. Below, we invostigate several critical aspects.
How Large Should the Context Be? The effectiveness of additional global context varies across datasets.
For instance, Gleason images contain over 25M pixels cach, strongly benefiting from extended spatial context. We represent in Fig. 0 the performance improvements relative to the sliding-window baseline across various spatial extents and downsampling ratios. We observe diminishing returns beyond a global context of 1024 pixels with fixed downsampling ratio, espocially given that comes a high cost due to quadratic growth of the attention matrix; for this reason, we were not able window. Conversely, excessively coarse global windows (e-g., downsampling ratios beyond 8x for fixed global size) provide limited benefit. This highlights dataset-dependent trade-off between memory, coverage, and resolution.
run the experiments with a 4096 J global
How many Tokens Should One Use? Figure explores the impact of the token count. Performance gain saturates after relay tokens, a common behaviour for register/glos /global tokens (Zhang et al. 2021 Fig. 3). Computation grows quadratically a moderate pace: only +226% from 32 tokens.
Figure 6: Impact of Global Window Extent. Performance function of the global window size in pixels at full image resolution. Two vari- ants are tested: fixed downsampling ratio (L 4) and a fixed global window size (256 x 256).
12
aunu Moooonnno (%)
9
Datasets
Downsampling
Gleason
Cityscapes
Fixed /
Archacoscape
To 256 256
URUR
256
512
1024
2048
4096
Global window extent (pixels)
10

---

Published in Transactions on Machine Learning Research (01/2020)
60
Perfomance
30
GFLOPS
Figure 7: Impact of Relay Count.
55
20
We evaluate several configurations for a SwinV2 model on Archacoscape.
50
10
0
8
16
32
# of relay tokens
Relay 1
Relay 2
Relay 3
Relay 4
Figure 8: Relay Tokens Attention Maps. We compute the average al- tention between the relay tokens and each of the patch tokens of the local and global windows, averaged for all heads, blocks, and 320 random images of Cityscapes. Specialised, complemen- tary spatial patterns emerge.
Local
Global
Where Do Relay Tokens Look? In Fig. 8 visualise the average attention maps of four relay tokens in the local and global windows; more illustrations be found in the Appendix. Two notable patterns emerge:
Radialisation: Each relay token consistently specialises in a particular direction of the image (e.g., NE, NW, SE, SW corners). This phenomenon persists across blocks and heads, presumably because token-wise MLP operations favour consistent specialisation.
Local Global Harmonisation: As a given relay token focuses in a corner of the local window, also gathers global information in the surrounding area of the global window, This suggests that the networks processing each window are able use the relay token to gather and propagate information in spatially consistent manner. Moreover, the global attention avoids the centre of the image, where the local window already contains more fine-grained information.
Which Objects Benefit Most From a Large Context? We measure which object categories gain most from relay tokens compared to a sliding-window baseline (SW) using the relative IoU error improvement: (loUrelay LoUgw)/(1 loUgw).
The classes of Cityscapes with the most gains are medium-scall objects (truck +44%, bus +39%, train +33%, motorcycle +25%, car +20%) classes defined by their function rather than appearance (road +-39%, sidewalk +24%). Smaller objects with fine-grained structure (fence +13%, pole +6%, traffic light +9%, and traffic sign +8%) gain less from coarse global cues. Similar trends appear across datasets: ancient Khmer reservoirs in Archaeoscape (+16%), greenhouses (+15%) in URUR, and malignant cells (+35%) in Gleason also notably benefit from the enhanced context.
4.4 Ablations
We evaluate the impact of different design choices on the performance and efficiency of our proposed approach.
Architecture Ablation. We report the impact on performance several key architectural design choices
in Tab. 0
Losses. Removing either the global loss galobal or the consistency loss conns degrades performance by approxinately 1.0 to 1.9 mloU points, respectively. Simultaneously omitting both losses leads to larger performance drop of 2.5 points, demonstrating their individual yet complementary contributions.
11

---

Published in Transactions on Machine Learning Research (01/2026)
Table 6: Impact on Performance. Effect individual design choices evaluated in two representative settings: conf. A ViT on Gleason, and conf. SwinV2 on Archaeoscape. Grey denotes unchanged configuration from the default.
A
B
Relay Tokens
49.1
57.8
2 relays
49.3
51.2
Sliding Window
33.2
51.9
8 relays
46.3
55.7
L
47.2
52.1
Token Concatenation
46.8
no Calobal
47.8
50.9
Shared Projectors
49.1
49.6
Leous Galobal
48.1
48.6
Different Projectors
46.3
57.8
Table 7: Impact Efficiency, We evaluate on the Cityscapes validation set the performance and efficiency of different relay-token variants: Fewer Blocks match the bascline compute, and Parallel Relays facilitate parallelisation.
Model
Variant
mTOU
FLOPs
Throughput (img/s)
ViT
Full
53.0
995
VIT + Relay
Full
61.0
13.6
465
VIT + Relay
Fewer Blocks
55.0
6.8
898
VIT + Relay
Parallel Relays
56.4
13.6
488
Number of Relay Tokens. We vary the number of relay tokens, testing configurations with 2, 4 (default), and 8 tokens. Using only 2 relay tokens yiclds insufficient improvements over the bascline, while employing 8 relay lokens is slightly less effective than 4. This finding aligns with observations on register token parameterisation for segmentation tasks (Darcet et al. 2024 Fig. 8). Similar trends occur across other datasots, although results might differ for datasots with more intricate multi-scale interactions.
Token Concatenation. An alternative multi-scale approach is to concatenate tokens from local and global windows before processing them through single network (without relay tokens). While straightforward for plain VIT architectures, this approach is complex for Swin models duc to their window-based attention mechanism. It also significantly increases the computational footprint (approximately four times the baseline, twice that of relay tokens). Despite this additional resource requirement, concatenation performs worse than relay tokens, suggesting that relay tokens are more effective capture multi-scale features than full self-attention between patches.
Shared Projectors. The networks processing the local and global windows share over 99% of their weights, except the relay token and in some cases, the projectors. Whether or not sharing projectors is beneficial to the model is very dataset specific: on Gleason using different projectors causes loss of 2.8% (VIT) while it improves the performance by +8.2% on Archaeoscape (SwinV2). This underlines how different datasets can have widely different scale-invariance propertics, which can justify (in the case of Gleason and URUR) a shared processing across scales.
Adaptive Positional Encoding. Inspired by ScaleMAE (Reed al., (023), we explore using positional encoding whose scale depends on the downscaling factor. Contrary expectations, this approach noticeably degraded performance. We hypothesise this is due to incompatibility with positional encodings learned during the model's initial pretraining.
Relay Token Variants. By design, introducing relay tokens approximately doubles the number of transformer blocks and the computational load. To ascertain whether the gains come from additional compute or from improved cross-scale interaction, we run two targeted experiments in Tab.
12

---

Published in Transactions on Machine Learning Research (01/2026)
Fewer Blocks. We keep only the first six out of twelve transformer blocks in both the local and global branches, thus matching the computational cost of the baseline model (ViT-S with sliding window). This variant still perform better than the VIT baseline, indicating that relay tokens provide benefits beyond pure compute scaling. While it underperforms the full model, this variant offers a strong accuracy accuracy-effcienncy trade-off in compute-conslrainee settings.
Parallel Relay. limitation of our default design that global and local windows are processod sequentially, which reduces parallelism. We therefore consider paralles relay variant in which global and local branches are executed concurrently, each with its updated relay representation is obtained by averaging the tokens produced by both branches. This design improves parallelization relative to the sequential global to local scheme, at the cost of weaker cross-scale coupling. Specifically, we replace Eqs. 14 and 5
copy of the relay tokens. The
in the paper by:
[bbt3 prellv(sllbal) pelobalj = % bllock (Ioorhy,, sgloball) s6+1
(11)
bb4
( /bocell
Pb ablock (Ifoelay, sbocal))
(12)
Mt1 = 0,5 relay(global) + bo7 pollyylbcal)
(13)
This yields only modest throughput gains, suggesting that GPU utilisation is already efficient with the the original sequential relay. However, the performance improvement relative to the VIT baseline drops significantly, suggesting that sequential relay feedback is important for delivering effective, scale-adaptive feedback.
Conclusion
We introduced relay tokens, wight extension of vision transformer models, enabling effective multi-scale reasoning for large-image segmentation. With minimal additional parameters and straightforward imple- mentation, relay tokens significantly boost segmentation accuracy across diverse datasets and domains. Our method offers efficient, easily deployable alternative to specialised multi-scale or large-scale segmentation architectures, enabling practitioners to scamlessly integrate multi-scale reasoning capabilities into existing transformer-based workflows.
Acknowledgements and disclosure of funding
The experiments conducted in this study were performed using HPC/AI resources provided by GENCI-IDRIS (Grants 2024-AD011014481RII 2024-AD011011196R1, 2025-AD011014713322)
This work was supported by the ANR project sharp ANR-23-PEIA-000S in the context the PEPR IA, by Hi! PARIS under the France 2030 program (ANR-23-IACH 13-1000-00055 and by Paris Ile-de-France Region through We thank Nicolas Dufour, Antoine Guédon, and Lucas Ventura for inspiring discussions and valuable feedback.
the DIM PAMIR network IDD--IIMAPAMMR-22555444
References
P. Burt and Adelson. The Laplacian pyramid as compact image code. IEEE Transactions on Communications, 1983.
Liang-Chieh Chich Chen, Jonathan T Barron, George Papandreou, Kevin Murphy, and Alan L Yuille. Semantie image segmentation with task-specific edge detection using cnns and discriminatively trained domain transform. In CVPR, 2016.
Liang- Chich Chen, George Papandreou, Lasonas Kokkinos, Kevin Murphy, and Alan L Yuille, Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 2017a.
Liang-Chieh Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. 1706. 2017b.
13

---

Published in Transactions on Machine Learning Research (01/2026)
Liang- Ching-CCich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Harlwig Adam. Encoder-decoder with atrous separable convolution for semantic image seguentation. In ECCV, 2018.
Wuyang Chen, Ziyu Jiang, Zhangyang Wang, Kexin Cui, and Xiaoning Qian. Collaborative global-local networks for memory-efficiet segmentation of altra-hight resolution images. In CVPR, 2019.
Ho Kei Cheng, Jihoon Chung, Yu- Wing Tai, and Chi-Keung Tang. CascadePSI:: Toward class-agnostic and very high-resolution segmentation via global and local refinement. In CVPR, 2020.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Euzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The Cityscapes dataset for semantic urhan scene understanding. In CVPR, 2016.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In ICLR, 2024.
Henghui Ding, Xudong Jiang, Ai Qun Liu, Nadia Magnenat Thalmann, and Gang Wang. Boundary-aware feature propagation for scene segmentation. In ICCV, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa. Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Serdar Erigen. Fermet-Former: Segmentation by efficient-Reshle with attention-boostin gates and attention- fusion networks. In CVMI, 2024.
Yunxiang Fu, Meng Lou, and Yizhou Yu. SegMAN: Omni-scale context modeling with state space models and local attention for semantic segmentation. arXin:24 12. 1890, 2024.
Feng Gu, Nikolay Burlutskiy, Mats Andersson, and Lena Kajland Wilén. Multi-resolution networks for semantic segmentation in whole slide images. In MICCAI, 2018.
Shaohua Guo, Liang Liu, Zhenye Gan, Yabiao Wang, Wuhao Zhang, Chengjie Wang, Guannan Jiang, Wei Zhang, Ran Yi, Lizhuang Ma, al. ISDNet: Integrating shallow and deep networks for efficient ultra-high resolution segmentation. In CVPR, 2022.
Ritwik Gupta, Shufan Li, Tyler Zhu, Jitendra Malik, Trevor Darrell, and Karttikeya Mangalam. xT: Nested tokenization for larger context in large images. 1CML, 2024.
Dongchen Han, Xuran Pan, Yizeng Han, Shiji Song, and Gao Huang. FLaiten Transformer: Vision Transformer using Focused Linear Attention. In ICCV, 2023.
David Joon Ho, Dig Vijay Kumar Yarlagadda, Timothy M. D'Alfonso, Matthew G. Hanna, Anne Grabenstetter, Peter Ntiamoah, Edi Brogi, Lee K. Tan, and Thomas J. Fuchs. Deep multi-magnifiiationn networks for multi-class breast cancer image segmentation. Computerised Medical Imaging and Graphics, 2021.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In ICML, 2021.
Jitesh Jain, Jiachen Li, Mang Tik Chin, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer rule universal image segmentation. In CVPR, 2023.
Deyi Ji, Feng Zhao, Hongtao Lui, Mingyuan Tao, and Jieping Ye. Ultra-high resolution segmentation with ultra-rich context: A novel benchmark. In CVPR, 2023.
Davood Karimi, Guy Nir, Ladan Fazli, Peter C. Black, Larry Goldenberg, and Septimiu E. Salcudean. Deep learning- based gleason grading of prostate cancer from histopathology images role of multiscale decision aggregation and data augmentation. IEFF Journal of Biomedical and Health Informatics, 2020.
Diederik P Kingma and Jimmy Ba. ADAM: A method for stochastic optimization. ICLR, 2015.
Alexander Kirillov, Eric Mintun, Nikhila. Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In 1CCV, 2023.
S. Lazebuik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In CVPR, 2006.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In CVPR, 2021.
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, al. SWIN transformer v2: Scaling up capacity and resolution. In CVPR, 2022.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. CVPR, 2015.
1lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. JCLR, 2019.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dalaset for semantic inderstanding of street scenes. In CVPR, 2017.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without. supervision. TMLR, 2024.
Yohann Perron, Vladyslav Sydorov, Adam P. Wijker, Damian Evans, Christophe Pottier, and Loic Landrieu. Archaeoscape: Bringing aerial laser scanning archaeology to the deep learning era. NewrIPS 2024 Dataset and Benchmarks Track, 2024.
PyTorch 2.7. POnaaeel documentation. httpp://docs... wwwpsspytttooooooo httpr///.g....e//aaaa////////aaa optim. Ir_ scc.euulerr.... ReduceLRROPPlaeeuu html Accessed: 2025-08-04.
Colorado J Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell. Scale-MAR: A scale-aware masked autoencoder for multiscale geospatial representation learning. In CVPR, 2023.
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. ImageNet-211K pretraining for the masses. In NewrIPS Datasets and Benchmarks Track, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmenta- tion. In MICCAI, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. ImageNet large scale visual recognition challenge. IJCV, 2015.
Rüdiger Schmitz, Frederic Madesta, Maximilian Nielsen, Jenny Krause, Stefan Steurer, René Werner, and Thomas Rösch. Multi-scale fully convolutional neural networks for histopathology image segmentation: From nuclear aberrations to the global tissue architecture. Medical Image Analysis, 2021.
Tiancheng Shen, Yuechen Zhang, Lu Qi, Jason Kuen, Xingyu Xie, Jianlong Wn, Zhe Lin, and Jiaya Jia. High quality segmentation for ultra high-resolution images. In CVPR, 2022.
Haopeng Sun, Yingwei Zhang, Lumin Xu, Sheng Jin, and Yiqiang Chen. Ultra-high resolution segmentation via bounary-enhanced patch-merging transformer. Xrx: 12. 10.1111,, 2024.
Towaki Takikawa, David Acuna, Varun Jampani, and Sanja Fidler. Gated-SCNN: Gated shape CNNs for semantic segmentatton. In ICCV, 2019.
Andrew Tao, Karan Sapra, and Bryan Catanzaro. Hierarchical multi-scale attention for semantic segmentation. arXiv:200..1 10821, 2020.
Loic Themyr, Clément Rambour, Nicolas Thome, Toby Collins, and Alexandre Hostettler. Memory transformers for full context and high-resolution 3d medical segmentation. International Workshop on Machine Learning in Medical Imaging, pp. 121 130. Springer, 2022.
Loic Themyr, Clément Rambour, Nicolas Thome, Toby Collins, and Alexandre Hostettler. Full contextual attention for multi-resolltiinn transformers in semantic segmentation. In WACV, 2023.
Hiroki Tokunaga, Yuki Teramoto, Akihiko Yoshizawa, and Ryoma Bise. Adaptive Weighting Multi-Field-On VieW CNN for Semantic Segmentation in Pathology. In CVPR, 2019.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Sai Wang, Yutian Lin, Yu Wu, and Bo Du Toward real ultra image segmentation: Leveraging surrounding context to cultivate general segmentation model. In NeurIPS, 2024.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.
Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In CVPR, 2021.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. PVTv2: Improved baselines with pyramid vision transformer. Computational Visual Media, 2022.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. SegFormer: Simple and efficient design for semantic segmentation with transformers. NewrIPS, 2021.
Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. In ICCV, 2023.
Pengchuan Zhang, Xiyang Dai, Jianwei Yang, Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Cao. Multi-scale vision longformer: A new vision trausformer for high-resolution image encoding, In ICCV, 2021.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyn Liu, and Xinggang Wang. Vision Mamba: Efficient: space model. In ICML, 2024a.
visual representation learning with bidirectional
Xizhou Zhu, Xue Yang, Zhaokai Wang, Hao Li, Wenhan Dou, Junqi Ge, Lewei Lu, Yu Qiao, and Jifeng Dai. Parameter-inverrs image pyramid networks. In NewrlPS, 2024b.
16

---

Published in Transactions on Machine Learning Research (01/2026)
A-1 Implementation
We provide in Algorithm A-1 an extended implementation of ViT with relay tokens.
Algorithm A-1: VIT with Relay pseudocode.
(a) ViT with Relay, Minimal implementation.
(b) ViT with Relay. Auxiliary functions.
class V1T relrrlay (on, Nodule):
import torch
def ... init___ (self, B, D. P. in_., out_c, R):
import torch.nn
from einops import rearrange
self.p
self.proj = mn (P * P in_c, D)
patchify(img, patch_size):
self. blocks nn ModuleList
(B, C, -> (B, #patches, patch_eize"2
(Transformer(0) for in rBng)(B)]) self.out_head nn. Linerr (D, P * * out_c)
return rearrange(
self.relay un Paramator (torch. randn (R, D))
ing,
'b (hp p) (up q) -> (hp wp) (c p q)", p-patch_ size, q-patch_size
def fsrv,rd(self,, img_loc, img_glo): BS (i_.loc..ize(0) # batch size
P_ patchify(ing_ self.P)
def unpatchifyypattches, patch_size, H, W) (B, #patches, batch_size"2 out, channels) -> (B, out_chan, H, M)
r_glo patchify patchify(ing_glo (img_glo, self.P)
f_loc self.proj(p.1ec (Ps_ene D, f_glo self.proj(t_glo) (Ps_ D,
return rearrange(
patches,
'b (hp up) (oc p q)
(hp (xp q)
f.r repeat repeat(self.reeeg) BS)
hp-Ht // patch_aize, size, up-W patch_size, p-patch_ qupatch_size size
for block in (atettoorc.cat(ffr.... f f_r_glo = block(torch.ck (ff_r,,r, f_glo), ,)) f_glo, f_r = f.r_glol:, R:], f_r_glol:, :8)
class (mensformerBBoocxm...oo Module): (ondul))))
def __iiii___sett dim):
f f_r_loc = block(torch.cat (f_ ,_loc),,)) f_loc, f = f_r_loc[; R:], f_r_loc[; :R]
_)__._i)tt___
colf.atte. nn. ww..huttbeeddtteetiionnc.ii... num, heads-H, batch_ffrst-TTee self self.mlp nn Sequential nn Liinearrdin, 4 dim), nn ReLU(), nn. Linerr dim, dim)
out_patches self. out (ut__eea((f_lo) out unpatchify (out, (out__atche__,, self.P)
out
def forward x): aelf attnxa, [0] self mll())
repeat batch_size):
Replicate relay tokens across
#the batch dimension:
return reg [None] expand (batch__iae,s -1, -1)
17

---

Published in Transactions on Machine Learning Research (01/2026)
A-2 Additional Analysis
Attention Map In Fig. we present average attention maps of VIT trained on Cityscapes with 2, 4 and relay tokens. For all configurations, we observe strong coordination between scales: each relay token focuses on a well-defined part of the local window and the surrounding area of the global window. The radialisation observed with relay tokens mostly holds with most token attending to a single cardinal direction of the window. Interestingly, one of the relay tokens attends the centre part of both the local and global window. This may allow the centre of the local window to benefits from long distance contextual information contain in the global window.
Relay
Relay
Local
Global
Relay
Relay 2
Relay
Relay 4
Local
Global
Relay 1
Relay 2
Relay 3
Relay 4
Relay 5
Relay 6
Relay
Relay 8
Local
Global
Figure A-1: Attention. Attention maps of the relay tokens for ViT trained on Cityscapes with 2, 4, 8 relay tokens.
Model Size In Tab. A-1 provide the performance of approach with different SwinV2 backbone sizes the Cityscapes dataset. We observe that our approach with relay token show similar scaling to the standard SwinV2 model.
FLatten Transformer performance In Tab. A-2 we compare the memory usage and throughput of
the Swin configuration of FLatten Transformer Han et al. 2023 on Cityscapes (batch size = 64). We observe that adding relays yields higher performance gains with a lower memory footprint compared
18

---

Published in Transactions on Machine Learning Research (01/2026)
Model
Tiny
Small
Base
Large
SwinV2
64.9
68.2
68.1
67.0
SwinV2 + relay token
73.5
75.1
75.1
74.1
Table A-1: Impact of model size on performance on Cityscapes.
GPU memory
Throughput (full img/s)
Model
mloU
(GB)
Flatten Swin-S/ 256
68.0
26
10.6
Flatten Swin-S 512
73.5
9.5
Flatten Swin-S/ 256 + RELAY
75.3
48
4.9
Table A-2: Memory Usage and Throughput of Flatten Swin on Cityscapes.
simply increasing the sodding-window size, consistent with our observations Fig- 5 of the main paper. While linear-attention models can scale to larger windows with limited computational overhead and thus remain faster than relay-based variants, they incur substantially higher memory costs in our experiments.
A-3 Datasets
We provide in Fig. A-2 the colour codes for the classes of our evaluated datasets.
Cityscapes
road
sidewalk
building
wall
fence
pole
traffic light
traffic sign
vegetation
terrain
sky
person
rider
car
truck
bus train
motorcycle
bicycle
ignored
Archaeoscape
Mounds
Water
Temples
Background
ignored
URUR
others
building
farmland road
greenhouse
woodland
bareland
water
ignore
Gleason
Benign
Type
Type 4
Type
Figure A-2: Colourmaps. We provide the colour code for the classes of all the dataset evaluated.
19


Reviews:
[{"review_id": "juFPs6mhxw", "reviewer": "Reviewer_k6WA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper tackles the challenge of ultra-high-resolution semantic segmentation by introducing relay tokens - a small set of learnable query tokens shared between local and global transformer branches. The local transformer blocks process patches from the high-resolution image, while the global transformer blocks operate on patches from a downsampled version of the same image.\nThe method interleaves global and local transformer blocks such that the relay tokens are first updated by the global block and then passed to the local block, allowing efficient information exchange across scales.\n\nThe paper demonstrates competitive performance across multiple segmentation benchmarks, supported by both quantitative and qualitative results. The ablation studies are thorough, analyzing the effect of the number of relay tokens, the loss components, and the global window size. The visualization of average attention maps further confirms that different relay tokens attend to spatially distinct image regions, as expected.\n\nThe main limitation of the paper is computational cost: the proposed approach roughly doubles the FLOPs compared to single-scale baselines, since it requires running both local and global transformer branches (albeit with shared weights). The contribution would be stronger if the paper had investigated whether similar performance could be achieved with reduced compute - for example, by reducing the number of transformer layers or by using relay tokens less frequently.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims are well supported by experimental results and ablations. The proposed relay-token mechanism is evaluated on Cityscapes, Archaeoscape, URUR, and Gleason, and compared against a wide range of baselines including ViT, Swin Transformer, Vision Mamba, and several specialized multi-scale architectures. The paper also contrasts different scaling strategies - sliding windows, registers, downsampling, and decision fusion (the closest baseline, which fuses local and global predictions via averaging) - and reports both accuracy and peak GPU memory usage.\n\nRelay tokens increase the parameter count by only ~2%, while providing consistent performance gains. The main trade-off lies in higher computational cost (approximately 2× FLOPs). The qualitative attention maps are intuitive, showing that relay tokens specialize in distinct spatial regions, and the ablations convincingly demonstrate the importance of the loss terms, the number of relay tokens, and insight into whether local/global projectors should be shared.", "audience_yn": "Yes", "audience_explanation": "This paper would appeal to TMLR readers interested in vision transformer architectures, multi-scale modeling, and efficient segmentation for large images. The proposed method is practical, general, and clearly positioned within ongoing research on scalable, context-aware transformer designs. Its insights are relevant to both researchers developing new architectures and practitioners working with ultra-high-resolution visual data.", "requested_changes": "The paper is well written and provides a clear discussion of the proposed method, its performance, computational cost (in terms of memory and FLOPs), and the intuition behind how relay tokens facilitate cross-scale information exchange. However, one clear drawback is that the approach roughly doubles the FLOPs compared to single-scale baselines and is inherently sequential, since it requires interleaving global and local transformer blocks.\n\nCritical change for acceptance:\nThe authors should evaluate whether relay tokens can enable similar segmentation performance with reduced computation, for instance by using fewer transformer blocks or by performing relay exchanges only in a subset of layers to allow parallelization of local and global blocks. Such an ablation would clarify whether the observed improvements arise from the relay-token mechanism itself or simply from increased computation. Currently, most baselines (except the decision fusion method) use fewer FLOPs than the proposed approach, which makes it harder to contextualize the contribution despite comparable parameter counts.\n\nRecommended change for clarity:\n- Add a figure illustrating the spatial relationship between the cropped global predictions and the local features used in the losses in Equations (9) and (10). This would make the training setup and the alignment between global and local supervision much easier for readers to follow.\n\nMinor edits (non-critical):\n- Figure 3: Increase vertical spacing between the “Relay tokens” and “Sliding win.” rows to avoid overlapping text.\n- Figure 7 and Table 5: Clearly specify the metric being reported as performance (e.g., mIoU) in captions or labels.", "broader_impact_concerns": "This work focuses on a technical contribution to vision transformer efficiency and scalability, without direct societal or ethical implications. The proposed relay-token mechanism is a general modeling improvement and does not involve sensitive data, human subjects, or method-specific considerations.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UTb7IlNvql", "reviewer": "Reviewer_Drrm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes relay token as method to improve the semantic segmentation of ultra-high-resolution images. Relay token uses a global branch that works on a down-sampled image patch to provide global information and a fine-grained local branch that works on local image patch. The relay tokens are shared between the global and local branches to transfer information between them. Relay token is general across vision Transformer models and requires to add only a small number of parameters. Experiment results show that relay token improves the accuracy of the semantic segmentation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I really like this paper due to its succinct presentation, simple idea, good generality, and extensive experiments. The strong points of the paper include\n\nS1: Existing semantic segmentation methods for ultra-high-resolution images are discussed comprehensively along with their limitations (i.e., Figure 1).\n\nS2: The relay token method is simple and general across different vision Transformer model.\n\nS3: The experiment results are strong, showing that relay token effectively improves the accuracy of semantic segmentation. Other details, such as the resolution of the two branches, the number of relay tokens, and the designs, are also evaluated.\n\nS4: Relay token is compared with other methods that adopt the global-local idea, and reasons are provided why the relay token is a better choice.", "audience_yn": "Yes", "audience_explanation": "This paper proposes a simple, lightweight yet general and strong method for the semantic segmentation of ultra-high-resolution images. Many audiences of TMLR will be interested to learn it.", "requested_changes": "R1: Please add some efficiency experiments regarding (i) the training time of relay token and (ii) the inference speed with relay tokens.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WR0RurDW9K", "reviewer": "Reviewer_EHcX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a simple and effective method for segmenting ultra-high-resolution images for vision transformers. \nIt simultaneously processes the image by the local and global branches, with the introduced relay tokens to preserve local details and global awareness. \nExperiments demonstrate that the proposed method improves the performance of multiple vision transformers with only a small number of additional parameters.\n\nstrengths:\n1. The writing is clear and easy to follow.\n2. The proposed method is simple and effective, and it is easy to implement for various vision transformers.\n\nweaknesses:\n1. Although the additional parameters are small, the additional computational cost is huge. \nAccording to Table 4, the GFLOPs will be doubled after applying the method to vision transformers. \nThis may limit the practical applications of the proposed method. \n2. The experiments only apply the method to some basic vision transformers, such as ViT and Swin. \nIt would be more convincing if the authors could apply the method to some advanced vision transformers, such as GPWFormer [1] and BPT [2].\nIf not, the authors should provide some discussion on why the method is not suitable for advanced vision transformers.\n3. The introduction is too short. To make the paper more self-contained and friendly to readers, the authors should expand the introduction section by providing more background information and related works.\n4. The paper mentioned \"The quadratic token–token interaction in vanilla transformer layers renders off-the-shelf ViT-like models infeasible for UHR semantic segmentation\". \nHowever, there are some efficient attention mechanisms that can reduce the computational cost of attention, such as Linformer [3], Efficient Attention [4], etc.\nThe authors should discuss the advantages of the proposed method compared to these efficient attention mechanisms, and provide some experimental results if possible.\n\n[1] Ji, Deyi, Feng Zhao, and Hongtao Lu. \"Guided patch-grouping wavelet transformer with spatial congruence for ultra-high resolution segmentation.\" Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence. 2023.\n\n[2] Sun, Haopeng, et al. \"Ultra-high resolution segmentation via boundary-enhanced patch-merging transformer.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 7. 2025.\n\n[3] Wang, Sinong, et al. \"Linformer: Self-attention with linear complexity.\" arXiv preprint arXiv:2006.04768 (2020).\n\n[4] Shen, Zhuoran, et al. \"Efficient attention: Attention with linear complexities.\" Proceedings of the IEEE/CVF winter conference on applications of computer vision. 2021.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The author evaluates the proposed method and baselines on multiple datasets, with ablation studies to validate the effectiveness of each component.", "audience_yn": "Yes", "audience_explanation": "Ultra-high resolution semantic segmentation is a practical problem, and researchers working on this topic will be interested in this paper.", "requested_changes": "Please see the weaknesses.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)