SHF: Symmetrical Hierarchical Forest with Pretrained Vision Transformer Encoder for High-Resolution Medical Segmentation
Authors: Enzhi Zhang
Peng Chen
Rui Zhong
Du Wu
Jun Igarashi
Isaac Lyngaas
Xiao Wang
Masaharu Munetomo
Mohamed Wahib
Venue: NeurIPS 2025 spotlight ()
Keywords: Hierarchical Forest; High-resolution; Long-sequence; Transformer; 3D Imaging; Segmentation
Primary area: deep_learning
TL;DR: A lightweight approach that adaptively patches the input image to increase token information density and encode hierarchical spatial structures into the input embedding.

Abstract: This paper presents a novel approach to addressing the long-sequence problem in high-resolution medical images for Vision Transformers (ViTs). Using smaller patches as tokens can enhance ViT performance, but quadratically increases computation and memory requirements. Therefore, the common practice for applying ViTs to high-resolution images is either to: (a) employ complex sub-quadratic attention schemes or (b) use large to medium-sized patches and rely on additional mechanisms within the model to capture the spatial hierarchy of details. We propose Symmetrical Hierarchical Forest (SHF), a lightweight approach that adaptively patches the input image to increase token information density and encode hierarchical spatial structures into the input embedding. We then apply a reverse depatching scheme to the output embeddings of the transformer encoder, eliminating the need for convolution-based decoders. Unlike previous methods that modify attention mechanisms \wahib{or use a complex hierarchy of interacting models}, SHF can be retrofitted to any ViT model to allow it to learn the hierarchical structure of details in high-resolution images without requiring architectural changes. Experimental results demonstrate significant gains in computational efficiency and performance: on the PAIP WSI dataset, we achieved a 3$\sim$32$\times$ speedup or a 2.95\% to 7.03\% increase in accuracy (measured by Dice score) at a $64K^2$ resolution with the same computational budget, compared to state-of-the-art production models. On the 3D medical datasets BTCV and KiTS, training was 6$\times$ faster, with accuracy gains of 6.93\% and 5.9\%, respectively, compared to models without SHF.

SHF: Symmetrical Hierarchical Forest with Pretrained Vision Transformer Encoder for High-Resolutie Medical Segmentation
Enzhi Zhang 1.2
Peng Chen Rui Zhong' Du Wu?
Isaac Lyngaas
Jun Igarashi
Xiao Wang
Masaharu Munetomo'
Mohamed Wahib²
Hokkaido University, Sapporo, Japan
RIKEN Center for Computational Science, Hyogo, Japan OOkk Ridge National Laboratory, Tennessee, USA {zhangenzhi, zhongrui, munetomo]@iic wtt.ai.......aa....oomoooo (peng. chen, du, wu, jigarashi, mohamed, attia)Oriken jp (lyngaasir, wangx2)@ornl.gee gov
hokudai jp.
Abstract
This paper presents novel approach to addressing the long-sequence problem in high-resolution medical images for Vision Transformers (ViTs). Using smaller patches as tokens can enhance ViT performance, but quadratically increases com- putation and memory requirements. Therefore, the common practice for applying VITs to high-resolution images is either to: (a) employ complex sub-quadratic attention schemes or (b) use large to medium-sized patches and rely on additional mechanisms within the model to capture the spatial hierarchy of details. We propose Symmetrical Hicrarchical Forest (SHF), a lightweight approach that adap- tively patches the input image increase token information density and encode hierarchical spatial structures into the input embedding. We then apply a reverse depatching scheme the output embeddings of the transformer encoder, eliminat- ing the need for convolution- based decoders. Unlike previous methods that modify attention mechanisms retrofitted to any ViT model to allow to learn the hierarchical structure details in high-resolution images without requiring architectural changes. Experimental results demonstrate significant gains in computational efficiency and performance: the PAIP WSI dataset, achieved 3~~2xx speedup 2.95%~7.033% increase in accuracy (measured by Dice score) at 64K" resolution with the same computational budget, compared to state-of-the-as production models. On the 3D medical datasets BTCV and KiTS, training was 6x faster, with accuracy gains of 6.93% and 5.9%, respectively, compared models without SHF.
complex hierarchy of interacting models, SHF be
Introduction
Recently, Transformers have been rapidly adopted in the field of computer vision [ 2 Building on the self-attention mechanism, Vision Transformers (ViTs) and their variants have achieved significant advancements in various image classification and downstream visual tasks [ Text tokens are atomic, semantically distinct, and rich in information, whereas visual tokens are geometrically related and sparse in semantics. In other words, feeding a sequence of image patches to transformer encoder deprives the self-attention mechanism of direct information about spatial hierarchy.
"Coresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The loss of spatial hierarchy information becomes more pronounced when working with high- resolution or multi-dimensional medical images, as the spatial hierarchy becomes more detailed and intricate 8 This requires the use of very small patches so that the self-attention mechanism can capture local features [9. However, using smaller patches quadratically increases the computational and memory costs of self-attention, prompting several approaches to address this issue by modifying the model architecture to help it learn the hierarchical structure in images. Most of these approaches Model-hierarchicr solutions involve training ViTs hierarchically, with multiple transformers operating at different resolution levels [10, 8 11 [2]. While this approach improve model performance, it also increases training time and memory usage. Additionally, managing multiple interacting transformers adds complexity, requiring extensive hyperparameter tuning at each resolution level. Attention- hierarchical solutions alter the patching scheme at the self-attention stage to represent hierarchical features, seen in Swin [33 14), MVïT [15], , and MV (16). Although these methods are more parameter-efficient than standard ViTs 4 they introduce additional spatial operations, increasing model complexity and reducing multi-modal capabilities.
fall into two broad categories: model-hierarchhiall and attentionn-hierarchical
From the above summary, two questions arise: (1) How can we design a patching strategy that uses the fewest possible patches to represent the original image, thus increasing the information density of each patch to maintain model performance, without altering the structure of the ViT model? (2) If an effective patching strategy could be developed to relay spatial hierarchical information to the model, could we leverage a post-encoder adaptive dispatching strategy during inference to eliminate the need for post-encoder convolution decoding?
To address the two questions above, this paper proposes Symmetrical Hierarchical Forest (SHF), which adapts the patching strategy to the hierarchical details of each training example. This approach enables the ViT to capture hierarchical local spatial information typically derived either from post- encoder convolutions (e.g. the convolution decoder used in the SAM 2 model (17) or hierarchical architecture features (i.e. Swin 14 and HIPT (8) We downscale patches regions with fewer details, aligning them to the patch size used for regions with more details. However, without expert knowledge of hyperparameters, we rely solely on hierarchical forests to extract/represent the spatial hierarchy information. Using the SHF scheme, we demonstrate that the model receives sufficient information about the spatial hicrarchy, allowing us to eliminate additional model components (such as U-Net 188 convolution decoding blocks) that would otherwise be required. Notably, the use of additional components (e.g. the convolution decoder in SAM 2 (77) can lead to high memory demands to store activations for high-resolution masks (e.g. over 20GB of memory for storing mask activations when using SAM 2 on 64K2 images). By employing a transformer encoder-only design with hierarchical forest and post-depatching, we can use smaller patch sizes, enabling self-attention to capture the spatial hierarchy more effectively than larger patches and additional mechanisms (such as U-Net or convolution decoding blocks). As an added benefit, our method simplifies model design and allows for swapping in different encoders, as it is a lata-based approach that operates on the input and output of the transformer encoder without modifying the encoder itself.
The contributions of this work are as follows:
Symmetrical Hierarchical Forest. By applying the Symmetrical Hierarchical Forest (SHF), we extract the hierarchical information directly and eliminate the expert knowledge required for tuning hyperparameters. Additionally, we completely discard the convolution decoder, signifi- cantly reducing the computational and memory overhead (~75% GPUs) of mask processing in high-resolution segmentation tasks. Finally, by downscaling redundant regions in the input image space, we achieve a quadratic reduction in the computational cost of the ViT encoder.
Long Context Segmentation. To demonstrate the efficiency of SHF, we conducted experiments on high-resolution medical imaging datasets, retrofitting state-of-the-art (SoTA) models such as SAM 1 [19] and SAM 2 [17] with our SHF scheme in place of their convolution decoders. When comparing models retrofitted with SHF to SoTA models without SHF, high-resoluticn pathology datasets (e.g. the PAIP dataset, ranging from 5122 to 64 pixels) and 3D MRI datasets (BTCV, KiTS) benefit from the efficiency of SHF, allowing patch sizes as small as 2x2 pixels and 2x2x2 voxels. At the same performance level, we achieve 3x to 32x speedup on the PAIP dataset, or, with the same computational budget, a 7.03% increase in Dice score at 64 resolution. On 3D medical imaging datasets, such as BTCV and KITS, we see ~6x training speed improvement along with performance increases of 6.93% and 5.9%, respectively, compared to SoTA models without SHF.

---

16384"63.128 13.12%%
Dice Score: 100%
Dice Score:80.99
Dice Scorc:67.639
32768 (7768"Q..122 @3.125%
Dice Score: 100%
Dice Score:81 1 1.43%
Dice Score:62.34%
655362 @3.125%
Dice Score: 100%
Dice Score:82.96
Dice Score:6l.6
Input High-Res Image
Ground truth
SHF-SAM
SAM
Figure 1: We compared the segmentation difference between SAM 10 and SAM retrofitted
with our SHF scheme (SHF-SAM) instead of the original convolution decoder for PAIP [20 at 16, 3842, 32, 768?, and 65, 5362 resolutions. At the same GPU budgets. SHF-SAM can go down to batch size of 8x8 (vs. 1, 024x1, 024 at best for SAM before going OOM). As result, SHF-SAM can extract and express mask details better than SAM, with the gap accuracy favoring SHF-SAM as the resolution gets higher.
Simplicity and Low-Overhead. Unlike existing methods that modify the self-attention or transformer encoder mechanisms, our solution preserves the original self-attention mechanism. This ensures seamless retrofitting of SHF into any vision transformer. SHF is low-overhead pre-processing and post-pres post-processing solution that is further amortized over epochs, making the overhead effectively negligible.
The rest of this paper is organized as follows: Section 2 reviews related work. Section B presents the methodology. Section describes the experimental setup. Section reports the evaluation results. Finally, Section 6 concludes the paper and outlines future work.
2 Related Work
The computational cost of self-attention increases as the patch size decreases. To mitigate this, several strategies have been developed. Sequence parallel methods, including Deep-Speed Ulysses [2I], LightSeq [22], RingAttention 23 LLS (44, FlashAttention (25), and 266 Linear approximation
methods, such as spectral attention 27 20 low-rank approximation 30 31 sparse attention matrix
sampling [32) infrequent self-attention updates [37 38, combinations of these (39]. These
methods reduce the computational load of the attention mechanism; however, excessive reduction

---

Dataset
1:Hlera-Edges
2-3: Hierarchical Forest Patching
4: Training Stage
VIT Image Encoder
(x, - (1))
(Mg 4)
x: Image
Lerain (1.233)- (12331))
5: Inference Stage (w/ De-patching) Gan nn nnm
(x dos (x
Xezi b)
Image Encoder
y: Mask
Yt yi, Leees (y',y)
Tes: (Kg 4)
f3
(x,y) S (Xa.
Train Stage Patching
Patching
Z-order
Dewn-sampling
Inference De-patching
Figure 2: Overview of SHF. SHF begins with the original image and ends with feeding the extracted patches (tokens) into intact transformer-bas based model. In real training example the SAM [19 model using 512x512 images from the PAIP (20 liver cancer dataset, SHP reduces the number of patches from 4,096 to 512 (each of size 4x while maintaining the same Dice score. This results in an ~8 reduction in sequence length and a 77.53 speedup in end-to-end training.
lead to performance loss, reported in the literature 40
Hierarchical training of ViTs,
where multiple transformers are trained at different resolution levels [10 g 11 12]. However, using multiple transformers increases training time and memory usage, and managing multiple
interacting transformers is complex. Recently, quadtrees have been used in image segmentation to reduce attention cost, e.g. quadtree/octree attention patch pre-processing 441) Both of those approaches employ quadtrees, but involve additional model complexity or need expert knowledge in the processing stage for hyperparameters.
3 Methodology
3.1 Vision Transformers and Attention
The self-attention mechanism in transformers computes attention scores A between input tokens, forming the attention matrix. Let x E RN denote a sequence of feature vectors of dimensions F. A transformer is function T RN RNF defined by the composition of L. transformer layers Ti(.).... (-(-) as follows:
(1) A:(-) is the self-attention function. The function f((.) transforms each feature independently of the others and is usually implemented with small two-layer feedforward network. Formally, the input sequence æ is projected by three matrices Wq E RTXD, WK RFDD, and Wv R to corresponding representations Q, K and V. Thus, the attention scores calculated follows:
Tr((z) = f((((z)) +x)
Q zWQ =WK, K, V aWv, A., Softmax ((Q.K)//Va,
(2)
where Qi and Ky are query and key vectors for tokens and j. and d, is the dimension of the key vectors. The complexity the attention matrix is O(N2), where N is the sequence length, We further assume that the input is the content of square image with resolution of Z, that is, let

---

Table 1: Speedup of SHF end-to-end training for PAIP dataset the same segmentation quality as the baseline. We use the highest dice score of the baseline models, SAM and SHF-SAM.
Resolution 512 512
Model-Patch
Seconds/Image
GPUs Sequence Length
Dice Score (%)
Speedup (x)
SHF-SAM-16
0.03616
256
71.03
7.75x
SAM44
0.28041
1
16,384
68.98
SHF-SAM-16
0.09913
1
1,024
73.65
1,4 1,
3.86x
SAM-8
0.38261
8
16,384
66.56
SHF-SAM-16
0.1031
8
1,024
74.17
4, 090 x 4, 096
15.7x
SAM-32
1.6183
64
16,384
71.05
SHF-SAM-16
0.3512
16
2,048
76.21
8, 192 8, 192
7.17x
SAM-64
2.5168
128
16,384
67.31
SHF-SAM-16
0,3672
32
2,048
76.89
16, 384
16, 384
15.4x
SAM-128
5.6714
256
16,384
67.63
SHF-SAM-32
0.3826
64
2,048
76.08
32, 768 32, 768
23.8x
SAM-256
9.1213
512
16,384
62.34
SHF-SAM-6
0,4013
256
2,048
75.33
65, 65, 536
32.3x
SAM-1024
12.9833
1,024
16,384
61.68
I E R_x2 and by assuming that patches arise from the uniform grid patch method of patch size p. Thus the sequence N (). The total computation and memory cost of attention scores defined in Eqn 2 according to resolution and patch size is O((z/ P).). This complexity shows the difficulties of increasing the resolution while decreasing the patch size P with the uniform grid patch strategy.
3.2 Symmetrical Hierarchical Forest (SHF)
In the following paragraphs, we describe how SHF works, following the steps outlined in Fig.
Enera-Edges Detection. We aim to use different methods to extract hierarchical details, as this would allow to augment the dataset. This, however, is different from traditional augmentation applied the image level; we augment by providing the model with different views of the spatial hierarchy for each image, thereby giving the model better opportunity to learn the hierarchy. To use different ways to extract the hierarchical details in the image x E X, as Fig. 11, we use a Gaussian blur with kernel A and Canny [44 edge detection with threshold to the original input images x E X. The Gaussian blur with kernel & smooths the irrelevant details, and the Canny edge detection with threshold extracts the grayscale edges xe of the image. To generate different spatial structures that can also recover the original inputs, during our experiments, we randomly choose the threshold to be in the range [100,200], and the kernel size randomly set to one of [3.5.7,9,11,13]]
Hierarchical Forest. In the Hierarchical Forest stage (Fig. 2),, we build several quadtrees (octrees in 3D) from each ze; Te undergoes recursive tree partitioning. To construct the tree T, we create tree nodes T, representing specific regions where n is the number of leaf nodes. The leaf nodes T Tnts-1 is defined recursively as follows:
Ta Ta [T. (TI.
in 2 N T.J. arg max V (TT, [t)
Tuts-1
where V is the criterion we use to differentiate the different levels of detail. We choose the maximum sum of pixel values among the tree nodes by arg max ((T*((i) [i]), {T, .... Th are the new
child nodes after the subdivision of [") s 2n is the number of subdivisions and is the number of dimensions. T can be used for both 2D and 3D tasks, for example = 4. 8 which means quadtree avoid unnecessary padding or dropping
and octree, respectively 45 46). In our implementation, due to varying sample lengths, we control the number of splits at the leaf nodes to keep each sample the same sequence length N. This approach fully utilizes the GPU resources without discarding sample information. The sequence length N is set to [1024, 4096, 8192, 16384, 16384, 16384] with respect resolutions, which practically allows the input x, to be subdivided all the way down the 2 x 2 patch size level.
After building the tree, as a property of quad/octrees, visiting all the tokens (appearing as leaves in the tree) from left to right gives the token sequence as a Z-order space- space-filling curve in the image
space
47
This operation(
(>) results in sequence of image patches shown in step Fig. 3. We
5

---

not only use spatial trees to encode images but we also encode masks y. Using the same tree and z-order of the mask will also result in sequence of mask patches yı- Since the spatial trees are built from the image, the sequence length of the image and mask patches (2i, Ue) should be the same N.
Image Encoder. After completing hierarchical forest patching, we obtain an image patch sequence and a mask patch sequence of the same length, allowing us to (continuously) train the ViT model. We use the well-known pre-trained image encoder from SAM in steps Fig. 4 and then continue training the model on our dataset after retrofitting our SHF scheme into the SAM model. SAM uses an MAE (2] pre-trained ViT [4) minimally adapted to process high-resolution inputs. For 3D MRI data tasks, we use the SAM [17] encoder and similarly retrofit SHF into SAM 2. When retrofitting SHF into both SAM and SAM 2, we remove the convolution decoder that is part of the original SAM and SAM 2 models. It should be noted that during the training phase of the model, we do not directly generate masks y but instead generate encoded masks V/-- The advantage of this approach is that it saves memory, which would otherwise be needed for high-resolution masks, and reduces the backpropagation compute costs associated with the heavy decoders. We summarized the objective function as follows:
M Cuiera (fo (Trrece k. D))) Tsym ki Di))
min E (RAD)eD #
(4)
Where M is total number of samples, 9 represents the trainable weights, Liiera denotes the Dice loss, (ks, 1) specify the Gaussian filter size and Canny threshold, and Teree = Taym are the recursive tree operators applied to both the input and the mask y.
Symmetrical Depatching. The function of depatching (3)) is to upscale the sequence obtained from
the ViT into a mask during the inference stage. By depatching, we can eliminate the need for general U-Net or lightweight decoders and calculate the backpropagation gradient directly at the output of the encoder (in Fig. 5). This is because we have observed that the information in the mask itself is sparser compared to the input image (see the Fig. Therefore, when reconstructing the mask the evaluation stage, we simply use the same quad/octree structure derived from the image to linearly upscale the patches at the corresponding positions in the sequence.
4 Experimental Setup
4.1 High- Righh-eesoutton Medical Image Datasets: PAIP, BTCV, & KITS
PAIP: 20 is a high-resolution, real-world liver cancer pathology dataset, with sample resolutions up to 64 sigmificantly surpassing those of conventional image datasets. PAIP contains 2, 457 Whole- Slide Images (WSIs). When lower resolutions are needed, we downscale the images to uniform sizes of [512, 1024, 4096, 8192, 16384, 32768, 65536] square pixels. For training, we randomly select 70% of samples, 10% for validation, and 20% for testing. All datasets are shuffled and normalized to [0.0, 1.0) as input for the model. The BTCV: challenge 48 for 3D multi-organ segmentation includes 30 subjects with abdominal Computed Tomography (CT) 449 scans, with 13 organs annotated by experts. Each CT scan consists of 80 to 225 slices, each with 5122 pixels. The multi-organ segmentation task is defined as a 13-class segmentation problem, where the average dice score across all classes is typically reported. Although BTCV has lower resolution compared to the PAIP dataset (512² vs. 64K²), remains a widely used benchmark the high-resolution medical segmentation community. KiTS: the 2019 Kidney and Kidney Tumor Segmentation Challenge (KiTS19 (50)) aimed to develop algorithms for segmenting kidneys and kidney tumors in contrast-enhancee 3D CT scans. The dataset consists of 300 anonymized scans in NIfTI (.ni.gz)) format, each manually annotated with kidney and tumor labels. Each scan has a typical resolution of 512x512 pixels per slice, with the number of slices varying based on patient anatomy. KiTS is common benchmark for 3D MRI segmentation [51]
4.2 Evaluating Models: Baselines Proposed
Baseline Model: We use the well-known segmentation model SAM [1 which employs Vision Transformer (ViT) encoder as its backbone for segmentation tasks. SAM offers ViT variants (ViT- Base (b), ViT-Large (l), and Vig-Huge (h)), each with 12, 24, 32 transformer layers, respectively.

---

(a) Edge Image
(b) 7=4.0 7=4.07
(c) 1=1024
(d) 1=2000,
(c) Edge MRI
(f) L=260, 766.06
(g) L= 2=22.47 (h) 1=2052,
Figure 3: Spatial subdivision of an edge image with a resolution of 1024x 1024. L represents the sequence length, and 7 denotes the compression ratio relative the grid patches. SHF significantly compresses the image, with higher dimensions resulting in higher compression ratio.
These weight configurations -b, I, and h- are pre-trained using Masked Autoencoders (MAE) [2]. Unlike Unet-style models [52 [8 53 [ the SAM mask decoder is lightweight, containing only two convolutional layers. However, reconstructing high-resolution masks from the latent space requires additional upscaling layers, leading to increased memory usage.
As defined in Eqn 4 our approach utilizes the tree structure from the patching stage in SHF, reducing the need for decoder training. We also experimented with the updated SAM2 T7 encoder ViT model, adapting it for 3D MRI data tasks, such as BTCV and KiTS19. Unlike SAM2 original video frame processing, patch embedding stage. Unet-shaped models, such as UNETR [18), TransUnet pattern with transformer convolutional layers encoders to extract image details, connected to decoders via skip connections. These decoders upscale the representation vectors from the dense latent space to match the mask's size. For comparison with SHF, we test Unet 52 with pre-trained weights from the timm package, along with the SAM-pretrained
Proposed Model:
3D convolutional layers for voxel processing in the
53 and SWIN Unet
1 follow contraction-expaane
UNETR 18 and TrensUnet (53) models.
Performance Metrics: Computational performance is reported in seconds per image for end-to-end training. Segmentation accuracy is evaluated using the Dice score, which quantifies the overlap between predicted and ground truth segmentation masks. It defined as:
Dice(X.Y) xXnl//(X|++ [Y]
(S)
where and the being compared, and [XnY| is the size of their intersection.
Evaluation
5.1
Speedup: SHF vs w/o SHF on the same models, datasets, and comparable performance
Training with SHF is faster due to its ability to significantly compress the sequence length shown in Figure. E and eliminate the decoder component. As shown in Table SHF, which combines pre- processing and posppsrocssiigg steps on top of the pre-trained SAM baseline, achieves a geometric mean speedup ranging from 3.86 to 32.3x while maintaining comparable dice scores. This speedup is measured when both SHF and the baseline are trained for the same number of epochs. At the highest resolution of 64 R2 with training on 256/1024 GPUs, SHF delivers approximately a 32x speedup compared to naive SAM encode and reduction of 75% usage of GPUs. Such speedup
7

---

Table 2: Improvement in quality of segmentation (IQS) for the PAIP dataset against different baselines.
Resolution
Model
Patch
Seconds/Imag//GPI
GPUs
Sequence Length
Dice Score (%)
SAM
8+
0.3826
8
16,384
66.56
UNETR
8
1.0863
32
16,384
75.72
12424 1, 024 (IQS: +2.95%)
TransUNet
1.3247
8
72.38
UNet
0.0981
68.92
0.0991
L.044
SAM
324
1.6183
16,384
71.05
322
1,8613
128
16,384
4. 096 4, 096
TNatsUNet
2.1637
128
71.32
(IQS: +3.22%)
UNct
0.3712
16
64.11
SHF+SAM
2
0.3766
4,096
78.99
SAM
64
2.5168
128
16,384
67.31
UNETR
64
2.6618
256
16,384
75.27
8, 192 8, 192
UransUNet
2.3678
256
70.89
(IQS: +4.41% %))
UNet
1.2858
32
63.21
SHF+SAM
2'
1.5327
16
8,192
79.68
SAM
1284
5.6714
256
16,384
67.63
UNETR
128"
5.1179
512
16,384
75.89
384 16,
TNansUNet
6.1296
512
70.46
(IQS: +5.09%)
UNet
2.7825
256
62.97
SHF+SAM
3,2741
16,384
80.98
SAM
256
9.1213
512
16,384
62.34
UNETR
256
8.1896
1,024
16,384
74,96
32, 768 32, 768 (IQS: +6.47%)
Tran
10.001
1,024
69.88
UNer
4,2714
512
61.38
SHF+SAM
4"
3,4631
64
16,384
81,43
SAM
1,024
12.983
1,024
16,384
61.68
UNETR
1,024
13.218
2.048
16,384
75.31
65, 536 65, 536
UNensUNet
14.352
2,048
67.67
(IQS: +7.03%)
UNct
5.9611
1,024
59.69
SHF+SAM
3.6112
256
16,384
82.96
(a) Mask
(b) Recon-16,338
(c) Mask
(d) Recon-1,024
Figure 4: Reconstructed mask through the SHF build from the input 64K image with sequence length 16, 384 and 1, 024 with recovery accuracy 99.5% and 73. These high recovery accuracies suggest the information contained in the masks are sparse.
benefited from eliminating the heavy decoder, allowing more parallelization and reducing inference and back-propagation.
5.2 Segmentation Performance: SHF w/o SHF on the different models and datasets
Qualitative Results: Table 2 demonstrates the segmentation improvements across different models and PAIP resolutions. At comparable resolutions, SHF achieves nearly 8x reduction patch size while maintaining the same computational complexity. This results in an average 5.5% improvement in dice score over the original model. Additionally, these improvements come with training time speedups of up to 4.6x. At high resolution (64K²), SHF lacks decoder during training, offering substantial computational and GPU memory savings compared to SAM, which relies on an upsam- pling restoration mask. Consequently, SAM requires shorter sequence length at high resolutions, as performs more upsampling to generate masks. For instance, as shown Table moving from 2 to 64/2 resolution results in slight 2% performance drop for SAM. Table presents 3D MRI

---

Table 3: Segmentation of BTCV [48] for multi-organ segmentation and KiTS19 [50] for Kidney
Tumor Segmentation on single GPU. Time indicates the end-to-end runtime to achieve the corre- sponding Dice Score.
Datset
Model
Patch Size
Time
Speedup (x)
Dice Score (%)
U-Nct [52
N/A
843.99 90 Seconds
9.04x
80.2
U-Mamba [52]
N/A
8,016.24 Scconds
0.95 x
83.51
Trans Net [SS
N/A
3113.25 Seconds
2.45x
83.8
BTCV
48
UNETR [18
4
8386.56 Seconds
0.91
89.1
Swin UNETR [56]
4
5861.93 Seconds
1.30x
89.5
SAM2
4
7637.28 Seconds
1.0x
82.77
SHF-SAM2
2
1067.88 Seconds
7.15x
89.71
U-Net 52
N/A
243.7 Minutes
1.99x
83.23
U-Mamba 52]
N/A
969.0 Minutes
0.5x
86.22
CoTr [55]
N/A
488.7 Minutes
0.99
84.59
UNETR [8
8
513.6 Minutes
0.94x
86.45
KiTS 50
inFormer [51
8
876.5 Minutes
0.55x
75.85
Swin UNETR 51
83
748.3 Minutes
0.65x
81.27
Swin UNETR-V2 [51
83
766.4
Minutes
0.63x
84,14
S7
83
483.1
Minutes
1.0x
81.35
SHF-SAM2
2
187.3 Minutes
2.58x
87.25
segmentation results for BTCV and KiTS datasets at 512" resolution. Instead of processing each 2D slice independently and reconstructing the final 3D prediction as in previous works [57] [53, we replace the 2D convolution patch embedding with 3D convolution for voxel processing. As shown in the tables, SHF outperforms the prestrained SAM2 Hiera image encoder, yielding 5.9% higher dice score and requiring 4x less computational resources.
Visual Results: We compare the segmentation quality across different high resolutions (16K², 32K2 64K²)) for the baseline models SAM, and our proposed SHE. The segmentation results summarized in Fig. The first column shows the original input, with the label indicating the resolution. The red square highlights a small portion of the image, approximately 3. 3.125% of the entire slide. The second column presents the ground truth, followed by the predictions from different models. At higher resolutions, all models can capture the general segmentation areas. However, due the limitations of heavy convolution- based decoders and uniform grid patching, only large patch sizes are feasible, such as the 16K patch size used by SAM and UNETR at 64K2 input resolution. In contrast, the same 64K² resolution, SHF can use smaller patch sizes, as small as 8", significantly improving the quality of the detailed masks.
5.3 SHF vs HIPT hierarchical model: training from scratch the same datasets
To demonstrate the versatility of SHF, com- pare its classification performance on the PAIP dataset with HIPT [8) SoTA highly advanced hierarchical multi-resolution model specifically designed for microscopie pathology classifica- tion. For this experiment, we restructured the PAIP dataset, originally intended for segmen- tation, into six organ-based categories. Fach category consists of 40 samples, with 28 for training, 8 for testing, and 4 for validation. For HIPT, we resized all samples three resolution scales ([256, 1024, 16384)) and set the patch sizes for each scale to [16, 256, 4096), adhering to the original settings. For SHF, we used only the 16 resolution images for classification. Instead of relying on decoder for segmentation, we added an output channel dedicated to class prediction. In Table 4 SHF achieves significant accuracy improvement (>8%) over HIPT, even when using a vanilla ViT model and the same computational budget. At high resolution (16K2), HIPT is limited to a patch size of 40962 before running OOM, whereas SHF can use patch sizes as small as 22 in the highest-resolution regions. This substantial accuracy boost, despite SHF using basic ViT, indicates
Table Classification (Top-1 accuracy) of vanilla ViT, HIPT [8) and SiT-VIT on PAIP dataset (16, 3842 res.)
Model
GPUs
Patch Size
Accuracy
VIT H HIPT
128
4, 096
68.97
128
[16, 256 4, 096
72.69
SHF-VIT VIT--96
4, 090
69.11
SHF-VIT-2
128
22
$0.14

---

that: a) SHF avoids random patch dropping or padding. and b) smaller patch sizes play a more crucial role in improving performance than model complexity.
Conclusion and Future Work
This paper presents SHF, a lightweight and efficient method for enhancing ViT performance on high-resolution images. By increasing token information density and encoding hierarchical spatial structures through a hicrarchical patching strategy and reverse depatching, SHF enables standard ViTs to handle long sequences effectively without requiring architectural changes. Experimental results on both 2D and 3D medical imaging tasks demonstrate significant improvements computational efficiency and segmentation accuracy, establishing SHF as practical solution for high-resolution medical image analysis. In future work, we aim to extend SHF to broader scientific domains. including materials discovery 59), brain neuron structure detection [60, 61], and non-medical
CT applications [62 64). Additionally, we plan to enhance SHF's performance on multimodal
problems 665 to support more complex and diverse applications.
7
Acknowledgment
This manuscript has been co-authored by UT-Battelle, LLC, under contract DE-ACO5-000R2272. with the US Department of Energy (DOE). The U.S. Government retains a nonexclusive, worldwide license to publish or reproduce the published form of this manuscript, or to authorize others to do so, for U.S. Government purposes, as acknowledged by the publisher. This research was partially supported by the ORNL AI Initiative sponsored by the ORNL Director's 's Research and Development Program. This work was supported by JSPS KAKENHI under Grant Number JP21K17750.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2333.00777 087744 2023.
[2] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, 2017.
[4] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 17X. 2021.
[5] Jie Hu, Li Shen, and Gang Sun. Squeezz-andd-exciatioon networks. Proceedings of the IEEE conference on computer vision and patlern recognition, 2018.
[6] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. Proceedings of the IEEE conference computer vision and pattern recognition, 2018
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Ross Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European Conference on Computer Vision, 2020.
[8] Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In 2022 IEEF/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 16123-16134, New York, NY, USA, 2022. IEFE, doi: 10. 109/ CVPR52688 200..00..22226212
10

---

[9] Robin Strudel, Ricardo Garcia, Ivan Laptev. and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. Proceedings of the IEEE/CVF international conference on computer vision, pages 7262-7272, 2021.
[10] Yuqi Si and Kirk Roberts. Three-level hierarchical transformer networks for long-sequence and multiple clinical documents classification, 2021. URL wtttttttttstti....t//aaaa/ wwtpxivv..rg/bbb////00
08444
[11] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: Cross-attentinn multi- scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 357-366, New York, NY, USA, 2021. IEER.
[12] Lili Yu, DÅąniel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. Megabyte: Predicting million-byte sequences with multiscale transformers, 2023.
[13] Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-unet: Unet-like pure transformer for medical image segmentation. European conference on computer vision, pages 205-215 Springer. 2022.
[14] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[15] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6824-6835, 2021.
[16] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEF/CVF conference computer vision and pullern recognition, pages 4804 2022.
[17] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything images and videos. arXiv preprint arXiv:2408.00714 2024. URL
00704
[18] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett I Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEF/CVF winter conference on applications of computer vision, pages 574-584, 2022.
[19] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander Berg, Wan- Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026. 2023.
[20] Yoo Jung Kim, Hyungjoon Jang, Kyoungbun Lee, Seongkeun Park, Sung-Gyu Min, Choyeon Hong, Jeong Hwan Park, Kanggeun Lee, Jísoo Kim, Wonjae Hong, Hyun Jung, Yanling Liu, Haran Rajkumar, Mahendra Khened, Ganapathy Krishnamurthi, Sen Yang. Xiyue Wang, Chang Hee Han, Jin Tae Kwak, Jianqiang Ma, Zhe Tang, Bahram Marami, Jack Zeineh, Zixu Zhao, Pheng-Ann Heng, RAijdiger Schmitz, Frederic Madesta, Thomas RÄúsch, Rene Werner, Jie Tian, Elodie Puybareau, Matteo Bovio, Xiufeng Zhang. Yifeng Zhu, Se Young Chun, Won- Ki Jeong, Peom Park, and Jinwook Choi. Paip 2019: Liver cancer segmentation challenge. Medical Image Analysis, 67:101854, 2021. ISSN httpp//cccccccccooccccc//c//aa/a
doi: https://doi.orr/10000 1016/ j.media. 2oaedia.2020100130 101854. URL hww.e////e.wwcccccceeeeee..cc. $1361841520302188
[21] Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Shuaiwen Leon Song, Samyam Rajbhandari, and Yuxiong He. Deepspeed ulysses: System optimizations for enabling training of extreme long sequence transformer models, 2023.
11

---

[22] Dacheng Li, Rulin Shao, Anze Xie. Eric P. Xing, Joseph E. Gonzalez, lon Stoica, Xuezhe Ma, and Hao Zhang. Lightseq: Sequence level parallelism for distributed training of long context transformers, 2023.
[23] Hao Liu, Matei Zaharia, and Pieter Abbeel. Ring attention with blockwise transformers for near-infinite context, 2023.
[24] Xiao Wang. Isaac Lyngaas, Aristeidis Tsaris, Peng Chen, Sajal Dash, Mayanka Chandra Shekar, Tao Luo, Hong-Jun Yoon, Mohamed Wahib, and John Gouley. Ultra-long sequence distributed transformer, 2023. URL httpt://ta//ax......om
org/ aabb//311.1 .23822.
[25] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher RÅT. Flashattention: Fast and memory-efficient exact attention with io-awareness. In NeurlPS: Proceedings of the 35th Neural Information Processing Systems Conference, New York, NY, USA, 2022. Association for Computing Machinery. doi: 1044850/ARXX..205.442. 14135. URL https
//arxiv.org wwt.:or///bbb/o555m/ 14135
[26] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023.
[27]
Tri Dao, Beidi Chen, Nimit S Sohoni, Arjun Desai, Michael Poli, Jessica Grogan, Alexander Liu, Aniruddh Rao, Atri Rudra, and Christopher Re. Monarch: Expressive structured matri- ces for efficient and accurate training. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4690-4721. PMLR, 17-23 Jul 2022. URL htttpp::/ppoceeeeeeeee dao22a. html
mlr press/v162/
[28] Deyu Bo, Chuan Shi, Lele Wang, and Renjie Liao. Specformer: Spectral graph neural networks meet transformers. In The Eleventh International Conference on Learning Representations, 2023. URL httphttts///peeeriiaee wwtp:/eeetooooeeooeeee http//oorrmm/1dopppsttso.....
[29] Devin Kreuzer, Dominique Beaini, Will Hamilton, Vincent Léourrnaau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neu- ral Information Processing Systems, volume 34, pages 21618-211 21689, Curran Associates, Inc., 2021. URL 144412220055300bccae666777777771
nwwrips ctcccc//aaaa faseerrfflllspppppeeeee wtt.f/lle///ppe/22////iiiee pdf
[30] Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers. arXiv preprint 1rXiv:2009.14794, 2020.
[31] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers rnns: Fast autoregressive transformers with linear attention. In Proceedings of the 37th Inter- national Conference on Machine Learning, ICML'20, New York, NY, USA, 2020. Association for Computing Machinery.
[32] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers, 2019. https:stt///// http:/awwww.w.oraaaaaaaaa htrgg/bss19904.1 10509
[33] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer.
In The International Conference Learning Representations (ICLR), New York, NY, USA, 2020, Association for Computing Machinery. doi: 10.48550/ARXIV2200..1 044451. URL https
// 1/arxiv.orz larxiv.org/aa httplv.orrgabbb///0/ee abs 2001 04451
[34] Aurko Roy, Mohammad Saffar, Ashish Vaswani, and David Grangier. Efficient Content-Based Sparse Attention with Routing Transformers. Transactions of the Association for Computational Linguistics, 9:53-68, 02 2021. ISSN 2307-387X. doi: 10. 16. a 00353, URL https //doi.org/10.1111 www./t1ccccccccc tacpl//tclllaaaaaaa _00333
[35] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint a5Xiv:2004.05150, 2020.
12

---

[36] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, al. Big bird: Transformers for longer sequences. Advances in Neural Information Processing Systems, 33: 17283-17297, 2020.
[37] Chengxuan Ying, Guolin Kc, Di He, and Tie- Yan Liu. Lazyformer: Self attention with lazy update, 2021.
[38] Markus N. Rabe and Charles Staats. Self-attention does not need o(n²) memory, 2022.
[39] Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying sparse and low-rank attention. Advances Neural Information Processing Systems, 34:17413-1744 17426,, 2021.
[40] Han Shi, Jiahui Gao, Xiaozhe Ren, Hang Xu, Xiaodan Liang. Zhenguo Li, and James Kwok. Sparsebert: Rethinking the importance analysis in self-attention. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9547-9557 New York, NY, USA, 2021. PMLR. URL http://proceedii
mlr
html
[41] Shitao Tang, Jiahui Zhang, Siyu Zhu, and Ping Tan. Quadtree attention for vision transformers. 2022.
arXiv preprint
[42] Moritz lbing, Gregor Kobsik, and Leif Kobbelt. Octree transformer: Autoregressive 3d shape generation on hierarchically structured sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2697-2706, 2023.
[43] Enzhi Zhang, Isaac Lyngaas, Peng Chen, Xiao Wang, Jun Igarashi, Yuankai Huo, Masaharu Munetomo, and Mohamed Wahib. Adaptive patching for high-resolution image segmentation with transformers. In SC24: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2024.
[44] John Canny. A computational approach edge detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMII-(6):679-699 1986. doi: 10. (0.1109/TPAMI.190 4767851.
[45] Hanan Samet. The quadtree and related hierarchical data structures. ACM Computing Surveys (CSUR), 16(2):187-2600 1984.
[46] Hal Finkel and Jon Bentley. Quad trees: a data structure for retrieval on composite keys. Acta Informatica, 4(1):1-9, 1974.
[47] Timothy M. Chan, Sariel Har-Peled, and Mitchell Jones. On Docality-sensitiie orderings and their applications. SIAM Journal on Computing, 49(3):583-600 2020. doi: 10. 1 13/1199912224499. URL htttt::://aattt org/10 10. 1137 119M/2449113
[48] B Landman, Z Xu, J Igelsias, M Styner, T Langerak, and A Klein. Miccai multi-atlas labeling beyond the cranial worksho and challenge. In Proc. MICCAI Multi-Atlas Labeling Beyond Cranial Vaulta-TWorkkhhop Challenge, 2015.
[49] Du Wu, Peng Chen, Xiao Wang, Issac Lyngaas, Takaaki Miyajima, Toshio Endo, Satoshi Matsuoka, and Mohamed Wahib. Real-time high-resolution x-ray computed tomography. In Proceedings of the 38th ACM International Conference on Supercomputing, pages 110-123, 2024.
[50] Nicholas Heller, Fabian Isensee, Klaus H Maier- Hein, Xiaoshuai Hou, Yucheng Xie, Fengyi Li, Yang Nan, Guangrui Mu, Zhiyong Lin, Miofei Han, Zhenlin Peng, Chunmei Sun, Ziyu Wu, Xiaojie Qiu, Zaiyi Chen, Chaojie Liang, Erick Remer, Jennifer C Teh, Shengkai Cui, Arveen Kalapara, Valentina Sandfort, Zhoubing Xu, Neal Schenkman, Alexander Kutikov, Joel Rosenberg, and Fakhran. The kits19 challenge data: 300 kidney tumor cases with clinical context, semantic segmentations, and surgical outcomes.
13

---

[51] Fabian Isensee, Tassilo Wald, Constantin Ulrich, Michael Baumgartner, Saikat Roy, Klaus Maier-Hein, and Paul F Jaeger. nnu-net revisited: call for rigorous validation in 3d medical image segmentation. arXiv preprint 2rXiv:2404.09556, 2024.
[52] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention- 20/5: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241 Springer, 2015.
[53] Jieneng Chen, Yongyi Lu, Qing Yu, Xiaomeng Luo, Ehsan Adeli, Yan Wang, Le Wang, and Chenyang Lu. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2 /02. 04306, 2021.
[54] Jun Ma, Feifci Li, and Bo Wang. U-mamba: Enhancing long-range dependency for biomedical image segmentation. arXiv preprint arXiv.2401.04222 2024.
[55] Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L.. Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. CoRR, abs/2102.04306, 2021.
[56] Yucheng Tang, Dong Yang, Wenqi Li, Holger R. Roth, Bennett A. Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In CVPR, pages 20698-20708. IEEE, 2022.
[57] Yuyin Zhou, Lingxi Xie, Wei Shen, Yan Wang, Elliot Fishman, and Alan L Yuille. A fixed- point model for pancreas segmentation in abdominal ct scans, In International conference on medical image computing and computer-assistee intervention, pages 693-701. Springer, 2017.
[58] Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for accurate and interpretable prediction of material properties. Physical review letters, 120 120(14): 145300,, 2018.
[59] Yue Liu, Tianlu Zhao, Wangwei Ju, and Siqi Shi. Materials discovery and design using machine learning. Journal of Materiomics, 3(3):159-177, 2017.
[60] Alessandro Motta, Manuel Berning, Kevin M Boergens, Benedikt Staffler, Marcel Beining, Sahil Loomba, Philipp Hennig, Heiko Wissler, and Moritz Helmstaedter. Dense connectomic reconstruction in layer 4 of the somatosensory cortex. Science, 366(6469)xcaay31 134, 2019.
[61] Michal Januszewski, Jörgen Kornfeld, Peter H Li, Art Pope. Tim Blakely, Larry Lindsey, Jeremy Maitin ShepandShepard, Mike Tyka, Winfried Denk, and Viren Jain. High-preeiste automated reconstruction of neurons with flood-filling networks. Nature methods, 15(8):605-610, 2018.
[62] Du Wu, Enzhi Zhang, Isaac Lyngaas, Xiao Wang, Amir Ziabari, Tao Luo, Peng Chen, Kento Sato, Fumiyoshi Shoji, Takaki Hatsui, al. Paradigm shift in infrastructure inspection tech- nology: Leveraging high-perfie nigh-performance imaging and advanced ai analyties to inspect road infrastructure. arXiv preprint ar X5X.:2500..3955. 2025.
[63] Peng Chen, Mohamed Wahib, Xiao Wang, Takahiro Hirofuchi, Hirotaka Ogawa, Ander Biguri, Richard Boardman, Thomas Blumensath, and Satoshi Matsuoka. Scalable fop decomposition for cone-beam et reconstruction. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16, 2021.
[64] Peng Chen, Mohamed Wahib, Shinichiro Takizawa, Ryousei Takano, and Satoshi Matsuoka. lfdk: A scalable framework for instant high-resolution image reconstruction. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-24, 2019.
[65] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PmLR, 2021.
14

---

[66] Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Yang. Etc: Encoding long and structured inputs in transformers. arXiv preprint arXiv:2004.08483, 2020.
[67] Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33: 17283-17297, 2020.
[68] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. arXiv preprini arXiv:2001.04451, 2020.
[69] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
[70] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are mms:: Fast autoregressive transformers with linear attention. International conference on machine learming, pages 5156-5165. PMLR, 2020.
[71] Jieru Mei, Liang- Chich Chen, Alan Yuille, and Cihang Xie. Spformer: Enhancing vision transformer with superpixel representation. arXiv preprint arXiv:2401.02931, 2024.
[72] lya Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint urXiv:1711.051001 2017.
[73] The Frontier supercomputer. htttps///ww. olcf. ornl goroo/oor/ootterr///////
[74] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou. Three things everyone should know about vision transformers. In European Conference on Computer Vision, pages 497-515. Springer, 2022.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explained the improvements in training speed and model performance in the introduction, which can be seen in the numerical values in Table Table Table 3 and T or the image quality in Figure
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of this application some scenarios, especially when subdivision is wrong, in the supplementary material.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[NA]
Justification:
This paper does not contain theoretical proofs and results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Sec step-by-step code and pseudo code in the supplement material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We added a Python pseudocode and instructions in the supplement material and will make the code public once accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so âÄIJNoâÄī is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/nipsscc/ppli wttpcccppppccceeee.ce..e....leel guides/ wwt////ooeimisiiioooooooo solooocccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We described the framework in the Fig. added the step-by-step algorithm and detailed settings in the supplement material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We added statistical significance for the small scale in the experiments in the supplement material. We hope the reviewer understands that the computational resources for high-resolution experiments are huge.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide necessary hardware requirement in Tab. I Tab. 2
And in the
supplement material, we provide detailed resource requirements and scripts for launching
the jobs.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This paper focuses high-resolution image segmentation.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All datasets are public.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development this research does involve LLMs.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neunee /0tps///cooooeeeeeee/222//////// for what should or should be described.
22

---

Appendix
Appendix Contents
Summary of Methods for Training Long Sequences
Discussion and Ablation Studies
B.
Sequence length 7. and compression ratio 7
B.2
Image Information Loss in Patching and Depatching
B.3
Training the SHF Algorthm: Hyperparameters and Loss
B.4
Learning From the Encoded Image Space
B.5
Experimental Setup
B.6
Compared to Adaptive Patching Methods
Generalization of SHF to Non-Medical Domains)

---

A Summary of Methods for Training on Long Sequences
Table 5 provides a summary of recent approaches that address the long-sequence challenges encoun- tered by VIT models when processing high-resolution images.
Table 5: A summary of relevant long sequence training methods for solving the quadratic attention through the reduce the amount of work. Here N = sequence length.
Appeoach
Method
Demerits
Complesio (Best)
Implocentation
OIN)
Longooener
(+) Benter tine complecry vs
((
Some Models Foried Py
Self attention Impleneens
Spanity levels insuficienn gains
matersliz.
Atteitonn Approsination
Tie Rmm to Revomer
M1
(4) m prive tim ompies ON Ing V]
Some Micale WT Fornnt Py. Torch
Nel antertion ingirment Goa
My.
[)] Highander deriveiass
Spane Altation (
(1) Innoodaced spane ficiccinonss of OINVN)
Some Mocels WT Forted Py Torch
Self attcciona ling lereene
the ditectio.
Soa
( Higher complediy.
[e) neggtiioo or
ONN
Poroed Py-
antertonooon Inglenent
Assumption atentiin
Torch
Goa
SPFormer X (rodiction)
(4) Iregular uikets.
(e) No adguution resolation
Pauu regines
Madel wi Pain PyTrech
Model Implementation
Hie. Trarsformer 1o NTw Clasiffction)
(+) Indegendert hyperpac. tining of O1N N)
Model w/ Plain PyTooch
Model Implemestatio
bizam. malch.
No sugeen cumpassy
Hlenachical
Vir.
Implonetaaic
6) Comples token fuson scheme in
daal beasch VITs
HIM (CSeciiaaion)
(-) Maak rituaiive bince in ftanno in ONN the heeaachy.
Midldl wi Team Tiech
Mosel Implanctation
() High crel for tnáning muliiplc mod-
MEGABYTE (Peddcctoon
(4) Suppost multi modales.
Model Plain PyToech
Model Implenentation
High roiining multiple
Spice Model
MEGARRTTS (Pediction)
Suepeet multi-madalirs
O
PyTlcchh
Misiel Implomenation
fox muliiple mod.
Prodiction)
(+) Suepeet of multi-madslic
O
Model sel Pain PyTlochh
Model Implemetttion
() High cost for traning nuliiple mod. ds
Hish-resolution
MEGABYTE EM (Podliction)
(4) Suppeei of muli--odalii-
Midel wi Plan PyToech
Model Implementation
6- High onet for miiinng mumpple mod- ds.
TIRTARYT D (Prsditioo)
(+) Stepert of mannmaaallyy
Mistel wi Fiom By
Moxel Implomenttion
() High cost fox trainigg maitple mod ds.
(+) Artention mechunkem
Largels roduces compotation mainbeis quality.
Ours
Symmetrical Eleerachical Forest
(Seamentation & Cliss)
talls in an Image
(1) Erfclency depends on level of do- O(ksg N)
Any Midel wo Plain By- Torch
Image Prospreccsiigg
(-) bak, samantiss are independent of
edge infrrmation.
B Discussion and Ablation Studies
B.1
Sequence length L and compression ratio
Fig.P illustrates how different sequence lengths affect the compression ratio of the original image, using the same input image and edge extraction algorithm. The first column displays an edge image with resolution of 1024 x 1024 pixels. We evaluated sequence lengths correspond to average patch sizes of [31.7, 4.3, 9.17] pixels under the grid patch configuration. The resulting compression ratios y are [1.07, 12.18, 27.66). For image segmentation, a sequence length of 1024 appears to provide the best balance for images at 1K resolution. In 3D, the compression ratios increase to [6.06, 22.47, 47.76), as higher dimensionality Icads to sparser information within each patch, thereby making experiments in higher visual dimensions feasible.
[256, 1024, 2050), which
The key reason SHF can handle small patch sizes at high resolutions is its ability to reduce sequence length through hierarchical forest patching. In Fig. [ we illustrate how the sequence length can be adjusted by tuning the threshold of the split value without substantially affecting prediction performance. The split value governs both the total sequence length and the distribution of patch sizes. The first row of Fig. 5 shows that halving the split value [100, 50, 20) leads to roughly proportional change in patch size distribution, with average patch sizes [12.77, 19.17 17, 29.88). This demonstrates linear relationship between the split value and the average patch size. In contrast, for the uniform grid patching strategy, sequence length grows quadratically (((3))) Using

---

hierarchical forest patching, however, observe an approximately linear increase in average sequence length as the average patch size decreases. In 3D, the average patch sizes [6.27, 11.79, 17.86 are larger due to increased dimensionality, which results in sparser information per patch and makes experiments in higher visual dimensions feasible.
B.2 Image Information Loss in Patching and Depatching
In Fig. 4 we present the tree structure generated from the input image using the patch partitioning strategy, showing the effects of compressing and reconstructing the original mask. When decoding the mask through spatial partitioning. the accuracy can theoretically reach an upper limit of 99.5% with perfect predictions. However, as our experiment' sequence mask achieves only 62.97%, this tree-structured reconstruction s impact on decoding performance is relatively minor.
Notably, SHF differs from convolution-base believe this is due to two types of losses: the first is texture loss, where incorrect model regression generates inaccurate textures, leading noise artifacts in the SHF image. The second is structural loss, observed as an amplification of noise in patches with varying structures introduced during the Dispatching stage. We attribute both losses to the loss of geometric relationships between patches during compression. Fig. 5 illustrates the degradation of these geometric properties.
decoder in terms of image quality degradation. We
B.3 Training the SHP Algorithm: Hyperparameters and Loss
Since depatching happens in the evaluation stage, algorithm is also divided into the training stage and the evaluation stage. Let's first look at the training stage, where L represents the length of the hierachical patching sequence, K represents the optional kernel size, and la represent the lower and higher threshold of canny edge detection, f is the model, æ is the input, 0 is the trainable parameter, 1) is the dataset, N the number of batches, and F. is the total number of epochs required. Here, we provide a Python pseudocode in Code Listing for reference.
def bui faild_hheene huild_hierachicalfforee
(self):
h, self .ww.dddmaaana..................d shape
root Rect (0,,,,, self. nodes = [[root, root. contains (self, ( domain) while len nodes) < self fixed_length: (self nodes, key=lambda x:x[1])
bbox, value
idx self nodes index ([bbox, value]) if bbox. gexx.get_size () [0] 2: break
x1. x2 y2,y2 abox.get_coordd
It Rect(x1. int (xx1+x2)))) int ((y1+y))))) y2)
v1 1t. containn domain)
It Rect Dect(inne ((x1+x2))) x2, (nt((r)y2)/2)) y2)
v2 contains domain)
lb Rect int((x1=x2/2)), y1, int ((y1+y))) )2))
v3 lb. Do.contains (omaii)innm.
rb Dent(intee (ix(x(x)))))))) x2, y1, (nt1(y)))) ) /2))
v4 rb. concatinin domain)
self nodes self nodes [dx] [[it,vi], [rt, v2], [1b, v3],
rb, v4]]
self nodes [idx+1:]
Code Listing 1: The implementation the proposed hierarchical forest-building algorithm in Python.
As shown in the training stage of overview in Section 2 the input image first goes through Gaussian smoothing and canny edge detection to get Ze, then the edge image ve goes through hierarchical patching to get the forest Tr and an encoded sequence of length L, and we also use T, to encode mask y to get yp- Then, we use the model to train on data with Tp as input and Уp as mask. After obtaining the predicted encoded sequence mask 1/ for the evaluation stage, we send it to the hierarchical dispatching stage and reconstruct the prediction ÿj. Then, we can calculate the dice score between ground truth y and ÿ.

---

We train the model using the Adam V optimizer 72 with an initial learning rate of le-4 over 800 epochs. The first 20 epochs are allocated for learning rate warm-up. followed by a decay by factor of 0.1 at epochs 400 and 600. By epoch 800, the model converges. To maximize training speed, we select the largest possible batch size within the GPU memory limits. The loss function combines Dice loss and binary cross-entropy (BCE) loss:
L(U.0 v) = Lee (0.9) (1 w) -idice (Ū.
w
Mi log(ūs) + (1 Y/s log(1 2t)]
(6)
2 2 1= (ūi Ui) + D d ;= Ui +
+ (1 w) (1
where L( y) represents the combined loss function comprised of weighted sum of cross-entropy (CE) loss and dice loss. The parameter w controls the balance between CE loss and dice loss, set to 0.5 in our experiments. To stabilize calculations, the smoothing term is set 1.0.
B.4 Learning From the Encoded Image Space
key concern is to ensure is that the spatial structure learned in the embeddings ğ corresponds to the spatial structure of the sequence We assume that if the output prediction can completely match the compressed mask ye. then there should be a minimal loss in translation to the original mask through the spatial structure. To achieve this, we expect the regression ability of the transformer to learn the implicit spatial matching from the embedding space to the geometric pixel space, as empirically observed by a direct decrease in the loss. Since, for the positional encoding, we chose trainable embedding vector with an initial value of zero, we speculate that if the order of patches, after hicrarchical forest patching, affects the learning, then the trainable positional encoding may start to learn the positioning of tokens as manifested by the Z-order curve of the quad/octree.
B.5 Experimental Setup
Hardware. All experiments were conducted on the Frontier Supercomputer 773 ORNL. Each Frontier node is equipped with 64-core AMD EPYC CPU and four AMD Instinct MI250X GPUs, each with 128 GB of memory. The four MI250X GPUs on a node are interconnected via AMD Infinity Fabric at 50 GB/s. Nodes are connected through the Slingshot-11 interconnect with 100 GB/s bandwidth, across a total of 9,408 nodes.
Software. For the software stack, we used the Py PyTorch 2.4 nightly build (03/16/2024), ROCm v5.7.0, M1Open 19.09.0, and RCCL v2.13.4 with the libfabric v1.15.2 plugin. This configuration enabled efficient multi-GPU training and high-performance communication across nodes for large- scale experiments.
B.6 Compared to Adaptive Patching Methods
key motivation for developing SHF was to overcome the core robustness issues and significant hyperparameter sensitivity inherent in the AP approach. Rather than merely fine-tuning hyperparame- ters, SHF introduces a more fundamental solution by reframing hyperparameter choice as a structured, hierarchical feature extraction problem. This design choice is the key its superior performance. We will add a table to the main paper showing the performance gains of SHF over AP, particularly at high resolutions and on the large-scale ImageNet- 1K dataset. Furthermore, we will provide visualizations in the appendix that clearly demonstrate how different parameters influence the feature extraction process across the hierarehy. To demonstrate the effectiveness of SHF over AP, in hish-resolttion images, the following table shows the Speedup of SHF end-to-end training for the PAIP dataset at the same segmentation quality as the baseline. We use the highest dice score of the baseline models SHE and AP.
3

---

(a) Input image
(b) Ground Truth
(c) Reconstruct mask
(d) Hierarchical Forest Patched Im- age
Hicrarchical Mask
Forest
Patched
(f) Hicrarchical Forest Patched Pre- diction
Figure 5: An illustrative example of image processed using hierarchical forest patching, along with the corresponding reconstructed mask derived from the patching mask. During compression from the image domain the sequence domain, the geometric relationships between patches lost. Consequently, transformers must learn from sequences that do not preserve spatial structure and cannot rely inherent geometric information.
B.7 Generalization of SHF to Non-Medical Domains
To evaluate the generalization of SHF beyond medical imaging, we conducted experiments on the widely used ImageNet dataset, as summarized in Table [6] It is important to note that SHF was originally designed for altre-higg- medical images; therefore, its performance on ImageNet is not intended to replicate state-of-the-ar results. Nevertheless, we include both our results and those reported in [74 for comparison. shown Table SHF-ViT outperforms the standard VIT by leveraging hierarchical information. However, when only single tree with fixed hyperparameters used, performance falls below that of the basic ViT, indicating that fixed hyperparameters can limit generalization.
Table 6: Comparison of Vision Transformer (ViT) model accuracies on ImageNet- K.
Model
Sequence Length
Validation Acc.
V2 Acc.
ViT-B/16 74
196
82.2%
72.2%
Tree-Vits ViT--/16
256
77.3%
65. 65.1%
SHF-VTT-B/16
196
82.7%
72.8%
ViT-L/16 [74]
196
83.0%
72.4%
ViT-LVI
256
77.8%
65.7%
SHF-VIT-L/16
196
83.4%
73.1%

---

Courr of Tots Procise of PAP
Couet Tots Preciee PAP
Covm of Trea Phnnkes HAI
100
250-
40-
Counte
106
134
Patlles
(a) 512, Avg patch size=
L 256, Avg patch 1ize=19.17 (c) 128, Avg voxel size= 29.88
Count of Tota Patchea el PAP
Count Total Patcha PAP
Court of Total Patcha PAP
100
Gavite
Counte
100
20
755
(d) 512, Avg voxel size=6.27 (e) L 256, Avg voxel size=11.79 (i) 128, Avg voxel size=17.86
Figure 6: Average patch size [1..77, 19.17, 29.88] of training images with 1024 resolution in PAIP and average octree voxel size 66.27, 11.79, 17.86) training images in KiTS lead to empirical linear scaling of the corresponding sequence length [512, 256, 128). Compared normal distribution, 3D voxel size distribution is preferred to small patches, which is considered better compression ratio.
2D patch size in


Reviews:
[{"review_id": "QFi1RNBN7E", "reviewer": "Reviewer_QK6b", "summary": "This manuscript presents the Symmetrical Hierarchical Forest (SHF) framework, which proposes a quadtree/octree‐based preprocessing and postprocessing strategy that injects hierarchical spatial priors into an unmodified Vision Transformer, enabling it to handle gigapixel 2D pathology slides and volumetric 3D scans. This change dramatically reduces compute and memory compared to vanilla ViTs: by greedily splitting only the most “edge-rich” regions into smaller patches and leaving smooth areas coarse, the authors report significant improvements in both runtime and segmentation performance across different evaluation settings.", "questions": "- I would suggest the authors consider the inclusion of an additional self-attention cost addressing baseline to strengthen their performance/runtime tradeoff claim.\n\n- I suggest improving the clarity of the paper by moving the related works section to be after the introduction, expanding on it a bit more thoroughly, and generally improving the order and relevance of the figures in the manuscript, moving those that are not addressed in the main text to the appendix (as well as referencing them appropriately within the appendix).", "limitations": "Authors should include a discussion on the limitations and potential failure modes of the approach, like cases in which the detail score (the sum of edge pixel values) may overlook semantically important but low‐contrast regions, potentially missing subtle lesions or structural features not captured by simple edge detection methods.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n\n- SHF’s quadtree patching concept is elegant and model-agnostic, and thus can wrap around any pretrained ViT, avoiding custom attention or encoder modifications\n\n- The authors extensively demonstrate the runtime and performance gains of their methods with respect to to vanilla SAM/SAM 2, HIPT and common UNet-style baselines\n\n- Approach is shown to be versatile across 2D pathology and 3D MRI\n\nWeaknesses:\n\n- I find the general structure/flow of the paper to be a bit confusing and awkward for several reasons. (1) the related works section is located at the very end of the paper, when it is customary to place it after the introduction to better frame the contributions: pushing it to the end makes it harder for readers to situate SHF relative to prior approaches. (2) Figure 3 is never cited in the main text: it took me some time to understand it was being adressed in the appendix, where it is also not explicitly mentioned in the corresponding section (Appendix 1.1). It should be either moved entirely to the appendix or discussed in the main manuscript.  (3) Figure 1 is only referenced at the very end of the paper, well into section 4.2. The figure should be either moved further down the paper or discussed/referenced earlier on to avoid disrupting the narrative flow of the document.\n\n- While the related work survey lists many strategies for mitigating the quadratic cost of self-attention, the experiments mostly compare SHF to vanilla SAM/SAM 2, common UNet-style baselines, and only consider HIPT in one of their experiments as a cost effective baseline. The manuscript would benefit from including at least one other approach that also addresses cost of self-attention to patch size tradeoff, to better highlight the superiority of SHF.", "was_revised": true, "final_justification": "The rebuttal effectively resolved my primary concerns. The authors' plan to restructure the manuscript for improved clarity, along with the expanded discussion of alternative self-attention approximations and their additional comparisons in the appendix, strengthen the positioning of SHF in the broader context of efficient ViT adaptations. I also appreciate the detailed discussion of limitations, particularly around edge-based splitting and its potential failure modes. These additions round out the work, and in light of these improvements, I am raising my score to 5 (Accept).", "rebuttal": "We sincerely thank you for your constructive feedback on our manuscript. All these additions and clarifications will be incorporated into the final manuscript. Below, we address the points you have raised:\n## 1.\tRegarding the paper's structure and figure references: \n\nThank you for your valuable suggestions concerning the organization of the paper and the clarity of figure references. In the revised manuscript, we will relocate the \"Related Work\" section to improve the flow and will ensure that all figures are properly referenced and explained in the text.\n## 2.\tRegarding the computational cost of self-attention: \n\nWe appreciate you raising this important point. To provide a clear analysis, we have compiled a comprehensive table that compares the time complexity and performance trade-offs associated with various self-attention and linear attention mechanisms. While we believe this information is highly relevant, we will place it in the appendix due to page limitations in the main body of the paper.\n\n_A summary of relevant long sequence training methods that reduce the amount of work. *N* = sequence length._\n\n| **Approach** | **Method** | **Merits & Demerits** | **Complexity (Best)** | **Model** | **Implementation** |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| Attention Approximation | Longformer [beltagy2020longformer] ETC [ainslie2020etc] | **(+)** Better time complexity vs Transformer. \\ **(-)** Sparsity levels insufficient for gains to materialize. | O(N) \\ O(N√N) | Some Models w/ Forked PyTorch | Custom Self-attention Implementation |\n| | BigBird [zaheer2020big] Reformer [kitaev2020reformer] | **(+)** Theoretically proven time complexity. \\ **(-)** High-order derivatives | O(NlogN) | Some Models w/ Forked PyTorch | Custom Self-attention Implementation |\n| | Sparse Attention [child2019generating] | **(+)** Introduced sparse factorizations of the attention. \\ **(-)** Higher time complexity. | O(N√N) | Some Models w/ Forked PyTorch | Custom Self-attention Implementation |\n| | Linformer [katharopoulos2020transformers] Performer [choromanski2020rethinking] | **(+)** Fast adaptation \\ **(-)** Assumption that self-attention is low rank. | O(N) | Some Models w/ Forked PyTorch | Custom Self-attention Implementation |\n| | SPFormer [mei2024spformer] (Prediction) | **(+)** Irregular tokens. \\ **(-)** No adaptation to high resolution. | O(P²) \\ P:num of regions | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| Hierarchical | Hier. Transformer [si21] (Text Classification) | **(+)** Independent hyperpara. tuning of hierarc. models. \\ **(-)** No support for ViT. | O(NlogN) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| | CrossViT [chen2021crossvit] (Classification) | **(+)** Better time complexity vs standard ViT.  \\ **(-)** Complex token fusion scheme in dual-branch ViTs. | O(N) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| | HIPT [Chen22] (Classification) | **(+)** Model inductive biases of features in the hierarchy.  \\ **(-)** High cost for training multiple models. | O(NlogN) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| | MEGABYTE [yu2023megabyte] (Prediction) | **(+)** Support of multi-modality. \\ **(-)** High cost for training multiple models. | O(N^4/3) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| High-resolution | xT [gupta2024xt] (Prediction) | **(+)** Support of 8K resolution . \\ **(-)** Lack of hierarchical relation in tokens. | O(NlogN) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| | HiFormer [heidari2023hiformer] (Prediction) | **(+)** Support of multi-modality. \\ **(-)** High cost for ultra-resolution. | O(NlogN) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| **Ours** | **Symmetrical Hierachical Froest** (Segmentation & Class.) | **(+)** Attention mechanism intact.  \\ **(+)** Largely reduces computation cost; maintains quality.  \\ **(+)** Efficiency depends on level of details in an image. \\ **(-)** task semantics are independent of edge information. | O(log²N) | Any Model w/ Plain PyTorch | Image Pre-processing |\n\n\n## 3.\tRegarding the limitations and potential failure modes: \nYour observation is astute and highlights a key aspect of our method. The behavior of the tree-node splitting is indeed influenced by the choice of the score function, which in turn can impact downstream performance. To address this, we have expanded the appendix to include a detailed discussion of potential failure modes in specific scenarios and an analysis of the limitations of our approach on non-medical datasets, using ImageNet-1K as a case study.\n\nRegarding generalization to other domains, we conducted experiments on the widely available ImageNet dataset. It's important to note that SHF was designed for ultra-high-resolution medical images, so its performance on ImageNet was not designed to replicate SoTA's experimental results. Even so, we still used our own results and those from a public paper [1] for cross-comparison. As can be seen in the table below, SHF-ViT outperforms basic ViT due to its use of hierarchical information. However, if only a single tree is used, i.e., fixed hyperparameters are used, it will be weaker than basic ViT. This is because fixed hyperparameters can lead to generalization performance deficiencies.\n\n_Training on Imagenet common dataset to show the generalization.(300 epoch, top-1 accuracty)_\n\n| Model | Sequence Length | Validation Accuracy | V2 Accuracy |\n| :--- | :--- | :--- | :--- |\n| **ViT-B/16[1]** | 196 | 82.2% | 72.2% |\n| **Tree-ViT-B/16** | 256 | 77.3% | 65.1% |\n| **SHF-ViT-B/16** | 196 | 82.7% | 72.8% |\n| **ViT-L/16[1]** | 196 | 83.0% | 72.4% |\n| **Tree-ViT-L/16** | 256 | 77.8% | 65.7% |\n| **SHF-ViT-L/16** | 196 | 83.4% | 73.1% |\n\n\n[1] Touvron H, Cord M, El-Nouby A, et al. Three things everyone should know about vision transformers[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 497-515.\n\nAll these additions and clarifications will be incorporated into the final manuscript."}, {"review_id": "Jv9J7hrmMr", "reviewer": "Reviewer_QeSt", "summary": "This paper proposes  Symmetrical Hierarchical Forest, which is used to enhance ViT models for high-resolution medical image segmentation. SHF uses hierarchical structures to patch the image, which emphasizes fine local detail in critical regions. The method also introduces a symmetrical dispatching strategy to recover full-resolution outputs. The authors combine SHF into existing ViT-based segmentation models and show improvements in speed and memory usage, along with better segmentation performance across several datasets.", "questions": "Please refer to the weakness.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n -   The paper is well written and easy to follow. The figures are visually appealing, and the code is provided in the supplementary materials, making the work reproducible.\n\n -   The proposed model achieves a significant speed improvement in medical image segmentation. I agree with the authors that this is an important application area, as 3D medical images typically require sliding-window inference, which is computationally intensive.\n\nWeaknesses:\n\n-    The idea of using a hierarchical architecture is not particularly novel. From my perspective, the paper essentially explores ways to cluster the semantic content of patches—a direction with a substantial body of prior work. For instance, SPFormer employs superpixels instead of patches and performs well on real-world image segmentation. In the medical imaging domain, models like HPFormer have also been proposed. I would encourage the authors to discuss more explicitly how their approach differs from and advances beyond these prior methods.\n\n-    The experimental setup is somewhat unclear. For example, regarding SAM2: was it evaluated directly using the pretrained model, or was it fine-tuned on the medical image datasets? I am concerned that the observed performance gains might simply be due to the use of a stronger segmentation backbone. Additionally, the baselines used for comparison seem outdated—could the authors include more recent and competitive methods for a fairer evaluation?\n\n-    There are a few typos in the paper. For instance, on line 225, \"prestrained\" should likely be \"pretrained.\" It would be helpful if the authors could proofread and correct such errors.", "was_revised": true, "final_justification": "One of the key challenges in medical image processing is the high computational cost associated with high-resolution images, particularly in 3D datasets. This paper presents an effective architectural solution to address this issue by significantly improving efficiency without compromising performance. Given its practical relevance and demonstrated benefits, I believe the proposed approach makes a valuable contribution and raises my score to Accept.", "rebuttal": "We sincerely thank you for your feedback and for pointing out the typos, which will be fixed in the revised version.\n\n## 1. Regarding Novelty Compared to Hierarchical Models: \n(We first apologize for not finding HPFormer in the high-resolution medical image, so the answer is based on HiFormer, which is also a hierarchical high-resolution medical approach published in 2024. Please correct us if we were wrong.)\n\nWe appreciate the opportunity to clarify the novelty of our Symetical Hierarchical Forest (SHF) framework. Its primary advantages over existing hierarchical models are twofold:\n\n1. SHF is model-agnostic. Unlike architectures like HiNet or the more recent SPFormer, which require fundamental changes to the model's backbone, especially the self-attention mechanism. However, our method introduces a lightweight token selection module that leaves the core attention mechanism untouched. This design choice is significant as it allows for direct compatibility with a wide range of pre-trained ViT models and facilitates its application in multi-modal settings and future work.\n\n2. SHF achieves a significantly larger operational scale. Our framework can process inputs at extreme resolutions, reaching up to 65,535 with a patch size of 8. We believe this high-resolution capability is a substantial advance beyond what has been demonstrated in previous hierarchical high-resolution approaches (4K~8K).\n\n_A summary of relevant long sequence training methods that reduce the amount of work. *N* = sequence length._\n\n| **Approach** | **Method** | **Merits & Demerits** | **Complexity (Best)** | **Model** | **Implementation** |\n| :--- | :--- | :--- | :--- | :--- | :--- |\n| Attention Approximation | Longformer [beltagy2020longformer] ETC [ainslie2020etc] | **(+)** Better time complexity vs Transformer. \\ **(-)** Sparsity levels insufficient for gains to materialize. | O(N) \\ O(N√N) | Some Models w/ Forked PyTorch | Custom Self-attention Implementation |\n| | BigBird [zaheer2020big] Reformer [kitaev2020reformer] | **(+)** Theoretically proven time complexity. \\ **(-)** High-order derivatives | O(NlogN) | Some Models w/ Forked PyTorch | Custom Self-attention Implementation |\n| | Sparse Attention [child2019generating] | **(+)** Introduced sparse factorizations of the attention. \\ **(-)** Higher time complexity. | O(N√N) | Some Models w/ Forked PyTorch | Custom Self-attention Implementation |\n| | Linformer [katharopoulos2020transformers] Performer [choromanski2020rethinking] | **(+)** Fast adaptation \\ **(-)** Assumption that self-attention is low rank. | O(N) | Some Models w/ Forked PyTorch | Custom Self-attention Implementation |\n| | SPFormer [mei2024spformer] (Prediction) | **(+)** Irregular tokens. \\ **(-)** No adaptation to high resolution. | O(P²) \\ P:num of regions | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| Hierarchical | Hier. Transformer [si21] (Text Classification) | **(+)** Independent hyperpara. tuning of hierarc. models. \\ **(-)** No support for ViT. | O(NlogN) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| | CrossViT [chen2021crossvit] (Classification) | **(+)** Better time complexity vs standard ViT.  \\ **(-)** Complex token fusion scheme in dual-branch ViTs. | O(N) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| | HIPT [Chen22] (Classification) | **(+)** Model inductive biases of features in the hierarchy.  \\ **(-)** High cost for training multiple models. | O(NlogN) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| | MEGABYTE [yu2023megabyte] (Prediction) | **(+)** Support of multi-modality. \\ **(-)** High cost for training multiple models. | O(N^4/3) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| High-resolution | xT [gupta2024xt] (Prediction) | **(+)** Support of 8K resolution . \\ **(-)** Lack of hierarchical relation in tokens. | O(NlogN) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| | HiFormer [heidari2023hiformer] (Prediction) | **(+)** Support of multi-modality. \\ **(-)** High cost for ultra-resolution. | O(NlogN) | Custom Model w/ Plain PyTorch | Custom Model Implementation |\n| **Ours** | **Symmetrical Hierachical Froest** (Segmentation & Class.) | **(+)** Attention mechanism intact.  \\ **(+)** Largely reduces computation cost; maintains quality.  \\ **(+)** Efficiency depends on level of details in an image. \\ **(-)** task semantics are independent of edge information. | O(log²N) | Any Model w/ Plain PyTorch | Image Pre-processing |\n\n## 2. Regarding the Strength of the Segmentation Backbone: \n\nThank you for this question. We would like to clarify that all performance metrics reported for SHF-SAM2/SAM2 and SAM2/SAM2 were obtained after fine-tuning on the respective datasets. The comparisons presented in Tables 1-4 are designed to directly contrast SHF-SAM/SAM-2 with the baseline SAM/SAM-2, thereby demonstrating that our proposed method provides a clear performance gain on top of this already powerful segmentation backbone. We will add an explicit statement in the main paper to make this clearer. Additionally, our selection of UNETR and Swin UNETR as baselines was motivated by their established status as strong, widely-used models in recent medical imaging segmentation research.\n\nAll these additions and clarifications will be incorporated into the final manuscript."}, {"review_id": "8jhlGma9Zr", "reviewer": "Reviewer_N2tG", "summary": "This paper proposes Symmetrical Hierarchical Forest (SHF), a lightweight and architecture-agnostic preprocessing approach designed to improve the efficiency and accuracy of Vision Transformers (ViTs) on high-resolution medical image segmentation tasks. The core motivation stems from the high computational cost of applying ViTs to long input sequences, which typically arises when using small patches to preserve fine details. SHF addresses this by adaptively partitioning input images into non-uniform patches using a quadtree-based scheme guided by edge detection, resulting in fewer, information-rich tokens. This hierarchical tokenization captures local spatial detail while reducing sequence length, enabling standard ViTs to process high-resolution inputs without architectural changes or heavy decoders. During inference, a symmetric depatching process reconstructs the output mask directly from the encoder’s output. The method is demonstrated on 2D (PAIP) and 3D (BTCV, KiTS) medical imaging benchmarks, where it achieves improvements in segmentation accuracy and training speed, performing better than several previous baselines.", "questions": "The SHF framework relies on Canny edge detection with Gaussian smoothing to guide the hierarchical patch partitioning. While this is computationally lightweight, it is a hand-crafted heuristic that may not align well with semantic relevance in all medical contexts. Could the authors provide more analysis or ablation studies demonstrating how robust this approach is across diverse datasets or imaging conditions?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The following are the strengths of the paper, \n\n1. The paper addresses a well-motivated and practically important problem: applying Vision Transformers to high-resolution medical image segmentation, where the quadratic cost of self-attention with small patches becomes prohibitive. The proposed solution is relevant to real-world applications such as whole-slide image (WSI) analysis and 3D organ segmentation in clinical pipelines, where efficiency and precision are both critical. \n\n2. The proposed Symmetrical Hierarchical Forest (SHF) is conceptually straightforward, involving adaptive patching of the input based on quadtree decomposition of edge maps. This approach leads to a significant reduction in sequence length without modifying the transformer architecture or self-attention mechanism. The design choice to operate entirely in the input/output space, leaving the encoder unchanged, is notable for its practicality and ease of integration. \n\n3. The method is evaluated on both 2D (PAIP) and 3D (BTCV, KiTS) datasets and consistently shows improvements in both segmentation accuracy and computational efficiency.  \n\n4. SHF eliminates the need for convolutional decoders by reconstructing segmentation masks directly through a depatching operation that mirrors the hierarchical patching tree. This results in lower memory usage and model simplicity. For example, on 64K² input resolutions, SHF saves around 20GB+ memory. \n\n \n\nThe following are the weaknesses of the paper, \n\n1. The method depends on Canny edge detection and Gaussian blurring to determine which regions are subdivided during patching. While computationally efficient, these heuristics are not semantically grounded and may fail in cases where important regions lack strong gradients (e.g., smooth tumors or blurry boundaries). This reliance on hand-crafted edge cues limits robustness and introduces a potential mismatch between perceptual detail and clinical relevance. \n\n2. The main idea, adaptive patching of high-resolution images using quadtrees, has appeared in closely related work such as the Adaptive Patching Framework (APF) [1]. The differences made in this paper, such as reduced hyperparameter reliance and decoder-free inference, are incremental rather than fundamentally new. While the execution is well-done and the integration is clean, the conceptual novelty is moderate. \n \n---\n[1] Adaptive Patching for High-resolution Image Segmentation with Transformers (https://arxiv.org/abs/2404.09707)", "was_revised": true, "final_justification": "I am increasing my rating to 5: Accept.\n\nThe paper addresses an important challenge in applying ViTs to high-resolution medical images and proposes a practical, architecture-agnostic solution. While the core idea of adaptive patching is not entirely novel, the execution, particularly the decoder-free design, symmetric depatching, and consistent efficiency gains, is well-motivated and effective. The rebuttal further clarified robustness and comparative advantages, making this a solid contribution with practical value.", "rebuttal": "We sincerely thank you for your constructive feedback on our manuscript. Below, we address the points you have raised:\n\n## 1.Regarding Robustness and Potential Mismatch: \n\nThank you for this question.  The behavior of the tree-node splitting is indeed guided by the chosen score function, and its specific focus (e.g., edge detection) can influence the performance on downstream tasks. To transparently address this, we have expanded our appendix to include a detailed discussion of this behavior, including an analysis of potential failure modes and an examination of our method's limitations on a non-medical benchmark like ImageNet-1K.\n\nRegarding generalization to other domains, we conducted experiments on the widely available ImageNet dataset. It's important to note that SHF was designed for ultra-high-resolution medical images, so its performance on ImageNet was not designed to replicate SoTA's experimental results. Even so, we still used our own results and those from a public paper [1] for cross-comparison. As can be seen in the table below, SHF-ViT outperforms basic ViT due to its use of hierarchical information. However, if only a single tree is used, i.e., fixed hyperparameters are used, it will be weaker than basic ViT. This is because fixed hyperparameters can lead to generalization performance deficiencies.\n\n_Training on Imagenet common dataset to show the generalization.(300 epoch, top-1 accuracty)_\n\n| Model | Sequence Length | Validation Accuracy | V2 Accuracy |\n| :--- | :--- | :--- | :--- |\n| **ViT-B/16[1]** | 196 | 82.2% | 72.2% |\n| **Tree-ViT-B/16** | 256 | 77.3% | 65.1% |\n| **SHF-ViT-B/16** | 196 | 82.7% | 72.8% |\n| **ViT-L/16[1]** | 196 | 83.0% | 72.4% |\n| **Tree-ViT-L/16** | 256 | 77.8% | 65.7% |\n| **SHF-ViT-L/16** | 196 | 83.4% | 73.1% |\n\n\n[1] Touvron H, Cord M, El-Nouby A, et al. Three things everyone should know about vision transformers[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 497-515.\n\n## 2.Regarding Ablation Studies on Robustness: \nWe appreciate your valuable suggestion to demonstrate robustness. In response, we have performed a direct comparison against the Adaptive Patching (AP) method. A key motivation for developing SHF was to overcome the core robustness issues and significant hyperparameter sensitivity inherent in the AP approach.\n\nRather than merely fine-tuning hyperparameters, SHF introduces a more fundamental solution by reframing hyperparameter choice as a structured, hierarchical feature extraction problem. This approach is what enables the superior performance. We will add a table to the main paper showing the performance gains of SHF over AP, particularly at high resolutions and on the large-scale ImageNet-1K dataset. Furthermore, we will provide visualizations in the appendix that clearly demonstrate how different parameters influence the feature extraction process across the hierarchy.\n\n\nTo demonstrate the effectiveness of SHF over AP, in high reoslution images, the following table shows the \n_Speedup of SHF end-to-end training for PAIP dataset at the same segmentation quality as the baseline. We use the highest dice score of the baseline models SHF and AP._\n\n| **Resolution** | **Model-Patch** | **Sec/Image** | **Seq Length** | **Dice Score (%)** | **Speedup (Sec/Image)** |\n| :--- | :--- | ---:| :--- | :--- | :--- |\n| 16K 128 GPUs | **SHF-16** | 0.3672 | 2,048 | 76.89 | **4.47x** |\n| | AP-16 | 1.6421 | 2,116 | 77.43 | |\n| 32K 256 GPUs | **SHF-32** | 0.3826 | 2,048 | 76.08 | **5.64x** |\n| | AP-32 | 2.1567 | 2,116 | 76.13 | |\n| 64K 256 GPUs | **SHF-64** | 0.4013 | 2,048 | 75.33 | **9.87x** |\n| | AP-64 | 3.9614 | 2,116 | 75.32 | |\n\nAll these revisions will be incorporated into the final manuscript."}, {"review_id": "qUuBj6DKPq", "reviewer": "Reviewer_ptuA", "summary": "This paper introduces an efficient, tree-based, adaptive patchifier that uses larger patches in flat areas of the image and smaller patches (down to 2x2) in regions with many edges. This results in a significant speedup on high-resolution images while maintaining similar performance to the SAM baseline.", "questions": "- Include comparisons to additional large image segmentation frameworks, such as ViTMamba, xT, GLNet, and SGNet, to strengthen the experimental section.\n- Clarify how the adaptive patch sizes produced by the tree operator are processed by the vision encoder.\n- Consider providing pseudocode for Equation 3 to improve clarity.", "limitations": "It is unclear how the method would adapt to other domains than just medical images.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nUsing quad/octrees on image edges is an elegant approach to constructing an adaptive patchifier. The benchmarks show substantial improvements compared to ViT-based approaches, both in terms of performance and speed.\n\nThe decoder-less architecture offers additional speedup \"for free\" thanks to the use of small patches in important areas.\n\nWeaknesses:\n\nUsing edges to guide adaptive patching works well on whole slide images with large flat regions, but it is unclear how the method would adapt to noisy images or images with a lot of texture. For instance, could it generalize to satellite images?\n\nThe benchmarks lack comparison to other large image segmentation frameworks. This includes linear transform variants (as correctly identified in the related work): ViTMamba[1] or xT[2], as well as more specialized approaches such as GLNet[3] and SGNet[4].\n\nIt is unclear how the different patch sizes generated by the tree operator are passed to the vision encoder.\n\nClarity:\n- Similarly, how are 1024 patches embedded in Table 4: is it only a conv2d, or is downsampling used?\n- Equation 3 is not clear; pseudocode would probably be more helpful.\n- Table 3: \"[16, 2562, 4, 0962]\" is very hard to read.\n\n- [1] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision Mamba: Efficient visual representation learning with bidirectional state space model.\n- [2] Ritwik Gupta, Shufan Li, Tyler Zhu, Jitendra Malik, Trevor Darrell, and Karttikeya Mangalam. xT: Nested tokenization for larger context in large images.\n- [3] Wuyang Chen, Ziyu Jiang, Zhangyang Wang, Kexin Cui, and Xiaoning Qian. Collaborative global-local networks for memory-efficient segmentation of ultra-high resolution images.\n- [4] Sai Wang, Yutian Lin, Yu Wu, and Bo Du. Toward real ultra image segmentation: Leveraging surrounding context to cultivate general segmentation model.n important areas.", "was_revised": true, "final_justification": "I believe most of the concerns raised by other reviewers and myself have been addressed:\n- Clarity: The authors propose to incorporate pseudocode and Python snippets.\n- Robustness/generalizability: The authors evaluated their approach on entirely different domains, demonstrating similar gains.\n- Evaluation against more recent competitive models: U-Mamba and HiFormer have been added to the evaluation.\n\nSubject to the promised revisions, I see no further grounds for rejection and recommend acceptance.", "rebuttal": "We sincerely thank you for your constructive feedback and valuable suggestions.\n## 1. Regarding Comparisons to Other Frameworks: \n\nThank you for bringing these reference works.  We will incorporate the suggested work by Chen et al. [1] into our \"Related Work\" section. Furthermore, we will add the following comparison table (with U-mamba [1][2], a SoTA variant of ViTMamba that is in particular tuned for high resolution medical imaging) to the manuscript to clearly delineate the significant distinctions between our approach and other relevant frameworks.\n\n_Segmentation of BTCV for multi-organ segmentation and KiTS19 for Kidney Tumor Segmentation on a single GPU. *Time* indicates the end-to-end runtime to achieve the corresponding Dice Score._\n\n| **Dataset** | **Model** | **Patch Size** | **Time** | **Speedup (x)** | **Dice Score (%)** |\n| :--- | :--- | :---: | ---:| :---: | :---: |\n| BTCV [landman2015miccai] | U-Net [ronneberger2015u] | N/A | 843.90 Seconds | 9.04x | 80.2 |\n| | U-Mamba [1][2] | N/A | 8,016.24 Seconds | 0.95x | 83.51 |\n| | SAM2 [ravi2024sam2] | 4³ | 7637.28 Seconds | 1.0x | 82.77 |\n| | **SHF-SAM2** | **2³** | **1067.88 Seconds** | **7.15x** | **89.71** |\n| KiTS [Heller2019kits19] | U-Net [ronneberger2015u] | N/A | 243.7 Minutes | 1.99x | 83.23 |\n| | U-Mamba [1][2] | N/A | 969.0 Minutes | 0.50x | 86.22 |\n| | SAM2 [ravi2024sam2] | 8³ | 483.1 Minutes | 1.0x | 81.35 |\n| | **SHF-SAM2** | **2³** | **187.3 Minutes** | **2.58x** | **87.25** |\n\n[1] Isensee, Fabian, et al. \"nnu-net revisited: A call for rigorous validation in 3d medical image segmentation.\" International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2024.\n\n[2] Ma J, Li F, Wang B. U-mamba: Enhancing long-range dependency for biomedical image segmentation[J]. arXiv preprint arXiv:2401.04722, 2024.\n\n\nA key differentiator of our work is its application to medical image segmentation at an ultra-high resolution of 64K. In contrast, many existing methods, including some of those referenced, were primarily evaluated on tasks with lower resolutions (e.g., 4K). We hypothesize that scaling these existing approaches to the extreme computational demands of 64K resolution while maintaining performance would be a significant challenge.\n\n\n## 2. Regarding Clarification of Patch Size and Pseudocode: \n\nTokens of different patch sizes generated by SHF will first be downsampled to the same size, and the subsequent process is the same as standard ViT.\n\nTo thoroughly clarify our methodology, we will enhance the appendix by including both detailed pseudocode and illustrative Python code snippets. We believe this will provide a comprehensive and unambiguous explanation of our implementation.\n\n## 3. Adapt to other domains than just medical images:\n\nYes, SHF can be adapted to other domains. By the time we finish this paper, we focused on the medical imaging domain. Here is a result from the high-resolution microscopy X-Ray Computed Tomography (XCT) projections and reconstructions task. The sample was collected from the national highway network of the Kansai urban region of western Japan. These samples include delamination, cohesive failure, insufficient asphalt coverage around aggregates, and cracking of the aggregates themselves. Each resampled sub-volume is randomly cropped along the z-axis to a depth between 50 and 120 slices, resulting in 3D volumes of approximately 8K² pixels per slice.\n\n_Segmentation of simulated XCT dataset for multi-classes segmentation at the fine-tuning stage._\n\n| **Datset** | **Model** | **Patch Size** | **GPU (hours)** | **Epochs** | **Dice (%)** |\n| :--- | :--- | :---: | :---: | :---: | :---: |\n| 780 unique volumes | U-Net [ronneberger2015u] | N/A | 1,280 | 500 | 58.38 |\n| | Swin UNETR [isensee2024nnu] | 256² | 5,120 | 1,000 | 63.74 |\n| | SAM 2 [ravi2024sam2] | 128² | 5,120 | 1,000 | 85.98 |\n| | **Our Model** | 2² | 5,120 | 1,000 | **94.79** |\n\n\n\nRegarding generalization to the standard normal resolution benchmark, we conducted experiments on the widely available ImageNet dataset. It's important to note that SHF was designed for ultra-high-resolution medical images, so its performance on ImageNet was not designed to replicate SoTA's experimental results. Even so, we still used our own results and those from a public paper [1] for cross-comparison. As can be seen in the table below, SHF-ViT outperforms basic ViT due to its use of hierarchical information. However, if only a single tree is used, i.e., fixed hyperparameters are used, it will be weaker than basic ViT. This is because fixed hyperparameters can lead to generalization performance deficiencies.\n\n_Training on Imagenet common dataset to show the generalization.(300 epoch, top-1 accuracty)_\n\n| Model | Sequence Length | Validation Accuracy | V2 Accuracy |\n| :--- | :--- | :--- | :--- |\n| **ViT-B/16[1]** | 196 | 82.2% | 72.2% |\n| **Tree-ViT-B/16** | 256 | 77.3% | 65.1% |\n| **SHF-ViT-B/16** | 196 | 82.7% | 72.8% |\n| **ViT-L/16[1]** | 196 | 83.0% | 72.4% |\n| **Tree-ViT-L/16** | 256 | 77.8% | 65.7% |\n| **SHF-ViT-L/16** | 196 | 83.4% | 73.1% |\n\n\n[1] Touvron H, Cord M, El-Nouby A, et al. Three things everyone should know about vision transformers[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 497-515.\n\nAll these revisions will be incorporated into the final manuscript."}]

Author rebuttal:

Meta-review: Initially, reviewers found our Symmetrical Hierarchical Forest (SHF) framework elegant and practical but expressed concerns on the robustness and generalisation of the proposed work. After rebuttal, all the reviewers have been convinced by the authors' feedback and gave all positive ratings. The meta reviewer agrees with the recommendation of the reivewers and makes the final decision.
Decision: Accept (spotlight)