Distributed Parallel Gradient Stacking(DPGS): Solving Whole Slide Image Stacking Challenge in Multi-Instance Learning
Authors: Boyuan Wu
ZEFENG WANG
Xianwei Lin
Jiachun Xu
Jikai Yu
Zhou Shicheng
Hongda Chen
Lianxin Hu
Venue: ICML 2025 poster (2025)
Keywords: Multi-Instance Learning ，Distributed Training，Gradient Compression，Whole Slide Image Analysis，Medical Image Classification
Primary area: applications->health_medicine
TL;DR: We propose DPGS-DMGC, the first parallel MIL framework enabling gradient-stacked training for non-stackable medical data, achieving 31× speedup, 99.2% smaller models, and +9.3% accuracy.

Abstract: Whole Slide Image (WSI) analysis is framed as a Multiple Instance Learning (MIL) problem, but existing methods struggle with non-stackable data due to inconsistent instance lengths, which degrades performance and efficiency. We propose a Distributed Parallel Gradient Stacking (DPGS) framework with Deep Model-Gradient Compression (DMGC) to address this. DPGS enables lossless MIL data stacking for the first time, while DMGC accelerates distributed training via joint gradient-model compression. Experiments on Camelyon16 and TCGA-Lung datasets demonstrate up to 31× faster training, up to a 99.2% reduction in model  communication size at convergence, and up to a 9.3% improvement in accuracy compared to the baseline. To our knowledge, this is the first work to solve non-stackable data in MIL while improving both speed and accuracy.

Distributed Parallel Gradient Stacking Solving Whole Slide Image Stacking Challenge in Multi-Instance Learning
Boyuan Wu
Zefeng Wang'
Xianwei Lin
Jiachun Xu Lianxin Hu
Jikai Yu
Shicheng Zhou
Hongda Cheng'
Abstract
Different length MIL Data
A
Model
Optimizer
Whole Slide Image (WST) analysis is framed as Multiple Instance Learning (MIL) problem, but existing methods struggle with non-stackable data due to inconsistent instance lengths, which de- grades performance and efficiency. We propose Distributed Parallel Gradient Stacking (DPGS) framework with Deep Model-Gradient Compres- sion (DMGC) to address the problem. DPGS en- ables lossless MIL. data stacking for the first time, while DMGC accelerates distributed training via joint gradient-model compression. Experiments Camelyon16 16 and TCGA-Lung datasets demon- strate up to 31x faster training and maximum 9.3% accuracy improvement compared to base- line. To our knowledge, this is the first work to solve non-stackable data in MIL while improving both speed and accuracy.
Updatx
Gradion 1
Clauitier
Legits
Loss
Function
Different length MIL Data
Model Pool
DMGC
Manager
B
Sparse
Model I (GPU
L.Lggts
Gradiant Pood
Loss
Function
Global Model
Weight
(GPt)
DMGC
Optinieer
Figure I. A: Traditional MIL methods cannot stack data due to the varying lengths the data. B: The proposed framework paral- lelizes the MIL models and stacks the gradients of these models, thereby simulating data stacking and addressing this issue.
Shao et al. (021)) leveraging Transformers to model in- stance correlations, exemplify high-accuracy solutions in this domain. However, these methods are unable ad- dress the inherent non-stackability of MIL data. Due the variation in the number of instances per bag (ranging from hundreds to thousands), conventional approaches are unable to stack bags of different lengths into batches, necessitat- ing sequential processing of individual bags, which brings two critical bottlenecks: (1) Low training efficiency: The inability to utilize GPU parallelism leads prohibitively long training times for large-scale WSI datasets Wen et al.) (2025 Bailly et al. (2022). (2) Unstable gradient estimation: Sequential gradient updates rely on single-bag statistics, introducing bias across non-identically and independently distributed (non-IID) bags and hindering model convergence Li et al. (019). These two problems severely restrict the performance and scalability of MIL models.
1. Introduction
Multi-Instance Learning (MIL) has become essential in Whole Slide Image (WSI) analysis for pathological diag- nosis, it can handle bags containing hundreds to thou- sands of patch instances. WSIs, with ultra-high resolutions (> 1-0,0000x1000 10,,00 pixels) and sizes up to several GBs, offer rich histological information but introduce significant computational challenges, especially in cancer subtype clas- sification. MIL addresses this by dividing WSIs into patches, selecting informative instances form variable-length bags, extracting instance-level features, aggregating them into bag-level embeddings, and performing classification. MIL frameworks such as CLAM et al. (2021), which identifies critical regions via interpretable heatmaps, and TlansMIl.
'School of Information Engineering, Huzhou University, 313000 Huzhou, China "College of Engineering, Carnegie Mellon University, 15213 Pittsburgh, United States America. Corre- spondence to: Zefeng Wang <zefeng, wang @ zjhu.edu.cn>.
To tackle these challenges, we propose Distributed Parallel Gradient Stacking (DPGS) and Deep Model-Gradiess Com- pression (DMGC). As shown in Figure 1, DPGS assigns variable-length bags to submodels for parallel gradient com- putation and aggregates gradients to emulate batch stacking without changing the architecture. DMGC compresses gra-
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Distributed Parallel Gradient StackinggDPGGS::Solvvin Whole Slide Image Stacking Challenge in Multi-Instance Learning
dients and parameters jointly, leveraging sparsity update only sparse parameters-reddeee communication by 99.2% at a 99, 99.99% discard rate while maintaining convergence. On Camelyon16 and TCGA-Lung DPGS+DMGC improved training speed by up to 31x and accuracy by up to 9.3% over the unstacked MIL baseline.
stacking, reducing computational efficiency, destabilizing gradient estimation, and ultimately degrading convergence speed and model accuracy.
2.2. Parallel Computing and Distributed Training
Distributed deep learning employs parallelization to address
The main contributions of this study are as follows:
efficiency and scalability challenges |Aill et al. 20118); Dai et al. (2019); Langer et al. (2020). Data parallelism remains the mainstream strategy: Goyal et al. (017)) achieved 90% scaling efficiency across 256 GPUs via linear learning rate scaling. DistBelief et al. (2012) confirmed feasibility for large-scale applications. For models exceeding single- device memory, model parallelism Rasley et al. (2020) par- titions parameters across devices. Hybrid methods, such as Megatron -LM Narayanan et al. (2021), combine tensor, pipeline, and data parallelism (PTD-P) to sustain through- put under memory constraints. Decentralized approaches Lin et al. (2017) address privacy concerns via asynchronous updates and the removal of parameter servers Recht et al. 2011)
Distributed Parallel Gradient Stacking for MIL: The first algorithm enabling efficient, lossless MIL data stacking with multi-GPU(or single-GPU)/ multi-node training support.
Deep Model-Gradienn Compression: A novel compres- sion strategy jointly optimizing gradient and model parameter sparsity, achieving up to 99.2% communica- tion reduction while maintaining convergence.
Thoroughly validated on public WSI datasets (Came- lyon16, TCGA-Lung) and distributed learning envi- ronments, offering an efficient solution for large-scale MIL training.
However, existing frameworks are ill-suited for MIL. Al- though DistBeliei Dean et al. (012)) Megatron- LM Shhecyy et al. (2019), and DeepSpeed Wang et al. (2024a) excel in LLMs and standard tasks, their Transformer centric designs and batch-based assumptions misalign with MII., While P-MIL Xu al. (2017) and MIBP al, (2012) offer tai- lored solutions, they lack neural MIL compatibility or adapt- ability to modern models. This incompatibility brings two challenges: (1) architectural mismatch with MIL's instance- level interactions, and (2) inefficient use of resources due to variable-length bag- bag-instance structures that break batch uniformity.
The paper's organization is as follows: Section II reviews MIL methods, distributed training. and gradient compres- sion; Section III details DPGS and DMGC design with mathematical proofs; Section IV evaluates the framework on multiple WSI datasets across varied configurations; and Section V concludes with future research directions.
2. Relate Work
2.1. Multi-Instance Learning
MIL. provides framework for analyzing complex data structures containing multiple instances. Existing MIL paradigms typically partition WSIs into localized instances and perform bag-level predictions by aggregating instance- level features. Attention- based MIL methods leverage at- tention mechanisms for precise feature aggregation, demon- strating strong performance in medical imaging applications Shao et al.) 2) Zhuu et al.) (2024); Lu al. (2021); Qu et al. (2024), Tang et al. (2024), Wang et al. (2024b) lsse et al. 2018). llse et al. (2018) pioneered an attention-basee
2.3. Gradient Compression
Gradient compression techniques reduce communication overhead in distributed training through sparsification Abbaa hamyan et al. (2021): ; Chen et al.) (2018): : Dryden et al. (2016); Wang al. (2020), quantization, low-rank approxi- mation, and error compensation. Top-k sparsification Aji & Heafield 2017) and Deep Gradient Compression (DGC) Lin et al. (017)) achieve high compression ratios by preserving critical gradients. Quantization methods such as 1-bit SGD Seide et al. (2014) and QSGD Alistarh et al. (2017) balance communication cost with model fidelity. Low-rank approxi- mation Vogels al. (2019) reduces transmission dimensions via matrix factorization, while error feedback mechanisms Karimireddy et al. (2019) preserve convergence by accumu- lating untransmitted gradients. Hybrid approaches like DGC Lin et al. (0177) combine sparsification with momentum correction to maintain accuracy under extreme compression.
MIL architecture that improved both accuracy and inter- pretability on biomedical datasets. [Tang et al. (2024) in- troduced pseudo-bag construction and dual-layer attention mechanisms to mitigate data scarcity and overfitting, en- hancing model generalizability. Graph-based approaches Zheng et al. (2022); Li et al. (2024); Chan et al. (2023); Hou et al. 2022); Zhao et al. 2023) further model instance interactions using graph convolutional networks, achieving ofat--of-the-ae results in cancer subtyping and diagnosis. However, these methods do not address a key MIL lim- itation: variable bag lengths hinder efficient mini-batch
Despite these advances, gradient compression remains largely unexplored in MIL. contexts. Current MII. method-
2

---

Distributed Parallel Gradient Stackinng Whole Slide Image Stacking Challenge in Multi-Instance Learning
Different length Bag
Subprocess
Parallel Gradient Stacking for Multi-Instance Learning
Data
Main Process
Model Copy
Model
Weight queue
Copy
Copy
Offine
Feature Extractor
Inir Copy
Delta model
Loss Function
Updatee
Verifleation process
Label
Global Model
Sum Up
Gradient
Nut updated
Thread
Thread
Thread
Grudient of Residual
Discarded Gradient
k% Gradient
Update
Global optimizer
Sorted Sum Gradient
Clip
T
Gradient quene(Sparse compression)
Gradlent Pool
Figure 2. The proposed DPGS framework begins by parallelizing models and feeding variable-length data packets for gradient computation. Gradients from parallel models undergo DMGC compression prior to transmission the master server. The master server aggregates gradients, updates models, applies DMGC compression to weights, and returns them to training servers. Red-numbered annotations in the figure depict the training loop process, with model replication occurring solely during initialization. Multi-threaded validation runs concurrently with the master process.
ologies typically operate in non-distributed settings, over- looking the optimization potential of gradient manipulation in distributed environments.
efficiency of parallel computation and model training. For example, consider two bags:
X1 {11.1. T1.4,
(I)
X₂ {I2.1 22.2 ................................ 22
(2)
3. Method
where I1 * Iz. As a result, X, and X₂ cannot be directly stacked into batch tensor X, presenting challenges for par- allel computation and accelerating training in MIL models.
This study proposes gradient-based approach to address MIL sample stacking with inconsistent bag lengths. A dis- tributed gradient aggregation architecture enables parallel processing, improving stacking efficiency and accelerating training. The proposed DMGC reduces bandwidth via dual sparsity (gradients weights) and improves convergence. These innovations resolve ntn-stackable data issues and en- hance WSI model performance. See Figure and Algorithm 1 for details.
3.2. Distributed Parallel Gradient Stacking
Given model parameters ((0), a sample set is distributed across P sub-processer M for parallel computation. Each sub-process processes sub-sample Xp, sub-label Lp, com- putes loss Lp and gradient VoCp. sends the gradient the master, and clears its local state without updating. The mas- ter using stacking function S(.), updates the parameters, and broadcasts the updated ((++1), The overall training process DPGS(00)) X, P,n, M) is described as:
3.1. Preliminaries
In MII,, datasets typically consist of multiple bags, with each bag containing several instances. A dataset can be represented as = {(X,, Ly) where X, = H represents the j-th bag, which contains I, instances, and L,j the label of the bag. Here, T denotes the i-th instance of the j-th bag. The number of instances may vary across different bags X. This inconsistency in bag length makes it challenging to stack the instances of multiple bags into unified tensor during the training process, thus affecting the
P
1 g(++1) = =g(() -n P (o,
xp), (3)
where the gradient stacking function S(-) is defined as:
(VoLi, VoL2. VoLp) = 1 Σ VoC, (4)
p=1
3

---

Distributed Parallel Gradient Stackinng Whole Slide Image Stacking Challenge in Multi-Instance Learning
3.2.1. EQUIVALENT DERIVATION OF DPGS AND MINI-BATCH
Algorithm DPGS Algorithm with DMGC
Input:
X Input data samples
In DPGS, each of the P subprocesses handles bag Xp and computes the loss Ep and its gradient VeCp. If the loss function is an average, the main process averages all subprocess gradients to obtain the global gradient:
Y: Ground truth labels for the training data
P: Number of subprocesses for parallel computation
1. : Learning rate
M Model function used for training
k: Top-K ratio for gradient compression
P Σ
thr Gradient clipping threshold
V Latcked
V Cp
(5)
Initialize parameters: (()), @Comp, resp
p
Training Service:
while not converged do
DDMGC +- Decomppess
In traditional mini-batch training using the mean-based loss function, the P samples {X1, .., } a single mini-batch B, and the total loss is computed as:
((t) + 0 1) + DDMMGC
are stacked into
resp +-
for each p p= P do
X, + Subset of X
P
Gp t VoL(M (Xp), Xp), Goomb Gp + TCSp for each E len (onnGoomm do
battch p
Σ
Cp
(6)
p=1
Uhr, + 4% of GGombl
Then, the mini-batch gradient calculated:
mask + p Grombl
thrp p
Gmask P
mask
P
P
respli] + Grombli end for
Gmakk
CoChachh = Vo
P
P
Cp
=
P
V
cCp
(7)
p
p=1
p=1
GDMGC Clip(Gp Gymaask tthrdii))
GComp + Compress(GDMGG
By comparison, we find:
Send GComp to master
Gp +0
end for
VoCratch = VoLunced
8
Wait for synchronization with Master Service end while
Thus, these two methods are equivalent in model training. After obtaining the gradients, we update the model weights:
Master Service:
while not converged do
for each p p= p= P do
Receive GComp from Training Service GDMGC + Decompress (Comp)
1 ( g(++)= 0(10 -7
V/Co p=1
(9)
end for
Galobal + 1) 1 +p=iGhaee
GDMGC
((++1) +- g()) - Gglobal
Here, we use SGD for the mathematical equivalence deriva- tion, but in practice, any optimizer can be used to update the model.
ODMGE + (((+1) - (1t)
@Comp + Compress(@DMM)) Send @Comp to all Training Service
Check convergence condition
3.3. Deep Model-Gradient Compression
end while
Distributed training incurs high bandwidth usage due gra- dient and weight transmission Dean et al. (2012). DGC Lin et al. (2017) addresses this by sending only targe-magnitude gradients and accumulating residuals, achieving dual spar- sity. We propose DMGC, an improved version of DGC that transmits weight deltas instead of full weights (Algorithm 1). further reducing bandwidth without sacrificing training stability.
3. VIRTUAL BATCH IN DMGC
DMGC and related locally accumulated gradient compres- sion methods (c.g., DGC) transmit only gradients exceeding a preset threshold, while accumulating sub-threshold up- dates locally until they surpass it, drastically reducing com- munication bandwidth. Through multi-round accumulation,
4

---

Distributed Parallel Gradient StackinngDDGGS):Skkvkinn Whole Slide Image Stacking Challenge in Multi- Learning
this process is mathematically equivalent to using larger batch sizes. Within the DPGS framework, the learning rate can still follow linear scaling rules despite gradient accu- mulation. The following derivation clarifies this behavior under DPGS.
due the inefficiency of processing variable-length bags.
DPGS enhances efficiency via gradient stacking and dis- tributed computation. Each epoch processes P bags in [N/P] rounds. Each round includes: 1. Parallel forward- backward passes synchronized by the slowest gradient com- putation, costing max forward + Thackward). dient aggregation, costing Toomm (1/Crano) + T Tupdate- 3. Global parameter updates, fixed at Tupdate
Consider the update process for a specific component (6+) of the model parameters 9. Assume there are P subprocesses, each processing batch X, and computing its correspond- ing loss Lp and gradient VoLp. After introducing DMGC within the DPGS framework, only gradients that exceed the threshold are transmitted, while other gradients are accu- mulated locally in the subprocesses. After local gradient accumulations, a global update is performed, and the mathe- matical expression for this is:
Sparse gra-
The parallel time complexity is:
N (maaxTiid + Towa) + Tc/C,++ + Ta). (12)
T parallet = p
The theoretical speedup ratio is approximately:
P
P
0 b4r 0000 TT
Σ
V (1) c((tt)
(10)
(13)
PdT
I+a
where
Here, 1) represents the learning rate, is the number of subprocesses, b is the batch size each subprocess handles (in MIL, b- 1), and I' the gradient accumulation inter- val. This equation indicates that within a gradient update interval of T, each subprocess accumulates gradients from T iterations, which are then applied to single global up- date. Originally, the effective batch size per iteration is P. b, but after T accumulations, is equivalent to expanding the batch size to P b T. Therefore, each global update implicitly uses a larger virtual batch size of b.
Teomm M - (1/Cratio) ) + Tuplate (14)
max(Trorwad: T baakwardd)
with near-linear scaling when a < 1.
This analysis solely addresses per-iteration time reduction. Actual training duration depends on convergence speed. While increasing effective batch size via gradient accumula- tion (e.g.. P in DPGS) may reduce noise and accelerate convergence, excessive noise suppression risks suboptimal training, causing empirical rates to deviate from linear scal- ing. For experimental results, please refer to 4.2.2
In large-batch training, learning rate scaling is typically required for stability. However, as shown above, the learning rate 1) x T and the scaled batch size P.b-T effectively cancel out. Thus, methods based on local gradient accumulation (c.g., DMGC) inherently follow the linear scaling rule Goyal al. (2017), requiring no significant adjustment to maintain stability and convergence.
4. Experiments
4.1. Data
Table 1. Original data distribution of the datasets employed. No- tably, the Camelyon 16 test set utilizes an officially predefined split, comprising 38. .2.2%% tumor class and 61.7 %% normal tissue class. In contrast, the TCGA-Lung dataset was randomly partitioned with an 8:2 training-to-tee test ratio. IFL:Instance Feature Length
3.4. Time Complexity Analysis
We analyze the computational efficiency of the DPGS frame- work by comparing time complexities with conventional se- quential training, noting that reduced time complexity does not directly imply faster end-to-end training since gradient quality and optimizer dynamics also influence convergence rates.
CLASSIFICATION
TYPE
TYPE II
IFL
C16 MULTICCLLE
160 40.11%
239 59.89%
512
C16 IMAGINET
160 40.11%
239 89%
256
Let N be the number of bags, with Torward Thacwaadd and Tupiase denoting the per-bag times for forward, backward propagation, and parameter updates. Communication over- head is Tromm & M (1/ /rrann)), where M is the number of parameters, and Cratio is the gradient compression ratio.
TCGA-LUNG MULTISCALE
534 51.05%
512
98.9
1024
TCGA-LUNG UIN2
534 51.05%
512 48.95%
1536
Experiments two public WSI datasets, using varied fea- ture extraction strategies, effectively yielded four datasets for analysis: Camelyon16 16 (C16) with single-scale ResNet50 (ImageNet- )retraind)) and multiscale features, and TCGA- Lung with multiscale features obtained from DS-MIL Li et al. (202T) and single-scale features extracted by UIN2.
Sequential training's total time is:
Tirrill = N (I forward + Thakkward + T )
(11)
5

---

Distributed Parallel Gradient Stackingg Whole Slide Image Stacking Challenge in Multi-Instance Learning
Table 2. Experiments compare MII. methods' accuracy/convergence time with/without DPGS on C166Multiscale,, C16 ImageNet, and TCGA-Lung Multiscale, demonstrating DPGS's dual improvement in accuracy and convergence speed. Table annotations: indicates model covvevnveeee (accuracy equals original data distribution, no convergence time recorded)" while "B" specifies batch size achieved performance levels.
C16 MULTISCALE
TCGA-LUNG MULTISCALE
C16 IMAGENET PRETRAIN
FRAMEWORK
METHOD
ACCURACY
TIME (S)
ACCURACY
TIME (S)
ACCURACY
TIME (S)
ABMIL Dav AL SOIE
91.47%
405.5 (0_16)
92.38% +0.25
636.3 (0-22)) (3-.3(0-222)
75.05% +1:22
1 1177.2 (9_110) +38.5
MEANMIL (SAA
61.71%
92.23% 20.40
883.1 (4-52) 2007
61.71%
CLASSIC
TRANSMII
93.80% 10.32
304.2 (B-te) _6.1
92.39% 10:92
397.5 (B=sz)
1178
80.48% 1
1163.1 (8-10)
1152. (71210) 116.6
CLAM-ME ME
93.02% $0.95
499.3 (M-10) +19.s
92.60% +o..9
503.2 (0-00)
78.91%
+3.00
ACMIL DHANO ET AL
2014
94.40%
20.04
953.4
(R-10) 4e5.7
93.13% +:0.4
711.2 (R-az))
2e 28
81.31%
2735.3 (R=10) #40.2
RRTMIL
TANG BT AL. 5004
94.57% +0.85
2462.1 (0-36)
+10.s
93.26% +0342
1923.8 8
(0
+41.4
82.53%
+0.31
3099.1 (#-11) #21.8
ABMIL
92.23% 10.54
1512.8
(8-10)
1.00
91.43% 1.0.36
1465.0 (0-att)
1.90.
82.48% 13.70
9040.5 (R=10) 1
MEANMIL
61.71%
90.00% +o.ss
1
(0_32)
61.71%*
PADDING
TRANSMIL
94.04% 10.34
2912.8
100.3
93.82% 20.00
1054.1 (8_)
1:1.3
84.38%
2:222
9754.0
(R-10) 2200.1
CLAM-MB
93.80% #0.42
2737.1
(4:30)
#54.7
92.00% +0.st
1063.2 (0
#21.5
83.28%
+:130
(1::36) 9700.
+194.0
ACMIL
92.63%
798.6 6
93.33% 10:47
2077.8 8
84.77%
1:030
9552.4
(8-10) 1108.7
RRTMIL
93.02% 10.00
1299.3
(8-10)
1.39.
93.62%
1.072
2872.1 1 (Rmd8)
1.5.22
84.90%
1,0211
10023.1
(R-16) 2192.1
ABMIL
84.58% +1.58
3811.9 (7--)) #25.1
92.45% +0.19
3564.7 (0-6)
+18.8
73.84% +1.00
22054.2 (R-1) +101.0
MEANMIL
61.71%
87.46% #5:22
4016.4 (0mm))
430.3
61.71%
SAMPLING
TRANSMIL
89.93%
3721.6 (8-4)
92.33% 13.70
4316,3 (B-x)
79.89%
30281, . (Bhm4 1
CLAM-MB
86.24% 13.92
3012.7
(7-1) 22:1
92.24% 23:32
4416.2 (R=8)
190.1
75.15% 2:4.5
28231,3 (R-() 1:21.1
ACMIL
91.86% +3.25
1088.8 (0:24)
exus
93.34% +x002
6461.2 (0:a)
#00.1
84.62% +3.01
20112.2 (M::4) #25L.
MRTMIL
91.74%
6050.6 6 (1-1)
93.66% 13:01
6901. 2 (1-5)
84.79% 14.00
32211. (R-1) 1:15
ABMIL.
94.04% % 10:34
29.3 (9m.00 1o.
93,90% 10.13
20.0 (8-02)
84.38%
11:01
92.4 (8-e)
MEANMIL
71.33% 20.02
104.1 (0-16) m9.1
92.23% +0.32
14.1 (71))
+2.
65.63%
+0.90
122.2 (P-16)
=19.1
DDGC+
TRANSMIL
95.23% =0.22
115.2 (8:11) #12.
94.70% #0.41
48.6 (4:)2))
45.9
85.97%
1:449
158.1 (Ah=m) +16.2
CLAM-MB
93.82% =0.54
28.1 (ML-6)
+2.
93.23%
+0.15
40.1 (0_10)
#7.0
83.59
+1.25
109.3 (SILL)
=9.2
ACMIL
95.87% % 20:41
49,9 (B-16)
94.79%
40.09
67.3 (5-32)
87.71%
20.02
61.2 (B--2) 1:7.9
RRTMIL
95.97 1.079
615.7 (B-10)
116.5
94.90% 10..s
570.1 (B-52)
192.1
87.99%
710.2 :B-16) _.6.1
Multiscale features were derived from multiple magnifica- tions (e.g.. 20x, 5x) ) and concatenated into feature pyra- mid. This approach preserves tissue details across scales while optimizing feature usage via local attention, enhanc- ing both classification and lesion localization. TCGA-Lung UIN2 refers to features extracted by the UIN2 feature ex- tractor on the TCGA dataset [Chen et al. (2024). As shown CChen
As shown in Table 2, DPGS boosts ABMIL accuracy on TCGA-LUNG from 92.38% 93.90% and speeds up con- vergence by 31.8x (636.3s to 20.0s). On Camelyon16- Multiscale, MeanMIL improves by 9.62% (61.71% to 71.33%). All baselines show accuracy and convergence gains, highlighting DPGS's generalizability. Table 3 con- firms further performance improvements with the modern feature extractor UN12 Chen et al. (022))Furthee evidence of the generalisability of our approach.
in Table 1, the scale, distribution, and feature diversity of these datasets provide robust benchmark for evaluating model robustness.
4.2. Results of DPGS
Table 3. Performance of TCGA-LUNG Features Extracted by UN12 on ABMIL
4.2.1. DPGS VS PADDING VS SAMPLING VS CLASSIC
Dataset
TCGA-UN12
This study evaluates the DPGS framework and DMGC through controlled experiments using three datasets from Section 4.1. Metrics include classification accuracy and con- vergence time, defined per Jaranii & Bijarchi (2024) as the point when total loss remains below the 95th per- centile across trials. Our experiments compare three base- lines: 1. Classic 2. Padding: Applics zero-padding to shorter bags for uniform length. 3. Sampling: Randomly selects fixed number of instances per bag.
Framework
ACC
Time
Classic
95.30% 0.71%
895.96 (B=1)
1. 25.76
Padding
96.10%
0.65%
2767.9 (B=32)
1: 54.42
Sampling
95.12% +: 0.11%
233.75 (R-32)
17.25
DPGS+DMGC
96.90 % +: 0,31%
32.10 (B=32)
+ 4,31
Further analysis reveals Bag Padding's 's limitations: while achieving a 1.37%-7.43%% accuracy boost on Camelyon16- PaageNet-Pretraiin it suffers a 3.2x8.4x slower conver- gence compared to other baselines. We attribute this to: (1) excessive zero-padding lowering information density
Experimental Sctup: 4 nodes, 10% DMGC retention, 4XNVIDIA V100 GPUs (32GB), 1000 Mbps bandwidth (non-parallel baselines use a single GPU).
6

---

Distributed Parallel Gradient StackinggDDGGS):Skkiiin Whole Slide Image Stacking Challenge in Multi- Learning
and complicating convergence, and (2) data dimensional ex-
60
pansion increasing computation and memory costs. Table 2 confirms DPGS's 's dual advantage over Bag Padding in both
Keep Rate 10%
Keep Rate 1%
accuracy and convergence, validating its optimized training dynamics and improved generalization.
Time (S/10Epoch)
Keep Rate 1
40
For Sampling, although this approach accelerates conver- gence, reduces classification aceuracy and increases vari- ance (see supplementary table). This may compromise data integrity- mathematically distinct from standard minibatch sampling-as it violates positive bag completeness and risks omitting critical instances, especially in larger batches Shap- cott et al.] (2019). In contrast, our method achieves a better trade-off between efficiency and accuracy.
30
20
10
2
3
4
5
7
Number of Nodes
Figure 3. Synchronous parallel algorithms face scalability plateaus due to communication and synchronization overheads. While expanding nodes reduces training costs, this gain is offset by in- creased communication. Higher gradient compression shifts the plateau inflection point to larger scales. In addition, as per Equa- tion 12, larger datasets prolong computational benefits from node scaling under fixed communication costs, delaying the plateau and raising the bottleneck inflection point.
Notably, variations in bag length (57k/105k for C16 Multi- scale/ImageNet) significantly impact processing speed, ex- plaining the latency observed with Bag Pooling. Dataset size also affects the optimal batch size (TCGA=32 vs. C16=16), highlighting the importance of tuning hyperparam- eters based on scale. Differences in method performance become less pronounced with simpler features but more evident as feature complexity increases.
As demonstrated in Table 4, it is important to recognize that the proposed methodology is not limited to multi-node dis- tributed systems; it can also be implemented single-node computing devices. By using multi-process virtualization to simulate nodes, DPGS can operate in distributed manner on single device. Furthermore, the improved communica- tion efficiency within a single machine's memory can lead to higher computational efficiency.
4.2.2. THE NODE EXPANSION EFFICIENCY OF DPGS
This study provides theoretical and empirical analysis of scalability in distributed training. As shown in Figure 3, with fixed batch size (B = 8). DMGC keep rates (10% 1%, 0.1%), and 1000 Mbps bandwidth on the Cl6-Multiscale dataset, ABMIL training time decreases nearly linearly as node count increases (K I 1-8), but shows significant non- linear reduction beyond K 4 (at 1% keep rate). Aligned
4.2.3. BATCH SIZE GAIN FOR DPGS
Table 4. A comparison of ABMII. performaace in single- vs. multi- machine training shows that the method supports single-GPU op- eration via multi- process parallelism (virtual nodes), maximizing resource use without requiring multi-GPU setups. Intra-device communication improves efficiency, leveraging memory band- width that exceeds network limits.
This study further investigates the regulatory mechanism of batch size on MIL model performance. Under tradi- tional non-parallel frameworks constrained by hardware limitations that preclude batch stacking, we systematically analyze the dynamic relationship between batch scale and model efficacy within the DPGS framework by varying gradient stacking magnitudes under identical experimental configurations.
Dataset
C16-Multiscale
Configure
4 Nodes on 1 GPU
4 Nodes on 4 GPUs
Figure 4 presents the results across the three datasets dis- cussed in Section 1. In the TCGA-LUNG dataset, all com- parative models exhibit monotonically increasing accuracy as the stacking quantity increases from 8 to 32. However, performance degrades when the batch size exceeds a critical threshold (N 32). Notably, model sensitivity to batch size varies significantly. When the batch size increases from 4 to 32 in the Camelyon MuMMutiscale dataset, accuracy and convergence time trends diverge across models. More- over, optimal batch sizes vary substantially between datasets. Comparative analysis reveals a notable distinction between TCGA-LUNG (simple-strus structured data) and Camelyon 16- ImageNet-Pretraal Pretraii (complex, lower-quality data): models
Acc
93.71% = 0.11% 133.32 = 17.47
93.83% 0.09%
Time
142.95 + 11.73
with Equation 12, we posit that this phenomenon originates from fundamental scalability constraints of distributed sys- tems: when the node scale surpasses K > (keep rate 1%), the proportion of cross-node gradient synchronization time (Teomm) in the total training duration progressively in- creases. At this critical juncture, the diminishing returns from reduced computation time per iteration are offset by the rising communication overhead, pushing overall training efficiency into plateau phase.
7

---

Distributed Parallel Gradient StackinggDPGGS::Solvvin Whole Slide Image Stacking Challenge in Multi- Learning
TCGA Lung Multiscale
Camelyon 16 Multiscale
Camelyon 16 ImageNet Pretrain
Methods
Accuracy (%)
Accuracy (%)
ABl MIL
Accuracy
93
80
Trans MIL
70
AB MIL Boodinne
70
Raseline
Trans MIL Baseline
Buscline
8
16
64
16
32
4
8
16
32
Batch Size
Batch Size
Batch Size
TCGA Lung Multiscale
Camelyon 16 Multiscale
Camelyon 16 ImageNet Pretrain
800
Methods
400
1000
AB MIL.
(S)
600
Time (S)
Time (8)
750
Trans MIL
400
200
500
CLAM
Bascline
200
250
Mena MIL Baseline
CLAM Basetine
16
32
Batch Size
Batch Size
Batch Size
Figure 4. The dashed line signifies the baseline of accuracy and time when this framework is not employed. The incomplete line indicates that the model did not converge on this dataset, consequently, a convergence time is not available. The corresponding accuracy represented by the high value in the test data distribution.
yield significantly greater accuracy gains (4.68%-9.33% 0.4%-1.399 1.3%% with batch size scaling in the complex dataset.
Table 5. ablation analysis conducted evaluate the racy and convergence time of ABMIL and TransMIL integrated with DPGS under Mbps network bandwidth. The experimen- tal results demonstrate that the DPGS framework combined with DMGC achieves fastest convergence speed while maintaining competitive accuracy performance.
This discrepancy is attributable to two factors: (I) Complex datasets offer more room for accuracy improvement due to lower baseline performance; (2) Stacking-drived error esti- mation reduces gradient noise more effectively, improving the consistency of parameter updates, especially in noise- intensive gradients associated with complex data patterns.
Model
Method
Accuracy
Time (s)
DPGS+ None
93.84% #0.08
544 +37
4.3. Results of DMGC
ABMIL
DPGS+DGO La 2017
93.83% 12012
249 115
DPGS+DMGC
93.83% =0.09
142 #11
4. ABLATION STUDY FOR DMGC
DPGS+None
95.23% 20.00
1311 =90
This study systematically validates the core advantages of DMGC through controlled ablation experiments. Under fixed network bandwidth (20 Mbps) and computational node scale of N 4 on the Cl6-Multiscale dataset, five independent experimental replicates were conducted, and mean values along with standard deviations were calcu- lated. As shown in Table 5, when using the combined DPGS+DMGC strategy, the ABMIL and TransMIL mod- els achieve 73.9% and 76.2% reductions in convergence time, respectively, compared the DPGS baseline methods, while maintaining identical classification accuracy.
TransMIL
DPGS+DGC
95.20% =0.08
652 +55
DPGS+DMGC
95.23% 2:000
313 119
bandwidth conditions, DMGC achieves superior conver- gence speed compared to both DGC and non-compression methods, with average acceleration factors of 1.73x over DGC and 3.08x over uncompressed approaches. These results confirm that DMGC maintains stable model perfor- mance under stringent bandwidth limitations while signif- icantly improving communication efficiency compared DGC.
4.3.2. BANDWIDTH AFFECTS DMGC
4.3.3. DISCARD RATE IMPACTS ON DMGC.
To investigate algorithmic robustness under network band- width constraints, we establish controlled network environ- ments using writch-based rate limiting and conduct multi- bandwidth experiments on ABMIL models with the C16- Multiscale dataset. Figure 5 shows that under various low-
This study analyzes gradient discard rates in DMGC through controlled experiments. As Table 6 showed Under 10Mbps bandwidth constraints, DMGC maintains stable conver- gence even 99.99% discard rates, demonstrating excep-
8

---

Distributed Parallel Gradient StackinggDDGGS::Sllvvinn Whole Slide Image Stacking Challenge in Multi-Instance Learning
340
Sowe Ioo mniGe
Table 6. Impact. of DMGC Dropout
Rates
ABMII
344
Performance(C16-Me (aa.aaececCMMMMMMscaaaalll www-MMMticcae.MMMaaa Comm: nication size(Byte).
Commu-
CR: Compression ratio(original size
Newwork Bandwalth
brsthg 142
compressed size)
1006
Keep Rate
Time (8)
Acc. (%)
Comm. (xle3)
CR
10M
0.01
5256
93.8go2
8.3
117.5
1342
0.1%
1380
93. 93.8
10.7
91.6
501tps
739
1 1%
355
93. 8
42. 22.945.
22.8
10%
4094ss
94. 0
90.446.5
10.8
20%
470g33
94. Ogao
244. 1,73
4.4
Time (s)
30%
531434
94. Oga0
384.5279
2.5
40%
564233
93.8g02
440.0
2.2
Figure.5 5. The convergence times of the ABMIL model with dif- ferent compression methods on the C16-M Multiscale dataset under varying bandwidths. should be noted that the accuracy in this experiment remained nearly unchanged, with a variance of less than 0.03%. can be observed that the bandwidth decreased, the advantage of this method over DGC and the non-compression approach became increasingly pronounced.
50%
621+30
93.8.a.
697.24om
1.4
60%
445.26
93.8.00
733.1
1.3
70%
390,26
94. 0,00
884.9.96
80%
441
93.8g0.1
923.5g94
1.1
90%
724.57
93. 8,02
937.6x108
1.0
100%
731434
93.820.
979.72175
1.0
tributed training, the synchronization overhead still limits GPU utilization. Future work will explore asynchronous distributed computing to mitigate synchronization latency and further accelerate large-scale MII. workflows. Com- bining adaptive task scheduling with asynchronous updates has the potential to improve the scalability of ultra-large histopathology datasets, paving the way for distributed MIL at petahyte scale.
tional robustness, A non-monotonic convergence pattern emerges: optimal speed at 99% discard rate vs. suboptimal performance 30%, stemming from the balance between Low discard rates preserve gradient tensor semantic integrity but significantly increase network transmission latency; 2. High discard rates reduce communication overhead at the cost of increased gradient sparsification.
Further analysis demonstrates DMGC's dual compression (gradient sparsification + weight sparsification) significantly improves communication efficiency. Experiments show 117.5-fold parameter compression ratio versus baseline. However, diminishing marginal returns in compression effi- ciency are observed in low discard rate regimes. This phe- nomenon occurs due to heterogeneous sparse weights across distributed nodes. Only parameters that exceed retention thresholds are updated, which reduces weight compressibil- ity and leads to decline marginal utility.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Abrahamyan, L., Chen, Y., Bekoulis, G., and Deligiannis, N. Learned gradient compression for distributed deep learning. IEEE Transactions on Neural Networks and Learning Systems, 33(12):7330-744, 2021.
5. Conclusion
This study introduces DPGS, distributed MII. framework that integrates gradient stacking with dual compression (DMGC), achieving the following:
Aji, A. F. and Heafield, K. Sparse communica- tion for distributed gradient descent. arXiv preprint arXiv:/704 770.05501, 2017.
1. Lossless batch processing of variable-length data via distributed gradients.
Alistarh, D., Grubic, D.. Li, J., Tomioka, R., and Vojnovic, M. Qsgd: Communication tization and encoding. Advances neural information processing systems, 30, 2017.
sgd via gradient quan-
2. A 99.2% bandwidth reduction through joint gradient- weight compression, shown in Table 6.
3. Up to 31x faster training and 9.3% accuracy im- provements over baselines on the Camelyon16 and TCGA-Lung datasets.
Anil, R., Pereyra, G., Passos, A., Ormandi, R., Dahl, G. E., and Hinton, G. E. Large scale distributed neural net- work training through online distillation. arXiv preprint arXiv: /804 00325,, 2018
While DPGS optimizes the efficiency of synchronous dis-

---

Distributed Parallel Gradient Stackingg Whole Slide Image Stacking Challenge in Multi-Instance Learning
Bailly, A., Blanc, C., Francis, É., Guillotin, T., Jamal, F,, Wakim, B., and Roy, P. Effects of dataset size and interac- tions on the prediction performance of logistic regression and deep learning models. Computer Methods and Pro- grams in Biomedicine, 213:106504, 2022.
Jahani-Nasab, M. and Bijarchi, M. A. Enhancing conver- gence speed with feature enforcing physics- thysics-informed neu- ral networks using boundary conditions as prior knowl- edge. Scientific Reports, 14(1):23836, 2024.
Karimireddy, S. P., Rebjock, Q., Stich, S., and Jaggi, M. Error feedback fixes signsgd and other gradient compres- sion schemes. In International Conference on Machine Learning, pp. 3252-3261. PMLR, 2019.
Chan, T. H., Cendra, F. J., Ma, L., Yin, G., and Yu, L. Histopathology whole slide image analysis with hetero- geneous graph representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15661-15670, 2023.
Langer, M., He, Z., Rahayu, W., and Xue, Y. Distributed training of deep learning models: A taxonomic perspec- tive. IEEE Transactions Parallel and Distributed Sys- tems, 31(12):2802-25 2818, 2020.
Chen, C.-Y., Choi, J., Brand, D., Agrawal, A.. Zhang, W., and Gopalakrishnan, K. Adacomp: Adaptive residual gradient compression for data-parallel distributed train- ing. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Li, B., Li, Y., and Eliceiri, K. W. Dual-stream multiple instance learning network for whole slide image clas- sification with self-supervised contrastive learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 14318-14328, 2021,
Chen, J., Ding, T., Lu, M. Y., Williamson, D. F., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine, 30(3): 850- 850-862, 2024.
Li, C. H., Gondra, I., and Liu, L. An efficient parallel neural sework-based multi-instance learning algorithm. The Journal of Supercomputing, 62:724-740, 2012.
Li, J., Chen, Y., Chu, H., Sun, Q. Guan, T., Han, A., and He, Y. Dynamic graph representation with knowledge- aware attention for histopathology whole slide image analysis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 11323- 11332, 2024.
Dai, J., Wang, Y., Qiu, X., Ding, D., Zhang, Y., Wang. Y., Jia, X., Zhang, C. L., Wan, Y., Li, Z., et al. Bigdl: A distributed deep learning framework for big data. In Proceedings of the ACM symposium on cloud computing, pp. 60-60,20 2019.
Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., et al. Large scale distributed deep networks, Advances in neural information processing systems, 25, 2012.
Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z. On the convergence of fedavg on non-iid data, arXiv preprint 1907:0299000288 2019.
Lin, Y., Han, S., Mao, H., Wang. Y., and Dally, W. J. Deep gradient compression: Reducing the communica- tion bandwidth for distributed training. arXiv preprint 1712v 2017.
Dryden, N., Moon, T., Jacobs, S. A., and Van Essen, B. Communication quantization for data mata-parallel training of deep neural networks. In 2016 2nd Workshop on machine learning in hpc environments (MLHPC), pp. 1-8. IEEE, 2016.
Lu, M. Y., Williamson, D. F., Chen, T. Y., Chen, R. J., Bar- bieri, M., and Mahmood, F. Data-efficient and weakly su- pervised computational pathology whole-slide images. Nature biomedical engineering, 5(6):555-570, 2021.
Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A.. Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 1:XXvv/700..0 777.0066.7777 2017.
Narayanan, D., Shoeybi, M., Casper, J., LeGresley, P., Pat- wary, M., Korthikanti, V., Vainbrand, D., Kashinkunti, P., Bernauer, J., Catanzaro, B., et al. Efficient large-scale language model training on gpu clusters using megatron- Im. In Proceedings of the international conference for high performance computing, networking, storage and analysis, pp. -11,,0211
Hou, W., Yu, L., Lin, C., Huang. H., Yu, R., Qin, J., and Wang, I.. H 2-mil: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 36, pp. 933-941, 2022.
Qu, I.., Ma, Y., Luo, X., Guo, Q., Wang, M., and Song, Z. Rethinking multiple instance learning for whole slide image classification: A good instance classifier is all you need. IEEE Transactions on Circuits and Systems for Video Technology, 2024.
Ilse, M., Tomezak, J., and Welling, M. Attention deep multiple instance learning. In International conference on machine learning, pp. 2127-2136. PMLR, 2018
10

---

Distributed Parallel Gradient Stackinng Whole Slide Image Stacking Challenge in Multi-Instance Learning
Rasley, J., Rajbhandari, S., Ruwase, O., and He, Y. Deep- speed: System optimizations enable training deep learn- ing models with over 100 billion parameters. In Proceed- ings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 3505-3506, 2020.
Niigh-Performance Parallel and Distributed Computing, pp. 113-124, 2020.
Wen, J., Wen, J., and Fang, M. Msamil-net: An end-to-end multi-scale aware multiple instance learning network for efficient whole slide image classification. arXiv preprint arXiv:2503.08555 2025.
Recht, B., Re, C., Wright, S., and Niu, F. Hogwild!: A lock- free approach to parallelizing stochastic gradient descent. Advances in neural information processing systems, 24, 2011.
Xu, Y., Li, Y., Shen, Z., Wu, Z., Gao, T., Fan, Y., Lai, M., and Chang, I.-C. Parallel multiple instance learning for extremely large histopathology image analysis. BMC bioinformatics, 18: 1-11- 2017.
Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. 1-bit stochas- tic gradient descent and its application to sata-parallel distributed training of speech dnns. In Interspeech, vol- ume 2014, pp. 1058-1062 Singapore. 2014.
Zhang, Y., Li, H., Sun, Y., Zheng, S., Zhu, C., and Yang, L. Attention-challenginng multiple instance learning for whole slide image classification. In European Conference on Computer Vision, pp. 125-143. Springer, 2024.
Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J., Ji, X., et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136-2147, 2021.
Zhao, W., Wang, S., Yeung. M., Niu, T., and Yu, L. Mulgt: Multi-task graph-transformer with task-aware knowl- edge injection and domain knowledge-drine driven pooling for whole slide image analysis. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 3606-3614, 2023.
Shapcott, M., Hewitt, K. J., and Rajpoot, N. Deep learning with sampling in colon cancer histology. Frontiers in bioengineering and biotechnology, 7:52, 2019.
Zheng, Y., Gindra, R. H., Green, E. J., Burks, E. J., Betke, M., Beane, E., and Kolachalama, V. B. A graph- transformer for whole slide image classification. IEEE transactions medical imaging, 41(11):3003-3015 2022.
Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., and Catanzaro, B. Megatron-Im: Training multi- billion parameter language models using model paral- lelism. arXiv preprint arXiv:1909.0800 08050. 2019.
Zhu, W., Chen, X., Qiu, P., Sotiras, A., Razi, A., and Wang. Y. Dgr-mil: Exploring diverse global representation in multiple instance learning for whole slide image classifi- cation. In European Conference on Computer Vision, pp. 333-351. Springer, 2024.
Tang, W., Zhou, F., Huang, S., Zhu, X., Zhang, Y., and Liu,
B. Feature re-embedding: Towards foundation model- level performance in computational pathology. In Pro- ceedings of the IEEE/CVF Conference Computer Vi- sion and Pattern Recognition, pp. 11 11343-11333 11352, 2024.
Vogels, T., Karimireddy, S. P., and Jaggi, M. Powersgd: Practical low-rank gradient compression for distributed optimization. Advances in Neural Information Processing Systems, 32, 2019.
Wang, G., Zhang, C., Shen, Z., Li, A., and Ruwase, O. Domino: Eliminating Communication in LLM Train- ing via Generic Tensor Slicing and Overlapping. arXiv preprint arXiv:2409. 1 15241, 2024a.
Wang, H., Luo, L., Wang. F., Tong, R., Chen, Y.-.W, Hu, H., Lin, L., and Chen, H. Rethinking multiple instance learning for whole slide image classification: A bag-level classifier is good instance-level teacher. IEEE Transac- tions on Medical Imaging, 2024b.
Wang, L., Wu, W., Zhang, J., Liu, H., Bosilca, G., Her- lihy, M., and Fonseca, R. Fft-based gradient sparsifica- tion for the distributed training of deep neural networks. In Proceedings of the 29th International Symposium on


Reviews:
[{"review_id": "EgAx3f6qM8", "reviewer": "Reviewer_M5Bd", "summary": "The manuscript introduces a framework designed to overcome the challenges posed by variable-length bags in Multiple Instance Learning (MIL) for whole slide image analysis. The framework was called Distributed Parallel Gradient Stacking (DPGS), distributes the gradient computation across multiple processes and stacking gradients. This method simulates mini-batch training without the need for data padding. The authors also propose Deep Model-Gradient Compression (DMGC), which jointly compresses both gradients and model weights to reduce communication overhead significantly. Experiments on the Camelyon16 and TCGA-Lung datasets indicate improvements in training speed and accuracy over standard approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1：Could you clarify the integration between DPGS and DMGC, specifically how the compression interacts with the distributed gradient stacking process?\n\nA1：\n    We sincerely appreciate the reviewer's valuable feedback. We apologize for the lack of clarity in describing the relationship between the components in our original manuscript. Regarding your question, we provide the following clarification:\n    The DMGC module is responsible for the compression and transmission of gradients within the gradient-based DPGS framework. Specifically, during distributed training, after worker nodes have completed their local gradient computations, the gradients are first processed by the DMGC module for compression before being transmitted to the main server. Consequently, the main server receives not the full gradients but rather the sparsified gradients compressed by DMGC.\n    On the main server side, DPGS aggregates these compressed sparse gradients to form new global gradients. It is noteworthy that due to the high compression ratio of DMGC, the aggregated gradients remain sparse (the extent of sparsity is contingent on the keep rate). This inherent sparsity enables selective model updates, i.e. only a subset of weights require updating rather than full parameter updates. The system leverages this property to transmit only the sparse weight updates back to the worker nodes, thereby significantly reducing communication overhead (this represents one of DMGC's key innovations compared to DGC[1]).\n\nWe sincerely appreciate your valuable feedback. If you have any further questions about the paper, please do not hesitate to ask us.\n\nQ2：Could you provide additional details regarding your hyperparameter choices—such as the selection of compression thresholds and the batch sizes used in different settings?\n\nA2：\n    We sincerely appreciate the reviewer's valuable feedback.  Regarding your question, we provide the following clarification:\n    Regarding the experiments presented in Table 2 and Figure 4, all DPGS+DMGC tests were conducted with 4 worker nodes and 1 master process, using a fixed gradient compression ratio of 10% (retaining 10% of gradients; results with different compression ratios can be found in Table 4). All frameworks (except the Classic framework which cannot support batch stacking) were evaluated with multiple batch sizes, employing a stepwise B-value configuration adapted to each dataset's scale (C16: B=1/4/8/16/32; TCGA-LUNG: B=1/8/16/32/64). The complete DPGS+DMGC results across different B-values are presented in Figure 4.\n    Within the DPGS+DMGC framework shown in Table 2 and Figure 4, the master process maintains a fixed gradient buffer size of 4, while worker nodes dynamically adjust their gradient accumulation steps according to the batch size (e.g., when batch size is 16, both the gradient buffer and accumulation steps are set to 4, making the effective gradient update equivalent to batch size 16). All experiments used the Adam optimizer with default momentum parameters, with a base learning rate of 0.001 that follows the linear scaling rule (L=0.001×Batch_size) as established in Goyal et al.'s work on large minibatch SGD[2]. Regarding the experiments in Table 3, all tests employed the aforementioned learning rate configuration with a fixed batch size of 16.  As described in the manuscript, we maintained consistent network bandwidth across all tests, and applied identical compression ratios (retaining 10% of gradients) for both DGC and DMGC methods.  All other DPGS-related experimental settings remained unchanged from those previously specified.\nFor the experiments depicted in Figure 3, DPGS maintained the same configuration while DMGC used a fixed 10% compression ratio, with identical learning rates and optimizer settings as described earlier.  The only modification involved varying the number of nodes, and we recorded training time (not convergence time) at every 10-epoch interval, maintaining a constant batch size of 16 throughout these experiments.\nConcerning Table 4 and Figure 5, the DPGS configuration remained consistent with our baseline setup except for deliberate variations in two parameters: network bandwidth and compression ratio.  These experiments were likewise conducted with a fixed batch size of 16.\n\nWe sincerely appreciate your valuable feedback. If you have any further questions about the paper, please do not hesitate to ask us. We would be deeply grateful if you find our revisions satisfactory and might consider adjusting your overall evaluation.\n\n[1]Lin, Y., Han, S., Mao, H., Wang, Y., and Dally, W. J. Deep gradient compression: Reducing the communication bandwidth for distributed training. URL http://arxiv.org/abs/1712.01887.\n[2]Goyal, P., Dollar, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., and He, K. Accurate, large minibatch SGD: Training ImageNet in 1 hour. URL http://arxiv.org/abs/1706.02677.", "claims_and_evidence": "The submission does support its claims with both theoretical derivations and empirical evidence. The authors provide theoretical Justification that gradient stacking is mathematically equivalent to traditional mini-batch training. The empirical results on Camelyon16 and TCGA-Lung datasets show improvements in convergence time and prediction accuracy.", "methods_and_evaluation_criteria": "The combination of DPGS and DMGC addresses the challenge of handling variable-length bags in MIL. Experiments conducted on Camelyon16 and TCGA-Lung provide a measure of the proposed method’s effectiveness in real-world scenarios. The evaluation criteria include both prediction accuracy and convergence time, which assesses not just the model’s performance but also its efficiency.", "theoretical_claims": "The authors derive that aggregating gradients computed on separate bags (via DPGS) is mathematically equivalent to computing the gradient on a mini-batch formed by stacking the data. The authors also present a theoretical time complexity analysis that compares the parallel training time with sequential training. The derivation provides a reasonable estimate of speedup and highlights potential scalability.", "experimental_designs_or_analyses": "The combination of DPGS and DMGC addresses the challenge of handling variable-length bags in MIL. Experiments conducted on Camelyon16 and TCGA-Lung provide a measure of the proposed method’s effectiveness in real-world scenarios. The evaluation criteria include both prediction accuracy and convergence time, which assesses not just the model’s performance but also its efficiency.", "relation_to_broader_scientific_literature": "The paper bridges gaps between MIL-specific challenges and the broader advances in distributed deep learning and gradient compression.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "The paper enables efficient parallel processing despite variable instance lengths in MIL. The combinations of DPGS and DMGC improve training efficiency and model accuracy.\n\nThe manuscript provides a derivation demonstrating the equivalence of gradient stacking to traditional mini-batch training. The time complexity analysis strengthen the theoretical foundation of the approach.\n\nThe results, presented in tables and figures, effectively demonstrate both convergence speed improvements and accuracy gains.\n\nThe integration between DPGS and DMGC could be better articulated. It is unclear how the compression interacts with the distributed gradient stacking.\n\nAdditional details regarding hyperparameter choices (e.g., selection of compression thresholds, batch sizes in different settings) would further strengthen reproducibility.", "other_comments_or_suggestions": "No.", "questions_for_authors": "1. Could you clarify the integration between DPGS and DMGC, specifically how the compression interacts with the distributed gradient stacking process?\n\n2. Could you provide additional details regarding your hyperparameter choices—such as the selection of compression thresholds and the batch sizes used in different settings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MwQiYPxxaU", "reviewer": "Reviewer_g8fv", "summary": "This paper introduces Distributed Parallel Gradient Stacking (DPGS), a framework designed to address the challenge of non-stackable data in Multiple Instance Learning (MIL) for Whole Slide Image (WSI) analysis. The authors propose two key components: (1) DPGS, which enables parallel processing of variable-length MIL bags by distributing them across multiple GPUs and aggregating their gradients, and (2) Deep Model-Gradient Compression (DMGC), which reduces communication overhead during distributed training through joint compression of gradients and model parameters. Experiments on Camelyon16 and TCGA-Lung datasets demonstrate significant improvements in both training speed (up to 31× faster) and classification accuracy (up to 9.3% increase) compared to baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1:The paper calls non-stackable MIL data a bottleneck citing 'prohibitively long training times',yet current methods train adequately on single GPUs within half-day for these datasets\n\nA1:Thank you for reviewing.Clarifications:This is an inexact error \"long training times\":Our method shows superior efficiency under identical conditions. While some MIL methods use 24GB GPUs, repeated runs (tuning/ablation) require 10-100× more time,making efficiency critical.Current medical datasets (thousand-scale) remain smaller than natural image sets (e.g., ImageNet).As [1] shows,larger datasets improve generalization,suggesting medical data will grow.Our framework accelerates training through multi-node/single-GPU parallelism and enables federated learning for multi-center collaboration,getting ready for the future\n\nQ2:The reported performance improvements lack comparisons with simpler alternatives like uniform sampling that could potentially achieve comparable results more efficiently.\n\nA2:Thank you for reviewing.For benchmarks, we added uniform sampling tests (currently ABMIL-only due to time constraints).Results show it speeds convergence but reduces accuracy and increases variance(see table:https://drive.google.com/file/d/1sjgW2z6Y9pbgwI9v6TwaJBZcwBd4PC4O/view?usp=sharing),compromising data integrity—mathematically distinct from standard minibatching by breaking positive bag completeness and potentially missing key instances,with larger batches worsening these effects [2] .Our method maintains better efficiency-accuracy tradeoffs. \n\nQ3: While DMGC enhances communication efficiency, its multi-GPU dependency and unproven performance on typical clinical hardware (especially low-GPU setups) pose practical concerns.\n\nA3:Thank you for reviewing.To clarify:Our method supports single-GPU operation through multi-process parallelization (virtual nodes) to maximize resource utilization.This optimizes usage without multi-GPU setups. Intra-device communication enhances efficiency as memory bandwidth exceeds network limits. We compared ABMIL performance using 4 nodes on single 4060 8G vs. multi-machine V100 32G configurations(https://drive.google.com/file/d/1biUVuB6COd79X3NADdsiwm9va2p1Mpyd/view?usp=sharing), with detailed implementation analysis to be included in revision.\n\nQ4:Unclear feature extraction:The\"multi-scale\"features referenced for both datasets are not adequately defined,obscuring their relation to current computational pathology feature extraction methods. \n\nA4:Thank you for reviewing.Multi-scale features were obtained from DS-MIL's repository (https://github.com/binli123/dsmil-wsi/issues/49), extracted at multiple magnifications (e.g.20×,5×) and concatenated into a feature pyramid.This maintains tissue details across scales while optimizing feature use via local attention, enhancing both classification and lesion localization.\n\nQ5:Outdated baselines:The paper uses older MIL methods but omits recent advanced MIL approaches representing current state-of-the-art in computational pathology and pathology foundation models for feature extraction.\n\nA5:Thank you for reviewing.Key clarifications:Our focus is a general MIL framework addressing non-stackable data bottlenecks, not a new model, hence using classical models (ABMIL,TransMIL) for comparison.New tests compare with recent models (RRT-MIL,ACMIL), showing training efficiency gains (ACMIL1.47%ACC and 19.1XTime improve/RRTMIL 1.4%ACC and 3.99XTime improve)(Full table::https://drive.google.com/file/d/1VODIlXC0Qd1wXPao16yJFCReRAYj3MBV/view?usp=sharing).Due to time constraints,the full experiments will be provided in the revised version.\n\nQ6:Limited dataset analysis:While using established datasets, the paper should better discuss their characteristics and potential impact on results. \n\nA6: Key findings:Bag length variation (Δ57k/105k for C16 Multiscale/ImageNet) affects speed, explaining Bag Pooling's lag.Dataset size dictates optimal batch (TCGA=32 vs C16=16), showing scale's hyperparameter impact.Method differences fade with easier features (TCGA) but intensify with hard ones (C16 ImageNet).\n\nQ7:The study's omission of current foundation models (e.g., UNI, CONCH, PLIP) that achieve near-perfect performance on these datasets with simple MIL architectures requires justification.\n\nA7:Thank you for reviewing.Our core contribution addresses MIL's data stacking limitations through framework innovation (not feature extraction), justifying initial internal comparisons.Following your advice, we've added comparisons(https://drive.google.com/file/d/1Xs2YqWd7SCp4Te8PaqnUbjXEzdf3p6Vd/view?usp=sharing);Due to time,the full experiments will be provided in the revised version.\n\n[1]Bailly,et al. \"Effects of dataset size and interactions on the prediction performance of logistic regression and deep learning models.\" Computer Methods and Programs in Biomedicine 213 (2022): 106504.\n[2]Shapcott M,et al.Deep Learning With Sampling in Colon Cancer Histology.Front Bioeng Biotechnol.2019;7:52.", "claims_and_evidence": "The paper makes several claims that are supported by experimental evidence but lack critical context:\n1.\tProblem significance: The authors frame non-stackable MIL data as a critical bottleneck for training efficiency, claiming that sequential processing leads to \"prohibitively long training times.\" However, from my experience, many current MIL methods can be trained on a single high-memory GPU (e.g., RTX3090 with 24GB memory) in reasonable timeframes (half a day) for the datasets used in this paper.\n2.\tAcceleration and accuracy improvement: The paper claims up to 31× faster training and 9.3% accuracy improvement. While the experimental results in Table 2 support these numbers, the comparison is against sequential processing methods without consideration of simpler alternatives like uniform sampling approaches that could enable standard batch training. \n3.\tMathematical equivalence of DPGS to mini-batch training: The mathematical derivations showing DPGS is equivalent to mini-batch SGD are sound, but this equivalence raises questions about why standard mini-batch approaches with fixed-size bags (through patch sampling) couldn't achieve similar results with less complexity.\n4.\tCommunication efficiency of DMGC: The ablation studies support the claim that DMGC achieves significant communication reduction. However, this actually leads to another major question that existing works can be trained with a single GPU, yet the proposed method requires multi-GPU for training. In clinical settings, researcher may not have access to multi-GPU setups.", "methods_and_evaluation_criteria": "The idea of this work is timely and interesting. And the method design seems reasonable. However, the evaluation has several significant limitations:\n1.\tUnclear feature extraction: The \"multi-scale\" features referenced for both datasets are not adequately defined, making it impossible to understand their relationship to current feature extraction approaches in computational pathology.\n2.\tOutdated baselines: The paper considers older MIL methods (ABMIL, MeanMIL, TransMIL, CLAM-MB) while omitting including more recent and advanced MIL approaches that represent the current state-of-the-art in computational pathology and pathology foundation models for feature extraction.\n3.\tMissing comparisons to simpler alternatives: The paper did not evaluate simpler baselines such as uniform sampling that could potentially address the variable-length bag issue with much less complexity and computational overhead.\n4.\tLimited dataset analysis: While the paper uses established datasets (Camelyon16 and TCGA-Lung), it could further benefit from adequately discussing the dataset characteristics or how they might influence the observed results.", "theoretical_claims": "The theoretical foundations of the paper are generally sound.", "experimental_designs_or_analyses": "Please address the questions in the Methods and Evaluation Criteria section. \nAdditional questions:\n1.\tLimited analysis of computational requirements: There is insufficient analysis of how results might change with different hardware configurations, particularly more accessible setups with fewer or less powerful GPUs. \n2.\tImpact of foundation models: In the field of computational pathology, multiple foundation models have been proposed yet I see no use here in this paper. From the community’s experience, using foundation models as feature extractors and a simple MIL model (such as ABMIL), the performance on these two datasets could easily goes up to near 100%. I believe it would be better to take these into consideration.", "relation_to_broader_scientific_literature": "The proposed method contributes to the entire community of computational pathology, facilitating efficient and effective frameworks for whole slide image analysis. The speed-up in terms of training time and batch training are interesting and considered for the modern MILs for the first time.", "essential_references_not_discussed": "The Related Work section in this paper is comprehensive, yet some minor discussion is needed:\n1.\tThe paper doesn't adequately discuss simpler alternatives to handling variable-length bags, such as uniform sampling strategies.\n2.\tThe MIL approaches used as baselines do not represent the current state-of-the-art.", "other_strengths_and_weaknesses": "Strengths:\n1.\tThe mathematical derivations showing equivalence to mini-batch training are sound.\n2.\tThe gradient compression approach (DMGC) offers an interesting extension to existing gradient compression techniques.\n3.\tThe ablation studies provide useful insights into the factors affecting performance within their framework.\nWeaknesses:\n1.\tHigh resource requirements: Although the speed-up performance is impressive, the method requires multiple GPUs and high-bandwidth connections, significantly limiting its practical applicability in many research and clinical settings.\n2.\tMissing comparisons to simpler alternatives: The paper doesn't evaluate simple alternatives such as uniform sampling that could potentially achieve similar results with much less complexity.\n3.\tOutdated baselines: The paper relies on comparisons with older MIL methods rather than current state-of-the-art approaches.\n4.\tUnclear feature extraction: The \"multi-scale\" features referenced throughout the paper are not adequately defined.\n5.\tLimited relevance given foundation models: The paper doesn't acknowledge or compare against foundation models that have demonstrated near-perfect performance on the same datasets.", "other_comments_or_suggestions": "1.\tTypos should be carefully filtered: for example, in the first paragraph in Introduction, “followed by classification.MIL framework”.\n2.\tMeanwhile, please check the indent in the paper. For example, in the first paragraph of Introduction, “exemplify high-accuracy solutions in this domain.However,”.\n3.\tCould you put Figure 1 at the top of Page 1?", "questions_for_authors": "1.\tCould you provide a clear definition of the \"multi-scale\" feature extraction process used in your experiments, including architectures and implementation details?\n2.\tHave you compared your approach with simpler methods like uniform sampling from each WSI to create fixed-length bags that could be trained with standard batch processing?\n3.\tCould you explain your experimental setting and the reason why you did not consider current foundation model-based approaches (UNI [1], CONCH [2], PLIP [3], etc.) that have demonstrated state-of-the-art performance on the same datasets?\n4.\tWhat is the minimum hardware configuration required to achieve meaningful benefits from your approach compared to single-GPU training?\n5.\tWhy did you choose to compare against older MIL methods rather than more recent approaches that might represent stronger baselines?\nI will consider raising the overall recommendation score if these questions are solved in the rebuttal phase.\n[1] Chen, Richard J., et al. \"Towards a general-purpose foundation model for computational pathology.\" Nature Medicine 30.3 (2024): 850-862.\n[2] Lu, Ming Y., et al. \"A visual-language foundation model for computational pathology.\" Nature Medicine 30.3 (2024): 863-874.\n[3] Huang, Zhi, et al. \"A visual–language foundation model for pathology image analysis using medical twitter.\" Nature medicine 29.9 (2023): 2307-2316.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "mTeQXENBOC", "reviewer": "Reviewer_7pdf", "summary": "This paper proposes a framework called DPGS combined with DMGC to address the non-stacked data problem in MIL, particularly in WSI analysis. DPGS addresses the inefficiencies due to variable-length instance bags that prevent effective batch stacking by parallelizing MIL models and stacking gradients instead of raw data, thereby achieving significant training speedups. DMGC further enhances performance by compressing both gradients and model weights. Experimental results on Camelyon16 and TCGA-Lung datasets demonstrate up to 31× faster training and 9.3% accuracy improvement over baseline models.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1:Table 2 shows Bag Padding's unexpectedly long convergence time versus Classic. The convergence time calculation method also requires clarification.\n\nA1:Thank you for reviewing.Bag Padding's longer convergence time (Table 2) stems from large packet length spans (C16 MULTISCALE: 223-57,318; C16 IMAGENET: 487-105,775), requiring excessive padding and computation time that outweighs batch benefits. Larger batches mitigate but don't eliminate this penalty.\n\nQ2:Comparing DPGS with recent federated MIL approaches and including asynchronous distributed MIL studies would enhance the literature review's comprehensiveness.\n\nA2:Thank you for reviewing.We acknowledge the need to compare with async distributed/federated MIL methods. Following your guidance, we systematically searched Web of Science using:\"Federated Learning\" AND \"multi-instance learning\"\"asynchronous distributed\" AND \"multi-instance learning\"Both queries returned null results, further confirming our study's novelty.\n\nQ3:The method's novelty is limited, building on existing gradient accumulation, distributed training, and compression techniques.\n\nA3:Thank you for reviewing.We appreciate the reviewer's rigorous evaluation. While acknowledging prior work (gradient accumulation/distributed training), our core contribution addresses MIL's fundamental bottleneck - non-stackable data - which has constrained recent NN-based MIL studies to inefficient batch_size=1 training[2][3].Our three-level innovation:Problem: First lossless MIL data stacking enabling batch_size scalability (orders-of-magnitude efficiency gain)Method: DMGC extends DGC via gradient sparsity for compressed weight distribution (lower communication overhead)Framework: First adaptation of distributed concepts to MIL's variable-length sequences (bridging general/MIL-specific methods)\nWe fully agree these innovations build upon existing technologies, much like how Transformer builds upon self-attention mechanisms. However, proposing systematic solutions to domain-specific critical problems constitutes significant innovation in itself.\n\nQ4:While the experimental design is robust, adding stronger baselines would improve accuracy validation. \n\nA4:Thank you for reviewing.To address your suggestions and improve rigor, we added comparisons with two recent MIL models (RRTMIL, CVPR 2024[2]; ACMIL, ECCV 2024[3]). Preliminary results (below) show our framework retains significant training efficiency advantages over these models (ACMIL1.47%ACC and 19.1XTime improve/RRTMIL 1.4%ACC and 3.99XTime improve). (Full table:https://drive.google.com/file/d/1VODIlXC0Qd1wXPao16yJFCReRAYj3MBV/view?usp=sharing)Ongoing experiments will be fully reported in the revision.\n\nQ5:DPGS+DMGC: Why no same-B comparison?MEANMIL:Why missing time data in Table 2?\n\nA5:Thank you for reviewing.Our tiered B-value design (C16: 1/4/8/16/32; TCGA-LUNG: 1/8/16/32/64) reflects real-world possible batch size selection. Classic only supports B=1 (no stacking),therefore, no comparison can be made; we'll correct Bag Padding labels and provide (https://drive.google.com/file/d/1_DUaRWWV7AcXjODUHF9RfFp_GsJFThd-/view?usp=sharing).For MeanMIL's missing timing:Unstable convergence prevented standard timing calculation [1].Non-convergent cases used class priors (marked *) as conservative estimates.Convergence time shown as \"-\" with explanatory notes.\n\nQ6:Clarify the trade-offs between gradient stacking and asynchronous updates.\n\nA6:Thank you for reviewing.Our design employs synchronous distributed training rather than asynchronous updates due to two key factors:\nAsynchronous updates cause stale gradients that led to significant training instability in our preliminary experiments.\nWhile synchronous updates require coordination overhead, our DMGC method minimizes communication costs while preserving stability through gradient stacking.\n\nWe greatly appreciate your feedback. Should you have any additional questions, please don't hesitate to ask. We'd be most grateful if you find our revisions satisfactory and consider adjusting your evaluation.\n\n[1]Jahani-Nasab, Mahyar, and Mohamad Ali Bijarchi. \"Enhancing convergence speed with feature enforcing physics-informed neural networks using boundary conditions as prior knowledge.\" Scientific Reports 14.1 (2024): 23836.\n[2]Tang, Wenhao, et al. \"Feature re-embedding: Towards foundation model-level performance in computational pathology.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[3]Zhang, Yunlong, et al. \"Attention-challenging multiple instance learning for whole slide image classification.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.", "claims_and_evidence": "The paper claims to enable lossless data stacking in MIL, accelerate training, and improve model accuracy. However, the experimental data appear to contain discrepancies. For instance, inconsistencies exist between the experimental data presented in Table 2 and the descriptions within the main text. Notably, the convergence time of the Bag padding method in Table 2 significantly exceeds that of the Classic method (non-parallel methods restricted to a single GPU), which appears to be anomalous. Furthermore, the precise methodology for computing the convergence time evaluation metric is not adequately elucidated.", "methods_and_evaluation_criteria": "The methodology proposed in this manuscript offers the potential to accelerate the training of giga-pixel Whole Slide Image (WSI) analysis, thereby possessing significant research implications", "theoretical_claims": "N/A. The demonstration of equivalent derivations between DPGS and traditional mini-batch training is straightforward.", "experimental_designs_or_analyses": "The experimental design and analysis are generally robust. However, the validation of model accuracy could benefit from the inclusion of stronger baselines. Furthermore, it is imperative to ensure the veracity of the experimental data, eliminating any typographical errors within the tables, as these significantly undermine the credibility of the reported results.", "relation_to_broader_scientific_literature": "The manuscript applies widely adopted techniques, such as gradient accumulation, distributed training, and gradient compression, to the task of giga-pixel WSI analysis. The connections to prior works are well-articulated.  A discussion on how DPGS compares to recent federated MIL approaches would strengthen the literature review.", "essential_references_not_discussed": "The paper provides a comprehensive overview of related works. Given the paper's emphasis on distributed training and gradient compression, incorporating recent studies on asynchronous distributed MIL and federated MIL frameworks would furnish a more comprehensive perspective of the field.", "other_strengths_and_weaknesses": "Strengths:\n- The paper proposes DPGS to stack giga-pixel Whole Slide Images (WSIs) with inconsistent instance lengths, thereby accelerating training.\n- Experimental results demonstrate that the proposed method significantly enhances both training speed and accuracy.\n\nWeaknesses:\n- The novelty is somewhat incremental, as the method leverages established gradient accumulation, distributed training, and gradient compression techniques.\n- The computation methodology for metrics such as convergence time remains undefined.\n- Numerous data errors significantly compromise the credibility of the experimental results.", "other_comments_or_suggestions": "Clarify the trade-offs between gradient stacking and asynchronous updates.", "questions_for_authors": "-  Why is the Time data for the MEANMIL method unavailable in Table 2? \n- Why is the DPGS+DMGC method not compared under identical \"B\" conditions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "MYw5m9ML0r", "reviewer": "Reviewer_psN8", "summary": "This paper proposes a Distributed Parallel Gradient Stacking (DPGS) framework and a Deep Model Gradient Compression (DMGC) technique to address the non-stackable data issues caused by variable bag lengths in whole slide image (WSI) multiple instance learning (MIL). The core innovations include: 1) Enabling lossless batch processing through distributed sub-model parallel computation and gradient aggregation; 2) Jointly compressing gradients/parameters and leveraging gradient sparsity to update only sparse parameters, reducing communication overhead by 99.2% while maintaining convergence; 3) Experimental results demonstrate 31× acceleration in training speed and 9.3% improvement in accuracy on Camelyon16 and TCGA-Lung datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: Most of the claims made in the manuscript are supported by clear and convincing evidence. The shortcoming is that the motivation of this paper is not strongly substantiated. \n\nA1:\nWe sincerely appreciate the reviewer's valuable feedback.\nRegarding training efficiency, we clarify that \"excessively long\" was intended to highlight the relative efficiency difference between existing methods and our approach under the same conditions. We have added reference [1] to support this.\nFor the second point, we agree that a clearer theoretical link between non-IID data and convergence difficulties is needed. We’ve cited [2], which proves: (1) non-IID data causes gradient bias, delaying or preventing convergence, and (2) appropriate gradient aggregation strategies (e.g., mean normalization) mitigate this effect. Our gradient aggregation mechanism is based on these insights.\n\nQ2:\n    However, it is not clear how “convergence” is defined, which may not be sufficient to support the claim that the instability of gradient estimation and the difficulty of convergence in previous methods have been resolved.\n\nA2:\nWe sincerely appreciate the reviewer's valuable feedback.\nWe define convergence using the threshold-based criterion from [3], where the model is considered converged when the total loss remains consistently below a fixed threshold. This method aligns with standard practices and ensures objective stability assessment. We derived the threshold by analyzing the 95th percentile of the loss distribution across multiple trials, ensuring robustness.\n\nQ3：\n    The explanation as to why existing distributed training frameworks (such as Megatron-LM, DeepSpeed) or gradient compression methods (such as DGC) cannot be directly applied to MIL models is not clear.\n\nA3:\nWe sincerely appreciate the reviewer's valuable feedback.\n    The existing distributed frameworks (e.g., Megatron-LM/DeepSpeed) indeed excel in traditional LLM scenarios but face two fundamental challenges when applied to MIL:\n(1) Architectural Compatibility: Current frameworks are primarily optimized for structured architectures like Transformers (e.g., tensor parallelism with intra-layer partitioning, as in Megatron-LM). However, MIL’s multi-instance interaction mechanism, due to its distinct design philosophy and structure, requires different parallelization strategies, making these frameworks suboptimal for MIL.\n(2) Data Characteristics: MIL’s hierarchical \"bag-instance\" data (variable-length, non-directly stackable) violates the conventional distributed training assumption of \"batch homogeneity.\" This leads to inefficient data sharding and memory wastage when using existing frameworks. Our proposed method is specifically tailored to MIL’s unique computational paradigms and data topology.\nThus, in the MIL domain, existing distributed training frameworks cannot be efficiently applied.\n\nQ4:\n    Although bags of different lengths cannot be directly concatenated into a batch tensor, to my knowledge, the built-in collate_fn function in PyTorch’s DataLoader can load multiple bags within a single batch, which means that batch training is possible on a single GPU. What advantages does the proposed DPGS offer over this training approach?\n\nA4:\nWe sincerely appreciate the reviewer's valuable feedback.\nAlthough PyTorch’s collate_fn provides flexibility, preprocessing steps like Bag Pooling are still needed to form batch tensors, potentially causing information loss and additional overhead. DPGS eliminates these steps, as shown in Table 2, where it significantly outperforms pooling-based methods in efficiency.\n\nQ5：How does DMGC ensure that gradients critical for early-stage convergence (e.g., large-magnitude or directionally consistent gradients) are retained during sparsification?\n\nA5:\nWe sincerely appreciate the reviewer's valuable feedback.\nDMGC preserves essential gradients through:\n(1) Magnitude-based Gradient Filtering: We retain the top k% largest gradients, which dominate optimization (Table 4 shows that even retaining just 0.01% allows convergence).\n(2) Momentum Accumulation: Historical gradients stabilize training by suppressing noise and fluctuations. These mechanisms collectively preserve key gradients while minimizing ineffective updates.\n\nThank you for your valuable feedback. If you have further questions, feel free to ask. We would really appreciate it if you find our revisions satisfactory and adjust your evaluation.\n\n[1]Wen, Jiangping, Jinyu Wen, and Emei Fang. \"MsaMIL-Net: An End-to-End Multi-Scale Aware Multiple Instance Learning Network for Efficient Whole Slide Image Classification.\" arXiv preprint arXiv:2503.08581 (2025).\n\n[2]Li, Xiang, et al. \"On the convergence of fedavg on non-iid data.\" arXiv preprint arXiv:1907.02189 (2019).\n\n[3]Jahani-Nasab, Mahyar, and Mohamad Ali Bijarchi. \"Enhancing convergence speed with feature enforcing physics-informed neural networks using boundary conditions as prior knowledge.\" Scientific Reports 14.1 (2024): 23836.", "claims_and_evidence": "Most of the claims made in the manuscript are supported by clear and convincing evidence. The shortcoming is that the motivation of this paper is not strongly substantiated. For instance, the article claims, \"There are two critical bottlenecks: (1) Slow training speed: The inability to utilize GPU parallelism leads to prohibitively long training times for large-scale WSI datasets; (2) Unstable gradient estimation: Sequential gradient updates rely on single-bag statistics, introducing bias across non-identically and independently distributed (non-IID) bags and hindering model convergence.\" However, the second point lacks support from both references and experimental evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method(s) and/or evaluation criteria (e.g., benchmark datasets) are appropriately justified for the current problem or application.", "theoretical_claims": "Yes, I have examined the principles of the method proposed in the article, primarily the Deep Model-Gradient Compression outlined in Section 3.3. After reviewing the references cited in this section (Dean et al., Lin et al.), I find the principles underlying the proposed method to be sound and without issue.", "experimental_designs_or_analyses": "Yes, I have checked the validity of experimental designs and analyses from Section 4.1 to Section 4.3. The experimental design and analysis conducted for the proposed method in the article are methodologically sound and empirically valid.", "relation_to_broader_scientific_literature": "The Distributed Parallel Gradient Stacking (DPGS) framework proposed in this paper represents the first work to achieve simultaneous improvements in both speed and accuracy while addressing the issue of non-stackable data in Multiple Instance Learning (MIL). This holds significant importance for the current development of large-scale pre-trained pathology foundation models. The training method introduced in this paper has the potential to enhance the performance of existing Slide-level pre-trained foundation models, such as those referenced in [1][2][3].\n\n[1] A whole-slide foundation model for digital pathology from real-world data (Nature 2024)\n[2] A pathology foundation model for cancer diagnosis and prognosis prediction (Nature 2024)\n[3] SlideChat: A Large Vision-Language Assistant for Whole-Slide Pathology Image Understanding (CVPR 2025)", "essential_references_not_discussed": "To the best of my knowledge, the key contribution of the proposed Distributed Parallel Gradient Stacking (DPGS) framework in this paper is unique. It introduces the DPGS framework combined with Deep Model Gradient Compression (DMGC) technology to address the issue of non-stackable data caused by varying bag lengths in Whole Slide Image (WSI) multiple instance learning. The related work is comprehensively reviewed, but the motivation presented in the introduction may require further support from additional literature or experimental evidence.", "other_strengths_and_weaknesses": "Strengths:\n1.The paper introduces a novel approach (DPGS) to address the critical challenge of non-stackable data in MIL by leveraging distributed gradient stacking instead of traditional bag padding, enabling lossless MIL batch processing while improving both speed and accuracy\n2.The proposed method achieves up to 31× faster training and 9.3% accuracy gains on widely used medical datasets (Camelyon16, TCGA-Lung). This holds a certain significance for the development of foundational models that require large-scale pre-training in the field of computational pathology.\n\nWeaknesses:\nAs I am not familiar with work in the field of distributed computing or training in the general machine learning area, I conservatively point out the following shortcomings of the work:\n1. The introduction of the paper mentions a key bottleneck of unstable gradient estimation faced by traditional MIL model training. To address this issue, the paper designed experiments to compare the convergence times of different methods. However, it is not clear how “convergence” is defined, which may not be sufficient to support the claim that the instability of gradient estimation and the difficulty of convergence in previous methods have been resolved.\n2. The explanation as to why existing distributed training frameworks (such as Megatron-LM, DeepSpeed) or gradient compression methods (such as DGC) cannot be directly applied to MIL models is not clear.", "other_comments_or_suggestions": "Since I am not familiar with work in the field of distributed computing or training in the general machine learning area, I do not have any other comments or suggestions.", "questions_for_authors": "1.Although bags of different lengths cannot be directly concatenated into a batch tensor, to my knowledge, the built-in collate_fn function in PyTorch’s DataLoader can load multiple bags within a single batch, which means that batch training is possible on a single GPU. What advantages does the proposed DPGS offer over this training approach?\n2.How does DMGC ensure that gradients critical for early-stage convergence (e.g., large-magnitude or directionally consistent gradients) are retained during sparsification?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Existing MIL approaches for Whole Slide Image (WSI) analysis face inefficiency and performance drops due to non-stackable data (varying instance lengths). This paper proposes DPGS, a framework enabling lossless MIL data stacking, paired with DMGC, a joint gradient-model compression method for distributed training. Experiments on Camelyon16 and TCGA-Lung datasets show 31× faster training, 99.2% smaller model communication size, and 9.3% higher accuracy versus baselines. 

During rebuttal, the authors effectively addressed reviewer concerns, and the clarifications led to unanimous positive ratings from reviewers. The AC also concurs with the recommendation to accept this paper.
Decision: Accept (poster)