Pushing the Limits of Gradient Descent for Efficient Learning on Large Images
Authors: Deepak Gupta
Gowreesh Mago
Arnav Chavan
Dilip Prasad
Rajat Mani Thomas
Venue: Accepted by TMLR ()

Abstract: Traditional deep learning models are trained and tested on relatively low-resolution images (< 300 px), and cannot be directly operated on large-scale images due to compute and memory constraints. We propose Patch Gradient Descent (PatchGD), an effective learning strategy that allows us to train the existing CNN and transformer architectures (hereby referred to as deep learning models) on large-scale images in an end-to-end manner. PatchGD is based on the hypothesis that instead of performing gradient-based updates on an entire image at once, it should be possible to achieve a good solution by performing model updates on only small parts of the image at a time, ensuring that the majority of it is covered over the course of iterations. PatchGD thus extensively enjoys better memory and compute efficiency when training models on large-scale images. PatchGD is thoroughly evaluated on PANDA, UltraMNIST, TCGA, and ImageNet datasets with ResNet50, MobileNetV2, ConvNeXtV2, and DeiT models under different memory constraints. Our evaluation clearly shows that PatchGD is much more stable and efficient than the standard gradient-descent method in handling large images, especially when the compute memory is limited. Code is available at https://github.com/nyunAI/PatchGD.

Published in Transactions on Machine Learning Research (8/2024)
Pushing the Limits of Gradient Descent for Efficient Learning on Large Images
Deepak K. Gupta*
Deepak.gupta@iitione
Tronsmute AI Lab (Temmin Hub), Indian Institute of Technology (ISM), Dhanbad, India
Gowreesh Mago"
gkrma27177@mmiillcoo com
Tronsmude AI Lab (Termin Hub), Indian Institute of Technology (ISM), Dhanbad, India
Arnav Chavan"
amwa..haavon@hyyunacccom
Tronsmute Al Lab (Tezmin Hub), Indian Institute of Technology (ISM), Dhanbad, India Nyun AI, India
Dilip Prasad
dilip. pro.addrrt..roo
Department of Computer Science, UT The Arctic University of Norway,
Tromso, Norway
Rajat Mani Thomas
mmt44003@qator
wwd.comnell.cccoommmmm mma1:0000Bigttrrrmeddd..mddddddd
edu
Department of Physiology and Biophysics, Weill Cornell Medicine--Qaaar
Doha, Qalar
Reviewed on OpenReview: https: // openreview. net/ forum? id= 6dS1 jhdemD
Abstract
Traditional deep learning models are trained and tested on relatively low-resolution images (< 300 px), and cannot be directly operated large-scale images due to compute and memory constraints. We propose Patch Gradient Descent (PatchGD), effective learning strategy that allows us to train the existing CNN and transformer architectures (hereby referred to as deep learning models) on large-scale images in an end-to-end manner. PatchGD based on the hypothesis that instead performing gradient-based updates on an entire image at once, should be possible to achieve a good solution by performing model updates on only small parts of the image time, ensuring that the majority of covered over the course of iterations. PatchGD thus extensively enjoys better memory and compute efficiency when training models large-scale images. PatchGD is thoroughly evaluated PANDA, UltraNNIST, TCGA, and ImageNet datasets with ResNet50, MobileNet ConvNeXtV2, and DeiT' models under different memory constraints. Our evaluation clearly shows that PatchGD is much more stable and elficient than the standard gradient-deseent method in handling large images, especially when the compute memory is limited. Code available at https://eee https://github.co http:/ggthhub.ooonnnneee htt.:/ooommyuuA//aaacccai P/
1
Introduction
In the field of computer vision, deep learning models have emerged the fundamental framework for advanced feature extraction, significantly outperforming traditional algorithms. For a comprehensive overview of the
Equal contribution.

---

Published in Transactions on Machine Learning Research (8/2024)
Figure 1: Example nanoscopy image (left) of a mouse kidney cryo-section approximately 1/12th of the area of single field-of-view of the microscope, chosen to illustrate the level of details at different scales. The bottom right images show that the smallest features in the image of relevance can be as small as a few pixels (here 5-8 pixels for the holes) (Villggs-
Hernández et al. 2022
0600 5070
Accuracy (in fraction)
Figure 2: Performance comparison of standard CNN and PatchGD (ours) for the task of classification of Ul- traMNIST digits of size x12x512 pixels using ResNet50 model. Two different computational memory budgets of 16 GB and 4GB are used, and is demonstrated that PatchGD is relatively stable for the chosen image size, even for very low memory compute.
0.6
0.4
Gradient descent (16 GB)
PatchGD (16 GB)
0.2
Gradient descent (4 GB)
PatchGD (1 GB)
20
40
60
80
100
Epochs
evolution of the associated models and their applications across wide range of scientific domains, we refer
to the reviews presented in Khan et al. (2020): Li et al. (002aa) Alzubaidi et al. (2021) 0g Khan et al. (022
Shamshad et al. (222))
With the recent lechnological developments, very large images are obtained from domains like microscopy
Schermelleh et al.) 2019), medical imaging (Aggarwal al. 2021). and earth sciences (Hunng et al., 2018 Amani al.) 2020), and the challenge of using deep learning models on big data to analyze such images is immense. For example, images obtained from high-content nanoscopy | can be large as 6000 x 6000 or even more (Villegas-Heránnde
[Khater et al.
2020
et al. 2022), with the smallest scale of features being only few pixels in size. Clearly, processing these large images with such fine details prohibits the use of any image downsampling algorithm, and using existing models on such high-dimension images is computationally infeasible.
Most prevailing deep learning models are trained on datasets such as ImageNet, which mainly comprise of low-resolution (< 500 pixels) images. Most research efforts have focused on achieving stale-of-the-arl performance these datasets. However, applying these models to high-resolution images results in a quadratic increase in activation size, requiring significantly more training compute and memory. Morcover, limited GPU memory makes it impractical to process such large images with such models.
This paper introduces a novel training framework for deep learning models aimed at handling very large images. The definition of "large images" depends on the available computational memory for training. For instance, training ResNet50 model with 10, 000 10, 000 size images would be challenging with a 48 GB GPU memory, but training the same model with 512 512 size images would be feasible with 12 GB GPU memory, However, when limited to 4 GB GPU memory, even 512 512 size images may be considered too large.
Figure 2 illustrates the above issue on the task of classification of the UltraMNIST digits (Gupta et al., 2022 into one of the 10 predefined classes labeled from 0-9. More details on the UltraMNIST dataset and classification problem are provided in the supplementary material. The semantic relationship between different parts of the images and the large variation in spatial feature size makes this problem difficult for traditional models, particularly when dealing with large image sizes or low procossing memory. In this study,
refer Figure in supplementary material
2

---

Published in Transactions on Machine Learning Research (8/2024)
focus on images of size 512 x 512 pixels and examine the problem under two computational memory budgets: GPU memory limits of 4 GB and 16 GB.
For the base model, we use 50 (He et al. 2016) and employ the standard training approach. We refer to it as Gradient descent (GD). Note that the term GD is used here as a generic notation to refer to the class of gradient-based optimizers popularly used in deep learning (such as SGD, SGD with momentum (Bengio al. 2013) and Adam (Kingma and Ba 2014), among others), and it is not necessarily restricted to stochastic gradient descent method. For the results demonstrated in Figure 2 used the Adam optimizer. We also present results using the proposed training method, called Patch Gradient Descent (PatchGD), which is a scalable training approach for building neural networks with large images, low memory compute, or both.
PatchGD's effectiveness is demonstrated in Figure 2 where it outperforms GD in both 16 GB and 4 GB memory limits. The performance difference 4% at 16 GB but significantly increases to 13% at GB, simulating rcal-world challenges with large images. Training a ResNet50 model with 512 512 images using only 4 GB memory leads to inferior performance, as shown in Figure 2 However, PatchGD is stable even at this low memory regime, and this can be attributed to its design which makes it invariant to image size to large extent. We explain the method in more detail later and present experimental results on various image sizes, highlighting PatchGD's ability to adapt existing CNN models for large images with limited GPU memory.
Contributions.
We present Patch Gradient Descent (PatchGD), a novel strategy to train neural networks on very large images in an end-to-end manner. PatchGD is an adaptation of the conventional feedforward- backpropugation optimization.
Due to its inherent ability to work with small fractions of given image, PatchGD is scalable on small GPUs, where training the original full-scale images is not possible.
PatchGD reinvents the existing training pipeline for deep learning models to large images in very simplified manner and this makes it compatible with any existing CNN architecture any conventional gradient-based optimization method used in deep learning. Morcover, its simple design allows it to benefit from the pre-training of the standard CNNs on low-resolution data.
2
Related Work
This paper seeks to enhance the ability of existing deep-learning models to handle large images. Previous research in this direction is scarce, with most studics focusing on histopathological datasets, which are a popular source of large images. Many of these studies rely on pixel-level segmentation masks, which are not always available. For instance, Tizuka et al. (2020) : Liu et al. (2017) use patchwise segmentation masks perform patch-level classification on whole slide images, and then apply an RNN to obtain the final image label. Meanwhile, BBratz et al. (2022) uses goblet cell segmentation masks for patch-level feature extraction. However, these approaches require labeled segmentation data, are computationally expensive, have limited feature learning, and are more susceptible to error propagation.
Another set of methods focuses on building a compressed latent representation of the large imput images using existing pre-trained models or unsupervised learning approaches. For example, Brancati et al. (002 uses a model pro-trained on Imagenet to construct latent block, which is further passod to an attention head to do the final classification. Other similar variants include using a U-net autoencoder to build the latent features (Lai et al. 2022). using encoding strategies involving reconstruction error minimization and contrastive learning (Tellez et al. 2018), and getting stronger latent representations through multi-task learning Tellez et al. (2020) A critical limitation of this class of methods is that the encoding network created from unsupervised learning is not always a strong representative of the target task.
There exist several mothods that use pro-trained models derivod from other tasks as feature extractors and the output is then fod to a classifier. Example methods include using Texture Network (CAT-Net) and Google Brain (GB) models feature extractors (Kosaraju et al. 2022), or additionally using similar
3

---

Published in Transactions on Machine Learning Research (8/2024)
datasets for fine-tuning [Brancati al. 2021). Although these methods gain an advantage from transfer learning, such two-stage decoupled pipelines propagate errors through under-represented features, and the performance of the model on the target task is hampered. In this paper, we propose a single-step approach that can be trained end-to-end on the target task.
Several studies have aimed to identify appropriate patches from large images and utilize them efficiently for image classification. Naik et al. (2020) suggests constructing a latent space using randomly selected tiles, but this approach fails to maintain semantic coherence across tiles and overlooks features spread across multiple tiles. Campanella et al. (2019) considers this
multii-nsstacee learning approach, assigning labels to top-K probability patches for classification. Pinckaers et al. (2022); Huang et al. (2022) propose patch-based training but employ streaming convolution networks. [Sharma et al. (2021) clusters similar patches and performs cluster-aware sampling for whole-slide image (WSI) and patch classification. Cordonnier et al. (2021) uses a patch scoring mechanism and patch aggregator network for final prediction but involves downsampling for patch scoring, potentially leading to the loss of patch-specific features crucial for WSI. Papadopoulos eet al. (202T) progressively increases resolution and localizes regions of interest, disregarding the rest similar to performing hard adaptive attention. DiPalma et al. (2021) trains a high-resolution teacher model and applics knowledge distillation to lowcr-resolution version of the same model. Katharopoulos and Flcuret (2019) performs attention sampling on downsampled images and derives an unbiased estimator for gradient updates. However, their attention method involves downsampling, potentially losing vital information. It is important to note that these methods, utilizing patch selection and knowledge distillation, are independent of our work and can be combined with it. However, such integration beyond the scope of this paper.
With the recent popularity of Transformer- methods, Chen et al. (0022) proposed self-supervised learning objective for pre-training large-scale vision transformers at varying scales. Their method involves a vision transformer that leverages the natural hierarchical structure inherent in WSI. However, their method requires a massive pre-training stage which is not always feasible. Also, their method is specific to WSI rather than more general image classification and involves training multiple large-scale transformors. Our method, on the other hand, largets more general image classification lasks and does not involve large-scale pre-training, rather it directly works over any existing CNN model.
To overcome the memory challenges in model training, systems-lovel methodologies have been explored. Jain et al. (2020); Kirisame el al.] (2020) propose approaches like lensor re-malerialization to expand available memory, which can be used for both baselines and our PatchGD method, making them complementary.
This work primarily focuses on efficiently training large-scale images on low-compute GPUs. Gholami et al. 2018 ; Dryden et al. (2019); Oyama et al. (2020) address the image size issue by scaling deep learning models across multiple GPUs. There exist other model parallelism methods that can help to handle larger models or activations (Rajbhandari
al.] 2020 Shoeybi et al.) 2019). These approaches distribute a model across multiple GPUs, effectively managing memory issues by leveraging the collective memory capacity. However, PatchGD is designed to help models operate under low memory settings and it helps to push the limits of each single GPU beyond its conventional capacity. Thus, PatchGD and model parallelism are orthogonal techniques that can be combined to further enhance the performance, allowing for the distribution of large models across multiple GPUs while maximizing the utilization of each GPU's capacity through PatchGD.
Two approaches in the current literature enable training larger model on low-memory GPU, activation checkpointing (Chen el al., 2016) and activation oflloading (Rhu et al., 2016 Cui et al. 2016). Checkpointing saves only the necessary activations during forward propagation and recomputes the remaining from the saved ones needed during backpropagation. Offloading shifts part of the activations to the CPU when the GPU memory starts to approach its full capacity and retrieves them when needed. While both these approaches have been effective, they lead to a significant. increase in latency in the learning process. More importantly, the gain in batch size during training is significantly lower when compared to PatchGD, We shed light on both these aspects through numerical experiments later in this paper.
4

---

Published in Transactions on Machine Learning Research (8/2024)
Patching
patches
Update Z with position vector
Repeat
Iniiiaize emply
Pipeline for the filling of Z block, also referred Z-filling
Model Update
Panching
Sample le and posiiin vectors
position
Model Inference
module using
82
Backprepopation
02
(b) Model update and model inference.
Figure 3: Schematic representations of the pipelines demonstrating working of different components of the PatchGD process.
3 Approach
3.1 General description
Patch Gradient Descent (PatchGD) is novel approach for training deep learning models with high-resolution images. It's a modification of the standard seofforwand-bcckkppagation mothod. PatchGD is built on the hypothesis that, instead of applying gradient-based updates to the entire image simultaneously, similar results can be achieved by updating the model in small image segments, while ensuring the full image coverage over multiple iterations. Even if only portion of the image is used in each iteration for gradient updates, the model is still trainable end-to-end with PatchGD.
Figure 3 presents the schematic representation of the PatchGD pipeline. The central idea behind PatchGD is to construct the Z block, which is a deep latent representation of the entire input image. Although only a subset of the input used to perform model updates, captures information about the entire image by combining information from different parts of the image acquired from the previous update steps. Figure Ba illustrates the use of the block, which is an encoding of an input image X using a model parameterized by weights 0,. The input image is divided into n number of patches, and each patch is processed s, such that each patch in the input
independently using 0.. The size of Z is always enforced to be space corresponds to the respective x 1 x segment in the Z block.
The filling of Z is carried out in multiple steps, with each step involving the sampling of k patches and their positions from X and feeding them to the model as a batch for processing. The output from the model along with the corresponding positions are then used to fill the respective parts of After sampling all n x n patches of X, the completely filled Z obtained. PatchGD utilizes this concept of Z Z-flling during both training and inference stages.

---

Published in Transactions on Machine Learning Research (8/2024)
To create an end-to-end model, we incorporate small subnetwork that consists of conrolatiional/sllaaatettttt and fully-connected layers. This subnetwork processes the information contained in Z and converts it into a o-dimensional classification probabilities. It is worth noting that the computational cost of adding this small subnetwork is minimal. The Figure 3b illustrates the pipelines for both model training and inference stages. During training, the model components 01 and 0g are updated. At every model update step, only a fraction of the patches are updated in while the rest retained from the previous state the model. Subsequently, we use the partially updated to calculate the loss function value and update the model parameters using backpropagation.
Why docs PatchGD work? PatchGD operates on the principle that patches remain independent until the Z block, with no interactions required prior to this stage. Unlike conventional methods, information exchange at patch boundaries is deferred. Nevertheless, the negligible loss incurred as a result of this delayed interaction is minimal in comparison to the patch size. This effect diminishes further when applied large images and correspondingly large patches.
3.2 Mathematical formulation
In this section, we present a detailed mathematical formulation of the proposed PatchGD approach. For simplicity, we describe it in the context of image classification.
Let So R-MxXxC + Ie denote a model parameterized by that takes an input image X of spatial size M M and C channels and computes the probability of it belong each of the e pre-defined classes. To train this model, the following optimization problem is solved.
min L(f(0:X),)),
(1)
where X,y E D represents the data samples used, and L(-) represents the loss function. Conventionally, this problem is solved using mini-batch gradient descent method where at every step, the model weights are updated using the average of gradients computed over a batch of samples, denoted here as S. Based on this, the model update at the stlsstp is =00=1)11]
dl(x)
& B
(2)
d
X
where a and B denote the learning rate and the size of the batch used, respectively. As can be seen in Eq. 2 if the size of image samples 8 E S is very large, it will lead to large memory requirements for the respective activations, and under limited compute availability, only small values of B, and sometimes not even a single sample, fit into the GPU memory. This should clearly demonstrate the limitation of the gradient descent method when handling large images. This issue is alleviated by PatchGD approach.
Algorithm Filling of the Z block (referred as Z-filling)
Input: Input image X E MxMC Pre-trained feature extractor fo,; Patch size p, (M/p) Initialize: Z E Rmxmxx requires_gradients
False
repeat
Na.b +- patch (xtractor(X,a.b)
# Extract patch at location (a,b)
Xa.6 E DXxxxC
2n,b +- So, (Xa,)); 2, E Rxxxx
# Extruct embedding from each patch # U Update the positional embedding
Z(a, -Zabb
until all patches sampled
Return Z =0
PatchGD. As described in Section 3.1 PatchGD avoids model updates on an entire image sample in one go, rather it computes gradients using only part of the image and updates the model parameters. In this regard,

---

Published in Transactions on Machine Learning Research (8/2024)
Algorithm 1 Model Training for 1 iteration
Input: Batch of imput images X' E BB Pre-trained feature extractor fo,, Classifier head 90. Patch size p, Inner iterations ç, Patches per inner iteration k, Batch size B, Learning rate a, Grad. Acc. steps
#M² + p² Ç x k
2: Initialize: Z t- 08xmxmxe, U₁ <- 0, U2 < 0
3: Z + Z-filling(X (X, .p) forall X E X True
4: Sounures_graient((ff
5: for j:: to do
for in X' do
(Px.j. u} t paach_ sampler(X,
# Sample k patches from each image
Px, E PRxxxxxx
2 So (Px.))
# Extruct embedding from each palch # Update the positional embeddings # Classifier prediction over updated Z
10:
Z(c] t-2
11:
Yprod + 90, (Z)
12:
C calculate calculate_loss(( pyrred))))
13:
U. t U1 +1 U2 + U2 + dc/d02
# #Acummulate gradients across batch of patches
14:
end for
15:
if j%b 0 then
16:
U1 U1/e, U2 +- U2/c
17:
0, - 0 aU1, 02 t 02 - uU2
# Update weights every inner iterations
18:
U - 0, U2 +0
19:
end
20:
end for=0
the model update step of PatchGD can be stated as
dC(x,)
( ((:--1)
a L XES, PEPX. doa.-11
Σ
k B
Here, i refers a mini-batch iteration within a certain epoch. Further, j denotes the inner iterations, where at every inner iteration, k patches are sampled from the each input image X X (denoted as Px.5) and the gradient-basee updates are performed as stated in Eq. Note that for any iteration i, multiple inner iterations are run ensuring that the majority of samples from the full set of patches that are obtained from the tiling of X are explored.
In Eq. 3. (1.0) denotes the initial model for the inner itcrations on S; and is equal to ((i-1,), the final model state after ( inner iterations of patch-level updates using S,-1- For a more detailed understanding of the model update process, please see Algorithm T As described earlier, PatchGD uses an additional sub-network that looks at the full latent encoding for the input batch X. Thus, the parameter set 0 is extended 0 6,], where the base CNN model and the additional sub-netwond
fe, and g0,, respectively.
Amgorithm 1 describes model training over one batch of B images, denoted as X E BBxMXMXC As the first step of the model training process, Z corresponding to X is initialized. The process of filling of Z is described in Algorithm 2 For a patch indexed by position v and denoted as X,, the respective Z(u] is updated using the put obtained from fer- Note here that 0 loaded from the last state obtained during the model update on the previous batch of images. During the filling Z, no gradients are stored for backpropagation.
Next, the model update process is performed over series of inner-iterations, where at every step j e 21, & patches are sampled per image in X and the respective parts of Z are updated. Next, the partly updated Z is processed with the additional sub-network 02 to compute the class probabilities and the corresponding loss value. Based on the computed loss, gradients are backpropagated to perform updates of 0, and 0z. Note that we control here the frequency of model updates the inner iterations through an additional term c. Similar to how batch size of mini-batch gradient descent introduces noise and adversely affects the convergence process, we observed that gradient update per inner iteration leads to sometimes poor convergence. Thus, introduce gradient accumulation over inner steps and
7

---

Published in Transactions on Machine Learning Research (8/2024)
Table 1: Performance scores on the UltraMNIST dataset with images of size 512 512 obtained us- ing ResNet50 architecture. Patch size for PatchGD is 256.
Table 2: Performance scores on the UltraMNIST datasel with 512 x 512 images obtained using Mo- b2 architecture. Patch size for PatchGD 256.
Method
Batch size
Memory (GB)
Acc. %
Method
Batch size
Memory (GB)
Acc. %
GD
27
16
65.2
GI)
30
16
67.3
GD
26
16
50.5
GD
16
64.3
PatchGD
16
69.2
PatchGD
83.7
GD
2
4
53.6
GD
4
67.7
GD
2
4
52.5
GD
4
60.0
PatchGD
4
63.1
PatchGD
10
4
74.8
update the model accordingly. Gradients are allowed to backpropagate only through those parts of Z that are aclive the jth inner-iteration. During inference phase, Z filled using the optimized fa; as described Algorithm 2, and then the filled version of Z is used compute the class probabilities for input X using 90:
4
Experiments
We showcase the effectiveness of PatchGD) on benchmark datasets with large images and multiple scales, and ablation studies on multiple models, datasets and use cases.
4.1 Experimental setup
Datasets. We perform thorough evaluation on two datasets, UltraMNIST (Gupta al., 2022 and PANDA (Blltnn al. 2022). ; and also conduct additional experiments using TCGA-NSCI.C (Chen al. 2022 and ImageNet Deng et al. 2009) datasets. Details about the datasets are presented in the supplementary part of the paper.
CNN models. We assess PatchGD on ResNet50 and MobileNet architectures. ResNet50 serves as a backbone for diverse computer vision tasks, while MobileNetV2 is a lightweight architecture for edge devicos. We also conduct experiments with ConvNext V2, a stale-of-the-arl vision model, as well as provide preliminary results for generative modeling.
Implementation details. We employ consistent hyperparameters throughout experiments and report classification accuracy for UltraNNIST and ImageNet tasks and additionally Quadratic Weighted Kappa (QWK) F on the PANDA dataset. For TCGA-NSCI,C, we comply with the previous baselines and report the mean and standard deviation of AUC across a 10-fold cross-validation set. Both the baselines and PatchGD are implemented using PyTorch. We consider GPU memory constraints of 4GB, 16GB, and 48GB to simulate common limits and measure latency on an NVIDIA 40GB A100 GPU and an NVIDIA 24GB L.4 GPU. Additional details are described in the supplementary material.
4.2 Results
UltraMNIST classification. The performance of PatchGD UltraMNIST has already been shown in Figure 2 More detailed results are presented in Tables D and 2 For both the architectures, PatchGD improves the standard gradient descent method (abbreviated GD) by large margins. Our approach employs additional sub-network 90., and it be argued that the gains reported in the paper are due to it. For this purpose, extend the base CNN architectures used in GD and report the respective performance scores in Tables D and 2 as GD*
For both architectures, PatchGD outperforms GD as well as GD* by large margins. For ResNet50, the performance difference even higher for low memory constraint. At. 4 GB, while GD seemss unstable with
2QWK: metric histopathological images and PANDA dataset.
3GD refers to baseline being extended with the same sub-network geg
8

---

Published in Transactions on Machine Learning Research (8/2024)
Table 3: Performance scores obtained using Resnet50 on PANDA dataset for Gradient Descent (GD) and
Patch Gradient Descent (PatchGD).
Method Resolution
Patch Size
Batch Size
Mem.
(GB)
Throughput (imgs/sec)
Accuracy %
QWK
Baseline
512
27
16
618.05
44.4
0.558
PatchGD
512
128
521.42
44.9
0.576
PatchGD
512
64
200
16
341.87
52.1
0.616
Baseline
2048
16
39.04
34.8
0,452
PatchGD
2048
128
32.52
53.9
0.627
Baseline
2048
6
48
39.04
49.4
0,625
PatchGD
2048
128
56
48
32.52
56.2
0.667
Baseline
1096
48
9.23
50.0
0.611
PatchGD
4096
256
26
48
9.62
59.7
0.730
Table 4: Comparison of PatchGD (ResNet50 architecture) with cxisting methods at 4096 image size and
48GB memory constraint.
Method
Accuracy
QWK
HIPT Chen et al. 2022
34.8
0.388
HIPT--
49.3
0.531
ABNN Brancati et al
2021
48.2
0.593
C2C Sharma et al.
2021
50.9
0.668
PatchGD
59.7
0.730
performance dip of more than 11% compared to the 16 GB case, PatchGD is significantly more stable. For MobileNet V2, the difference between PatchGD and GD is even higher at 16GB case, thereby clearly showing that PatchGD blends well with even light models such as MobileNet For MobileNet there is no drop in model performance when going from 16 GB to 4 GB, which demonstrates that MobileNet can work well with GD even at low memory conditions. Nevertheless, PatchGD still performs significantly better. The underlying reason for this gain can partly be attributed to the fact that since PatchGD facilitates operating with partial images, the activations are small and more images per batch are permitted. We also observe that the performance scores of GD* are inferior compared to even GD. ResNet50 and Mobilenet V2 are optimized architectures and speculate that the addition of plain convolutional layers in the head of the network is not suited due to which the overall performance is adversely affected.
Prostate Cancer Classification (PANDA). Table 3 presents the results obtained on PANDA dataset for three different image resolutions. For all experiments, we maximize the number of images used per batch while also ensuring that the memory constraint not violated. For images 512 x 512, we see that PatchGD, with patches of size 128 x 128, delivers approximately the same performance as GT) (for both accuracy as well as Quadratic Weighted Kappa (QWK) metric at 16 GB memory limit. However, reducing the patch size and thus increasing the batch size leads to very sharp gain in the scores of PatchGD. For similar memory constraint, when images of size 2048 2048 pixels are used, GD scores approximately 10% lower while PatchGD shows a boost of 9% in accuracy.
Two factors contribute to the performance gap between GD and PatchGD, Firstly, GD faces a bottleneck with batch size due increased activation size higher-resolution images, allowing only image per batch. Gradient accumulation and hicrarchical training were explored but did not improve performance significantly. Increasing the memory limit helped mitigate the issue of using only image per batch. Secondly, the optimized receptive field of ResNet50 is not well-suited for higher-resolution images, resulting in suboptimal performance. PatchGD demonstrates superior accuracy and QWK compared to GD on the PANDA dataset. when handling large images end-to-end. In terms of inference latency, PatchGD performs comparably to GD. The smaller activations in PatchGD offset the slowness caused by chwce image processing. PatchGD shows potential for real-time inference in applications requiring large image handling.
We further present a comparison of PatchGD with the existing methods designed for handling large images, and the results are presented in Table 4 We used Resnet50 at. 4096 image resolution and 48GB GPU
9

---

Published in Transactions on Machine Learning Research (8/2024)
memory constraint for training. Note that almost all works that exist on handling large images are not designed to work with memory constraints, and if put in such applications, these lead to unstable performance scores. For example, although the vision transformer backbones of HIPT are pre-trained on large medical datasets, the performance of the model in the momory-constrained setting is lowest among the 4 methods presented in the table. For HIPT, all the layers of the vision transformer backbones are trainable, and a batch size of only fits in the memory. This is the primary reason for the significant drop in the performance of the method. The original HIPT model is trained with large batch sizes over a set of GPUs, however, in our muenory-constranned setup, it is not possible. The performance of ABNN and C2C is relatively better, however, they are still significantly lower than the PatchGD training of simple architecture. C2C employs attention modules in the head of the network, and we believe with such additions, the performance of PatchGD could be boosted even further. Nevertheless, we see from the presented results that for memory-cons(raineed settings, PatchGD performs significantly better than any other existing method when it comes to handling large images.
Since HIPT uses a transformer model, one possible way to enhance its performance under low memory setting is to use layer normalization and implement gradient accumulation over a series of iterations. We conducted an experiment with gradient accumulation over 12 steps, referred to as HIPT-L in Table This led to an equivalent batch size of 60, Although the convergence was slow, the performance of the model boosted from 34.8 to 49.3. This clearly demonstrates that transformers with gradient accumulation could work well even at low batch sizes. Nevertheless, we still see a significant performance gap of more than 10% between HIPT and our approach. Moreover, transformers are known to be data-hungyy and one important thing to note here is that the pre-trained HIPT model are using in this paper is already heavily trained a very large medical datasct comprising training images from variety of medical datasets. On the contrary, our model is only pre-trained on standard ImageNet and no additional pre-training is done. This clearly makes our approach stand out when compared to HIPT in the sense that it is applicable for low memory as well as relatively low training data regimes as well.
TCGA-NSCLC classification with no pre-training. To further demonstrate the elficacy of PatchGD on established large image benchmarks, we study the task of TCGA-NSCLC dataset classification and compare our solution with popular approachos such as HIPT (Chen et al. 2022) and CLAM-SB Lu et al. 2021) among others. Related results are presented in Table 5 Note here that the HIPT model uses 3-stage Transformer an external large-scale histopathologica dataset first and then fine-tuned on TCGA-NSCLC data. Further, it uses the images at a gigapixel scale, Further, CLAM-SB uses a multistage processing approach where a segmentation map is first obtained, followed by creating embedding of small patches. Attention pooling is then used to assign weight to each patch which together are then served to a classification model. The other basclines listed in Table 5 similarly also use multistage processing with additional pretraining done on external datasets for boosted discriminative power.
model with a VIT backbone which is pre-trained
For our study on PatchGD, we employed lightweight ConyNeXt- Tiny model without any pretraining external histopathological datascts. Initially, we established two basclines (Baselinc-1 and Bascline-2) by training the model using standard backpropagation techniques. We then applied the PatchGD methods for training. Despite using only 4K images, no external pretraining, and a simple tiny backbone, PatchGD outperformed HIPT and other methods designed for handling large images, demonstrating its superiority. Notably, Basclinc-2 also outperformed GCN-MIL and MIL, highlighting the offectiveness of the ConvNeXt architecture and the simplicity of the training process. We believe that incorporating pre-training on external data, done in HIPT, could further enhance the performance of the PatchGD solution.
PatchGD with transformer architecture. The simplicity of our PatchGD approach allows its easy integration with transformer architecture as well. We demonstrate its compatibility with transformer architectures through experiments on the ImageNet dataset using DeiT-Tiny architecture. Results related to these are presented in Table We observe that PatchGD couples well with the chosen model and leads to improved performance over the respective baselines obtained using conventional training. Note that the
baseline is optimized using the high-performing training procedure described in Touvron et al. (022) and the performance is further optimized across 3 different image resolutions.
10

---

Published in Transactions on Machine Learning Research (8/2024)
Table 5: Performance comparison PatchGD with HIPT and other baseline models for the task of TCGA-
NSCLC subtyping dataset.
Method
Model
Image Size
AUC
Standard Deviation
Baseline-1
ConvNeXt-V2 Tiny
78.0
3.7
GCN-MIL Zhao et al.
2020
VAE-GAN = Graph CNN
83.1
3.4
MIL [Lu et nl. 2021 2021
89.2
4.2
Baseline-2
ConvNeXt-V2 Tiny
4096
90.4
4.3
DS-MIL [Li ot al.
2021b
Patching +
Resnet18 1 Aggregation
92.0
2.4
CLAM-SB Lau et al 2021
Patching + Resnet50 + Attention Clustering
92.8
2.1
HIPT Chen et al.
Patchiug 3 Transformer
95.2
2.1
PatchGD
ConvNeXt-V2 Tiny
4096
97.0
1.7
Table 6: Accuracy Comparison for ImageNet Deit Tiny architecture with Gradient Descent and PatchGD.
PatchGD
# Classes
# Samples / Class
Baseline Accuracy (%)
Accuracy (%)
221
384
512
512
25
100
85.76
88.68
88.72
90.71
25
200
88.41
90.32
90.16
92.12
25
500
90.08
92.00
92.18
93.14
25
1000
91.28
93.12
93.20
95.44
10
100
85.40
88.40
25
100
88 88.68
90.74
100
100
76.90
78.20
500
100
73.65
70.82
For transformer backbones, we have observed that the performance of the model better when the head is also a transformer rather than a CNN model. Table 7 presents a comparison of CNN and transformer heads for the classification task on the PANDA dataset. For the transformer head, we use single multi-headed self-attention layer with 2 heads each of 192 channels followed by linear layer. The CNN head uses 3 conv-relu-bn blocks with kernel size of 3 x 3 and 256 channels followed by a linear layer. We consistently see that the transformer head works better.
Handling natural images (ImageNet). To understand how PatchGD works with natural images, we study its performance on ImageNet dataset for different choices of the number of classes as well as the number of samples per class. This follows the results discussed the earlier section. We conduct these experiments using DeiT-Tiny transformer architecture and the results are reported in Table To study the effect of the number of samples, we fix classes to 25. Interestingly, we observe that PatchGD outperforms the standard training approach by around 2% accuracy, significant improvement in the context of ImageNet training.
We further examined the performance of PatchGD across different numbers of classes, keeping the number of samples per class fixed at 100 (Table 6 Interestingly, PatchGD outperformed the baseline approach when dealing with fewer classes. However, when the number of classes increased to 500, the baseline method performed better. This discrepancy arises bocause, for low-resolution images such as those the ImageNet dataset, the small information loss at the edges of the patches becomes significant when there many classes and limited samples per class. Our initial findings suggest that this issue be mitigated to some extent by using overlapping patches, although this increases computational demands. Nonetheless, our observations indicate that PatchGD is the preferred choice for natural images in low-data regimes.
PatchGD vs. Activation Checkpointing vs. Activation Offloading. As has been described throughout this paper, PatchGD aims at better utilization of GPUs, by facilitating to training of deep learning models with larger images (leading to larger model activations) on smaller GPUs. Two other popular approaches aiming at fitting larger model smaller GPUs activation checkpointing (Chen et al. 2016) and
11

---

Published in Transactions on Machine Learning Research (8/2024)
Table 7: Performance scores obtained using DeiT-Small on PANDA dataset for Gradient Descent (GD) and Patch Gradient Descent (PatchGD).
Method
Resolution
Patch Size
Batch Size
Head
Mem. (GB)
Accuracy %
QWK
Bascline
512
22
16
48.4
0.596
PatchGD
512
128
136
CNN
44.9
0.576
PatchGD
512
128
136
Transformer
48.7
0.599
Baseline
2048
48
48.6
0.612
PatchGD
2048
128
CNN
48
48.9
0.589
PatchGD
2048
128
32
Transformer
48
57.4
0.702
presents a comparison of PatchGD with these methods. We present the comparison of ResNet550 architecture on PANDA datasct different image resolutions on an NVIDIA 16 GB L4 graphies card. For gradient checkpointing, employ chunk sizes of and 6.
activation offloading RRhu al. 2016). Table
PatchGD outperforms checkpointing and offloading approaches, particularly with 2K resolution images, where the margin of superiority is significantly larger. Under the selected memory constraint at this resolution, both baseline methods can only handle a maximum batch size of per iteration, with activation offloddigg managing only 2. In contrast, PatchGD can handle batch sizes 14. For smaller images, all methods can increase the batch size, but PatchGD still delivers the best performance. This clearly demonstrates that PatchGD is more effective in utilizing GPU resources. Additionally, is worth noting that PatchGD, checkpointing, and offloading orthogonal methods and be combined to fit larger models smaller GPU resources.
Table Performance comparison of PatchGD against, Activation Offloading (Rhu al. 2016) and Activation Checkpointing (Chen al. 2016) on PANDA dataset using ResNet50 model and NVIDIA L4 16 GB GPU,
Method
Image
Batch Size
Peak Memory (GB)
Training time / image
iteration(ms)
Accuracy
Boseline
2048
8.6
716
34.8
Activation Off-londing
2048
14.6
2603
42.1
Gradient Checkpointing, chunks=4
2048
14.1
1007
46.0
Gradlent Checkpointing, chunks=6
2048
13.1
998
48.0
PatchGD
2048
14
15.1
930
56.2
Baseline
G12
27
14.6
42
44.4
Activation Off-loading
512
32
14.9
161
46.8
Gradient Checkpointing, chunks=d
512
52
14.8
59
46.2
Gradient Checkpointing, 6hunks=6
512
72
14.6
59
44.7
PatchGD
612
200
14.7
79
52.1
Additional study. In this section, we show some additional experiments to further prove the advantages of PatchGD. Training recipes and hyperparameters are provided in the supplementary material.
Role of ind-to-end training. Table g| shows that freezing the backbone leads to reduced performance, highlighting the key role of end-to-end training in PatchGD, Other existing methods can fine-tune the network
Table 9: Performance scores on PANDA dataset (2048 x 2048) at 24 GB memory budget for differ- ent choices of the feature extractor: pretrained and frozen on ImageNet, trained on PANDA using GD and frozen, and fully trainable.
Table 10: Performance scores for ConvNext V2 on PANDA dataset.
Meshod
Image Size
Batch Size
(GB)
Acc. %
PatchGD
2048
40
49.3
Fenture extractor
Batch Size
QWK
Accuracy%
Raseline
2048
45.1
PatchGD
44.0
ImageNet (frozen)
50
0.538
44.6
Baseline
512
50
16
43.9
PANDA (frozen)
50
0.612
50.0
40
0.662
56.0
12

---

Published in Transactions on Machine Learning Research (8/2024)
end-to-end, but only on low-resolution images, whereas PatchGD enjoys fully-trainable end-to-end training even at higher resolutions.
Additional architectures. Beyond the experiment on TCGA-NSCLC task, we also conducted an additional experiment with ConvNext- V2, a state-of-the-ar image classification model, on PANDA dataset and the results are presented in Table IQ. PatchGD outperforms the baseline at higher resolution (2048) while performing competitively at a relatively low resolution too (512). This shows that PatchGD can take advantage of the higher representation power of newer CNN architectures.
On aftention-based head module for CNNs. We have shown in the paper the working of a CNN backbone with a CNN head as well as Transformer backbone with a Transformer head. Here, we study whether using & head with atlention module could be beneficial for the learning of CNNs. For this purpose, we experimented with ResNet50 backbone and PatchGD and we replaced the CNN head with an attention-based MLP to see the effect. For the attention head, we employed a single-layer multi-head attention module with 64 heads, and for each pixel of the latent corresponding to a loken, we concatenated trainable CLS token for final classification. Additional details are presented in the supplementary material. Compared to the base performance accuracy of 52.1% with CNN head, the accuracy of the model with the attention head improved to 53.6%. This clcarly shows that using an attention module can help to enhance the performance of PatchGD results. Further, we anticipate that for larger images, where the spatial size of the L1-block is larger, this improvement will be even more.
Hyperparameter study. Table 11 shows the impact of patch sam- pling on PatchGD's performance. We find that using smaller sampling fractions per inner itcration leads to improved accu- racy, which is counter-intuitive since smaller fractions provide less image context. This behavior may be attributed to regular- ization noise induced by smaller patch batch sizes, benefiting the convergence process. However, further research is needed for comprehensive understanding. Additionally, the fraction of the image seen in one pass does not significantly affect per- formance except when it is low, insufficient context impedes convergence.
Table 11: Sampling ablation on PANDA
dataset. Memory limit is 16 GB, Image size and patch size are 2048 and 128 respectively.
Sampling
Max Sampled
Accuracy % QWK
50
100
42.3
0.538
30
100
49.9
0.613
10
100
53.9
0.627
10
70
53.1
0.624
10
50
53.9
0.622
10
30
51.1
0.610
We also explore the influence of the gradient accumulation length parameter on PatchGD, and the results are available in the supplementary material. We observe that performing gradient- based updates per inner iteration yields superior performance in our experiment. However, the choice of e depends on the number of inner steps Ç. For large Ç values, values greater than are preferred. For instance, for processing 2K resolution images with a patch size of 128 128, e = ( proves effective. Establishing an empirical relation between { and e is subject for future research. We have also observed that using the models trained with GD as the initial models in PatchGD can improve overall performance. However, there are instances where model training on GD is not possible. In such scenarios, one could use low-resolution models trained on GD or even conventional pre-trained models. Nevertheless, the effect of each of these choices needs to be thoroughly studied.
On other tasks. PatchGD can also be adapted for other tasks such as segmentation and generative modeling, among others. Our early results related to generative modeling look promising and pave way for future research. More details are presented in the supplementary material.
5 Conclusions
In this paper, we introduced Patch Gradient Descent (PatchGD), a novel training strategy for deep learning models that effectively handles large images even with limited GPU memory. PatchGD updates the model using partial image fractions, ensuring comprehensive context coverage over multiple steps. Through various experiments, we demonstrated the superior performance of PatchGD compared standard gradient descent, both in handling large images and operating under low memory conditions. The presented method and
13

---

Published in Transactions on Machine Learning Research (8/2024)
experimental evidence highlight the significance of PatchGD in enabling existing deep learning models to effectively process large images without compute memory limitations.
6 Limitations
While our numerical experiments have showcased the effectiveness of PatchGD, there are still limitations in terms of comprehensively understanding its generalization and stability. Additionally, our method's relative slowness compared standard gradient descent is a minor drawback, particularly when real-time training is crucial. However, this limitation does not affect the inference speed, making bottleneck only in specific scenarios prioritizing real-time training.
Gradient bias in PatchGD. PatchGD introduces gradient bias in both the forward and backward passes, unlike methods such as activation checkpointing and activation offloading. During the forward pass, the bias arises because the classífier operates on stale z-vectors, which are derived from previous iterations. This results in suboptimal feature representations since the z-vectors do not accurately reflect the latest model updates. Unlike activation chockpointing or offloading, which recompute or store exact intermediate activations, PatchGD's dependence on these delayed s-vectors can lead to discrepancies between the computed and true activations.
In the backward pass, gradient bias occurs because some z-vectors do not propagate gradients. This incomplete gradient flow results from PatchGD's 'ss strategy of updating only a subset of veveors during each iteration. Additionally, due to overlapping receptive fields, neighboring patches can influence these z-vectors, leading to an uneven gradient propagation and an approximation that deviates from the true gradient.
To mitigate these biases, several strategies can be employed. Using smaller patch sizes reduces the forward pass bias by ensuring that z-vectors are updated more frequently, thereby decreasing the staleness effect. Introducing overlapping patches helps in capturing more accurate gradients by minimizing boundary effects and ensuring more uniform gradient propagation. Incorporating momentum in stochastic gradient descent. (SGD) can help average out the bias over multiple iterations by leveraging historical gradient information to smooth out the noise introduced by the gradient bias.
Empirical evaluations show that while PatchGD offers significant memory savings, the introduced gradient bias results in noisier gradient updates. However, this bias does not significantly impact overall training performance and convergence. The benefits of reduced memory usage and the ability to train larger models with PatchGD outweigh the impact of gradient bias. We acknowledge the presence of this bias and recommend further studies to quantify and refine these strategies, enhancing the effectiveness of PatchGD in training large-scale models efficiently.
7 Future work
This paper has established the foundational concept of patch gradient descent to enable training CNNs using very large images and even when only limited GPU memory is available for training. The results as well insights presented in the paper open doors to several novel secondary research directions that could be interesting in terms of improving the efficacy well as the acceptance of the presented method in broader scientific community. We list some such directions here.
Sealing to gigapizel images al small compule memory. An ambitious but very interesting application of PatchGD would be to be able to process gigapixel images with small GPU memory. We can clearly envision this with PatchGD but with additional work. One important development needed is to extend the PatchGD learning concept to multiple hierarchical Z blocks, thereby sampling patches from the outer block to iteratively fill the information in the immediate inner Z block and on.
Enhanced receptive field. So far, PatchGD has been looked at only in the context of being able handle very large images. However, different side of its use that with almost the same architecture, it builds smaller receptive build, thereby zooming in better. We speculate that in this context,
14

---

Published in Transactions on Machine Learning Research (8/2024)
PatchGD could also help in building better discriminative models with lighter CNN architectures. Clearly, this would be of interest to the deep learning community and needs to be explored.
Broader Impact
The broader impact of this work lies particularly in its potential to extend the capability of deep learning models. By addressing the challenge of training models on large-scale images with limited computational resources, our approach opens up opportunities for researchers and practitioners with constrained hardware setups to tackle complex problems in healthcare, agriculture, and environmental monitoring, where high- rosolution images play crucial role in decision-making processes. Morcover, our approach contribute to reducing the environmental footprint of deep learning by enabling efficient training on low-power devices, thus promoting sustaina bility in the development and deployment of deep learning models. In summary, our work has the potential to empower diverse communities, drive sustainable development, and accelerate scientific progress. It is essential to approach these advancements with a conscientious mindsct, taking into account the broader societal impact and proactively working lowards an inclusive and responsible deployment, deep learming technologies. With our work, it is also important address the potential risks and challenges. Issues related to data privacy, bias, and fairness should be carcfully addressed to prevent any unintended negative consequences. Additionally, the potential for misuse malicious applications of deep learning models should be acknowledged and proactively addressed through robust security measures and ethical guidelines.
References
A. Khan, A. Sohai, U. Zahoora, and A. S. Qureshi. A survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review, 53:5455-5511 2020.
Zewen Li, Fan Liu, Wenjie Yang, Shouheng Peng, and Jun Zhou. A survey of convolutional neural works Analysis, applications, and prospects. IEEE Transactions on Neural Networks and Learning Systems, pages 1-21, 2021a.
L. Alzubaidi, J. Zhang, A. J. Humaidi, A. Al-Dujaili, Y. Duan, ). Al-Shamma, J. Santamaría, M. A. Fadhel, M. Al-Amidic, and L. Farhan. Review of deep learning: concepts, cnn architectures, challenges, applications, future directions. Journal of Big Duta, 8, 2021.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM Computing Surveys, 54(10):200, 2022. doi: 10. 10.1145/3505244.
Fahad Shamshad, Salman Khan, Syed Waqas Zamir, Muhammad Haris Khan, Munawar Hayat, Fahad Shahbaz Khan, and Huazhu Fu. Transformers in medical imaging: A survey. orXie preprint XXX:2220100987. 2022.
URL httph:tp:///ttt statps://arxiv.orrr org/ abs 221/222211...2
Ismail M Khater, Ivan Robert Nabi, and Ghassan Hamarneh. A review of super-resolution single-molecule localization microscopy cluster analysis and quantification methods. Palterns, 13)): 2020.
Lothar Schermelleh, Alexia Ferrand, Thomas Huser, Christian Eggeling, Markus Sauer, Oliver Biehlmaier, and Gregor PC Drummen. Super-resolution microscopy demystified. Nature cell biology, 21(1):72-84, 2019.
Ravi Aggarwal, Viknesh Sounderajah, Guy Martin, Daniel SW Ting, Alan Karthikesalingam, Dominic King, Hutan Ashrafian, and Ara Darzi Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ digital medicine, 4(1):65, 2021.
Yanbo Huang, Zhong-xin Chen, YU Tao, Xiang-zhi Huang, and Xing-fa fa Gu. Agricultural remote sensing big data: Management and applications. Journal of Integrative Agriculture, 1799):1915-1991, 2018.
Meisam Amani, Arsalan Ghorbanian, Seyed Ali Ahmadi, Mohammad Kakooei, Armin Moghimi, S Mohammad Mirmazloumi, Sayyed Hamed Alizadeh Moghaddam, Sahel Mahdavi, Masoud Ghahremanloo, Saeid Parsian, et al. Google carth engine cloud computing platform for remote sensing big data applications: A comprehensive review. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 0455266-5500 2020.
15

---

Published in Transactions on Machine Learning Research (8/2024)
Luis E
Vishesh Dubey, Mona Nystad, Jean- Tinguely, David A Coucheron, Firehun T Dullo, Anish Priyadarshi, Sebastian Acuña, Azeem Ahmad, José M Mateos, et al. Chip-based multimodal super-resolution microscopy for histological investigations of cryopreserved tissue sections. Light: Science & Applications, 11(1):1-17, 2022.
Deepak K. Gupta, Udbhav Bamba, Abhishek Thakur, Akash Gupta, Suraj Sharan, Ertugrul Demir, and Dilip K. Prasad. Ultramnist classification: A benchmark to train cmns for very large images. arXiv, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Compuler Vision and Pollern Recognition, pages 770 778, 2016.
Yoshua Bengio, Nicolas Boulange=-Lwaandowski and Razvan Pascanu. Advances in optimizing recurrent networks. In 2013 IEEE international conference on acoustics, speech and signal processing, pages 8624 8628. IEEE, 2013.
Diederik P Kingma and Jimmy Ba. orXiv:1412.688 6980, 2014.
Adam: A method for stochastic optimization. ar preprinl
Osamu lizuka, Fahdi Kanavati, Kei Kato, Michael Rambeau, Koji Arihiro, and Masayuki Tsuneki. Deep learning models for histopathological classification of gastric and colonic epithelial tumours. Scientific Reports, 10(1):1504, Jan 2020. ISSN 2045-2322. doi: (0.1088/s41598-1 org/ 10111.1000///1118-111
00008//415900000-5446777 020-58467-9, URL https://ddo
58467-9
Yun Liu, Krishna Gadepalli, Mohammad Norouzi, George E Dahl, Timo Kohlberger, Aleksey Boyko, Subhashini Venugopalan, Alcksci Timofcev, Philip Q Nelson, Greg S Corrado, et al. Detecting cancer metastases on gigapixel pathology images. arXiv preprint 1XX:::1033.00 02442, 2017.
Jon Braatz, Pranav Rajpurkar, Stephanie Zhang, Audrew Y Ng, and Jeanne Shen. Deep learning- sparse whole-slide image analysis for the diagnosis of gastric intestinal metaplasia. arXiv preprint arXiv:2201.01449, 2022.
Nadia Brancati, Giuseppe De Pietro, Daniele Riccio, and Maria Frucci. Gigapixel histopathological image analysis using attention-based neural networks. IEEE Access,
2021.
Zhi-Fei Lai, Gang Zhang, Xiao-Bo Zhang, and Hong- Tao Liu. High-resolution histopathological image classification model based on fused heterogeneous networks with self-supervised feature representation. BioMed Rescarch International, 2022:1007713, Aug 2022. ISSN 2314-6133. doi: 00.1155/2022/8007713
Too
URL https://doi org/10 /155/20222/00777777
David Tellez, Geert J. S. Litjens, Jeroen A. van der Laak, and Francesco Ciompi. Neural image compression for gigapixel histopathology image analysis. IEEE Transactions on Pallern Analysis and Machine Intelligence, 43:567-578, 2018.
David Tellez, Diederik Höppener, Cornelis Verhoef, Dirk Grüinagen, Pieter Nierop, Michal Drozdzal, Jeroen Laak, and Francesco Ciompi. Extending unsupervised neural image compression with supervised multitask learning. In Medical Imaging with Deep Learning, pages 770 783. PMLR, 2020.
Sai Chandra Kosaraju, Jeongyeon Park, Hyun Lee, Jung Yang, and Mingon Kang. Deep learning-based framework for slide-based histopathological image analysis. Scientific Reports, 12, Nov 2022. doi: 10.1038/ 411598-02--231 26-6-00. URL https://doi org/10. 0200///44119-002-232--6
Nikhil Naik, Ali Madani, Andre Esteva, Nitish Shirish Keskar, Michael F. Press, Daniel Ruderman, David B. Agus, and Richard Socher. Deep learning-enabled breast cancer hormonal receplor status determination from base-level h&te stains. Nature Communications, 11(1):5727, Nov 2020. ISSN 2041-1723. doi:
10.1038/s41467-0000 19334-3. URL htttp:://aoi.rrg//// http:////1..1110/11156-000000000 19334-3
Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor Reuter, David S Klimstra, and Thomas J Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, 25(8):1301-133) 2019.
16

---

Published in Transactions on Machine Learning Research (8/2024)
H. Pinckaers, B. van Ginneken, and G. Litjens. Streaming convolutional neural networks for end-to-end learning with multi-megapixel images. IEEE Transactions on Pullern Analysis 8; Machine Intelligence, 44 (03):1581- 1590, mar 2022. ISSN 1939-3539. doi: TAL100/TPAAML..2000001539
Shih-Chiang Huang, Chi-Chung Chen, Jui Lan, Tsan- Yu Hsich, Huci-Chich Chuang, Meng-Yan Yao Chien, Tao-Sheng Ou, Kuang-Hur Hua Chen, Ren-Chin Wu, Yu-Jen Liu, Chi-Tung Cheng, Yu-Jen Huang, Liang- Wei Tao, An-Fong Hwu, I-Chieh Lin, Shih-Hao Hung, Chao-Yuan Yeh, and Tse-Ching Chen. Deep neural network trained on gigapixel images improves lymph node metastasis detection in clinical settings. URL
Nature Commanications,, 3341), Jun 2022. ISSN 2041-1723. doi:
https://doon ott...rggg/.// 10001000//444470022223330
Yash Sharma, Aman Shrivastava, Lubaina Ehsan, Christopher A. Moskaluk, Sana Syed, and Donald E. Brown. Cluster-to-conquee A framework for end-to-end multi-instance learning for whole slide image classification. In International Conference on Medical Imaging with Deep Learning, 2021.
Jean-Baptiste Cordonnier, Aravindh Mahendran, Alexey Dosovitskiy, Dirk Weissenborn, Jakob Uszkoreit, and Thomas Unterthiner. Differentiable patch selection for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 2351-2231 2260, 2021.
Athanasios Papadopoulos, Pawel Korus, and Nasir Memon. Hard-attention for scalable image classification. Advances in Neural Information Processing Systems, 34:14694 14707, 2021.
Joseph DiPalma, Arief A. Suriawinata, Laura J. Tafe, Lorenzo Torresani, and Saced Hassanpour. Resolution- based distillation for efficient histology image classificaaton. Artificial Intelligence in Medicine, 119: 102136, 2021. ISSN 0933-36333 doi: /ttps://doo..rgg/ 101.///0....///.0001./...../..// w0t.md..222..1022222...2 URL https://www http:////nenccettrrreee///////// http:///ew..tcooooceeecceeeeea/a
Angelos Katharopoulos and François Fleuret. Procossing megapixel images with deep attention-sampling models. In International Conference on Machine Learning, pages 3282 3291. PMLR, 2019.
Richard J. Chen, Chengkuan Chen, Yicong Li, Tiffany Y. Chen, Andrew D. Trister, Rahul G. Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 16144-16155 June 2022.
Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Picter Abbeel, Joseph Gonzalez, Kurt Keutzer, and Ion Stoica. Checkmate: Breaking the memory wall with optimal lensor rematerialization. Proceedings of Machine Learning and Systems, 2:497-511, 2020.
Marisa Kirisame, Steven Lyubomirsky, Altan Haan, Jennifer Brennan, Mike He, Jared Roesch, Tianqi Chen, and Zachary Tatlock. Dynamic tensor rematerialization arXiv preprint arXiv:2006.00 09616, 2020.
Amir Gholami, Ariful Azad, Peter Jin, Kurt Keutzer, and Aydin Bulue. Integrated model, batch, and domain parallelism in training neural networks. In Proceedings of the 30th on Symposium on Parallelism in Algorithms and Archileclures, pages 77 87-86,20 2018.
Nikoli Dryden, Naoya Maruyama, Tom Benson, Tim Moon, Mare Snir, and Brian Van Essen. Improving strong-scaling of cnn training by exploiting finer-grained parallelism. In 2019 IEFE International Parallel and Distributed Processing Symposium (IPDPS), pages 210-220. IEEE, 2019.
Yosuke Oyama, Naoya Maruyama, Nikoli Dryden, Erin McCarthy, Peter Harrington, Jan Balewski, Satoshi Matsuoka, Peter Nugent, and Brian Van Essen. The case for strong scaling in deep learning: Training large 3d cnns with hybrid parallelism. TEEE Transactions on Parallel and Distributed Systems, 3247)21641-1652, 2020.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Nehworking, Storage and Analysis, pages 1-16. IEFE, 2020.
17

---

Published in Transactions on Machine Learning Research (8/2024)
Mohammad Shocybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXie preprinl arXiv:1909.0003: 08053, 2019.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1004 061 74, 2016.
Minsoo Rhu, Natalia Gimelshein, Jason Clemons, Arslan Zulfiqar, and Stephen W Keckler. vdnn: Virtualized deep neural networks for scalable, memory-effiiiett neural network design. In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 1-13. IEEE, 2016.
Henggang Cui, Hao Zhang, Gregory R. Ganger, Phillip Gibbons, and Eric P. Xing. Geeps: scalable deep learning on distributed gpus with a gpu-specialized parameter server. In Proceedings of the Eleventh European Conference Computer Systems, EuroSys '16, New York, NY, USA, 2016. Association for URL httpp:://oi...oo
Computing Machinery. ISBN 9781450342407. doi: 10.1145 01114///20111.299900
1145/2901318 0445/2991388.29113333 2901323
Wouter Bulten, Kimmo Kartasalo, Po-Hsuan Cameron Chen, Peter Ström, Hans Pinckaers, Kunal Nagpal, Yuannan Cai, David F' Steiner, Hester van Boven, Robert Vink, et al. Artificial intelligence for diagnosis and gleason grading of prostate cancer: the panda challenge. Nature medicine, 28(1):154-163, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the LEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255. Icce, 2009.
Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Mattoo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering, ((6):555-570) 2021.
Yu Zhao, Fan Yang, Yuqi Fang, Hailing Liu, Niyun Zhou, Jun Zhang, Jiarui Sun, Sen Yang, Bjoern Menze, Xinjuan Pan, and Jianhua Yao. Predicting lymph node metastasis using hislopathological images based on multiple instance learning with deep graph convolution. In Proceedings of the LEEE/CVE /CFF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4837 4846, June 2020.
Bin Li, Yin Li, and Kevin W. Eliceiri. Dual-stream multiple instance learning net network for whole slide image classification with self-supervised contrastive lcarning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recognition (CVPR), pages 14318 14328, June 2021b.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357, PMLR, 2021.
Robert Geirhos, Jörn- Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelligence, 2(11): 665-673, 2020.
Supplementary Material
A Datasets
A.1 PANDA
The Prostate c.Ncer graDe Assessment Challenge Bulten et al. (022) consists of one of the largest publically available datasets for Histopathological images which scale to a very high resolution. It is important to mention that we do not make use of any masks as in other aforementioned approaches. Therefore, the complete task boils down to taking an input high-resolution image and then classifying them into 6 categories based on the International Society of Urological Pathology (ISUP) grade groups. There are a total of 10.6K images which are split. into train and test sets in the ratio 80:20.
18

---

Published in Transactions on Machine Learning Research (8/2024)
A.2 UltraMNIST
This is synthetic dataset generated by making of the MNIST digits. For constructing image, 3-5 digits are sampled such that the total sum of digits is less than 10. Thus image be assigned label corresponding to the sum of the digits contained in the image. Each of the 10 classes from 0-9 has 1000 samples making the dataset sufficiently large. Note that the variation used in this dataset is an adapted version of the original data presented in Gupta et al. (202)). with background noise removed so that any shortcut learning is avoided Geirhos et al. (2020). Since the digits vary significantly in size and are placed lar from each other, this datasel fit. well lerms of learning semantic coherence in a image. Moreover, it poses the challenge that downscaling the images leads significant loss of information. While even higher resolution could be chosen, later demonstrate that the chosen image size is sufficient to demonstrate the superiority of PatchGD over the conventional gradient descent method.
3
Figure 4: Sample PANDA and UltraMNIST dataset images used for training PatchGD.
A.3 TCGA
The TCGA-NSCIC dataset, known as The Cancer Genome Atlas-Non-Smin Cell Lung Cancer, encompasses two distinct lypes of lung cancer: Lung Adenocarcinoma (LUAD), with 522 cases, and Lung Squamous Cell Carcinoma (LUSC), with 504 cases, with total number of image files 3220. The data was split in stratified manner using the patient cases, into train and test set in the ratio 80:20, making sure there is no data Icakage from train to test. The whole slide images are used to evaluate the performance of baseline and PatchGD in classifying the lung cancer subtypes.
Training Methodology and Hyperparameters
For Tables 1,2,3,5,7,8,99110 presented in the main paper, all models are trained for 100 epochs with Adam optimizer and a peak learning rate of 1e-3. A learning rate warm-up for 2 epochs starting from 0 and linear decay for 98 epochs till half the peak learning rate was employed. The latent classification head consists of convolutional layers with 256 channels in each. We perform gradient accumulation over inner ilerations for better convergence, in the case of PANDA. To verify if results are better, not because of an increase in parameters (coming from the classification head), basclines are also extended with similar hcad. GD*, for MobileNet UltraMNIST, refers to the baseline extended with this head.
In the case of low memory, as demonstrated in the UltraMNIST experiments, the original backbone architecture is trained separately for 100 epochs. This provides a better initialization for the backbone and is further used in PatchGD as mentioned in Tables 1 and 2.
For bascline PANDA at 2048 resolution, another study involved gradient accumulation over images, which was done for the same number of images that can be fed when the percent. sampling 10% i.e. 14 times since 2048x2048 image with patch size of 128 and percentage sampling of 10 percent can have maximum batch size of 14 under 16GB memory constraint. That is to say, the bascline can virtually process a batch of 14 images. This, however, was not optimal and the peak accuracy reported was in the initial epochs due to the loading of the pre-trained model on the lower resolution after which the metrics remained stagnant. (aceuracy: 31%,1%, QWK:0.357).
For Table 4 presented in the main paper, we use the training strategies as mentioned in the respective works. The training strategy TCGA is similar to what is employed on the PANDA dataset in Table 5.
19

---

Published in Transactions on Machine Learning Research (8/2024)
Figure 5: Sample PANDA images along with their latent space Z. It can be seen that the latent space clearly acts as rich feature extractor.
For the ImageNet experiments in Table 6, follow the exact training recipe given in Touvron et al. (2021]. This includes 300 epoch training regime with cosine decay and combination of multiple image augmentation!]
Convergence of bascline models. For all the baseline experiments reported in the paper, we have also investigated extended training of the baseline to match the training time of the corresponding PatchGD experiments. However, has been consistently observed that the configurations reported the paper are the most optimal and the baseline converged within the initial 100 epochs for all the configurations. This clearly confirms that the gain reported by PatchGD is not due to the additional training time associated with this method.
C On other tasks
Generative modeling. PatchGD can be used for generating large-scale images with broad semantic context, which can be beneficial for data augmentation in fields such as deep learning for medical imaging. Farly
See Table Touvron al. 2021
20

---

Published in Transactions on Machine Learning Research (8/2024)
Table 12: Comparison with normalization techniques at 2048 image size and 48GB memory constraint with Resnet50 backbone.
Method
Batch Size
Setting
Accuracy
BatchNorm
49.4
GroupNorm
Groups = 32
50.3
Grad. Acc.
Steps
44.1
PalchGD
56
56.2
Table 13: Influence different. number of gradient accumulation steps e on the performance PatchGD.
Model
Dataset
Memory
Image
Patch
Accuracy
MobileNetv2
UltraMNIST
16 GB
512
256
83.7
MobileNetv2
UltraNNIST
16 GB
512
256
81.5
MobileNetv2
UltraMNIST
16 GB
512
256
81.1
Resnet50
PANDA
4GB
512
41.9
Resnet50
PANDA
4GB
512
64
50.5
Resnet50
PANDA
4GB
512
32
45.0
Resnet50
PANDA
48GB
4096
256
56.9
Resnet50
PANDA
48GB
4096
256
32
59.7
results using StyleGAN-2 the CIFAR-10 dataset showed that our method generated patches of 16 16 which were stitched together and analyzed by the discriminator, leading to comparable FID score of 6.3 to the standard GD's FID score of 6.1. We believe this small performance gap can be eliminated with hyperparameter optimization. We consider that the potential of PatchGD in generative modeling can be maximized by generating large images with various semantic contexts, although this needs to be explored further.
PatchGD for segmentation. We discuss here how PatchGD be used for tasks such segmentation any other encoder-decoder tasks We have discussed generative modeling already, and since the setup would be something similar, present here an understanding of how the PatchGD formulation would unfold for tasks such segmentation. For the task of segmentation as well, we have two sets of weights 01 and 0g that constitute the encoder and the decoder, respectively. Here, the encoder generates a Z-block and the decoder is used to generate the segmentation map from the Z-block. Similar to the classification problem, PatchGD operates on each image over a course of multiple inner ilerations. At each inner iteration, patches are sampled from image and accordingly passed through and the output then used update the respective parts of Z. Further, & c-dimensional vectors patches that are used to update parts of the segmentation map y, and the process is repeated. Note that similar to 7-filling, this process also requires y-filling before the model updates of the encoder and decoder are performed over patches. For this purpose, we can first train a segmentation model on lower-resolution images of the chosen task and then use its encoder and decoder, and starting models for the PatchGD learning process.
sampled from Z and passed through the decoder to generate mask
D Comparison with normalization techniques
Batch normalization methods also influence the covergence of deep learning models at low batch sizes. However, PatchGD outperforms these techniques as well and we present a comparison is presented in Table
E
Gradient Accumulation Study
We also highlight an ablation study on the effect of changing the gradient accumulation steps e as presented in Table 13
The gradients accumulated and weights updated only after steps. The ablations were
21

---

Published in Transactions on Machine Learning Research (8/2024)
conducted for different epsilon settings, image and patch sizes, and memory constraints. We found that for smaller patch sizes, employing gradient accumulation steps greater than 1 is essential, with significant gains observed as the patch size to image size ratio decreases. Despite this promising trend,e remains a hyperparameter requiring further tuning. Morcover, exploring the nuanced relationship between accuracy and steps is an essential aspect for future investigation optimizing PatchGD. In case of UltrMNNIST dataset 612 image size, best performance observed < for patch size of 256. For PANDA two variations were tried for image size 512 and image size 4096 with best results obtained and 32 respectively.
F Applications in Time Series Classification
Extending the concept PatchGD the 1-dimensional case, we find the application in time series classification. For this task, take the example of UCT Human Activity Recognition Dataset. A set of inertial signals at 128 unique time stamps are used to predict the action being executed (sitting, walking, etc.). For the baseline model, we use a basic 1-d Convolutional Network with 64 kernels each of size 3 and a linear layer at the end which achieves an accuracy of 88.9%. The model is trained using Adam as an optimizer with a constant learning rate of 1c-3 for 30 epochs with 32 batch size. The counterpart PatchGD-inspired approach involved the same 1-d convolutional network as the encoder with an intermediate latent vector, with other common hyperparameters being kept the same. The time series is broken into chunks temporally, each chunk being of length 16. Each inner iteration consists of sampling 25% of the total chunks with gradient updates enabled. The model is updated at the final iteration. Impressively, the approach achieves similar accuracy of 88.5%. The results are promising and yet again demonstrate the wide application to other tasks where PatchGD can be applied. Although this needs to be investigated further.
22


Reviews:
[{"review_id": "mqixTtXNob", "reviewer": "Reviewer_qfar", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The experiments show that the proposed method achieves higher accuracies than vanilla (S)GD, at different maximum memory limits, on two academic datasets of images of high resolution (UltraMNIST and PANDA).\n- The paper includes additional (and limited, see next) experiments performed on the ImageNet100 dataset at a smaller resolution, showing that the method is applicable not only to high resolution images, but also on natural images of more standard image resolutions..\n\n**Weaknesses**\n- My main concern is that all comparison are done purely limiting the maximum amount of memory, but no details are provided regarding the training time. The proposed method has a couple of hyperparameters that can scale it cost significantly, compared to vanilla SGD. Namely, the number of inner iterations and the number of sampled patched in Algorithm 1. These hyperparameters have a significant impact in accuracy (see Table 8), but as I mentioned are completely neglected from a training time perspective. Suppose that to achieve the results reported in the paper, the total training time with PatchGD becomes m times larger than GD. What would the results of GD be if one trained simply for m times longer?\n- Most of the experiments are done in rather small datasets (PANDA and UltraMNIST). The paper does include a few extra experiments on some other datasetts, but given that most of the experiments are conducted on the smaller ones, is not clear whether the proposed approach will generalise to more realistic scenarios.\n- There are many baselines that are completely ignored from the paper. For instance, activation checkpointing / offloading, which is currently available and relatively easy to implement in most frameworks (e.g. PyTorch, JAX), or training in lower & mixed precision.\n- Many state of the art works in image classification use Vision Transformers, instead of ConvNets. The paper includes some experiments using PatchGD with DeiT, but it's unclear how to use the proposed approach with a pure Transformer arquitecture.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method to limit the memory requirements when training convolutional neural networks for image classification on large resolution images. The method assumes that the representation of different patches, extracted from the high resolution image, are roughly independent in the earlier layers of the network. Thus, at a given point in training, it avoids having to re-compute the representation of all patches of the image before a given layer, when the parameters haven't changed significantly (an hyperparameter decides how often they are recomputed). The paper tests the proposed method, PatchGD, and compares it against vanilla (S)GD on different academic datasets, with images of relatively high resolution (ranging from 512 to 4096 image size).", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Provide clear total training and inference cost (either in a given hardware, or FLOPs) of the proposed method and the baseline, for a fair comparison.\n- Include experiments with well stablished baselines, as mentioned in the weaknesses; and using well stablished benchmarks (e.g. ImageNet).", "broader_impact_concerns": "No major ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xAVy9NH9Ce", "reviewer": "Reviewer_Epb9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Motivation**:\n+ S1. The problem of training on higher resolution images is interesting and well motivated, especially in the context of high resolution medical images.\n\n**Methodology**:\n+ S2. The proposed method is also conceptually simple at its core, and seems like a natural approach for handling high resolution inputs.\n- W1. However, the specific instantiation of the proposed approach in Algorithms 1 & 2 seems overly complex, with several several inner iterations per batch, each sampling a different subset of Z-block patches, and with several gradient accumulation steps composed with the inner loop.\n- W2. The motivation for restriction head network to a convolutional architecture was not very clear to me.\n- W3. While basic experiments combining PatchGD with a ViT network did not work well in Table 7, there are probably much simpler baselines to experiment with, especially since ViT networks already include a patchification; e.g., what about detaching a subset of the input tokens from the computational graph, or randomly masking/dropping a subset of tokens? what about increasing the patch size to reduce the number of tokens for high resolution images? how about hierarchically processing progressively larger regions?\n- W4. The value of the proposed approach is not clear; it remains to be verified whether it is necessary to train at a high resolution for the entire duration, especially since recent results in CV suggest that you can reap most advantages of high resolution images by only increasing image resolution during the last few epochs of training (e.g., DINOv2). It is also not clear whether PatchGD is a necessary strategy when one can leverage tools such as activation checkpointing or model parallelism.\n\n**Experiments**:\n+ S3. Several convolutional architectures and image classification datasets were considered; including combining GD baselines with an additional convolutional head.\n+ W5. Concerns about the nature of the results and the baselines. Firstly, adding an additional convolutional head to the GD baselines should not degrade performance as in Tables 1 & 2, suggesting perhaps that these experiments were conducted with suboptimal hyperparameters. Secondly, GD performance should not degrade with the GPU memory. You can maintain the same baseline using gradient accumulation, especially since PatchGD employs gradient accumulation. \n+ W6. The experimental setting is somewhat contrived, and in this case, there is no reason not to include similar results on more standardized image classification datasets; e.g., ImageNet where you artificially constrain the GPU memory, and where well established baselines are available for the considered architectures.\n\n**Clarity**:\n- W7. The exposition, specifically the discussion of the method in Algorithms 1 and 2, and the associated text, could be improved to reduce reliance on cumbersome notation", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposed PatchGD, a strategy for training convolutional networks for image recognition with very large image resolutions. The approach splits the ConvNet into two subnetworks, backbone and head. Specifically, the high resolution image is partitioned into a set of non-overlapping patches (similar to patchification in ViT); each patch is then independently projected by the backbone network into an embedding vector to construct a \"Z-block,\" which is a latent 2D feature map. The convolutional head then operates on the entire Z-block to output an image class. To enable training on high resolution images while limiting activation memory, a large subset of embedding vectors in the Z-block are detached from the computational graph during backpropagation.\n\nExperiments conducted with UltraMNIST and PANDA datasets using ResNet50, MobileNet-v2, ConvNeXt, and ViT models. The approach is found to work well with convolutional networks, but perhaps ill-suited in its current instantiation for transformer models. Numerical results demonstrate the ability to train on high resolution images by using gradient accumulation over several \"inner loop\" steps, where a subset of features in the Z-block are resampled in each inner iteration.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "+ Please consider simplifying Algorithms 1 & 2, or at least increasing comments to improve clarity.\n+ Could you clarify why the head $\\theta_2$ is restricted to a conv-net.\n+ Please consider simpler baselines for the ViT networks with GD (AdamW); e.g., a) detaching a subset of the input tokens from the computational graph, and/or b) randomly masking/dropping a subset of tokens, and/or c) increasing the patch size to reduce the number of tokens for high resolution images.\n+ Please include GD baselines using gradient accumulation with the batch size found to work best for each model/dataset in memory-restricted settings, rather than reducing the batch size and degrading performance.\n+ Please consider motivating PatchGD compared to standard tools such as activation checkpointing or model parallelism.\n+ Please consider adding results on other datasets with well-established baselines for the considered architectures, e.g., ImageNet or CIFAR100.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R2ZhTRbi0z", "reviewer": "Reviewer_QP2J", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper shows convincingly that the method reduces memory and gives good results. My main concern is that the paper does not contain adequate baselines for memory-efficient training methods. For example I could not find any mention of checkpointing, which is a standard trick to reduce memory consumption. Similarly, the authors don't mention FullyShardedDataParallel (FSDP), pipeline parallelism, or simply offloading activations to CPU memory during the forward pass. In my view the authors should at least benchmark against checkpointing and activation offloading.\n\nAn important reference that is missing: \"ZeRO: Memory Optimizations Toward Training Trillion Parameter Models\"\n\nUnlike the mentioned methods, PatchGD introduces bias to the gradients. This comes both from the forward pass (because the classifier is run on stale z-vectors) and the backward pass (because some z-vectors don't propagate a gradient, while pixels in neighbouring patches still could influence those z-vectors due to overlapping receptive fields). The paper should discuss gradient bias.\n\nSome details are missing. E.g. it is not clear to me what the memory cost of the Z-buffer is, though it is probably small. I could not find information on where these buffers are stored. Are they in GPU memory?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents PatchGD, a memory-efficient method for training CNNs on large images. The basic idea is to split the network into a patch-wise feature extractor and a final classifier, and to keep a buffer Z that stores the outputs of the feature extractor. For one batch of patches, the feature extractor is run and the corresponding vectors in Z are updated, before the classifier is run on the full Z (which also contains some stale feature vectors). The backprop is done, propagating gradients only through the z-vectors most recently computed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The paper should really include a discussion on methods like checkpointing, FSDP, and activation offloading. It should also discuss and perhaps study gradient bias, which PatchGD has but these methods don't. Finally, it should benchmark compute and memory performance as well as accuracy vs checkpointing, and activation offloading.", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "XBEA1F44u5", "reviewer": "Reviewer_ngg3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors provide a simple method of learning for very large inputs by splitting the input data along patches during training. They also show conclusively that their method outperforms naïve large-input training, including when gradient accumulation is applied.\n\nThe major weakness of the paper is that the only comparison is with naïve training as opposed to methods more specialized for large image training. Even in the case of the PANDA challenge, the QWK score of 0.730 presented by the authors is too far below the 0.862 score achieved in the original challenge to be considered a fair comparison. More rigorous comparisons with SOTA baselines are necessary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes patch gradient descent, a simple method of dividing gradient descent updates of large data into patches by using positional embeddings to locate each patch in the overall input.\nThe authors demonstrate the superiority of their patch-based method over standard gradient descent using the original image sizes, including when using gradient accumulation.\nThe authors also provide results for multiple model architectures and datasets, providing more extensive proof for their conclusions for multiple GPU memory budgets.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The experiments require comparisons with the current SOTA methods for fair comparison. Although the authors have demonstrated superiority over naïvely using large images during training, it is difficult to conclude that they have demonstrated superiority over other methods.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents PatchGD, a method designed to reduce memory requirements when training CNNs on large-resolution images. The approach involves splitting the network into two parts, processing patches of the image independently, and using a buffer to store intermediate features. The buffer is used in its partial form in a stochastic manner for efficiency. The method is tested on various datasets and shows promising results in terms of memory efficiency and accuracy.

Reviewers highlighted several strengths, including the relevance of the problem and the novelty of the approach. However, they also raised concerns about the lack of comparison with other memory-efficient methods, the complexity of the proposed algorithm, and the limited scope of experiments. These concerns were adequately addressed in the rebuttal phase, where the authors provided additional baselines, clarified the methodology, and expanded the experiments.

Given the importance of memory efficiency in deep learning and the promising results shown by PatchGD, I believe this paper makes a valuable contribution to the field. While some limitations remain, such as the potential complexity of the approach and its generalizability to transformer architectures, these do not detract significantly from the overall contribution.
Decision: Accept (poster)