Memory Optimization for Deep Networks
Authors: Aashaka Shah
Chao-Yuan Wu
Jayashree Mohan
Vijay Chidambaram
Philipp Kraehenbuehl
Venue: ICLR 2021 Spotlight (2021)
Keywords: memory optimized training
memory efficient training
checkpointing
deep network training

Abstract: Deep learning is slowly, but steadily, hitting a memory bottleneck. While the tensor computation in top-of-the-line GPUs increased by $32\times$ over the last five years, the total available memory only grew by $2.5\times$. This prevents researchers from exploring larger architectures, as training large networks requires more memory for storing intermediate outputs. In this paper, we present MONeT, an automatic framework that minimizes both the memory footprint and computational overhead of deep networks. MONeT jointly optimizes the checkpointing schedule and the implementation of various operators. MONeT is able to outperform all prior hand-tuned operations as well as automated checkpointing. MONeT reduces the overall memory requirement by $3\times$ for various PyTorch models, with a 9-16$\%$ overhead in computation. For the same computation cost, MONeT requires 1.2-1.8$\times$ less memory than current state-of-the-art automated checkpointing frameworks. Our code will be made publicly available upon acceptance.

Published as a conference paper at ICLR 2021
MEMORY OPTIMIZATION FOR DEEP NETWORKS
Aashaka Shah', Chao-Yual Jayashree Mohan', Vijay Chidambaram 1.2 Philipp Krähenbühh University of Texas at Austin
MWware Research (aahaaka, cywu, jaya, vi jay, philkr)ecs wtt.kk//essuuuteas........ wtw.us.eassee.... edu
ABSTRACT
Deep learning is slowly, but steadily, hitting a memory bottleneck. While the ten- sor computation in top-of-the-line GPUs increased by 32x over the last five years, the total available memory only grew by 2.5x. This prevents researchers from exploring larger architectures, training large networks requires more memory for storing intermediate outputs. In this paper, we present MONEТ, an automatic framework that minimizes both the memory footprint and computational overhead of deep networks. MONET jointly optimizes the checkpointing schedule and the implementation of various operators. MONET is able outperform all prior hand- tuned operations as well as automated checkpointing, MONET reduces the overall memory requirement by 3x for various PyTorch models, with a 9-16% overhead in computation. For the same computation cost, MONET requires 1.2-1.8x less memory than current ofate-of-the-ar automated checkpointing frameworks. Our code is available at httpp:::///........t. //tpthggttb..../// com/ ut sas sastaa/aaaallla/eeee
INTRODUCTION
Deep networks are widely used in domains ranging from image classífication (Krizhevsky et al., 2012 Simonyan & Zisserman 2015 He al., 2016 to video recognition (Wu al. [2019] Fe-
ichtenhofer et al.] [2019) or natural language processing (Devlin et al., 2019] Yang et al.) 2019). However, training deep networks resource-intensive. In particular, the amount of GPU memory bottlenecks training many deep networks (Dong et al. 2016]] Kim et al. 2016 Chen et al. 2018 Child et al. 2019). This bottleneck requires either modifying the network architecture or scaling training to multiple nodes, incurring significant overheads.
We presents MONET, an automatic framework to minimize memory footprint for deep networks. MONET jointly optimizes global compute-graph-level techniques (such as checkpointing) and lo- cal techniques (such as memory-efficient implementations of individual operator). At the heart of MONET is theoretical analysis that enables joint optimization and provides tight bounds mem- ory consumption. We analyze the memory consumption and computational cost of general for- ward and backward pass under changing local operator implementations and global checkpointing schedule. Specifically, we are able to tightly bound the peak memory consumption for network for- ward, backward, and recomputation stages. MONET uses these constraints optimize for the most efficient forward and backward implementation both locally and globally under a fixed memory bud- get. We linearize all memory bounds, and express both implementation selection and checkpointing as a 0-1 integer program, which we solve using standard solvers.
We conduct extensive experiments, demonstrating that MONET significantly outperforms ex- isting automatic frameworks that use local or global techniques. On multiple architectures (ResNet (He et al., 2016)) VGG (Simonyan & Zisserman 2015). UNet (Ronneherger et al., 2015). GoogleNet (Szegedy et al.) 2015), MobileNet-V2 V2 (Sandler et al. 2018)), memory budgets (5-10 GB), and network configurations (multiple resolutions), MONET consistently achieves lower mem- ory footprints at equivalent or lower computational overhead. MONET reduces the overall memory requirement by 3x for various models, with a 9-16% overhead in computation. For the same com- putation cost, MONET requires 1.2-1.8x less memory than the current ttate-of-the-ee automated checkpointing framework. The results achieved by MONET demonstrate the power of jointly opti- mizing global checkpointing schedules and local operator implementations.

---

Published as a conference paper at ICLR 2021
Activatativtioon?
Checkpoint?
Input-Activation
BN
BN
MONET
Corv
ReLU
Conv
+
ReLU
Checkpoint?
Inplace?
Checkpoint
Bit-Activation
Bit-Activatio?
Figure 1: Memory Optimized Network Training (MONeT), automatic framework that mini- mizes the memory footprint of deep networks by jointly optimizing global and local techniques.
RELATED WORK
There are two broad families of approaches to reduce the memory footprint of a deep network during training: operator-level implementation changes, and global, graph-level optimizations. The novel aspect of MONET is that it is able to combine both approaches and find the optimal mix of local and global techniques for a given network.
Operator-Specific Optimizations. Researchers have found creative ways to implement individ- ual operators or groups of operators in a more memory-efficient manner. Standard deep learning frameworks (Tia et al. 2014 Collobert al. 2011 Paszke al., 2019 Abadi al. 2016) provide different implementations of certain operators that trade computation for intermediate memory use. These implementation are chosen according to local search heuristics, and are not globally optimal. Gist (Jain et al. 2018) proposes several hand-crafted optimizations such as storing only ReLU signs. RevNets (Gomez al. 2017) redesigns ResNet (He et al. (016)) architecture making each network block reversible, thereby eliminating the need to store intermediate activations for backpropagation. Memory-efficient DenseNets (Pleiss et al.) 2017) reduce memory utilized for feature maps by re- computing all intermediate feature maps during the backward pass with small compute overhead. In-place activated batchnorm (Bulô et al., 2018) or ReLU layers use output activations to compute their gradients, thus reusing single memory buffer for the gradient computation in consecutive lay- ers. Mixed precision training (Micikevicius et al.) 2018) uses half precision (FP16) instead of single precision (FP32) for all tensors and arithmetic during training, reducing the memory by nearly half. While training at precision lower than FP16 results in loss of training quality (Banner et al. 2018), prior work like backpropagation with approximate activations (Chakrabarti & Moseley (2019) care- fully quantize certain intermediate outputs (activations) to 4 bits, resulting in sigmificant memory savings. Although these hand-crafted techniques independently result memory savings, there is no one-size-fits-all recipe, and different implementations perform best on different architectures. In contrast, MONET automatically finds the best implementation for each forward and backward operator given memory budget.
Checkpointing, Chen et al. (2016) proposed dividing a network into different segments, dropping all intermediate outputs within each segment, and recomputing them later. Chen et al. use / equal segments, trading memory savings for the cost of an extra forward pass. Checkmate (Jain et al., 2019) solves the problem in a more general setting, using an mixed-integer linear program solver to decide which layers to recompute for given network. Like Checkmate, our work optimizes a checkpointing schedule, but on different computation graph. Our computation graph allows for the optimization of an entire execution plan jointly finding a checkpointing schedule and the best implementation of each forward and backward operator. In Checkmate, changes in operator implementation induce different computation graph, and could thus not directly be optimized. | highhighhts some of the difficulties of adding operator optimizations into Checkmate.
Appendix
In summary, while much work has been done on local optimizations (operator implementations) and global compute- campute-graph--aee techniques (automated checkpointing), MONET the first system to jointly optimize given architecture using both local and global techniques.
2

---

Published as a conference paper at ICLR 2021
Algorithm 1: Forward Pass
Algorithm 2: Backward Pass
Input : Inputs, 9, a schodule (s. r).
Input
Loss gradients, inpuls, 0, 8 (s.
Output: Ouiput lensor
Output: Ouiput lensar
s* = fx Saved tansers for bockward x/
F. (loss gradients] /+ local backward tensors //
[inputs, 0) /- Tocal: tensora for "orward
k
3
L=S²
=**
/+ Local forward tensors +/
for 1...
ats forward: (F.)
s*
=
Saved tensors
5
for 1... do
6
if then
2; forward:
Remove all lensors from L that are not used later
Add z; L.
Remove all tensers from not used later then
if then
ef
Add z4 to SN
11
Add to g ye backward, (L,
/x use 24 E
return L
Add yu
Remove temsors from that are not used later
Figure 2: Schematic overview of the forward and backward passes. The algorithms include ag- gressive memory savings by greedily freeing unused tensors, and allow for general checkpointing schedule (8, r) to be executed.
PRELIMINARIES
Let the forward pass of a CNN with parameters è be expressed as a directed-acyclie graph (DAG), where each node E {l...., .N} corresponds to operator forward,, and edges (i,j) E: E specify the data-flow dependencies, ie.. the output of operator i is used as input in operator j. Without loss of generality, computational dependency ji, E E implies i < j. Let N, {i: (i,j) E} be the set of all incoming edges of an operation j.
We will first discuss the forward pass through network and the basic form of backward pass using checkpointing. The backward pass reverses all computational dependency expressed in our DAG, and induces certain dependencies on forward activations. We call these checkpoint dependencies DE. They are either saved or recomputed depending on a schedule (s,r). Checkpointing creates trade-off between computation and memory consumption. To highlight this trade-off, we formally compute the amount of memory consumed in both forward and backward passes, which allows us to optimize for the ideal execution plan in Sec. We provide a reference the notations introduced in this section and the next along with their explanations in Appendix A
The Forward Pass. Alg. T shows a general overview of the forward pass in a deep network, as
implemented in standard deep learning frameworks (Jia et al.) 20144 Collobert al. 2011 Paszke
et al. 2019 Abadi et al., 2016). The algorithm proceeds in increasing order of index t. Each operator forward, depends on set of tensors L stored in local memory. These tensors include model pa- rameters O computational dependencies. Ni, and tensors stored for later forward operators, i.e. skip or residual activations (He et al., 2016). At each iteration, we add any output tensors of forward, to the local memory L. Early deep learning frameworks (Jia et al. [ Collobert et al. (011 strictly grew the set of local tensors leading to an unnecessarily high memory consumption. Modern graph-based frameworks (Paszke al. 2019:; Abadi et al.) 2016) reduce the memory footprint by aggressively pruning local memory L and freeing any tensor that no longer used in later compu- tations. Some output activations 2; used in the backward pass, and have to be saved for later. We use a checkpointing schedule N to determine which. Formally, N E {0, 1} indicates whether the activation of node i is stored during the forward pass. An activation which is not stored will be recomputed if it is needed during the backward pass.
Analyzing peak memory consumption of the forward pass. Only the forward, operator (Alg. LL. allocates memory. All other operators perform mere bookkeeping existing tensor. It is thus sufficient to study the peak memory consumption m. in forward, for each node i. Let L,, S.N be the set of local tensors L and saved tensors S while calling forward, respectively. L, includes all parameters and computational dependencies for this and later forward passes L, θ U {xy j E N₁, for any t 2 and j < i}. L, is constant and computed ahead time. The schedule N determines the of saved tensors S.N {xz sN
for < i}. In addition, each forward

---

Published as a conference paper at ICLR 2021
operator uses certain amount of workspace memory C₁ to store intermediate results. memory consumption of a forward operator is thus
The total
m; = + | + |SN U L - c; + |as +
|ayl+ +
lzylsy
(1)
ZyEh,
JKEzy
where |.| refers to the memory consumed by tensor or set of tensors. Most of the memory consumption is constant and does not depend on the schedule.
The Backward Pass. The backward pass proceeds in a reverse order, as summarized in Alg. backward of each node k depends on a set of gradient tensors f. and forward tensors {z; : D&). Any gradients required by the current and later backward passes are stored in local memory L. Dependencies D3 may either stored in S or need to be recomputed from checkpoints in S Recomputation involves forward computation of one overhead, and allows for a new set of tensors 5*-1 to be saved. After recomputation, all dependen- cies D kept in memory. The backward operation produces gradient for each input tensor of the original forward operation, which added L if required for later backward computation. We aggressively remove tensors in L that are not required.
nodes, which increases computational
Analyzing the peak memory consumption of the backward pass. Peak memory consumption my again only depends on the forward, (Alg. [ and backward, (Alg. [.. operations. For the backward, operation, let Cg be the workspace memory, L. be the set of gradient tensors stored, newly saved tensors. Here
Dy {24 : } } be the forward tensors used, and 5 be the Le and D, can be pre-computed. The total memory consumption for the backwardg call is
nu = cτ + |uvl + +1** U U. U Da = è + |ivel + Σ Ew1++ + Σ E
E
Σ
s (2)
I.ED2
Here again, only the last term depends on the checkpointing schedule, while the rest a constant.
Analyzing the peak memory consumption of the recomputation. Finally, the peak memory m for the forward, call (Alg. 2 L. 7 depends on the set of local tensors I,, checkpoint dependencies D, saved tensors S, and gradient tensors L, named LA DR. S. 5k-1 Ly respectively, Following the forward pass:
m. = C
2 +
+
|s
UL Σ
Lx
12,l
Σ
|zyl Σ j>i
1
JERREETUUD
Unlike the forward pass, L is no longer constant, but depends on past saved tensors and future recomputations in (s)r): T θ (æz N for any 2 i with rf = and j < <i).
In the next section, we show how to take this formalization the forward and backward pass, and find an optimal execution plan including checkpointing schedule (s, forward, implementations, and backward, implementations, under a fixed memory budget.
4 METHOD
Our goal is to find global checkpointing schedule (s, r) and local forward//backward,, implemen- tations that jointly minimize the computation cost within memory budget M. We show how to express this optimization in a 0-1 integer program and efficiently solve it. To this end, we linearize any peak memory consumption constraints, ensure that the checkpointing schedule is valid, and solve to minimize a computation cost objective. We keep track of the three contributors to memory and computational cost forward pass, backward pass, and recomputation of forward operators.
Memory Constraints. Consider the case of basic checkpointing using only single implementation for forward; and backwardg. The memory consumption of the forward [ and backward 2 pass are linear in and thus efficiently expressed in integer program. However, recomputation depends both on -- and , a non-lineat manner through the local memory L This joint dependence on optimization variables gives rise to quadratic constraints, which cannot directly be incorporated into
4

---

Published as a conference paper at ICLR 2021
an integer program. For simplicity in this derivation, bound the set of local tensors from above. assuming every future tensor is recomputed. We give more information about this in Appendix [ The upper bound L is constant, yielding linear upper bound m of the recomputation memory
m. analogous to Eq. 3 The set of memory constraints is thus
my M
V/
and ri S M
Vk and
m
M
Ve.
(4)
To enable operator optimization, we bit-vector ö indicate the selection of an operator imple- mentation. We add to the constraints which allows us to jointly optimize checkpointing (s, r) and operator implementations 8.
Forward Operator Optimization. Let each forward operator forward; have multiple different implementations I3 = {a, b, c....). For examples, convolution may be implemented using matrix multiplication, the Winograd algorithm (Winograd 1980), Fourier transform, etc. (Chetlur al.) 2014). All implementations follow the same DAG structure, and thus use the same dependencies, Ng- However, each implementation trades workspace memory { (rf.,,-- ......> in different manner. Our experiments show that this trade-off is often complex.
for computational efficiency
Our goal is to represent the peak memory when using multiple forward, implementations in the forward pass and recomputation. Let δū E {0, 1] indicate that implementation for forward; in the forward pass. Each forward operator should use exactly one implementation Ee, 1. The choice of implementation determines the operator's computational cost Σ, 7 and workspace memory C; Σ, c, Analogously, each recomputation of forward; during backward,; chooses between implementations o E {0. when needed Σ, 8 r, with equiv- alent cost estimates >, o and workspace memory use ci = E, c In this formulation, all additional memory requirements remain linear and are directly integrated into the linear memory constraints their linear relaxations (equation 4
I4 is used
Backward Operator Optimization. Let each backward operator backward, have set of differ- ent implementations i. {a, .......). Each implementation again trades workspace memory (2,,,),.... for computational cost {4f, 4 .... While gradient tensors follow the fixed DAG structure, different implementations may depend on different forward activations (Dg, Dh....) For example, in-place activated operators (Bulo et al. (2018) depend on their output activation, while reg- ular operators use the input activation. This change in the dependency structure makes optimizing for backward-operator implementations challenging.
We again aim to represent memory in terms of implementations for each backward, operator. Let 6k.. E {0, 1} indicate that implementation a E 4 used at node k in the backward pass. Each backward operator should use exactly one implementation Σ, 6x,1 = 1, with a computational cost Σ, Af &g.1 and workspace memory é₁- & The workspace memory adds a linear constraint to the memory consumption my equation
The biggest changes to the optimization problem, comes from the changing dependency structure. DE is no longer constant. Instead, the implementation of backward operator changes the set of computational dependencies D; obtained from D To deal with this changing dependency struc-
ture, we use the indicator vector da to select memory contribution of dependencies from the chosen implementation. This changes the backward memory consumption
my = Σ de + +laal + L +
Σ
8e. U
(5)
and the corresponding peak recomputation memory m. to
m = c3 + |as| + | + Σ 8 T. D
(6)
Note, the last term of equation and equation are quadratic in the original optimization variables sh-1 which determines s and 8g,1- However, for binary variables, it can be linearized using an auxiliary variable (see Appendix C.4). We show the full equation expansion in Appendix C.1
5

---

Published as a conference paper at ICLR 2021
Checkpointing Constraints. The computational dependencies of forward and backward operators impose strict constraints on the checkpointing schedule. Any schedule violating these constraints cannot be exccuted. Recomputation r requires saved o or recomputed r dependencies j E N, and only previously stored or recomputed tensors can be saved: and
+
V1.k-
Furthermore, all forward tensors Dj required by backward, need to be stored or computed
8,,2 VAL.EED
(8)
Objective. Our goal is minimize the amount computation required for the forward and back- ward pass. This is represented the sum of computational costs of all operators:
ΣΣ dea +
ΣΣ
B the
ΣΣ A:
forward pass
backward pass
recomputation
Objective equation 9 with constraints equation 4 equation equation S and definitions equation T equation [ equation form our final optimization objective. It jointly solves for the optimal imple- mentation of each forward and backward operator, well an efficient checkpointing schedule.
5 EXPERIMENTS
Implementation Details. We develop MONET in PyTorch v1.5. and solve the joint optimization problem using the Gurobi (2014) solver. Appendix D provides more implementation details and a full list of optimized operators.
The UNet experiments use 608x416 inputs following prior work (Jain et al. 2019). All other exper- iments use 224 x 224 inputs following conventions (Krizhevsky et 2012 Simonyan & Zisserman, 2015 He et al, 2016). Batch size for the experiments is fixed to be the maximum at which the model can trained using baseline PyTorch on 16 GB GPU. Since Checkmate's 's (Jain et al. 2019) ex- ecution engine is built for TensorFlow, and an official Gist (Jain et al.) 2018) implementation is not available, we reimplement them in PyTorch for our comparisons. Our Checkmate implementation is competitive, it uses the original Checkmate solver and has the same network structure as MONET. Checkmate does not optimize for operator implementations like convolutions, so we show its run- time using the default convolution algorithm (Checkmate-D) For a stronger comparison, show the runtime of a Checkmate schedule that is post-optimized to greedily run the fastest convo- lution algorithm (Checkmate-Ol) Wherever not explicitly specified, we compare with Checkmate-O All checkpointing schedules are run using the same software implementations and costs are profiled on the same hardware (NVIDIA P100 GPUs). In order compare against operator-specific opti- mizations, we reimplement all Gist techniques in PyTorch and run them on our execution engine. See Appendix for more details about our baseline implementations.
also
Detailed Comparison to Baselines. (a) Checkpointing: Table [ compares the memory savings obtained by MONET and Checkmate for five different models when computational overhead over PyTorch is fixed to be 10%. MONET schedules 2-3x less memory than PyTorch. For the same computational overhead, MONET uses 1.2-1.8x less memory than Checkmate.
Fig. shows detailed runtime-memory trade-offs of MONET PyTorch and Checkmate for different models. We plot the average iteration time of training as % overhead over PyTorch for
50
GoogleNet
UNet
VGG-16
MobileNet-V2
PyTorch
15.1
14,9
14.3
14.1
14.5
Checkmate (Jain et al. 2019)
10.5
9.9
5.8
MONeT
5.7
6.9
5.2
5.5
4.8
Table 1: Memory usage comparison (in GB) for fixed compute overhead. At 10% compute overhead over PyTorch, MONeT uses 2-3x less memory than PyTorch. At the same overhead, MONeT can train models using 1.2-1.8x less memory than Checkmate.

---

Published as a conference paper at ICLR 2021
40
80
80
Pylorch
30
30
60
60
30
Checkmate-D
Overhead
Checkmate-I
40
10
29
MONeT
10
20
20
10
0.4 0.6 0.8
0.4 0.6 0.8
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6 0.8
Memory ratio
Memory ratio
Memory ratio
Memory ratio
Memory ratio
(a) ResNet-50 (184) (b) GoogleNet (320)
(c) UNet (11)
(d) VGG-16 (176) (c) Mobile-V2 (272)
Figure 3: Comparing MONeT with PyTorch and Checkmate. MONeT reduces memory by 3x compared to Pyroc,, with 9-16% compute overhead. It achieves better memory-compute trade- off than default Checkmate-D and conv-optimized Checkmate-O.
5 GB
6 GB
7 GB
8 GB
9 GB
10 GB
ResNett-50
Checkmate
8.96
12,01
10.78
4.54
2.98
MONeT-NoOp
1.18
0.46
0.14
0.09
0.06
0.07
MONeT
7.24
3.84
0.73
0.70
0.31
0.11
GoogleNet
Checkmate
12.72
4.56
4.32
3.92
0.86
MONeT-NoOp
0.10
0.11
0.07
0.07
0.07
0.07
MONeT
3.53
0.47
0.54
0.31
0.25
0.24
VGG-16
Checkmate
0.002
0.002
0.001
MONeT-NoOp
0.001
0.000
0.000
MONeT
0.003
0.003
0.003
0.003
0.003
Table 2: Solver time (in hours) to reach 5% close to optimal solution. MONeT-NoOp reaches a 5% close-to-optimal solution 1.6x-117x faster than Checkmate. МONeТ gets close to 5% of the optimal solution only in a few hours, and up-to 16x faster than Checkmate for larger models.
MONET and Checkmate schedules. The memory budgets range from GB to 10 GB, or equiva- lently, 0.33x to 0.70x PyTorch memory consumption. Batch size for these models is mentioned in paranthesis. For models, MONET reduces memory usage by 3x (0.33 memory ratio) as com- pared to baseline PyTorch with 9 16% compute overhead. For the same memory budget, MONET schedules are up-to 34% faster than Checkmate schedules. Note that we measure the empirical per- formance of the schedules running memory using the solver values; this is important since Checkmate does not consider workspace cost and overestimates its savings.
GPUs instead of just providing simulation of runtime and
For networks with individual memory-intensive layers, like VGG-16, operator optimization be- comes even more important for reducing memory; Checkmate can reduce memory for VGG-16 only up to 7 GB, whereas MONET with its optimizations is able to run VGG-16 with 5.5 GB mem- ory. The small runtime improvement of MONET schedules over PyTorch for VGG-16 and UNet higher memory budgets is mainly because of choosing faster convolution algorithms. MobileNet- V2 uses depthwise convolutions, and hence does not significantly benefit from joint convolution- optimization. As a result, the performance of MONET and Checkmate is closer for MobileNer- V2. We provide additional results for MONET on a memory-intensive model, 3D- (Çiçek et al.) 2016), in Appendix [1, for which we observe consistent memory reduction to 0.54x of PyTorch memory with an overhead of 8.86%.
For our evaluation, we cap the solver time to 24 hours for both MONET and Checkmate, and run the schedule thus obtained on our execution framework. At tighter memory budgets for non- non-linear models like ResNet-50 and GoogleNet, Checkmate unable to find feasible solution within couple of hours. In contrast to Checkmate, MONET finds the execution plans efficiently. For all the models and memory limits that evaluate, MONET reaches 5% close-to-optimal solution within

---

Published as a conference paper at ICLR 2021
VGG-16 (176)
ResNet50 (184)
GoogleNet (320)
MobileNerV2 (256)
UNet (11)
mem overhead
mem overhcad
mem
overhcad
mem
overhead
mem
overhead
Gist
0.76
44.34
0.58
105.69
0.52
35.94
0.69
153,98
0.73
38,26
MONeT
0.39
9.11
0.33
11.94
0.33
15.77
0.34
8.80
0.35
11,51
Table 3: Memory ratio and overhead (%) over Py PyTorc for Gist and MONeT. MONET obtains
1.4x-2.1x higher memory savings over Gist across models. Number in parenthesis after model name shows the batch size.
Overhead (5)
10
9.21
9.3
12
11.78
11.29
39.48
32.58
10.67
37
22,67
8
6.99
6.37
10
6
5.53
8.78
8.45
17
-0.70
-2.18
none
conv
Ouf
int
all
none
conv
out
int
all
none
conv
out
int
all
(a)
ResNet-50
(b) GoogleNet
VGG-16
Figure 4: Ablation results for memory ratio 0.53. Lowest compute overhead across models is seen
only when all optimizations are jointly optimized.
a few hours or sometimes even minutes. Table shows the time it takes for the solver to reach 5% close to the optimal solution, for Checkmate, MONET-NOOP (MONET with checkpointing enabled but operator-opuimization disabled), and MONET. MONET-NOOP converges to close-to-optimal solution 1.6x-117.4x faster than Checkmate. For larger models, MONET'S solver converges close-to-optimal solution up to 27 x faster than Checkmate. Note that running a solver is one-time cost for a model- once a MONET schedule has been solved for, it can be used by everyone train the model for different purposes with different batch sizes. The cost (typically seconds to hours) tiny compared to the efforts and costs to develop a model for distribution in most cases. See Appendix H for more discussion regarding solver times, problem statistics, and full
(b) Operator optimizations: Table 3 shows the comparison of MONET with Gist. While MONET can determine range of memory-runtime tradeoff', purely operator-ptimnzaaioo--bsse schemes like Gist only provide a single memory-runtime data point. For MONET, we show the memory- runtime data point with the most memory saving. MONET uses 1.4x x less memory than Gist for multiple architectures while maintaining full full-precision. Overall, Gist provides impressive memory savings, but incurs a high computation cost to achieve the savings.
While we get similar memory saving results for reimplemented-GGis as Jain et al. (2018) for VGG- 16, our compute overhead results are higher. This could be because of evaluations frameworks (PyTorch v/s CNTK) and different GPU models (Nvidia P100 v/s Nvidia Maxwell GTX Titan X). Gist uses dense to sparse conversion using cusparsesdensee2cs in one of its techniques. For the first ReLU-Conv layer in VGG-16 (shape (22C7744, 256)). ), this function takes 144ms, which itself is 10% of the VGG-16 execution time. We see similar results for other networks. To ensure fair comparison, we focus on the maximum memory savings obtained by MONET with Gist, while reporting the compute overhead for completeness.
different
Ablation Experiments. Fig. 4 shows additional ablation experiments. We show the % compute overhead over PyTorch on ResNet-50, GoogleNet, and VGG-16 for different types of MONET checkpointing schedules with a memory budget of 8 GB with no operator optimizations en- abled, with only one type of operator optimization enabled (conv-optimized, output-activated op- timized, anternediate-activvtee optimized), and with all optimizations enabled. Schedules which do not jointly optimize convolution algorithms are run with greedily post-optimized convolution algorithm. Plots for other models look similar to that of ResNet-50 and GoogleNet. The only dif- ference between 'none' and 'conv' is that convolution algorithms are jointly optimized in the latter. However, this fact leads to significant improvement in compute time for all cases. fact, convolu- tion algorithms have complex workspace memory compute characteristics, reserving slightly more memory for convolution workspace while checkpointing can allow for a much faster convolution (see Appendix M. This makes it important to jointly optimize conv algorithms with checkpointing. Similarly, autput-activated optimization also provides significant benefits over vanilla checkpoint- ing, since it effectively reduces the number of recomputations required. For memory-intensive net-
8

---

Published as a conference paper at ICLR 2021
-10*
Py PyTchh ms)
Poroh (14.7 (B)
MONeT-NoOp (939 ms)
1,5
MONeT (908 ms)
Mem (MB)
MONeT 1800 GB)
0.5
-10
Overhead Mem. Diff (MB)
-10
2
50
100
150
200
250
300
Network Layer Index
Figure 5: Detailed case study on ResNet-50. Top memory usage along execution (forward and
backward). Middle: memory saving of MONeT over PyTorch for each layer. Bottom: compute overhead of MONeT over PyTorch. MONeT saves memory in early layers to reduce peak memory. Most compute overhead happens recomputation during backward (right-hand-sie of the figure).
works, internediate-activaae optimization becomes more important. Jointly optimizing all strate- gies together gives the least computational overhead. See Appendix G for detailed ablation plots.
Detailed Case Study. The top graph of Fig. 5 shows memory usage while executing PyTorch, MONET without operator optimization, and MONET for ResNet-50 at batch size 184. As the training progresses along network layers represented forward pass outputs, Ieading to an increasing memory footprint. MONET reaches peak memory of 8 GB, whereas PyTorch requires 14.7 GB. Stored forward outputs freed up after other as backward pass proceeds, leading to reduced usage of memory. According to the checkpointing schedule, MONET saves only subset of the outputs stored by Py'Torch, resulting in the memory saving shown in the middle graph for layer outputs that are not stored. The bottom graph shows the per-layer compute overhead of recomputation of MONET over PyTorch. For MONET, later layers which are backward operators result in a recomputation of the forward, and have higher overhead.
X-axis, PyTorch and both MONET schedules store
CONCLUSION
We present MONET, a system automatically reduce memory requirements for training deep net- works. MONET jointly optimizes local (operator-level) and global (graph-level) optimizations to yield a compute- and memory-efficient checkpointing schedule. MONET reduces memory usage by 3x over PyTorch, with 9 16% compute overhead. It uses 1.2-1.8x less memory than the state-of-ther -htt----thhe-rt automated checkpointing framework for the same computational cost. Our experi- mental results show that MONET leads to better memory- computation trade-offs compared to the state-of-the-art.
ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers for their feedback. Aashaka Shah and Vijay Chidambaram were partially supported by donations from VMware and Google. Chao- Yuan Wu was partially supported by a Facebook Fellowship. Jayashree Mohan supported by Microsoft Research Fellowship. The results presented in this paper were obtained using the Chameleon testbed supported by the National Science Foundation

---

Published as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenherg, Rajat Monga, Sherry Moore, Derek Gordon Murray, Benoit Steiner, Paul A. Tucker, Vijay Va- sudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: A system for large-scale machine learning. In OSDI, 2016.
Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting system for convex optimization problems. Journal of Control and Decision, 2018.
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. In NewrIPS, 2018.
Samuel Rota Bulò, Lorenzo Porzi, and Peter Kontschieder. In-place activated batchnorm for memory-optimizze training of dnns. In CVPR, 2018.
Ayan Chakrabarti and Benjamin Moscley. Backprop with approximate activations for memory- efficient network training. In NewrIPS, 2019.
Liang-Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L. Yuille. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. TPAMI, 2018.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. CoRR, 2016.
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catan- zaro, and Evan Shelhamer. cudnn: Efficient primitives for deep learning. arXiv preprint arXiv:/410.0755 2014.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, 2019.
Özgün Çięek, Ahmed Abdulkadir, Soeren S Lienkamp, Thomas Brox, and Olaf Ronneberger. 3D U-Net: learning dense volumetric segmentation from sparse annotation. In MICCAI, 2016.
Ronan Collobert, Koray Kavukcuoglu, and Clément Farabet. Torch7: matlab-like environment
for machine learning. In BigLearn, NeurIPS workshop, 2011.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. NAACL, 2019.
Steven Diamond and Stephen Boyd. CVXPY: A Python-embeddee modeling language for convex optimization. Journal of Machine Learning Research, 2016.
Chao Dong. Chen Change Loy. Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 2016.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast networks for video recognition. In ICCV, 2019.
Aidan N. Gomez, Mengye Ren, Raquel Urtasun, and Roger B. Grosse. The reversible residual network: Backpropagation without storing activations. In NeurlPS, 2017.
Gurobi. Gurobi optimizer reference manual, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Animesh Jain, Amar Phanishayee, Jason Mars, Lingjia Tang, and Gennady Pekhimenko. Gist: Efficient data encoding for deep neural network training. In ISCA, 2018.
Paras Jain, Ajay Jain, Aniruddha Nrusimha, Amir Gholami, Pieter Abbeel, Kurt Keutzer, Ion Stoica, and Joseph E. Gonzalez. Checkmate: Breaking the memory wall with optimal tensor rematerial- ization. CoRR, 2019.
10

---

Published as a conference paper at ICLR 2021
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross B. Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature em- bedding. In ACM MM, 2014.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. ICLR, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: imperative style, high-performance deep learning library. In NeurIPS, 2019.
Geoff Pleiss, Danlu Chen, Gao Huang, Tongcheng Li, Laurens der Maaten, and Kilian Q. Wein- berger. Memory-efficient implementation densenets. CoRR, 2017.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mo- bileNetV2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
Shmuel Winograd. Arithmetic complexity of computations. SIAM, 1980.
Adrian Wolny, PyTorch 3D-UNet. 2019.
Adrian Wolny, Lorenzo Cerrone, Athul Vijayan, Rachele Tofanelli, Amaya Vilches Barro, Marion Louveaux, Christian Wenzl, Susanne Steigleder, Constantin Pape, Alberto Bailoni, Salva Duran- Nebreda, George Bassel, Jan U. Lohmann, Fred A. Hamprecht, Kay Schneitz, Alexis Maizel, and Anna Kreshuk. Accurate and versatile 3d segmentation of plant tissues cellular resolution. bioRxiv, 2020.
Chao-Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. CVPR, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In NewrIPS, 2019.

---

Published as a conference paper at ICLR 2021
A
NOTATIONS
Table gives a list notations used in the paper along with explanations.
Notation
Meaning
Tensors
2;,
Tensor which is the output of forward, in forward pass and during recomputation. Gradient tensor which the oulput of backward, backward pass.
y/
Sets
s
Set of stored tensors after forward, in forward pass. (N = num backward operatons) Set of parameters and forward tensors created till forward node required as computational dependencies forward, later forward passes.
Li
De
Set of forward pass tensors required as computational dependencies for backwarda.
sk-11
Set of stored forward pass tensors right before calling backwardæ.
La
Set of gradient tensors created before backward node k, and required as computa- tional dependencies for backward,, and later backward passes.
s
Set of stored tensors available to recomputation of forward, before computing backwardg.
L.
Set of parameters and forward tensors created till forward node required as computational dependencies for forward, and later forward recomputations to be done before backward,.
Set of implementations for operator forward.. Set of implementations for operator backwardg.
Ix
Solver variables
Indicate if output of forward; stored in memory the forward pass.
Indicate if output of forward, stored in memory when computing backwarde.
Indicate if forward, is recomputed before computing backwardæ.
8
Indicate if forward, uses implementation I, in forward pass. Indicate if forward; backwardk.
87,
implementation / E I, when recomputed before
8g Memory formulations
Indicate if backward, uses implementation / Ix-
m
Peak memory of forward; forward pass. Peak memory of forward; when it is recomputed before backwardg. Peak memory of backward..
m ma
Operator costs
Workspace memory of operator forward, executed using implementation E I, Workspace memory of operator backward, executed using implementation E fx. Compute cost of operator forward, executed using implementation Ia. Compute cost of operator backward; executed using implementation I E Iz.
2
T
Table 4: Notations used in paper with explanations. Notations with only in subseript/supprseritt generally relate to the forward pass, with only & relate to the backward pass, and with both and relate to the recomputation phase.
B BOUNDS ON LOCAL MEMORY
In Section 3 we mentioned that local memory L is dependent on solver variable r:
L θ (xy J N₁ for any 2 i with rf 1 and j < i).
In order to remove this dependence, we can get an upper bound Ł on L by assuming that all future tensors after will always be recomputed, that is, r 1, Vt > i, and
L C L OU (z, j E Nr for any t 2 i and j < i}.
Our experiments also use this upper bound. It is possible to tighten the upper bound by noting that r may be 1 only in the case when i < k. That is, forward node will not be recomputed before computing backward of node & node / lies after node k. Thus, tighter bound to Lţ follows
T i e u (x, j E N, for any & and and i) T.7.
12

---

Published as a conference paper at ICLR 2021
C DETAILED CONSTRAINTS
C.I. EXPANDED BACKWARD PASS MEMORY CONSTRAINTS
Sec. formulates backward peak memory my and recomputation peak memory m sum of mem- ory of a set of tensors. We expand the memory formulation and represent in the terms of optimiza- tion varaible here:
L ŌR + lBkl +al++10 +1 + Σ b U
mick =
Σ
CR + lyxl + Σ l + Σ Σ
8u.1k0i +
Σ
Σ
kiils
(10)
21 E
mE
x. DE
m, = G + | |ما| + Σ 1B D'A
G + |ay|
[i +
S4.1st "|ayy + ΣΣ
8u +
>>
(11)
jei 2 JD
jzi
C.2 COMPLETE MEMORY CONSTRAINTS
In this section, we present the complete memory constraints which we use for MONET optimization. These constraints include the recomputation variable rf. which was excluded from the main text to make understanding simpler. As discussed in Sec. [3] the peak memory of a forward, recomputa- tion before computing backwardę is denoted by m.. This represents the recomputation memory (renamed to mfae) when forward; is actually recomputed, that is, r 1. When this is not true, the peak memory (m only depends on stored checkpoints 5 checkpoint dependencies for DR. and gradient tensors f.g. Thus,
i = Q + | + [اxl + 1s U. D
U.
cr + r + [ual 1oy+ Ett1=t Ekej+ +
Eojlog (12)
jeizzeen
je D j>i
m
|èxl + 1ist-11 U
= |ial
+ Σ
Layl +
Σ jSRzYEDe
|a,l+ +
Σ 8 l5
(13)
D
Local memory LE can be bounded by Le. which gives us m To add forward operator optimiza- tions to m we recall the trade-off between workspace memory and compute time. We replace the workspace memory contributor r c; in equation 12 with Σ, cl.
The complete memory constraints are:
m, S M
442 and vi < M V4 and mm S M
Ve.i and ms: S M
(14)
C.3 IN-PLACE CONSTRAINTS
show how to represent the decision computing an operator using an in-place or out-of-place implementation. If an operator like ReLU uses in in-place implementation, its input tensor is over- written with its output. this case, its input tensor cannot be stored or used as input to a computation in this stage. This needs be reflected in constraints. We introduce two new binary variables to model in-place computations: d. represents if forward; recomputed in-place when computing backwardæ. P represents that the output of forward, has been computed and will not be overwrit-
ten by any other forward node recomputations in this stage. of is true, then p will be false else
p will be the same as rý. where Nr. Further, 8 will also be false if o: is true. This can be written the form of boolean constraints as follows:
1 2 1
2 and
D < 2 20
and
k- 8
< 2
(15)
13

---

Published as a conference paper at ICLR 2021
The checkpointing constraint changes, with pf replacing r the RHS. Further, of (or p²) can only be true if forward, (or forward,) is actually recomputed prior to computing backward node k. Thus,
p
and s
(16)
C.4 CONSTRAINT LINEARIZATION
The memory constraints we introduce in Section 4 contain quadratic terms in the form of z; Zjj with zú, I E {0, The quadratic terms cannot directly be incorporated into an integer program. However, we can linearize these terms by replacing each quadratic term 2; 2z by auxiliary variable ax E {0, 1} and introducing additional linear constraints 0g, 2 I2 + 2j - 1, dz,j S Zi. and oi.j S Zzj. After this substitution for all quadratic terms, all constraints in MONET are linear.
D IMPLEMENTATION
We develop MONET in the PyTorch (v1.5.1) framework. We use PyTorch's default Autograd pack- age for backward implementation of elementary functions when the autograd implementation is stateless. In all other cases, we implement custom forward and backward functions leveraging Py- Torch ATen library functions to flexibly support multiple operators and execution schedules. Each backward operator implementation is annotated with its computational dependencies, which is gen- erally the input or the output of its corresponding forward operator. Certain backward operators implementations may have dependencies on intermediate activations generated in the forward pass. For example, an intermediate-activaeed ReLU backward uses an encoded bitmask representing the sign of forward operator's 's input. We annotate this as an intermediate storage node and add it to our optimization problem, with strict recomputation dependency of the interemediate storage node on its creator node. Our operator optimizations select from different backward operator implementa- tions, convolution algorithms, in-place operators etc. We split the convolution backward operator into two a parameter-gradie gradient operator does not have any computational dependency on the forward pass, we can agres- sively free the forward input tensor right after the parameter-gradient is computed. We also reuse BatchNorm statistics in case of their recomputation. For our experiments, we limit ourselves to full precision training as quantization or lower precision computations introduce additional noise into SGD and change its convergence properties. We solve the joint optimization problem using the CVXPY (Diamond & Boyd 2016 Agrawal et al., 2018) solver with Gurobi (2014) backend.
operator followed by an input-gradient operator. Since the input-
MONET workflow. We obtain the forward pass dependencies in MONET by JIT tracing a model to obtain its graph. We profile each layer for workspace memory and compute cost, and obtain memory usage of the tensors from their shape and type. Note that the workspace memory for many convolu- tion operators in VGG-16 is greater than 2GB, making important factor model. Unlike prior approaches like Checkmate, we account for this workspace memory in our optimization problem, bringing the memory model very close to actual memory allocation. We phrase a boolean inte- ger programming problem using the generated graph and the profiled compute cost and workspace memory and solve using the CVXPY (Diamond Boyd [0166] Agrawal et al.) 2018)) modeling language and GUROBI (Gurobi 2014) solver. The solution is used to generate schedule that can be run by the MONET scheduler.
Operator optimizations. We divide operator optimizations according to the different type of imple- mentations they select from. (1) Output-activare activatedi: Backward calculation of operators like ReLU and BatchNorm can have computational dependency either on on their forward node's inputs or outputs. (2) Intermediate-activatie
Backward of ReLU has computational dependency on a 1-bit encoding of the sign of its forward node's input. Backward of MaxPool calculated using an intermediate 8- bit output-shaped tensor which contains the kernel-index of the maximum element. (3) Convolution algorithmss: We choose from forward and 6 backward cuDNN convolution algorithms. (4) Implace operations: The solver can choose to do inplace computation for operators like ReLU forward. We discuss constraints for in-place operator selection in C.3. All MONET experiments enable in il-paaee operation selection.
14

---

Published as a conference paper at ICLR 2021
E BASELINE IMPLEMENTATIONS
Checkmate implementation. We reimplement Checkmate (Jain et al. 2019) in PyTorch for our comparisons. We use the Checkmate solver as-is to obtain Checkmate schedules. Since Checkmate does not provide an execution engine for PyTorch, we run the generated Checkmate schedules on our own execution framework. Our inference engine uses the same operator implementations for Checkmate and MONeT. We have released our Checkmate implementation with the MONET code.
Gist implementation. Gist (Jain et al. 2018) is an operator- beesat memory-efficien scheme for training DNNs. encodes stashed forward tensors into smaller tensors which require less memory. Jain et al. (2018) evaluate Gist using CNTK on an Nvidia Maxwell GTX Titan X GPU. Since we implement MONET in PyTorch and have access to an Nvidia P100 GPU, a direct comparison with the numbers in the Gist paper is not possible. As an official Gist implementation is not available, reimplement on PyTorch and evaluate its exccution using MONET 's execution framework.
We implement all Gist optimizations Binarize (intermediate encodings for ReLU-Pool layers), Sparse Storage Dense Compute (compress and store sparse convolution inputs in ReLUUCCov layers as sparse storage), Delayed Precision Reduction (storing stashed non-compressed tensors in FP-16, but computing in FP-32), and Inplace (performing ReLU operators in-place wherever possible) over MONET's execution framework. In Gist, the Sparse Storage Dense Compute (SSDC) technique creates a sparse storage tensor in the Compressed Sparse Row (CSR) representation using the Nvidia cuSPARSE library. The dense storage is reshaped into a 256-sized column tensor before storing in a sparse format, allowing the column index of CSR representation to be saved in 8 bits instead of using 32 bits (termed Narrow Value Optimization in the paper). We also implement SSDC using with CUDA Toolkit version 10.1
Nvidia's s cuSPARSE library (function using PyTorch's C++ extensions.
In their paper. Jain et al. (2018) use the most memory-efficient convolution algorithms in Gist and compare its memory saving against a baseline which also chooses the most memory-efficient con- volution algorithm. Using memory-efficient convolution algorithms, our Gist reimplementation can train VGG-16 16 with 0.55x of the PyTorch-requiree memory (1.81x memory footprint), which is close to the data presented by Jain et al. (2018). However, it is 59% slower than when convolution selection is enabled, in which case it can train using 0.76x of the Py foorch-required memory. Since implementing Gist using memory-effeiient convolutions is not optimal in terms of compute time, implement Gist to use PyTorch's convolution selection algorithm. For all models other than VGG- 16 and UNet, see similar memory savings for Gist with memory-efficient convolutions and with invollution-electtonn enabled. We have released our Gist implementation with the MONET code.
F ON OPERATOR SELECTION FOR CHECKMATE
In this section, we briefly explain the difficulties of including operator selection directly into check- mate (Jain al. 2019). We will refer directly notation and equations in the checkmate paper (arxiv v3; 14 May 2020). The most direct way to incorporate operator selection into checkmate to introduce an auxiliary variable RY,, (0,1) that refers to re-computing layer time t us- ing implementation v. Most constraints in equation could stay the same, given Re, =.. R., and loss (la) E.E, E. C,, Some of our operators produce a different kind of checkpoint (e.g. binary activated ReL.Us), which could be handled in check-mate by splitting Sy The main issues in checkmate arise in the memory modeling and its relaxations (eq 4.5,7). The memory consumed by a specific checkpoint may depend on the operator implementation: DEPS[K] and USERS[i] both depend on the operator implementation (output activated, input activated, ...). In short, the checkmate computation graph is dynamic and depends on operator implementations. The most direct way to address this is to mem mem.ffreedyff (vx) = Σ. R,,, mem mem_fredd (1%) in a implementa- tion dependent way mem freed, (vk), and select the right version dependent on the operator used. Likewise, we need to extend FREE,
account for different operator implementations in Ri Likewise the product in equation (5) will now go over all implementations R using different USERS sets. This leads to a linear blowup in the number of constraints, and number of auxil- iary variables, leading to mem freed, (vx) = E. Ry mem freed,(vs) is a quadratic constrain that further needs to be re- solved using additional auxiliary variables. Given that Checkmate already pushes the limits of
least quadratic expansion
computational costs. Furthermore,
15

---

Published as a conference paper at ICLR 2021
Compute overhead (%)
20
40
NoOp
15
lnt
15
Out
10
Com
20
10
10
5
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
1
0.4
0.6
0.8
Memory ratio
Memory ratio
Memory ratio
Memory ratio
Memory ratio
(a) ResNet-55 (184)
(b) GoogleNet (320)
(c) UNet (11)
(d) VGG-16 (e) Mobile-V2 (272)
Figure 6: Ablation results on ResNet-50, GoogleNet, UNet, VGG-16, MobileNet-V2
current solvers, it is unlikely able to handle this explosion in constraints and variables, without significant modifications. MONET in the other hand represents the compute-gee graph more compactly and efficiently integrates different operator implementations.
G DETAILED ABLATIONS
Fig. 6 shows a detailed plot of our ablation experiments comparing the compute overhead of vari- ants of MONET across range of memory limits. Y-axis shows the compute overhead over PyTorch and X-axis shows the memory ratio to a PyTorch model. All variants which are not conv-optimized greedily post-optimized to use the fastest convolution. We see that MONET with no opera- tor optimization (NoOp) is generally slower than the other variants for all models and memory limits. Convolution and output-activated optimizations are both important in reducing compute overhead. MobileNet-Vi V2 uses depthwise separable convolutions, and hence does not significantly benefit from convolution-optiiizatooo ReLU operators, for which we have not implemented internediate-actiiatie backward optimization. interemediate-activv activated optimizations provide memory savings in memory-inie -intessve models, allow- ing models like VGG-16 16 reach memory savings which not attainable by other optimizations. All optimizations together result in the least compute overhead for any model or memory limit.
Further, MobileNet-V V2 has ha hardtanh operators instead of
H SOLVER TIME AND ILP STATISTICS
Solver runtime. MONET's solver runtimes vary for different models and different memory limits. We evaluate schedules obtained using solver times set to maximum of 24 hours. For moderate memory limits, both MONeT and Checkmate achieve an optimal solution before 24 hours. For tighter memory limits, the solution obtained by MONeT and Checkmate may not be most optimal. For multiple models and memory limits, Table Sec. [5) shows the time it takes for the solver to reach 5% close to the optimal solution for Checkmate, MONET-NOOP (MONET with checkpointing enabled but mperator-opiimizztion disabled), and MONET. We add the data for MobileNet- V2 and UNet 5 which also follow similar pattern. We also provide Table T which shows the time taken by the solver to reach 2% close to the optimal solution. We note that it has similar behavior as the time taken the solver to reach 5% close to the optimal solution. MONET-NON close-to- to-timal solution 1.3x-139x faster than Checkmate. For larger models, MONeT's solver converges to a 2% close-to-optimal solution up to 16x faster than Checkmate. At tighter memory limits for MobileNet-V2, the Checkmate solver reaches 2% close-to-optimal solution faster than MONET, but is still much slower than MONET-NOOP.
converges to 2%
ILP statisties. For different models, Table 7 shows the solver statistics after presolving for the problem formulated by Checkmate, MONET-NOOP and MONET for a 10 GB memory limit. It shows the number of forward operators the model and the number of constraints and variables for each solver. MONET-NOOP, which is MONET with only checkpointing enabled and without using operator optimization, has on average 50% fewer constraints and 67% fewer variables than Checkmate. Jointly-optimized MONET has slightly larger number of constraints, and on average
16

---

Published as a conference paper at ICLR 2021
GB
GB
9 GB
10 GB
ResNet--50
Checkmate
8.96
12.01
10.78
4.54
2.98
MONeT-NoOp
1.18
0.46
0.09
0.07
MONeT
7.24
3.84
0.73
0.70
0.31
0.11
GoogleNet
Checkmate
12.72
4.56
4.32
3.92
0.86
MONeT--NOp
0.10
0.11
0.07
0.07
0.07
0.07
MONeT
3.53
0.47
0.54
0.31
0.25
0.24
MobileNet-V2
Checkmate
2.16
2.88
1.16
0.29
0.34
0,14
MONeT-NoOp
0.04
0.02
0.08
MONeT
0.37
0.28
0.52
0.05
0.06
0.03
UNet
Checkmate
0.149
0.031
0.022
0.020
0.010
0.009
MONcT-NoOp
0.048
0.002
0.002
0,002
0.002
0,002
MONeT
0.363
0.064
0,028
0.027
0,024
0.006
VGG-16
Checkmate
0.002
0.002
0.001
MONeT-NoOp
0.001
0.000
0.000
MONeT
0.003
0.003
0.003
0.003
0,003
Table 5: Solver time (in hours) to reach 5% close to optimal solution. MONeT-NoOp reaches
5% close-to-optimal solution 1.6x-117x faster than Checkmate. MONeТ gets close 5% of the optimal solution only in a few hours, and up-to 16x faster than Checkmate for larger models.
5 GB
6 GB
7 GB
8 GB
GB
10 GB
ResNet-50
Checkmate
16.44
13.43
11.91
5.74
3.81
MONeT-NoOp
2.06
1.28
0.16
0.08
0.07
MONeT
12.64
3.00
3.60
0.62
GoogleNet
Checkmate
15.08
4.93
5.04
3.92
0.90
MONeT-NoOp
0.10
0.11
0.07
0.07
0.07
0.07
МONет
5.47
5.34
0.31
0.25
0.24
MohileNet-V2
Checkmate
2.16
2.88
1.16
0.29
0.34
0.14
MONeT-NoOp
0.43
0.37
0.02
0.02
0.10
0.09
МONeТ
9.49
5.33
1.53
0.14
0.18
0.05
UNet
Checkmate
0.243
0.031
0.027
0.021
0.011
0.009
MONeT-NoOp
0.181
0.003
0.003
0.002
0.002
0.002
MONeT
5.001
0.204
0.164
0.069
0.083
0.027
VGG-16
Checkmate
0.003
0.002
0.001
MONeT-NoOp
0.001
0.000
0.000
МONeT
0.004
0.006
0.004
0.003
0.003
Table 6: Solver time (in hours) to reach 2 % close to optimal solution. MONeT-NoOp reaches
a 2% to-setal- solution 1.3x-139x faster than Checkmate. MONeT reaches 2% close-to- optimal solution within few hours in most cases, and up to 27x faster than Checkmate for larger models.
40% fewer variables than Checkmate. MONET's formulation more efficient, and might be the reason that it reaches good solution faster than Checkmate.
17

---

Published as a conference paper at ICLR 2021
Checkmate
MONoT-NoOp
MONeT
Fwd ops
Constraints
Variables
Constraints
Variables
Constraints
Variables
GoogleNet
215
719,327
519,252
221,630
104,673
781,747
362,640
ResNet-50
175
473,592
344,659
344,652
167,238
487,842
229,431
Mobilen. V2
153
337,316
247.033
153,828
74,579
241,478
115,047
65,715
48,744
32,273
15,982
73,624
36,548
VGG-16
40
25,334
18,968
12,772
6,306
21,918
11,234
Table 7: ILP statistics for Checkmate, MONeT-NoOp, and MONeT. MONeT-NoOp has on av- erage 50% fewer constraints and 67% fewer variables than Checkmate. MONeT has slightly larger number of constraints, on average 40% fewer variables than Checkmate.
I CONVOLUTION ALGORITHMS
Fig. shows the complex workspace memory-compute trade-off for different convolution algo- rithms. The memory used is not always inversely proportional to the compute requirement. Jointly optimizing convolution algorithms enables MONET make the best decisions for which convolu- tion algorithm to select.
J APPLICABILITY TO MEMORY-INTENSIVE MODELS
To further show MONET's applicability to memory- inteeseee models, we evaluate it on 3D- UNet (Çiçek et al. 2016), fullyyconvolutiooaal model for volumetric images. Fig. [ presents the runtime-memory trade-off for MONET on 3D-UNet. We used commonly used 3D- UNet implementation (Wolny) 2019, Wolny et al.] 2020) with training configuration similar to 3DUnet.conf wetoofocal.bouccco on a GB P100 GPU. MONET reduces memory usage to 0.54 PyTorch, while incurring 8.86% overhead in compute time. At memory ratio of 0.81, MONET incurs almost compu- tational overhead, because makes use of operator optimizations and is able to bring down the recomputation cost to zero.
provided in the repository and batch size of 22, which just
Time (ms)
0.4
Overhead (%
10
0.2
0
2
0.4
0.6
0.8
Workspace memory (GB)
Memory ratio
Figure 7: Memory compute for 7 convo- lution algorithms with 256 66x56x6 in- put, 3x3 kernel, 64 output channels.
Figure 8:
Runtime-memory trade-off curve for 3D-UNet using MONET. The green point denotes the PyTorch baseline.
18


Reviews:
[{"review_id": "zgoGYctB5q5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary:\n\nThe paper provides a framework (MoNET) to perform automatic memory optimization targeting deep neural networks. Their technique jointly optimizes the checkpointing schedule and the choice of individual ops to find the implementation with the least possible runtime overhead under certain memory constraints. They formulate the problem as a ILP problem where the objective is to minimize runtime subject to strict memory constraints. They show that the solutions achieve 3x reduction in memory footprint compared to a pytorch implementation with minimal runtime overhead.\n\n## Strengths:\n* Making joint decisions on checkpointing and on the implementation of individual ops. This enables the authors to make both global and local decisions and to exploit the synergies in both.\n* Experiments on multiple NN models to show the impact of their technique.\n* Easy to read formulation of the ILP problem.\n\n## Weaknesses and Questions for authors:\n* Solution times and scalability\nThe authors do not explicitly mention the solution times for their ILP formulation. Also, it is unclear to me how the ‘checkmate’ framework can be slower compared to MoNET (as authors claim in page 7), since checkmate is only solving for the optimal checkpointing schedule.  I am also curious to know the scalability of the system for larger problems, specially since the authors are doing a joint optimization. Statistics about ILP problem sizes would be helpful.\n* Questions about the formulation\n  * Why do you need to compute $r_i^k$ for $i > k$? The gradient computations of op $k$ only depends on ops with numbers > $k$ (Algorithm 2)\n  * How do you get the computational efficiency of ops? If fusion of ops happens inside the compiler, then adding up the computational efficiency of individual ops may not realistically model the execution behavior.\n  * The formulation does not take into account possible compiler optimizations such as fusion, tiling etc. for both memory consumption and runtime calculations.\n* Questions about experiments\n  * It’s unclear what implementations of ops are used in the PyTorch implementation? Are these default ops or did you do a post-optimization similar to what you did for checkmate?\n  * What if you set memory ratio = 1 and compare it with the pytorch implementation, what is the overhead?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "wOb533kplA4", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Optimizing Neural Network Memory is broadly done through two channels - 1)  local operator level - like storing only signs of a ReLU function or bit quantization \n2) global graph level - like optimizing checkpointing schedule for a given compute graph. These channels are usually orthogonal/independent. \n\nIn this paper, the authors propose MONet which tries to find the best checkpointing schedule that can jointly optimize both the above channels. \nThe authors create an auxiliary graph to encapsulate operators and perform schedule optimization on the new graph rather than the usual graph in existing frameworks.\n\nTo obtain the new graph, they first perform a theoretical quantification of the peak memory consumption of the forward pass,backward pass and recomputation.\nThen, under a fixed memory budget M, we try to optimize the operators for computational efficiency.\n\nEach operator has multiple implementations that trade-off workspace memory 'c' and compute efficiency \\tau and we can exactly use only one implementation per operator. \nSo we end up with a linear combination of compute costs for all operations that looks like \n\n\\sum_i\\sum_l(\\tau_i^l \\delta_il) where \\delta is an indicator of which implementation to choose.\n\nWith 3 such double summations, the authors come up with a join objective function which constraints the overall memory sum to <=M \nand minimizes the compute cost objective above (which can be solved by a linear program).\n\nOn several popular architectures like ResNet-50, VGG-16 and UNet etc, MONeT outperforms Checkmate by abt 1.3-2x on memory and full-blown PyTorch by abt 2-3x.\n\nPros:\n1. Thoretically optimized\n2. Experiments on multiple CNN architectures\n\nCons:\n\n1. Effetcively no comparisons against any other schemes except Checkmate.\n2. No other architectures optimizations like Transformers or large scale FFNs.", "extra_scores": {}, "extra_text": {}}, {"review_id": "lehSWIkOa4d", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Training deep learning models is becoming increasingly challenging due to a memory bottleneck that limits the size of the feature maps that can be stored. The paper presents an automatic framework (MONET) that minimizes the memory footprint for deep networks. The novelty of MONET is that it jointly optimizes over: (a) global compute graph level techniques (such as checkpointing) and (b) local techniques (such as memory-efficient implementations of individual operators). While there are several existing works that focus separately on optimizing global techniques (e.g. the work on “Checkmate”) or local techniques, MONET is the first to jointly optimize over global and local techniques.\n\nThe memory constraints are carefully analyzed for the forward and backward passes, and expressed as a 0-1 integer program, which is then solved using the state-of-the-art solver Gurobi. The experimental evaluation confirms the theoretical gains provided by the solution of the optimization problem. In particular, the authors compare with a vanilla implementation in PyTorch, with the Checkmate-D (default Checkmate that uses global techniques), and with Checkmate-O (post-optimized to greedily run the fastest convolution algorithm). It is interesting to notice that MONET offers significant gains in all cases, even over Checkmate-O, underlying the need to perform the joint optimization in order to obtain the best schedule.\n\nI advocate for the acceptance of the paper. The memory bottleneck is an acute problem in deep learning, and the paper provides a practical solution that can alleviate this problem to some extent. I encourage this line of work and hope to see such schedule optimization become a standard option in deep learning frameworks. The paper is very well written, with a clear description of the approach and convincing experimental evaluation, while providing abundant references about related work.\n\nMinor comments, questions for the authors:\nThe memory requirements for some of the networks shown in the experimental evaluation are still small. While I understand that the GPU memory is limited to 16GB and there was a desire to compare against vanilla implementations, I think it would be interesting to show how models that require a lot of memory for training can benefit from MONET. For example, the UNet that was used in the paper seems to be the 2D version. The 3D UNet can require up to several hundred GB without optimizations (for varying batch sizes). Would it be possible to include even a simple estimate of the possible gains when using MONET? What would be the computational tradeoff to train it on the GPU used in the paper?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "T6n8h4FcX1f", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors present MONeT, an automatic approach to jointly optimize operator cost and checkpoint scheduling for deep learning on a fixed memory budget.  The paper thoroughly defines the problem, relevant previous work, and the MONeT framework.  Given a fixed GPU memory budget, MONeT solves an integer program in order to jointly minimize the computational overhead of checkpointing with various operator implementation.  This approach is intuitive, as previous approaches, such as the recently proposed CheckMate, only optimize the checkpoint schedule.  The derived integer program is also a nontrivial extension of previous work.  With MONeT implemented in PyTorch, a large number of empirical results are presented, which show the superiority of MONeT compared to CheckMate, and show memory savings (versus impressively slight overhead) compared to PyTorch.\n\nThe paper is well written, the description of computational cost and the derivation of the integer program were interesting, and results are very compelling (and easy to understand).  One area where the paper could be improved is on the notation used throughout Section 4 (see below for suggestions), which was difficult to follow due to the density of the section, the large number of different variables/variants of variables described, and some implicit definitions.  There are also a few small details and discussions which seem warranted, but, overall, I enjoyed reading this paper.\n\nSpecific comments:\n\n-\"n Checkmate, changes in operator implementation induce a different computation graph\" <- While this is\ntechnically true, only the cost associated with that operator changes,\nyes?  In this way, Checkmate could run multiple passes over the\ncomputed static graph with different operator costs, but this approach\nwould require an expoential (in the number of operators with varying\ncosts) number of evaluations.\n\n-\"We reimplement Checkmate in PyTorch\" <-This is non-trivial, so please include some re-implementation details.\n\n-Please include the version of PyTorch which was forked for the Monet  and Checkmate implementations in Section 5.\n\n-As CheckMate currently only supports tensorflow, it would be very helpful if the authors could also release the source for CheckMate in PyTorch when the source for MONeT is released.\n\n-Could you comment on the solver runtime needed to solve the integer\nprogram in monet, in contrast to the solver runtime for the MILP in\ncheckmate?\n\n-How do open-source solvers compare to the runtime for the commerical\nGurobi solver for the integer programs solved in monet?\n\n-\"We measure the empirical performance of the checkpointed schedules\nrunning on GPUs instead of just providing the solver values; this is\nimportant since Checkmate doesn’t consider workspace cost and\noverestimates its savings... Hence,\nwe show the results with solver running for 1 day for both MON E T and\nCheckmate. In contrast, MON E T finds the execution plans efficiently,\nits 1-hr solution already close to the 1-day solution with a small\ndifference of 1-2%.\"  <- This paragraph of worded somewhat vaguely; is\nthis to say 24 hours were included in execution time?\n\n-For VGG-16 in the ablation study in Figure 4, PyTorch exceeds\ndevice-memory for this dataset, yes? Is this the reason why monet\nachieves negative overhead; i.e., faster execution time than PyTorch\nitself?\n\nComments/questions regarding notation:\n\n-What is variable r in the schedule (s,r)?  It *seems* like r is the\n indicator function for activations which require recomputation.  Is\n that correct?  If so, please (please) state this explicitly in the\n paper.  If not, please define r.\n\n-A supplementary table for variables would be helpful.  It took some\n time to find a definition for $y$ in Equation 2.  While I eventually\n found one in Algorithm 2 (please define this explicitly inline,\n preceding Equation 2), a table would have made this much easier,\n especially considering how dense Section 3 is.\n\n-By the time the reader gets to Section 4, the table of variables\n becomes mandatory (there are different values of L and S with various\n superscripts, subscripts, and hats, it is very difficult to recall\n which is which and to look back in the dense text for their\n definitions).\n\n-Also, if possible, please standardize notation by the three\n categories: \n(a) peak\n forward pass memory consumption\n(b) peak backward pass memory consumption\n(c) peak recomputation memory consumption\nso that a reader can ascertain what collection Ls, Ds, and Ss are\nbeing referred to.  I understand there is overlap between these three\ncategories, but there must be some organizational way to more\neasily refer to these variables without having to research for their\ndefinitions when reading later sections of the paper.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agree that Monet proposed in the paper which optimizes for both local and global memory saving in Deep learning models is theoretically sound and experimentally convincing.
Accept!
Decision: Accept (Spotlight)