Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
Authors: Frank Schneider
Felix Dangel
Philipp Hennig
Venue: NeurIPS 2021 Poster (2021)
Keywords: deep learning
optimization
stochastic optimization
SGD
visualization
debugger

Abstract: When engineers train deep learning models, they are very much "flying blind". Commonly used methods for real-time training diagnostics, such as monitoring the train/test loss, are limited. Assessing a network's training process solely through these performance indicators is akin to debugging software without access to internal states through a debugger. To address this, we present Cockpit, a collection of instruments that enable a closer look into the inner workings of a learning machine, and a more informative and meaningful status report for practitioners. It facilitates the identification of learning phases and failure modes, like ill-chosen hyperparameters. These instruments leverage novel higher-order information about the gradient distribution and curvature, which has only recently become efficiently accessible. We believe that such a debugging tool, which we open-source for PyTorch, is a valuable help in troubleshooting the training process. By revealing new insights, it also more generally contributes to explainability and interpretability of deep nets.

COCKPIT: A Practical Debugging Tool for the Training of Deep Neural Networks
Frank Schneider" University of Tübingen Maria-von- Lan-L-iinen--taalfe 6 Tübingen, Germany f. wtwpeeiieeeunnitteiinneeeeeeeeee
Felix Dangel' University of Tübingen vonii-voo--innen- Tübingen, Germany f.
Philipp Hennig
University of Tübingen &
MPI for Intelligent Systems Tübingen, Germany
philipp. hennip.eeiipe.. htwp:luiiitubbiinnn..mm
Abstract
When engineers train deep learning models, they are very much "flying blind". Commonly used methods for real-time training diagnostics, such as monitoring the train/test loss, are limited. Assessing network's 's training process solely through these performance indicators akin to debugging software without access inter- nal states through debugger. To address this, we present COCKPIT, collection of instruments that enable closer look into the inner workings a learning machine, and a more informative and meaningful status report for practitioners. It facilitates the identification of learning phases and failure modes, like ill-chosen hyperparame- ters. These instruments leverage novel higher-order information about the gradient distribution and curvature, which has only recently become efficiently accessible. We believe that such a debugging tool, which we open-source for PYTORCII is valuable help in troubleshooting the training process. By revealing new insights, it also more generally contributes to explainability and interpretability of deep nets.
Introduction and motivation
Deep learning represents new programming paradigm: instead of deterministic programs, users design models and "simply" train them with data. this metaphor, deep learning is meta- programming form, where coding replaced by training. Here, ponder the question how provide more insight into this process by building a debugger specifically designed for deep learning.
Debuggers are crucial for traditional software development. When things fail, they provide access to internal workings of code, allowing a look "into the box". This is much more efficient than re-running the program with different inputs. And yet, deep learning is arguably closer to the latter. the attempt to train a deep net fails, a machine learning engineer faces various options: Should they change the training hyperparameters (how?); the optimizer (to which one?); the model (how?); just re-run with a different seed? Machine learning toolboxes provide scant help to guide these decisions.
Of course, traditional debuggers can be applied to deep learning. They will give access to every single weight of neural net, or the individual pixels of its training data. But this rarely yields insights
Equual contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Loss Curve
Loss Landscape
Alpha Distribution
Mini Batch Train Lows
1.1
Normalized Loss
1.0
Density
0.9
0.8
0.7
12
16
6.
1.0
0.5
0.0
0.5
1.0
Iteration
Normalized Step Length
Figure Illustrative example: Learning curves do not tell the whole story. Two different optimization runs (--) can lead to virtually the same loss curve (left). However, the actual optimization trajectories (middle), exhibit vastly different behaviors. In practice, the trajectories are intractably large and cannot be visualized directly. Recommendable actions for both scenarios (increase/decrease the learning rate) cannot be inferred from the loss curve. The a-distribution, one COCKPIT instrument (right), not only clearly distinguishes the two scenarios, but also allows for taking decisions regarding how the learning rate should be adapted. See Section 3.3 for further details.
towards successful training. Extracting meaningful information requires a statistical approach and distillation of the bewildering complexity into manageable summary. Tools like TENSORB [1] or WEIGHTS & BIASES [6] were built part to streamline this visualization. Yet, the quantities that are widely monitored (mainly train/test loss & accuracy), provide only scant explanation for relative differences between multiple training runs, because they do not show the network's 's internal state. Figure illustrates how such established learning curves
describe the current state of the model - whether it is performing well or not while failing to inform about training state and dynamics. They tell the user that things going well or badly, but not why. The situation is similar to flying a plane by sight, without instruments to provide feedback. It is not surprising, then, that achieving state-of-the-ar performance in deep learning requires expert intuition, or plain trial & error.
We aim to enrich the deep learning pipeline with a visual and statistical debugging tool that uses newly proposed observables as well several established ones (Section 2). We leverage and augment recent extensions to automatic differentiation (i.e. BACKPACK [12] for PYTORCH [33]) to efficiently access second-order statistical (e.g. gradient variances) and geometric (e.g. Hessian) information. We show how these quantities can aid the deep learning engineer tasks, like learning rate selection, as well as detecting common bugs with data processing model architectures (Section 3).
Concretely, we introduce COCKPIT, a flexible and efficient framework for online-monitoring these observables during training in carefully designed plots we call "instruments" (see Figure 2). To be of practical use, such visualization must have manageable computational overhead. We show that COCKPIT scales well to real-world deep learning problems (see Figure 2 and Section 4). We also provide three different configurations of varying computational complexity and demonstrate that their instruments keep the computational cost well below factor of 2 in run time (Section 5), It available as open-source code,² extendable, and seamlessly integrates into existing PYTORCII training loops (see Appendix A).
2 COCKPIT'S instruments
Setting: We consider supervised regression/classffcatiin with labeled data (x, y) E X x Y gen- crated by distribution P(æ, y). The training set D = {(x,. yn) = I,...,NY NN consists of N i..dd samples from P and the deep model f : ④ x X + Y maps inputs I,, to predictions ún by param- eters 0 E R This prediction is evaluated by a loss function L: Y x Y
which compares to the E(f ().f y) dP(z, y) by empirical approximation through (D( N E ((f(e, x.,) yn) = N ". e, (0). which
label yn- The goal is minimizing inaccessible expected risk Lr(0)
Pttpps: pottps://ot :/ http:/uu..cooottaneeeeeeeeeeeeee

---

Cooget for 500 CEAR 300 CICC
XTLP 103
GANDENTS
CINNNILRR
Figure 2: Screenshot of COCKPIT's 's full view while training the ALL-CNN-C [41] on CIFAR-100 with SGD using a cyclical learning rate schedule. This figure and its labels are not meant to be legible, but rather give impression of COCKPIT's the information currently tracked by most practitioners. The individual instruments are discussed in Section and observations described in Section 4. An animated version can be found in the accompanying GitHub repository.
experience. Gray panels (bottom row) show
in practice though can only be stochastically sub- sub-sampled on mini-batches B [L.....N].
(0
1
Σ (0)
(1)
|
nEB
As is standard practice, we use first- and second-order information of the mini-batch loss, described by its gradient (0) and Hessian HB(O).
1
98 ())
Σ (O), DEB
Hs(0)=
Σ (O). nEB
|B]
[
9n
Design choices: To minimize computational and design overhead, we restrict the metrics to quantities that require additional model evaluations. This means that, training step + with mini-batches B, Be+1 and parameters 0,, 0.41, we may access information about the mini-batch losses CB, (O,) and LB.,, (0++1), but cross-terms that would require additional forward passes.
Key point: (0) (e(0)) (0), and (f) are just expected values of distribution over the batch. Only recently, this distribution has begun attract attention [15] as its computation has become more accessible [8; 12]. Contemporary optimizers leverage only the mean gradient and neglect higher moments. One core point of our work is making extensive use of these distribution properties, trying to visualize them in various ways. This out-of-the-box support for the carefully selected and efficiently computed quantities distinguishes COCKPIT from tools like TENSORBOARY that offer visualizations as well. Leveraging these distributional quantities, we create instruments and show how they can help adapt hyperparameters (Section 2.1), analyze the loss landscape (Section 2.2), and track network dynamics (Section 2.3). Instruments can sometimes be built from stready-computedd information or are efficient variants of previously proposed observables. To keep the presentation concise, we highlight the instruments shown in Figure 2 and listed in Table 1. Appendix C defines them formally and contains more extensions, such as the mean GSNR [27], the early stopping [29] and CABS [4] criterion, which can all be used in COCKPIT.

---

Table 1: Overview of COCKPIT quantities. They range from cheap byproducts, nonlinear transformations of first-order information and Hessian- based measures. Some quantities have already been proposed, others first be considered in this work. They categorized into configurations economy C business
C full based on their run time overhead (see Section for a detailed evaluation).
Name
Short Description
Config
Pos. in Figure 2
Alpha
Normalized step size on a noisy quadratic inter-
economy
top left
polation between two iterates O.. 0c+1
Distance
Distance from initialization ||0, 08ll
economy
middle left
UpdateSize
Update size ||0++1 Oel12
economy
middle left
GradNorm
Mini-batch gradient norm g (0)ll2
economy
bottom left
NormTest
Normalized fluctuations of the residual norms |9s 9n 22, proposed in [9]
economy
top center
InnerTest
Normalized fluctuations of the 9n In's parallel components along IB. proposed in [7] Same as InnerTest but using the orthogonal components, proposed in [7]
economy
top center
OrthoTest
economy
top center
GradHistid
Histogram of individual gradient elements, {g. (0,)) )==.....
economy
middle center
TICDiag
Relation between (diagonal) curvature and gra- business dient noise, inspired by [43]
bottom right
HessTrace
Exact or approximate Hessian trace, business Tr (O)), inspired by [50]
middle right
HessMaxFV
Maximum Hessian eigenvalue, Amax (H)())), full inspired by [50]
top right
GradHist2d
Histogram of weights and individual gradient
full bottom center
elements, {(0). (. (e)))
Bug types: We distinguish three types of bugs encountered in deep learning. Implementation bugs low-level software bugs that, for example, trigger syntax errors. Training bugs result in unnecessarily inefficient or even unsuccessful training. They can, for example, stem from erroneous data handling (see Section 3.1), the chosen model architecture (see Section 3.2), or ill-chosen hyperparameters (see Section 3.3). Prediction bugs describe incorrect predictions a trained model on specific examples. Traditional debuggers are well-suited to find implementation bugs. COCKPIT focuses on efficiently identifying training bugs instead.
2.1 Adapting hyperparameters
One big challenge in deep learning setting the hyperparameters correctly, which is currently mostly done by trial & error through parameter searches. We aim to augment this process with instruments that inform the user about the effect that the chosen parameters have on the current training process.
Alpha: Are we crossing the valley? Using individual loss and gradient observations at the start and end point of each iteration, we build noise-informed univariate quadratic approximation along the step direction (i.e. the loss as function of the step size), and assess to which point on this parabola our optimizer moves. We standardize this value a such that stepping the valley-floor is assigned 0, the starting point is at -1 and updates to the point exactly opposite of the starting point have a 1 (see Appendix C.2 for a more detailed visual and mathematical description of a). Figure/ 1 illustrates the scenarios a +1 and how monitoring the a-distribution (right panel) can help distinguish between two training runs with similar performance but distinet failure sources. By default, this COCKPIT instrument shows the a-distribution for the last 10 % of training and the entire training process (e.g. top left plot in Figure 2). In Section 3.3 we demonstrate empirically that, counter-inten intuitiittuitivell, it is generally not good idea choose the step size such that is close zero.

---

Distances: Are we making progress? Another way to discer the trajectories in Figure is by measuring the L2 distance from initialization [31] and the update size [2: 16] in parameter space. Both are shown together in one COCKPIT instrument (see also middle left plot in Figure 2) and are far larger for the blue line in Figure 1. These distance metrics are also able to disentangle phases for the blue path. Using the same step size, it will continue to "jump back and forth" between the loss valley's s walls but at some point cease to make progress. During this "surfing of the walls", the distance from initialization increases, ultimately though. it will stagnate, with the update size remaining non-zero, indicating diffusion. While the initial "surfing the wall"-phase benefits training (see Section 3.3), achieving stationarity may require adaptation once the optimizer reaches that diffusion.
How steep is the wall? The update size will show that the orange trajectory is stuck. But why? Such down can result from both a bad learning rate and from loss landscape plateaus. The gradient norm (bottom left panel in Figure 2) distinguishes these two causes.
Gradient norm:
Gradient tests: How noisy is the batch? The batch size trades off gradient accuracy versus computational cost. Recently, adaptive sampling strategies based on testing geometric constraints between mean and individual gradients have been proposed [9; 7J. The norm, inner product, and orthogonality tests use a standardized radius and two band widths (parallel and orthogonal to the gradient mean) that indicate how strongly individual gradients scatter around the mean. The original works use these values to adapt batch sizes. Instead, COCKPIT combines all three tests into a single gauge (top center plot of Figure 2) using the standardized noise radius and band widths for visualization. These noise signals can be used to guide batch size adaptation on- and offline, or to probe the influence of gradient alignment on training speed [37] and generalization [10; 11; 27].
2.2 Hessian properties for local loss geometry
An intuition for the local loss landscape helps in many ways. can help diagnose whether training is stuck, to adapt the step size, and explain stability or regularization [18; 23J. The key challenge is the large number of weights: Low-dimensional projections of surfaces can behave unintuitively [30], but tracking the extreme average behaviors may help in debugging, especially if first-order metrics fail.
Hessian eigenvalues: A gorge or a lake? In convex optimization, the maximum Hessian eigenvalue crucially determines the appropriate step size [38]. Many works have studied the Hessian spectrum in machine learning [e.g. 17; 18; 30; 35; 36; 50]. In short: curvature matters. Established [34] and recent autodiff frameworks [12] can compute Hessian properties without requiring the full matrix. COCKPIT leverages this to provide the Hessian's largest eigenvalue and trace (top right and middle right plots in Figure 2). The former resembles the loss surface's 's sharpest valley and can thus hint at training instabilities [23]. The trace describes notion of "average curvature", since the eigenvalues A relate to it by E. Ai- (r)H)(O)), which might correlate with generalization [22].
TIC: How do curvature and gradient noise interact? There is an ongoing debate about curvature's link to generalization [e.g. 14; 21; 24]. The Takeuchi Information Criterion (TIC) [42; 43] estimates the generalization gap by a ratio between Hessian and non-central second gradient moment. It also provides intuition for changes in the objective function implied by gradient noise. Inspired by the approximations in [43], COCKPIT provides mini-batch TIC estimates (bottom right plot of Figure 2).
2.3 Visualizing internal network dynamics
Histograms are a natural visual compression of the high-dimensional [B] x D individual gradient values. They give insights into the gradient distribution and hence offer a more detailed view of the learning signal. Together with the parameter associated to each individual gradient, the entire model status and dynamics can be visualized in a single plot and be monitored during training. This provides more fine mee-iniieed view of training compared to tracking parameters and gradient norms [16].
Gradient and parameter histograms: What is happening in our network? COCKPIT offers a uni- Additionally, combined histogram of murumeler-grudien pairs [[0), 9n (0)))== ......,, provides two-dimensional look into the network's gradient and parameter values in a mini-batch. Section 3.1 shows an example use-case of the gradient histogram; Section 3.2 makes the case for the layer-wise variants of the instruments.
variate histogram of the gradient elements {g.. (0)) |.=..... (
5

---

Gradient Element
Gradient Element
0
100
10
10
10
10
10
(a) Normalized Data
(b) Raw Data
Figure 3: Same inputs, different gradients; Catching data bugs with COCKPIT. (a) normalized ((0, 1)) and (b) raw ((0, 255)) images look identical in auto-scaled front-ends like MATPLOTTLIR'S
imshow. The gradient distribution the 3C3D model, however, is crucially affected by this scaling.
3 Experiments
The diverse information provided by COCKPIT can help users and researchers in many ways, some of which, just like for a traditional debugger, only become apparent practical use. In this section, we present few motivating examples, selecting specific instruments and scenarios in which they are practically useful. Specifically, we show that COCKPIT can help the discern between, and thus fix, common training bugs (Sections 3.1 and 3.2) that are otherwise hard to distinguish they lead the same failure: bad training. We demonstrate that COCKPIT can guide practitioners to choose efficient hyperparameters within single training run (Sections 3.2 and 3.3). Finally, we highlight that COCKPIT'S instruments can provide research insights about the optimization process (Section 3.3). Our empirical findings are demonstrated on problems from the DEEPOBS [39] benchmark collection.
3.1 Incorrectly scaled data
One prominent source of bugs is the data pipeline. To pick relatively simple example: For standard optimizers work at their usual learning rates, network inputs must be standardized (i.e. between zero and one, or have zero mean and unit variance [e.g. 5). If the user forgets to do this, optimizer performance is likely to degrade. It can be difficult identify the source of this problem it does not cause obvious failures, NaN or Inf gradients, etc. We now construct semi-realistic example, to show how using COCKPIT can help diagnose this problem upon observing slow training performance.
By default", the popular image data sets CIFAR- 10/100000 [25] are provided as NUMPY [20] arrays that consist of integers in the interval [0, 255]. This data, instead of the widely used version with floats in [0, 1], changes the data scale by a factor of 255 and thus the gradients as well. Therefore, the optimizer's 's optimal learning rate is scaled well. In other words, the default parameters of popular optimization methods may not work well anymore, or good hyperparameters may take extreme values. Even if the user directly inspects the training images, this may not be apparent (see Figure 3 and Figure in the appendix for the same experiment with VGG16 on IMAGENET). But the gradient histogram instrument of COCKPIT, which has deliberate default plotting range around [-1, to highlight such problems, immediately and prominently shows that there is an issue.
Of course, this particular data is only placeholder for real practical data sets. While this problem may not frequently arise in the highly pro-procssedd packaged CIFAR- 10, it is not a rare problem for practitioners who work with their personal data sets. This is particularly likely in domains outside standard computer vision, c.g. when working with mixed-type data without obvious natural scales.
3.2 Vanishing gradients
The model architecture itself can be source of training bugs. As before, such problems mostly arise with novel data sets, where well-working architectures are unknown. The following example shows how even small (in terms of code) architecture modifications may severely harm the training.
html

---

Network
Parameter 0
Parameter 4
Parameter 10
Gradient Element
Gradient Element
100
10
10
10
10
10"
102
10"
10
10
10
10
102
10*
(a) Network Histogram
(b) Layer-wise Histograms
Figure 4: Gradient distributions of two similar architectures on the same problem. (a) Distri- bution of individual gradient elements summarized
the entire network. Both seem similar. (b) Layer-wise histograms for a subset of layers. Parameter O is the layer closest to the network's ' input, parameter 10 closest its output. Only the layer-wise view reveals that there are several degenerated gradient distributions for the orange network making training unnecessary hard.
Test Accuracy
0.75
MNIST MLP
SVHN 3e3d
F-MNIST MLP
CIFAR- 3c3d
0.25
-0.6
-0.4
-.2
0.4
0.6
Median &
Figure 5: Test accuracy as function of standardized step size a. For four DEEPOBS problems (see Appendix E), final test accuracy is shown versus the median a-value over the entire training. Marker size indicates the magnitude of the raw learning rate, marker color identifies tasks (see legend). For each problem, the best-performing setting highlighted by vertical colored line.
Figure 4a shows the distribution of gradient values of two different network architectures in blue and orange. Although the blue model trains considerably better than the orange one, their gradient distributions look quite similar. The difference becomes evident when inspecting the histogram laser-wise. We can see that multiple layers have a degenerated gradient distribution with many elements being practically zero (see Figure 4b, bottom row). Since the fully connected layers close to the output have far more parameters (a typical pattern of convolutional networks). they dominate the network-wide histogram. This obseures that major part of the model effectively unable train.
Both the blue and orange networks follow DEEPOBS's 3C3D architecture. The only difference is the non-linearity: The blue network uses standard ReLU activations, while the orange has sigmoid activations. Here, the layer-wise histogram instrument COCKPIT highlights which part of the architecture makes training unnecessarily hard. Accessing information fayer-wise is also essential due to the strong overparameterization deep models where training can happen in small subspaces [19]. Once again, this is hard to do with common monitoring tools, such the loss curve.
3.3 Tuning learning rates
Once the architecture is defined, the optimizer's learning rate is the most important hyperparameter to tune. Getting it right requires extensive hyperparameter searches at high resource costs. COCKPIT'S instruments can provide intuition and information to streamline this process: In contrast to the raw learning rate, the curvature standardiedd step size a-quantity (see Section 2.1) 1) has natural scale.
Across multiple optimization problems, we observe, perhaps surprisinly................ that best runs and indeed all good runs have median > 0 (Figure 5). This illustrates fundamental difference between stochastic optimization, as typical for machine learning, and classic deterministic optimization. Instead of locally stepping "to the valley floor" (optimal in the deterministic case), stochastic
7

---

optimizers should overshoor the valley somewhat. This need to "surf the walls" has been hypothesized before [e.g. 47; 49] as property of neural network training. Frequently, learning rates are adapted during training, which fits with our observation about positive a-values: "Overshooting" allows fast early progression towards areas of lower loss, but it does not yield convergence in the end. Real-time visualizations of the training state, as offered by COCKPIT, can augment these fine-tuning processes.
also indicates major challenge preventing simple automated tuning solutions: The optimal a-value is problem-dependent, and simpler problems, such as a multi-layer perceptron (MLP) on MNIST [26], behave much more similar to classic optimization problems. Algorithmic research on small problems can thus produce misleading conclusions. The figure also shows that the a- a-gauge is not sufficient by itself: extreme overshooting with a too-large learning rate leads to poor performance, which however can be prevented by taking additional instruments into account. This makes the case for the cockpit metaphor of increasing interpretability from several instruments in conjunction. By combining the a-instrument with other gauges that capture the local geometry or network dynamics, the user can better identify good choices of the learning rate and other hyperparameters.
Figure 5
4 Showcase
Having introduced the tool, we can now return to Figure 2 for closer look. The figure shows a snapshot from training the ALL-CNN-C [41] on CIFAR- 100 using SGD with a cyclic learning rate schedule (see bottom left panel). Diagonal curvature instruments are configured to use an MC approximation in order to reduce the run time (here, = 100, compare Section 5).
A glance at all panels shows that the learning rate schedule is reflected in the metrics. However, the instruments also provide insights into the early phase of training (first 100 iterations), where the learning rate is still unaffected by the schedule: There, the loss plateaus and the optimizer takes relatively small steps (compared to later, as can be seen in the small gradient norms, and small distance from initialization). Based on these low-cost instruments, one may thus at first suspect that training was poorly initialized; but training indeed succeeds after iteration 100! Viewing COCKPIT entirely though, it becomes clear that optimization in these first steps is not stuck at all: While loss, gradient norms, and distance in parameter space remain almost constant, curvature changes, which expresses itself in a clear downward trend of the maximum Hessian eigenvalue (top right panel).
The importance of early training phases has recently been hypothesized [16], suggesting logarithmic timeline. Not only does our showcase support this hypothesis. but it also provides an explanation from the curvature-based metrics, which in this particular case are the only meaningful feedback in the first few training steps. It also suggests monitoring training at log-spaced intervals. COCKPIT provides the flexibility to do so, indeed, Figure 2 has been created with log-schedules tracking events.
As a final note, we recognize that the approach taken here promotes an amount of manual work (monitoring metrics, deliberately intervening, etc.) that may seem ironic and at odds with the paradigm of automation that is at the heart of machine learning. However, we argue that this might be what is needed at this point in the evolution of the field. Deep learning has been driven notably by scaling compute resources [44], and fully automated, one-shot training may still be some way out. To develop better training methods, researchers, not just users, need algorithmic interpretability and explainability: direct insights and intuition about the processes taking place "inside" neural nets. To highlight how СоCKPIT might provide this, we contrast in Appendix F the COCKPIT view of two convex DEEPOBS problems: noisy quadratic and logistic regression on MNIST. In both cases, the instruments behave differently compared to the deep learning problem in Figure 2. particular, the gradient norm increases (left column, bottom panel) during training, and individual gradients become less scattered (center column, top panel). This is diametrically opposed to the convex problems and shows that deep learning differs even qualitatively from well-understood optimization problems.
5 Benchmark
Section 3 made a case for COCKPIT as an effective debugging and tuning tool. To make the library useful in practice, it must also have limited computational cost. We now show that it is possible to compute all quantities at reasonable overhead. The user can control the absolute cost along two dimensions, by reducing the number of instruments, or by reducing their update frequency.
8

---

Track Interval
Time Overhead
16
64
256
baseline
Configuration
economy
1.5
1.2
business
2
1.3
1.1
Base- line
Loss
Dis-
Grud
Update
Norm Test
Alpha Ortho
Inner Test
Grad
Hess Trace Diag
TIC
full
21
6.1
2.3
1.3
1.1
Norm
Size
Test
Histld
(a) Overhead COCKPIT instruments
(b) Overhead COCKPIT configurations
Figure 6: Run time overhead for individual COCKPIT instruments and configurations as shown on CIFAR-10 10 3C3D on a GPU. Left: The run time overheads for individual instruments are shown multiples of the baseline (no tracking). Most instruments add little overhead. This plot shows the overhead in one iteration, determined by averaging over multiple iterations and random seeds. Right: Overhead for different COCKPIT configurations. Adjusting the tracking interval and re-using the computation shared by multiple instruments can make the overhead orders of magnitude smaller. Blue fields mark settings that allow tracking without doubling the training time.
All benchmark results show SGD without momentum. COCKPIT's quantities, however, work for generic optimizers and can mostly be used identically without increased costs. One current exception Alpha which can be computed more efficiently given the optimizer's update rule."
Complexity analysis: Computing more information adds computational overhead, of course. How- ever, recent work [12] has shown that first-order information, like distributional statistics on the batch gradients, can be computed on top of the mean gradient at little extra cost. Similar savings apply for most quantities in Table 1, as they are (non-) mon-))inear transformations of individual gradients. subset of COCKPIT's quantities also networks on a classification task with C classes, the additional work is proportional to C' gradient backpropagations (i.e. C 10 for CIFAR- 10, C' 100 for CIFAR-1000 Parallel processing can, to some extent, process these extra backpropagations in parallel without significant overhead. fall back Monte Carlo (MC) sampling approximation, which
second-order information from the Hessian diagonal. For ReLU
this is longer possible, reduces the number of extra backprop passes to the number of samples (1 by default).
While parallelization is possible for the gradient instruments, computing the maximum Hessian eigenvalue inherently sequential. Similar to Yao et al. [50], we use matrix-free Hessian- products by automatic differentiation [34], where each product's costs are proportional to one gradient computation. Regardless of the underlying iterative eigensolver, multiple such products must be queried to compute the spectral norm (the required number depends the spectral gap to the second-lare largest eigenvalue).
Run time benchmark: Figure 6a shows the wall-clock computational overhead for individual instruments (details in Appendix E)." As expected, byproducts are virtually free, and quantities that rely solely on first-order information add little overhead most roughly 25 % on this problem). Thanks to parallelization, the ten extra backward passes required for Hessian quantities reduce to less than 100 overhead. Individual overheads also do not simply add up when multiple quantities are tracked, because quantities relying on the same information share computations.
To allow a rough cost control, COCKPIT currently offers three configurations, called economy. business, and full, in increasing order of cost (cf. Table l). As basic guideline, we consider factor of to be an acceptable limit for the increase in training time and benchmark the configurations'
"This is currently implemented for vanilla SGD. Otherwise, COCKPIT falls back to less efficient scheme. MC-sampled approximation
Hessian/generalized Gauss-Newton has been used Figure to reduce the prohibitively large number of extra backprops on CIFAR- 100 (C = 100).
"To improve readability, we exclude HessMaxEV here, because its overhead is large compared other quantities. Surprisingly, we also observed significant cost for the 2D histogram on GPU, is caused by an implementation bottleneck for histogram shapes observed in deep models. We thus also omit GradHist 2d here, as we expect it to be eliminated with future implementations (see Appendix E.2 for detailed analysis and further benchmarks). Both quantities, however, part of the benchmark shown in Figure 6b.

---

run times for different tracking intervals. Figure 6b shows a run time matrix for the CIFAR- 10 3C3D problem, where settings that meet this limit are set blue (more problems including IMAGENET are shown in Appendix E). Speedups due to shared computations are easy to read off: Summing all the individual overheads shown Figure 6a would result in a total overhead larger than 200 %, while the joint overhead (business) reduces to 140 %. The economy configuration can easily be tracked at every step of this problem and stay well below our threshold of doubling the execution time. COCKPIT'S full view, shown in Figure 2, can be updated every 64-th iteration without major increase in training time (this corresponds to about five updates per epoch). Finally. tracking any configuration about once per epoch which is common in practice adds overhead close zero (rightmost column).
This good performance largely due to the efficiency of the BACKPACK package [12], which we leverage with custom and optimized modification, that compacts information layer-wise and then discards unneeded buffers. Using layer-wise information (Section 3.2) scales better to large networks, where storing the entire model's individual gradients all at once becomes increasingly expensive (see Appendix E). To the best of our knowledge, many of the quantities in Table 1, especially those relying on individual gradients, have only been explored on rather small problems. With COCKPIT they can now be accessed at a reasonable rate for deep learning models outside the toy problem category.
Conclusion
Contemporary machine learning, in particular deep learning, remains a craft and an art High dimensionality, stochasticity, and non-convexity require constant tracking and tuning, often resulting in painful process of trial and error. When things fail, popular performance measures, like the training loss, do not provide enough information by themselves. These metries only tell whether the model is learning, but not why. Alternatively, traditional debugging tools can provide access to individual weights and data. However, in models whose power only arises from possessing myriad weights, this approach is hopeless, like looking for the proverbial needle in haystack.
To mitigate this, we proposed COCKPIT, a practical visual debugging tool for deep learning. It offers instruments to monitor the network's 's internal dynamics during training, in real-time. In its presentation, we focused on two crucial factors affecting user experience: Firstly, such a debugger must provide meaningful insights. To demonstrate COCKPIT'S utility, we showed how it can identify bugs where traditional tools fail. Secondly, it must come at a feasible computational cost. Although COCKPIT uses rich second-order information, efficient computation keeps the necessary run time overhead cheap. The open- source PYTORCH package can be added to many existing training loops.
Obviously, such a tool is never complete. Just like there is no perfect universal debugger, the list of current instruments naturally incomplete. Further practical experience with the tool, for example in the form of a future larger user study, could provide additional evidence for its utility. However, our analysis shows that COCKPIT provides useful tools and extracts valuable information presently not accessible to the user. We believe that this improves algorithmic interpretability helping practitioners understand how to make their models work but may also inspire new research. The code is designed flexibly, deliberately separating the computation and visualization. New instruments can be added easily and also be shown by the user's preferred visualization tool, e.g. TENSORBOARD. Of course, instead of just showing the data, the same information can be used by novel algorithms directly, stee-stepping the human in the loop.
Acknowledgments and Disclosure of Funding
The authors gratefully acknowledge financial support by the European Research Council through ERC SIG Action 757275 / PANAMA; the DFG Cluster of Excellence "Machine Learning New Perspectives for Science", EXC 2064/1, project number 390727645; the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FK7: 011$18039A) )) and funds from the Cyber Valley Initiative of the Ministry for Science Research and Arts of the State of Baden- Württemberg. Moreover, the authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Felix Dangel and Frank Schneider. Further, we are grateful to Agustinus Kristiadi, Alexandra Gessner, Christian Fröhlich, Filip de Roos, Jonathan Wenger, Julia Grosse, Lukas Tatzel, Marius Hobbhahn, and Nicholas Krämer for providing feedback to the manuscript.
10

---

References
[1] Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowwee R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R. Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., and Zheng, X. TensorFlow: Large-Scale Machine Learning Heterogeneous Systems, 2015. URL https: /ww..w.w....... tensorfloww org/.2
[2] Agrawal, A. M., Tendle, A., Sikka, H., Singh, S., and Kayid, A. Investigating Learning in Deep Neural Networks using Layer-Wise Wise Weight Change, arXiv preprint: 2011. 0011106700000 067006775. 2020. 5
[3] Bahamou, A. and Goldfarb, D. A Dynamic Sampling Adaptive-SGD Method for Machine Learning. arXiv preprint: 1912. 1335 2019. 24
[4] Balles, L.,, Romero, J., and Hennig, P. Coupling adaptive batch sizes with learning rates. In Conference on Uncertainty in Artificial Intelligence (UAI), 2017. 3, 17, 20, 21
[5] Bengio, Y. Practical recommendations for gradient-based training of deep architectures. Neural Networks: Tricks of the Trude (2nd ed.). pp. 437-478, 2012. 6
[6] Biewald, L. Experiment Tracking with Weights and Biases, 2020. URL htt.t:://// http::///wwwww.. com/. Software available from wandb.com. 2
Bollapragada, R., Byrd, R. H., and Nocedal, Adaptive Sampling Strategies for Stochastic Optimization. SIAM Journal on Optimization, 28:3312-3343, 2017. 4, 17, 22, 23, 24
[8] Bradbury. J.. Frostig, R., Hawkins, P.. Johnson, M. J., Leary, C., Maclaurin, D., and Wanderman-Milnee S. JAX: composable transformations of Python+NumPy programs, 2018. URL UtLhttp:///itthb google/ jax. 3
[9] Byrd, R. H., Chin, G. M., Nocedal, and Wu, Y. Sample Size Selection in Optimization Methods for Machine Learning. Math. Program.. 134(1):127-155, 2012. ISSN 0025-5610. 4, 5, 17,
[10] Chatterjee. S. Coherent Gradients: An Approach Understanding Generalization in Gradient Descent- based Optimization. International Conference on Learning Representations (ICLR), 2020.
[11] Chatterjee, S. and Ziclinski, P. Making Coherence Out of Nothing At All: Mcasuring the Evolution of Gradient Alignment. arXiv preprint: 2008. 01217, 2020. 5
[12] Dangel, F., Kunstner, F., and Hennig, P. BackPACK: Packing more into Backprop. In International Conference on Learning Representations (ICLR), 2020. 2, 3. 9, 10, 14
[13] Deng, J., Dong, W., Sacher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A Large-Scale Hierarchical Image Database. In IEEF conference computer vision and pattern recognition, pp. 248-255. IEFE, 2009. 26
[14] Dinh, L., Pascanu, R., Bengio, S., and Bengio. Y. Sharp Minima Can Generalize For Deep Nets. In International Conference on Machine Learning (ICML), 2017. 5
[15] Faghri, F., Duvenaud, D., Flect, D. J., and Ba, J. A Study of Gradient Variance in Deep Learning, arXiv preprint: 2007. 04532. 2020. 3
[16] Frankle, J., Schwab, J., and Morcos, A. S. The Early Phase of Neural Network Training. In International Conference on Learning Representations (ICLR), 2020. 5,8
[17] Ghorbani, B., Krishnan, S., and Xiao, Y. An Investigation into Neural Net Optimization via Hessian Eigenvalue Density. In International Conference on Machine Learning (ICML), 2019. 5
[18] Ginsburg, B. On regularization of gradient descent, layer imbalance and flat minima. arXiv preprint: 2007. 09286, 2020. 5, 26, 27
[19] Gur-Ari, G., Roberts, D. A., and Dyer, Gradient Descent Happens in Tiny Subspace. arXiv preprint: 0812.04754, 2018. 7
[20] Harris, C. R., Millman, K. J., van der Walt, J., Gommers, R., Virtanen, P., Cournapeau, D.,, Wieser, E., Taylor, J., Berg. S., Smith, J., Kern, R., Picus, M., Hoyer, S., van Kerkwijk, M. H., Brett, M., Haldane, A., del Río, J. F. Wiebe, M., Peterson, P., Gérard-Marchanl Marchant, P., Sheppard, K., Reddy, T., Weckesser, W., Abbasi, H., Gohlke, C., and Oliphant, T. E. Aray programming with NumPy. Nature, 58577825:3377--332 2020. 6

---

[21] Hochreiter, S. and Schmidhuber, J. Flat Minima. Neural Comput., 9():1142, 1997. 5,
[22] Jastrzebski. S., Arpit, D., Astrand, O., Kerg, G., Wang, H., Xiong, C., Socher, R., Cho, K., and Geras, K. Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization. arXiv preprint: 2012. 2012.14199. 14193, 2020. 5
[23] Jastrzebski, S., Szymezak, M., Fort, S., Arpit, D., Tabor, J., Cho, K., and Geras, K. The Break- Even Point on Optimization Trajectorics of Deep Neural Networks. In International Conference on Learning Representations (ICLR), 2020. 5
[24] Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, T. P. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In International Conference on Learning Representations (ICLR), 2017. 5
[25] Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, 2009. 6, 28
[26] Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- learning applied document recognition. In Proceedings of the IEEE, volume 86, pp. 2278-2324, 1998. 8, 15, 28
[27] Liu, Bai, Y.. Jiang, G., Chen, T., and Wang, H. Understanding Why Neural Networks Generalize Well Through GSNR of Parameters. International Conference on Learning Representations (ICLR), 2020. 3, 5, 17, 26
[28] Mahsereci, M. and Hennig, Probabilistic Line Searches for Stochastic Optimization. Journal of Machine Learning Research, 18(119):1-53 2017. 20
[29] Mahsereci, M., Balles, L., Lassner, C., and Hennig, P. Early Stopping without a Validation Set. arXiv preprint: 1 703.0000000 09580, 2017. 3, 17,21
[30] Mulayoff, R. and Michaeli, T. Unique Properties of Flat Minima in Deep Networks, In International Conference on Machine Learning (ICML), 2020. 26, 27
[31] Nagarajan, V. and Kolter, J. Z. Generalization in Deep Networks: The Role of Distance from Initialization. arXiv preprint: 1901. 01 672, 2019.
[32] Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu. B., and Ng. A. Y. Reading Digits Natural Images with Unsupervised Feature Learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. 28
[33] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L.,, Bai, and Chintala, PyTorch: An Imperative Style, High-Performance Deep Learning Library. Neural Information Processing Systems (NeurIPS), 2019. 2
[34] Pearlmutter, B. A. Fast Exact Multiplication by the Hessian. Neural Computation, 6(1):147-160, 1994. 5, 9, 24, 25
[35] Sagun, L., Bottou, L., and LeCun, Y. Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond. arXiv preprint: 1 61 1 07476, 2017.
[36] Sagun, L., Evei, U., Guney, V. U., Dauphin, Y., and Bottou, L. Empirical Analysis of the Hessian of Wver-Parametrized Neural Networks. arXiv preprint: 1766. 04454, 2018. 5
[37] Sankararaman, K. A., De, S., Xu, Z., Huang, W. R., and Goldstein, T. The Impact of Neural Network Overparameterrizaionn Gradient Confusion and Stochastic Gradient Descent. In International Conference on Machine Learning (ICML). 2020.
[38] Schmidt, M. Convergence rate of stochastic gradient with constant step size. 2014. URL https: pdf. 25
/ //www. cs ubc. -t/-sa/hssddtmmmmmcmcocm Documents/ 2o1._oootes_oonsttattsseeeeeeeee
[39] Schneider, F., Balles, I.., and Hennig, P. DeepOBS: Deep learning Optimizer Benchmark Suite. In International Conference on Learning Representations (ICLR), 2019. 6, 14, 28
[40] Simonyan, K. and Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition.
In 3rd International Conference on Learning Representationss ICLR, 2015. 26
[41] Springenberg, J. T.. Dosovitskiy, A., Brox, T., and Riedmiller, M. Striving for Simplicity: The All Convolutional Net. arXiv preprint: 1412. 6896, 2015. 8,
12

---

[42] Takeuchi, K. The distribution of information statistics and the criterion of goodness of fit of models. Mathematical Science, 153:12-18, 1976. 5, 25
[43] Thomas, V., Pedregosa, F., van Merriënboer, B., Manzagol, P.-.., Bengio, Y., and Roux, N. L.. On the interplay between noise and curvature and its effect optimization and generalization. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2020. 5, 17.
[44]
Thompson, N. C., Greenewald, K.. Lee, K., and Manso, G. F. The Computational Limits of Deep Learning. arXiv preprint: 2007. 05558, 2020.
[45] Vaswani. S., Mishkin, A.. Laradji. I., Schmidt. M., Gidel, G., and Lacoste-Julien, S. Painless Stochastic Gradient: Interpolation, LenchSearch, and Convergence Rates, Neural Information Processing Systems
(NeurIPS), 2019. 20
[46] Warsa, Wareing, T., Morel, Meghee, J., and Lehoueq, Krylov Subspace Iterations Deterministic k-Eigenvalue Calculations. Nuclear Science and Engineering NUCL SCI ENG, 147, 05 2004. doi: 10. 10.13182/NSE04-1. 24
[47] Wu, Y., Ren, M. Liao, R., and Grosse, R. B. Understanding short-horizon bias in stochastic meta- optimization. International Conference on Learning Representations (ICLR 2018), 2018. 8
[48] Xiao, H., Rasul, K., and Vollgraf, R. Fashion-MNIST: Novel Image Dataset for Benchmarking Machine
Learning Algorithms, arXiv preprint: 1 1708. 07747, 2017. 28
[49] Xing, C., Arpit, D., Tsirigotis, C., and Bengio, Y. A Walk with SGD. arXiv preprint: 1802. 08 770, 2018.
8, 20
[50] Yao, Z., Gholami, A., Keutzer, K., and Mahoney, M. W. PyHessian: Neural Networks Through the Lens of the Hessian. In IEEE International Conference on Big Data (Big Dataj, 2020. 4, 5, 9, 17, 24, 25
13


Reviews:
[{"review_id": "kDuJBX7B90", "reviewer": "Reviewer_PQdu", "summary": "Authors propose a tool for monitoring and debugging the training process of deep learning models. The tool visualizes/graphs a list of training related mathematical quantities that may improve understanding of the model, its hyperparameters and its training behavior.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors present a collection of tools for visualizing, monitoring and debugging training process of deep neural networks.\n\nI believe that contribution in this area can be very valuable. \n\nHowever, this paper does not really evaluate the toolkit authors propose. The paper does not evaluate different tools in the toolkit and their contribution to the monitoring and debugging process.\n\nIt is possible that Cockpit is a great toolset. However, this cannot be ascertained from the paper.\nThe illustrative examples shown in sections 3.1 and 3.2 provide some motivation to use the tool, but they do not evaluate user experience. \nI realize that tool(kit) evaluation is (very) hard. Running user studies to evaluate the tools takes time and resources. The situation is further complicated when the tool can be used by limited audience of deep learning practitioners. \nHowever, lack of evaluation with actual users makes it harder to know the benefits and drawbacks of the toolkit and its component views.\n\nI have limited experience in training and debugging deep learning models and therefore I cannot evaluate if Cockpit is valuable and which tools in Cockpit are useful.\n\nChanged rating based on the authors' response and discussion.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I think that authors address the limitations of the tool as much as they can without formal user study. The limitations would be more apparent with actual user study.\nI agree with authors that their tool would likely to have positive societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "UWSf6-W6Cd1", "reviewer": "Reviewer_EHzW", "summary": "The paper proposes to visualize 12 different metrics derived from the neural network training process, with the intention to help deep learning practitioners debug issues encountered during neural network training. These metrics are derived from the loss, the parameters, and first/second-order information from the parameters' gradients. The authors demonstrate the effects of incorrectly scaled training data, vanishing gradients due to use of a suboptimal activation function, and various learning rates on these metrics. The authors show the run-time performance of computing the metrics, and conclude with a demonstration of the metrics on a successful training run of All-CNN-C on CIFAR-100.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Originality\nThe idea of visualizing the training progress of a neural network is not new, and tools like TensorBoard allow building dashboards to report various kinds of progress. Many of the metrics used in the work also come from existing work (as cited in Table 1). Nevertheless, the software in this work conveniently allows the users to utilize these metrics in one place. The paper also provides some suggestions for how these disparate metrics might be interpreted together to get a fuller picture.\n\n## Quality\nThe authors show that useful information can be derived from the metrics proposed in the paper. The set of metrics seems reasonably well thought-out and the software package appears complete. However, I found the following lacking about the experiments and the analysis of the work:\n- The neural architectures and datasets used to evaluate the work are all fairly small, consisting of MLPs or CNNs with a handful of layers. The datasets used are CIFAR-10, SVHN, MNIST, and Fashion-MNIST. For these smaller models and datasets, it is already fairly well-understood how to train them effectively. Since they are small, it is also much cheaper to use black-box hyperparameter optimization methods rather than trying to debug issues like a suboptimal learning rate. The work would be improved if it evaluated using larger datasets and more complex architectures and objectives, especially those that are considered more difficult to train like GANs or Transformers.\n- The authors propose that the work can tell the user why neural network training is failing. Indeed, the authors show the effect that incorrectly scaled data and the use of a suboptimal activation function can have on the metrics computed by the provided tool. However, it is not clear how the user should be able to determine the underlying cause from what is shown by the tool.\n  - For incorrectly scaled inputs leading to a different gradient distribution, it is not clear that the changed gradient distribution has to be bad. There are also other potential causes, like incorrectly scaled initial weights.\n  - For vanishing gradients, there are other potential causes like not using batch normalization which could cause the same issue. Also it seems plausible that some patterns for layer-wise histograms may seem intuitively bad but nevertheless turn out to work well in practice.\n- Similarly, the authors suggest using a standardized step size to determine whether the learning rate is too small or too big. However, Figure 5 shows that certain step sizes are good for certain problems but bad for others, making it a difficult measure to use when provided with a new problem.\n\n## Clarity\nThe paper is clearly written and straightforward to understand. It may make more sense to put the Benchmark section after the Showcase section, since the Showcase section references a figure near the start of the paper and also is more related to the Experiments section.\n\n## Significance\nI think the tool itself can be useful for deep learning practitioners, but I don't believe that this paper clearly demonstrated exactly in which ways the tool would be useful.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe the authors have been straightforward about what the work can do, including limitations.\n\nI would suggest to the authors that they gather more experience from a variety of users who can use the Cockpit tool to debug real-world issues with their neural network training. Alternatively, the authors can try to gather common bugs found in neural architecture design and training and reproduce them using the tool, to determine tell-tale signs for these common bugs inside Cockpit. The authors can then automatically provide suggestions for what may be going wrong inside the tool, or produce a guidebook or manual for how to debug using it. \n\nA more formal user study, even small in scale, to show that users can benefit from using the tool, would also be quite helpful.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "NViBuG9MGx9", "reviewer": "Reviewer_6WN2", "summary": "This manuscript describes an open-source tool for providing useful training-related metrics and insights for deep learning models called Cockpit. Unlike current mainstay deep learning training visualization tools such as TensorBoard, Cockpit provides more in-depth metrics about the model's training state, such as a normalized weight update size, gradient norm, and eigenvalues of hessian. The authors provides motivations for computing and visualizing these metrics, making a case for their usefulness in diagnosing problematic model training. In addition, the authors show that these metrics can be computed with relatively low overhead. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n- Cockpit goes beyond traditional deep learning monitoring and visualization tools by providing a way to get a suite of 1st-order (gradient) and 2nd-order (hessian) metrics related to a training neural network's weights. There are tools such as TensorBoard and SageMaker Debugger which provide training-related metrics, but none that I know provides metrics as powerful or as up-to-date as the list shown in Table 1 of the paper. I believe the paradigm of this paper will be a valuable addition to the field of deep learning model tooling. \n\nQuality \n- Overall, this paper is properly structured, well written, and easy to follow. It provides not only the theoretical basis for the metrics included in Cockpit, but also several practical use cases. \n- There is the weakness that the practical use cases (Section 3) cover only a subset of the metrics listed in Table 1 and the subset includes only the more basic metrics such as the gradient histogram and alpha (normalized step size). It would strengthen the paper to provide practical use cases for the more advanced metrics such as HessMaxEv and TICDiag.\n\nSignificance\n- Debugging model training issues is a common problem experienced by ML / neural network researchers and engineers. Having a more powerful tool to give insights to the reasons for problematic model training and their remedies will shorten the research/development cycle and help improve model quality.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations\n1. As stated above, the metrics described in Table 1 and Section 2 includes a wide range of basic and advanced metrics, but Section 3 provides practical use cases only for the more basic metrics. Providing examples of how the more advanced metrics may be helpful to model debugging will strengthen the entire paper significantly. This is also important considering that the more advanced metrics such as HessMaxEv and TICDiag are more expensive to compute and hence require justification for their usefulness.\n2. I would like to see more description / discussion of the extensibility of the tool. Due to the rapid pace of the AI/DL field, it's likely that new metrics / quantities will become useful for model training debugging. Hence it is important for the tool to have an extensible API. Is this already the case for Cockpit? If so, what types of information are accessible via the Cockpit API (in combination with the underlying PyTorch library)? Giving some examples would be good. \n3. The word \"debugging\" has overloaded meanings in the field of deep learning. it sometimes refer to troubleshooting low-level bugs such as incorrect implementation of the underlying mathematical formulae / model structure / loss function; it sometimes refer to understanding \"why the model is not training properly\", which has more to do with hyperparameter choices; it may also refer to understanding why a trained model makes wrong predictions on given examples. This manuscript is mainly about the 2nd type. The authors might want to clarify that. \n4. Figure 2 contains multiple subplots that are too small to read and appreciate. The authors refer to different subplots of Figure 2 in various places of Section 2. Unfortunately the English used to refer to the subplots is hard to follow. I suggest the authors use the words \"middle\" and \"center\" consistently. \"middle\" is with regard to the vertical dimension, \"center\" is with regard to the horizontal dimension. The vertical dimension should always precede the horizontal dimension (e.g., \"bottom left\", \"middle center\", \"middle right\", \"top center\"). In addition, I suggest adding a column to Table 1 called \"location in Figure 2\", so it'll be easier for readers to cross reference Table 1 with the figure. \n5. Page 3, 1st paragraph. It is unclear why the authors contrasts Cockpit with TensorBoard, and they describe the latter as \"just a collection of plots\". The primary GUI of Cockpit, as shown in Figure 2, is indeed a collection of plots, albeit containing both scalar curves, histograms and other types of data. TensorBoard as a GUI can also visualize these different types of graphs (see the Scalars, Histograms, and Distributions dashboard of TensorBoard). TensorBoard/TensorFlow doesn't have out-of-the-box support for the list of quantities shown in Table 1, which is the main distinguishing factor of this tool. The authors might want to clarify that.\n6. I can't find a place in the paper where optimizers other than SGD is talked about. Given other optimizers (e.g., ADAM, RMSProp) are also widely used, I feel the author should briefly talk about how the use of the more complex optimizers may affect the usage of Cockpit.\n6. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "9rbqVjWZFKz", "reviewer": "Reviewer_68Eo", "summary": "The core contribution of Cockpit is a piece of software that provides 12 quantities meant to help practitioners debug issues with their model's training. Some of these quantities are novel (to my knowledge), while many of them are existing metrics taken from the literature (although often with improved computational methods or a slightly different setting).\n\nThey provide explanations for each quantity and how it's meant to be used, as well as some case studies on how these quantities might help a practitioner debug their model.\n\nFinally, they provide benchmarks demonstrating that the computational overhead of computing these quantities is not overly burdensome on the model.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I like this paper a lot. I think it's tackling a very important (and mostly unaddressed!) issue with neural network training - how do we figure out why our model isn't performing well? I would argue that tools like Tensorboard and the metrics they report (e.g. gradient norm) have had a significant impact, and I'd hope that this tool could extend that even further.\n\nIn addition, the paper is fairly convincing that the quantities they've chosen may be useful to practitioners, and this is backed up by both the descriptions of the quantities as well as the case studies in the experiments section. The appendix also makes it clear that determining the computation of these quantities was done carefully (although much of the heavy lifting here seems to be done by the underlying package BackPack).\n\nI'd particularly like to highlight the \"alpha\" quantitity, which is meant to measure whether the gradient is undershooting or overshooting. AFAIK, measuring this quantity in their manner is a novel contribution, and they provide some experiments + intuition for how this quantity can be used to troubleshoot your network's learning rate.\n\nIn addition, I appreciate the polished software package (with documentation + examples), as well as close integration with a popular machine learning framework (PyTorch).\n\nMy primary criticism is that I wish these quantities were explored more thoroughly. Most of the examples are done on relatively toy datasets (CIFAR-10, SVHN, F-MNIST, CIFAR10). However, considering the benchmarks in this paper, it seems that it would be feasible to explore these quantities in a more realistic setting. In particular, the observation that the \"alpha\" quantity is >0 seems quite interesting to me, I would like to see it explored more.\n\nSome nitpicks/minor questions\n- There are these ... random thin black lines around some of the figures, such as near Figure 1. I assume they aren't intentional.\n- I think both the paper/appendix as well as the documentation would benefit from an explanation of how each quantity may be commonly used to debug training.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews for this paper are quasi-unanimous, in the sense that the one reviewer who voted to reject has said they'd be OK with the paper being accepted. 
I do agree with some of the reviewers that experiments on larger models and more case studies would be nice, and part of me wants to recommend that this paper be rejected with the feedback to make those changes and resubmit. 
However, another part of me is worried that, with the reviewing process being as noisy as it is, that might result in this paper being rejected in the future - especially since I do think reviewers / ACs tend to have (unfairly so) a higher bar for these types of contributions.
Given that, I will recommend acceptance. 
I would really like to see a version of this paper with those changes made, however.

Decision: Accept (Poster)