Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks

1 Introduction and motivation . Deep learning represents a new programming paradigm : instead of deterministic programs , users design models and “ simply ” train them with data . In this metaphor , deep learning is a metaprogramming form , where coding is replaced by training . Here , we ponder the question how we can provide more insight into this process by building a debugger specifically designed for deep learning . Debuggers are crucial for traditional software development . When things fail , they provide access to the internal workings of the code , allowing a look “ into the box ” . This is much more efficient than re-running the program with different inputs . And yet , deep learning is arguably closer to the latter . If the attempt to train a deep net fails , a machine learning engineer faces various options : Should they change the training hyperparameters ( how ? ) ; the optimizer ( to which one ? ) ; the model ( how ? ) ; or just re-run with a different seed ? Machine learning toolboxes provide scant help to guide these decisions . Of course , traditional debuggers can be applied to deep learning . They will give access to every single weight of a neural net , or the individual pixels of its training data . But this rarely yields insights ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . towards successful training . Extracting meaningful information requires a statistical approach and distillation of the bewildering complexity into a manageable summary . Tools like TENSORBOARD [ 1 ] or WEIGHTS & BIASES [ 6 ] were built in part to streamline this visualization . Yet , the quantities that are widely monitored ( mainly train/test loss & accuracy ) , provide only scant explanation for relative differences between multiple training runs , because they do not show the network ’ s internal state . Figure 1 illustrates how such established learning curves can describe the current state of the model – whether it is performing well or not – while failing to inform about training state and dynamics . They tell the user that things are going well or badly , but not why . The situation is similar to flying a plane by sight , without instruments to provide feedback . It is not surprising , then , that achieving state-of-the-art performance in deep learning requires expert intuition , or plain trial & error . We aim to enrich the deep learning pipeline with a visual and statistical debugging tool that uses newly proposed observables as well as several established ones ( Section 2 ) . We leverage and augment recent extensions to automatic differentiation ( i.e . BACKPACK [ 12 ] for PYTORCH [ 33 ] ) to efficiently access second-order statistical ( e.g . gradient variances ) and geometric ( e.g . Hessian ) information . We show how these quantities can aid the deep learning engineer in tasks , like learning rate selection , as well as detecting common bugs with data processing or model architectures ( Section 3 ) . Concretely , we introduce COCKPIT , a flexible and efficient framework for online-monitoring these observables during training in carefully designed plots we call “ instruments ” ( see Figure 2 ) . To be of practical use , such visualization must have a manageable computational overhead . We show that COCKPIT scales well to real-world deep learning problems ( see Figure 2 and Section 4 ) . We also provide three different configurations of varying computational complexity and demonstrate that their instruments keep the computational cost well below a factor of 2 in run time ( Section 5 ) . It is available as open-source code,2 extendable , and seamlessly integrates into existing PYTORCH training loops ( see Appendix A ) . 2 COCKPIT ’ s instruments . Setting : We consider supervised regression/classification with labeled data ( x , y ) ∈ X× Y generated by a distribution P ( x , y ) . The training set D = { ( xn , yn ) | n = 1 , . . . , N } consists of N i.i.d . samples from P and the deep model f : Θ×X→ Y maps inputs xn to predictions ŷn by parameters θ ∈ RD . This prediction is evaluated by a loss function ℓ : Y× Y→ R which compares to the label yn . The goal is minimizing an inaccessible expected risk LP ( θ ) = ∫ ℓ ( f ( θ , x ) , y ) dP ( x , y ) by empirical approximation through LD ( θ ) = 1N ∑N n=1 ℓ ( f ( θ , xn ) , yn ) : = 1 N ∑N n=1 ℓn ( θ ) , which 2https : //github.com/f-dangel/cockpit in practice though can only be stochastically sub-sampled on mini-batches B ⊆ { 1 , . . . , N } , LB ( θ ) = 1 |B| ∑ n∈B ℓn ( θ ) . ( 1 ) As is standard practice , we use first- and second-order information of the mini-batch loss , described by its gradient gB ( θ ) and Hessian HB ( θ ) , gB ( θ ) = 1 |B| ∑ n∈B ∇θℓn ( θ ) ︸ ︷︷ ︸ gn ( θ ) , HB ( θ ) = 1 |B| ∑ n∈B ∇2θℓn ( θ ) . ( 2 ) Design choices : To minimize computational and design overhead , we restrict the metrics to quantities that require no additional model evaluations . This means that , at training step t→ t+ 1 with mini-batches Bt , Bt+1 and parameters θt , θt+1 , we may access information about the mini-batch losses LBt ( θt ) and LBt+1 ( θt+1 ) , but no cross-terms that would require additional forward passes . Key point : LB ( θ ) , gB ( θ ) , and HB ( θ ) are just expected values of a distribution over the batch . Only recently , this distribution has begun to attract attention [ 15 ] as its computation has become more accessible [ 8 ; 12 ] . Contemporary optimizers leverage only the mean gradient and neglect higher moments . One core point of our work is making extensive use of these distribution properties , trying to visualize them in various ways . This out-of-the-box support for the carefully selected and efficiently computed quantities distinguishes COCKPIT from tools like TENSORBOARD that offer visualizations as well . Leveraging these distributional quantities , we create instruments and show how they can help adapt hyperparameters ( Section 2.1 ) , analyze the loss landscape ( Section 2.2 ) , and track network dynamics ( Section 2.3 ) . Instruments can sometimes be built from already-computed information or are efficient variants of previously proposed observables . To keep the presentation concise , we highlight the instruments shown in Figure 2 and listed in Table 1 . Appendix C defines them formally and contains more extensions , such as the mean GSNR [ 27 ] , the early stopping [ 29 ] and CABS [ 4 ] criterion , which can all be used in COCKPIT . Bug types : We distinguish three types of bugs encountered in deep learning . Implementation bugs are low-level software bugs that , for example , trigger syntax errors . Training bugs result in unnecessarily inefficient or even unsuccessful training . They can , for example , stem from erroneous data handling ( see Section 3.1 ) , the chosen model architecture ( see Section 3.2 ) , or ill-chosen hyperparameters ( see Section 3.3 ) . Prediction bugs describe incorrect predictions of a trained model on specific examples . Traditional debuggers are well-suited to find implementation bugs . COCKPIT focuses on efficiently identifying training bugs instead . 2.1 Adapting hyperparameters . One big challenge in deep learning is setting the hyperparameters correctly , which is currently mostly done by trial & error through parameter searches . We aim to augment this process with instruments that inform the user about the effect that the chosen parameters have on the current training process . Alpha : Are we crossing the valley ? Using individual loss and gradient observations at the start and end point of each iteration , we build a noise-informed univariate quadratic approximation along the step direction ( i.e . the loss as a function of the step size ) , and assess to which point on this parabola our optimizer moves . We standardize this value α such that stepping to the valley-floor is assigned α = 0 , the starting point is at α = −1 and updates to the point exactly opposite of the starting point have α = 1 ( see Appendix C.2 for a more detailed visual and mathematical description of α ) . Figure 1 illustrates the scenarios α = ±1 and how monitoring the α-distribution ( right panel ) can help distinguish between two training runs with similar performance but distinct failure sources . By default , this COCKPIT instrument shows the α-distribution for the last 10 % of training and the entire training process ( e.g . top left plot in Figure 2 ) . In Section 3.3 we demonstrate empirically that , counter-intuitively , it is generally not a good idea to choose the step size such that α is close to zero . Distances : Are we making progress ? Another way to discern the trajectories in Figure 1 is by measuring the L2 distance from initialization [ 31 ] and the update size [ 2 ; 16 ] in parameter space . Both are shown together in one COCKPIT instrument ( see also middle left plot in Figure 2 ) and are far larger for the blue line in Figure 1 . These distance metrics are also able to disentangle phases for the blue path . Using the same step size , it will continue to “ jump back and forth ” between the loss valley ’ s walls but at some point cease to make progress . During this “ surfing of the walls ” , the distance from initialization increases , ultimately though , it will stagnate , with the update size remaining non-zero , indicating diffusion . While the initial “ surfing the wall ” -phase benefits training ( see Section 3.3 ) , achieving stationarity may require adaptation once the optimizer reaches that diffusion . Gradient norm : How steep is the wall ? The update size will show that the orange trajectory is stuck . But why ? Such slow-down can result from both a bad learning rate and from loss landscape plateaus . The gradient norm ( bottom left panel in Figure 2 ) distinguishes these two causes . Gradient tests : How noisy is the batch ? The batch size trades off gradient accuracy versus computational cost . Recently , adaptive sampling strategies based on testing geometric constraints between mean and individual gradients have been proposed [ 9 ; 7 ] . The norm , inner product , and orthogonality tests use a standardized radius and two band widths ( parallel and orthogonal to the gradient mean ) that indicate how strongly individual gradients scatter around the mean . The original works use these values to adapt batch sizes . Instead , COCKPIT combines all three tests into a single gauge ( top center plot of Figure 2 ) using the standardized noise radius and band widths for visualization . These noise signals can be used to guide batch size adaptation on- and offline , or to probe the influence of gradient alignment on training speed [ 37 ] and generalization [ 10 ; 11 ; 27 ] .

Summary: The core contribution of Cockpit is a piece of software that provides 12 quantities meant to help practitioners debug issues with their model's training. Some of these quantities are novel (to my knowledge), while many of them are existing metrics taken from the literature (although often with improved computational methods or a slightly different setting).  They provide explanations for each quantity and how it's meant to be used, as well as some case studies on how these quantities might help a practitioner debug their model.  Finally, they provide benchmarks demonstrating that the computational overhead of computing these quantities is not overly burdensome on the model.