Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks

1 Introduction and motivation . Deep learning represents a new programming paradigm : instead of deterministic programs , users design models and “ simply ” train them with data . In this metaphor , deep learning is a metaprogramming form , where coding is replaced by training . Here , we ponder the question how we can provide more insight into this process by building a debugger specifically designed for deep learning . Debuggers are crucial for traditional software development . When things fail , they provide access to the internal workings of the code , allowing a look “ into the box ” . This is much more efficient than re-running the program with different inputs . And yet , deep learning is arguably closer to the latter . If the attempt to train a deep net fails , a machine learning engineer faces various options : Should they change the training hyperparameters ( how ? ) ; the optimizer ( to which one ? ) ; the model ( how ? ) ; or just re-run with a different seed ? Machine learning toolboxes provide scant help to guide these decisions . Of course , traditional debuggers can be applied to deep learning . They will give access to every single weight of a neural net , or the individual pixels of its training data . But this rarely yields insights ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . towards successful training . Extracting meaningful information requires a statistical approach and distillation of the bewildering complexity into a manageable summary . Tools like TENSORBOARD [ 1 ] or WEIGHTS & BIASES [ 6 ] were built in part to streamline this visualization . Yet , the quantities that are widely monitored ( mainly train/test loss & accuracy ) , provide only scant explanation for relative differences between multiple training runs , because they do not show the network ’ s internal state . Figure 1 illustrates how such established learning curves can describe the current state of the model – whether it is performing well or not – while failing to inform about training state and dynamics . They tell the user that things are going well or badly , but not why . The situation is similar to flying a plane by sight , without instruments to provide feedback . It is not surprising , then , that achieving state-of-the-art performance in deep learning requires expert intuition , or plain trial & error . We aim to enrich the deep learning pipeline with a visual and statistical debugging tool that uses newly proposed observables as well as several established ones ( Section 2 ) . We leverage and augment recent extensions to automatic differentiation ( i.e . BACKPACK [ 12 ] for PYTORCH [ 33 ] ) to efficiently access second-order statistical ( e.g . gradient variances ) and geometric ( e.g . Hessian ) information . We show how these quantities can aid the deep learning engineer in tasks , like learning rate selection , as well as detecting common bugs with data processing or model architectures ( Section 3 ) . Concretely , we introduce COCKPIT , a flexible and efficient framework for online-monitoring these observables during training in carefully designed plots we call “ instruments ” ( see Figure 2 ) . To be of practical use , such visualization must have a manageable computational overhead . We show that COCKPIT scales well to real-world deep learning problems ( see Figure 2 and Section 4 ) . We also provide three different configurations of varying computational complexity and demonstrate that their instruments keep the computational cost well below a factor of 2 in run time ( Section 5 ) . It is available as open-source code,2 extendable , and seamlessly integrates into existing PYTORCH training loops ( see Appendix A ) . 2 COCKPIT ’ s instruments . Setting : We consider supervised regression/classification with labeled data ( x , y ) ∈ X× Y generated by a distribution P ( x , y ) . The training set D = { ( xn , yn ) | n = 1 , . . . , N } consists of N i.i.d . samples from P and the deep model f : Θ×X→ Y maps inputs xn to predictions ŷn by parameters θ ∈ RD . This prediction is evaluated by a loss function ℓ : Y× Y→ R which compares to the label yn . The goal is minimizing an inaccessible expected risk LP ( θ ) = ∫ ℓ ( f ( θ , x ) , y ) dP ( x , y ) by empirical approximation through LD ( θ ) = 1N ∑N n=1 ℓ ( f ( θ , xn ) , yn ) : = 1 N ∑N n=1 ℓn ( θ ) , which 2https : //github.com/f-dangel/cockpit in practice though can only be stochastically sub-sampled on mini-batches B ⊆ { 1 , . . . , N } , LB ( θ ) = 1 |B| ∑ n∈B ℓn ( θ ) . ( 1 ) As is standard practice , we use first- and second-order information of the mini-batch loss , described by its gradient gB ( θ ) and Hessian HB ( θ ) , gB ( θ ) = 1 |B| ∑ n∈B ∇θℓn ( θ ) ︸ ︷︷ ︸ gn ( θ ) , HB ( θ ) = 1 |B| ∑ n∈B ∇2θℓn ( θ ) . ( 2 ) Design choices : To minimize computational and design overhead , we restrict the metrics to quantities that require no additional model evaluations . This means that , at training step t→ t+ 1 with mini-batches Bt , Bt+1 and parameters θt , θt+1 , we may access information about the mini-batch losses LBt ( θt ) and LBt+1 ( θt+1 ) , but no cross-terms that would require additional forward passes . Key point : LB ( θ ) , gB ( θ ) , and HB ( θ ) are just expected values of a distribution over the batch . Only recently , this distribution has begun to attract attention [ 15 ] as its computation has become more accessible [ 8 ; 12 ] . Contemporary optimizers leverage only the mean gradient and neglect higher moments . One core point of our work is making extensive use of these distribution properties , trying to visualize them in various ways . This out-of-the-box support for the carefully selected and efficiently computed quantities distinguishes COCKPIT from tools like TENSORBOARD that offer visualizations as well . Leveraging these distributional quantities , we create instruments and show how they can help adapt hyperparameters ( Section 2.1 ) , analyze the loss landscape ( Section 2.2 ) , and track network dynamics ( Section 2.3 ) . Instruments can sometimes be built from already-computed information or are efficient variants of previously proposed observables . To keep the presentation concise , we highlight the instruments shown in Figure 2 and listed in Table 1 . Appendix C defines them formally and contains more extensions , such as the mean GSNR [ 27 ] , the early stopping [ 29 ] and CABS [ 4 ] criterion , which can all be used in COCKPIT . Bug types : We distinguish three types of bugs encountered in deep learning . Implementation bugs are low-level software bugs that , for example , trigger syntax errors . Training bugs result in unnecessarily inefficient or even unsuccessful training . They can , for example , stem from erroneous data handling ( see Section 3.1 ) , the chosen model architecture ( see Section 3.2 ) , or ill-chosen hyperparameters ( see Section 3.3 ) . Prediction bugs describe incorrect predictions of a trained model on specific examples . Traditional debuggers are well-suited to find implementation bugs . COCKPIT focuses on efficiently identifying training bugs instead . 2.1 Adapting hyperparameters . One big challenge in deep learning is setting the hyperparameters correctly , which is currently mostly done by trial & error through parameter searches . We aim to augment this process with instruments that inform the user about the effect that the chosen parameters have on the current training process . Alpha : Are we crossing the valley ? Using individual loss and gradient observations at the start and end point of each iteration , we build a noise-informed univariate quadratic approximation along the step direction ( i.e . the loss as a function of the step size ) , and assess to which point on this parabola our optimizer moves . We standardize this value α such that stepping to the valley-floor is assigned α = 0 , the starting point is at α = −1 and updates to the point exactly opposite of the starting point have α = 1 ( see Appendix C.2 for a more detailed visual and mathematical description of α ) . Figure 1 illustrates the scenarios α = ±1 and how monitoring the α-distribution ( right panel ) can help distinguish between two training runs with similar performance but distinct failure sources . By default , this COCKPIT instrument shows the α-distribution for the last 10 % of training and the entire training process ( e.g . top left plot in Figure 2 ) . In Section 3.3 we demonstrate empirically that , counter-intuitively , it is generally not a good idea to choose the step size such that α is close to zero . Distances : Are we making progress ? Another way to discern the trajectories in Figure 1 is by measuring the L2 distance from initialization [ 31 ] and the update size [ 2 ; 16 ] in parameter space . Both are shown together in one COCKPIT instrument ( see also middle left plot in Figure 2 ) and are far larger for the blue line in Figure 1 . These distance metrics are also able to disentangle phases for the blue path . Using the same step size , it will continue to “ jump back and forth ” between the loss valley ’ s walls but at some point cease to make progress . During this “ surfing of the walls ” , the distance from initialization increases , ultimately though , it will stagnate , with the update size remaining non-zero , indicating diffusion . While the initial “ surfing the wall ” -phase benefits training ( see Section 3.3 ) , achieving stationarity may require adaptation once the optimizer reaches that diffusion . Gradient norm : How steep is the wall ? The update size will show that the orange trajectory is stuck . But why ? Such slow-down can result from both a bad learning rate and from loss landscape plateaus . The gradient norm ( bottom left panel in Figure 2 ) distinguishes these two causes . Gradient tests : How noisy is the batch ? The batch size trades off gradient accuracy versus computational cost . Recently , adaptive sampling strategies based on testing geometric constraints between mean and individual gradients have been proposed [ 9 ; 7 ] . The norm , inner product , and orthogonality tests use a standardized radius and two band widths ( parallel and orthogonal to the gradient mean ) that indicate how strongly individual gradients scatter around the mean . The original works use these values to adapt batch sizes . Instead , COCKPIT combines all three tests into a single gauge ( top center plot of Figure 2 ) using the standardized noise radius and band widths for visualization . These noise signals can be used to guide batch size adaptation on- and offline , or to probe the influence of gradient alignment on training speed [ 37 ] and generalization [ 10 ; 11 ; 27 ] .

Summary: This manuscript describes an open-source tool for providing useful training-related metrics and insights for deep learning models called Cockpit. Unlike current mainstay deep learning training visualization tools such as TensorBoard, Cockpit provides more in-depth metrics about the model's training state, such as a normalized weight update size, gradient norm, and eigenvalues of hessian. The authors provides motivations for computing and visualizing these metrics, making a case for their usefulness in diagnosing problematic model training. In addition, the authors show that these metrics can be computed with relatively low overhead.