Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes to visualize 12 different metrics derived from the neural network training process, with the intention to help deep learning practitioners debug issues encountered during neural network training. These metrics are derived from the loss, the parameters, and first/second-order information from the parameters' gradients. The authors demonstrate the effects of incorrectly scaled training data, vanishing gradients due to use of a suboptimal activation function, and various learning rates on these metrics. The authors show the run-time performance of computing the metrics, and conclude with a demonstration of the metrics on a successful training run of All-CNN-C on CIFAR-100.
main_review: ## Originality
The idea of visualizing the training progress of a neural network is not new, and tools like TensorBoard allow building dashboards to report various kinds of progress. Many of the metrics used in the work also come from existing work (as cited in Table 1). Nevertheless, the software in this work conveniently allows the users to utilize these metrics in one place. The paper also provides some suggestions for how these disparate metrics might be interpreted together to get a fuller picture.

## Quality
The authors show that useful information can be derived from the metrics proposed in the paper. The set of metrics seems reasonably well thought-out and the software package appears complete. However, I found the following lacking about the experiments and the analysis of the work:
- The neural architectures and datasets used to evaluate the work are all fairly small, consisting of MLPs or CNNs with a handful of layers. The datasets used are CIFAR-10, SVHN, MNIST, and Fashion-MNIST. For these smaller models and datasets, it is already fairly well-understood how to train them effectively. Since they are small, it is also much cheaper to use black-box hyperparameter optimization methods rather than trying to debug issues like a suboptimal learning rate. The work would be improved if it evaluated using larger datasets and more complex architectures and objectives, especially those that are considered more difficult to train like GANs or Transformers.
- The authors propose that the work can tell the user why neural network training is failing. Indeed, the authors show the effect that incorrectly scaled data and the use of a suboptimal activation function can have on the metrics computed by the provided tool. However, it is not clear how the user should be able to determine the underlying cause from what is shown by the tool.
  - For incorrectly scaled inputs leading to a different gradient distribution, it is not clear that the changed gradient distribution has to be bad. There are also other potential causes, like incorrectly scaled initial weights.
  - For vanishing gradients, there are other potential causes like not using batch normalization which could cause the same issue. Also it seems plausible that some patterns for layer-wise histograms may seem intuitively bad but nevertheless turn out to work well in practice.
- Similarly, the authors suggest using a standardized step size to determine whether the learning rate is too small or too big. However, Figure 5 shows that certain step sizes are good for certain problems but bad for others, making it a difficult measure to use when provided with a new problem.

## Clarity
The paper is clearly written and straightforward to understand. It may make more sense to put the Benchmark section after the Showcase section, since the Showcase section references a figure near the start of the paper and also is more related to the Experiments section.

## Significance
I think the tool itself can be useful for deep learning practitioners, but I don't believe that this paper clearly demonstrated exactly in which ways the tool would be useful.
limitations_and_societal_impact: I believe the authors have been straightforward about what the work can do, including limitations.

I would suggest to the authors that they gather more experience from a variety of users who can use the Cockpit tool to debug real-world issues with their neural network training. Alternatively, the authors can try to gather common bugs found in neural architecture design and training and reproduce them using the tool, to determine tell-tale signs for these common bugs inside Cockpit. The authors can then automatically provide suggestions for what may be going wrong inside the tool, or produce a guidebook or manual for how to debug using it. 

A more formal user study, even small in scale, to show that users can benefit from using the tool, would also be quite helpful.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The core contribution of Cockpit is a piece of software that provides 12 quantities meant to help practitioners debug issues with their model's training. Some of these quantities are novel (to my knowledge), while many of them are existing metrics taken from the literature (although often with improved computational methods or a slightly different setting).

They provide explanations for each quantity and how it's meant to be used, as well as some case studies on how these quantities might help a practitioner debug their model.

Finally, they provide benchmarks demonstrating that the computational overhead of computing these quantities is not overly burdensome on the model.
main_review: Overall, I like this paper a lot. I think it's tackling a very important (and mostly unaddressed!) issue with neural network training - how do we figure out why our model isn't performing well? I would argue that tools like Tensorboard and the metrics they report (e.g. gradient norm) have had a significant impact, and I'd hope that this tool could extend that even further.

In addition, the paper is fairly convincing that the quantities they've chosen may be useful to practitioners, and this is backed up by both the descriptions of the quantities as well as the case studies in the experiments section. The appendix also makes it clear that determining the computation of these quantities was done carefully (although much of the heavy lifting here seems to be done by the underlying package BackPack).

I'd particularly like to highlight the "alpha" quantitity, which is meant to measure whether the gradient is undershooting or overshooting. AFAIK, measuring this quantity in their manner is a novel contribution, and they provide some experiments + intuition for how this quantity can be used to troubleshoot your network's learning rate.

In addition, I appreciate the polished software package (with documentation + examples), as well as close integration with a popular machine learning framework (PyTorch).

My primary criticism is that I wish these quantities were explored more thoroughly. Most of the examples are done on relatively toy datasets (CIFAR-10, SVHN, F-MNIST, CIFAR10). However, considering the benchmarks in this paper, it seems that it would be feasible to explore these quantities in a more realistic setting. In particular, the observation that the "alpha" quantity is >0 seems quite interesting to me, I would like to see it explored more.

Some nitpicks/minor questions
- There are these ... random thin black lines around some of the figures, such as near Figure 1. I assume they aren't intentional.
- I think both the paper/appendix as well as the documentation would benefit from an explanation of how each quantity may be commonly used to debug training.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Authors propose a tool for monitoring and debugging the training process of deep learning models. The tool visualizes/graphs a list of training related mathematical quantities that may improve understanding of the model, its hyperparameters and its training behavior.
main_review: Authors present a collection of tools for visualizing, monitoring and debugging training process of deep neural networks.

I believe that contribution in this area can be very valuable. 

However, this paper does not really evaluate the toolkit authors propose. The paper does not evaluate different tools in the toolkit and their contribution to the monitoring and debugging process.

It is possible that Cockpit is a great toolset. However, this cannot be ascertained from the paper.
The illustrative examples shown in sections 3.1 and 3.2 provide some motivation to use the tool, but they do not evaluate user experience. 
I realize that tool(kit) evaluation is (very) hard. Running user studies to evaluate the tools takes time and resources. The situation is further complicated when the tool can be used by limited audience of deep learning practitioners. 
However, lack of evaluation with actual users makes it harder to know the benefits and drawbacks of the toolkit and its component views.

I have limited experience in training and debugging deep learning models and therefore I cannot evaluate if Cockpit is valuable and which tools in Cockpit are useful.

Changed rating based on the authors' response and discussion.
limitations_and_societal_impact: I think that authors address the limitations of the tool as much as they can without formal user study. The limitations would be more apparent with actual user study.
I agree with authors that their tool would likely to have positive societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This manuscript describes an open-source tool for providing useful training-related metrics and insights for deep learning models called Cockpit. Unlike current mainstay deep learning training visualization tools such as TensorBoard, Cockpit provides more in-depth metrics about the model's training state, such as a normalized weight update size, gradient norm, and eigenvalues of hessian. The authors provides motivations for computing and visualizing these metrics, making a case for their usefulness in diagnosing problematic model training. In addition, the authors show that these metrics can be computed with relatively low overhead. 
main_review: Originality
- Cockpit goes beyond traditional deep learning monitoring and visualization tools by providing a way to get a suite of 1st-order (gradient) and 2nd-order (hessian) metrics related to a training neural network's weights. There are tools such as TensorBoard and SageMaker Debugger which provide training-related metrics, but none that I know provides metrics as powerful or as up-to-date as the list shown in Table 1 of the paper. I believe the paradigm of this paper will be a valuable addition to the field of deep learning model tooling. 

Quality 
- Overall, this paper is properly structured, well written, and easy to follow. It provides not only the theoretical basis for the metrics included in Cockpit, but also several practical use cases. 
- There is the weakness that the practical use cases (Section 3) cover only a subset of the metrics listed in Table 1 and the subset includes only the more basic metrics such as the gradient histogram and alpha (normalized step size). It would strengthen the paper to provide practical use cases for the more advanced metrics such as HessMaxEv and TICDiag.

Significance
- Debugging model training issues is a common problem experienced by ML / neural network researchers and engineers. Having a more powerful tool to give insights to the reasons for problematic model training and their remedies will shorten the research/development cycle and help improve model quality.
limitations_and_societal_impact: Limitations
1. As stated above, the metrics described in Table 1 and Section 2 includes a wide range of basic and advanced metrics, but Section 3 provides practical use cases only for the more basic metrics. Providing examples of how the more advanced metrics may be helpful to model debugging will strengthen the entire paper significantly. This is also important considering that the more advanced metrics such as HessMaxEv and TICDiag are more expensive to compute and hence require justification for their usefulness.
2. I would like to see more description / discussion of the extensibility of the tool. Due to the rapid pace of the AI/DL field, it's likely that new metrics / quantities will become useful for model training debugging. Hence it is important for the tool to have an extensible API. Is this already the case for Cockpit? If so, what types of information are accessible via the Cockpit API (in combination with the underlying PyTorch library)? Giving some examples would be good. 
3. The word "debugging" has overloaded meanings in the field of deep learning. it sometimes refer to troubleshooting low-level bugs such as incorrect implementation of the underlying mathematical formulae / model structure / loss function; it sometimes refer to understanding "why the model is not training properly", which has more to do with hyperparameter choices; it may also refer to understanding why a trained model makes wrong predictions on given examples. This manuscript is mainly about the 2nd type. The authors might want to clarify that. 
4. Figure 2 contains multiple subplots that are too small to read and appreciate. The authors refer to different subplots of Figure 2 in various places of Section 2. Unfortunately the English used to refer to the subplots is hard to follow. I suggest the authors use the words "middle" and "center" consistently. "middle" is with regard to the vertical dimension, "center" is with regard to the horizontal dimension. The vertical dimension should always precede the horizontal dimension (e.g., "bottom left", "middle center", "middle right", "top center"). In addition, I suggest adding a column to Table 1 called "location in Figure 2", so it'll be easier for readers to cross reference Table 1 with the figure. 
5. Page 3, 1st paragraph. It is unclear why the authors contrasts Cockpit with TensorBoard, and they describe the latter as "just a collection of plots". The primary GUI of Cockpit, as shown in Figure 2, is indeed a collection of plots, albeit containing both scalar curves, histograms and other types of data. TensorBoard as a GUI can also visualize these different types of graphs (see the Scalars, Histograms, and Distributions dashboard of TensorBoard). TensorBoard/TensorFlow doesn't have out-of-the-box support for the list of quantities shown in Table 1, which is the main distinguishing factor of this tool. The authors might want to clarify that.
6. I can't find a place in the paper where optimizers other than SGD is talked about. Given other optimizers (e.g., ADAM, RMSProp) are also widely used, I feel the author should briefly talk about how the use of the more complex optimizers may affect the usage of Cockpit.
6.
review_title: null

Meta-review: The reviews for this paper are quasi-unanimous, in the sense that the one reviewer who voted to reject has said they'd be OK with the paper being accepted. 
I do agree with some of the reviewers that experiments on larger models and more case studies would be nice, and part of me wants to recommend that this paper be rejected with the feedback to make those changes and resubmit. 
However, another part of me is worried that, with the reviewing process being as noisy as it is, that might result in this paper being rejected in the future - especially since I do think reviewers / ACs tend to have (unfairly so) a higher bar for these types of contributions.
Given that, I will recommend acceptance. 
I would really like to see a version of this paper with those changes made, however.

Decision: Accept (Poster)