0 DeepTracker : Visualizing the Training Process of Convolutional Neural Networks

ing with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2018 Association for Computing Machinery. 2157-6904/0/0-ART0 $0 https://doi.org/10.1145/nnnnnnn.nnnnnnn ACM Transactions on Intelligent Systems and Technology, Vol. 0, No. 0, Article 0. Publication date: 0. ar X iv :1 80 8. 08 53 1v 1 [ cs . C V ] 2 6 A ug 2 01 8 0:2 Dongyu Liu, Weiwei Cui, Kai Jin, Yuxiao Guo, and HuaminQu may take a couple of weeks. However, domain experts often have to repeat this process several times with slightly different settings to obtain a satisfying network, which may take several weeks or even months. To accelerate this process, experts have to understand the training processes further to check whether they are on the right track, find latent mistakes, and make proper adjustments in the next trial. Visualizing the concealed rich training dynamics (e.g., the changes of loss/accuracy and weights/gradients/activations over time) is of vital importance to the understanding of CNN training process. Unfortunately, CNNs usually contain a large number of interacting and non-linear parts [7] and recently become wider and deeper [17, 35, 37, 38]. Both of these bring considerable difficulties for experts in reasoning about CNN training behaviors. Many previous studies investigate what features have been learned by a CNN in one (e.g., usually the last one) or several representative snapshots during the training process [4, 11, 12, 21, 27, 30– 32, 36, 41, 43]. However, little research focuses on visualizing the overall training dynamics. One recent work [26] visualizes the training process of deep neural networks, but it is not tailored for CNNs and not scalable enough to analyze the CNNs that are not only wide but also deep. Besides, tools like TensorBoard, Nvidia Digits, and Deeplearning4j1 are able to show some high-level training dynamics, such as loss and the mean of weights in a layer. However, these tools can neither handle industry-level training (i.e., training a large CNN on a very large dataset) nor answer complex questions. For example, how the model’s performance on each class of images changes over time? How the changes of parameters impact the classification results for each class? Given so many layers or image classes, which of them are worth paying more attention to and what is the best manner to support comparative analysis? With these concerns, we are in urgent need of a scalable visualization solution to conducting more advanced analytical tasks. To this end, we have to deal with two major challenges. First, the system needs to handle the large-scale training log data. Typically, millions of CNN parameters and tens of thousands of validation images are involved in a training process. In addition, the training is an iterationbased process that usually requires a million iterations to complete, which makes things worse, because the parameters and classification results need to be recorded for every few iterations. In our experiments (Sec. 6), a sampled training log may easily exceed a couple of terabytes per training. To allow an interactive exploration of the data at such scale, it requires not only an effective data storage and index mechanism but also a scalable visualization technique. Second, the log information is heterogeneous. The full log contains structure (e.g., neural network), numeric (e.g., neuron weights), image (e.g., validation dataset), and nominal data (e.g., classification results). Given that significant insights are often hidden underneath the complex relationships among these data, our system also needs to present all these types of data intuitively and assist experts in their analysis tasks. To address these challenges, we use a downsampling method to store the raw data, and then preprocess and organize these data in a hierarchical manner. We also design several efficient index mechanisms to support real-time interactions. To help experts identify the iterations of interest quickly, we propose an application-specific anomaly detection method. We also integrate many filtering and aggregation approaches to reduce the amount of presenting information and ensure preserving the noteworthy information. For visualization, we design a set of hierarchical small multiples that is combined with a network structure layout to facilitate an effective exploration of the entire training log from different levels of detail. To reveal the complex correlations among neuron weights, validation images, and training iterations, we further introduce a cube-style 1TensorBoard: https://www.tensorflow. org/get_started/summaries_and_tensorboard; Nvidia Digits: https://developer.nvidia. com/digits; Deeplearning4j: https://deeplearning4j.org/visualization ACM Transactions on Intelligent Systems and Technology, Vol. 0, No. 0, Article 0. Publication date: 0. DeepTracker: Visualizing the Training Process of Convolutional Neural Networks 0:3 visualization. The cube integrates the small multiples and a matrix-based correlation view, thereby allowing experts to effectively slice and dice the data from different perspectives. The main contributions of this paper are summarized as follows: • A systematic characterization of the problem of visualizing the rich dynamics in CNN training processes, and a thorough discussion and summary of the design requirements and space. • A visual analytics system that integrates a tailored large data storage and index mechanism, an anomaly iteration detection algorithm, and a set of well-designed visualization techniques. • A couple of new visualization and interaction techniques including hierarchical small multiples, grid-based correlation view, and cube-style visualization.