## 2 Materials and Methods
### State-of-the-Art Visualization Tools for Convolutional Neural Networks  
A plethora of state-of-the-art visualization tools for convolutional neural networks exists, both academic and commercial. Some are general purpose applications for all types of data, others are focused solely on image data. Most of the available tools are limited to networks generated with a certain framework like _TensorFlow_[20] or require a proprietary platform like Facebook to use. As an add-on for _TensorFlow, Tensorboard_[21] is an obvious candidate. It provides visualization for machine learning experimentation but focuses on metrics such as loss and accuracy, showing the model graph and histograms of weights, biases, and their evolution over model training time. Users have to incorporate the _TensorBoard_ commands in the _TensorFlow_ code of the actual model training. Although it has a multitude of options, there is no dedicated module for image classification or semantic segmentation visualization. Therefore, an analysis of given models is not possible, making _TensorBoard_ no stand-alone solution. The intention of _Tensorflow Playground_[22] is to illustrate the general workings to beginners by interactively visualizing the weights and flow of a small neural network. _TensorFlow Graph Visualizer_, a _TensorFlow_ component tool visualizing deep learning model architectures and their underlying dataflow graphs, is examined by [23]. It uses graph transformations and decoupling of non-critical nodes on a low-level directed dataflow graph to produce an interactive high-level structure diagram for model explanation. The tool was used on the Inception network for image classification to demonstrate its performance. Although it simplifies the interpretability of deep learning models, it has no visual components for an explicit explanation of image classification and semantic segmentation models. _ShapeShop_, a rather basic tool for general model understanding of both multi-layer perceptrons and convolutional neural networks, is presented by [24]. Only simple basic shapes are implemented and no maps are returned, though, limiting its explanation capability. In a more recent publication [25], the same authors propose an interactive system called _SUMMIT_, that summarizes and visualizes which features a deep learning model has learned and how those features interact to make predictions.  
They propose activation aggregation to identify important neurons and neuron-influence aggregations to analyze the relationships among these neurons. These two techniques are combined to synthesize a novel attribution graph revealing and summarizing crucial neuron associations as well as substructures contributing to a model's predictions. However, they focus on classification and include no visualization options for semantic segmentation. A tool for beginners and students proposed by [26] is _CNN EXPLAINER_, which focuses on explaining the mathematics behind CNNs with visual depictions. No semantic segmentation is integrated. _NeuralDivergence_ is an interactive visualization system developed by [27] which uses activation distributions as a high-level summary of what a model has learned. It enables users to interactively summarize and compare activation distributions across layers and classes, helping them gain a better understanding of neural network models--yet there are no maps, only graphs of activation distributions. Another tool is _ActiVis_[28], an interactive visualization system for interpreting large-scale deep learning models and results. By tightly integrating multiple coordinated views, such as a computation graph overview of the model architecture as well as a neuron activation view for pattern discovery and comparison, users can explore complex deep neural network models at both the instance- and subset-level. _ActiVis_ has been deployed on Facebook's machine learning platform.